Dynamic Loading (dlopen)

The dynamic linker resolves shared libraries at load time — before main() runs. But sometimes you need to load code at runtime: plugin systems, optional features, and language extension modules all need to discover and load .so files on the fly. POSIX provides the dlfcn.h API for this.

The API

#include <dlfcn.h>
 
// Load a shared library into the process
void *handle = dlopen("libplugin.so", RTLD_NOW | RTLD_LOCAL);
 
// Look up a symbol (function or data) by name
void (*func)(int) = dlsym(handle, "plugin_init");
 
// Call it
func(42);
 
// Unload when done
dlclose(handle);
 
// Get human-readable error string
const char *err = dlerror();

dlopen() returns an opaque handle. dlsym() searches that library’s symbol table for a name and returns its address. The caller casts the void* to the right type — there’s no type safety. This is FFI at its most fundamental.

RTLD Flags: Controlling Symbol Visibility

The second argument to dlopen() controls two orthogonal things: when symbols resolve and where they’re visible.

When: lazy vs immediate

Flag	Behavior
RTLD_LAZY	Resolve symbols on first use (via PLT stubs). Faster load, fails later.
RTLD_NOW	Resolve all symbols immediately at dlopen() time. Slower load, fails fast.

Where: local vs global

This is where things get interesting — and where Python extension modules care deeply.

Flag	Behavior
RTLD_LOCAL	Symbols are private to this library. Other dlopen()’d libraries can’t see them. (default)
RTLD_GLOBAL	Symbols are added to the global scope. Other libraries (including future dlopen() calls) can resolve against them.

RTLD_GLOBAL effectively merges the library’s symbols into the process-wide namespace. This is powerful but dangerous — name collisions between independently-loaded libraries become silent corruption.

How Python Uses dlopen()

When you write import my_module and my_module is a C/Rust extension (a .so file), CPython loads it via dlopen():

// Simplified from CPython's dynload_shlib.c
handle = dlopen("my_module.cpython-311-x86_64-linux-gnu.so", RTLD_NOW | RTLD_LOCAL);
init_func = dlsym(handle, "PyInit_my_module");
init_func();  // calls the module's init function

Two critical details:

1. RTLD_LOCAL — each extension’s symbols are private. Two extensions can internally use a function called helper() without colliding.

2. The interpreter itself is loaded with RTLD_GLOBAL — Python symbols (PyObject_New, PyErr_SetString, etc.) are in the global scope. Extensions can call Python API functions without explicitly linking libpython.

This creates an asymmetry: extensions can use Python symbols (because the interpreter exported them globally), but extensions can’t see each other’s symbols (because each is loaded locally).

Why this matters for PyO3

PyO3 extensions are Rust shared libraries that call Python C API functions. On Linux, those symbols come from the interpreter process itself (loaded globally). The extension does not need to link libpython.so — and in fact must not, because:

• The interpreter already has these symbols loaded
• Linking libpython.so into the extension would create duplicate symbols — two copies of PyObject_New, two copies of PyGILState_Ensure, etc.
• The dynamic linker picks one arbitrarily, but internal state (like the GIL) isn’t shared between copies
• Result: crashes, deadlocks, or silent corruption

The extension-module Cargo feature in PyO3 tells it to skip linking libpython, relying on the interpreter’s globally-exported symbols instead. See the Bazel PyO3 note for how this interacts with the build system.

Platform differences

Platform	Extension gets Python symbols from	Build requirement
Linux	Interpreter process (RTLD_GLOBAL)	Do NOT link libpython (extension-module feature)
macOS	Same mechanism, but needs -undefined dynamic_lookup linker flag	Tell linker to accept undefined symbols at link time
Windows	python3.lib import library	Must link against python3.lib (no RTLD_GLOBAL equivalent)

dlopen() vs Load-Time Linking

	Load-time (ld.so)	Runtime (dlopen)
When	Before main()	Any time during execution
Declared in	ELF DT_NEEDED entries	Source code
Symbol resolution	GOT	dlsym() by name string
Failure mode	Process won’t start	dlopen() returns NULL
Use case	Known dependencies	Plugins, optional features, language extensions

Both ultimately use the same kernel mechanism — mmap() to map the .so into the process address space with appropriate page permissions — but the control flow and error handling differ significantly.