Usage of dlsym()/dlopen()

Question

I wrote next program:

#include <iostream>
#include <dlfcn.h>

int main(int argc, char** argv)
{
    typedef void* (*fptr)();
    fptr func;
    void *handle = dlopen(0, RTLD_NOW);
    std::cout << dlerror() << std::endl;    
    *(void **)(&func) = dlsym(handle, "__libc_start_main");
    std::cout << dlerror() << std::endl;

    std::cout << handle << " " << func << "\n";

    dlclose(handle);
return 0;
}

and try to compile in next way:

g++ -rdynamic main.cpp -ldl -o test

When I run this program I don’t see any message. Why?

Thank U for attention.

Answer 1

Your process is faulting because dlerror() is only valid to call in an error condition, which you never validated actually happened prior to invocation.

From the Linux docs:

The function dlerror() returns a human readable string describing the most recent error that occurred from dlopen(), dlsym() or dlclose() since the last call to dlerror(). It returns NULL if no errors have occurred since initialization or since it was last called.

In other words, your dlopen succeeded, so NULL is returned from dlerror(). That NULL is then sent as a char * to std::cout, and kerboom.

Bottom line: check your error conditions before invoking dlerror(). Try this instead:

#include <iostream>
#include <dlfcn.h>

int main(int argc, char** argv)
{
    typedef void* (*fptr)();
    fptr func;

    void *handle = dlopen(0, RTLD_NOW);
    if (handle == nullptr)
    {
        std::cout << dlerror() << std::endl;
        exit(EXIT_FAILURE);
    }

    func = (fptr)dlsym(handle, "__libc_start_main");
    if (!func)
    {
        std::cout << dlerror() << std::endl;
        exit(EXIT_FAILURE);
    }

    std::cout << handle << " " << func << "\n";

    dlclose(handle);
    return 0;
}

Answer 2

This is (likely) undefined behavior:

std::cout << dlerror() << std::endl; 

...unless dlerror is actually non-null (which it probably isn't). It should be:

char* error = dlerror();
if (error != 0)
    std::cout << error << std::endl;

...or:

void *handle = dlopen(0, RTLD_NOW); 
if (handle == 0)
    std::cout << dlerror() << std::endl;

Also, you probably should abort if dlerror is non-null (or handle is null), because calling dlsym on an null handle is also undefined behavior.

See Why does std::cout output disappear completely after NULL is sent to it.