dynamic library function cast

Question

A newbie question about shared library: In C, when loading a dynamic library, we use dlopen and then dlsym to find a symbol or a function. Now let say, the function we look for in the dll is typed as:

int add(int a, int b);

but if we cast it to another type, say,

typedef int (*sum)(int a, int b, int c);

what will happen? Will C runtime complain about it?

Thanks!

Answer 1

Long story short, this is undefined behaviour. Calling a function with the wrong number of parameters in C (which you can do by casting function pointers even without dlopen/dlfree) produces undefined behaviour.

For callee-clean calling conventions, like stdcall, using the wrong number or type of arguments will produce a stack imbalance (the callee adjusts the stack to the wrong place). A stack imbalance will quickly kill the program (if you're lucky) by corrupting the return address and local variables of the caller.

For caller-clean calling conventions, the effect is much like calling e.g. printf with the wrong number of arguments: the function may behave erratically by using garbage arguments, but your program might not blow up. (This in general is not desirable; an attacker could exploit this to take control of your program, for example).