Long story short, this is undefined behaviour. Calling a function with the wrong number of parameters in C (which you can do by casting function pointers even without dlopen
/dlfree
) produces undefined behaviour.
For callee-clean calling conventions, like stdcall
, using the wrong number or type of arguments will produce a stack imbalance (the callee adjusts the stack to the wrong place). A stack imbalance will quickly kill the program (if you're lucky) by corrupting the return address and local variables of the caller.
For caller-clean calling conventions, the effect is much like calling e.g. printf
with the wrong number of arguments: the function may behave erratically by using garbage arguments, but your program might not blow up. (This in general is not desirable; an attacker could exploit this to take control of your program, for example).