Type casts in C and type safety

Question

In chapter 6.2.1 of Mitchell's book (Concepts in Programming Languages), it mentioned that:

Type Casts. Type casts allow a value of one type to be used as another type. In C in particular, an integer can be cast to a function, allowing a jump to a location that does not contain the correct form of instructions to be a C function.

so I was up to use this non-safety and doing something unusual I tried something like this (pseudo-code):

int x = 0;
print "loop";
x();

to create an infinite loop. I tried by changing and testing, but I couldn't cope with that. How can I do something or everything else like these stuffs?

Thanks in advance

Answer 1

@Soroush, here's an example that might help you better understand what's going on behind the scenes:

#include <stdio.h>

int main(void)
{
    printf("begin\n");
    printf("loop\n");

    // declare a function pointer
    int (*loopPtr)();
    // set the function pointer to the current function
    loopPtr = main;
    // skip over the first printf();
    loopPtr += 22;
    // call the new location
    loopPtr();
}

For me, it works on x86_64 when compiled with clang -O0 (well, it works until the stack is exhausted since this is infinite recursion and each function call chews through stack space).

I determined the offset 22 by compiling, then disassembling and subtracting the address of the start of main() from the address of the second printf().

First, I compiled it:

clang -O0 test.c

Then disassembled it:

otool -tv a.out

...which produced this output:

[...]
_main:
0000000100000ee0    pushq   %rbp
0000000100000ee1    movq    %rsp,%rbp
0000000100000ee4    subq    $0x20,%rsp
0000000100000ee8    leaq    0x00000073(%rip),%rdi
0000000100000eef    movb    $0x00,%al
0000000100000ef1    callq   0x100000f40
0000000100000ef6    leaq    0x0000006c(%rip),%rdi
0000000100000efd    movl    %eax,0xf4(%rbp)
0000000100000f00    movb    $0x00,%al
0000000100000f02    callq   0x100000f40
[...]

_main: indicates the entrypoint of the main() function, whose first address is 0x100000ee0. The first callq instruction corresponds with the first printf() call, which I want to skip, so I chose the address just after that: 0x100000ef6. 0x100000ef6 minus 0x100000ee0 is 22 decimal.

Answer 2

That's not how it works.

In C you can cast an integer value to a function pointer value and then call such a function pointer

void (*ptr)() = (void (*)())42;
ptr();

but this will probably just result in a crash unless you know exactly what you are doing, i.e. you already know in some way that, at address 42, starts a function with that signature; having a fixed address like in this example may happen in system programming, while it's extremely rare in application programming.

What actually happens more often than it would be desirable (especially in Windows programming) is to pass around function pointers casted to some integers (callbacks in LPARAM/WPARAM anyone?), and then cast them back to function pointers to actually use them.

Also, what happens when such casts/calls are performed is outside the scope of the C standard, that leaves the implementations free to do what they want in this regard.

Answer 3

Well, x() doesn't magically call a function at that address. I think he means something like:

typedef void (*functionPtr)();

int x;
//...
functionPtr foo = (functionPtr)x;
//or
functionPtr goo = (functionPtr)&x;
foo();

Answer 4

Your book is misleading. The only thing that is allowed by the standard is the cast operation, and this only if the value fits. Execution of such a function pointer is undefined behavior in most cases. You'd have to know your system very well to make that possible. So the second part of your cited paragraph:

In C in particular, an integer can be cast to a function, allowing a jump to a location that does not contain the correct form of instructions to be a C function.

is just not true in that form. In particular all decent modern systems don't allow you to execute data, you'd have to set special flags for a page such that it can be considered to contain executable code.