Does this abuse of function declarations invoke undefined behavior?

Question

Consider the following program:

int main()
{
    int exit();
    ((void(*)())exit)(0);
}

As you can see, exit is declared with the wrong return type, but is never called with the incorrect function type. Is this program's behavior well-defined?

Answer 1

MSVC has no problem with this program, but gcc does (at least gcc 4.6.1). It issues the following warnings:

test.c: In function 'main':
test.c:3:9: warning: conflicting types for built-in function 'exit' [enabled by default]
test.c:4:22: warning: function called through a non-compatible type [enabled by default]
test.c:4:22: note: if this code is reached, the program will abort

And, as promised, it does crash when run. The crash is no accident of an incorrect calling convention or something - gcc actually generates an undefined instruction with the opcode 0x0b0f to explicitly force a crash (gdb disassembles it as ud2 - I haven't looked up what that the CPU manual might say about the opcode):

main:
.LFB0:
    .cfi_startproc
    push    ebp
    .cfi_def_cfa_offset 8
    .cfi_offset 5, -8
    mov ebp, esp
    .cfi_def_cfa_register 5
        .value  0x0b0f
    .cfi_endproc

I'm reluctant to say that gcc is wrong in doing this because I'm sure the people who write that compiler know a lot more about C than I do. But here's how I read what the standard says about it; I'm sure someone will point out what I'm missing:

C99 says this about conversions of function pointers (6.3.2.3/8 "Pointers"):

A pointer to a function of one type may be converted to a pointer to a function of another type and back again; the result shall compare equal to the original pointer. If a converted pointer is used to call a function whose type is not compatible with the pointed-to type, the behavior is undefined.

In an expression, the identifier exit evaluates to a function pointer.

The sub-expression ((void(*)())exit) converts the function pointer that exit evaluates to into a function pointer of the type void (*)(). Then a function call is made through that pointer, passing the int argument 0.

The standard library contains a function named exit that has the following prototype:

void exit(int status);

The standard also says (7.1.4/2 "Use of library functions"):

Provided that a library function can be declared without reference to any type defined in a header, it is also permissible to declare the function and use it without including its associated header.

Your program doesn't include the header containing that prototype, but the function call made through the converted pointer uses the 'declaration' provided in the cast. The declaration in the cast isn't a prototype declaration, so we need to determine if the function type of exit as defined by the standard library and the function type of the converted function pointer in your program are compatible. The standard says (6.7.5.3/15 "Function declarators (including prototypes)")

For two function types to be compatible, both shall specify compatible return types. ... If one type has a parameter type list and the other type is specified by a function declarator that is not part of a function definition and that contains an empty identifier list, the parameter list shall not have an ellipsis terminator and the type of each parameter shall be compatible with the type that results from the application of the default argument promotions

It seems to me that the converted function pointer has a compatible function type - the return type is the same (void) and the type of the single parameter is int after the default argument promotions. So it appears to me that there's no undefined behavior here.

---

Update: After a little more thought on this, it might be reasonable to interpret 7.1.4/2 to mean that a 'self-declared' library function name must be declared correctly (though not necessarily with a prototype, but with a correct return type). Especially since the standard also says that "All identifiers with external linkage in any of the following subclauses ... are always reserved for use as identifiers with external linkage" (7.1.3).

So I think a reasonable argument can be made that the program has undefined behavior.

Answer 2

I'd say that this is potentially ill-defined in either case.

My argument would be that the resulting generated code of two calls ((void(*)())exit)(0); and exit(); could well be different. Thus, in case when int exit() is only declared (the one that you are interested in), the primary issue could be that the binary layouts of int exit(void) and void exit(int) are not necessarily the same.

In case if int exit() would also be defined, there would most likely be crash for the following reason. There is a number of calling conventions out there, and the problem may show up, for instance, when the space for the return value is reserved on stack. Accordingly, when ((void(*)())exit)(0); is used, then obviously no space on stack would be reserved (specifically for the return value) by the compiler, while the function itself (int exit()) doesn't know about that, and therefore will still try to push the int return value into the expected memory cell in the run-time (the one that should have been reserved, but wasn't), which would definitely end-up as a crash.

Answer 3

I think this has defined behavior. The relevant parts of the standard are about the parameters (p6, a bit lengthy) and about the type:

If the function is defined with a type that is not compatible with the type (of the expression) pointed to by the expression that denotes the called function, the behavior is undefined.

All of this always talks about two different entities, one the function expression that is evaluated and second the function that is called. The identifier that gives raise to the expression (your falsely declared exit) never enters the game. So in your case the function is called correctly and there is no UB.

In general it would break a lot of code if that would be UB, namely for code that stores function pointers in arrays, e.g, and then calls the functions through casts as yours according to some additional knowledge about the context.

Just one nitpick, I think that you should to do the favor to the compiler and give a prototype in such a case. The argument conversion form 0 is trivial, and correct in this case, but really very error prone.

((void(*)(int))exit)(0);

would be better.

Update: In view of Michael's answer I agree that the above would have been true if you would have done all of that with a non-library function. But 7.1.3 p1 clearly forbids do use the identifier exit different from the prototype declared in the header, and then p. 2 states

If the program declares or defines an identifier in a context in which it is reserved (other than as allowed by 7.1.4), or defines a reserved identifier as a macro name, the behavior is undefined.