Once an integer overflows, using its value results in undefined behavior. A program that uses the result of an int
after the overflow is invalid according to the standard -- essentially, all bets about its behavior are off.
With this in mind, let's look at what's going to happen on a computer where negative numbers are stored in two's complement representation. When you add two large 32-bit integers on such a computer, you get a negative result in case of an overflow.
However, according to C++ standard, the type of malloc
's argument, i.e. size_t
, is always unsigned. When you convert a negative number to an unsigned number, it gets sign-extended (see this answer for a discussion and a reference to the standard), meaning that the most significant bit of the original (which is 1
for all negative numbers) is set in the top 32 bits of the unsigned result.
Therefore, what you get is a modified version of your third case, except that instead of "wildcard bit #
" it has ones all the way to the top. The result is a gigantic unsigned number (roughly 16 exbibytes or so); naturally malloc
fails to allocate that much memory.