Compiler does not care. From "man sqrt" on Linux:
The sqrt() function returns the non-negative square root of x. It fails and sets errno to EDOM, if x is negative.
If you want compilation to fail if a-b is either not a compile time constant or is a negative compile time constant you can use static_assert or BOOST_STATIC_ASSERT, whichever is available in your environment:
static_assert(a - b > 0.0, "a must be > b");
return sqrt(a - b);