Your formula is wrong. You are either dividing by zero or taking the square root of a negative number.
Check your formula!
Additional info:
NaN is "Not a number". It is an IEEE floating point value that signals an invalid results, like log(-1), or sqrt(-4).
Additionally, know that Positive Infinity and Negative Infinity are also floating point values.