Your code seems legit, let's look at your test cases in math:
Case 1:
a=2; b=3; c=5;
s=(2+3+5)/2.0
= 5.00
And you have area = sqrt(s*(s-a)(s-b)(s-c))
See there is (s-c) in the formula, which turn out to be (5.00 - 5) = 0 In this case area = 0.00, which is correct.
Case 2:
a=2; b=3; c=6;
s=(2+3+6)/2.0
= 5.50
in terms of (s-c), you have (5.50 - 6) = -0.5
sqrt of a negative number gives you the "math domain error"
The above results implies that these numbers cannot form legit triangles. There is nothing wrong with your code or the formula. However, make sure your test cases are legit before you test your code next time.
I hope it helps =]