The calculation is correct, but the order of the points isn't. Remember that the cross product is anticommutative, i.e, a x b = - (b x a)
.
In your particular example of a cylinder, you're defining the points a
and c
from left to right(when seen from the front). With your current normal calculation, the normal should be inwards. If you want it outwards, you can just reverse the order of the normal calculation. A possible solution is: normal = (a - b) x (c - b)
.
PS.: The usual definition for a triangle normal (in books) is (b - a) x (c - a)
, which is equivalent to the one above.