Newton's method is defined in terms of the iteration
xi+1 = xi - f(xi) / f'(xi)
(where f'(x) is the first derivative of f(x)). To find the inverse root of r, one needs to find the zero of the function f(x) = x - 1/sqrt(r) (or, equivalently, f(x) = x2 - 1/r). Simply take the derivative, plug it in to the definition of the iteration step, simplify, and you have your answer.
Actually, the exact form used in the code comes from using a third equivalent form:
f(x) = x-2 - r
See this article for the detailed steps of the derivation. It's also derived in the Wikipedia article on fast inverse square root.