atan should get number (double, float or long), but you are providing matrix. You should use
orientation = atan( grad_y.at<float>(i,j) / grad_x.at<float>(i,j) ) * 180/PI ;
I wrote 'float' in my example since I don't know what type you are actually using. Replace it if necessary.
Same problem in sqrt.
Edit (after edit of question):
You are using sqrt in the wrong way. See its documentation. It should be:
sqrt(gradAdd, magnitude);
Even better will be to use function magnitude instead of all the multiplications and square roots:
magnitude(grad_x, grad_y, magnit);
Also I recommend you not to give names to matrices that are functions of OpenCV. This will create confusion.