I can neither predict nor promise or verify that you'll receive exactly the same results as Mathematica: There are some very large and some very small numbers involved, and you might hit the limitations of the double
precision.
But in this case, the main reason for the error is most likely a very simple and very subtle one:
double coeff = G*sunMass/pow(pow(y[0],2)+pow(y[1],2)+pow(y[2],2),3/2);
The last exponent is 1. You are performing an integer division there, and 3/2 yields 1. You could change this to
double coeff = G*sunMass/pow(pow(y[0],2)+pow(y[1],2)+pow(y[2],2),3.0/2.0);
and see whether you get closer to the perfect circle.
BTW: Computing x² as pow(x,2) is very inefficient. If efficiency may be an issue here (and I assume that it is), you should consider writing this as something like
double yy0 = y[0]*y[0];
double yy1 = y[1]*y[1];
double yy2 = y[2]*y[2];
double coeff = G*sunMass/pow(yy0+yy1+yy2,3.0/2.0);