Solving a linear system Ax=b with gradient descent means to minimize the quadratic function
f(x) = 0.5*x^t*A*x - b^t*x.
This only works if the matrix A is symmetric, A=A^t, since the derivative or gradient of f is
f'(x)^t = 0.5*(A+A^t)*x - b,
and additionally A must be positive definite. If there are negative eigenvalues,then the descent will proceed to minus infinity, there is no minimum to be found.
One work-around is to replace b by A^tb and A by a^t*A, that is to minimize the function
f(x) = 0.5*||A*x-b||^2
= 0.5*x^t*A^t*A*x - b^t*A*x + 0.5*b^t*b
with gradient
f'(x)^t = A^t*A*x - A^t*b
But for large matrices A this is not recommended since the condition number of A^t*A is about the square of the condition number of A.