What is conjugate gradient descent?

Question

What is Conjugate Gradient Descent of Neural Network? How is it different from Gradient Descent technique?

I came across a resource, but was unable to understand the difference between the two methods. It has mentioned in the procedure that:

the next search direction is determined so that it is conjugate to previous search directions.

What does this mean? Also, what is line search mentioned in the web page?

Can anyone please explain it with the help of a diagram?

Answer 1

What does this sentence mean?

It means that the next vector should be perpendicular to all the previous ones with respect to a matrix. It's like how the natural basis vectors are perpendicular to each other, with the added twist of a matrix:

$\mathrm {x^T A y} = 0$ instead of $\mathrm{x^T y} = 0$

And what is line search mentioned in the webpage?

Line search is an optimization method that involves guessing how far along a given direction (i.e., along a line) one should move to best reach the local minimum.