What are problems of many hidden layers?

Question

Is there any problem if we use too many hidden layers in Neural Network? Can anyone simply describe what problems can occur if we have too many hidden layers.

Answer 1

The most important problem is so called "vanishing gradient phenomen", it is easy to verify (both theoretically and practically) that it is impossible to efficiently train more than one hidden layer (assuming traditional backpropagation, without deep learning/neocognitron/convolutional network) as computed gradients/derivatives are more and more smoothed. The "responsibility" for the error is lost with each additional layer.

Overfitting (as @Floris incorrectly stated) is not the main probolem here, as the same problem would come from the number of hidden units (in fact it would occur more commonly by increasing number of units in one hidden layer than by increasing number of hidden layers).