Back propagation algorithm when we have two outputs

Question

I have a big problem I want to implement my neuronal neutwork with 2 neurons outputs. Sth like that :

And I want to use backpropagation algorithm, but I don't know how to calculate a error, because I have a output with 2 neurons, when I have a only one neuron on a output that's very easy to use a backpropagation algorithm from one exit error, but with two neurons? I thinking about calculate error for every output seperately but then I must calculate seperately back propagation for 2 cases and I get "two different hidden layers" (For every neuron in hidden layer I have a weights for two cases). Mayby anyone knows some better solutions?

I will be very gratefull for any help.

Answer 1

Logically thinking, the first layer of weights should give you a representation (the hidden layer) that is useful for predicting both outputs. So, this layer should be updated based on the error made in both outputs. But the next layer of weights are separate for each output node, so should get separate weight updates.

So, on second layer weights, the weight updates will be calculated separately based on the respective outputs. For the first layer of weights, I would first calculate error derivatives backpropagating from each output separately and then simply combine them to get the final error derivative. Then apply learning rate to get the weight updates.

Watch out for the dynamic range of your outputs. For example, if one output is producing some real value of range [0,10] and another is producing values in range [-1000,1000] then your updates will be dominated by the one with larger range. You can

• add a preprocessing step that would change your data set to have same dynamic range in both outputs. Also, add a postprocessing step to restore the actual range.
• formulate the error functions for each output so that they produce error values of same dynamic range.