How can I calculate or monitor the training of a neural network in pybrain?

Question

I have a neural network n pybrain,with two inputs,a hidden layer and a output layer.I use the following to train:

trainer = BackpropTrainer(net,ds)
trainer.trainUntilConvergence()

net is the neural network and ds is the train data.

My question is if and how I can calculate the time needed to complete the training or how can I monitor the progress of the training.Thanks.

Answer 1

You could always subclass BackpropTrainer (source code here) and override trainUntilConvergence if using maxEpochs , track the percentage of completeness using the ratio between epochs and epochs.

If not using maxEpochs you could always make an educated guess of the number of epochs remaining based on the average rate of change in the validationerrors and the size of continueEpochs. Or merely just examine the rate of change in validationerrors. If you wanted to map epochs to time you would have to profile the time of each epoch and store them.

Answer 2

Nothing to add to the previous comment except for the code I use for it:

maxepochs=20
results=[]
for i in range(len(maxepochs)):
    aux = trainer.train()
    results.extend(aux)
    plt.figure()
    plt.scatter(range(len(results[0])),results[0])
    plt.draw()

You would get a new plot at every cycle. Is not very nice, but it works for me.

Hope I'd help you