How to train neural networks with large sized data sets?

Question

I have a dataset size of ~500000 with input dimension 46. I am trying to use Pybrain to train the network but the training is extremely slow for the whole dataset. Using batches of 50000 data points, each batch takes more than 2 hours for training.

What are caveats of optimizing the network design so that the training is faster?

Answer 1

Here are some of the things that influence your training speed:

• Number of weights in your network
• Speed of your CPU
• Package you are using (mostly engine it is working on, in PyLearn this is Theano)
• If all your data fits in memory or you are reading from disk in between batches

With regards to network design the only thing you can really do is make the network more shallow to reduce the number of weights. To reduce the number of epochs there might be other options like adding residual connections but that will not decrease the training time of 1 epoch.

Without more information it is unclear where the bottleneck is, but 20 hours for one epoch seems a bit high. The easiest and biggest improvement you will be able to get is to use a good GPU, which should be possible using pylearn since it is built on top of Theano.

Answer 2

Use some data set analysis (like Principal Component Analysis) trying to figure out the underlying structure of your data set. You could probably reduce the dimension without losing too much information.

Answer 3

2 hours for a batch is too long even with a CPU. How many layers you have? How many neutrons you have? Have you tried a single layer? Try to reduce the complexity of your model.

You might want to train a softmax regression, just to get an lower bound for your computation power. If you find your machine struggle even with softmax regression, you'll need to upgrade your hardware.