Outputs always equal in a network trained with pybrain to approximate a function

Question

Using the code below:

tf = open('defl_07h.csv','r')

for line in tf.readlines():
    data = [float(x) for x in line.strip().split(';') if x != '']
    indata =  tuple(data[:1])
    outdata = tuple(data[1:])
    ds.addSample(indata,outdata)

net = buildNetwork(ds.indim,20,ds.outdim,recurrent=True)
t = BackpropTrainer(net,learningrate=0.01,momentum=0.5,verbose=True)
t.trainOnDataset(ds,10)
t.testOnData(verbose=True)

Get the same outputs as follows:

out: [3.479 ] correct: [11.86 ] error: 35.12389858 out: [3.479 ] correct: [12.1 ] error: 37.16423359 out: [3.479 ] correct: [12.28 ] error: 38.73228485

then the network structure created:

Module: in
-connection to hidden0
- parameters [-1.9647867  -0.41898579 -0.24047698  0.6445537   0.06084947 -3.17343892
  0.25454776 -0.45578641  0.70865416 -0.40517853 -0.22026247 -0.13106284
 -0.71012557 -0.61140289 -0.00752148 -0.61770292 -0.50631486  0.95803659
 -1.07403163 -0.87359713]
Recurrent connections
Module: bias
-connection to out
- parameters [ 0.55130311]
-connection to hidden0
- parameters [-0.31297409 -0.2182261  -0.70730661 -1.65964456 -0.18366456  0.52280203
 -0.03388935  0.61288256  2.49908814  0.53909862 -0.56139066  0.06752532
 -0.71713239 -1.4951833   0.84217369  0.16025118  0.01176442 -0.59444178
  0.85652564  1.60607469]
Recurrent connections
Module: hidden0
-connection to out
- parameters [ 1.00559033 -0.02308752 -2.51970163  0.39714524  0.33257302 -0.6626978
 -0.53004298 -1.0141971  -0.95530079 -0.66953093 -0.00438377 -1.1945728
  0.99363152 -1.17032002  0.03620047 -0.21081934  0.2550164  -1.65894533
  0.20820361 -1.38895542]
Recurrent connections
Module: out
Recurrent connections

where could be the error?

Answer 1

I see a few problems with your code sample and they cause your net not learning anything meaningful. Here are some problems with your code:

1. Your data is not normalized! Neural networks are sensitive to the range of data fed into them. In your case, you supplied a range with a min of ~2500 and a max of ~36000, when your input and targets should be normalized between the -1,1 or 0,1 range (depending on if you used Tanh or Sigmoid).
2. You chose BackpropTrainer which generally needs to be trained a lot more than 10 epochs. Try 100 or 1000. (Or try RPromMinusTrainer, which I personally like.)

Both of these problems caused your net to just learn to spit out a single value.

I changed your code a little to include normalization and the RPropMinusTrainer with a decent amount of iterations (RProp- needs a lot fewer epochs):

from pybrain.supervised.trainers import *
from pybrain.tools.shortcuts import buildNetwork
from pybrain.datasets import SupervisedDataSet, SequentialDataSet
from pybrain.structure import RecurrentNetwork
from pybrain.structure import TanhLayer
import numpy as np

ds = SupervisedDataSet(1, 1)

tf = open('defl_07h.csv','r')

for line in tf.readlines():
    data = [float(x) for x in line.strip().split(';') if x != '']
    indata =  tuple(data[:1])
    outdata = tuple(data[1:])
    ds.addSample(indata,outdata)

i = np.array([d[0] for d in ds])
i /= np.max(np.abs(i),axis=0)
o = np.array([d[1] for d in ds])
o /= np.max(np.abs(o),axis=0)

nds = SupervisedDataSet(1, 1)
for ix in range(len(ds)):
    nds.addSample( i[ix], o[ix])

# train with normalized
net = buildNetwork(nds.indim,5,nds.outdim,recurrent=True)
t = RPropMinusTrainer(net,verbose=True)
t.trainOnDataset(nds,30)
t.testOnData(nds, verbose=True)

I get the following output:

out:     [  0.234]
correct: [  0.168]
error:  0.00217289
out:     [  0.209]
correct: [  0.168]
error:  0.00083736
...
out:     [  0.986]
correct: [  0.914]
error:  0.00258013
out:     [  1.006]
correct: [  0.916]
error:  0.00405199
out:     [  0.985]
correct: [  0.924]
error:  0.00187248

Not perfect, but the predictions increase with the actual normalized values, just like your original (pre-normalized) data did.

PS: I know this is an old question, but I've hit this problem before in the past and thought it might help some people. So if you're in the NN trenches and getting nowhere, remember to ask yourself: is my neural net expecting normalized data? Am I training enough?