Encog Hopfield network training

Question

I am trying to recognize a single character from an image. The image data has been cleaned and has a clear distinct letter visible.

So when I add the trainigset to the hopfield network, it goed great as log as i only add 2. Once I add more, the patterns it has trained start to verlap and blur.

How can I Prevent this?

training the network:

public void trainNetwork() {
        network.reset();
        System.out.println("Training hopfield network");
        long startTimeLong = System.nanoTime();
        for (double[] ds : trainingInput) {
            network.addPattern(doubleArrayToBiPolarNeuralData(ds));
        }
        long endTimeLong = System.nanoTime();
        double durationInSec = (double) ((endTimeLong - startTimeLong) / Math.pow(10, 9));
        System.out.println("Finished training network in: " + durationInSec);
    }

    private BiPolarNeuralData doubleArrayToBiPolarNeuralData(double[] data) {
        BiPolarNeuralData patternData = new BiPolarNeuralData(neuroncount);
        if (data.length != neuroncount) {
            IndexOutOfBoundsException e = new IndexOutOfBoundsException("the size of the traingsinputs is different from the amount of input neurons");
            logger.error(e.getMessage(), e);
            throw e;
        }
        patternData.setData(data);
        return patternData;
    }

Result when training 2 characters:

Cycles until stable(max 100): 1, result=
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
      OOOOOOOOOOO                          ->         OOOOOOOOOOO                       
    OOOOOOOOOOOOOOO                        ->       OOOOOOOOOOOOOOO                     
   OOOOOOOOOOOOOOOOO                       ->      OOOOOOOOOOOOOOOOO                    
  OOOOOOOOOOOOOOOOOO                       ->     OOOOOOOOOOOOOOOOOO                    
  OOOOOOO  OOOOOOOOO                       ->     OOOOOOO  OOOOOOOOO                    
  OOOOO      OOOOOOO                       ->     OOOOO      OOOOOOO                    
  OOOOO      OOOOOOO                       ->     OOOOO      OOOOOOO                    
              OOOOOO                       ->                 OOOOOO                    
          OOOOOOOOOO                       ->             OOOOOOOOOO                    
     OOOOOOOOOOOOOOO                       ->        OOOOOOOOOOOOOOO                    
   OOOOOOOOOOOOOOOOO                       ->      OOOOOOOOOOOOOOOOO                    
  OOOOOOOOOOOOOOOOOO                       ->     OOOOOOOOOOOOOOOOOO                    
 OOOOOOOOO    OOOOOO                       ->    OOOOOOOOO    OOOOOO                    
 OOOOOOO     OOOOOOO                       ->    OOOOOOO     OOOOOOO                    
 OOOOOO      OOOOOOO                       ->    OOOOOO      OOOOOOO                    
 OOOOOOO    OOOOOOOO                       ->    OOOOOOO    OOOOOOOO                    
 OOOOOOOOOOOOOOOOOOOO                      ->    OOOOOOOOOOOOOOOOOOOO                   
 OOOOOOOOOOOOOOOOOOOO                      ->    OOOOOOOOOOOOOOOOOOOO                   
 OOOOOOOOOOOOOOOOOOOO                      ->    OOOOOOOOOOOOOOOOOOOO                   
  OOOOOOOOOOOOOOOOOOO                      ->     OOOOOOOOOOOOOOOOOOO                   
   OOOOOOOOO  OOOOOOO                      ->      OOOOOOOOO  OOOOOOO                   
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           

Results when training all characters:

Cycles until stable(max 100): 3, result=
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
      OOOOOOOOOOO                          ->      OOOOOOOOOOOOOOO                      
    OOOOOOOOOOOOOOO                        ->      OOOOOOOOOOOOOOOO                     
   OOOOOOOOOOOOOOOOO                       ->      OOOOOOOOOOOOOOOOO                    
  OOOOOOOOOOOOOOOOOO                       ->     OOOOOOOOOOOOOOOOOO                    
  OOOOOOOOO OOOOOOOOO                      ->      OOOOOOOOOOOOOOOO                     
 OOOOOOOO     OOOOOOO                      ->      OOOOOO    OOOOOO                     
 OOOOOOO      OOOOOOO                      ->      OOOOOO     OOOOO                     
 OOOOOOO                                   ->      OOOOOO     OOOOO                     
 OOOOOOO                                   ->      OOOOOO     OOOOO                     
 OOOOOO                                    ->      OOOOOO    OOOOOOO                    
 OOOOOO                                    ->      OOOOOO      OOOOO                    
 OOOOOOO                                   ->      OOOOOO     OOOOOO                    
 OOOOOOO                                   ->     OOOOOOO     OOOOOO                    
 OOOOOOO      OOOOOOO                      ->     OOOOOOO     OOOOOO                    
 OOOOOOO      OOOOOOO                      ->     OOOOOOO     OOOOOO                    
 OOOOOOOO    OOOOOOOO                      ->     OOOOOOO    OOOOOOO                    
  OOOOOOOOO OOOOOOOOO                      ->     OOOOOOOOOOOOOOOOOO                    
  OOOOOOOOOOOOOOOOOO                       ->      OOOOOOOOOOOOOOOO                     
   OOOOOOOOOOOOOOOO                        ->      OOOOOOOOOOOOOOOO                     
    OOOOOOOOOOOOOO                         ->       OOOOOOOOOOOOO                       
      OOOOOOOOOOO                          ->        OOOOOOOOO O                        
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           
                                           ->                                           

It always resolves to this same composite blob. What am I doing wrong?

Answer 1

The maximum amount of patterns that can be learned in a hopfield network while giving an acceptable number of mistakes is called its capacity. The capacity is a function of the logarithm of the total number of neurons in the net, meaning that if you want more patterns, you have to increase the amount of neurons in the network. Also, the composite blobs are called mixed states (or sometimes spin-glass states, depending on what type of blob is shown). When a Hopfield network is put into a starting state, it tends to drive itself to the local energy minimum. Sometimes that minimum is not the trained pattern, but a state that is a mix of several patterns that were used in the training. Usually, these mixed states have a higher energy than the trained patterns, but if the starting state is nearer the mixed state, it will tend to drive itself to that local minimum. Sometimes including some noise to the network could avoid these local minimums and get the network to the most approximate trained state. You could include noise by generating a random number and only perform the sign operation if that number is above certain threshold.

In conclusion, adding neurons and noise to the network could help you solve your problem.