Encog AI Framework: Backpropagation with Gaussian Noise Injection

Question

I have been tinkering with standard Multi-Layer Perceptrons and the Backpropagation algorithm in Encog for two weeks now, both via workbench and via Java code. My next job will require inserting noise in input patterns, like in this paper: http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6033567 (PCA and Gaussian noise in MLP neural network training improve generalization in problems with small and unbalanced data sets)

Basically, I need to (it is a binary classification problem): 1 - Transform the input patterns using Principal Component Analysis (PCA) 2 - Use Backpropagation to train an MLP, with a trick: Insert a different white noise in each training pattern on each epoch.

What is the more straightfoward way to do this noise injection using the Java version of Encog? Does any of the available training algorithms involve artificial noise injection?

PS.: The full algorithm for the paper I cited is

    1. Apply PCA to decorrelate the variables

    2. Initialize the system architecture 

    3. Set k, max number of epochs and min error 

    4. Begin training - While epoch counter 
        a. Randomly draw an input pattern (vector x) without replacement for presentation 

        b. Inject noise into input pattern 

            1. For every variable from the input pattern 

                a. Draw g from a Gaussian distribution. g ~ N(0,1) 

                b. Calculate n = k * g

                c. Add ninto input pattern x 

        c. Present the input pattern 

        d. Adjust the system parameters 

        e. If training stopping criterion has been reached then 

            1. Stop training 

        f. Otherwise 

            1. Increment epoch counter 

            2. Go to 4.a

Answer 1

I think the best way to do this is going to be to create a class that implements the MLDataSet interface. You would then provide a regular BasicMLDataSet (or other data set) to your new version of MLDataSet. For the size() method you would return the number of training patterns that you want trained per iteration. Then for each call to the to your new MLDataSet to return a MLDataPair you randomly select a pair from the provided data set, you would then clone this element, and add the noise as described, and return it.

Does that sounds like it would accomplish what the paper is describing? If you end up implementing this and would like to contribute it to the Encog project, that would be great. I might attempt it myself as well, if you do not.

Answer 2

I studied the Backprop class and came up with another way that seemed more general and direct. I created an ErrorStructure interface, a NoisyBackpropagation class and a NoisyGradientWorker.

The first class generalizes many techniques of noise injection (a theme that got renewed attention last month, from I what I saw in some scientific papers).

The second class is just the Backprop with a ErrorStructure property. The third class is a GradientWorker that receives a NoiseStructure as parameter and injects noise in the training process.

Tricky bits:

• Some ErrorStructure implementations (in particular one that I hope to publish) will require complex initialization parameters derived from the training set. I couldn't use the more general kind of MLDataSet. This is not strictly necessary, but I'd like to use it someday in the future.
• I had to add the SSJ: Stochastic Simulation in Java package as a dependency or NoiseStructure computing would take too long. I don't know how much of Encog's speed will be hampered by SSJ.

By the way, the MLDataSet adjustement solution seems very useful to some advanced re-sampling schemes.