Query regarding neural network model

Question

I used the Neural Network Toolbox in matlab to train my data. I used four training algorithms, Scaled Conjugate Gradient (SCG), Gradient Descent with momentum and adaptive learning back-propagation (GDX), Resilient Back Propagation (RBP) and Broyden-Fletcher-Goldfarb-Shanno quasi-Newton back propagation (BFG). I have fixed the seeds at different points and obtained the accuracy. This is what I get:
The first column contains the size of the feature set. I have added features and increased the size of feature set to analyse the performance.

Initially I have ranked the features and then taken the top 8 feature as one set, the top 16 feature as the next set and so on. The first number before the '-' , is the performance of the algorithm on the training set, the second number after the '-' is the accuracy of the testing set. The train and test set has been divided into 60 and 20 respectively. The other 20 is the validation set. The learning algorithms each have been run with the same seed values to fix the accuracy.

I have fixed the seed to obtain each of the results btw, Like I have used rng(1), rng(10), rng(158), rng(250) and averaged the results to obtain one single pair of train-test accuracy, and I have done this for each pairs.

As you can see I am getting the same accuracy for all feature set size for each of the individual training algorithm. The same data shows perturbation in SVM chen I change the set size. What does this mean?

Answer 1

To debug this case, I suggest you try the following steps:

1. Reduce the features step-by-step until you end up with using just 1 feature and see whether the accuracy changes or not.
2. Add a sine-wave and a random noise to the feature set and see whether it effects any of these optimization algorithms.
3. Re-evaluate how you selected or derived these features, check if these are highly correlated.
4. Are your classification targets highly imbalanced? If so, then under/over sample them to achieve a more balanced training set. Then check the performance of you algorithm after training over this balanced dataset.

As already highlighted by Jan van der Vegt, its extremely odd that changing the no of features from 8 to 40 has no impact on test set accuracy.