Which value of output should be taken in multiple sessions of training Neural Network

Question

Suppose I am using Neural Network for a 2 class classification. After training the network with the training set, I want to predict the class label of a dataset with no class label. Now with retraining, the same dataset gives different result. For example in one training session, a sample gave the output of it belonging to class 1 while in the other session it gave the output of it belonging to class 2. Then which value should be taken as the correct one?

Answer 1

This is normal behaviour of most classifiers. You are not guaranteed 100% accuracy in machine learning, and a direct consequence is that classifiers make mistakes. Different classifiers, even if trained on the same data, can make different mistakes. Neural networks with different starting weights will often converge to slightly different results each time.

Also, perhaps in your problem the classification is an artificial construct over some spectrum (e.g. "car" vs "van" or "safe" vs "dangerous") in which case the mistake in one case is entirely reasonable and expected?

You should use the value from the classifier that you trust the most. To establish which one that is, use cross-validation on a hold-out set (where you know the true labels), and use the classifier with the best accuracy, or other metric, such as logloss or area under ROC. Which metric you should prefer depends on the nature of your problem, and the consequences of making a mistake.

Alternatively, you could look at averaging the class probabilities to determine the best prediction - perhaps one classifier is really confident in the class assignment, and the other is not, so an average will go with the first classifier. Some kind of model aggregation will often boost accuracy, and is common in e.g. Kaggle competitions when you want the highest possible score and don't mind the extra effort and cost. However, if you want to use aggregation to solve your problem, again you should test your assumptions using validation and a suitable metric so you know whether or not it is really an improvement.