Stacked Model performance?

Question

I am currently working with a dataset that seems very easily separable and I have an accuracy of 99% for SVM (NN-98%, RF-98%, DT-96-97% and I have checked for leakage & overfitting). As part of my project, I am also learning how to implement a hybrid model but its accuracy is also 99% (1 record misclassified). It seems to be misclassifying the same record as SVM and no other algorithm seems to get that record correct. Probability wise- RF and Logistic Regression give a probability of 0.5-0.7 but all other models give 0.9-1 probability for the wrong prediction. I was wondering what I should do now. Are there certain techniques I can use to reduce the probability to below 0.5 in the stacked model so that it isn't classified the way it currently is? I am using the StackingCVClassifier for implementation and this is a binary classification project.

I am currently using SVM, LR, RF as base models and NB as the meta-model.

My other question is if there is even value in making a hybrid model if the accuracy is the same as SVM's?

Thank you!

Answer 1

Are there certain techniques I can use to reduce the probability to below 0.5 in the stacked model so that it isn't classified the way it currently is?

It's generally not a good idea to try to bias the classifier in order to deal better with some specific instances, because it's likely to make it weaker in with some (possibly many) other instances. The way to do that would be oversample in the training set the instances which will help predict the target instance correctly, but that's a terrible idea, don't do it! :)

In general it's perfectly normal to have some errors, the data can contain noise or even sometimes annotation errors.

My other question is if there is even value in making a hybrid model if the accuracy is the same as SVM's?

If the data is really easy to classify, it's totally possible that stacking learners doesn't improve performance: if the performance of a single model reaches the maximum obtainable on this data, clearly there's nothing to improve.

However it's also possible that the benefit of stacking is not visible in this particular test: in this hypothesis the meta-model is indeed better in general that a single learner, but the test set just doesn't contain any instance on which this can be seen.

A way to check this is to reduce the size of the training set: by making things harder for the classifiers, it's possible that the weaknesses of the individual learners will show up.