How to select the best model from validation/training/holdout accuracy score

datascience.stackexchange https://datascience.stackexchange.com/questions/68501

  •  09-12-2020
  •  | 
  •  

Question

I have made my own function to log all the attempts at hyperparamter tuning, the following information is gathered from a 10 fold cross validation.

But I am struggling to work out which model is best.

+----+-------------------+-------------------+-------------------+-------------------+
|    | Validation        | Val StDev         | Train Err         | Holdout           |
+----+-------------------+-------------------+-------------------+-------------------+
| 0  | 0.899393281242345 | 0.02848162454242  | 1                 | 0.903572035647507 |
+----+-------------------+-------------------+-------------------+-------------------+
| 1  | 0.902138248122673 | 0.029520127182082 | 1                 | 0.924233269690891 |
+----+-------------------+-------------------+-------------------+-------------------+
| 2  | 0.899394809813502 | 0.025173568322695 | 0.918663669124935 | 0.909288194444444 |
+----+-------------------+-------------------+-------------------+-------------------+
| 3  | 0.897228682965021 | 0.030714865334356 | 1                 | 0.908866279069768 |
+----+-------------------+-------------------+-------------------+-------------------+
| 4  | 0.909270070641525 | 0.027056183719667 | 1                 | 0.924575965355467 |
+----+-------------------+-------------------+-------------------+-------------------+
| 5  | 0.891001537843704 | 0.032846295144796 | 1                 | 0.903091978426922 |
+----+-------------------+-------------------+-------------------+-------------------+
| 6  | 0.895784577401649 | 0.032132196798841 | 1                 | 0.884848484848485 |
+----+-------------------+-------------------+-------------------+-------------------+
| 7  | 0.88188105768332  | 0.033952319596936 | 0.910316431235621 | 0.903091978426922 |
+----+-------------------+-------------------+-------------------+-------------------+
| 8  | 0.920584479640099 | 0.027520133628529 | 0.936918085663145 | 0.924575965355467 |
+----+-------------------+-------------------+-------------------+-------------------+
| 9  | 0.887347364032516 | 0.030797370441503 | 0.900531622363285 | 0.903091978426922 |
+----+-------------------+-------------------+-------------------+-------------------+
| 10 | 0.876942399956479 | 0.043318142049256 | 1                 | 0.868971836419753 |
+----+-------------------+-------------------+-------------------+-------------------+
| 11 | 0.900452899973248 | 0.033120692366442 | 0.924324942576064 | 0.886904761904762 |
+----+-------------------+-------------------+-------------------+-------------------+
| 12 | 0.889635597754135 | 0.023388005619559 | 1                 | 0.91413103898301  |
+----+-------------------+-------------------+-------------------+-------------------+
| 13 | 0.899270803213788 | 0.026083641971929 | 1                 | 0.91413103898301  |
+----+-------------------+-------------------+-------------------+-------------------+
| 14 | 0.886812435037192 | 0.033456079535065 | 0.904149376999805 | 0.898247322297955 |
+----+-------------------+-------------------+-------------------+-------------------+
| 15 | 0.884982783710439 | 0.029572747271092 | 0.901110070564378 | 0.903091978426922 |
+----+-------------------+-------------------+-------------------+-------------------+
| 16 | 0.896948178627522 | 0.026483462928863 | 0.919101870462519 | 0.91413103898301  |
+----+-------------------+-------------------+-------------------+-------------------+
| 17 | 0.891278476391404 | 0.030334266939915 | 1                 | 0.888614341085271 |
+----+-------------------+-------------------+-------------------+-------------------+
| 18 | 0.909092365260866 | 0.031848098756772 | 1                 | 0.898247322297955 |
+----+-------------------+-------------------+-------------------+-------------------+
| 19 | 0.889032812279866 | 0.028580171007027 | 1                 | 0.903572035647507 |
+----+-------------------+-------------------+-------------------+-------------------+
| 20 | 0.890821503056501 | 0.03250894068403  | 0.935473681065598 | 0.889619742654119 |
+----+-------------------+-------------------+-------------------+-------------------+
| 21 | 0.908662067002155 | 0.02515678884091  | 1                 | 0.91413103898301  |
+----+-------------------+-------------------+-------------------+-------------------+
| 22 | 0.894626358857844 | 0.038437447908009 | 0.921035110317957 | 0.907956547269524 |
+----+-------------------+-------------------+-------------------+-------------------+
| 23 | 0.904503845292009 | 0.03112232540355  | 0.922738180662704 | 0.903091978426922 |
+----+-------------------+-------------------+-------------------+-------------------+
| 24 | 0.893363641701947 | 0.032000273114453 | 1                 | 0.897729496966138 |
+----+-------------------+-------------------+-------------------+-------------------+
| 25 | 0.891379560352061 | 0.032525437349441 | 0.916932513590361 | 0.892891134050809 |
+----+-------------------+-------------------+-------------------+-------------------+
| 26 | 0.905999138236311 | 0.027801373368529 | 1                 | 0.913722347684612 |
+----+-------------------+-------------------+-------------------+-------------------+
| 27 | 0.892155761699392 | 0.028942257106962 | 1                 | 0.898740310077519 |
+----+-------------------+-------------------+-------------------+-------------------+
| 28 | 0.90547992240768  | 0.034616407726398 | 1                 | 0.888072054527751 |
+----+-------------------+-------------------+-------------------+-------------------+
| 29 | 0.854504389713449 | 0.045352425614533 | 1                 | 0.816845180136319 |
+----+-------------------+-------------------+-------------------+-------------------+
| 30 | 0.854329853134155 | 0.047548722046064 | 0.912076317693916 | 0.828930724012203 |
+----+-------------------+-------------------+-------------------+-------------------+
| 31 | 0.919465770658208 | 0.029180215562696 | 0.931409372636061 | 0.938637698179683 |
+----+-------------------+-------------------+-------------------+-------------------+
| 32 | 0.90057318637927  | 0.026049221971696 | 1                 | 0.919367283950617 |
+----+-------------------+-------------------+-------------------+-------------------+
| 33 | 0.895446367880531 | 0.033041859254943 | 1                 | 0.913722347684612 |
+----+-------------------+-------------------+-------------------+-------------------+
| 34 | 0.884125762352326 | 0.03697539365293  | 0.897337858027344 | 0.897729496966138 |
+----+-------------------+-------------------+-------------------+-------------------+
| 35 | 0.883233039971663 | 0.034111835608482 | 0.893720586886425 | 0.913722347684612 |
+----+-------------------+-------------------+-------------------+-------------------+
| 36 | 0.898831686569679 | 0.042604871968562 | 1                 | 0.863619885443604 |
+----+-------------------+-------------------+-------------------+-------------------+
| 37 | 0.909826578207203 | 0.029949272123959 | 1                 | 0.898247322297955 |
+----+-------------------+-------------------+-------------------+-------------------+
| 38 | 0.881913627468284 | 0.036432833984006 | 0.893183972214204 | 0.918992248062016 |
+----+-------------------+-------------------+-------------------+-------------------+
| 39 | 0.893848891847337 | 0.02592119349599  | 1                 | 0.90842259006816  |
+----+-------------------+-------------------+-------------------+-------------------+
| 40 | 0.855511803338288 | 0.045600097937187 | 1                 | 0.816845180136319 |
+----+-------------------+-------------------+-------------------+-------------------+
| 41 | 0.856261861628324 | 0.046589739419035 | 1                 | 0.811284379041902 |
+----+-------------------+-------------------+-------------------+-------------------+
| 42 | 0.892329922600147 | 0.027143842917071 | 1                 | 0.91413103898301  |
+----+-------------------+-------------------+-------------------+-------------------+
| 43 | 0.883558035554916 | 0.031100578488573 | 0.903223129534581 | 0.898740310077519 |
+----+-------------------+-------------------+-------------------+-------------------+
| 44 | 0.853943064191891 | 0.045073764302981 | 1                 | 0.816845180136319 |
+----+-------------------+-------------------+-------------------+-------------------+
| 45 | 0.888853496341911 | 0.027961101762991 | 1                 | 0.913722347684612 |
+----+-------------------+-------------------+-------------------+-------------------+
| 46 | 0.897113661181162 | 0.036022289370872 | 0.921272431864044 | 0.897729496966138 |
+----+-------------------+-------------------+-------------------+-------------------+
| 47 | 0.891468488082473 | 0.028579769797201 | 1                 | 0.91413103898301  |
+----+-------------------+-------------------+-------------------+-------------------+
| 48 | 0.898831686569679 | 0.042604871968562 | 1                 | 0.863619885443604 |
+----+-------------------+-------------------+-------------------+-------------------+
| 49 | 0.902910285816319 | 0.025031947763032 | 1                 | 0.908866279069768 |
+----+-------------------+-------------------+-------------------+-------------------+
| 50 | 0.876865550417646 | 0.04283766065777  | 0.941524627838129 | 0.868971836419753 |
+----+-------------------+-------------------+-------------------+-------------------+
| 51 | 0.88091084510941  | 0.030430567351611 | 0.902847063239986 | 0.893421723610403 |
+----+-------------------+-------------------+-------------------+-------------------+
| 52 | 0.881680572698977 | 0.031246268640075 | 0.902192589536189 | 0.892891134050809 |
+----+-------------------+-------------------+-------------------+-------------------+
| 53 | 0.855040900830318 | 0.046858153140763 | 1                 | 0.816845180136319 |
+----+-------------------+-------------------+-------------------+-------------------+
| 54 | 0.894438862703006 | 0.040891854948011 | 0.920792225979698 | 0.907956547269524 |
+----+-------------------+-------------------+-------------------+-------------------+
| 55 | 0.891904027574454 | 0.031645714271463 | 1                 | 0.898740310077519 |
+----+-------------------+-------------------+-------------------+-------------------+
| 56 | 0.886050671635633 | 0.032472146105505 | 0.899909848784576 | 0.908866279069768 |
+----+-------------------+-------------------+-------------------+-------------------+
| 57 | 0.881980625144301 | 0.03563700286777  | 0.892637674766258 | 0.918992248062016 |
+----+-------------------+-------------------+-------------------+-------------------+
| 58 | 0.891270767582537 | 0.03142082216981  | 1                 | 0.898740310077519 |
+----+-------------------+-------------------+-------------------+-------------------+
| 59 | 0.910664399342078 | 0.025582535272637 | 0.927959603815499 | 0.893421723610403 |
+----+-------------------+-------------------+-------------------+-------------------+
| 60 | 0.888100544552359 | 0.026544114270193 | 1                 | 0.918597857838364 |
+----+-------------------+-------------------+-------------------+-------------------+
| 61 | 0.896690074843623 | 0.034624649065343 | 1                 | 0.913292822803036 |
+----+-------------------+-------------------+-------------------+-------------------+
| 62 | 0.887338053809583 | 0.030621509271507 | 1                 | 0.918992248062016 |
+----+-------------------+-------------------+-------------------+-------------------+
| 63 | 0.897753456871316 | 0.025062963044505 | 0.916156729004089 | 0.91413103898301  |
+----+-------------------+-------------------+-------------------+-------------------+
| 64 | 0.896456912702229 | 0.030808038532288 | 0.907756656209691 | 0.909288194444444 |
+----+-------------------+-------------------+-------------------+-------------------+
| 65 | 0.883460118896986 | 0.037339407800874 | 0.89706586511388  | 0.897729496966138 |
+----+-------------------+-------------------+-------------------+-------------------+
| 66 | 0.878395246090765 | 0.034089155345539 | 0.896519056428997 | 0.913722347684612 |
+----+-------------------+-------------------+-------------------+-------------------+
| 67 | 0.910664399342078 | 0.025582535272637 | 0.927959603815499 | 0.893421723610403 |
+----+-------------------+-------------------+-------------------+-------------------+
| 68 | 0.895947305636744 | 0.027392857919265 | 1                 | 0.91413103898301  |
+----+-------------------+-------------------+-------------------+-------------------+
| 69 | 0.891138031159789 | 0.032237486772592 | 1                 | 0.903572035647507 |
+----+-------------------+-------------------+-------------------+-------------------+
| 70 | 0.885255027194677 | 0.031984990817382 | 0.90200648514247  | 0.903572035647507 |
+----+-------------------+-------------------+-------------------+-------------------+
| 71 | 0.908735508831696 | 0.027210512830753 | 1                 | 0.913722347684612 |
+----+-------------------+-------------------+-------------------+-------------------+
| 72 | 0.88713294586216  | 0.029978466529712 | 1                 | 0.903572035647507 |
+----+-------------------+-------------------+-------------------+-------------------+
| 73 | 0.901055027327092 | 0.033359546159533 | 0.923523486623107 | 0.887502446662752 |
+----+-------------------+-------------------+-------------------+-------------------+

Traing err - the average accuracy score of the estimator fit and predicted on the training data

Validation - the average accuracy score of the esitmator fit on the training and predicted on the testing data set.

Val StDev - the standard deviation of the accuracy score based on the esitmator fit on the training and predicted on the testing data set.

Holdout (testing) Error - the holdout accuracy score (completly unseen data)

With this information in this format - how would one determine the best choice of esimator? There is so much information out there on model selection it is hard to take it all in.

Is my goal to produce the smallest variance between Training error and vaidation error? Ie bringing the lines of the learning curve as close together as possible.

Is a 100% accuracy on the Traing error a bad thing? (its bascially memorised the data) and should I ignore all these models?

Shoud I use the 1 sd rule to select the esimators that are 1 SD from the max mean validation score... and then choose the model with the lowest variance between holdout and validation?

Is there a better way decide which model is best? What model would you select?

Was it helpful?

Solution

Is my goal to produce the smallest variance between Training error and vaidation error?

Normally your goal is to have a model that performs best in production, for that you normally split the train/hold out in a way that reproduce the best in the hold out set. Have a look at this answer from here What would I prefer - an over-fitted model or a less accurate model? They talk a bit about model selection.

Is a 100% accuracy on the Traing error a bad thing?

When you have this you probably have "overfitting". Your model is predicting unrealistically good results in the training set that most probable it won´t able to achieve in the test set.

It is not necessarily a bad thing, but it is suspicious.

With this information in this format - how would one determine the best choice of estimator?

For your case, I would recommend choosing the one that achieves best in hold out without overfitting.

Note: Your dataset it is way too small. 830 rows of data is almost nothing. In order to validate your model properly I will suggest Leave One Out Cross Validation: "In LOOCV we divide the data set into two parts. In one part we have a single observation, which is our test data and in the other part, we have all the other observations from the dataset forming our training data." from here

What is happening in your test is that 2 or 3 rows of your data make a big impact in the result that is why you have a big variance between the different folds and parameters. It is hard to say which model is better when validating this way.

In summary, my suggestion is that you do LOOCV and then select the one that has better performance in test.

(I am assuming that you can do random split since there is no patter such as it is a time series)

Licensed under: CC-BY-SA with attribution
Not affiliated with datascience.stackexchange
scroll top