Metric to use to choose between different models - Hyperparameters tuning [closed]

Question

Closed. This question needs to be more focused. It is not currently accepting answers.

---

Want to improve this question? Update the question so it focuses on one problem only by editing this post.

Closed 29 days ago.

Improve this question

I'm building a Feedforward Neural Network with Pytorch and doing hyperparameters tuning using Ray Tune. I have train, validation and test set, train and validation used during the training procedure. I have different versions of the model (different learning rates and numbers of neurons in the hidden layers) and I have to choose the best model. But I'm unsure on which metric should I use to choose the best model. Basically I don't know the XXXX in this line of code:

analysis.get_best_config(metric='XXXX', mode='min')

Should it be test loss or validation loss?

Answer 1

Essentially, the function of your testset is to evaluate the performance of your model on new data. It mimics the situation of your model being put into production. The validation set is used for optimizing your algorithm.

Personally I would recommend tuning your algorithm using your validation set and using the hyperparameters of the training epoch with the lowest validation loss. The accuracy of the model using these hyperparameters on your testset can be used to estimate the performance of your model in the 'real world'.