Augmentation on test dataset and validation dataset

Question

I'm training a segmentation model (computer-vision). Thus, my dataset contains images and masks (binary segmentation of objects).

I'm augmenting the training dataset (applying random crop, rotation or shift etc.) to get a larger dataset. I don't apply augmentation on test and validation dataset.

Should I use augmentation on the validation dataset or the test dataset too ?

Answer 1

Your test and validation dataset should reflect the type of data you would expect when you deploy your model in the actual setting. So usually you do not apply augmentation to the validation and test dataset, since in the real setting you will not receive some strange augmented images.

Another way to think of it is if you apply augmentation to your validation dataset then you will actually measure how your model will perform on augmented data and not on 'real' data. You will use this to inform the best settings for your model so you will end up with a model tuned to perform well on augmented data.