Data augmentation: rotating images and zero values

Question

A lot of people rotate images to create a larger training set for neural networks. For most nets, all of the inputs have to be the same size so the image rotation function has to crop the newly rotated images to match the input size. So, say you have $32x32$ resolution images and do 45 degree rotations. Some of the output images will look diamond shaped with black (zero values) in the corners. So, my question is: should you leave these zero values alone or change them in some way and if so, how?