using
resize(image, image, Size(640, 360), 0, 0, INTER_CUBIC);
after read(image) seems to be the best solution to solve this problem. So the total (test) code becomes:
VideoCapture cap("path/to/file");
cap.set(CV_CAP_PROP_FRAME_WIDTH, 640);
cap.set(CV_CAP_PROP_FRAME_HEIGHT, 360);
Mat image;
cap.read(image);
resize(image, image, Size(640, 360), 0, 0, INTER_CUBIC);
namedWindow("firstframe", 1);
imshow("firstframe", image);
waitKey(0);
If anyone knows of a better way, please let me know.