This seems to be a bug, or missing feature, and it's hard to come up with a good workaround.
One somewhat hacky solution is to force a resize of the video widget:
size = self.video.size()
self.video.resize(0, 0)
self.video.resize(size)
but I wouldn't bet on this working on all platforms.
A more reliable workaround would be to put the video-widget inside a container widget with a black background, and then simply hide/show the video-widget when stopping/starting the media.