Ok, so it works well in PyQt, not so well in PySide.
I'm using a QPixmap wrapped in a QPixmapItem that gets added to the scene. To update the frame, I change the contents of the pixmap object and call update() on the scene.
Performance is good enough for video (although I don't need high frame rates for this project).
In PySide I ran into weird issues when I used more than 1 pixmap item, in PyQt it works just fine.