It sounds like IE doesn't use the same file as other browsers. I don't remember if IE support OGV files but if not IE should use the mp4 file you provide. It seems then your mp4 hasn't a delay set for autostart.
I have used sometimes a program called MP4box to mux or demux the file so it can start without loading everything.