flash/AS3 netstream loading/buffering very slow, what am I missing?

Question

My custom movieplayer uses the following code for playing video. It takes a really long time for the clip to start playing, but once it does, you can skip directly to the end. I have a feeling that there is some knowledge I'm missing in how buffers and preload works. Could somebody send me in the right direcion?

private function init(e:Event = null):void {
    connection = new NetConnection();
    connection.addEventListener(NetStatusEvent.NET_STATUS, doNetStatus);
    connection.addEventListener(IOErrorEvent.IO_ERROR, doIOError);
    connection.addEventListener(SecurityErrorEvent.SECURITY_ERROR, doSecurityError);
    connection.connect(null);
    stream = new NetStream(connection);
    stream.addEventListener(AsyncErrorEvent.ASYNC_ERROR, doAsyncError);
    stream.addEventListener(NetStatusEvent.NET_STATUS, doNetStatus);
    stream.addEventListener(IOErrorEvent.IO_ERROR, doIOError);
    stream.client = this;
    video = new Video(1024, 576);
    mc = new MovieClip();
    mc.addChild(video);
    stage.addChild(mc);
    mc.addEventListener(MouseEvent.CLICK, onClick);
    video.attachNetStream(stream);
    stream.bufferTime = 5;
    stream.receiveAudio(true);
    stream.receiveVideo(true);
    stream.play(SITEURL + vidID +".mp4");
}

You can see the player in action here: http://joon.be/serve/ngjd_player.swf

Answer 1

Apparently the video's weren't streaming because I needed to run QT FastStart on them. This solved the problem, netstream is now running as expected.

Answer 2

You can download qt-faststart.exe and then in admin command promt run:

qt-faststart.exe "source.mp4" "fixed.mp4"

Took about 15-30 seconds for 10 minutes long video. Credit: https://articulate.com/support/article/mp4-movie-doesnt-begin-playing-until-it-has-fully-downloaded

I also tried this free opensource converter HandBrake with Web Optimized checked. (took all 10 minutes to convert)