Live Streaming ( Socket ) - How to sync audio and video?

Question

1- Which one is better to use for streaming video ? TCP or UDP socket and why?

2- While streaming live, audio and video are coming from the server separately, so how can i make sure that the video i display and the audio I play on the device are in sync?

Answer 1

I wrote a voice chat application a while ago and TCP was out of the question, UDP multicasting is really the only way to go if you're looking for near-realtime data flow. There's two main issues with streaming stuff over UDP though:

1. The dropped packets. In the case of audio, it's a pretty easy fix. Usually the dropped packets won't make an audible difference (the packets are decompressed individually). However, when dealing with video, especially if the video is compressed (it usually is), figuring out a proper transfer protocol that ensures network robustness is a daunting task to say the least, especially if you're doing this from scratch. Video frames are split up in various packets. Figuring out what to do when these packets are missing is tough.
2. Synchronization between audio and video. This is a very tough problem and I suggest reading up on protocols such as RTSP (Real-Time Streaming Protocol). This is not an easy task, but here's some introductory info: http://www.cs.columbia.edu/~hgs/rtsp/ - sometimes it's done by sending separate sync packets (some protocols send these over TCP) that tell the player how the sound should match up with the video.

Answer 2

I would do UDP. However it depends on what you want. UDP will drop packets rather than wait (TCP). The trade off is whether you want a stable, but sometimes slow and costly, or one that is efficient, but sometimes may not get delivered. The choice is yours when it comes to how you want to implement it and how you are using it.

Answer 3

Today even youtube streams over HTTP ... here is a nodejs app which streams a file to the browser client ... use as a starting point to live stream video with audio nicely in sync

// usage 
// do following on server side (your laptop running nodejs)
// node this_file.js
//
// then once above is running point your browser at
//    http://localhost:8888
//
// of course your browser could be on your mobile or own custom app



var http = require('http'),
    fs = require('fs'),
    util = require('util');

var path = "/path/to/audio/or/video/file/local/to/server/cool.mp4"; // put any audio or video file here

var port = 8888;
var host = "localhost";

http.createServer(function (req, res) {

  var stat = fs.statSync(path);
  var total = stat.size;

  if (req.headers.range) {   // meaning client (browser) has moved the forward/back slider
                                         // which has sent this request back to this server logic ... cool
    var range = req.headers.range;
    var parts = range.replace(/bytes=/, "").split("-");
    var partialstart = parts[0];
    var partialend = parts[1];

    var start = parseInt(partialstart, 10);
    var end = partialend ? parseInt(partialend, 10) : total-1;
    var chunksize = (end-start)+1;
    console.log('RANGE: ' + start + ' - ' + end + ' = ' + chunksize);

    var file = fs.createReadStream(path, {start: start, end: end});
    res.writeHead(206, { 'Content-Range': 'bytes ' + start + '-' + end + '/' + total, 'Accept-Ranges': 'bytes', 'Content-Length': chunksize, 'Content-Type': 'video/mp4' });
    file.pipe(res);

  } else {

    console.log('ALL: ' + total);
    res.writeHead(200, { 'Content-Length': total, 'Content-Type': 'video/mp4' });
    fs.createReadStream(path).pipe(res);
  }
}).listen(port, host);

console.log("Server running at http://" + host + ":" + port + "/");