Transmission of images (image descriptors) over internet

Question

I need to send images or image descriptors from a client (smartphone) to a server (processing machine). The server tries to recognize the images/features in a video stream and sends back the IDs of the detected ones + maybe some additional data. Normally the recognition process does not last more than several seconds (considering a big number of images to be recognized). Ideally the processing server responds within milliseconds.

The images to be detected are sent in the "setup phase" (no problem if there's a big delay) and then the sampled video frames, where the recognition process is performed on, are being sent with a specific frequency, let's say 5 frames/second. (of course the frequency is variable)

What is the best communication protocol for implementing this? The code is going to be written in C/C++ but I'm more interested in how to workflow (the concept) would look like rather than code implementation.

Is HTTP sufficient for this? How about RTSP or maybe something else? Please keep in mind that the data goes from a smartphone (where the internet connection is not exceptional) to a processing machine (server, fast internet connection).

Thank you!

Edit: Thanks for your answers. Actually I was looking for a comparison between the existing communication protocols that can implement my specific need. As I said, I'm not interested in the complexity of the code that would implement the "connection". I would have liked to see some advantages/disadvantages between them, relative to my use case. On the other hand, the server performing the recognition has to be compliant with the communication protocol (+API) implemented by the application running on the smartphone, nothing more. That means I don't care how the server does it's job as long as it is able to understand client requests and return an answer that can be understood by the application making the request.

Something that I forgot to mention (my bad) is that I'm interested in ALL the communication protocols that are providing support for implementing this use-case.

Answer 1

You might want to consider Websockets, and since you are using C++, I highly recommend Websocketpp for the server-side implementation. Websocketpp has an intuitive object-oriented interface that will allow you to handle every event when data has been received,and can handle a huge number of concurrent clients. It is written to allow for multi-threading as well. In my particular case, I wrote a real-time, multi-threaded video game server on top of Websocketpp, so I'm sure that it works. Websocketpp is built on top of the Boost library, and the reason why it shines, in my opinion, is because the API is well-written and easy to use. The library comes with excellent sample code and the developer actively maintains his project. By using a mature websockets implementation, you won't be doing any polling that burns CPU. You'll be able to handle stuff asynchronously both on the client side and the server side. Websockets is also natively available using Javascript with most web browsers, but if you aren't using a web browser or webview, you can find freely available Websocket clients for different languages.

To use the Websocketpp server, you register some callback handlers for common events.

• on_open() gets called whenever a new client makes its first connection to the server.
• on_close() gets called when a client terminates its connection.
• on_message() gets called whenever a connected client sends data to the server.

Because it is easy to register handlers for these functions, you really don't have to worry much about the protocol and you can just concentrate on your application-specific logic.

To answer your updated question, HTTP is not well-suited to streaming, but has been used quite extensively because it used to be the case that there was no good alternative. If you research AJAX stuff, you'll find that HTTP was used with a "long polling" technique. Perhaps you'll find this post quite useful: In what situations would AJAX long/short polling be preferred over HTML5 WebSockets?

I believe the accepted answer is so good that you should just refer to that one so that I don't have to repeat the merits of why I think you should choose Websockets.

Answer 2

RTSP (Real Time Streaming Protocol) is not what you need because it was designed with a very different idea in mind. Bascically what you need is a protocol that allows you to upload files (or parts of them).

You can use a low-level protocol like TCP which allows you to stream whatever you want and however you want or, you can use a higher-level protocol like HTTP given this protocol includes the Chunked transfer encoding mechanism (a mechanism designed to allows you to send chuncks (pieces of a file) to the upload server one-by-one).

In both cases you can upload images and videos to a server. HTTP is easier because it is a simple protocol and you can find lots of code samples to upload files, indeed when you upload files to youtube for example, you are using HTTP (no chuncked in this case). Anyway, I recommend to you use the Chunked transfer encoding mechanism because it allows you to resume an upload (after a pause or an internet disconnection).

Another good reason to use HTTP (over TCP) is that the server-side code will be easier given that all webservers can help you to focus on the real problem, taking away the unwanted details.

So, HTTP is perfectly suitable for your requirement, however you should take in account that no protocol has a built-in way for throttling. That means that if you want to send 5 frames/second then, you will need to write the code to control that.

However, if you require a higher control, you can use TCP directly.

Using TCP (this is pseudocode)

socket = new Socket(SERVER_IP, PORT, TCP_PROTOCOL);
nextFrame = 0;
while(videoManager.IsThereFrame(nextFrame)){
   buffer = videoManager.ReadFrameAsByteArray(nextFrame++);
   if(AmISendingTooFast) waitForAWhile()

   socket.Write(buffer);
   if(error) repeat or save the nextFrame to try again later.
}
socket.Close();

Conclusion

HTTP is perfect for what you need, use it. If your requirements are more complex and you want to have extra full control then, use TCP but remember is will be harder and you won't have a webserver to aid you.

I hope this helps you.