Convert java mjpeg stream bytes to an image

Question

I'm receiving an mjpeg stream from the WiRC.

The WiRC document describes the following about the stream:

Camera image format      JPEG
Camera image resolution  CIF: 352 × 288

The documentation describes the following:

Packet specification

The protocol uses UDP packets to transfer the MJPEG stream. The MJPEG stream consists of independent JPEG frames. A JPEG frame is sent in several packets. The fragment size is determined by the server application.

The first 16 bytes of a packet is the header. The packet header has four fields containing 32-bit words in network byte order (big-endian).

name        offset  width    description
version     0       32 bit   protocol version and flags  
frame num   4       32 bit   bit index of the JPEG frame in the stream
offset      8       32 bit   offset of the packet data in the JPEG frame
length      12      32 bit   number of data bytes in the packet beyond the header

Version field

Flags are encoded in the upper 16 bit of the version field, the lower 16-bit contains the version number (mind the host byte order during interpreting the version field).

name                  bits     description
reserved flag bits    31..17   these bits shall be ignored
last packet flag      16       if set this is the last packet of a JPEG frame
version information   15..0    Protocol version, expected value is 0x5503

I'm using the following code for decoding the stream to an image:

int offset = ((int)(bytes[8]  & 255) << 24) |
             ((int)(bytes[9]  & 255) << 16) |
             ((int)(bytes[10] & 255) <<  8) |
             ((int)(bytes[11] & 255));
int length = ((bytes[12]  & 255) << 24) |
             ((bytes[13]  & 255) << 16) |
             ((bytes[14]  & 255) <<  8) |
             ((bytes[15]  & 255));
long frame = ((bytes[4]  & 255) << 24) |
             ((bytes[5]  & 255) << 16) |
             ((bytes[6]  & 255) <<  8) |
             ((bytes[7]  & 255));

System.out.printf("Version: 0x%02X 0x%02X", bytes[2], bytes[3]);

Boolean last = (bytes[1]   &   1) == 1 ? true : false;

System.out.println(" Offset: "+offset+" Length: "+length);
System.out.println("Lastpacket: "+last + " framenum: "+frame);
System.out.println();

Bitmap bmp=BitmapFactory.decodeByteArray(bytes,32,length);

However this keeps returning a message that the BitmapFactory failed

Any ideas or sugestions?

---

I get the following responce on the console:

UDP received stuff
Version:  0x5503    Offset:     0    Length: 7584
Lastpacket: true    framenum: 223

Edit: Changed code and added console outcome

Answer 1

I found my own bug. I was building images this way:

Bitmap bmp=BitmapFactory.decodeByteArray(bytes, 32, length);

Where it should have been:

Bitmap bmp=BitmapFactory.decodeByteArray(bytes, 16, length);

The rest was done right.

However thanks guys.

Answer 2

JPEG images are fragmented over UDP Packets and UDP Packets can be out of order.

So you'll have to look at the headers:

• Frame number: Same number - same frame
• Offset : Ordering inside frame
• Last Packet flag: The frame number of this packet is the last in the frame.

Now you have to work out how to put them together. I guess offset will start from 0 for each frame, so it should be fairly easy to come up with something.

Remember to build in some timeout mechanism, since UDP is not reliable.

For example: If you have Frame x still not complete and Frame x+1 is already complete, just ditch it. Or you will end up having a memory full of incomplete images :)

----EDIT----

You may want to consider using a ByteBuffer. It has very convenient methods to deal with byteorder and int / long whatever conversion from bytes.

Answer 3

For better use of mjpeg stream, you should use libjpeg library. or this demp project http://www.anddev.org/resources/file/1484