Compressing data before sending it - controll characters?

Question

In a multiplayer game setting, I was going to use zlib to compress larger strings before sending them. I placed the resulting data back into strings, which are to be sent as byte streams using TCP.

My problem is, that I need to place control characters into the string as well. For example, I need to add the original string length (in plain text) to the front of the compressed string, and seperate it from the compressed data using some symbol, like "|".

But I can't find a way of knowing which bytes are actual content and which bytes are control characters. Are there any characters that a zlib-compressed-string will never contain (besides 0, which I can't use since it marks the end of a c-string) which I can use to seperate "metadata" and "compressed data"?

Answer 1

If you control both creation of compressed blobs and decompression of them, you can prepend the size to the compressed data (prrobably by reserving some bytes at the beginning of the buffer), and then when decompressing skip the size and pass the pointer to compressed data to the decompression utility. This way you don't have to worry about messing up your compressed data with the size: the decompression code never sees the bytes that carry the size information.

Answer 2

No, there are no byte values that cannot be contained in a zlib stream. However a zlib stream is self-terminating. By simply using inflate() to decompress the stream, you will find the end when it returns Z_STREAM_END. The bytes not consumed by inflate() are the next bytes immediately after the zlib stream. Upon completion of decompression you know both how many compressed data bytes there are in the stream, as well as how many uncompressed bytes were generated.

If you are simply processing your entire stream, with the zlib stream imbedded, sequentially, then there is no need to store either the compressed or uncompressed lengths anywhere. That information is inherent in the zlib data. You would only need to store such data if you had a need to process your stream non-sequentially, or with the desire to access other data in stream after the zlib data without having to decompress the zlib stream.

Answer 3

How are you doing the compression? If you're piping the output through an external program, there's not much you can do, but if you're using something internal, like a compressing streambuf, you should be able to output to the original streambuf when necessary. With filtering streams from boost::iostream, for example, you can write the length, then push the compression stream, write the data to be compressed, then pop the compression stream, and continue writing plain text. (You may have to insert a judicious flush here and there, before changing the filter stack.) Or you should be able to compress the parts you want to compress into a buffer, and use std::ostream::write to output them.