Finding the end of a zlib compressed stream

Question

I'm working on an NMDC client (p2p, DC++ and friends) with Qt. The protocol itself is pretty straightforward:

$command parameters|

Except for compression:

"ZPipe works by sending a command $ZOn| to the client. After $ZOn a ZLib compressed stream containing commands will follow. This stream will end with an EOF that ZLib defines. (there is no $ZOff in the compressed stream!)"

Here's the relevant code:

QTcpSocket *conn;
bool compressed;
QByteArray zbuffer;
QByteArray buffer;

// ...

void NMDCConnection::on_conn_readyRead() {
    // this gets called whenever we get new data from the hub

    if(compressed) {            // gets set when we receive $ZOn
        zbuffer.append(conn->readAll());


        // Magic happens here


        if( stream_is_complete ) {
            buffer.append(uncompressed_stream);
            buffer.append(remainder_of_data);
            compressed = false;
        }
    } else { 
        buffer.append(conn->readAll());
    };
    parse(buffer);
}

So, how do I get the values for stream_is_complete, uncompressed_stream, and remainder_of_data? I can't look for the next '$' because the stream can contain it. I tried looking for something resembling an EOF in the zlib documentation, but there is no such thing, in fact, every stream ends with a seemingly random character.

I also played around with qUncompress(), but that wants a complete stream, nothing less, nothing more.

Answer 1

Are you using zlib directly?

Totally untested...

z_stream zstrm;
QByteArray zout;
// when you see a $ZOn|, initialize the z_stream struct
parse() {
    ...
    if (I see a $ZOn|) {
        zstrm.next_in = Z_NULL;
        zstrm.avail_in = 0;
        zstrm.zalloc = Z_NULL;
        zstrm.zfree = Z_NULL;
        zstrm.opaque = 0;
        inflateInit(&zstrm);
        compressed = true;
    }
}
void NMDCConnection::on_conn_readyRead() {
    if (compressed) {
        zbuffer.append(conn->readAll());
        int rc;
        do {
            zstrm.next_in = zbuffer.data();
            zstrm.avail_in = zbuffer.size();
            zout.resize(zstrm.total_out + BLOCK_SIZE);
            zstrm.next_out = zout.data() + zstrm.total_out;
            zstrm.avail_out = BLOCK_SIZE;
            rc = inflate(&zstrm, Z_SYNC_FLUSH);
            zbuffer.remove(0, zstrm.next_in - zbuffer.data());
        } while (rc == Z_OK && zstrm->avail_out == 0);
        if (rc == Z_STREAM_END) {
            zout.truncate(zstrm.total_out);
            buffer.append(zout);
            zout.clear();
            buffer.append(zbuffer);
            zbuffer.clear();
            compress = false;
            inflateEnd(&zstrm);
        }
        else if (rc != Z_OK) {
            // ERROR!  look at zstrm.msg
        }
    }
    else // whatever
}

This incrementally decompresses (inflates) from qbuffer to qout, and stops when inflate says "no more".

Maybe it would be better to borrow from QuaZip instead.

Answer 2

The end of file is not special for zlib. The problem I see with your code is that you use "readAll()", which actually does not have any means of reporting "errors", such as end of file.

You should try to use "read()" instead, in a loop. If you read the stream and it returns 0 bytes read, you can be sure you've reached the end of stream (the other end has closed the connection). The buffer you've read, joining those of all the previous "reads" in the loop, will give you the complete buffer.