Send/Receive GZip compressed MSMQ messages in C#

Question

I am trying to send large objects (>30MB) to a MSMQ queue. Due to the large amount of data we are are tring to send the idea was to GZip the objects prior to sending them, then unzipping them on the receiving end.

However, writing the compressed stream to the message.BodyStream property seems to work, but not reading it out from there. I don't know what's wrong.

Message l_QueueMessage = new Message();
l_QueueMessage.Priority = priority;

using (MessageQueue l_Queue = CreateQueue())
{
    GZipStream stream = new GZipStream(l_QueueMessage.BodyStream, CompressionMode.Compress);

    Formatter.Serialize(stream, message);

    l_Queue.Send(l_QueueMessage);
}

The Formatter is a global property of type BinaryFormatter. This is used to serialize/deserialize to the type of object we want to send/receive, e.g. "ProductItem".

The receving end looks like this:

GZipStream stream = new GZipStream(l_Message.BodyStream, CompressionMode.Decompress);

object decompressedObject = Formatter.Deserialize(stream);

ProductItem l_Item = decompressedObject as ProductItem;

m_ProductReceived(sender, new MessageReceivedEventArgs<ProductItem>(l_Item));

l_ProductQueue.BeginReceive();

I get an EndOfStreamException "{"Unable to read beyond the end of the stream."} trying to deserialize at System.IO.BinaryReader.ReadByte()

Using the messageBodyStream property I actually circumvent the message.Formatter, which I don't initialize to anything, becasue I'm using my own ser/deser mechanism with the GZipStream. However, I am not sure if that's the correct way of doing this.

What am I missing? Thanks!

Answer 1

OK, I made this work. The key was to convert the decompressed stream on the receiver to a byte[] array. Then the deserialization started working.

The sender code (notice the stream is closed before sending the message):

using (MessageQueue l_Queue = CreateQueue())
{
    using (GZipStream stream = new GZipStream(l_QueueMessage.BodyStream, CompressionMode.Compress, true))
    {
        Formatter.Serialize(stream, message);
    }

    l_Queue.Send(l_QueueMessage);
}

The receiving end (notice how I convert the stream to a byte[] then deserialize):

using (GZipStream stream = new GZipStream(l_QueueMessage.BodyStream, CompressionMode.Decompress))
{
    byte[] bytes = ReadFully(stream);

    using (MemoryStream ms = new MemoryStream(bytes))
    {
        decompressedObject = Formatter.Deserialize(ms);
    }
}

Still, don't know why this works using the ReadFully() function and not the Stream.CopyTo(). Does anyone?

Btw, ReadFully() is a function that creates a byte[] out of a Stream. I have to credit Jon Skeet for this at http://www.yoda.arachsys.com/csharp/readbinary.html. Thanks!

Answer 2

In your original code, the problem is that you need to close the GZipStream in order for a GZip footer to be written correctly, and only then you can send it. If you dont, you end up sending bytes that can not be deserialized. That's also why your new code where sending is done later works.

Answer 3

Try to separate compressing and sending:

byte[] binaryBuffer = null;
using (MemoryStream compressedBody = new MemoryStream()) 
{
    using(GZipStream stream = new GZipStream(compressedBody, CompressionMode.Compress))
    {
        Formatter.Serialize(compressedBody, message); 
        binaryBuffer = compressedBody.GetBuffer();
    }
}

using (MessageQueue l_Queue = CreateQueue())        
{            
    l_QueueMessage.BodyStream.Write(binaryBuffer, 0, binaryBuffer.Length);
    l_QueueMessage.BodyStream.Seek(0, SeekOrigin.Begin);
    l_Queue.Send(l_QueueMessage);
}