SerializationException when deserializing

Question

This code deserialize object from SQLite. I'm get serialized object from DBinaryData (BLOB) field. But get System.Runtime.Serialization.SerializationException: end of stream encountered before parsing was completed. How to fix this?

    public void Dump()
    {
        try
        {
            const string databaseName = @"C:\Code\C#\WcfService\WcfService\mainDB.db3";
            SQLiteConnection connection = new SQLiteConnection(string.Format("Data Source={0};", databaseName));
            connection.Open();
            try
            {
                SQLiteCommand command = new SQLiteCommand("INSERT into 'dump' ('DTime', 'DBinaryData') VALUES ('" + DateTime.Now.ToString() + "', '" + GetSerializedMessages() + "')", connection);
                command.ExecuteNonQuery();
            }
            finally
            {
                connection.Close();
            }
        }
        catch (Exception e)
        {
            Logger.Log(e.Message);
        }
    }

    public void Restore()
    {
        try
        {
            const string databaseName = @"C:\Code\C#\WcfService\WcfService\mainDB.db3";
            SQLiteConnection connection = new SQLiteConnection(string.Format("Data Source={0};", databaseName));
            connection.Open();
            try
            { 
                SQLiteCommand command = new SQLiteCommand("SELECT * FROM dump ORDER BY DId DESC limit 1", connection);
                SQLiteDataReader reader = command.ExecuteReader();
                while (reader.Read())
                {
                     Queue<Message> deserializedData = GetDeserializedMessages((byte[])reader["DBinaryData"]);
                     var data = MergeQueueMessage(deserializedData);
                     Logger.Log(data.ToString());
                }
            }
            finally
            {
                connection.Close();
            }
        }
        catch (Exception e)
        {
            Logger.Log(e.Message);
        }
    }

    public byte[] GetSerializedMessages()
    {
        byte[] result = null;

        MemoryStream memoryStream = new MemoryStream();
        BinaryFormatter formatter = new BinaryFormatter();

        try
        {
            lock (MessageQueue.Instance.Messages)
            {
                formatter.Serialize(memoryStream, MessageQueue.Instance.Messages);
            }
            result = new byte[memoryStream.GetBuffer().Length];
            memoryStream.GetBuffer().CopyTo(result, 0);
        }
        catch (SerializationException e)
        {
            Logger.Log("Failed to serialize. Reason: " + e.Message);
        }
        finally
        {
            memoryStream.Close();
        }
        return result;
    }

    public Queue<Message> GetDeserializedMessages(byte[] source)
    {
        Queue<Message> messages = null;
        using (MemoryStream memoryStream = new MemoryStream(source))
        {
            BinaryFormatter formatter = new BinaryFormatter();
            messages = (Queue<Message>)formatter.Deserialize(memoryStream);
        }
        return messages;
    }

    private IEnumerable<Message> MergeQueueMessage(Queue<Message> source)
    {
        IEnumerable<Message> result = MessageQueue.Instance.Messages.Union(source, new EqualityComparator());
        return result;
    }

Answer 1

With your edit: here's a bug (not sure if it is "the" bug, though):

result = new byte[memoryStream.GetBuffer().Length];
memoryStream.GetBuffer().CopyTo(result, 0);

The length of the buffer is irrelevant. If is the memoryStream.Length that matters. Frankly, this should just be result = memoryStream.ToArray(); - which would give you the correct result.

---

And another bug in the SQL:

SQLiteCommand command = new SQLiteCommand("INSERT into 'dump' ('DTime', 'DBinaryData') VALUES ('" + DateTime.Now.ToString() + "', '" + GetSerializedMessages() + "')", connection);
command.ExecuteNonQuery();

Concatenation is never a good idea, but here it is fatal; since GetSerializedMessages() returns either null (on failure - not a good idea; should have just thrown) or a byte[], this does simple concatenation. If you concatenate a byte[] the output is not what you expect:

byte[] b = {1,2,3};
string s = "a " + b + " c";
// gives: "a System.Byte[] c"

that clearly doesn't contain the actual data you wanted, so is gibberish. Ideally you should be using parameters here for both the data and the date:

SQLiteCommand command = new SQLiteCommand("INSERT into 'dump' ('DTime', 'DBinaryData') VALUES (@when, @data)", connection);
// note: not sure if SQLiteCommand has an "AddWithValue", but the overall usage
// should be something like this
command.Parameters.AddWithValue("when", DateTime.Now);
command.Parameters.AddWithValue("data", GetSerializedMessages());
command.ExecuteNonQuery();

---

Finally: don't swallow problems; your serialization code should be (IMO) more like

public byte[] GetSerializedMessages()
{
    try {
        using(MemoryStream memoryStream = new MemoryStream())
        {
            BinaryFormatter formatter = new BinaryFormatter();
            // skipped: serialize etc
            return memoryStream.ToArray();
        }
    } catch(Exception ex) {
        Logger.Log("Failed to serialize. Reason: " + ex.Message);
        throw; // it doesn't stop being a problem just because we logged it
    }
}

---

The first thing to look at is whether the byte[] you get out (via reader["DBinaryData"]), is 100% identical to the byte[] you had when you originally serialized. If you don't have a test for that, all bets are off. From the error, it sounds like they're not identical - this could be because of:

• an error in the code that serializes and stores the data
• truncation inside the database storage
• truncation when reading the BLOB (some connections limit the amount fetched in one go)
• an error in the code that fetches and deserializes the data

The first two are totally fatal: if it is those - the data is toast.

A lazy way to compare two byte[] in an integration test is to compare the hex:

// here expected should be the raw data just after serializing; actual should
// be what you get after storing it in the db and fetching it back, using
// your code
Assert.AreEqual(BitConverter.ToString(expected), BitConverter.ToString(actual));

which gives a nice hex output of any delta. You don't show how you serialize and store the messages, so I can't tell you whether there are any obvious issues there, but please see http://marcgravell.blogspot.com/2013/02/how-many-ways-can-you-mess-up-io.html for a list of common issues here.

Finally, I strongly advise: stop using BinaryFormatter for this. See questions like this to see other people's pain: basically they can't get their data back after even minor changes (or sometimes just rebuilds). Contract-based serializers would be much safer - I lean towards protobuf-net, but I'm hugely biased.