OutOfMemory exception from BinaryFormatter.Deserialize coming from its internal StringBuilder call

Question

I have a .NET 4 WCF service that is sending the client some big objects (~115Mb) that get deserialized by the client. The first time the object comes in it deserializes fine. However, all subsequent calls throw an OutOfMemoryException. I've checked to make sure that all of my IDisposables are wrapped in using blocks. I've looked at the other questions similar to this such as BinaryFormatter outofmemory exception deserialization and Deserialize from MemoryStream throws OutOfMemory exception in C# . I've tried some of the solutions that people have recommended including using Simon Hewitt's Optimized Serializer. However, in the end, he still relies on BinaryFormatter for deserializing objects.

I caught the OutOfMemoryException and looked at the stack trace (see below). The trace appears to originate from an issue with the memory utilization in the StringBuilder class. I've read other articles about how StringBuilder can cause memory problems due to the (length * 2) algorithm they use when more space is needed.

at System.Text.StringBuilder.ToString()    
at System.IO.BinaryReader.ReadString()    
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectString(BinaryHeaderEnum binaryHeaderEnum)    
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()    
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)    
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)    
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream)

Is there a way to get BinaryFormatter to work differently and not use StringBuilder or is there a good alternative to BinaryFormatter that manages memory better?

Answer 1

I wouldn't recommend using BinaryFormatter for anything that size (in fact it would probably be a lot smaller if you weren't using binaryformatter). If it is fairly simple data such as tabular data or with some constraints like no circular references and so on, then rolling your own binary serialization with a simple binary writer, or using some off the shelf serializer like protobuf-net or json.net should be more compact and significantly faster.