How can I write a raw string of XML to a file reliably, to a base-64 encoded string?

Question

I have memory stream of raw XML produced by a DataContractSerializer. This string can be quite long, and generates files of 60mb+. I then read this into a string, and then encode in UTF8 bytes and then convert to a base 64 string. Here is a code snippet:

// serialize to a memory stream.
rawXML = new MemoryStream();

dcs = new DataContractSerializer(vToWrite.GetType(),
    null, 2147483647, false, false, null, new ClassResolver());

dcs.WriteObject(rawXML, vToWrite);
rawXML.Seek(0, SeekOrigin.Begin);

// read the stream into a raw string of XML
sr = new StreamReader(rawXML);
xmlString = sr.ReadToEnd();
rawXML.Close();
sr.Close();

// encode into base64
//bytesToEncode = Encoding.UTF8.GetBytes(xmlString);
//encodedText = Convert.ToBase64String(bytesToEncode);

// write string out
writer = File.CreateText(fullPath + "Module" + i + "D" +
    historicDataCount +  ".bin");

writer.Write(Convert.ToBase64String(
    Encoding.UTF8.GetBytes(xmlString)));// (encodedText);

writer.Close();
// end save mod day //     

The problem i'm getting is that sometimes I run into a out of memory exception when either reading the stream into the xmlString or when writing the string out using the stream writer (Thats what 'writer' is by the way).

I'm not quite sure how to solve this. I think maybe reading and converting the original stream in 'chunks' maybe the way forward, but i'm not sure how this will affect the reading back in of the file.

Answer 1

Here is a piece of code that uses System.Security.Cryptography.ToBase64Transform and converts an array of bytes to base64 in chunks

// write string out
byte[] data = Encoding.UTF8.GetBytes(xmlString);

using (StreamWriter writer = File.CreateText(fullPath + "Module" + i + "D" + historicDataCount +  ".bin"))
    using (ToBase64Transform transformation = new ToBase64Transform())
    {                    
        byte[] buffer = new byte[transformation.OutputBlockSize];

        int i = 0;    
        while (data.Length - i > transformation.InputBlockSize)
        {
            transformation.TransformBlock(data, i, data.Length - i, buffer, 0);
            i += transformation.InputBlockSize;
            writer.Write(Encoding.UTF8.GetString(buffer));
        }

        // final block
        buffer = transformation.TransformFinalBlock(data, i, data.Length - i);
        writer.Write(Encoding.UTF8.GetString(buffer));
    }

Answer 2

Since base64 converts three octets(bytes) into four encoded characters if you read multiple chunks of 3000 bytes (or any other 3 byte multiplier) there should be no problem

For more info about base64 encoding see http://en.wikipedia.org/wiki/Base64