sb = new StringBuilder(1024*1024*1024) > 'System.OutOfMemoryException' was thrown

Question

I need to append lots of string in a loop and this will create a very long string.
1GB should be long enough to store the string, therefore I do this:

var sb = new StringBuilder(1024 * 1024 * 1024);

but receives this error

'System.OutOfMemoryException' was thrown

any advice to append long strings?

I'm writing a backup tools for MySQL database. The tools will export data from database and write into a text file. The data contains BLOB and TEXT data type. So, will expect very long string.

This is the code: Code Updated 1

var conn = new MySqlConnection(Program.ConnectionString);
var cmd = new MySqlCommand();
cmd.Connection = conn;
conn.Open();

cmd.CommandText = selectSQL;

MySqlDataReader rdr = cmd.ExecuteReader();

while (rdr.Read())
{
    var sb = new StringBuilder(1024 * 1024 * 1024);
    for (int i = 0; i < rdr.FieldCount; i++)
    {
        if (sb.Length > 0)
            sb.AppendFormat(separator);

        if (rdr[i].GetType() == typeof(byte[]))
        {
            sb.AppendFormat(ConvertByteArrayToHexString((byte[])rdr[i]));
        }
        else
            sb.AppendFormat(rdr[i] + "");
    }
    WriteToFile(sb.ToString());
}

conn.Close();

Answer 1

The out of memory error may be caused by any number of reasons, based on how Windows virtual memory and how the .NET runtime works.

Do you really need to dump everything into the StringBuilder all at once? If you're writing to a file, you can do the operation in pieces at a time.

Use something like this as a starting point: int bufferSize = 65536;

using (StreamWriter sw = new StreamWriter("filename", true, System.Text.Encoding.UTF8, bufferSize))
{
    while(!finished)
    {
        string data = "foo"; //get next data here...
        sw.Write(data);
    }
}

`

References: What consumes less resources and is faster File.AppendText or File.WriteAllText storing first in StringBuilder?

Answer 2

Use a file stream to write long files. There's no need to keep the full string in memory at any one time.

Answer 3

You haven't asked for a StringBuilder with enough capacity to store 1 GB - you've asked for one with enough capacity to store 1024 * 1024 * 1024 characters, which'll require (roughly) 2 GB of memory. That's smaller than the maximum string length (2^31-1 characters), but larger than the heap space available in a 32-bit process.

You may not need a StringBuilder with that kind of capacity - others answers discuss that. Assuming you need a string that gigantic, you definitely don't need to pre-allocate your StringBuilder with that size. Just create a StringBuilder, and let it grow on its own:

var sb = new StringBuilder();

or pre-allocate something more reasonable, if you know you'll have huge amounts of data:

var sb = new StringBuilder(1024 * 1024);