C# Clear System Memory after use of byte[] Variable

Question

I'm downloading video files through a usb device. The program stores files in a byte[] array and then writes to new file using WriteAllBytes. But after the completion, System Memory is still in use even after I assign null to byte[].

Here is the code:

byte[] myByte = session.DownloadFile(InFile);
File.WriteAllBytes(OutFile, myByte);
myByte = null;

session.DownloadFile() is my own class it all works fine just a memory issue. The program crashes if memory used by program exceeds 1GB.

Answer 1

Instead of trying to bank on the collection of memory or the GC trying to free memory. You should instead be using streams instead of large byte arrays of data. This is what streams are built for.

You can use the CopyTo method on the stream size if your application is .NET 4.5. Else you can write the chunking code manually. You can find an example in How to use Stream.CopyTo on .NET Framework 3.5?

You should be trying to do something like this:

int blockSize = 8192; // The size of the array used to chunk, default is 4096

using(var fileStream = File.Open("C:\\path\\to\\destination.file")
using(var usbStream = session.OpenFileStream(InFile)) {
    usbStream.CopyTo(fileStream, blockSize);
}

This way your application will use the blockSize (in bytes) to copy data and never more.

This would require that the session object from your above example supports streaming I/O. Else, if you have written the session class re-work it to support streaming by returning a Stream instead of a byte[].

You will never have to worry about the size of the file you are copying crashing your program. You will be able to support whatever the underlying file system(s) supports.

Answer 2

As soon as a memory structure is no longer used, it is marked for deletion by the CLR. The CLR will decide when that memory is released.

You can explicitly set the value to null, or you can put the code that uses the variable in its own block, like so:

{
   byte[] myByte = ...
   File.WriteAllBytes(....);
}

Any variable that is declared within the block gets marked for deletion upon exiting the block.

But, to repeat: the CLR decides when the memory actually gets released, and it will do so when possible and necessary. In the .NET Framework 4.5.1, the large object heap also gets compacted when necessary.

So if you run into a memory issue, it is probably not due to the byte array not being cleaned up... Maybe the video file you are downloading is very big, or the WriteAllBytes function maybe requires too much memory.

Answer 3

If you're running in 32-bit mode then your process will have 2 GB of memory available to it (the other 2 GB is reserved for the system). However, in my experience, most 32-bit C# applications tend to run out of memory around the 1.5 GB mark.

If you're holding blocks of memory greater than 85 KB then these will go into the large object heap. Until recently this was never compacted, so it's possible to have plenty of free memory but not one range large enough to satisfy a single request for a large object. In this case you'll run out of memory.

If the file you're downloading is large then I'd recommend downloading it in chunks and writing it out to a temporary file. When you've finished downloading all the chunks rename the file to be the actual file you wanted in the first place. This will reduce the memory demands in your application, but it will require a bit more work.