How to handle buffering data read from the network?

Question

When reading data over the network, you specify a buffer to receive the data into:

byte[] b = new byte[4096];
socket.Receive(b);

Now my first thought is of course to reuse the receive buffer by declaring it as a member variable of the class. My next issue is that I have not received all of the data that I am expecting, so I need to buffer my data. This is easy to accomplish by keeping track of the count of bytes received, and specifying the offset:

socket.Receive(m_ReceiveBuffer, count, m_ReceiveBuffer.Length - count);

Now, the issue here is that if it is still not enough, I am guessing that I need to grow the buffer, which means copying memory, and continue to receive into this buffer. Assuming that something went wrong, this buffer would continue to grow, and if big enough messages are received, would run the system out of memory.

Any ideas how to properly handle this? Is there a better way of receiving the data than just fill, copy, grow, fill, copy, grow that I am talking about?

Answer 1

read in chunks:

const int ChunkSize = 4096;
int bytesRead;
byte[] buffer = new byte[ChunkSize];
while ((bytesRead = socket.Receive(buffer, 0, ChunkSize, SocketFlags.None)) > 0)
{
    byte[] actualBytesRead = new byte[bytesRead];
    Buffer.BlockCopy(buffer, 0, actualBytesRead, 0, bytesRead);
    // Do something with actualBytesRead, 
    // maybe add it to a list or write it to a stream somewhere
}

Answer 2

Before starting with SYstem.Net.Sockets.Socket, are you sure that you can't use System.Net.Sockets.TcpClient (or UdpClient) that does all the messy buffer work for you and transforms it to an easily managed stream?

If not, remember that the amount of data you recieve doesn't have to be equal to what you request, so you should always look at the return value from the recieve function. And, the only way to not run out of memory is by actually processing what you recieve.

Answer 3

First, separate the code into receiving the data and processing the data. The receive buffer should only hold the data until the code gets a chance to copy it out to the processing area. The processing area is where you will determine if you have received enough data to do something useful. Don't leave the data in the network receive buffer until this happens. For the network receive buffer, I think using a circular buffer will help you with your idea of reusing a buffer. Hopefully, you have an idea of the message sizes. That will help in determining the size of the buffer. If you assert (or something similar) when the read and write pointers of the circular buffer meet, then increase the size of the buffer. Once the size of the buffer is large enough, you should be able to read enough data out of the buffer into the processing code at a rate fast enough that the circular buffer doesn't overflow.