Choosing buffer size for FTP and HTTP transfers

Question

How does one choose the size of a buffer (bytes I read from or write to socket) for the maximum throughput when implementing a low-level HTTP and FTP transfer? My application should transfer data with HTTP or FTP on connections varying from 130 Kbps to 3 Mbps (I know the expected speed beforehand). Sometimes it's a one way transfer, sometimes it goes in both directions. Should I stick with some average buffer size or I must vary it depending on the connection speed?

Thanks.

Answer 1

First, get some measurements.

Then, after you have a reliable performance measurement, make changes to your buffer size and plot a graph of speed vs. buffer size.

Since you know the connection speeds in advance, you should be able to get some measurements of actual speeds with different actual buffer sizes.

The OS, protocol stack and network is too complex to work out an answer from first principles. You need to measure before you do anything.

Answer 2

Choose a buffer size over 8KB. 9000 is typically the largest MTU (maximum transmission unit) size used in even the fastest networks.

When you use a buffer larger than the MTU of the connection, the operating system will break it down in to MTU sized pieces as needed, and thus anything you use over the MTU will have little effect on network performance.

However, using a large buffer will likely have other effect on performance, if you're transferring files, then using large buffers may increase the read performance, thus improving the speed of your application.

So, Usually picking a nice round number like 16KB is a good idea. Definitely don't go under 1500, as this can negatively effect network performance (causing the operating system to sometimes send small packets, which decrease performance on the network).

Answer 3

You can get buffer size: httpSocket.getReceiveBufferSize(); I highly recommend writing this buffer to RAM in the process of reading. For example: ByteArrayOutputStream memoryFile. and only after filling the buffer write it to disk.