How to optimize upload routine using Delphi 2010?

Question

My yet to be released Delphi 2010 application allows users to upload their files to my servers. Right now I'm using HTTPS POST to send the files, the (simplified) algorithm is basically:

1. Split File into "slices" (256KB each)
2. For each slice, POST it to server

ie. for a 1MB file:

--> Get Slice #1 (256KB)
--> Upload Slice #1 using TidHTTP.Post()

--> Get Slice #2 (256KB)
--> Upload Slice #2 using TidHTTP.Post()

--> Get Slice #3 (256KB)
--> Upload Slice #3 using TidHTTP.Post()

--> Get Slice #4 (256KB)
--> Upload Slice #4 using TidHTTP.Post()

I'm using Indy 10. I (ab)used my profiler over and over and there are not much left to optimize except changing the upload routine itself.

I'm also using multi-threading, and even though I did my best to optimize my code, my benchmarks still tell me I can do better (there are other well optimized software that do achieve a much better timing...almost twice as fast as my upload routine!)

I know it's not my server's fault...here are the ideas that I still need to explore:

1. I tried grouping slices in a single POST, naturally this resulted in a performance boost (20-35%) but resuming capability is now reduced.

2. I also thought about using SFTP / SSH, but I'm not sure if it's fast.

3. Use web sockets to implement resumable upload (like this component), I'm not sure about speed either.

Now my question is: is there something I can do to speed up my upload? I'm open to any suggestion that I can implement, including commandline tools (if license allows me to ship it with my application), provided that:

1. Resumable upload is supported
2. Fast!
3. Reasonable memory usage
4. Secure & allow login/user authentication

Also, because of major security concerns, FTP is a not something I'd want to implement.

Thanks a lot!

Answer 1

I would suggest doing a single TIdHTTP.Post() for the entire file without chunking it at all. You can use the TIdHTTP.OnWork... events to keep track of how many bytes were sent to the server so you know where to resume from if needed. When resuming, you can use the TIdHTTP.Request.CustomHeaders property to include a custom header that tells the server where you are resuming from, so it can roll back its previous file to the specfiied offset before accepting the new data.