Uploading multiples files to Sharepoint with CSOM results in SocketException

Question

I've been using Client-Side Object Model in C# for Sharepoint2010 for a few weeks, and I've tried uploading files with File.SaveBinaryDirect and by adding the file to the list and then calling ExecuteQuery. Both methods are successful when I just upload just a few files with their size <= 3 MB.

However, let's say I try to upload 10,000 files of 10KB each using SaveBinaryDirect, it fails with a SocketException (Address already in use). It is called like 2000 times before it throws this exception, so it looks like it uses up all the ports before any Time_wait finishes waiting their default 240 seconds.

With the ExecuteQuery method, it works, but only because I upload them in batch, with the size of one batch always under 3 MB, and uploading never more than 20 files at once. So this method also has limitations: it can't upload files over 3 MB, and it also has the risk of throwing the SocketException. With 10,000 files, the method is only called 500 times. I'd guess that with a larger amount of files to upload, the method would be called over 2000 times too and throw the same error as the SaveBinaryDirect method.

Now, I don't know how to resolve the SocketException error in CSOM. Does anyone know how to resolve it?

What I could do is a try-catch, and everytime I catch this specific exception, I wait 4 minutes before uploading files again. Another thing I could do is to implement a Web client with socket using the options REUSE_ADDRESS, but I'm not sure if I'd be wasting my time, plus I'm really no expert in the matter.

OR

Does anyone know a reliable way to upload any number of files with varying size (though they must always be under the default max upload size of 50 MB) to Sharepoint? Or which APIs for Sharepoint2010 could offer me this possibility? Is it even possible to begin with?

I will probably try with the service WebCopy.asmx and/or the WCF service listdata.svc, but I'm not sure I'll be anymore successful with either of these services. They look very similar to the usage of Client.svc through CSOM.

Answer 1

So here's how the HttpWebRequest will try to keep reusing the same connection, although it appears that this method is not completely secure:

        HttpWebRequest request = (HttpWebRequest)WebRequest.CreateRequest(url);
        request.ConnectionGroupName = CONN_GROUP_NAME;
        request.UnsafeAuthenticatedConnectionSharing = true;

Setting a ConnectionGroupName is optional. You can seek more info here.

You can access the WebRequest in CSOM, and here's where:

        ClientContent ctx = new ClientContext(siteUrl);
        ctx.PendingRequest.RequestExecutor.WebRequest.UnsafeAuth...

Just be sure to have a PendingRequest first.

I hope this might be useful.