Java HttpGet, Too many open files in service error

Question

I am using the RESTFul HttpGet service from

import org.apache.http.client.methods.HttpGet;

I created a function to query a service on a different computer. The call works fine, however I run into a "too many open files" on the service. My side simply returns a 500 error, which I catch.

I spoke with the vendor, and they were quite adamant that the RESTful call should be made in a persistent way and that I am simply not freeing something and continued on that the problem is on my side.

I wrote a stress function, see below, to help isolate the problem. Too my knowledge, I am releasing everything. I am still new to Java so many I am just not seeing something that was not clear.

import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;

public void doStressTest()
{
    String strUri = "http://<ip address>:<port>/task?vara=dataa&varb=datab";
    HttpGet oRestyGet = new HttpGet();
    oRestyGet.addHeader("Accept", "application/xml");

    for (int iLoop = 0; iLoop < 1000; iLoop++)
    {
        CloseableHttpClient httpclient = HttpClients.createDefault();
        try
        {
            oRestyGet.setURI(new URI(strUri));
            try
            {
                CloseableHttpResponse response2 = httpclient.execute(oRestyGet);
                try
                {
                    String strResponseHeader = response2.getStatusLine().toString();
                    if (false == "HTTP/1.1 200 OK".equalsIgnoreCase(strResponseHeader))
                        return;

                    continue;
                }
                finally
                {
                    response2.close();
                }
            }
            catch (ClientProtocolException e)
            {
                e.printStackTrace();
            }
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
        finally
        {
            try
            {
                httpclient.close();
            }
            catch (Exception e)
            {
                e.printStackTrace();
            }
        }
    }

    try
    {
        oRestyGet.releaseConnection();
    }
    catch (Exception e)
    {
        e.printStackTrace();
    }
}

UPDATE: I thought that I add the text of what the vendor had to say, just in case it might help. I should add that in the stress test below I am opening a single HTTP GET request object, which for my "session" is persistent and used for the entire request.

This means that the system is running out of FDs for the user running the process. This happens when there are too many open sockets or FDs. In general with HTTP it is highly recommended the HTTP requests are sent in a persistent way, that is you have one HTTP connection for the entire session and not opening multiple HTTP connections then closing them for each request. More often then not you end up with lingering connections.

I am closing the "closeable" CloseableHttpClient, and nor do I think that I am required to have only one. I should add that "I" am not running out of anything, the service is. The vendor seems to say that the service is perfect as is the OS. What more can be done to isolate the problem?

UPDATE 2: (I sent the log file from the service to the vendor, and they had the following to say. Stalemate?)

It looks like the problem is that there are too many connections left open by your interface. I included the log file from a session by Firefox. You can see how I can make multiple HTTP requests over the same connection using a browser.

There is only one line where the initial connection is made

HTTP connection from [127.0.0.1]...ALLOWED

the subsequent GET requests are from the same connection.

...your logfile shows: HTTP connection from [192.168.20.123]...ALLOWED

for each GET request you make, and those connections are not closed.

UPDATE 3: I have access to the log that the service generates, although no access to the source. Java issues the connection and the GET requests as one package in response to the line:

CloseableHttpResponse response2 = httpclient.execute(oRestyGet);

I am issuing the httpclient.close(), which generates no log entry, so I suspect that the service simply does not respond to that.

As I am unfamiliar with the mechanics of the other end, maybe the service simply responds to events, and the problem is with CENTOS not handling the close() call properly. Either way, using this method, the problem is not mine. Proving it is another story. The alternative is some sort of other solution that frees resources properly.

Answer 1

Tom G was right in his response that the problem was on the vendor side, but the reason that I created another answer is that although the vendor did not close a file handle that was opened, the stress test still crashed.

I had to switch to Jersey, which solved the "httpclient.execute()" performing a open/close operation with every GET request, and then the vendor had to give me a build that fixed the issue on their end, hence two issues.

The apache framework appears to have problems, at least with my implementation, while the Sun framework (Jersey) worked nicely in my stress test.

Now I need to test with the real code.

Answer 2

If you're getting an HTTP 500 Internal Server Error, that has nothing to do with your client code, or how you manage HTTP connections. The problem is entirely on the vendor side.

Answer 3

I would use lsof to see what files are being held open. You could also look at netstat -n -a to see all of the connections that are open. Have you tried bumping up the open file limit to see if you can get around the problem that way?

Too many open files ( ulimit already changed )