How do you fix “Too many open files” problem in Hudson?

Question

We use Hudson as a continuous integration system to execute automated builds (nightly and based on CVS polling) of a lot of our projects.

Some projects poll CVS every 15 minutes, some others poll every 5 minutes and some poll every hour.

Every few weeks we'll get a build that fails with the following output:

FATAL: java.io.IOException: Too many open files
java.io.IOException: java.io.IOException: Too many open files
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)

The next build always worked (with 0 changes) so we always chalked it up to 2 build jobs being run at the same time and happening to have too many files open during the process.

This weekend we had a build fail Friday night (automatic nightly build) with the message and every other nightly build also failed. Somehow this triggered Hudson to continuously build every project which failed until the issue was resolved. This resulted in a build every 30 minutes or so of every project until sometime Saturday night when the issue magically disappeared.

Answer 1

This is Hudson issue 715 (http://issues.hudson-ci.org/browse/HUDSON-715). The current recommendation is to set the 'maximum number of simultaneous polling threads' to keep the polling activity down.

Answer 2

See https://wiki.jenkins-ci.org/display/JENKINS/I%27m+getting+too+many+open+files+error for what we need from you to fix this kind of problem.

Answer 3

Change system limits for per-process maximum open file descriptors? As in ulimit -n for the Java process?

Answer 4

I have experienced this problem with another Java application running on Debian, it went away when we downgraded to Java version 1.6.0.0. Java never closed unused connections, causing it to throw the exception.

Answer 5

One of the most common problem causing "Too many open files" is to have Active Directory plugin enabled and configured in Jenkins. There are known issues with this plugin which cause enormous number of threads to show up and "Too many open files" error in logs as well. After disabling it and switching to LDAP authentication I did not experience Jenkins to hang anymore.