Program runs out of memory after resolving many domains under a security manager?

Question

I have a service where domain names to be resolved come from untrusted sources. Recently, it kept crashing from running out of memory. I narrowed down the possible reasons, and came to the conclusion that it must be related to the recent flux of DNS requests. However, the service doesn't store anything after resolving domain names, so this seemed unlikely, but I tried spamming my service with requests that result in it resolving domain names, just in case. It did die from that. I then narrowed down the code to this after concluding that there is no memory being stored on my code's behalf:

import java.math.*;
import java.net.*;
class A {
    static {
        try {
            for (BigInteger i = BigInteger.ZERO; i==i; i = i.add(BigInteger.ONE))
                Inet4Address.getByName("a"+i+".dog");
        } catch (Exception e) {throw new RuntimeException(e);}
    }
}

I setup dnsmasq with this line in /etc/dnsmasq.conf to make the resolutions faster:

address=/dog/127.0.0.1

At first when I ran this, it survived for days, so it seemed this was not the problem. But then I ran it with my script that I use to start my service, which enables the security manager, and it crashes:

$ javac A.java && java -Xmx80m -Djava.security.manager A
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
Could not find the main class: A. Program will exit.

The security manager makes my program vulnerable to this denial of service attack. Why? How to fix it?

---

$ java -version
java version "1.6.0_27"
OpenJDK Runtime Environment (IcedTea6 1.12.6) (Gentoo build 1.6.0_27-b27)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

Answer 1

" InetAddress class has a cache to store successful as well as unsuccessful host name resolutions. By default, when a security manager is installed, in order to protect against DNS spoofing attacks, the result of positive host name resolutions are cached forever. When a security manager is not installed, the default behavior is to cache entries for a finite (implementation dependent) period of time. The result of unsuccessful host name resolution is cached for a very short period of time (10 seconds) to improve performance."

source: javadoc for InetAddress.java http://docs.oracle.com/javase/6/docs/api/java/net/InetAddress.html

I think it is caching all the hostnames resolved forever and causing outOfMemoryError finally.

You can try setting Java security properties control the TTL( time to live) and see if this helps in solving this problem

Answer 2

You are executing a long running code in a static block.

When JVM loads a class, it first executes the static block. My suspicion is that the JVM is unable to load the class because of your code (which is essentially a long running operation) and crashes.

Usually you do simple initialization steps in a static block. Move your code to the static main method and try again. See if it improves the run time ?

Answer 3

Why don't you want to do a basic profiling? Several issues are possible, so five minutes with a proper tool will definitely help and save your time.

1. Your example without SecurityManager

2. Your example with SecurityManager

As was expected memory is more heavily used when SecurityManager is enabled. I haven't setup any DNS infrastructure, so request rate was not that high.

Important to note:

• garbage collection takes some time
• in some cases default GC does not provide enough "throughput", so eventually memory is getting exhausted
• VM memory model is not that simple
• default VM does not provide and real-time performance guaranties

I would suggest to avoid any tweaking until you do some profiling on your environment with your settings. We can spend a lot of time here trying to suggest the root cause, but you'll find the answer much faster. Please try to use profiler and do not forget to share your results. This is quite an interesting topic.

Thank you.

Answer 4

I find that java.lang.OutOfMemoryError: Java heap space is often misleading and may mean your Perm Space is exhausted rather than your tenured heap for objects.

Try increasing your perm space with: -XX:MaxPermSize=128M

The security manager no doubt increases the number of classes loaded into Perm.

Answer 5

Is the crash reproducible with -Djava.compiler=NONE? I'm wondering if you are running into a JIT compiler bug.

Also, I'm curious if the failure is the same when running the for loop inside a static main method. I'm wondering if executing a long running for loop in the static block while the class is being loaded is a factor.

Answer 6

I still believe problem is related to OpenJDK.

There are 2 JDKs that are of production quality: one of Sun (now Oracle) and one of IBM.

Gentoo is fine Linux, but I think that you could not build high quality JDK from GNU open source. At least now. Sad, but true.

Try to reproduce it with Sun JDK. http://www.gentoo.org/doc/en/java.xml

Comment about static initialization block is worth attention.