“java.lang.OutOfMemoryError: unable to create new native thread” while running Storm topology

Question

I'm using kafkaspout to read messages from kafka queue into storm topology, It was running fine for a last whole week, but from yesterday it was giving me below error saying Out of memory. I have checked all the memory related parameters and memory leakages in code but all seems to be fine,

    java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method) ~[na:1.7.0_45]
    at java.lang.Thread.start(Thread.java:713) ~[na:1.7.0_45]
    at backtype.storm.util$async_loop.doInvoke(util.clj:421) ~[storm-core-0.9.0-rc3.jar:na]
    at clojure.lang.RestFn.invoke(RestFn.java:559) [clojure-1.4.0.jar:na]
    at backtype.storm.daemon.executor$eval9473$fn__9474.invoke(executor.clj:732) ~[na:na]
    at clojure.lang.MultiFn.invoke(MultiFn.java:167) ~[clojure-1.4.0.jar:na]
    at backtype.storm.daemon.executor$mk_executor$fn__9342.invoke(executor.clj:317) ~[na:na]
    at backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:316) [na:na]
    at         backtype.storm.daemon.worker$eval9968$exec_fn__6466__auto____9969$iter__9974__9978$fn__9979.        invoke(worker.clj:360) [na:na]
    at clojure.lang.LazySeq.sval(LazySeq.java:42) [clojure-1.4.0.jar:na]
    at clojure.lang.LazySeq.seq(LazySeq.java:60) [clojure-1.4.0.jar:na]
    at clojure.lang.Cons.next(Cons.java:39) [clojure-1.4.0.jar:na]
    at clojure.lang.LazySeq.next(LazySeq.java:92) [clojure-1.4.0.jar:na]
    at clojure.lang.RT.next(RT.java:587) [clojure-1.4.0.jar:na]
    at clojure.core$next.invoke(core.clj:64) [na:na]
    at clojure.core$dorun.invoke(core.clj:2726) [na:na]
    at clojure.core$doall.invoke(core.clj:2741) [na:na]
    at   backtype.storm.daemon.worker$eval9968$exec_fn__6466__auto____9969.invoke(worker.clj:360)   [na:na]
    at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.4.0.jar:na]
    at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
    at clojure.core$apply.invoke(core.clj:601) [na:na]
    at backtype.storm.daemon.worker$eval9968$mk_worker__10024.doInvoke(worker.clj:329) [na:na]
    at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.4.0.jar:na]
    at backtype.storm.daemon.worker$_main.invoke(worker.clj:439) [na:na]
    at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.4.0.jar:na]
    at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
    at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.0-rc3.jar:na]
    2014-04-11 13:56:11 b.s.util [ERROR] Async loop died!
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method) [na:1.7.0_45]
    at java.lang.Thread.start(Thread.java:713) [na:1.7.0_45]
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) ~[na:1.7.0_45]
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360) ~[na:1.7.0_45]
    at      java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132) ~[na:1.7.0_45]
    at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:641) ~[na:1.7.0_45]
    at com.netflix.curator.framework.state.ConnectionStateManager.start(ConnectionStateManager.java:83) ~[curator-framework-1.0.1.jar:na]
    at com.netflix.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:168) ~[curator-framework-1.0.1.jar:na]
    at storm.kafka.DynamicBrokersReader.<init>(DynamicBrokersReader.java:35) ~[stormjar.jar:na]
    at storm.kafka.trident.ZkBrokerReader.<init>(ZkBrokerReader.java:23) ~[stormjar.jar:na]
    at storm.kafka.trident.KafkaUtils.makeBrokerReader(KafkaUtils.java:26) ~[stormjar.jar:na]
    at storm.kafka.KafkaSpout.open(KafkaSpout.java:67) ~[stormjar.jar:na]
    at backtype.storm.daemon.executor$eval9403$fn__9404$fn__9419.invoke(executor.clj:504) ~[na:na]
    at backtype.storm.util$async_loop$fn__5748.invoke(util.clj:401) ~[na:na]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
    at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
    2014-04-11 13:56:11 b.s.d.executor [ERROR]
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method) [na:1.7.0_45]
    at java.lang.Thread.start(Thread.java:713) [na:1.7.0_45]
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) ~[na:1.7.0_45]
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360) ~[na:1.7.0_45]
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132) ~[na:1.7.0_45]
    at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:641) ~[na:1.7.0_45]
    at com.netflix.curator.framework.state.ConnectionStateManager.start(ConnectionStateManager.java:83) ~[curator-framework-1.0.1.jar:na]
    at com.netflix.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:168) ~[curator-framework-1.0.1.jar:na]
    at storm.kafka.DynamicBrokersReader.<init>(DynamicBrokersReader.java:35) ~[stormjar.jar:na]
    at storm.kafka.trident.ZkBrokerReader.<init>(ZkBrokerReader.java:23) ~[stormjar.jar:na]
    at storm.kafka.trident.KafkaUtils.makeBrokerReader(KafkaUtils.java:26) ~[stormjar.jar:na]
    at storm.kafka.KafkaSpout.open(KafkaSpout.java:67) ~[stormjar.jar:na]
    at backtype.storm.daemon.executor$eval9403$fn__9404$fn__9419.invoke(executor.clj:504) ~[na:na]
    at backtype.storm.util$async_loop$fn__5748.invoke(util.clj:401) ~[na:na]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
    at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
    2014-04-11 13:56:11 b.s.util [INFO] Halting process: ("Worker died")
    2014-04-11 13:56:11 b.s.util [INFO] Halting process: ("Worker died")

My storm cluster parameters are: nimbus.childopts -Xmx1024m worker.childopts -Xmx768m drpc.childopts -Xmx768m

Answer 1

The inability to create a new native thread has got nothing to do with the heap, it is actually a proxied message from the OS who is unable to allocate more native memory to create a new thread and/or process at OS level.