JVM abrupt shutdown on creating threads and doing network operation in them

Question

This is an issue with I am trying to fix, maybe a bug in JRE please vote to reopen this question. It's very selfish approach to mark something off topic if you are not competent enough to understand some problem.

Java Version

java version "1.7.0_45" Java(TM) SE Runtime Environment (build 1.7.0_45-b18) Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

Following both version of codes produces the same crash.

Client machine call a Servlet deployed in Tomcat 7 using a Server Maching that is running on Windows Server 2003 64-bit, Using Oracle 10g database and Hibernate 3.0.

When I try to put load on Servlet for load testing I wrote following two methods that just creates a thread and call Synchronous method but without any stacktrace of log My Process in Eclipse termiates.

I don't know the reason of crash if I create 500 threads then it crashes at 487 and If I create 1000 threads then it crashes at 874. I don't know what the real reason is. I have configured a StuckThreadDetectionValve to detect long running Threads.

https://stackoverflow.com/a/7484800/185022

I'll share all the updates here. Please give me a hint at least if you know that what could possibly be wrong.

** Both codes produces the same crash **

Code 1:

public class TokenTest {
    static int count = 0;

    public static void main(String[] args) {
        try {
            System.out.println("Start " + new Date());
            final ApplicationServiceClient app = WSClient.getClient("http://192.168.1.15:81/az-server", ApplicationServiceClient.class);
            final int max = 50;
            ExecutorService executor = Executors.newFixedThreadPool(max);

            for (int i = 1; i <= max; i++) {

                executor.execute(new Runnable() {
                    public void run() {
                        try {
                            System.out.print(++count +",");
                            app.verifyToken("7056451004350030976"); //Network Synchronous Operation, calls a Servlet that is deployed in another Server Machine, Using Tomcat 7.0
                        } catch (Exception ex) {
                            ex.printStackTrace();
                        }

                    }
                });
            }
            executor.shutdown();

            while (!executor.isShutdown()) {
                System.out.println("END " + new Date());
            }
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }   

}

Code 2:

public class TokenTest {
    static int count = 0;

    public static void main(String[] args) {
        try {
            System.out.println(new Date());
            final ApplicationServiceClient app = WSClient.getClient("http://192.168.1.15:81/az-server", ApplicationServiceClient.class);
            final int max=250;
            for (int i = 1; i <= max; i++) {
                Thread t = new Thread() {
                    public void run() {
                        app.verifyToken("7056451004350030976");//Network Synchronous Operation
                        System.out.print(count++);
                        System.out.print(", ");
                        if (count >= max)
                            System.out.println("\n" + new Date());
                    }
                };

                t.start();
            }

        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }



}

Note: The server is hosted on another Windows Server 2003 64-bit machine.

Updated It just terminates my Java process in Eclipse and If I give value like 5000 then it stuck on 2800 something and nothing happens neither it terminates my process. Here is a console snapshot.

Update* Thank to S.Yavari. He led to the right direction. After configuring Stuck_Thread_Detection_Valve on Tomcat and interrupting the Threads that Stuck Threads I am able to generate load of 500K clients on server without any problem and Also giving a Sleep of 100 miliseconds before starting new request.

Below is the code :

public class StageOne {

    final ApplicationServiceClient appClient = WSClient.getClient(Main.url, ApplicationServiceClient.class);
    final SecureWebServiceClient secureClient = WSClient.getClient(Main.url, SecureWebServiceClient.class);
    final static List<Long> times = new Vector<Long>();
    String token = "";

    public String authenticateClient() {

        token = appClient.authenticateClient("az_y", "az", "0");
        // System.out.println("Registration Response: " + token);
        return token;
    }

    private void getDefaultDepartmentName() {

        String deptname = appClient.getDefaultDepartmentName();
        // System.out.println("Department: " + deptname);
    }

    private void getLocation() {

        List<String> location = appClient.listLocation();
        // System.out.println("Location: " + location);
    }

    private void registerClient_() {
        secureClient.setToken(token);
        String respon = secureClient.registerClient("Buddy AZ", "443", "njuy", "1");

        // System.out.println(respon);

    }

    static int count = 0;

    public static void main(String[] args) {

        int pool = 500000;

        final ExecutorService execotors = Executors.newFixedThreadPool(pool);

        for (int i = 0; i < pool; i++) {
            try {
                Thread.sleep(100);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            execotors.execute(new Runnable() {

                @Override
                public void run() {

                    long startTime = System.nanoTime();

                    StageOne one = new StageOne();

                    one.authenticateClient();
                    one.getDefaultDepartmentName();
                    one.getLocation();
                    one.registerClient_();

                    long EndTime = System.nanoTime();

                    // System.out.println("Time in Seconds ("+ (++count) +")" + TimeUnit.NANOSECONDS.toSeconds(EndTime - startTime));
                    times.add(TimeUnit.NANOSECONDS.toSeconds(EndTime - startTime));
                }
            });
        }
        execotors.shutdown();

        while (!execotors.isShutdown()) {
            try {
                Thread.sleep(10);
            } catch (Exception e) {
                // TODO: handle exception
            }
        }

        Collections.sort(times);
        long sum = 0;
        for (int i = 0; i < times.size(); i++) {
            sum += times.get(i);
        }
        long result = sum / Math.max(1, times.size());


        System.out.println("Min Response Time: " + times.get(0));
        System.out.println("Average Response Time: " + result);
        System.out.println("Max Response Time: " + times.get(times.size() - 1));

    }

}

Answer 1

I think this is just a Threading problem. Use this code:

public class Test2 {
    public static final Object LOCK = new Object();
    static int count = 0;

    public static void main(String[] args) {
        try {
            System.out.println(new Date());
            final ApplicationServiceClient app = WSClient.getClient("http://192.168.1.15:81/az-server", ApplicationServiceClient.class);
            final int max=250;
            for (int i = 1; i <= max; i++) {
                Thread t = new Thread() {
                    public void run() {
                        synchronized (LOCK) {
                            app.verifyToken("7056451004350030976");
                        }
                        System.out.print(count++);
                        System.out.print(", ");
                        if (count >= max)
                            System.out.println("\n" + new Date());
                    }
                };

                t.start();
            }
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }
}

This method (verifyToken) must be thread safe, otherwise you will have so many problems using this method inside threads.

Answer 2

In code 1, the code is not crashing. It is working as designed. There doesn't seem to be any problem with that code.