Java Security Manager checkExec

Question

I'm having an issue trying to implement a security check in the Java Security Manager.

I seem to have misunderstood the purpose of the checkExec() method in the security manager. I thought it would get called when my application tries to invoke another Java application via the command line, but that doesn't seem to be happening.

Here's what I want to happen: An invoking application which uses my custom security manager executes "java SomeApp" on the command line. The security manager checks if this is OK and performs the appropriate action.

To try to accomplish this, I did the following:

1. Created a custom security manager which overrides the checkExec method. The method signature is public void checkExec(String cmd).

2. Created a test class, SecurityTest.

3. SecurityTest assigns my custom security manager as its security manager.

4. SecurityTest then runs java InvokeMe on the command line.

5. The custom security manager intercepts this and does something.

1 through 4 are fine, but 5 never happens. I've verified that my custom security manager is valid, it is successfully assigned to SecurityTest, and SecurityTest successfully runs InvokeMe. However, to the best of my ability to discern, the checkExec() method in my custom security manager never gets called.

Where am I going wrong? Is what I'm trying to do (have the security manager do something when an application invokes java SomeApp) even possible?

Thank you for your time.

EDIT:

Here is some code and output that results from running a test. This is rapid-prototype style, "crank it out and clean up later" style code so it's not going to be beautiful:

Custom security manager:

import java.util.*;
import java.io.*;
class DTESecurityManager extends SecurityManager {
    // instance variables

    DTESecurityManager() {
        super();
        //assign a bunch of variables
        System.out.println("TEST 1");
    }

    public void checkExec(String cmd) {
        if (cmd.toLowerCase().startsWith("java ")) {
            if (cmd.matches("(?i).*-djava.security.manager.*")) {
                throw new SecurityException("Cannot assign a security manager to invoked Java applications.");
            }
            StringBuffer intermediateCommand = new StringBuffer(cmd).insert(5, "-Djava.security.manager=DTESecurityManager ");
            String modifiedCommand = new String(intermediateCommand);
            try {
                Runtime rt = Runtime.getRuntime();
                Process pr = rt.exec(modifiedCommand);
            } catch (Exception e) {
                System.err.println("Error: " + e.getMessage());
            }
            try {
                FileWriter fstream = new FileWriter("Verification.txt");
                BufferedWriter out= new BufferedWriter(fstream);
                out.write("I worked.");
                out.close();
            } catch (Exception e) {
                System.err.println("Error: " + e.getMessage());
            }
            throw new SecurityException("Command was executed, but was modified to force invoked application to use this security manager.");
        }
    }
}

SecurityTest:

import java.io.*;

class SecurityTest {

    public static void main(String[] args) {
        try {
            System.setSecurityManager(new DTESecurityManager());
        } catch (Exception e) {
            System.out.println("Error: " + e.getMessage());
        }
        try {
            System.out.println("If we got this far, we had no problems setting our security manager.");
            System.out.println("Now let's try to invoke another Java application.");
            Runtime rt = Runtime.getRuntime();
            Process pr = rt.exec("java InvokeMe");
            System.out.println("I reached the end of my test without crashing. Now terminating.");
        } catch (Exception e) {
            System.out.println("Exception message: " + e.getMessage());
        }
    }

}

InvokeMe:

import java.io.*;

class InvokeMe {

    public static void main(String[] args) {
        System.out.println("I, InvokeMe, have been invoked successfully.");
        try {
        FileWriter fstream = new FileWriter("InvokeMeWasCalled.txt");
            BufferedWriter out= new BufferedWriter(fstream);
            out.write("It worked.");
            out.close();
        } catch (Exception e) {
            System.err.println("Error: " + e.getMessage());
        }
    }

}

Test output:

TEST 1
If we got this far, we had no problems setting our security manager.
Now let's try to invoke another Java application.
I reached the end of my test without crashing. Now terminating.

There is no indication that checkExec() is getting called -- no console output, and Verification.txt doesn't exist anywhere on my filesystem. (Of course, I'm not sure what a security manager subclass' behavior is supposed to be when performing actions like this -- so maybe that's normal. If that's the case, I need another quick-and-dirty method to test whether I'm hitting certain lines of code.) InvokeMeWasCalled.txt does exist, so that part is definitely working.

Answer 1

The issue is in your security manager's overriden checkExec. The cmd parameter passed to checkExec only contains the first "word" in the command being executed, so in your program, "java" is the cmd parameter. As a result, the string does not start with "java ", and checkExec does not execute the if block, resulting in a "clean" check from the security manager.

public void checkExec(String cmd) {
    System.out.println(cmd); // prints "java"
    if (cmd.toLowerCase().startsWith("java ") { ... } // is false
    // no security exception is thrown, allowing your command
}

Also, as a side note, the modified command execution will result in a StackOverflowError because of only the first "word" being passed to the method.

Answer 2

I know the code wasn't meant to be pretty, but it appears to be using a blacklist approach, ie "if we recognise that this is an unwanted pattern, then we throw an exception." That's not robust enough for the real world, and it relates to the actual problem ('cmd' not meaning what you thought it did).

For starters, after your 'if' block, there should be 'else super.checkExec()' to make the regular processing run if we don't want to do something special. That would have picked up your problem right there.