How do I support multiple server.pid files?

Question

I am running play on multiple machines in our datacenter. We loadbalance the hell out of everything. On each play node/VM I'm using Apache and an init.d/play script to start and stop the play service.

The problem is that our play websites are hosted on shared network storage. This makes deployment really nice, you deploy to one place and the website is updated on all 100 machines. Each machine has a mapped folder "/z/www/PlayApp1" where the play app lives.

The issue is that when the service starts or stops the server.pid file is being written to that network location where the apps files live.

The problem is that as I bring up 100 nodes, the 100th node will override the PID file with it's pid and now that pid file only represents the correct process ID for 1 out of 100 nodes.

So how do I get play to store the pid file locally and not with the app files on the network share? I'll need each server's PID file to reflect that machines actual process.

We are using CentOS (Linux)

Thanks in advance Josh

Answer 1

According to https://github.com/playframework/play/pull/43 it looks like there is a --pid_file command line option; it might only work with paths under the application root so you might have to make directories for each distinct host (which could possibly be symlinks)

I have 0 experience with Play so hopefully this is helpful information.

Answer 2

I don't even think it should run a second copy, based on the current source code. The main function is:

public static void main(String[] args) throws Exception {
    File root = new File(System.getProperty("application.path"));
    if (System.getProperty("precompiled", "false").equals("true")) {
        Play.usePrecompiled = true;
    }
    if (System.getProperty("writepid", "false").equals("true")) {
        writePID(root);
    }
    :
    blah blah blah
}

and writePID is:

private static void writePID(File root) {
    String pid = ManagementFactory.getRuntimeMXBean().getName().split("@")[0];
    File pidfile = new File(root, PID_FILE);
    if (pidfile.exists()) {
        throw new RuntimeException("The " + PID_FILE + " already exists. Is the server already running?");
    }
    IO.write(pid.getBytes(), pidfile);
}

meaning it should throw an exception when you try to run multiple copies using the same application.path.

So either you're not using the version I'm looking at or you're discussing something else.

It seems to me it would be a simple matter to change that one line above:

File root = new File(System.getProperty("application.path"));

to use a different property for the PID file storage, one that's not on the shared drive.

Although you'd need to be careful, root is also passed to Play.int so you should investigate the impact of changing it.

This is, after all, one of the great advantages of open source software, inasmuch as you can fix the "bugs" yourself.

---

For what it's worth, I'm not a big fan of the method you've chosen for deployment. Yes, it simplifies deployment but upgrading your servers is an all-or-nothing thing which will cause you grief if you accidentally install some dodgy software.

I much prefer staged deployments so I can shut down non-performing nodes as needed.

Answer 3

Change your init script to write the pid to /tmp or somewhere else machine-local. If that is hard, a symlink might work.