node.js process per match

Question

This is the sort of the same question as before:

node.js child processes

I'm asking about whether or not I should use a child process per match for my node.js game.

But I realized that previously I neglected to include some very important details.

The game allows players to manipulate game rules in certain limited ways. However this can still lead to infinite loops / memory leaks / stalls and crashes.

Is 1 process per match a scalable / reasonable idea?

Answer 1

If any single game process can eat up all the memory or CPU, this isn't scalable. If your server is an 8-core machine, eight games can take all of the CPU time, there's nothing you can do, except for monitoring processes via top and killing them as needed - but that would make for a bumpy server.

Now, if you manage to prevent this stuff in the first place (sounds like a better idea to me), it is viable. Each process will take upwards of 30mb of memory, so you'll need a beefy server for every couple hundreds. Look at http://site.nodester.com for an example, they seem to be running around 600 processes on a single machine. Their software stack is open source too: https://github.com/nodester/nodester

node v0.8 will bring Isolates (shared-nothing threads), which will probably use less resources than a child process.

A more "serious" solution to this would be using some kind of virtualization like OpenVZ that will allow you to set resource limits, then just keep a pool of virtual servers available, every game gets it's own. It's not as heavy as it looks, it's ~18mb overhead per server and you can host hundreds per machine, although it's a much more complex setup.

Answer 2

The short answer is no, it won't scale!

The long answer

Let's first look at scalability. I'm taking the definition proposed by wikipedia: "scalability is the ability of a system, network, or process, to handle growing amount of work in a graceful manner or its ability to be enlarged to accommodate that growth."

In case one of your processes can eat up as much CPU as it's granted by the scheduler (More details on the linux scheduler) your system won't scale in a "graceful manner"! So what you would need for scalability is a setup as proposed by Ricardo Tomasi above where every match would need it's own VM. But that's not graceful and taking cost into consideration this is no viable solution.

The problem why your system won't scale is the algorithm behind it, no architecture can fix that but the algorithm needs to be fixed.

Your options to fix the algorithm

• Try to use some blocking mechanism in the game loop
• Counters to detect infinite loops
• Build an event queue that has limited slots so that adding additional events will throw exceptions
• Use a time slots algorithm for your game loop somehow like every match can consume 1/count(matches) time of one node.js process (but avoid building your own scheduler)

And even when your algorithm is fixed, spawning a process for each match will eat up some MB of your limited RAM resource which is not graceful in the sense of scalability.

Answer 3

I think that actor model should definitely provide scalability:

• Use a process pool with a load balancing mechanism
• Use ZeroMQ or something similar to exchange message
  • you will need a few communication channels here
  • use request/response for handshaking and control
  • you can use multicasting for the main channel
  • you can also use publish/subscribe if you have specific use for it too

You'll probably need a master process to do balancing and control all of the processes, keep track of events etc. Using the actor model you can scale out across a network of machines and in fact you can even have peer-to-peer clusters if you desire to do so, might need to use RSA keys for authentication though. I'd recommend to start with a single master just waiting for processes to connect, the implement the worker skeleton and see how to implement the control side of things. Stick to master with 2 workers for starters, it's simpler to debug.

For the web front end you can also use a proxy, such as Nginx, which will call the master and then the master will tell it where to direct the new client. I suppose you would need to implement a UI module and then use it from within the worker. I mean that the master will present no UI, and your workers will listen on different ports, though Nginx will hide it away from the user (he will see no ports in the URL bar) and you could also implement a RESTful API on top of that too.

Answer 4

This is a hard question to answer, because we don't know what your game do ...

If it crash, all games will crash, so i assume having multiple process is a good idea.

But i dont see any other good reason why you should have multiple process. ( Maybe a ton of blocking operation like huge DB transaction, process huge file ... etc)

Like @Samyak Bhuta said, you could use forever or cluster to restart your process. We are using monit for this

Answer 5

There are multiple things to discuss here.

How many players can connect to a match?

What kind of database are you using? Does it have fast writes?

Restarting a process is the last solution, because if you have a game where everything should happen fast and you restart the process it will take a couple of seconds before the players reconnect to it.

I don't think one process per match is scalable, what happens when you have 50.000 matches at the same time for example? I would say that a better solution would be to group matches on a child process by 2 criteria:
a) by the match id (some kind of sharding algorithm)
b) if more and more players come popping up spin another process (or even more), based on the number of players.

It's really hard to decide what to do before making some tests on your game. You should really test it to see how it behaves several real matches (how much CPU it "eats", memory) and make some analysis based on the data. Nobody can say exactly what to do in these kinds of situations, because it depends on the project.