Question

I have a simple homogenous (one controller, several server), cluster deployment for 5 appscale VMs. Under testing they ran within the max and min boundaries I set for the number of appengine proceses I wanted to run. Then, under load, one node goes down. The rest keep running. My separate load balancer handles it well. I'm happy that the redundancy worked to give me resilience. Nice. But how do I bring that node bad back into play?

If the quick answer is "it's not supported", then my real question is: how would I go about hacking this, using cassandra as my datastore?

No correct solution

OTHER TIPS

There's a command called appscale-add-instances that will take care of this for you. There's more documentation here, but the summary is:

  1. Make sure your new machines have the same version of AppScale installed as the currently running machines.
  2. Write a YAML file that includes their IP addresses. An example of adding a single node would be:

    memcache: 192.168.10.3

    taskqueue_slave: 192.168.10.3

    appengine: 192.168.10.3

This indicates that we have one machine that runs the memcache, taskqueue, and appengine services.

  1. Run appscale-add-keypair --add_to_existing --ips your-yaml-file.yaml to get the SSH keys synced up on the new nodes. Skip this step if running on Amazon EC2, Google Compute Engine, or Eucalyptus.
  2. Run appscale-add-instances --ips your-yaml-file.yaml to add the new nodes!

Old answer (saved for posterity, and a walkthrough of how we implemented this support):

So the quick answer is "it's not supported". I was seeing the case of adding crashed nodes as very similar to the case of adding new nodes in general, so the way to hack AppScale to make this work is as follows:

  1. Check out the AppScale tools (AppScale/appscale-tools on github) and make a new command in bin called appscale-add-nodes, which takes a list of nodes to add to the currently running AppScale deployment.
  2. Add a new SOAP-exposed method in the main AppScale code (AppScale/appscale on github) in the AppController (AppController/djinn.rb), called add_nodes_from_ips (see djinnServer.rb for how to expose methods via SOAP and their definitions in djinn.rb to see how they implement it), which takes in a single argument, the list of IPs to start up as AppScale nodes.
  3. Have appscale-add-nodes connect to the Shadow AppController (whose IP address is in ~/.appscale/locations-appscale.yaml) and call your new SOAP-exposed method with the list of IPs to start AppScale on.

Your SOAP-exposed method should call start_appcontroller on each IP address, so that the AppController daemon is started there and can connect back to the other AppControllers. The list of roles to start is up to you - I'd recommend using the open role so that AppScale can decide what to use your new nodes for.

It's definitely a lot to take in, but I'm more than glad to update this answer in response to comments (and of course, I'll add a page on the AppScale wiki for each individual part of this, like how to add SOAP methods to the AppController in general).

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top