Running and maintaining a 'live' mega project

Question

I have looked around but couldn't find an answer for my requirements of managing a large online project. By large, I mean a web based service that depends on multiple back-end technologies, massive amount of diverse computations on the fly, a lot of storage requirements; and by live, I mean that on the front-end it tries to serve a massive user-base and has to be up all the time (I think Google would be a perfect example).

In my case, I am developing a forensics tool and I have to meet at least two requirements:

1. That my tool/ service is always on and available.
2. That I or my tool can keep on improving the tool/ service itself.

At this moment, I can't manage to meet both the above points together; if I do one, I violate the other, but I'm sure there's a "standard" way around. I am sure my requirements (even though they look contradictory to me at the moment) are no different from say what Google and Facebook do on a regular basis.

I hope it's clear by now that I am not asking abouthow to manage complexities/ fixes that have to be addressed as software projects tend to get bigger. I think this question answers that quite well- How to keep a big and complex software product maintainable over the years?

Answer 1

One 'standard' way of deploying with no downtime is to use green-blue deployments

You need

• a minimum of two servers running your website/app
• load balancing of requests between the servers

Then do the following

1. divide your servers into two groups, green and blue
2. take the green group out of load and wait for the connections to drop
3. deploy your updated version to the green servers.
4. put the green servers back in load
5. take the blue servers out of load
6. deploy your updated version to the blue servers.
7. put the blue servers back in load

Now there is some trickyness around point 4/5 where you have both versions of the application live at the same time. This can cause issues with requests from the same client hitting different versions of the service

You can handle this situation in a number of ways

• Sticky Sessions. once a user is connected to one group they stay with that group.

• Backwards compatibility. The new server response is backwards compatible with the old version

• Routing by request version. The client adds a version to the request, enabling the load balancer to route old requests to the old version servers and new requests to the new version.

  In this model you leave the old group un-upgraded until the clients have all upgraded.

A couple of notes:

Two is really not enough, if you want fail over during the deployment then you need more than one box live at any given time. But usually with large applications we are talking about 10's of servers in any group.

In my example I have both green and blue in load at the same time during normal operation. But I believe the traditional method is to have one only of the two live. Obviously this can mean having expensive boxes lying idle. But in these heady days of 'The Cloud' you can do clever tricks like spining up a whole new set of 'blue' boxes and retiring your 'green' ones