software disaster recovery when an engineer is suddenly unavailable

Question

Recently in my company we had a project where the deadline was very tight, and everything was going according to plan until I was unavailable due to extreme personal issues.

Eventually we missed the deadline by 4~5 days.

What are the usual recovery plans for these conditions? should my company attempt to outsource a developer to finish my work? even that could take a few days to find one?

Answer 1

It depends on the foreseeable duration of the unavailability, the remaining duration of the project, the way the tasks are distributed and the consequences of missing the deadlines.

Software developers are not an interchangeable at will. Developers build knowledge on the system as the system grows, and adding a new resource requires to cope with the missing contextual knowledge of the new resources.

Several good practices reduce the risks associated with sudden unavailability:

• peer reviews ensure that knowledge on the development are shared between several developers. In case of unavailability, the rest of the team could reorganise to take over, or in the worst case bring in a new coder and organise a knowledge transfer.
• integrated colocated teams that closely work together for making design decisions can mitigate unavailability in the same fashion. The shared knowledge on the overall design facilitates redistribution of work and briefing of newcomers.
• formal documentation could eventually help. However in practice this works well only if the documentation produced by one developper is used by another developper (or models used in code generation tools). Documentation that is only read by oneself frequently appears to be much more ambiguous. If a new developper has to take over such work, the self documentation might raise as much questions that it answers.

Bringing in new developers when there are tight deadlines is very challenging, because briefing the newcomers take time of the team, in a period where there's no spare time. If you're ill for 1 week it makes no sense. It's to be envisaged if the cost of briefing newcomers is compensated by the benefits of the new resource, typically if there are high costs of being late, or if it's not possible to postpone, or if the unavailability is over a longer period of time.

Answer 2

The usual plans for this is to build contingency into the deadline. Things happen, you often never hit deadlines. You being unavailable was just another hiccup in the carefully laid plans that never go according to plan!

Answer 3

This is called the "bus-factor". Basically, the risk posed to the project if a developer is hit by a bus. Having a developer unavailable for a week is a small hiccup compared to losing a developer permanently. It could be an accident or sudden illness, but less dramatically it could just be a core developer switching job on short notice or getting fired. Or a core developer getting transferred to another high-priority task in different department.

In short, you have to either plan for lowering the bus-factor, or you have to be prepared to mitigate it, e.g. by having buffers in deadlines or just be flexible enough to be able to push the deadline. What you usually cannot do is just outsource a complex task on short notice, or hire a new developer - it would almost always take more time to introduce a new developer to an existing system than to wait a week for a core developer to return. If a small team loses a member they will of course be slower, but if the also have to introduce a new member, they will be even slower.

You can lower the bus-factor by having a team continuous knowledge sharing and peer-reviews (or even pair-programming). But of course this is something which have to happen before the bus hits.

Answer 4

Any employee may become unavailable for a week, or a month, or forever, without any notice, at any time. Accident, illness, having had enough of this job, many other reasons. It is up to management to make sure that such an occasion is annoying, maybe expensive, but not a disaster.

If you have a team of ten, you might lose 10% of your team. The company should be able to handle that if the rest of the team is motivated (paying for overtime is highly motivating). If you are a team of one, then the work isn't going to be done. If you have other employees who can step in, the delay may be minimised. Hiring someone from the outside would be difficult, although you probably find some contractor who can start at a days notice for a few weeks at a very high hourly rate.

The best thing to do is to have contracts etc. in place so that a delay finishing a product is not a financial disaster. And to have a planned and achievable finishing date way ahead of the deadline. Having employees that can step in for each other helps (but can be problematic to achieve).

And if you have a deadline that must be achieved, then maybe the scope of the work is more flexible. In other words, drop features to meet your deadline.

Answer 5

One key way to reduce the "Bus Factor" that @JacquesB mentions above is pair programming as a core technique. (My own practice is to use the term "Lottery Factor" since it's less morbid but the effect is the same.)

Many developers hate pair programming; many managers hate it too, for entirely different reasons (some developers hate being forced to communicate with other humans for extended periods; some managers often feel incorrectly as though they are paying twice as much money for a single output).

But pair programming all-but-eliminates the risk of a single human point of failure, by ensuring that any given development task is performed and understood by at least two developers.

Answer 6

There are a number of ways I've seen this sort of thing handled:

Share the work out

The most obvious thing to do is share the work out among existing resource (assuming this is possible). How to ensure the developers hit the ground running is almost an answer in itself but ultimately, it boils down to properly recording requirements, designs and progress. Things like pair programming can also assist greatly here.

Push the deadline back or attempt to claw back time

Check with the customer to see if the deadline can be extended. Alternatively, it might be possible to gain addition development time by working evenings, weekends and holidays.

Drop other tasks

Are there any other non-critical tasks that can be temporarily dropped to make room?

Get ahead

Is there work planned after the development that can be brought forward such as documentation, test scripts and configuration?

Admit it could be late

Speak to the customer early. It might be possible to deliver in part - or at the very least, you could get a decent steer on the relative priorities of other things.

Additional resource

A possibility - but this itself carries risk. It will take time to get them up to speed and as they're temporary, they could just leave leaving you even worse off.

Check the critical path

If other parties are involved - check that they are still on target. There is little point in moving heaven and earth to get something finished if say, the test team are a month behind on testing things.

Accepting the realities of risk

There is a common phrase in the legal profession that states that hard problems create poor solutions. It can be tempting to try and get everybody to understand everything to cover all eventualities. This however, is a fools errand.

Developers should be spending quality time on their own developments. Consuming an ever increasing amount of time becoming au fait with other developments is a highly questionable activity. A reasonable middle ground might be to have a subject matter expert and a deputy.