How to deal with long iterative stories in SCRUM?

Question

We have a story "Scale service to X 000 transactions per second". It's unclear how to do that and how much time will it take. Usually in this case you can define a Spike to find out what obstacles you will face during implementation, what steps have to be done etc., so on the next Sprint planning you have enough information to define tasks properly and estimate the story.

However in case of our story it's iterative process. You run load tests, see that goal is not reached, profile, optimize something, run load tests again, profile, optimize, ... - research-implement, research-implement many times.

How to handle this type of stories?

Answer 1

How to handle this type of stories?

A spike is the best thing you can do to address a story that is still needing to be refined or is more iterative than a traditional story.

There is nothing wrong with using a spike for this type of work in the short term (iterative style stories such as optimizations or research tasks for example), that is exactly what they are for.

The other thing you may wish to consider is to put this style of work on a Kan Ban board if it appears to be ongoing.

I found this article insightful on the topic of stories: Writing good Stories

Answer 2

You can break that goal down into multiple tasks. Assuming you've not done load testing of this service previously. You need to build a load testing harness to generate load. Trying to understand what realistic load looks like. You need instrumentation of the application to understand where the bottleneck is. That gets you a good chunk of the way into the load testing segment and then you'll have identified a couple potential sources of where the bottleneck is and can iterate from there. Those first three tasks can be estimated independently and done independently.

From there the iterative portion of making changes seeing their effect on the system can be better judged since you will have information like the load test itself takes so long and making a build takes so long so you can only try say 20 ideas a day. It will also give you information like I need to double the current load vs 10x the current load which will inform you about the scope of necessary changes.

Answer 3

Like MisterPositive says, not all work fits the Scrum methodology.

Don't spike this in your regular sprint cycle - put some resources parallel to your other ongoing development into identifying the need first (if you're happily managing 10 TPS you might not need to worry about 10 kTPS right now).

Then those resources need to identify bottlenecks and research solutions, and the work to fix those bottlenecks can then be estimated into a regular sprint.

Absolutely does not spike to sprint to spike - it will wreck any velocity your regular sprints have - only put things into sprints when you are confident(ish) in the estimate (e.g., we don't include most things that are 13+ and end up breaking them down)

Answer 4

This is not a lot different from any other project. You have a large task that will be accomplished by doing a bunch of small tasks. Whether the small task is to create a database, develop a UI, or perform load testing is somewhat irrelevant for the purposes of planning.

In order to do this "story" (which is really an epic, but could also be looked at as a feature or product or project), you need to break it down into smaller pieces just like you would with any other project. Then, you prioritize the pieces, then start working on them.

For example, the first few stories might be to do the following:

1. develop load tests
2. run load tests to generate a list of candidates for optimization
3. prioritize the candidates

At that point you might have a dozen or so candidates. The next story would be to work on the highest priority candidate, then the next, then the next.

4. create first optimization
5. create second optimization
6. ... and so on

That's not the only way. You could, for example, set up a story for "create and run load tests", then "optimize biggest bottleneck", then just repeat those stories over and over.

---

That being said, not all types of software development or teams are a good fit for scrum. It's impossible to say what's best for your organization. If you have a top-notch team that can simply go into a room, lock the door, and be highly productive for two weeks, scrum is overkill. If you have a team full of less skilled, disorganized people, you may want to stick as close as possible to scrum.

The bottom line is to get the work done. If scrum helps, use scrum. If it doesn't help, find what does. The most important parts of scrum are the transparency and continuous improvement. The stories usually help with the former, and the retrospectives usually help with the latter. In between there's a lot of room to do whatever works best for your team.

Answer 5

I recently had a similar story to this. What you should understand is that with your current knowledge, you see the problem as requiring a series of random changes until you get the desired results, but you don't have to be in this state forever.

What I did was create a first story which had as its output a detailed list of concrete things to try. Then I had enough background knowledge to estimate how long it would take to try everything on the list, as well as which ones were most likely to work.

During the course of the trying stories, I discovered a couple more things to try, but for the most part I knew the upper bound of the effort at the end of the spike. There's still some uncertainty because the first thing you try might work, but the upper bound is much more accurate at that point.

You really will be a more knowledgeable person on a topic after a spike, if you do it right, and you should trust in that.