Large internal features on kanban

Question

This days I'm reading some tutorials/advices about Kanban process and Agile methodologies, but I have two big questions about this Kanban methodology.

What I understood is user stories are about adding bussiness value, like a feature which our customers can use. The main problem is we have a lot of features that looks trivial (simple form that manages synonyms for solr, for example) but are hard to implement and may take several days/weeks to complete, due to scaling/coordination details on our cluster (refreshing the synonyms using zk and reloading from database, for exmaple).

How do you break down this user story into smaller ones since Kanban specificies every user story has to be independent? Can I break it down into user stories about implementation details? (implement coordination algorithm, implement X storage service, etc)

And my last question is about architectural changes/refactoring. Since it doesn't add any business value (invisible to our customers) how do you reflect this tasks in this methodology?

Answer 1

Kanban is an approach that is one that focuses on trying to identify work in progress (and minimize it). Thats the core concept in it. It is based on the just in time workflows of Japanese car companies. Stuff that is stuck somewhere isn't moving as fast as it could through the workflow. Often it is paired with scrum in which case its known as scrumban though this doesn't have much to do with the agile principles.

For kanban to work well, the stories or tasks should be similar sized. Having a minivan go through the process that is designed to build scooter sized things will cause problems on an assembly line. So to it will have problems trying to push a task that takes a month through a workflow that wants to be pushing through tasks that are a week long. It can be done, but you're going to need to realize what it implies to the rest of the workflow.

Stories or tasks can have dependencies. Just one needs to make sure that the dependency is complete before the next one moves forward. Consider the situation where you have two tasks, A and B. A must be done before B can be coded. If you have both of these tasks in the backlog at the same time, someone might take B first - this would be bad as it would have a task that is some state that cannot be moved forward - work in progress. Furthermore, if A is started and is partly done then B is started you can again have the situation where there is work in progress that cannot move forward.

That is why there is an emphasis on independent tasks. However, that does not mean that all tasks must be independent - just that you shouldn't have two dependent tasks being worked on simultaneously or in the wrong order. The control for this is the ability to put things in the backlog.

With attention paid to similar sized stories moving through the process and limiting how much can be in a given column at one time it is then possible to give better feedback to the customer about how long it will take for something to go through the workflow.

Thus, you may absolutely break it down into implementation stories or tasks. If you are using a physical kanban board, consider using a particular color sticky note for a partial detail of a larger story so that these can be seen and understood (possibly including the dependency information on them) so that they can be seen and followed more quickly when trying to get the big picture of the board.

---

There is nothing in kanban that has a parallel with the oft seen applications of agile that attempt to minimize design time (note, there's nothing in the principles behind the Agile Manifesto that suggests no design either though it may be a reaction to the "Welcome changing requirements, even late in development" or "Working software is the primary measure of progress" being interpreted as "design is worthless" - but I doubt any of the signatories of the Manifesto would make that claim). When one looks at various demos of kanban boards one sees things like:

That first image has a 'specify' column. The second one shows an 'Analysis' and 'Design' column.

As described in the principles behind the Agile Manifesto the key thing is to have working software and work with the customer. The individual responsible for putting tasks into the backlog is perfectly capable of putting internal refactoring tasks in there. I've also heard of managers doing entire scrum style sprints that are 80% internal tasks when necessary, or have a Friday refactoring day where you work on code cleanup on Fridays. The wise manager/product owner will do this because without addressing the internal (non-enduser facing tasks) the size of the tasks that can go through become smaller and smaller (and estimations for similar sized things become bigger and bigger over time) as increased technical debt is accumulated on the product.

There is nothing in agile or kanban that says you cannot have tasks of the nature 'refactor library XYZ'. If the product owner approves of the task to go through (or however the workflow for putting things in the backlog is) then do it. As mentioned, if you don't do it, development will eventually grind to a halt.

I will point out that not doing this goes against the principles: "Continuous attention to technical excellence and good design enhances agility." By keeping the code base in a constant state of a given quality one can do a better job at predicting how long the entire project will take (rather than having the velocity get slower as things get harder because of the technical debt).

In theory agile processes are supposed to be constantly refactoring the code, though this isn't something I've often seen. In part it has to do with how empowered the programmers are to say "this is how long something will take". In the situation of consultants working on a project, there is often a fair bit of "this will take as long as I say it will take," and take into account the constant refactoring. Such a statement is harder to do with in-house development where its a "it will take a week" ➔ "can you get it done in 3 days if you cut corners/don't do that refactoring thing?" ➔ "well, yea..." ➔ "great! 3 days it is." This is not so much a problem with agile, but rather the empowerment of the developer in the organization.

---

Related reading:

• What is Scrumban?
• Lean Software Scrum-ban
• Lean Software Kanban bootstrap I will point out that this post is specifically addressing the very large project issue that deals with breaking down large stories into smaller ones.

Answer 2

I don't know anything specific to Kanban, but in agile in general, the culture is that you must be able to demonstrate new features working to your customer at the end of every iteration. This demonstration usually has to be complete, in the sense that it must have the UI, business logic and persistence processes working (which is often called a vertical slice).

However, in a case like yours where a "trivial" feature demands too much technical effort, your team may ask if your customer agrees to seeing only a partial demonstration by the end of an iteration. For instance, only the UI and business logic working, but without any persistence (since that is your bottleneck). If the customer agrees, then you can break the persistence into another story.

If your customer may not agree, you can try to break the story into even smaller pieces. Though in your case, breaking the story may not help. If it is a simple CRUD, you can break the C, R, U and D operations into individual stories, but you will still have your "weeks" of technical effort on the persistence layer when developing the first of these stories.

So, in the end, if your customer does not agree to the "partial vertical slices" idea, and if you can't break the story into smaller pieces, or breaking them doesn't help you, then you have one very likely outcome: You won't be able to deliver the feature in one iteration.

If this happens, and if your team is counting story points done per iteration (velocity), then what you have is an iteration with zero velocity. This is usually a sign that something in the team has to change. Which brings me to the answer to your last question (about refactoring):

From my (very limited and out of context) perspective, it looks like your team may be facing problems with leaving techcnical debt behind. The fact that you're using a persistence mechanisms that hinders your velocity may not be directly related to technical debt, it may really be necessary for the specific business your team is developing; however, if that is not the case, your team must be able to switch to another persistence mechanism as soon as the one you're currently using creates more problems than it solves.

For instance, you may use a NoSQL database if you want to reduce the effort of changing the "schema" (since NoSQL doesn't have any) or having to perform huge migrations (because you won't need to). But doing this switch may be labeled as "impossible" if your system is a mess. If your architecture is too coupled to the database you're currently using, the only way to switch is to refactor your system to be decoupled from the persistence mechanism.

So about how to fit refactoring within your methodology, I believe this topic is already covered by this other question, so I better not repeat myself here.

Answer 3

Kanban specificies every user story has to be independent

Where did you get this from? Kanban does not require the units of work to be "user stories", and there is nothing in Kanban forbidding you to develop a component first, and then a second component using the first, and so on. And each of this components can still follow a typical Kanban workflow, with analysis at the beginning, and tests and deployment at the end. You may just have to rethink what "deployment" means in this context.

If you have a big feature to be developed over many weeks, you will have to break this down into small subfeatures, or even better, components - at least, if you trying to develop any kind of reasonable architecture. Of course, some intermediate components may not provide any business value to your customer on its own. But actually, since you will develop tests for each subfeature or component, the working tests can be used to proof that a specific component is "ready". And the component provides "business value" for the developer in your team who is going to use the component. So the only thing you have to adapt in your interpretation of Kanban is just your "defintion of done", when it comes to internal components.

Said that, this should not stop you from thinking about how to organize the development process in a way you can deliver value to the final customer as early as possible, and in smaller steps. Using your example: a form with some complex algorithm in the background itself might be developed first with a very basic, simple "dummy" algorithm, and the algorithm is improved or refined in a second step. A storage service might need some kind of administrative console, and / or some kind of logging mechanics to make it work - something which counts as "business value", too.

Answer 4

This is a challenge in--dare I say, weakness of--many of the Agile methodologies. By focusing so strongly and intensely on user stories, user-visible value, and user feedback, they make it hard(er) to focus on "internal" improvements that work down "technical debt" or that prepare the project/codebase for future user-visible and user-appreciable improvements. Internal improvements are, almost by definition, not user-visible--certainly not at in a systematic, easy-to-value, or leading indicator sense.

In the grand scheme of things, Agile is a reaction to and/or flight from previous methodologies that were all-architecture-all-the-time and deeply pre-planned, with little in the way of incrementalism, feedback loops, or developer input. While the shift has been valuable, the resulting eschewing of much of the deep architecture, long-horizon planning, and internal focus that software projects (whatever their many and genuine flaws) used to have, you could argue that some previous virtues are harder to enjoy. The more extreme the form of Agile, the more likely it is to wholeheartedly disdain preparatory work.

If you want the pattern name for this, it's "throwing the baby out with the bathwater."