Agile Scrum - Importance of requirement gathering and documentation and suggestions to improve

Question

We are using agile scrum methodologies for developing and maintaining a product. Since we are a product company, we do not work with customer directly on day to day basis, but instead communicate with BA who gives us the requirement.

We are facing following challenges and wondering how to overcome them.

The product we are working on is complex and the BAs and the team is relatively new. When we get the requirement it is very high level. The acceptance criteria is also very high level and involves lot of discovery during the sprint. There are many touch-points which were never considered in the acceptance criteria and they always come up when the developer looks at the code. Due to this the sprint planning becomes very hard and takes away lot of development and QA time in discussion with BA on what should happen for each scenario.

1. Is this acceptable in agile scrum? If so, how can we plan in a better way given very little clarity on "WHAT" has to be done. Most of the times if we are unable to complete the work in a given release, additional teams are brought in to complete the work and this complicates it further.

2. I believe most of it is due to very little documentation and whatever we have is scattered. When a team member leaves the company, he takes away the knowledge and due to little or no documentation, the new team member (BA) has no where to refer to, and due to this the entire team suffers. So how much documentation is essential?

3. In this case, is it the responsibility of the BA to give proper acceptance criteria or the scrum team's(BA is also part of our scrum team) responsibility to navigate through such difficulties? The developers and the QAs are always in a time crunch due to the discovery process involved during the sprint. The quality also suffers due to this.

4. We also have to spare additional capacity for the bugs, customer cases. So at the end of it, it is usually the developers and QA who have to put extra hours to meet all the commitments.

Answer 1

Is this acceptable in agile scrum?

Define "acceptable". In some environments, it's fine. In most, it is prohibitively problematic.

That problem is that most Business Analysts are horrrrrible. Even when they understand the product well, and even when they can define and scope features, they rarely understand much of the implications of the feature to the codebase. So while it's not great to have ill-defined feature requests, it's unlikely you'll be able to fix that.

So how much documentation is essential?

None.

I am perhaps in the minority here, but I find documentation - especially over the long term - to be less than worthless. Documentation is a poor communication method. Many people are bad at making it. It doesn't allow you to tailor your message to different target audiences. It doesn't allow interaction to clarify ideas.

Worse, it's invariably incorrect/out-of-date. So not only did you spend a bunch of time and effort making these documents, they're actively lying to your staff.

In this case, is it the responsibility of the BA to give proper acceptance criteria or the scrum team's(BA is also part of our scrum team) responsibility to navigate through such difficulties?

It varies. To me, BAs (and other stakeholders) don't get to dictate what is being done in a sprint. It is an agreement between the team and the stakeholders made during sprint kickoff. That kickoff meeting should be "hey, I would like XYZ feature." "Okay, what does that mean? Do you need Foo?" sort of back and forth until everyone can agree what work is going to be done in the sprint, usually in the form of "In 2 weeks, Bob will be able to open the App and order a Pina Colada".

We also have to spare additional capacity for the bugs, customer cases. So at the end of it, it is usually the developers and QA who have to put extra hours to meet all the commitments.

Why?

You're negotiating with stakeholders during sprint kickoff about the commitments you're taking on, right?

If you commit to too much, then that's what you get. If you get told what you're doing this sprint, then maybe you should do agile.

Answer 2

I think what you are looking for is extreme programming.

Most of the times if we are unable to complete the work in a given release

The triangle of time / scope / cost allows you to increase one if you reduce another, not just force all three to be better, this results in the hidden fourth variable taking a hit i.e. quality.

additional teams are brought in to complete the work and this complicates it further

Adding more software developers to an already late software project does not make it finish sooner, see Brooks - Mythical Man Month. The hard part about building software is not typing the solution into the computer, but the people surrounding the problem space.

When a team member leaves the company, he takes away the knowledge and due to little or no documentation, the new team member (BA) has no where to refer to, and due to this the entire team suffers

By pair programming you can spread knowledge around your team. The idea is to have specialists, who are experts in a field, but not roles, where people are forced to stay within their designated areas.

In this case, is it the responsibility of the BA to give proper acceptance criteria or the scrum team's responsibility to navigate through such difficulties?

It is the responsibility of everyone on the team to collectively take ownership of solving the problem. It sounds like the Product Owner is not doing a good enough job on breaking down customer requirements into user stories but this might mean you just need to help them understand your technology more.

The developers and the QAs are always in a time crunch due to the discovery process involved during the sprint. The quality also suffers due to this.

Of course the quality suffers, this isn't rocket science. XP has a strict 40 hours work week. A crunch is sometimes necessary, although a lot less than most 1980's MBAs think, when an external deadline moves. But most of the time if you are being forced into a crunch then your business analysts aren't doing their job properly and are tring to get you to help cover that up for them and it will result in burnt out developers who produce worse code and are less productive in the long run.