Almost every reported bug is a high-priority bug [closed]

https://softwareengineering.stackexchange.com/questions/302763

08-12-2020
|

Question

I have noticed a pattern while working on several software projects: the big majority of the bugs reported had a high/very-high priority. I asked some colleagues about why this may be happening, and they mentioned that if a bug doesn't haven't that level of priority it is very rare that the Bug gets developer attention, which indeed makes sense.

So, I wanted to know if this problem is common or if I just had bad luck. I did a quick Google search, and I found that some teams implement Bug Reporting guidelines or have a separate "Bug Triage" team. If you have faced and solved this issue, what was the approach that worked for you?

This question is specifically about the "Priority Inflation" problem: If you face the scenario and what meassures result effective against this problem.

Solution

I asked some colleagues about what this may be happening, and they mentioned that if a bug doesn't haven't that level of priority it is very rare that the Bug gets developer attention, which indeed makes sense

Actually, if you ask me it does not. The more (used) levels of priority, the more information you have. If you effectively only have one priority, that's the same thing as having no priority at all.

And since you still have the same number of bugs to tackle, and the same amount of manhours in which to do it, it follows that some other heuristic will be used, possibly the null one - "first come, first served". And so you now have a bug priority metric, except that's the time of arrival and no longer under your control.

It can be a symptom of not enough resources being allocated to the bug fixing (there are some policies such as "No new features until the bugs are fixed" that can help there. Joel approves; understanding limits and consequences is a business decision).

In one project I worked, the incoming bugs were accumulated in a "no priority buffer" and every Monday we would review the bug list, estimate difficulty (a very rough estimate; more often than not we just put in, "Average") and sort them by available time. This did tend to bump down the list boring, uninteresting or thought-to-be-hard bugs; to offset that, supervisors and marketing had a certain number of credits per week that they could spend to bump the priority of favorite bugs, and were reimbursed for unsolved bugs (this set a limit on how much a developer-despised bug could be delayed).

It was also possible to merge, cancel and split bugs; I remember one module which was so hopelessly flawed that we sank some twenty or thirty bug reports into a single "rewrite this thing from scratch", which was then split in "clearly state the inputs and outputs of the wretched thing", "write tests to ensure inputs and outputs match the spec", and so on. The last item was "print the old code on recycled paper, bring it out on the lawn and set it on fire" (we did that, too. I remember how good it felt. We took turns on the eulogy; it was quite hilarious).

After some haggling, we had the week's to-do list, which was divided in "will do", "might do" and "can't do" that were bumped to next week. This is where some additional haggling came in: we had say fifty hours to allocate to bugs, and we were 95% sure to fix the first twenty. The management strongly wanted a twenty-first bug to be fixed and had no credits left; we would then offer to swap that bug with one on the "Will do" list, or someone would say "Get me off the FooBazFeature subteam for a couple of days and I'll do it", or we would say "We need more manpower".

The system satisfied no one, really, but this was believed (at least among the developers) to be a good sign.

Some additional negative patterns that turned up were the "Wishful Thinking" on the managers' part ("You stated bug 57212 requires eight hours. That is unacceptable. Make it four") and the "Debug by Fiat" ("Do whatever you want but these forty bugs must be fixed before the big demo next week. You cannot have more time, you cannot have more people"). Also the Boxer Syndrome ("I will work harder"), that tended to work very well for a short time, but usually led to a developer freaking out or leaving for greener pastures.

OTHER TIPS

If you have this problem where users are assigning ever-higher priority bugs then the only realistic solution is a triage mechanism. All bugs get reported with whatever priority they like, but some poor manager will have to go through every newly reported bug and reset its priority to a sensible level.

After a while your users will either get the message, or you can change the reporting system so that every bug has a default priority. If they want it escalated they will have to contact someone to bump it, which will require some justification. This fact alone will cause 99% of all bugs to be left un-escalated by the user.

Obviously you have more bugs than you can process, so maybe you need to embark on a bug-fix roundup to clear the backlog. This will show the users that their bugs will get fixed without needing them to be marked as super-super-dooper-really-no-honest-this-time-important.

DISCLAIMER: I have not yet had experience with user-reported bug priority shenanigans. I know the question asks for this, but it might help having the perspective of an outsider.

Your problem is not that you have too many high-priority bugs. Your problem is that you have too many people who have direct control over bug priority. If every user can directly assign a priority to their bug, they will almost automatically report their issue as high priority.

You could make it so bug priority has to be configured by a manager or a helpdesk drone, but this could lead to favoritism and social engineering, where a client gets artificially higher priority because of their status or because they know how to craft their messages to make them seem more important. It is also a lot more labor-intensive.

There is a middle ground, where your users have some control over the priority, but in a way that makes it harder to exploit the system. Essentially, you force your users to use a template for reporting bugs. They first select a category:

The program becomes unusable or crashes when I do something.
The program has a graphical defect that affects functionality.
The program does not allow me to do something I should be able to do.
The program allows me to do something I shouldn't be able to do.
The program gives the wrong result when I do something.
The program takes too long to do something.
The program has a graphical defect that does not affect functionality.
The program has a defect that does not fit in one of the above categories.

To give examples:

My iPhone crashes when I receive a message containing Hebrew symbols.
My Android lock screen is rotated in such a way that half of it falls of the screen.
My Android phone sometimes doesn't accept my lockscreen code, even though I entered the right code.
When I try to navigate to PhoneHub.net, my phone redirects me to an adult site.
When I open the Facebook app, it takes a minute to open, even on fast connections and with no other apps running.
Your app has a spelling error.
I found a security defect in your program and would like to report it.

As you can see, each of these errors has a different grade of severity, and the categories are roughly ordered based on this severity. You can then assign each bug a priority based on the category, who reports it and keywords that appear in the description. Bugs in category 7 should get their priority assigned manually.

Note that this can happen fully automatically, and you should let this happen automatically; in fact automation is the key here. Users are prone to overestimating their own importance to the business, so they don't see an issue in reporting their bugs as a higher priority than they should. they're less inclined to deliberately place their bug in a different category, because that requires them to basically lie about the bug.

Users might still enter their bugs in the wrong category of course. The first thing you should do is, from version 1.0, show a friendly message encouraging users to provide accurate information about the bug to help the developers to find and fix it faster. Most users will understand and stop misreporting bugs. Some users might still continue providing wrong information. When that happens, send those users a gentle reminder via mail that accurate information is important and to please not abuse the system. If they continue falsifying records, you give them a warning that they're willfully abusing the system and continued abuse will result in their bugs automatically being assigned a lower category. If they persist, you can adjust their bug multiplier.

You can see parts of this system in place in high-throughput support situations: giant tech companies like Microsoft, Facebook, Google, Gaming companies with a lot of users like Valve and Blizzard, certain governments,... While I'm not sure of the workings behind the scene, you notice that their user-facing support system has a similar interface to what I suggest here, with a strict category system.

As people have said, this is why the people reporting bugs don't get to assign priority. Your dev team should be in control of their own task assignment (within the scope set by higher management). So, someone further up says "work on this app, fix this feature, make it better at doing this", and the team gets to decide how to go about that, because they're the ones with the technical expertise required to assess how best to achieve what management wants.

The people reporting the bugs should be assigning impact or severity levels, which have defined scope. You can easily call people out for not sticking to agreed levels of severity, because you have material evidence for those levels. For example:

Complete loss of functionality
Partial loss of functionality
Widespread reduction in effectiveness
Localised reduction in effectiveness
Annoyance or hindrance (workaround exists)
Cosmetic
Nobody actually noticed, was found during obscure exploratory test

To start, you can use these levels as a blunt instrument to point out that a misalignment of some title text isn't a Level 1 bug because it's not rendering the entire application unusable. Once they get the idea, you can make it more fine-grained, and start debating whether the glitch in updating this one textbox is a Level 5 because you can fix it by right-clicking in the textbox a few times, or a Level 4 because it's slowing down everyone in Accounting so they get fewer forms processed per hour.

Once you've got useful, measurable information on how bad the bug is for your organisation, assigning priority becomes obvious. Whatever is currently causing the largest problem for the organisation - lost profits, damage to public image, employee unhappiness, whatever - is going to be High Priority, and you work down from there.

It goes a bit like this:

Mgr: what are you working on? Dev: this low priority tasks Mgr : shouldn't you be working on high priority tasks?

Client: when will my bug be fixed? Dev: it's low priority we have high priority tasks. Client: oh, well then set my bug status to high priority.

This will lead to ever escalating levels of priority. I see you've already gone beyond high priority to very high priority. What will be next are:

Super High Priority
Ultra High Priority
Very Super Ultra High Priority.
Very Super Ultra Mega High Priority.

etc. etc.

Yes, this is a normal process. As long as there is no cost involved with assigning priority, and the caller has influence, of course they will try get their issue resolved in the fastest way and set the highest priority.

There are basically 2 ways to counter this:

Take control away from the client regarding priority levels.
Associate a cost to the client for elevated priority levels.

One can add higher and higher priority levels to the system, especially if it is Jira. Giving the new high priorities increasingly silly but dire names can increase the Hawthorne effect bump in the quality of priority choices made by all parties. If the highest priority has a really outlandish name, the effect could be permanent. Eventually when someone chooses a priority, they have to weigh the social consequences of choosing the "bug of death" priority vs. getting it due attention. So, the highest priority is de facto reserved for something that happened to the CTO at home in front of his guests, or other incidents of equivalent visibility.

Introduce a cost to support requests. You can only allow a user to report X number of high priority items in a given period of time, Y number of medium priority items, and Z low priority.

Of course, that also means the dev team and management are going to have to then guarantee that a certain number of these will in fact be fixed - all the high priority items, most of the medium priority items, and (maybe) some of the low priority items within a certain timeframe.

But if this is going to work, management are actually going to have to buy into it, otherwise the whole exercise is a waste of time.

At the end of the day though, your particular situation seems to be a symptom of the a problem that your management aren't allocating enough resources to dealing with support issues. If issues were being dealt with in a timely manner, then I don't think this would be happening.

Something like this was implemented in the first company I worked for as the support process was dysfunctional and led to a situation where if everything is an emergency, nothing is. In our case, after getting the in-house situation under control, the new software development manager put a hard limit on the number of high priority issues a customer could lodge depending on how much they paid in support contracts. If they went over the limit, either they coughed up more cash or the support issue was lowered in priority.

This happens extremely often in large corporations where IT is seen as ancillary, and/or is outsourced. Business people don't understand software and don't care to, all that they care is that "their" bug gets fixed yesterday, regardless of how non-critical it is. They don't realise, or care, that there are a hundred other users also filing bugs, and a team of maybe 5 developers available to fix things.

This is exacerbated by poor management, especially non-IT managers who can't say "no" or who simply let business people set bug priority. (In either case, said manager isn't doing his/her job.) Most managers will prioritise the bug that they were last contacted about because "it's urgent"; the net result is that developers end up jumping from bug to bug, thus fixing a single bug takes longer (context switching), and at the end of the day everyone is unhappy. "When every bug is a high-priority bug, no bugs are high priority."

I've been in this situation, and generally the only way to escape it is to get out. Bug reporting guidelines are always ignored because users don't give a s**t. Attempting to introduce bug triage will either be resisted, or it will be implemented and then ignored the next time a user calls your manager to complain about "their" bug.

Basically, if the developers have no control of the priority, you've already lost.

As as company, you want to solve the issues with the highest importance/cost ratio. Users decides importance, Engineering decides cost. If users assigns the same importance to all bugs, then cost alone matters.

Practically speaking, this means you define priority as importance/cost, and do not allow users or developers to set that priority directly. Neither side has the full picture.

If users decide to rate all issues equally important, that's OK - but it means that engineering (cost) decides what's done. Explain to them that importance is the only way in which they can influence (but not decide) priority.

A few factors...

Often the person report the bug does not know the bigger picture and is unable to decide how significant the bug is.
Often low priority bugs never get triaged or considered, therefore it is better to give a priority that is too high and let the person doing the triage lower it.
As I developer I have often looked at bug report and found that there is a very high priority problem behind the bug, but the product manager doing triage does not care about the bug.

So I think all bug reports need to be looked at QUICKLY by one to two experienced developers, adding their thoughts to the bug report, then the bug needs to be triaged. Expecting the person that finds the bug to be able to set a useful priority at the point in time when they find it, is just asking too much.

It is quite possible that all the bugs mentioned are high priority. I've been on projects that were both underfunded and underspecified, and when testing began QA and users just gave up reporting low priority items, because they knew that spelling errors or glitches in usability were a waste of time if the core functionality of the project was completely hosed. If your automated baggage system is crashing carts together and destroying luggage, who cares if the font on the tags is 2pt too small?

In a situation like this, the project is failing. If you suspect that this is even a possibility, you need a heart to heart with the people reporting defects to find out. If people are inflating bug reports, then the other answers will help. If the bugs are as bad as reported, then you need to take extreme action.

Most has already been said, but I'd distill the "bug zoo" list down to something a bit less granular:

A: The bug stops the user dead in her tracks, gives wrong output, generally renders the system/feature/function unusable. That's a "high priority" bug.

B: Everything else. These are "negotiable" bugs.

NEGOTIABLE bugs fall under a variety of concerns (which you'd put in your own particular order):

1: Bugs that impact security;

2: Bugs that impact usability (fitness for intended purpose);

3: Bugs that impact aesthetics;

4: Bugs that impact performance (a subset of usability perhaps?);

5: Bugs that offend your sensibilities as a professional programmer;

6: Bugs that diminish USER TRUST;

So that's the utopian world that none of us actually live in. sigh To complete my answer to the OP's stated question, across the entire software industry, it is utterly common for development efforts to be in a status where every single bug is a priority-one-super-banger-special. And you know what they say about "special": when everyone is special, nobody is special.

Basically this issue is rooted in the problem of decentralizing prioritization. There should always be an odd number of people with ability to prioritize a team's workload, and 3 is too many. So that 1 person is responsible for effectively -triage. And it should be a manager/analyst, in consultation with a dev lead/architect. And the process is quite simple, and can be applied to feature requests too. What's the cost of doing the development? What's the value of the expected result to the business. Output of that function is the priority assigned.

Of COURSE every user wants their issue fixed first. And as soon as you allow that, chaos. You have no valid prioritization. You need this done by a SINGLE person with authority (consulting with others where necessary), who has VISIBILITY across ALL the issues and business needs, and competent enough to gather the required business and IT expert advice and thereby generate reasonably accurate estimates of the metrics above.

At the risk of stating the obvious I'm going to assume there is no bug tracking software that is setting bugs to high priority as default (or has been set to this setting).

I'm afraid, that in the absence of controls, this is the default scenario for multiple teams, clients etc reporting in. If the status quo is being abused then some sort of triage process is definitely in order.

A quick win which I've seen work very well in the past is that P1 (top priority bugs) spawn a plethora of management alerts. If the system has been abused, it is immediately bumped down. Or if it genuinely urgent, a conference call or physical meeting happens to address the issue as soon as possible.

I'm assuming here that you're talking about all bugs not just those from the initial development. If you're typically a green field development gun for hire, then it certainly isn't unusual that most bugs are high priority following the initial alpha release.

You can't just have a priority and expect everything to magically work out. Sometimes people figure it out on their own, but not always.

To correctly assign priorities, there must be a definition of exactly what constitutes each priority. These criteria must be objective, to avoid bug favoritism and political prioritization. If the objective criteria are not followed, you need to make the team follow it.

There's really no way around this - if you can't have objective criteria for what bug goes where, and if people willfully refuse to honor these criteria, then you might as well not have submitter-assigned priorities - either do without priorities, or have a third party assign priority as others suggested. Crowdsourced data only works if the submitters are cooperative, and don't actively sabotage the data collection.

If the difficulty arises from not being able to create objective criteria, you can use relative criteria:

Have a simple queue of all bugs. Users are able to insert bugs anywhere in the queue. They should only insert at a given position if they feel their bug is more important than everything below it. Bug fixers start from the top of the queue.
Assuming you already have a set of well-classified bugs, tell everyone that the condition for putting a bug in priority X is that it must be more important than all bugs in priority X-1.
Tell submitters that at no time can the number of bugs with priority X exceed the number of bugs with priority X-1.

But it sounds like your problem is not the definition, but the belief amongst submitters that low-priority bugs do not get fixed. Presumably, you cannot persuade them otherwise, since (from what you say) their belief is grounded in fact. Then, why do you make them submit these bugs? It ends up being nothing but busywork. You could, for instance, after a certain number of active bugs has been reached, tell everyone to not bother making reports unless they feel they found something more important than most outstanding bugs. Granted, this is just the queue solution with an upper limit on queue length.

Licensed under: CC-BY-SA with attribution

Not affiliated with softwareengineering.stackexchange