What to do when third party won't fix bug [closed]

Question

Closed. This question is off-topic. It is not currently accepting answers.

---

Want to improve this question? Update the question so it's on-topic for Software Engineering Stack Exchange.

Closed 3 years ago.

Improve this question

I've reported to a third party what I consider a very serious performance bug in a widely used product. Older versions worked fine. Newer versions are practically unusable except for what I consider trivial situations. I have provided test cases showing older version performance versus newer version performance.

After waiting a few weeks with no response, I was finally told this:

We appreciate you taking the time to report this problem. We are currently prioritizing problems that are impacting a broad set of our customers, so we may not be able to investigate this one immediately. We know this problem is important to you, so we will continue to monitor it.

Due to some recent upgrades to our development environment, it's very difficult to continue using the older version of this library. We use this library in many places, so completely ripping it out and replacing it would be hard to do. Plus, there aren't many options for another product.

My concerns are as follows:

1. Am I understanding how to use the new library correctly? Is my reproducible test case actually correct, or am I missing something?

2. What (if anything) can I do to convince this vendor to put more consideration into the problem?

3. If I can't get a resolution, what are my options? Revert to (really old) previous versions of the product? Replace this library with another product and end up rewriting a ton of battle hardened code?

So my thought on #1 is to get code reviews of my test case. I'm currently the senior software engineer at work. I supervise two other engineers, but only one is really qualified to review. I believe he has ran my test case and has not reported any issues with it. We've both been trying to understand the causes of the performance issues, though we have came to different conclusions. I thought about posting the test harness on here asking for a code review, but I'm not sure if that's the appropriate thing to do.

I should also note that I have two other bug reports with this vendor. Also with reproducible test cases. They also seem to be getting the same lackadaisical treatment, though this was the only one where they explicitly said it wasn't a priority.

So I'm just looking for guidance. I don't really see any great option as it is.

It was requested to make this more specific, so I will attempt to describe the problem itself...

Until recently, we had been using Visual Studio 2008 and ReportViewer 2008. Upgrading our environment has been planned for awhile. As a result of an unrelated (and as yet unfixed) bug in ReportViewer, we decided to go ahead with our upgrade, hoping that would fix the bug. It did not.

We kept ReportViewer 2008. Everything there works fine. However, newer versions of Visual Studio won't allow you to design reports for 2008. You either have to upgrade or you have to develop your reports in raw xml.

So the obvious choice was to just upgrade ReportViewer, all of our RDLC files, and everything should work. However, what I found was even the simplest of reports took twice as long to run. Other relatively short reports took 10x as long. Larger reports (though still what I consider reasonable) went from 30 seconds in ReportViewer 2008 to "never finishes" in every newer version I tried. I tried with 2012, 2015, and 2017, though I suspect 2010 has the same issue. I suppose it's more than just a performance bug, since a large enough report never finishes. And it also doesn't seem to give any kind of errors.

Other people have had similar issues with it taking 10x as long to run their reports. Suggestions include changing CAS security settings or changing dynamic grouping to static. I could not find any CAS settings that fixed the issue. As for grouping, my test case was using RowNumber as a group. I'm not sure whether that is considered dynamic since it takes a group name. As for the actual data in my report, everything else was static.

As for determining the problem, I have ran a profiler and determined newer ReportViewer versions do a massive amount of binary serialization. My other programmer seems to think vb expressions are causing an issue, though he hasn't explained why he thinks this. Not to say they aren't, but I think we need to know why.

My test harness was developed to run every combination of ReportViewer 2008, 2012, 2015, and 2017 against every .NET version of 3.5, 4.0, 4.5, 4.5.1, 4.5.2, 4.6 and 4.6.1. ReportViewer 2008 used a 2008 version of the RDLC. Everything else used the newer version.

The test harness showed the problem was connected to ReportViewer version, not .NET version. However, my other programmer keeps thinking it's .NET that is the issue.

As I see it, the options to weigh are:

1. Hope the vendor eventually looks at this, verifies there is an issue, and fixes it. Downside is in the mean time we have to either edit our reports directly in XML or work with them in VS2008, then pull them over to our current VS version (which is 2017)

2. Switch from ReportViewer to something else. The obvious one is Crystal. I have no experience with that product. This would require a massive amount of rewriting in our app, which I'm not fond of.

3. Revert back to VS2008/ReportViewer 2008. I'm ok with this, though that means converting some of our more recent code back to working with .NET 3.5. What bothers me about this route is future support from MS. Will ReportViewer 2008 continue to work on newer versions of Windows? Will .NET 3.5 continue to work?

First and foremost on my mind is making sure I'm not making a mistake. Maybe there's something I don't understand about newer ReportViewer versions. Maybe my understanding of the workarounds I've found is incorrect. Getting feedback from either my bug report to Microsoft or on the MSDN forums has been fruitless. I feel like I must be missing something, as this is such a widely used product that this should have been addressed a long time ago.

For those who are curious:

MSDN Thread

Bug Report

Answer 1

IMO, you shouldn't have upgraded your development framework if the newer versions drastically impact your project's performance.

In the first place, you should have tested newer versions of the framework before start implementing new code in this new development environment. In terms of process, I think this would be the correct way to go.

But, since you already upgraded and already have code written using the new framework, I suggest one of the following options (if necessary, you could provide more details/effort needed for each option so we could help evaluate which one is more feasible):

• For now, downgrade only the Reporting feature into a separate project, and deploy it separately, in order to clients be able so use the feature with good performance;
• Downgrade entirely to your previously working development environment, rewriting new code to work with older versions of .NET framework;
• Try to use some conversion tool to convert your reports to another reporting engine;
• Write a conversion tool to convert your reports to something else (in another engine);
• Investigate harder to check why your reports are lacking performance in this new version;
• Create a fallback report, so your clients still have something to use during your main Reporting feature is under maintenance;

You could combine options, e.g. use the first option while you plan to migrate your reports to another engine.

Answer 2

This is one of the drawbacks of using third-party libraries: as you get more and more dependent of them, you're at the mercy of the company which develops those libraries. Not only they can set the priorities as they want, but they can even decide to stop maintaining this library, or simply go bankrupt, leaving you with no much choices.

The choices made by another company are their choices. Some factors, such as being a very important customer or suggesting to pay a lot for them to work on what is important to you may help the negotiation. However, this is too specific to a given case, and is outside the scope of this site.

Back to technical aspects, there is a bunch of things you can work on to mitigate those risks:

Interfaces

The most important aspect consists of putting an interface between third-party libraries and your business logic. This would help switching to a different library if a competitor happens to develop a better one (or to your own implementation if you're able to develop your own).

Sometimes, this is extremely straightforward. Usually, small libraries which are used to perform a very specific and narrow task are just like that: abstracting them is very easy, and one can swap a library for another with nearly zero effort. Example: a library which generates QR code images from URIs.

In other cases, it is extremely complex. Frameworks or libraries on which practically all your code relies tend to become very interleaved with your code. Even if abstractions are used, it may not be that easy to swap one library/framework for another. Example: SharePoint.

Not only is it important to reduce the coupling between your code and third-parties, but it is also essential to be able to measure how difficult is it to swap a library if you're required to do it in the future. Separating business code and interface code into several packages gives you that hint, by showing how much code belongs to the business logic, and how much provides the interface with the third-party library. The more code you get in the interface package, the more you're in trouble.

Automated deployment

A specific problem where a new version of a library breaks what worked before can be avoided by following a bunch of practices related to the deployment of software.

In badly maintained environments, one of the major risks is that by upgrading a library to a new version by hand, one could find himself in a situation where it is hard to impossible to go back to a previous version. Problems could range from the downtime during the downgrade operation, to situations such as when the retailer doesn't distribute the old version of the library any longer, and you never kept any backup of the binaries. Or it may simply be impossible to reproduce all the steps which were performed on the server to install the old version in the first place, because nobody noted those steps.

In a well-maintained environment, with automated deployment, proper versioning of dependencies and an artifacts repository, switching from the newly deployed version to the older one could be as simple as a click on “Deploy” on one of the previous revisions of your app, the one which used the older version of the third-party library.

Migration

Finally, an important aspect is that before you migrate to another version of the library (or from a library to the competitor's one), you need to ensure that it (1) works correctly and (2) fits your needs.

When it comes to checking that it works correctly, you have to play with the library to see how it behaves, but also run the tests, which includes stress and load tests. I'm not talking here about the tests of your product (since you would be able to run them only after you migrate), but the tests written with the sole purpose of seeing how the third-party library performs the major scenarios you'll use.

Example 1: when migrating to a different library which generates QR codes, one can write a test which generates ten thousand images to see how if the new library outperforms the old one (or at least is not much slower).

Example 2: when migrating a site to a new version of SharePoint, if the major usage of the site is to store Word documents, one could write a test which would compare how two versions of SharePoint deal, in terms of performance and disk space, with this specific storage (for instance trying to study how the introduction of chunks for Office documents helps saving disk space, and what are its consequences on the CPU usage and overall performance).

Reading your question, it seems that having a few stress tests may have prevented you from upgrading all your assets to the new version of the library, and only then finding the performance issues.