PM opting for an overly-complex setup which nobody has experience with [closed]

softwareengineering.stackexchange https://softwareengineering.stackexchange.com/questions/352596

Question

Recently I started a project which didn't seem too hard to make, the concept was a fairly simple application that had to accept input every now and then (maybe 10x a day), and try to perform some operations on them and collect all results at the end. This application would then get a front-end web portal that customers could use to view the results, not exactly rocket science.

For this I initially made smart use of Python's built-in concurrency libraries (ThreadPoolExecutor) and use an easy-to-use library for the front-end (I chose Flask as it's easy for beginners and is relatively easy to maintain and test).

Once we were halfway the project, the PM stated we had to use third party message queue capabilities instead of threads and had to implement load balancing, what eventually ended up happening was that we eventually started working with Celery, Redis, RabbitMQ, Nginx, uWSGI and a bunch of other large third party services which nobody had any real experience with.

In the end this lead to a bunch of spaghetti code, untestable tasks (because of the complexity of third party libraries, patching the code didn't even work) and a bunch of headaches because nobody even knew what the added value of these services were.

Before you say "Yes you should use those services", keep in mind nobody knows how to use these or even knows what they do besides introduce race-condition plagued code.

What should I do about this? At this point it would simply be too costly to revert back to what we had and the PM is dead-set on using these services, even though the end-product is now worse off than it was in the beginning. Is there even any use in discussing this with him? Do I ask for more time? Or the harsh answer, am I just too stupid for my job?

Was it helpful?

Solution

Once we were halfway the project, the PM stated we had to use third party message queue capabilities instead of threads and had to implement load balancing

This isn't an appropriate thing for a PM to "state" unilaterally. Two reasons:

  1. Design decisions should be made by a technical resource and only in response to NFRs. So politely ask your PM if there is a new NFR and if you could please have details.

  2. If an NFR is being introduced halfway through the project, it should probably be done via a change control. The change control is very important from a governance perspective; it would not only be an input to your requirements, but also is an important input to QA's test cases, operations' deployment and support handbook, and (here is the really important part) the PM's schedule. If the new requirement introduces more work, the development team ought to have an opportunity to communicate new development estimates, and the PM will have to decide whether they can live with the new date, add more resources, or push back on the stakeholder who introduced the NFR.

Now if there really is a bona fide NFR, and there is no getting around it, it may also be appropriate to request new or different resources who have familiarity with the technologies that are being introduced, or request a training budget for some of your existing resources. So there is a cost aspect as well.

If you speak the PM's language – schedule and cost – I think you will get more traction than speaking about how developers feel about the resulting design. Those things have real impact.

A PM ought to know better than to introduce stuff like this on the fly with no governance, no controls, and no consensus. If they just don't get it, you may need to escalate to product management or program management, as (s)he is putting quality and schedule at risk unnecessarily.

OTHER TIPS

What would be stupid is to let yourself get death marched.

What you are describing is that you've lost critical feel. There is no sense of control and no clear way back to it.

The last thing you should do is work hard, keep your head down, and quietly suffer until they finally admit that the project is doomed.

What you should do is think very hard about what you have every right to expect.

If they want you to use technologies that you don't understand you should expect time to learn them. Don't be ashamed of what you don't know. Use your ignorance as a cudgel. When they demand you use something ask why. Don't accept 'because'. Don't accept 'modern best practices'. Don't accept 'scale-ability' without getting real, testable, expectations.

By testable I mean they HAVE to tell you how many requests per day/hour/minute they want it to be able to do. Make clear you intend to build something to exercise this system according to those specs.

This way you can use a 30 day free trial to prove that the latest wiz bang thing they want is worth it or if your better off sticking to what you already know.

Now keep in mind. It isn't the tools that introduced race-condition plagued code. You guys did that. You need to learn HOW you did that so you can undo that.

And no. It isn't too costly to revert back to what you had. The PM can't have what they want just by demanding it. You have to push back until you can effectively use what the PM wants or prove it's not what the project needs.

Seriously, just giving in to this is unprofessional and deadly to the project.

I've been here man. More then once. It does make you feel stupid. That's really not it. You're just lost.

Talk to the PM. Honestly. Lay it all out. Show you're willing to learn but don't want to be taken for a ride. Never ever ever design or code based on faith. Make the PM show you how to do what they want. Don't pretend you understand when you don't. Don't say it'll be done when it wont. If you're going to believe in something believe in yourself. You have to be willing to say NO.

If that doesn't work polish the resume because you're going to need it soon. One way or another.

This should really be on workplace.stackexchange.com, because the problem is not really a software development question, but about workplace relationships.

If you are sure that your simple approach would have worked and produced a working result rather quickly, then your PM is a destructive force in your company that should be removed. Figure out how to get the news to the level above him: That your team had a simple, working solution that had made good progress, and for reasons that nobody can explain your PM forced you to attempt a much more complex solution, using a multitude of tools that nobody knows, nobody understands, nobody knows if they are useful at all, and that unfathomable decision of your PM caused you all the trouble and causes the project to be late and not working.

Not knowing the context and the product strategy pursued by your management, it's difficult to objectively answer your question.

Here some objective arguments. It's however possible that it's not what you expected:

  • "Keep in mind that nobody know how to use these products yet".
  • Using only tools and techniques known perfectly will ensure high productivity. But it will considerably limit the ability to innovate. On some markets, it might be fatal to your product. For example, almost 30 years ago, I proposed to use windows 3.0 to develop a new version of a CAD product that was successful under MS-DOS. The product manager objected that this was not a proven environment, that it would be too complex and too difficult to learn for the team, and anyway, that "Windows will never be a mainstream environment"... I let you guess the success of his product 2 years later.
  • It's all a matter of costs and benefits. The cost of experimenting vs. the benefit of a scalability and deployability ensured by a third party that is experienced with huge installations and heavy workload.
  • The drawbacks of adding a new tech can be smoothened, with appropriate training, or initial support/coaching by an experienced consultant.

Ultimately, the economic choice is the responsibility of your product manager. Discuss with him the pros and cons, to ensure that he makes a well informed decision and does not underestimate the added complexity. And if he stays on his track, try to achieve your best: you've nothing to loose and in the worst case, you'll have a new tech on your CV.

There are two approaches to third-party libraries (and other components):

  1. Use as many of them as possible
  2. Use as few of them as possible

My approach is (2). It sounds like your approach too is (2), but the project manager likes the approach (1).

There are three ways you can handle this situation. Either you let the PM do whatever the PM wants, you try to convince the PM to change the approach to third-party libraries, or you vote with your feet and select another job.

If you want to convince the PM to change the approach, consider these arguments:

  • Time to learn. Each external library requires time to learn, in which time a competent programmer may write the desired functionality especially if a huge library was chosen just to do a very simple thing that could be done in few hundreds of lines of code.
  • Replaceability. If you have an external library, how do you ensure that if its development is stopped, you can replace it with another similar library? My solution is to avoid external libraries whenever I can, and whenever it isn't feasible, I write a simple wrapper to abstract the part of the programming interface I want. Usually the interface I want is much simpler than the interface the library offers. Then my code accesses the external library only through this wrapper, making replacements easy. Building your entire application on some framework is a big no-no. Servlets? Yes, they have been here for a long amount of time and continue to be here for the foreseeable future. Template engines? Yes, although they aren't exactly replaceable (you usually pick one and stay with that), the value they bring is huge, so select carefully -- and keep in mind that when switching template engines, you can have two template engines in the same application but you can't have two frameworks in the same application usually. Apache Struts? No, frameworks come to fashion and go out of fashion in a quick manner, and you usually can't have two frameworks in the same application.
  • Version hell. By selecting an external library, you have to update it to avoid security vulnerabilities, and updating it may break things. Well-designed components (such as Java JRE) are compatible with different versions, but my experience is that most libraries are crap due to imposing huge version hell. Also, component X may require Z version 1 and component Y may require Z version 2, and you may not necessarily be able to link Z version 1 and Z version 2 in the same application.
  • Security vulnerabilities. By selecting an external library, the number of easily exploitable security vulnerabilities against your application increases. Some could claim that in-house developed code resembles security through obscurity, but then again I would say it is still a form of security.
  • License issues. Each external library imposes its own license on parts of your program. For example, GPL libraries cannot be used in non-GPL programs, and LGPL libraries also require distribution of the source code along with binaries, which may take considerable amounts of bandwidth.
  • Application startup time. Each big external library slows down the startup time of your application. By writing a simple, lean library in-house, you can make the startup time of your application much quicker.
  • Memory footprint. By having X requiring Y requiring Z requiring A requiring B, you need X+Y+Z+A+B in the memory at the same time. By implementing only the equivalent of X, let's call it X', in-house, you need just X' in the memory. And usually the memory footprint of X' is less than the memory footprint of X.
  • Bug risk. The more lines there are in the external component, the higher is the risk that you will run into a bug that will be hard to fix due to the huge amount of code you need to understand. If you do the thing in-house, you usually do it with less lines of code (to do just what you need, nothing else), and thus, smaller risk of bugs.
  • Customizability. If I write SQL queries myself, I know what the query looks like and how well it performs on a given database engine and given set of indices. If, on the other hand, the SQL query is written by an external component, I don't know anything about its performance. I used to work in a company where each web page took several seconds to fetch. I suspected the cause was the Hibernate library they used fetched too much data automatically from the database when all you needed was one item and not all the items related to this particular item. I left the company before discovering the true cause of the slowness, because I didn't like the approach of using huge numbers of existing libraries.

Beware especially if a library calls itself a framework. This means the library requires you to build your entire application around itself. You generally cannot have two frameworks in the same application; they will fight with each other without peacefully coexisting. Web development utility library? Yes please, there are too few of these. If I ever find a better library than what I use now, I can use the newly found library in new code while continuing to use the old library in old code. Web development framework? A big honking NO!

I think your PM is aiming for a hard to manage system that will produce a lot of maintenance work while it is live, so it will ensure your income.

Personally, you seem to be stuck with python, just forget python for a while, don't code in python for a year, learn new stuff, you will see there are other languages that can do the same, and probably better.

As others have stated, learn the tools before you start coding with them. Maybe suggest that it would be good to evaluate the necessary stack together, based on research of different tools that seem suitable to the task. Or maybe ask how he came up with that list, he could have had help from someone who is up to date.

Developers should not be scared of learning to use new libraries, frameworks, technologies, etc. That is a core part of a developer's job description, and it is perfectly reasonable for someone to suggest that the team work with third party things that nobody has any experience with, or even to demand that the team do so if they're in a position to be making authoritative technical decisions for the team.

However, it isn't reasonable to expect that you can just pull a new technology (let alone several new technologies at once) into your stack and keep making progress. Significant time should have been scheduled to learn the ins and outs of the new approach and figure out a good design to incorporate the new pieces, during which no real progress on the actual product would be expected (from the people doing this learning/design work, which may or may not be the entire team, though if it's not there probably needs to be more scheduled time down the road for the people who learned to transfer knowledge to the rest of the team). That is the cost of making this sort of major change. Learning new technologies is part of the developer's job, but it's not something that just happens with zero time cost.

It sounds like that didn't happen from the question. People dove right in trying to somehow make build good implementations on top of technologies they didn't themselves understand. Of course the resulting code is terrible.

Try to convince your PM that the company will need to spend more time on this. It will either come in the form of stopping now, learning and assessing the new technologies, figuring out a good design and cleaning up the current implementation mess. Or it will come in the form of more time wasted on bugs, maintenance, costlier development, etc.

It's impossible to say whether the technical choices described in the question (load balancing, message queues, etc) are actually appropriate. I don't think that "no-one on the team has experience working with this before" is a good reason to absolutely rule out a decision, but it does increase the short-run cost of making that decision (which can alter the "best" decision for the context), and if your PM isn't considering that and is expecting the team to immediately become as productive as experienced people would be then you should push back on those grounds; they will be setting highly unrealistic project schedules, which isn't in anyone's best interest.

Licensed under: CC-BY-SA with attribution
scroll top