What is the most egregious waste of money you have seen, and what did you do about it? [closed]

softwareengineering.stackexchange https://softwareengineering.stackexchange.com/questions/18585

Question

Often we as programmers see large organisations wasting huge sums of money on bloated and inefficient solutions to problems. This pains me greatly because I like organisations to benefit from best of breed solutions. However, my abilities as a programmer are limited when it comes to influencing the key decision makers and often my perspective on the matter is constrained to my own little technical world.

So, my question is this. After encountering an egregious waste of money on some software and/or hardware that really got your goat, what did you do about it to get it fixed or were you doomed to bite the bullet and mutter forever under your breath? I'm interested in hearing your overall experiences and especially what lessons you learned about how to tackle this sort of thing in the future. Let's not name names, the experience of how to tackle the problem is more important than the actual offending product.

Was it helpful?

Solution

Paying for big, bulky, buggy commercial products, in the range of:

  • Application servers;
  • Testing tools;
  • Development environments.

when the open-source or light-weight alternatives are obviously superior.

My steps usually are:

  1. establish an alternative as a reference - e.g. "I will experiment with the app server X instead of app server Y. I had good experience with it, because (...).";
  2. sell this proposition to my colleagues - "I am now developing faster since server X restarts much faster and I don't wast all that time";
  3. sell this to the immediate manager - "Our team now develops faster since we're using server X. It all began as a little experiment but everyone likes it.".

OTHER TIPS

I've seen too many examples to name a favourite, but I've noticed a few general trends in my main field, web-development:

  1. Vanity Websites. These are websites that serve no useful purpose to anyone outside the small organisation that commissions them and are built around an obsessive compulsion with logos, photos of themselves and self-indulgent waffle. The worst part is these are usually public-sector funded and commissioned by people who have no clue about the web. (For instance, once had a NHS hospital trust who wanted to develop a mini-version of Facebook for their own staff intranet).

  2. Paid for is Best. The mindset that insists that paid-for software must intrinsically be better than open-source. After all, it's paid for, right? I've seen so many clients insist on making stupid choices simply because they work in a culture that automatically discounts anything open-source as a matter of policy.

  3. Design by committee. This is where a huge group of people have a "brainstorm" and then try to incorporate every crack-pot idea there is into the design, inevitably resulting in a ill-thought through mess that compromises on everything in favour of trying to please everyone (and by everyone they mean the committee making the decisions, not the people having to use the application).

  4. Consultants. This is where you pay a middle-man (who knows neither business practices nor software development) to get in the way and cream-off money by protracting the development process with confusing techno-babble and business-speak.

I don't see that anyone has mentioned this one yet.

Building your own solution when you could buy it.

Variations of this pattern:

  • not even considering the buy-vs.-build tradeoff
  • significant scope creep of the in-house solution
  • limited scope, but also limited utility of the in-house solution

My two favorites:

  1. Hiring consultants (freelance) just to add more production capacity, while they should invest in their own employees instead, by hiring consultants to bring new knowledge and coaching their existing people.

  2. Hiring project managers that manage other project managers that manage other project managers that finally (think) they manage the development team. While they should let the team self manage and focus on business instead. I've seen software projects where they had more project managers than developers. Imagine the meetings.

Limiting Long-term Raises and Bonuses

I think its taught in Business 101 to not give employees raises. A secondary case is to limit salaries of star performers because they need to fit inside of some certain salary range.

Eventually employees will realise their pay-scale is not in-line with their industry (or output). The people that have the resume and skills will eventually leave, and take with them all their knowledge and probably a few of their friends. The people remaining (who are the bottom performers) will have to pick up the slack and then spend more time hiring a new person (at market rate). So the company just traded a star employee for JR level one, and just lost all the "savings" of keeping the salaries low.

As this continues, the development team will struggle to stay on par, and will probably get worse and worse until something drastic is done.

This answer is somewhat different than most: not firing an employee soon enough, or stated differently, being overly tolerant of an employee's mistakes habits. These were things I've observed and couldn't do much about as a consultant.

  • The dev that poorly drove the design decisions of a project that led to its eventual rewrite (it was a complete mess).

  • The dev that sent sensitive unencrypted data to Google Charts because they thought it would be cool to show a pie chart (was a pie chart a requirement? Nope!).

  • The dev that consulted with a company in the past and accepted a position with them directly. He did an about face and turned into a prima donna that sought the Technical Lead position and went as far as talking to the lead's manager stating they thought it would be good for him to take over as the lead. Talk about audacity! A lot of devs no longer like the guy and he burnt a lot of bridges within his first 2 weeks as an employee. To top it all off he's a very green dev that only graduated 2 years ago but thinks he's awesome.

A few mistakes are understandable, but when there's a consensus between many devs about someone's attitude or skill level companies ought to get rid of them sooner rather than later.

Several times I witnessed management bringing in consultants for the sole purpose of spending money. Most of the time this happened at the end of the year when they were under-budget frantically trying to spend the money. Usually these consultants would be paid hundreds of dollars an hour and they would spend weeks on a PowerPoint presentation that would never be used.

There is a much bigger issue at play here.

Many companies have one goal - to increase shareholder wealth. What they produce is irrelevant. How they produce it is irrelevant. How much waste they produce is irrelevant. The cost to society and the planet is irrelevant.

So - go work for or start a company that does something of benefit for society / the planet.

Paying large software companies not only for their product, but for their "support."

I was working at a Government agency for a team that was deep in bed with Oracle. Over the course of many years, they had been paid bajillions of dollars for their software. Coming from a startup background this made no sense to me - "why not use MySQL or Postgres?" I was told that it was mainly because of the support that Oracle provides, if something goes wrong they help you find the solution quickly.

The support was an absolute joke. There was a problem where one web app kept crashing the whole system. It appeared to be a result of a slow database query with a combination of horribly written code (which was written by a team of consultants, which should be a whole other answer). A "task force" (groan) was assembled to pinpoint the issue and fix it. Included on the task force was an Oracle support member. Every day at EOB there would be a conference call where the task force members would update the rest of the team with the findings. It was a long enough call that nobody wanted to be on b/c it started at 5, and the Oracle person just made it worse. Why? Well, saying "person" isn't even correct. It was a number of people. It seemed like every two or three conference calls, the Oracle rep would be somebody new, who explained their predecessor was now on another project or had gone on vacation. The new people were never briefed by anybody at Oracle, so every time somebody new came in we had to waste ten minutes of the conference call explaining the problem again. Their contribution would then be asking for J2EE log files, which not only can any monkey read, but were also useless because the horribly written code was doing things like throwing IndexOutOfBounds exceptions when the programmer found errors in parsing XML.

Having programmers to 1st line phone support.

Having programmers do testing.

I know that this is an old question & I'll be lucky if 3 people read this answer, but it's a fun story to tell, so what the hell.

I came into a project (embedded systems, safety-critical firmware, very high stakes) and I was appalled by what I found. People using C (especially pointers) incorrectly, no static analysis, no code reviews, no testing other than "integrate it together, run it, beat on it, see what breaks."

I wrote a very long email my first week there (as a consultant). It was dicey because I was basically saying it was mis-managed, the developers were in over their heads, no process was being followed, etc. It should have gone to the corporate VP, but instead I sent it to the development manager who hired me. He was not entirely defensive about it, in fact he acknowledged many of the shortcomings & told me I wasn't the first to point them out (no kidding, right?)

To answer the crux of the original question: I offered to spend AT MOST 1 man-week getting Gimpel's Lint (PC-Lint / Flexelint) static analysis tool configured & running on their platform, and to run a full report of everything that was found. I told them I was absolutely sure that we'd find several lurking "timebombs" as a result.

They calculated my hourly rate, multiplied it by 40, and determined it was "too expensive to do that." Long story short, I left there within 60 days. About 3 years later, I learned of a product recall, the cost approached 9 figures ($100M), not to mention damage to the company's reputation.

I won't mention the company, the product, or the industry, but I still keep in touch with one of the engineers there, and when he explained to me what caused the recall, my eyes rolled - it was a problem that would have been caught by even a basic static analysis tool (accessing an array out of bounds). In fairness, I cannot say with certainty that the problem was in the code when I was there, but I'm sure if they'd spent the money on some kind of static analysis tool, that bug would not have escaped.

So they saved $295 by not buying PC-Lint (OK, they also saved a week of paying me, at most) - but I'm nowhere good enough to charge $100M for a week.

That's what I call a pretty damn big waste of money.

Reminds me of a joke that many of you may have already heard:

Ever heard the story of the giant ship engine that failed? The ship’s owners tried one expert after another, but none of them could figure but how to fix the engine. Then they brought in an old man who had been fixing ships since he was a youngster. He carried a large bag of tools with him, and when he arrived, he immediately went to work. He inspected the engine very carefully, top to bottom.

Two of the ship’s owners were there, watching this man, hoping he would know what to do. After looking things over, the old man reached into his bag and pulled out a small hammer. He gently tapped something. Instantly, the engine lurched into life. He carefully put his hammer away. The engine was fixed! A week later, the owners received a bill from the old man for $10,000.

"What?!" the owners exclaimed. "He hardly did anything!"

So they wrote the old man a note saying, "Please send us an itemized bill."

The man sent a bill that read: 

  Tapping with a hammer ........ $ 2.00

  Knowing where to tap ......... $ 9998.00

Effort is important, but knowing what you're doing makes all the difference.

Bloated development teams and terrible productivity in software companies.

This is a consequence of the common pattern in the business world: the importance of a manager is measured by the number of subordinates, therefore a manager's number one concern is not productivity but quite the opposite: poorer productivity is the best justification to hire more people.

In a company that sold software...giving salespeople full commission on all custom mods sold, so that selling something that already existed and we could just profit on was not nearly as profitable for them as selling one-offs. This was combined with moving the sales staff halfway across the country from the technical staff.

This also meant that we in Development couldn't possibly meet the sales deadlines, making customers unhappy, and had a great deal of difficulty in getting any core work done that would make the product better for everybody. The increased pressure caused code quality to decrease and hurt morale, particularly when we heard stories about the sales office (which I never confirmed).

Lots of us resented Sales, but in fact it wasn't their fault. They were going out and selling as much as they could, doing what they were rewarded for in accordance with the limits that had been placed on them. It was bad management that caused all these problems.

There are two that I have experienced.

  1. Cancelling a project that had a huge ROI for the business that was about 80% completed and then handing out 100 engraved and gold plated iPods to senior executives.

  2. Laying-off several hundred people and then the following day announcing substantial pay raises and bonuses for the senior executives.

These are not totally programming related, but most certainly wasted a lot of money, plus provided a slap to the face for all those involved.

I didn't get laid off, but I also didn't get a raise or an iPod either...

I've seen a couple of horrible outsourcing projects that succeeded in significantly increasing costs while either failing to increase or actually reducing efficiency.

In the worst case the new outsource team were put in place and up skilled but the existing on-shore team remained in place because the outsource team weren't trusted to actually do any of the critical work.

At this point the logical thing to do would obviously have been to accept failure and shut down the outsource team but because management weren't willing to publicly admit that it hadn't worked both teams were left in place (at a significant increase in cost with no increase in efficiency or usable capacity) until the whole thing could be buried.

In another instance development was outsourced and the original team laid off. Two years later they realised that it hadn't worked and paid to bring the whole lot in-house again only to find that in addition to the very significant costs of another handover, the impact of lost knowledge, recruitment fees, contract terminations and so on, the outsource organisation had lost a significant chunk of the source code.

(Note: I'm not saying outsourcing can't work, just that too many times people are seduced by potential savings and don't consider the realities of their new world, the change to process and working practices and so on which leads to majorly screwed projects)

Technical Debt

I've seen is the chronic "beating the dead horse" of legacy code. Or more to the point, from the viewpoint of the trenches, countless hours spent in maintenance mode when the whole team knows we should be in replacement mode.

What we've done.... is still on going. Trying to invoke positive change from within

Performance Testing

Simply, not doing it. Again, still working on the positive change from within.

I've been working with a few state institutions and they are amazing at wasting money on IT. From buying bloated middleware to solve extremely simple problems to paying thousands and thousands of dollars to a vendor to have them create a CSV. Without in-house people with sufficient experience it seems they either get fleeced on the upfront cost or on the maintenance.

In non-software companies (banks, insurance) with in-house IT the money comes from various business groups. The business groups directly gets sales pitch from vendors and will push it on to the IT. They are paying for the software/hardware and your salary so your protests will go no where.

  • Paying for bloated applications and middleware that costs mid-high five figures and doesn't even fit within the existing system architecture
  • Using expensive software like HP QualityCenter, BMC Remedy, HP LoadRunner etc where better and cheaper options are available
  • With multi city teams a lot of travel costs, sometimes for only a few hours of meeting
  • Paying for windows 7 license that somes with new machines and then paying again to downgrade to windows XP as the new SOE (designed in 2010) is still XP
  • Over capacity in hardware

I work in the performance testing profession and I witness (literally) millions of dollars a year being flushed down the drain by organizations for four reasons

  1. Hiring an outsourcer based on price alone, not qualifying skills and not regularly auditing the skills of the performance testers. Hiring an amateur performance tester is a lot like hiring an amateur plumber or an amateur electrician, it will take them a lot longer to work through basic tasks, a lot of checks and balances in process are lost, and when you do find out just how bad they were its horribly expensive to fix (in production). As a moderator for half a dozen forums in this field I observe regularly people showing up who lack fundamental skills in testing, communication, project management, development, systems analysis, etc... and they have simply been thrown at a tool. To the person that noted LoadRunner as a waste of money earlier, if you throw a fool at a tool there is only one result you should expect. The irony is that open source tools require an even more mature skill set to be successful with them.

  2. Not collecting performance requirements. This impacts the whole organization because you will have a different perspective on performance in Architecture, Platform engineering, Application Engineering, Functional QA and performance QA, none of which may actually match the business stakeholders (and frequently doesn't). This is a process problem for in many organizations the performance test team is asked both to collect the performance requirements and to test against them. For proper checks and balances you should do one and not the other. Related to 1 above with immature staff you will have people who cannot even recognize a proper performance requirement, do not have a measurement point to validate against with a load profile, and yet they are still building "scripts to run." This is a collosal waste of time and effort and does little to improve quality. Performance needs a common perspective across the organization and is not something that can just be tacked on at the end if it wasn't engineered in to begin with.

  3. Performance Test Environment Management. I cannot tell you how many organizations are delayed to to test environments not being ready to run at the time that the test organization is ready to proceed. Just in one client I can see this as a multi-million dollar problem in terms of hours lost while waiting

  4. Project managers who have no understanding of what performance testing is, what tasks are involved or the level of effort in place, but who are dictating how long the activities should take place. This leads to variances in the project schedule which are entirely related to how items were scheduled (and cost overruns as a result). This is directly related to 1 above as well for immature testers are not able to accurately project either the number and types of tasks nor how long the tasks should take. It is an axiom that if you allow someone who does not understand what you do and why you do it to dictate how you work and how long you will take, then this path will lead to failure. It happens all too often in performance testing.

Proprietary version control systems. Given the state of Git and Mercurial, I don't see why people would go for something with a gate keeper.

Not only do you have to pay for the VCS, you also have to pay per user. Additionally, your flexibility gets shot in the foot. You might as well wear a T shirt that says "I ♥ Vendor Lock In!!!"

I feel that it's just nuts these days to not use a free (D)VCS. If you want lots of add on perks to go with it, stuff like Kiln is available.

I don't think I'd go to work for someone that insisted on BitKeeper or similar.

I almost said the same thing about emulators, but products like Simics continue to offer significant advantages over free alternatives.

Status meetings & weekly reports

An organization I worked in was all about weekly status reports - rolled up at 3 different levels. The dev leads and test leads for each of the 4-6 projects in flight report their progress in a lengthy email, which then get rolled up by the next manager up, which in turn gets arbitrarily summarized by the next one.

The following business day, all project leads gather in a 1 hour meeting to go over the report.

Effectively one day each week is spent on reporting that week's progress. Keep in mind that this is all separate from the daily standups and weekly demo / retrospective meetings.

I work for a public body. There's really no way to adequately explain the level of waste that can go on when the workplace is so heavily legislated and unionised that sacking someone is practically almost impossible.

Managers play pass the parcel with bad staff, and hope to remove them all at once under cover of restructuring. Some bad staff are promoted, just to move them out of an area that needs improvement. Any good staff end up struggling constantly just to make up for the bad staff's work. Staff you wouldn't keep for 3 months forge 40 year careers. The amount of money they waste over such careers is astronomical.

I worked in the private sector previously, and saw a lot of waste, but public sector waste is a whole different sport, let alone ballgame.

It was suggested in a comment that establishing sinecures for underperforming staff would help. It would help in that it would limit the damage they could do, but wouldn't impact the root causes of the problem. I think the best thing would be the adoption of some private sector hiring and management procedures, and changes in legislation to make it easier for public bodies to let staff who underperform go. Unions should also change their policies in consultation with the government - their role of protecting their members is important, but they should recognise that sometimes their members are truly out of their depth, and should be moved on

One project I worked on with a large financial institution. There were huge amounts of conference calls daily, and I estimated that they burned about $100k per day just on conference calls. The project lasted about 2 years. They had tons of legacy systems and when the daylight savings changes were made a couple years ago, they paid Microsoft about half a million dollars to come up with a DST patch for NT 3.51.

We were having a smallish amount of work and barely making bills and payroll in a small shop in which I worked. The solution: hire an efficiency consultant and a personal secretary for the boss so he could go perform more "meat and potatoes" work.

Solve a budget shortfall by increasing spending...fail.

On the plus side - the efficiency expert supplied a dry erase board where we tracked our billable hours and paid hours...guess who had the least amount of billable hours.

Let's see, we once spent well over half a million dollars doing the work to win a million dollar contract. So much for the profit on that one. Some of us on the project proposal development team tried to point this out but it had become a thing of pride for our little company to win over the Fortune 500 companies we were competing with. We did win and lost money hand over fist onteh contract for that and other reasons but we had bragging rights.

As a government contractor once, I was forced to work unpaid overtime because the contract allowed it and the contractor got paid for my overtime. Not only that I was caught up on my work and spent 4 hours every Sunday surfing the Internet with no work to do. Needless to say I moved on very quickly after they started that nonsense.

Buying Clarity as our project management system, a commercial app that is so bad, 100% of the people who use it have begged to go back to our old home grown system (the one guy who liked and chose it has moved on to some other company), people even volunteered to work on their own time to add the reporting they wanted to our old system. But we have invested the money so we are stuck with it. In other words, refusing to ditch something that doesn't work just because it was expensive.

Sheer waste. An IT spend that had to be cut by many millions. So the way to do this was to fly the IT people in from all over the world. Put them up in a flash hotel for a week. Then in the building where the meetings were held, lay a new floor. Marble of course. And overnight, between the meetings each day, the building was redecorated. Thats every evening for a week.

Err... priorities anybody?

Fantasyland.

The company I work for paid $800 for a CHART FX License - It's not even my money yet I feel robbed.

http://www.softwarefx.com/sfxNetProducts/ChartFX/

Just for kicks, their software will place files all over the place including the registry and program files.... yep all that for some naff looking charts.

Licensed under: CC-BY-SA with attribution
scroll top