An older, experienced contractor used an SQLite DB for various queues - am I, a young dev, justified with feeling uncomfortable with it? [closed]

Question

Closed. This question is opinion-based. It is not currently accepting answers.

---

Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.

Closed last year.

Improve this question

Context: I'm an embedded dev with only 2 years of solid experience. I'm the sole technical employee of a startup of 4 people. We have an MVP of our product out and are getting ready to develop the next iteration of it. The original MVP was developed by a partnered contractor team, with one older embedded dev doing everything software. I joined the company too late to have any input into the design of the MVP. The product is a gateway-type device: embedded Linux, messages come in one way, some limited intelligence happens inside, they come out on the other side.

The 'problem': Everything in the system seems to be chucked into a single SQLite database. Processed and unprocessed messages live in the same table (with one field used to indicate which one they are), provisioning related things live another table, even the logging and debug is done by writing to yet another table in the database. The system is written largely in python but the biggest part of it is a massive class full of wrappers for complex SQL which seem to do the bulk of the data manipulation.

All of this makes me uncomfortable, especially the messages part as it looks like a classic case of "database as a queue" design anti-pattern. That being said I find it difficult to articulate exactly why this is wrong to my non-dev boss, other than vague and nebulous mentions of maintainability, difficulty in introducing changes because of lack of modularisation, as well as of lack of clarity in how data flows and is being processed. It doesn't help that the contractor has "authority of years of experience" over my judgment.

Am I justified in feeling uncomfortable about this design choice? I mean, the thing works in principle. I know that the desire to refactor can be pretty strong, irrational and should not always be acted upon given business constraints. But then... I kinda feel like starting mostly from scratch only ripping out useful bits of the first MVP would be cleaner and take less time than working with the system created so far. But is it just my younger-person-hot-headedness?

Answer 1

Your first problem is that you think of the design as "wrong." That's really not the right way to consider things. Rather, different designs make different design trade-offs. Any design has pros and cons that have weighed against each other. If there is a problem with the design it's not that is "wrong" but rather that it makes a poor choice of trade-offs. Don't think of it or explain it as, the design is wrong, rather think in terms of how a better trade-off might exist.

The fact that you are using SQLite and an embedded Linux system makes database as queue far less problematic then it would be in other circumstances. Database-as-queue has two major drawbacks:

1. Database operations are fairly expensive, reducing the performance.
2. New messages have to be detected by repeatedly querying the database to check for them.

However, SQLite operations are much less expensive than a typical database, so the performance loss is minimal. SQLite also supports a data changed notification that allows the program to detect changes to the database without repeatedly querying the database.

SQLite gives you a lot of functionality for free. Basically, you define your tables, and SQLite takes care of persisting them to disk, loading them back when you restart, ensuring that the file does not get corrupted, giving you powerful querying tools, allowing complex operations on the data. Furthermore, SQLite gives you all of this without configuring other applications, its just a library your code is using. If you don't use SQLite for all of this stuff, you'll have to implement that functionality yourself.

In my assessment, using SQLite in this way is a pretty sensible choice. Your response comes across to me as somebody who hasn't used databases enough to know their power. Of course, there may be other considerations I don't know that change that, or it could be the system is simply poorly constructed in other ways.

But if you do want to take your concerns to your boss:

1. Make a good faith effort to work with the system as it stands first, that will make it more plausible when you make the cast. Also, you'll have a much better idea of what does and doesn't work. And maybe you'll learn to love SQLite.
2. Be able to identify precise things your boss cares about: better performance, more features, etc. in making the case.

Answer 2

I think you are making a common junior-level programmer error in your judgement here. Because the solution is not the way you would have done it, you want to redesign it. One of the most important things you need to become comfortable with is that whether the design is the way you think it should be doesn't matter. It really, truly doesn't matter at all. All that matters is whether it works, whether it can be improved, and what what it will cost (time and money) to make those improvements.

You haven't mention a single concrete problem. You see a design you don't like. But why should your manager care? Have you produced any working production software solutions of a similar scale and complexity? They've got a perfectly good milking cow and you want to them to trade it in for some magic beans.

Let's start with the assumption that this software works. It's reliable and performs well. Customer satisfaction is high. Now you come in and tell the manager you want to rewrite it. Basically you are saying that the initial time and investment that went into building this is a sunk cost and that you want scrap a working, profitable piece of software for your unknown abilities.

Writing the code is just the beginning of getting to production. Most likely there are a lot of technical challenges that are solved in the current solution that you aren't aware of. Taking on a redesign means you will need to re-discover the things that were already solved. Along the way, you are bound to introduce new issues. There's a good chance you will create new production bugs. And you want your manager to sign up for that time, cost, and risk with zero concrete benefit? Do you understand why that's not a good choice on their part? Being non-technical actually makes it more likely the manager will allow you to do this, not less. It's much easier to convince a non-technical manager that an unnecessary rewrite is worth doing. A competent manager will know there's no virtually no upside to letting the junior developer rewrite the veteran's working solution.

Now, let's say the current implementation is buggy, slow, and/or unstable. Maybe the solution is brittle and making a few changes will take longer than a full rewrite. That's a different situation entirely. But you still might not win the argument: sometimes the cost of remediation is greater than the benefit. I've lost count of the times I've had to work within the confines of a bad design because a rewrite was not feasible or justifiable.

At this point in your career, you should be less worried about building it your way and more about learning from what others have done. It's not always going to be the case that you learn good ways to do things. My first few years as a developer were spent wrangling some pretty atrocious code within a poor design built with a dying platform. I hated it at the time but it was some of the best education I could have hoped for. I learned a lot about how not to do things and why good design matters. A wise person once told me: "smart people learn from their mistakes. Really smart people learn from the mistakes of others."

Answer 3

I can put myself in your shoes, and also in the contractor's.

Ask yourself the following questions:

First of all, how long will it take to change the design? Is it really worth it?

Effort in adding new functionality to the product.

Good points of the design as it is (this is the key as he has widely more experience than you, it could be designed for certain users, ...)

Have you had a handover of the product? Maybe that's what you need, you don't understand the product, motivation and its design.

It may not be so bad, just a little more info required