Should I introduce tools even if I might not need them right now

Question

I recently read two articles regarding two different approaches for managing the underlying technologies of a system. The first one was by engineers of reddit, explaining how they gradually extended their deployment and scaling mechanisms, starting from simple python scripts to a full AWS auto scaling solution. The general conclusion of the article was to gradually extend your infrastructure/technologies whenever you have to.

The other article was about kubernetes where the author stated that kubernetes is highly complex but is "essential complexity" because its advantages outweigh the complexity of the setup/keeping it running so you should introduce it early on.

Both views contradict each other in a way. Should I introduce essential complexity early to be future proof or gradually introduce it whenever I need it?

---

Reddit: The Evolution of Code Deploys at Reddit (June 2017).

Kubernetes: Is Kubernetes Overkill? (August 2017).

Answer 1

Some design decisions cannot be easily undone. That's why they are made early.

Despite the warnings about premature optimization, it's actually a good idea to make decent design decisions from the start, including having a reasonable architecture and making sensible use of data structures. Complexity is not necessarily a prerequisite; some of the most highly scalable and maintainable solutions are also the simplest.

However...

When you are a startup, the most pressing need is getting to market quickly, not engineering for massive scale. Some of the most successful startups in the world eventually re-engineered their systems to accommodate huge user bases, including Facebook, Twitter and the almighty Google. But they did that after they already had a large user base and bucketfuls of money, when it became a good problem to have.

Ninety five percent of all startups will never get that large, so engineering your systems from the ground-up for massive scale makes no sense at all.

Answer 2

A lot of people seem to think tools like Kubernetes are only for people who need "massive" scale. In reality, the problems a scheduler solves manifest at very modest volumes. If you try to roll your own solutions, you will very quickly find yourself reinventing the wheel to do things like rolling upgrades, health checks, capacity management, a/b testing, matching dev and test environments to production environments, hardware failures/upgrades, logging, metrics, and so forth. This is stuff that is useful even on 1-3 node clusters.

If you are a big company you can easily afford to maintain a homegrown system while you transition to something else. If you are a smaller company that can't just have a fully dedicated "new scheduler team," it is super painful to need the features of a scheduler, but not have the resources to move to one because they are all dedicated to maintaining your old homegrown system. If I could go back in time to when my team was first thinking it would be easier to use python scripts "just for now," that's what I would tell them.

Answer 3

From your description, I don't see a contradiction between the two, because the problems are vastly different, and I think you may be misinterpreting the meaning "essential complexity"

When the article about Kubernetes mentions "essential complexity", I interpret it as essential vs. accidental complexity*.

• Essential complexity - the complexity in the solution necessary due to the complexity of the problem domain. E.g. if doing an accounting application, the complexity regarding double entry bookkeeping would be essential complexity.
• Accidental complexity - the complexity introduced when we make a software application. An example of accidental complexity would be code to handle orphan removals in your relational database because your ORM didn't handle it well. Some accidental complexity is unavoidable, e.g. dealing with scaling.

So let's examine the problem of deployment for the two cases mentioned:

• Reddit - Essential complexity relates to posts and comments, etc. The complexity introduced in deployment scrips falls under accidental complexity.
• Kupernetes - Being a tool related to deployments in a scalable environment, these two things becomes essential complexity (I have very little experience with Kubernetes, so how well it actually solves these problems is not for me to comment on)

So if you talk about introducing a tool that you don't need right now but might need in the future, it sounds like unnecessary accidental complexity.

Remember, just as Robert Harvey wrote, for a startup, time to market should be your concern. And it will be perfectly fine to introduce some accidental complexity now in order to achieve a faster time-to-market. You might also introduce a tool that you need right now, but not in the future!

And be careful about making hard-to-change decisions early. Early is the time when you have the least knowledge to help you making the right decision. If you can set yourself up in a way where you can postpone such decisions, you will help yourself quite a lot.

And also remember that scaling problems can be dealt with gradually. Normally you don't need to deal with massive scale on day 1 (replacing an existing application with a large user base being the only exception I can think of right now)

*I believe the terms Accidental and Essential Complexity were introduced in the book, "No Silver Bullet", which I unfortunately haven't read yet