Overengineering can kill your product "Product people - Product managers, product designers, UX designers, UX researchers, Business analysts, developers, makers & entrepreneurs November 11 2021 False Guest Post, Mind the Product Mind the Product Ltd 1793 Product Management 7.172

Overengineering can kill your product

BY ON

Today’s post is not directed only to product managers. Founders, investors, or any other profile with enough skin in the game on any digital product or service could also take advantage of it.

I believe it because we will talk about one of the most prevalent issues when creating products: overengineering them. In my opinion, overengineering has killed more products than the absence of good development practices.

Before getting into detail, let me talk to you a bit about my background. Before being a product manager, I was an engineer. In fact, my formal training was in computer science. While during my career, I’ve always been closer to the business than to coding myself, I’ve created, scaled, and managed both engineering and product teams in equal measure.

So it’s not that I talk about overengineering from the outside. I’m guilty of having caused and suffered it. Because of that, I know it’s crucial to learn what it is, its costs, and how we can prevent it.

What is overengineering?

If we go to the strict definition of Wikipedia, over-engineering refers to the fact of designing a product in a more complex way than necessary:

Overengineering (or over-engineering, or over-kill) is the act of designing a product or providing a solution to a problem in an overly complicated manner, where a simpler solution can be demonstrated to exist with the same efficiency and effectiveness as that of the original design.

In the context of software, I like this definition from Paweł Głogowski:

Code or design that solves problems you don’t have.

Now you will think, who designs or codes to solve a problem that she does not have? It seems ridiculous, right? Well, hold on to the chair because, after two decades of career and having done it myself, I can assure you that over-engineering is not the exception; it is the norm.

Read this article on how to get design agencies and engineering teams to work together.

Overengineering causes

Nobody does it with bad intentions. On many occasions, overengineering happens because we try to anticipate the future and be ready for the unknown.

When we code a feature, it is easy to think that we can make it future-proof by investing a little more time “just in case”.

The reality is that nine times out of ten, that “just in case” never comes. But along the way, we have lost valuable time and increased the complexity of the project, something that we will carry throughout its whole life.

xkdc comic strip
The general problem — (Image: XKDC)

Another cause of overengineering is often a lack of experience. In general, the more senior you are, the less prone you are to overengineer because you’ve already lived through quite a few situations where artificial complexity has exploded in your face.

The learning curve with respect to the experience of an engineer usually follows a pattern very similar to this:

  1. You start by programming in a straightforward way.
  2. You discover a paradigm such as object-oriented programming and jump on the bandwagon.
  3. You read about design patterns and start looking for the opportunity to implement them in every situation.
  4. After a few years of having suffered unnecessary complexity, you go back to writing straightforward code.
Code Complexity vs. Experience
Code Complexity vs. Experience — (Image – flaviocopes)

Loosely defined requirements add to this problem as well. If an engineer does not have a well-defined problem, he will tend to overengineer to protect himself from uncertainty.

Boredom can also lead to overengineering. If an engineer does not have exciting challenges to face, he may end up complicating any problem simply by trying something new.

Overengineering consequences

When I said at the beginning of the article that overengineering kills startups, I was not kidding. It has two particularly perverse effects on any system.

On the one hand, it increases our development costs. If our engineers do not choose the simplest solution to address a problem, our costs in time and money increase, preventing us from iterating faster. Watch this MTP Hamburg talk on Iterating Your Voice Product, by Lisa Vigar.

On the other hand, it also increases your maintenance costs. Simple code is much easier to program, test, and modify. When we complicate it, the complexity can grow exponentially, impacting our iteration speed.

So I reaffirm myself. Overengineering kills products. Much more than the absence of good engineering practices. It is this way because to benefit from good practices you need to have product-market fit, while overengineering can prevent you from getting it in the first place.

Overengineering examples

The first that comes to mind is microservices-based architectures. They came like a wave a few years ago, and they should have taken down more projects than those they have consolidated.

I put them as an example of overengineering because they are not necessary in 99% of cases, especially for a startup that has to find market-fit and will benefit significantly from using a more straightforward architectural pattern like a “majestic” monolith.

If you succeed in finding market-fit and it turns out that you end up needing to switch to microservices due to scale problems, oh boy, that’s a good problem to have.

Premature optimization is often another typical example of overengineering. A common situation is preparing a system to absorb a large amount of traffic with an overly complicated infrastructure setup when you still don’t have users. In most cases, having a monolith running on a single server is all you will need to validate your business model.

One of the worst examples of premature optimization occurs when we spend a lot of time designing a system trying to prevent repeating ourselves in the future and having to throw away part of the work done.

It doesn’t matter how perfect your design or implementation is if you never see it working because you go broke before. The worst code on the planet that helps you validate a hypothesis is better than standing still for fear of not repeating yourself.

Related to the above, software rewrites are an obvious example of overengineering. Usually, engineers do not like to work on legacy codebases. Their natural tendency is to do everything from scratch. But as Joel Spolski wrote more than 20 years ago in Things you should never do, rewrites rarely serve their purpose and can even take your business away.

It’s obvious to say it, but your client doesn’t care how elegant your system is on the inside. Your client cares that you help him solve his problem. Every minute invested in not giving them value is a wasted minute.

How to prevent overengineering?

The best way to prevent over-engineering from my point of view is to turn your engineers into true product engineers.

We can achieve it by involving them in the day-to-day business, explaining the why after each initiative, and linking it with the metrics that matter for the organization and its vision.

Watch this MTP panel to find out more about defining metrics that matter.

We need to bring them closer to our users, inviting them to interviews and discovery sessions with them. You want your team to empathize intimately with your user’s problems, so they can quickly discard any engineering effort that won’t solve them as efficiently as possible.

Don’t expect your engineering team to be motivated to avoid complexity if you treat them as mere production chain resources whose sole task is to implement user stories from a backlog. They need to understand the why behind each decision.

Related to the above, define the problem well to reduce ambiguity. Your engineers need to know the why, but they also need to know what to expect. The more you can narrow down the problem, the less reason they will have to protect themselves from overengineering a solution. A good way to define the expectations of a system is by using service objectives using SLIs and SLOs.

Engineers are one of the most creative profiles in any company. If your team trusts your criteria, day-to-day ideas or initiatives will likely arise on their part that may show that they are thinking about a future “what if” scenario. When you have the intuition this could be the case, ask: How does this help solve our current user’s problems? What happens if we don’t do it now? The answers will help you to discriminate if it is a possible case of overengineering or not.

Lastly, more oriented towards founders, prioritize hiring senior engineers who have already had a few experiences in product companies. Look for war scars during interviews. Ask about their most painful experiences and how they dealt with them. Stick with those who put the user and simplicity ahead of simply technology solutions.

Mental models to prevent overengineering

YAGNI

The problem of overengineering is so prevalent in the industry that the engineers themselves have a term to refer to the situation of adding code “just in case”: YAGNI, an acronym for “You are not going to need it”.

YAGNI tries to prevent you from adding anything that is not strictly necessary to solve the problem you have in front of you because the reality is that most likely, “you won’t need it”.

KISS

The term KISS, “Keep it simple stupid”, refers to the fact that simple systems are easier to repair, evolve and maintain. Therefore, simplicity should be one of the goals of any design, avoiding any unnecessary complexity.

Worse is better

With Worse is better, we emphasize that there comes the point when having fewer options is preferable to having more. It’s this way because it can simplify the use of our product, making it attractive to a broader segment of the market.

In other words, it encourages us to maintain the minimally essential functionalities of a product, avoiding adding fat that can add complexity to it.

Conclusion

To wrap up, overengineering has the potential to destroy your startup, it can:

  • Add unnecessary complexity.
  • Increase development and maintenance costs.
  • Reduce your iteration speed.
  • Avoid you from getting market-fit.

Unfortunately, overengineering is no exception; it is the norm. For this reason, it is vital to know what it consists of and try to prevent it mainly by involving and bringing your engineers closer to your customers’ real problems.

Every minute we invest in development that doesn’t solve an actual customer problem is a minute wasted. Avoid falling into the “just in case” trap.

The graveyard is filled with exquisitely designed startups and products to scale to millions of users who never got the slightest bit of traction. Don’t become one of them.

Want to hear more from Simón? Have a read of his recent articles:

Today's post is not directed only to product managers. Founders, investors, or any other profile with enough skin in the game on any digital product or service could also take advantage of it. I believe it because we will talk about one of the most prevalent issues when creating products: overengineering them. In my opinion, overengineering has killed more products than the absence of good development practices. Before getting into detail, let me talk to you a bit about my background. Before being a product manager, I was an engineer. In fact, my formal training was in computer science. While during my career, I've always been closer to the business than to coding myself, I've created, scaled, and managed both engineering and product teams in equal measure. So it's not that I talk about overengineering from the outside. I'm guilty of having caused and suffered it. Because of that, I know it's crucial to learn what it is, its costs, and how we can prevent it.

What is overengineering?

If we go to the strict definition of Wikipedia, over-engineering refers to the fact of designing a product in a more complex way than necessary: Overengineering (or over-engineering, or over-kill) is the act of designing a product or providing a solution to a problem in an overly complicated manner, where a simpler solution can be demonstrated to exist with the same efficiency and effectiveness as that of the original design. In the context of software, I like this definition from Paweł Głogowski: Code or design that solves problems you don't have. Now you will think, who designs or codes to solve a problem that she does not have? It seems ridiculous, right? Well, hold on to the chair because, after two decades of career and having done it myself, I can assure you that over-engineering is not the exception; it is the norm.

Read this article on how to get design agencies and engineering teams to work together.

Overengineering causes

Nobody does it with bad intentions. On many occasions, overengineering happens because we try to anticipate the future and be ready for the unknown. When we code a feature, it is easy to think that we can make it future-proof by investing a little more time "just in case". The reality is that nine times out of ten, that "just in case" never comes. But along the way, we have lost valuable time and increased the complexity of the project, something that we will carry throughout its whole life. [caption id="" align="alignnone" width="550"]xkdc comic strip The general problem — (Image: XKDC)[/caption] Another cause of overengineering is often a lack of experience. In general, the more senior you are, the less prone you are to overengineer because you've already lived through quite a few situations where artificial complexity has exploded in your face. The learning curve with respect to the experience of an engineer usually follows a pattern very similar to this:
  1. You start by programming in a straightforward way.
  2. You discover a paradigm such as object-oriented programming and jump on the bandwagon.
  3. You read about design patterns and start looking for the opportunity to implement them in every situation.
  4. After a few years of having suffered unnecessary complexity, you go back to writing straightforward code.
[caption id="" align="alignnone" width="1375"]Code Complexity vs. Experience Code Complexity vs. Experience — (Image - flaviocopes)[/caption] Loosely defined requirements add to this problem as well. If an engineer does not have a well-defined problem, he will tend to overengineer to protect himself from uncertainty. Boredom can also lead to overengineering. If an engineer does not have exciting challenges to face, he may end up complicating any problem simply by trying something new.

Overengineering consequences

When I said at the beginning of the article that overengineering kills startups, I was not kidding. It has two particularly perverse effects on any system. On the one hand, it increases our development costs. If our engineers do not choose the simplest solution to address a problem, our costs in time and money increase, preventing us from iterating faster. Watch this MTP Hamburg talk on Iterating Your Voice Product, by Lisa Vigar. On the other hand, it also increases your maintenance costs. Simple code is much easier to program, test, and modify. When we complicate it, the complexity can grow exponentially, impacting our iteration speed. So I reaffirm myself. Overengineering kills products. Much more than the absence of good engineering practices. It is this way because to benefit from good practices you need to have product-market fit, while overengineering can prevent you from getting it in the first place.

Overengineering examples

The first that comes to mind is microservices-based architectures. They came like a wave a few years ago, and they should have taken down more projects than those they have consolidated. I put them as an example of overengineering because they are not necessary in 99% of cases, especially for a startup that has to find market-fit and will benefit significantly from using a more straightforward architectural pattern like a "majestic" monolith. If you succeed in finding market-fit and it turns out that you end up needing to switch to microservices due to scale problems, oh boy, that's a good problem to have. Premature optimization is often another typical example of overengineering. A common situation is preparing a system to absorb a large amount of traffic with an overly complicated infrastructure setup when you still don't have users. In most cases, having a monolith running on a single server is all you will need to validate your business model. One of the worst examples of premature optimization occurs when we spend a lot of time designing a system trying to prevent repeating ourselves in the future and having to throw away part of the work done. It doesn't matter how perfect your design or implementation is if you never see it working because you go broke before. The worst code on the planet that helps you validate a hypothesis is better than standing still for fear of not repeating yourself. Related to the above, software rewrites are an obvious example of overengineering. Usually, engineers do not like to work on legacy codebases. Their natural tendency is to do everything from scratch. But as Joel Spolski wrote more than 20 years ago in Things you should never do, rewrites rarely serve their purpose and can even take your business away. It's obvious to say it, but your client doesn't care how elegant your system is on the inside. Your client cares that you help him solve his problem. Every minute invested in not giving them value is a wasted minute.

How to prevent overengineering?

The best way to prevent over-engineering from my point of view is to turn your engineers into true product engineers. We can achieve it by involving them in the day-to-day business, explaining the why after each initiative, and linking it with the metrics that matter for the organization and its vision.

Watch this MTP panel to find out more about defining metrics that matter.

We need to bring them closer to our users, inviting them to interviews and discovery sessions with them. You want your team to empathize intimately with your user's problems, so they can quickly discard any engineering effort that won't solve them as efficiently as possible. Don't expect your engineering team to be motivated to avoid complexity if you treat them as mere production chain resources whose sole task is to implement user stories from a backlog. They need to understand the why behind each decision. Related to the above, define the problem well to reduce ambiguity. Your engineers need to know the why, but they also need to know what to expect. The more you can narrow down the problem, the less reason they will have to protect themselves from overengineering a solution. A good way to define the expectations of a system is by using service objectives using SLIs and SLOs. Engineers are one of the most creative profiles in any company. If your team trusts your criteria, day-to-day ideas or initiatives will likely arise on their part that may show that they are thinking about a future "what if" scenario. When you have the intuition this could be the case, ask: How does this help solve our current user's problems? What happens if we don't do it now? The answers will help you to discriminate if it is a possible case of overengineering or not. Lastly, more oriented towards founders, prioritize hiring senior engineers who have already had a few experiences in product companies. Look for war scars during interviews. Ask about their most painful experiences and how they dealt with them. Stick with those who put the user and simplicity ahead of simply technology solutions.

Mental models to prevent overengineering

YAGNI

The problem of overengineering is so prevalent in the industry that the engineers themselves have a term to refer to the situation of adding code "just in case": YAGNI, an acronym for "You are not going to need it". YAGNI tries to prevent you from adding anything that is not strictly necessary to solve the problem you have in front of you because the reality is that most likely, "you won't need it".

KISS

The term KISS, "Keep it simple stupid", refers to the fact that simple systems are easier to repair, evolve and maintain. Therefore, simplicity should be one of the goals of any design, avoiding any unnecessary complexity. Worse is better With Worse is better, we emphasize that there comes the point when having fewer options is preferable to having more. It's this way because it can simplify the use of our product, making it attractive to a broader segment of the market. In other words, it encourages us to maintain the minimally essential functionalities of a product, avoiding adding fat that can add complexity to it.

Conclusion

To wrap up, overengineering has the potential to destroy your startup, it can:
  • Add unnecessary complexity.
  • Increase development and maintenance costs.
  • Reduce your iteration speed.
  • Avoid you from getting market-fit.
Unfortunately, overengineering is no exception; it is the norm. For this reason, it is vital to know what it consists of and try to prevent it mainly by involving and bringing your engineers closer to your customers' real problems. Every minute we invest in development that doesn't solve an actual customer problem is a minute wasted. Avoid falling into the "just in case" trap. The graveyard is filled with exquisitely designed startups and products to scale to millions of users who never got the slightest bit of traction. Don't become one of them. Want to hear more from Simón? Have a read of his recent articles:

5 thoughts on “Overengineering can kill your product

  1. Excellent article. I’ve seen overengineering many, many times during my 25-year career. I’ve been hired on several occasions to fix/refactor/simplify software that had spiraled out of control due to developers who were too “clever” for their own good. Usually, the culprits are too much OO, inheritance, DI, design patterns, etc. ORM is also a significant problem in many cases. The only real solution to this problem is hiring the right people. This is easier said than done, but when I interview people I try to tease out any proclivities they may have towards overengineering, such as a person who likes to talk about design patterns, or who takes pride in being hyper-technical, etc.

  2. “The more you overtake the plumbing, the easier it is to stop up the drain.” (Quote from an old movie).

    That old saying applies to software engineering. However, under-engineering is also a minefield. “YAGNI” (You Ain’t Gonna Need It) makes sense when talking about writing future functionality today in a production app. It is foolish when talking about building in generic usage and extensibility for the current functionality.

    That illustrates the difference between professional software engineers and programmers – looking ahead to the next 1 to 3 years of what you have written. Is it performant? Is it extensible? Is it sufficiently genericized where it can be used for other vertical concepts with the same general purpose? When you start finding code blocks (objects, modules, etc.) that do essentially the same thing, but specifically tailored for different vertical uses, you have what amounts to duplicate code that could have been genericized had the software engineer or developer put some thought into it.

    There is a huge difference between “Keep It Simple, Stupid” and “Keep It Simply Stupid”. Aim for the former to find the sweet spot between bloated, hard to maintain code and overcomplicated code that “stops up the drain”.

  3. An unqualified KISS is probably the most useless principle I’ve ever heard of . In my experience, people who take it to their heart end up usually following the KIS (Keep It Stupid) principle.

    The KISS implies a simple solution can be easily found, provided one just quits making things complicated by thinking too much. Sure, we can do that. But in that case it is the *process* by which we come up with the solution that is simple. Not the solution itself.

    Simple solutions, on the other hand, are very, very hard to come by. Unless, of course, the domain is inherently simple, in which case you probably don’t need to worry about any of this in the first place. Finding a clean, elegant, simple solution that aligns well with the requirements takes a lot of time, sweat, tears, and many, many iterations. It won’t just happen magically as a result of trying to “keep it simple”.

    In the context of new products, a simple process is probably quite a bit more important than an internally clean, elegant, simple product. In other words, design simplicity is most likely a premature optimization if the product specs are still fluid and it is uncertain what the actual requirements are or will be.

    To sum up, simple process often results a messy and complicated solution but is relatively easy to come by with. Simple solutions, on the other hand tend to require lots of hard work, trial and error, and time.

    Before applying KISS, one should be, IMO, very explicit about what the ‘IT’ in the KISS principle means in the situation at hand. Is it the process or a solution that needs to be simple? You just cannot have it both ways.

  4. should “better than standing still for fear of not repeating yourself” be “better than standing still for fear of repeating yourself” ?

  5. Probably missing few core points:
    1. Without good development practices due to lack of Experience or plain old Rational Thought the only possible outcome is Operational Deficiency.

    Every Best Practice is context-dependent, thus having Deficient Resources makes them inapplicable in certain cases. Some teams can really suck with the same tech stack when others flourishing using it.

    2. Basic organizational anti-patterns, like Mushroom Management, and broken retrospective lead to rediculous outcomes.

    Even plain old Microservices and Micro-frontends can be a basis of Stovepiping and applying Mushroom Management. Usually, again, due to Lack of Competence and Sheer Hubris.

    3. “Premature Optimization” only used in context of over-engineering by those who didn’t read the book, but use Halo-effect cognitive bias to project compensated qualities onto the term itself. There are a lot of Psychological Compensational Needs under the hood.

    It’s like “Why Agile has nothing to do with Discipline ?” or “Why senior developers turning the project into a sandbox due to the lack of self-fulfillment ?” or “Why most of the MVP’s lack Concise and Validated Definition of Viability ?”

    Complex doesn’t mean Hard or Expensive. Simple doesn’t mean Easy or Cheap.

    Too often “over-engineering” is just an organizational and psychological issue and not an Engineering one.

    Stop operating on Feelings.
    Six Sense of the Fifth Body Anchor Point is not a reliable Key Performance Indicator.

Leave a Reply

Your email address will not be published. Required fields are marked *