For eight years I worked at a successful startup called Base CRM (now Zendesk). During this time I saw the company take different approaches to many problems startups have, but one remained unchanged: an unparalleled, obsessive focus on shipping new functionality. Having co-started a gig called Probe I’ve been thinking about a company as a whole and how things should have been different at Base. Here I’m going to share the lessons I learned about the hidden costs of constantly shipping features.
Disclaimer: although the things I write here may sound negative it wasn’t all that bad. We still were a really successful company, did many things right, we had a great team, a product used by thousands of salespeople. I’m just focusing on where we could’ve done better (myself included).
Beginnings at Base
The mentality at Base was always oriented towards product and shipping features. Even our motto said it: “It’s null till you ship it” – what better way to say that all your Research and Development (R&D) efforts should go to releasing new stuff to production? This mentality was brought by our CEO, the person behind the idea of building a new CRM – Uzi Shmilovici. He was the most experienced person in our very young team (I joined when I was 24). His experience was definitely something that helped to instil the value of shipping new product features in our minds.
Quickly the whole organization adopted and started living by this motto. To be perfectly fair: I think there was a lot of merit in it. Being a hardcore developer at that time I still believed that what we sell is the shipped product, not the technical solutions behind it, nor the automated tests, code coverage, etc. At least this is how I projected what this motto meant, and I’m quite convinced that others looked at it this way as well. I wasn’t 100% right.
Maturity and growth at Base
Over time the company grew. More customers, bigger funding rounds, higher ambitions. Our R&D suddenly became 40, then 100 people. All of them with the “it’s null till you ship it” mindset.
Things got more and more complicated as well. CRM is a broad topic and can mean many things. It tries to embrace the basics of many different sales processes that companies have, together with a broad set of other features like email or calendar integration. It took an enormous amount of work to ship new features that fit nicely into the existing ecosystem. Suddenly we had one-year projects on our roadmap (nicely hidden under three-month projects which didn’t end on time).
Additionally we landed a few huge deals, the famous customers you can’t afford to lose. It meant custom work embedded into already complicated functionality, and complexity grew fast. Big customers certainly sounded good on paper, but they came with a cost (complexity and a very hands-on relationship) that we weren’t really prepared for and that we didn’t manage to put boundaries on fast enough.
The difficulties started piling up fast, and before we knew it we had to deal with a lot.
When product problems pile up
We also added a lot of complications on the backend of the product. Moving to microservices meant getting rid of some problems of our legacy architecture, but it introduced us to the world of (arguably) harder ones, like keeping data consistent across services. Some we managed to cover up with money by just buying more processing power. Others we desperately needed to fix, especially those basic building blocks of our infrastructure which we knew didn’t work. One example would be a communication channel between our microservices without the at-least-once guarantee. Some messages would never reach the destination endpoints, which caused problems with data-consistency.
Of course, all of this was the effect of our success. We had new customers coming in, the metrics looked good, and we quickly moved up in the market. It was all because we were solving real user problems.
But the difficulties started piling up fast, and before we knew it we had to deal with a lot.
Watch this #mtpcon Digital 2020 keynote, Christian Idiodi: Every product problem is a people problem
The difficulties
On the product side, the complexity was quite overwhelming. We had a huge system with many moving parts, all of which interacted with each other. These interactions were also sometimes hard to understand. With the added factor of custom solutions for Big Customers™, more and more flaws in the interactions started showing up.
No decision paper-trail
There was little documentation about product decisions. If you’d been around for a long time (like myself), you had a fairly good understanding of how the product worked. Anyone else had to ask around. Even then, the original authors of ideas didn’t have all the answers, because the situation was often complicated and involved multiple people on the team.
Insufficient iteration
We didn’t revisit features often, either. The product specs were well thought through, sometimes crafted for months, but the truth is that almost nothing you build will be perfect the first time (and if it is, there’s almost certainly a generous amount of luck involved). It’s normal, and there’s nothing to be ashamed of. No matter how hard we plan and research, we build with imperfect information, and we do the best we can. Sadly, we had a lot more confidence than was wise in how well our features were initially designed (some might say hubris) and we didn’t revisit or iterate on existing features.
Technology built with more enthusiasm than experience
The backend suffered as well. We moved into a world of microservices very early on, and that’s not to say that microservices aren’t a power tool (I’m still a fan of them). However, you have to have expertise on how to do it well. Back then, we were inexperienced and just thinking about the outcomes and upsides, so we still pushed very hard to separate our architecture into microservices. In many cases we did it badly. We pushed it too fast, too early, and without really learning along the way. In effect, this meant we just repeated the same mistakes multiple times, rather than iteratively improving our architecture and processes as we moved forward.
We moved too slow with technical documentation as well. With a system as complex as this, it’s crucial to document your tech stack, code, and APIs. Yet with the pace of shipping new things, and a pressure to deliver working solutions, we neglected to document how things work and focused on just getting things out the door. In hindsight, we should have standardized many more things as we moved forwards. Without coding or operating standards, the cost to introduce new solutions grew.
Lessons learned
So far, I’ve just talked about the challenges we faced, and the pain we felt. There’s plenty you can take away just from that, but I’d like to make sure I’m also sharing some of the actual lessons I learned in the process, so that you don’t make the same mistakes.
Culture shapes everything
One thing it’s worth noting as a background to all of these lessons is that it’s important to recognise the huge and subtle impact of organisational culture. While some of our challenges were a result of us being a young, enthusiastic startup team, many could be traced back to the culture of “it’s null till you ship it”. That wasn’t just a saying, it was an idea that became embedded in how we thought about bugs, customers, growth, and technology.
I’m certainly not saying it was a bad saying or a toxic culture – it was instrumental in our growth and success! But I am saying that we didn’t pay enough attention to how it was shaping our organisation and our product practices. Be ready to adapt your organisation when your situation changes.
Iterate more
First and foremost: no matter how much time you put into your product design, iterate again. Your best, educated guess is not going to be enough, even if you spend six months perfecting it. The urge to have new functionality is huge – believe me, I know! – but make sure you spend time taking care of existing solutions. There are many improvements waiting. Look at your existing customers and what they tell you (via your support system or sales team), trust your team, and give them time to improve on solutions they previously crafted which could be delivering more value (or less friction).
Read how product leaders can make better decisions with iterative analytics
Recognise (and control) your technical debt
While you’re at it, gather quantitative data from your backend about errors and bugs. The complexity and friction you ignore in your software will inevitably haunt you when you want to try and build something new, so pay attention to your technical debt and take steps to control it. Bear in mind that some of your technical debt will be in the code, and some of it will be in the documentation or processes (or lack thereof). You need to address both, one way or another.
Beware of money with strings attached
Hopefully, you’ll be successful and will win big customers. However, beware of huge deals, because although the money is great, they often come with strings attached. Most likely you’re going to “pollute” your product with custom solutions that don’t really fit in, and which possibly shouldn’t even be a part of the product at all. It’s also going to force you to redesign your team structure somehow – perhaps create a Professional Services team – or else your roadmap is going to be a constant battlefield between what you want to ship and what your Big Customer™ wants. We thought these kinds of big customers would be a segway into a world of Enterprise sales, but it turned out they weren’t a guaranteed path. In fact, we lost our very first one without a clear benefit – no product improvements, and no references or stepping-stones for future Enterprise customers.
Speak up sooner
If you are an individual contributor or a team leader of a small group of people, make sure you voice your concerns clearly. Founders are (hopefully) focused on the most important metrics and setting up strategy (and for good reasons!), but it is your responsibility to make sure they know about the things you’re responsible for.
Product experience getting worse? More and more errors piling up on the backend side? Be open about it and ask for time to fix things, otherwise it’s you who’s going to have to deal with these issues all the time.
Discover more product learnings
If you liked this post from Michal, you might also enjoy some of our many case studies which feature real-world learnings from other product professionals. Explore our case study library today.
Comments 0