Learning from my own mistakes – how my product failed despite positive feedback from clients

The Beginning of BoostDev.ai

Exploring my experience as a founder of BoostDev.ai – a product that intended to help IT leaders better understand their teams by using data analysis of teamwork patterns. Why did it fail and what could have been done differently?

In today's IT world, many people want to create their own product

I was one of them – a successful project manager working for an American healthcare company. One evening, whilst strolling along the Mediterranean coast with a friend (an experienced CTO for a one European fintech company), I told him about our experience implementing metrics in development teams. We had begun measuring how much time teams spend on development, reviews, testing, and waiting for other departments – comparing processes across different teams, trying to understand what distinguishes highly effective teams (those delivering results faster with fewer defects) from others.

The Product Hypothesis

My friend exclaimed: "That's really interesting! Just think how much more you could analyse if you gathered data from other tools – messengers, calendars? Management could get far more insight into the black box of development!"

As a result, we came up with a product hypothesis that BoostDev (as our project came to be called) would be a product helping IT companies improve transparency and efficiency in development through analysing data collected from tools that teams use. Of course, such a definition contained many unknowns – which IT companies exactly, what problems were they facing? What data would the system analyse and from which tools?

After researching the market for similar products, I quickly concluded that the market was relatively fresh – the first similar product was launched in 2017, there were 3 major players in the market, and a few more smaller ones. However, the functionality and how these products were presented, alongside market assessment (the only public market research I was able to find) and growth rates of 7% per year, filled me with optimism – we were in the right place and we would definitely create a great product and be able to make money.

Market Research and Early Optimism

What also inspired our confidence was that competitors were receiving investments – investors believed in such products. The existence of other products helped me form my own vision of the product – in my head, I didn't always understand what exactly we would do and how, but the question of how we would differentiate ourselves also wouldn't leave my mind. "We'll be cheaper," I told myself, not finding any obvious unique market proposition.

I compiled a table of all competitors, listing all the functionality they offered to end users. The results disappointed me – I couldn't see an opportunity for BoostDev to take its own niche. At least, I couldn't see what could be done to make BoostDev look more attractive than its competitors. So I began analysing reviews of competitors, hoping to find something that would give me an idea.

A couple of reviews from companies that had tried trials offered by "competitors" but couldn't afford to pay for licenses because prices were too high for companies in markets outside the US proved promising.

During one of our brainstorming sessions, we also managed to come up with a couple of interesting ideas that later proved to be genuinely unique – as time showed, competitors were only beginning to move in this direction.

Nevertheless, at that point we still hadn't begun active development. The question of resources and team arose. The answer seemed pretty obvious – we needed investments to start work.

What I Would Do Differently Now – Before Investment

1. I would have spoken to those who use existing products – how do they use them, how do the solutions help them? Gather a profile of potential clients – who are they? What problems do they experience? What do they lack in existing solutions? How much do they pay?
2. I would have spoken to those who don't use such products, validated problems from the first group – the task would have been to understand the arguments, how difficult they are to overcome, understand the profile of "definitely no", "maybe", "yes" to better understand whom to target and assess the available SAM and SOM markets.
3. Sometimes you could speak with current or former employees of competitor companies – understand how they saw the product's strength, what was most important for the clients, what was less important, what difficulties they encountered? However, despite invaluable insights, one needs to be careful – ensure that such a person isn't violating NDA terms when speaking with you, or consult with a lawyer beforehand to avoid breaching confidentiality policies.

On a call with a manager from one of the companies offering help with finding investment for startups, we were offered to prepare an investment pitch deck and financial model. Overall, we spent about a month on a reliable PnL model, deck design to be looking solid, and gathering necessary insights for potential investors.

We spent over 2 more months on meetings with potential investors, but the absence of an MVP and any properly validated demand led to the standard response: "Let us know when you've made an MVP and can show that the model works."

Despite the valuable experience of meeting with investors, I understood that we needed to move forward with development even with investment limited to our personal funds.

In the meantime, we were communicating with "target" clients and, as it seemed to us, we were looking in the right direction – large IT companies were interested in studying IT department productivity and, not finding sufficiently flexible solutions on the market, were beginning to develop solutions internally. Also, we saw many companies that simply hadn't heard of such products but clearly showed interest. At the same time, we couldn't come up with our proposition clearly enough.

What I would do differently now:

1. I wouldn't waste time attracting investment until I definitely understand the business plan and product market, and hadn't compiled an accurate estimate of necessary expense needed
2. I would look for ways to validate hypotheses with the clients we were already communicating with – I would have built an MVP in the format of an Excel report with target metrics I see the most valuable
3. I would have analysed demand across different client groups with different propositions (Landing Page Test).
4. I wouldn't have been so optimistic that interest would convert into potential profit. I would have focused more on identifying current problems and how the product solves them.

Building the MVP

Next, I began hiring a team. We required at least 1 frontend and 1 backend developer. At that point, this didn't guarantee rapid MVP creation timelines, but it was better than simply seeking investment "for an idea", especially since it seemed to me that with a working MVP we would find it much easier to get market feedback.

In parallel, I began creating UX mockups. For the first functions in the MVP, I chose the most basic development metrics that major players offered.

Development revealed more and more cases that we hadn't considered in the original plan – the gap in development processes the different companies have was too large – it was difficult to create a "common" solution that would allow us to deliver the solution to a large number of clients.

Eventually, we began attracting early adopters and the system's first users. At that time, we chose clients from a list of industries they worked in, found decision-makers – CTOs, Heads of PMO, Chief Product Officers – and offered to tell them about a product that would help accelerate delivery by 30 percent through implementing 4 simple metrics. The conversion rate from messages was approximately 6 percent; nevertheless, we spoke with one new potential client every day. We felt we were becoming more confident in pitching skills, understanding of the problem, and client profile.

The problem was that during demos we were showing a system that wasn't yet fully ready. Despite positive feedback and client curiosity in trying BoostDev, we couldn't end calls by giving the client a login/password for the system.

We realised that at this stage we could collect and process data, but the system interface was insufficiently flexible and raw. We began sending users reports with metrics and our recommendations in PDF format, without an interactive interface. Clients were only required to provide us with read-only tokens from the necessary repositories to be analyzed. 

This allowed us to test the hypothesis quite quickly on 30+ clients who agreed to be early adopters. However, at the same time, this helped us understand that all those pain points and edge cases that we decided not to include in the MVP actually prevented clients from seeing the benefits of using the solution. At the same time, we understood that despite seeing demand (CTOs want to know more about teamwork and quickly audit the development process), we didn't understand how to offer them a tool for working with this data – for a separate another paid system, we clearly lacked functionality.

BoostDev required significant improvements both from a data analysis perspective and from a user scenarios perspective. It wasn't enough for clients to see metric charts in a separate window; they wanted to manage the process. They wanted us to be able to tell them about "today's" problems with one mouse click, what to pay attention to; they wanted a smarter commit management system with advanced analytics and broad functionality that could replace their current tools.

Team Burnout and Pause

At that point, about 1 year had been spent on BoostDev development from idea to realising the need for significant project restructuring after the MVP.

The team felt demotivated and tired. As a leader, I also understood that I lacked the expertise to help clients formulate those use cases that would help them solve the tasks they faced. In the end, we decided to pause product development until we had new strength and understanding of how to proceed.

What I would do differently now:

1. Having confirmed demand, I would have begun validating user scenarios – what data does the user want to see and how do they want to use it?
2. For confirmed user scenarios, I would have drawn UX mockups to understand what the client wants to see.
3. Whilst the system was being developed, I would have analysed client data manually
4. I wouldn't have been so confident in myself as someone who could simultaneously create a product, understand technical development specifics, and successfully sell. Saving money on the team didn't allow me to truly see the product's perspectives as a business, leading to insufficient quality and burnout.

Final Reflection

If I had initially made different decisions, this could have shortened the time to 3 months for hypothesis validation and saved efforts and resources for creating a successful product. Nevertheless, for me this was an experience that helped me see how business works, understand what to focus on next time, build an excellent network of new contacts, rid me of many illusions about my own capabilities, showing that team and people are the key to success.

Our main problem was the desire to convince ourselves of our own vision, developing the product in a vacuum and not receiving sufficient user feedback until the moment of MVP creation.