Building the Stacktrace for AI mistakes

🙋 Ways you can help - intros to AI teams 🙋
We’d like to speak with people with the following profile:

1/ Technical leader e.g. CTO, Head of AI/Engineering/Product
2/ They are building an LLM-based product and have users/revenue
3/ They have fewer than 100 employees

Please get in touch if you know anyone who fits the bill!

📉 Progress in January 📈
Our metrics for January were:

Monthly revenue: $300 (unchanged)
Customers: 1 (unchanged)

We failed to meet our target to grow from 1 to 2 customers. We retained our first customer, as they saw the value in even a very early version of our new product. However, we didn’t sign up a new customer. Reflecting on our main hypotheses going into January:

1/ Our “Stacktrace for AI mistakes” makes it faster for AI teams to action human feedback
This was mostly true. The concept was clear and resonated with AI teams as a time-saver. We booked 12 demos across January and customers clearly understood the potential value. Top-of-funnel was also a strong point. We held another ~35 discovery calls and we don’t take this for granted, given that getting customers to talk to you is the hill that many startups die on!

2/ Actioning human feedback on AI output quality is a significant resource commitment.
The evidence on this was mixed. This was likely the main cause of lost deals, so we write in more detail below about how we’re planning to respond.

🔬 Learnings about actioning human feedback on AI output 🔬
We held 12 demos in January but didn’t convert a new customer. The main objections were:

1/ Artanis takes too much engineering effort to integrate.
Human feedback about AI comes at irregular “bursty” intervals. Engineers may spend several days in a row looking at traces, followed by weeks of inactivity. However, the first version of our product wasn’t self-serve; we needed 2 weeks to onboard new customers. This meant there was a mismatch between the “bursty” nature of the problem and how quickly we could solve it. We’re now addressing this by making our product self-serve, so they can get up and running in minutes rather than days.

2/ We’re not collecting enough human feedback on AI output.
This was usually due to their product being pre-launch or not having enough usage. In these cases, our product doesn’t save much engineering time. This isn’t an objection we can address; it’s a qualification criterion that may impact our commercial viability. We’re taking a bet that this is a “small-but-growing” problem: AI products will pick up more usage over time, and that companies will collect more human feedback to improve their AI.

3/ We don’t want to pay upfront, can we do a free trial instead?
We lost deals because we require 1 month payment upfront and offer a refund if they’re unhappy, instead of offering a free trial like most software products. We don’t want to address this just yet, as it’s an intentional decision to select early adopters who most strongly need our product. We’d rather have 2-3 highly engaged customers than a larger number who’re less involved.

🏹 Goal for February - grow from 1 to 3 customers 🏹
We’re aiming to grow from 1 to 3 customers, with the following product changes:

1/ Let customers use Artanis on top of their existing tracing, rather than replacing it.
2/ Enable self-serve installation within an hour, rather than requiring a 2-week onboarding.
3/ Allow manual annotation of human feedback on traces, rather than requiring a fully automated connection.

These should shorten time-to-value, so we can deliver during the short “bursty” periods where they’re looking at traces. It should also reduce the engineering effort required from their team to integrate Artanis. 

🙏 Shout-outs 🙏
Special thanks for January go to:

Paul F - for being January’s intro MVP!
Zoe M - for the intro to Clement
Conrad - for continued support
Sogo - for the several-degrees-removed effort
Matt R - for the CS idea and DX suggestions
Marissa R - for thinking of Paraglide
JJ, Dominic, Sergey & Yiming - for very clear feedback after a demo
Csongor - for several good intros
Sivesh - for being a Value-Add-Investor
Hanna L - for proactive help with other startups
Ismail S - for the intro to Sanj
Bailey & Rod - for inviting us to speak at a great AI eng. event
Henry I - for connecting us to your CS team
Gunisha & Archana - for architectural advice

En Taro Tassadar,
Artanis Team