What Vistaly's AI Rebuild Teaches About Repair Loops in Discovery

Teresa Torres shows what it actually takes to make AI-generated opportunity solution trees trustworthy (evals and repair loops, not just a slicker prompt), LangChain publishes a real cost number for model routing in production agents (64% cost cut, no quality drop), and Notion keeps shipping the infrastructure layer for agents (skills libraries, shared memory) that most teams are still trying to bolt on themselves. ### Teresa Torres / Vistaly - Generating Opportunity Solution Trees With AI: Interview Synthesis, Evals, and Repair Loops **Source:** https://www.producttalk.org/generating-opportunity-solution-trees-with-ai-how-vistaly-rebuilt-its-product-around-interview-synthesis-evals-and-repair-loops/ **Credibility:** High (direct case study from Teresa Torres, the originator of the opportunity solution tree framework, documenting a real product rebuild) **What happened:** Vistaly rebuilt its product around AI-generated opportunity solution trees, using interview synthesis to automatically populate trees from raw customer conversations. Torres walks through how the team had to build evals and "repair loops" (mechanisms to catch and correct bad AI output) to make the auto-generated trees trustworthy enough for PMs to actually use in discovery work, rather than something that looks plausible but quietly misleads. **Key patterns:** - Interview synthesis: raw interview transcripts get processed into structured opportunities and needs that populate the tree automatically, instead of a PM manually tagging and categorizing - Evals built specifically for discovery artifacts, not generic LLM output quality, meaning the test suite checks whether the extracted opportunities actually reflect what the customer said - Repair loops: a feedback mechanism where mis-generated branches of the tree get flagged and corrected, which then presumably feeds back into improving future synthesis - The rebuild wasn't a feature bolt-on. Torres describes it as rebuilding the product "around" this capability, meaning the core workflow shifted to assume AI-assisted synthesis as the default path **Why it matters for PMs:** This is the most concrete example I've seen of what it actually takes to ship "AI does your discovery synthesis for you" without it becoming a trust liability. Everyone wants to automate interview synthesis. Almost nobody talks about the eval infrastructure required to make the output safe to act on. If you're building anything that summarizes or structures qualitative user research with AI, this is the blueprint: you need a way to systematically check if the AI got it right, and a loop to fix it when it didn't. **Critical questions:** - How much manual eval-writing and repair-loop-tuning did this take before the team trusted it enough to rebuild around it? That's the real cost of this approach, and it's not in the headline. - Does the repair loop improve the model's future output, or is it just a one-time fix for a single tree? If it's the latter, this doesn't scale past a handful of customers. - What's the failure mode when the eval misses something? Research synthesis errors are quiet (a wrong opportunity looks like a reasonable one), which makes them more dangerous than code bugs. **Action you could take today:** If your team is using AI to synthesize user interviews or research notes, pull up your last three AI-generated summaries and manually check them against the raw transcripts. Count how many material errors you find. That number is your actual trust gap.