Green tests aren't permission to merge
Every Dependabot PR sitting in your queue is waiting on a person to go verify it. Software Factory Ep. 2 moves that investigation into a sandbox. Plus why software factories aren't cracked yet, and Greg Wilson's Grafana-routed triage agent for incidents.
This week's video
Open a repo your team hasn't touched in a month and count the Dependabot PRs. Every one of them is waiting on a person. CI already ran and told you pass or fail, which isn't the same as safe to merge, because a green suite is a claim about the code your tests cover, not about the code the upgrade actually changed. Someone still has to go look, and that someone has forty other things in front of them. So the queue grows.
Episode 1's triage agent could only read the changelog and reason about that gap. This week it gets hands: a Mastra Workspace backed by an ephemeral Railway sandbox where it clones the repo, checks out the PR, runs the full suite, and then deliberately breaks the thing the changelog says changed, to check whether the suite actually notices. That turns a queue of unknowns into a queue of decisions, backed by test output nobody had to sit down and generate by hand.
Resources mentioned:
- Companion repo, stand up Station 2 yourself
- Mastra Workspaces and sandboxes
- Railway Sandboxes
- Announcing Mastra Factory
- Episode 1, the read-only triage agent
Green tests are not permission to merge
If you own a team's backlog, the interesting number in this episode isn't a token count or a build time. It's that a major-version bump had been sitting open for weeks before the sandbox cleared it. That's not a technology gap, it's a review bottleneck, and it's the same one showing up everywhere agents write more code than humans can verify: generation got cheap, verification didn't.
The honest twist is a 2x2, not a green checkmark. Tests green, changelog clean: merge, now proven instead of assumed. Tests green with a changelog that shows a real behavior change: still a hold, because a passing suite only tells you the suite doesn't cover what changed. More evidence earns more confidence. It doesn't automatically earn more autonomy, and that distinction is the whole argument for building this station at all, and for how much rope you hand any agent doing verification work on your team.
The isolation boundary is what makes it safe to delegate that investigation at all, which is the part I'd want answered before trusting this on a real codebase. Every PR gets its own ephemeral sandbox, so two audits run in parallel without touching each other's Postgres instance or gem bundle. Read-only token, nothing pushed, destroyed the moment the audit finishes. Scope grew this episode, the agent executes arbitrary commands now instead of just reading text, but blast radius didn't, because the only thing it can hurt is a VM that's gone in minutes.
Geoffrey Huntley posted something last week that I keep coming back to: software factories are real, but the factory part hasn't been cracked yet, and what's actually unsolved is a short list of non-agent problems, sandboxing, monorepo structure, reproducible builds, CI/CD, identity and secret management. Not a tokens-and-prompts problem. A systems engineering problem. This episode is exactly one item off that list, worked all the way through on a real repo instead of gestured at. The agent isn't smarter this week. The environment it's allowed to act in is better designed.
Curated Links & Tools
Routing a Grafana alert to a Mastra triage agent — Greg Wilson again, and it's the same triage-agent shape as this episode from the incident-response side instead of the dependency side. A Grafana alert fires, hits a webhook, and a deterministic seed pulls the failing metric and trace from Prometheus and Tempo before the agent gets four read-only tools to investigate further. A citation gate checks that every claim in the final Slack card actually matches evidence it gathered, so the card can't assert something it didn't verify. Same discipline as the sandbox audit: let the agent investigate freely, keep the write scope at zero, and don't let it report anything it can't source.
What's sitting in your team's queue right now because nobody has the two hours to go verify it? Reply and tell me.
If you're weighing how much verification work to hand an agent on your team, that's exactly the kind of autonomy decision I help Mastra teams make. Here's how I help →.
Damian

Add a comment: