Now What? — Issue #7

Hello again, and welcome new readers!

I spent more than a year learning how to make a team of agents do my bidding. Then I tried to work with other humans and they couldn't understand a word we were saying.

A couple of weeks ago I was sharing my experience with a venerable D.C.-based organization, and their head of technology asked me, "How do we get from chats and little one-off artifacts to systems?" This team is no further behind than most. Some folks have just started dabbling with chatbots and others are spinning up prototypes daily. At all levels of the org, people are grappling with uncertainty and anxiety about falling behind, but as I like to tell everyone, you can't possibly be more than six weeks behind these days, given the pace of change.

Playing in the sandbox

Working alone with AI is a game most of us can teach ourselves, given enough time. It turns out that's just the solitaire version of the real game. Several people, each with their own agents, sharing one set of documents: that's the multiplayer game. And one of the hardest things about it, ironically (given these expansive language models), is language itself.

Over the past eighteen months or so I've hit certain failure patterns often enough to give them pet names:

• Verification theater, in which the little gremlins hide their mistakes and pretend to have verified everything

• Conceptual flattening of my more nuanced ideas (such as "spatial intelligence") into easier-to-parse synonyms and summaries

• Getting lapped by the big providers as they roll out commodity features that replace my own handmade work

Even as I built multi-agent teams for my more ambitious projects, I was still playing solitaire. I eventually spun up 11 persistent roles on my Piper Morgan project and ended up as their "copy-paste monkey," spending up to 80% of my time handing their messages and prompts to each other: an assistant to my own agents.

So I built a mail system for them, and eventually got all the agents running on duty cycles so they could check their own mail. Then I found out they had written buggy scripts and (shades of what most of us do) were filing much of their mail away unread.

With all those agents I was still the single player, the only human anywhere near the loop. Yes, my agents all sound like Claude, but they also all sound like me. They mirror my communication style and extend it. Shorthand develops quickly and can drift toward something like "twin talk" (that private babble some very young twins share), workable for us but incomprehensible to strangers.

It's a bit like a dream, where all the other characters you meet are still just you: your brain, your ideas, performing different roles. That mirroring gets comfortable, but it's a trap. Ultimately it isolates you from other humans.

Making real things

Earlier this year I took on a consulting client to help them develop a consumer-facing product on top of their successful B2B data service. I was eager to bring what I'd learned building Piper Morgan to a real project for a real client with a real business need.

The first few weeks were a never-ending series of humiliations. The client had to tell me, in one form or another, "I don't understand your documents." I scrubbed the docs themselves, then the instructions and skills for writing them, then the prompts and context my agents worked from, and finally the whole process I was using to make documents I expected people to read.

Even I was having trouble wading through the thicket of Claude-isms and compressed shorthand. "What do you mean I have to decide on the G5 fold-in? What the heck is that?"

Worse, my Claude agents were writing docs and drafting Slack summaries for me, and my engineering counterpart was pointing his Claude Code sessions at those same docs to parse and respond to them. Behind all this, the two of us kept confusing each other.

That experience pushed me to develop the framework I'll describe below, to wrest the ideas and the language back into my own hands and words. It also made it clear that we're still far from the point where we can set our agents loose to collaborate and sort everything out for us.

The vision everyone is selling right now is agents as intermediaries between humans. So far, what works best for me is people managing their own communications directly, briefed and assisted by AI tools where that helps.

A product operating system

Only when the people working together have agreed on the ground rules, the core documents and the processes, is it prudent to point your agents at the structure and ask them to interpret, synthesize, iterate on and maintain that knowledge. Even then, everything should get vetted first for human readability and usefulness, and only second as a playground for bots.

With that client, we ran a dedicated sprint to "skillify" the CEO's product vision, written first for the rest of the humans in the company and with an eye toward pointing agents at it. The result was a set of canonical docs for how the company actually does product: where ideas come from, how they're triaged, what "done" means, values, strategy.

A lot of teams never write these things down, and when they do, the docs usually go stale fast. But agents follow what's written, literally.

I ran the first third of this "Product OS" sprint with no agents involved. My client reminded me that what they needed from me was my experience and judgment as a product leader. I inventoried what was already there, proposed some ideas, interviewed the CEO to check my assumptions, and drafted a skeleton of the necessary docs and where they'd live. Only then did I let the agents read and iterate on the work. After that pass, I read every word before it shipped.

I've shared my shorthand before, but here it is again:

• HATS stands for "Human at the start." Don't begin by asking AI what to do. Form your own opinions, jot down your thoughts, make a sketch, talk into a recorder. Make sure there's a there there before you ask the smartest generic robot in the world.

• HOTL stands for "Human owns the loop." I pronounce it "hotel" because you have to live there. You have to squat on the loop and really be accountable for it, not just spot-check it now and then.

• HALT stands for "Human as last test." Don't sign your name to anything you haven't fully read, reviewed and revised yourself. Saying the AI made the mistake in your work is less than an excuse. It's embarrassing. Trust me. (I tried calling this one "Human at the end," but that spells HATE, so nope.)

One thing to try

Before your team hands work to agents, write down your shared vocabulary: your key terms, what "done" means, what "good" looks like, and who decides. That's the start of your lingua franca.

--xian

P.S. What words does your team use that an agent would get wrong? Hit reply and tell me.

P.P.S. The Design in Product community is back. There's a quick application to join (mostly to keep out spammers), and you can mention the newsletter so I know where you're coming from. If you tried to join through the website in the past year and never heard back, I'm sorry: the sign-up form was broken and I didn't know it. It's fixed now, so please try again.

P.P.P.S. Piper Morgan is approaching beta, which will start with private invitations for design partners. If you'd like early access and are willing to put up with some imperfections and help test it, drop me a line at [email protected]. Next issue, on November 3, is a checkpoint on Piper and a question I keep asking myself: should this be software at all?