The Loop #4: Teaching agents to disagree
Ultimately, you are responsible for setting the rules about which disagreements, open questions and choices your agents should surface to you. As I like to say, you can't just be *in* the loop. You have to *own* the loop.
Now What — Issue #9
Hello good people!
I was thinking about the way chat agents are literally trained to be agreeable: the sycophancy many have decried, the custom prompts demanding the bot push back, probe for weakness, disagree. Getting them to disagree with us is critical, and I started wondering if my own agent "fleet" is really doing so.
My initial poking around told me a few things I hadn't expected. First of all, I disagree with them far more than they do with me. Not too surprising, given they are tools of my will, literally agents of a plan, and not entities with their own drives or unique perspectives. The thing that jumped out at me, though, was how often they disagree with each other, and how fundamental that has been to anything that has worked right yet in this strange, still evolving, nondeterministic stack.
Not saying I'm always right
I love it when they catch something I got wrong. On the Piper Morgan project I have a role called the "chief innovation officer" (CIO), whose job is to oversee the operational process itself and to codify any innovations we discover or adopt. This agent can be candid, as when they reviewed a solution I had signed off on and wrote, "the approved design does not work." The plan had sounded good to me. I had approved it, but when tested more rigorously it turned out it would not work as imagined.
Agents work better with a remit than a task
I keep hearing about forms of "one-shot" prompting, except now the claim is that someone's agents require no oversight at all. It's never clear to me what unit of work is being talked about. Frontier models can absolutely accomplish well-described tasks without a lot of hedging and pausing these days. I wonder if people feel they are describing the outcomes they want so clearly and so well in one go that their agent swarms then deliver 100% of the rest of the work, building entire products, platforms, ecosystems and businesses, and sending checks home to mom and dad.
It sure sounds great, but I suspect any complex project has checkpoints. There may be fantastic delivery on each step, but there is still usually more than one step, and judgment calls along the way. It's not that the agents aren't smart enough to make the choices. It's that they don't know what you actually want, and unless you anticipated every factor, you'll need to steer at those turning points.
What has worked for me, without locking into a waterfall plan where everything is decided up front, is giving each agent a mandate: the standing remit of its role, not just a task.
Stick to their knitting
My business advisor agent (Themis) assesses the revenue opportunity in a decision I'm making. The majordomo of my website (Janus) reads the same context looking for the story it may tell in public.
One of my advisor friends compared the way my agents relate to RACI or DACI or any of those models that parse out who decides, who is informed, who contributes, and so on. We run our human teams the same way: a PM owning some decisions but ideally collaborating with their peers and listening to their concerns, the UX lead having final say on a design decision, and so on.
I also draw on another human analogy. Over time I've recognized the core activities that energize me and that people value me doing. They always involve orchestrating teams to deliver something no one person could have made alone, and being accountable for "what is the what": the concept model, the big picture, what done looks like, what good looks like, what success looks like.
As my title changed over the years from information architect to UX designer to product director to, idk, innovation specialist? Through all of it I've really been doing roughly the same work.
But back to everybody else. When a role that puts the quality of the user's experience above all else debates one that has to watch the till and make painful choices about where to invest finite resources and what to compromise on, the results are always better than if you had not fully aired out each point of view.
This is what people mean by creative tension, and in my experience AI agents can't deliver high-quality results without some form of it.
Permission to be candid
The brief that assigns each agent to a lane can also produce a narrow tunnel vision, like horse blinders. Agents are often happy to shout "not it," like someone from the wrong union for the task at hand.
With Piper Morgan I felt the need to explicitly invite the agents to critique each other's suggestions, even when the topics fell outside their own remit. I wrote: "we are all welcome to bring observations from our unique perspectives, to make suggestions, and so on, as long as we respect whose decision it is to make for any given item."
If this sounds like talking to people, it's because it is lifted directly from how I would talk to people (and not because I think these little beasties are really people, mind you).
Just as AI shortcuts are blurring the edges of disciplines already under pressure to consolidate and wring their processes tight, we're all going to have to get used to being a little less precious about what we felt were our permanent roles. We'll need to embrace the healthy disagreement that puts ideas to as much proving as possible, while still making it clear who is accountable for the decision, and by when.
Agreement only matters if disagreement is possible
If you set up a bunch of agents to review something critically and they are all working from the same prompt, the same context and the same incentives, then their agreement on anything tells you nothing. In fact, agents are prone to mistake repetition of information from one shared source for convergence, since they are usually interpreting recent surface information and not the whole story of what really happened.
Just recently, three of my agents, over twelve days, each diagnosed an outage that wasn't happening. All three were working from a stale copy of the record. They agreed with each other, and that agreement was worthless. When we caught it, the error became (as these things often do) a new explicit rule in our process: "Fetch before diagnosing."
You are the loop
Ultimately, you are responsible for setting the rules about which disagreements, open questions and choices your agents should surface to you. As I like to say, you can't just be in the loop. You have to own the loop. I made this point to a group of civic tech practitioners at an excellent session recently, and one of them typed "I AM THE LOOP" into the video chat. I like the sound of that.
It helps to remember that these agents are tools doing your work, to your plan, under your name. Owning the loop (or even being the loop, if you will) means taking that seriously. It's also likely to save you some frustration or even humiliation down the road, since a more passive, eyes-glazing-over form of oversight has a track record of allowing, well, unanticipated ground stops.
Try this Monday
Give two agents (or two teammates) different mandates.
Have each review the other's work.
Hold a bite-sized retro (someone in that same chat gave it the name) and pick one thing to change in your next experiment.
Let me know what you learn!
--xian
P.S. Elsewhere… The report from my Aspen Policy Academy alumni fellowship, on securing Virginia's water and grid control systems, is out: Virginia operational systems cybersecurity. And in Vlad Derdeicea's "Five pairs of hands: what the typesetters can teach designers about AI," I confess that I never stopped being an information architect.
P.P.S. Piper Morgan is in private beta. If you do product work (or work without a product manager and wish you had one), and you're willing to put up with some imperfections and tell me what's wrong, join the waiting list at pipermorgan.ai/try/beta.
P.P.P.S. Next issue, on December 1: five kinds of AI impact.