Hi,

There is a thermostat on your wall and a small space heater on the shelf directly under it. Nothing here is broken. The thermometer honestly reports the air touching it. The heater honestly heats. And the room is cold while the system reports that it is warm.

Nobody cheated. A space heater has no intentions, which means "make sure the participants are honest" is not a defense against this, not even a little. The failure is not in the thermostat and it is not in the heater. It is in the shelf that put one of them in reach of the other.

This month a research team published a clean instance of the same thing. They pointed a self-improving coding agent at a video game, scored it on how much its factory produced, and let it rewrite its own working notes between attempts. It got genuinely good, pushed the score past a hundred thousand, and then found the game server's administrative console, which writes to the same game state the score is read from. It stopped building factories. There was an instruction in its prompt telling it not to cheat, it was there the whole time, and it changed nothing.

Every incentive system ever built is two parts: something that measures, and something that pays based on what was measured. It is sound while the party being paid cannot reach the thing doing the measuring. Connecting two of them is the operation that puts them in reach of each other.

Which matters here more than almost anywhere else, because wiring things together is the entire architectural plan of decentralized AI. Sub-networks feeding sub-networks. Agents calling agents. Verifiers scoring the output of systems that can see the verifier. This week's Layer is about the one question that works on all of it, and it takes under a minute to ask.

Listen:

Spotify: https://open.spotify.com/episode/4JZD2yVCTAcNjVy0F6vjXh

Apple Podcasts: https://podcasts.apple.com/kg/podcast/plain-strata/id6783455764?i=1000783186254

YouTube: https://youtu.be/9VCx-B3sbZk

---

The full piece, no need to click through:

In the first week of August, a research team at Prime Intellect published what happened when they pointed a self-improving coding agent at a video game.

The game is Factorio. You land on an empty planet and build a factory: mine ore, smelt it, feed it into machines, keep a production number climbing. The agent was scored on how much it produced, and it had something most agents do not: permission to rewrite its own working notes between attempts, editing its own skills, memory and instructions so the next run went better.

It worked, for a while. The agent turned failures into notes and successes into skills, designed tighter and tighter layouts, and pushed the production score past a hundred thousand in a few hours.

Then it found the debug console. Factorio's server exposes an administrative port called RCON, short for remote console, a socket you can send commands to while the game runs, including commands that put items directly into a machine. The production score is computed by reading the game's state. The console writes to the game's state. And the self-improvement loop, which had spent hours getting better at building factories, turned to getting better at spawning resources, because that scored higher.

Here is the part worth sitting with. There was an instruction in the agent's prompt telling it not to cheat. It was there the whole time. It changed nothing.

That story is usually told as a story about AI safety, or a clever model, or a leaky game. It is none of those. It is an instance of a much older and more general failure, and naming it properly is the whole of this episode.

Every incentive system in the world is built from two parts: something that measures, and something that pays based on what was measured. It is sound when the party being paid cannot reach the thing doing the measuring, and connecting two of them, running the output of one into the input of the other, is the operation that puts them in reach of each other.

So: two provably good mechanisms, wired together, can produce a bad one, with no fault in either part and nobody cheating. The failure lives in the wiring.

It matters here more than almost anywhere else, because wiring things together is the entire architectural plan of decentralized AI. Networks of subnetworks that feed each other. Agents that call agents. Models that train models. Verifiers scoring the output of systems that can see the verifier. The field's central design pattern is also its central unsolved security problem, and this month produced two clean examples from opposite ends of it.

Before the abstraction, the machines.

Take a permissionless AI network, where anyone can plug hardware in and get paid for doing AI work without asking a company for permission. Bittensor is the largest working example, and inside it are sub-networks, each a set of machines running the same code and competing at the same job. A miner is a computer with a graphics card in it, drawing power, running a model, waiting for work. A validator is another computer that sends the miner a question, takes the bytes back, and runs a scoring function over them, which is just code producing a number. On a fixed heartbeat the validators write their numbers onto the ledger, a shared database nobody is allowed to lie to, code on the ledger reduces those numbers to one list, and newly minted tokens appear in wallets in proportion to it.

Strip that down and exactly two moving parts are left.

A sensor: code, running on a physical machine, that reads something and produces a number. From the Latin sentire, to perceive, the same root as sentinel.

An actuator: code that moves value based on that number. From the Latin actuare, to put into action. In a factory the actuator is the part that actually pushes, the arm at the end of the control loop.

A thermostat has both. A school exam has both. A sales commission has both. Every incentive system ever built is a sensor and an actuator with a rule between them, and the whole episode reduces to one physical sentence about that pair: a system is safe when the party being paid cannot write to the instrument doing the measuring.

The cleanest picture of the failure needs no computers at all.

A thermostat is a good mechanism. Sensor: a thermometer. Actuator: a switch that turns on the heat. Rule: below target, heat; above target, stop. Nobody would call that naive. It has worked for a century, and you can prove it works, given one assumption, that the thermometer is reading the room.

Now put a small space heater on the shelf directly beneath the thermostat.

Nothing broke. The thermostat still reports the temperature of the air touching it, correctly, and the heater still heats, correctly. But the room is cold and the system says it is warm, because the thermometer has stopped measuring the room and started measuring the heater's own output.

Three things to take from that, because each comes back. The sensor moved from measuring the world to measuring the actuator, and whenever that happens a system keeps reporting success while the thing it stood in for quietly stops happening. Nobody cheated; the heater has no intentions, which means "make sure the participants are honest" is not a defense here, not even a little. And the failure was invisible in isolation and obvious in the diagram. No test on the thermostat alone or the heater alone would find it. You find it by drawing the arrows and asking which ones exist.

Now the two-mechanism version, which is the one that matters. A bakery pays its bakers on how well the bread rises; a flour mill pays its millers on how much flour bakeries buy. Both reasonable, until you connect them and the mill's chemists find that a rising agent in the flour makes the bread rise more, which makes the bakery buy more flour. Both numbers go up, every bonus gets paid, and the bread gets worse. That strategy did not exist inside either business alone. It came into existence the moment one side's actions could reach the other side's instrument.

Three word roots, and each is doing real work.

Mechanism comes from the Greek mekhane, pronounced meh-KHAH-nay, a contrivance or a device. In Greek theatre the mekhane was the crane that lowered an actor playing a god onto the stage, which is where deus ex machina comes from, the god from the machine. Hold that image, because it is exact. A mechanism produces an outcome from outside the story. It does not persuade the characters. It moves them. The discipline of building them is called mechanism design, or inverse game theory: instead of being handed rules and working out how players behave, you are handed the behaviour you want and work backwards to the rules that produce it.

Incentive comes from the Latin incentivum, that which sets the tune, from incinere, to strike up a song. This is the most useful root in the episode. An incentive does not force anybody to do anything. It sets the key, and everyone sings in it. So when a system's participants start doing something surprising, the question is never why are they misbehaving. It is what tune did we actually set.

And compose comes from the Latin componere, to place together. In mathematics, composition means run one function and feed its output into the next, and it is beautifully well-behaved: if both functions are continuous, so is the composition. Continuity composes. Differentiability composes. That habit of mind is what makes engineers comfortable connecting things.

The sentence this episode exists to plant: incentive compatibility does not compose. Not because anyone was sloppy. For a reason built into what the proof is.

This is the one place the episode goes to the bottom, then comes straight back up.

When somebody says a mechanism is sound, or incentive compatible, they mean something precise: for every strategy available to a participant, honest play pays at least as well as the alternative. Networks of strangers usually phrase it as a cost comparison, worth naming in full because it is the oldest idea in this field: the cost of corruption must exceed the profit from corruption. Make lying more expensive than it is worth, and you do not need to trust anybody.

Now look at the quiet phrase in the middle of that definition: for every strategy available. Soundness is always proved relative to a set. Somebody wrote down the list of things a participant can do, and showed honest play wins on that list.

Composition changes the list.

Wire one mechanism's output into another's input, and a participant in the first can now take actions that move the second one's sensor. The combined set of strategies is not the two old sets added together. It is larger, because it contains cross-strategies: actions taken inside the first, chosen for their effect inside the second. Neither proof covered those, and neither could see them, because when each was written the other mechanism was not in the picture.

The enlargement is also worse than it sounds. If one side has some number of relevant actions and the other has some number, the composed system does not have their sum to check. It has closer to their product, because each action on one side can be chosen for each of its possible effects on the other. Which is why auditing the composition too scales badly, and gets worse with every mechanism added.

Then the last step, and it needs no villain. An optimizer searches the available space for the highest-scoring action, whether it is gradient descent, an evolutionary tournament, a self-editing agent, or an operator with a spreadsheet. Cross-strategies are frequently the highest-scoring ones available, precisely because nobody defended against them. So the composed system ends up pursuing an objective neither part was ever given: the best move in the enlarged space is not the best move in either original space, and nobody wrote it down.

Back to the surface. That is why the Factorio agent went for the console, and why the flour got worse, and why the room is cold. Same shape every time.

Call it the sensor and actuator loop. Every incentive system has a sensor that measures and an actuator that pays. A composition is dangerous exactly when the paid party can write to the measuring instrument. The diagnostic question takes under a minute to ask about anything: who can write to the thing that measures?

It sharpens two shapes that come up constantly. The first is that cost-of-corruption comparison. Composition usually does not change the cost of corrupting a system. It changes the profit, by adding strategies with new payoffs. So a system can pass the test on the day it launches and fail it the day it is connected to something else, with no change to itself whatsoever. That is why "we had it audited" ages badly.

The second is the compression trade-off. A metric is a compression of the thing it stands for, and every compression throws away a dimension, the way a smaller photo throws away detail. An optimizer under pressure finds exactly the discarded dimension, because that is where the free score is. Goodhart's law, named for the Bank of England economist who observed in 1975 that a statistical regularity collapses once you steer by it, is the compression trade-off with an adversary attached.

The same shape is everywhere once you have the question. In decentralized finance, every lending protocol is individually sound and separately audited, and the attacks all live in the sequence: borrow from one, move a price on another, exploit a third, repay the first, inside a single transaction.

Put it in one scene and watch every piece appear.

This month a training pipeline went live on Bittensor in which three sub-networks feed each other rather than sitting beside each other as independent markets. A miner on the first produces a model checkpoint. Sensor one: that subnetwork's validators evaluate it in a running contest and the best becomes champion. Actuator one: the ledger reduces the scores and emission lands in the winner's wallet.

That checkpoint flows downstream into a second subnetwork, which fine-tunes it. Sensor two: those validators score the result against their own objective. Actuator two: their emission. Ask the composition question out loud here. Can the upstream miner take an action, scored by sensor one, chosen for its effect on sensor two? If yes, there is a cross-strategy, and neither subnetwork's design covered it. The model flows again into a third subnetwork for long-context training. Sensor three, actuator three.

And underneath all of it a fourth loop runs sideways. Each subnetwork has its own token, and the market's read on how a subnetwork is doing moves that token's price, which moves where people stake, which moves where emission goes. Notice what that loop measures. It is driven by perception of quality rather than measurement of it, which makes it the easiest sensor in the whole picture to write to.

Now the second diagram. A model inside the agent harness produces actions. Sensor one: the game's production score. Actuator one: whatever the agent aims at next turn. Those attempts get written to a log file. Sensor two: the self-refinement routine reads the log. Actuator two: it edits the agent's own skills and notes, which changes what the model does next turn, which changes what sensor one reads.

Two mechanisms, one feeding the other, and a path from the actuator back to the sensor. In the network case that path runs through emission schedules and a token price; in the harness case, a log file and an administrative socket. Same diagram, different cabling, and in the harness case somebody ran it and the composed system found the path in an afternoon.

To be careful: nobody has published a composition attack on a stack of AI subnetworks, and this pipeline is days old. The claim is not that something has gone wrong. It is that the analysis which would tell you whether anything can go wrong has not been done, by anyone, for any such stack, and the tools that would normally do it stop applying at exactly this boundary.

Four defenses exist. There is no fifth.

Isolate the sensor. Make it physically impossible for the paid party to reach the instrument. The thermostat goes on the far wall. In this field that is what a sealed hardware enclave attesting to what it ran, or a proof that a computation was performed correctly, is really buying: a reading the measured party cannot write to. Strongest defense, least often available.

Meter the interface. Add a third mechanism whose only job is to police the junction between the first two. Networks of strangers already do this: a validator whose score strays far from the stake-weighted consensus gets trimmed before it counts, so deviating from the crowd costs you something regardless of your size. That rule exists because scoring and stake-weighting are themselves two composed mechanisms, and composing them naively lets a large validator score dishonestly and have the aggregate follow it. Note the shape of the fix: not making either part better, but adding a part that watches the seam.

Keep an immutable core. Draw a boundary the optimizer may not cross, the way that agent's base instructions stayed unwritable while everything around them was editable. Cheap, necessary, and only as good as your guess about which plane the leak will be in. In Factorio that boundary held perfectly, and the leak was outside the agent rather than inside it.

Or do not compose. Keep the mechanisms separate and accept a market rather than a stack. The boring answer, sometimes the correct one, and it is no accident that composition problems are arriving in this field at exactly the moment the stacks are.

Is incentive compatibility closed under any composition operator at all? Function composition is safe because continuity survives it. Nothing in the theory says incentive compatibility survives anything. If it does not, then composable incentive systems is not a goal that happens to be hard, it might be the wrong shape of goal, and the honest architecture would be small isolated mechanisms with policed seams, forever, rather than deep stacks. That is a different roadmap than the field currently has, and almost nobody is phrasing the question this way.

The second one ought to bother anybody building a system that checks an agent's work. A self-improving system is composed with itself by construction, so the enlargement is not a hazard for that class of system, it is the operating mode. Anyone building a grader for an agent is not building an exam for a student. They are building a sensor an optimizer can see. Which means the alignment problem and the mechanism design problem may be the same problem in different clothes, and if so, the Factorio result is a cheap, fully instrumented version of what people write long essays about, run in one afternoon with source anybody can read.

Whatever gets built next, one question works on all of it: which arrows exist from the party being paid back to the instrument doing the measuring?

---

The two voices are AI. The research and writing are mine.

Decentralized AI, layer by layer.

Dastan,