There has been quite a lot of "chatter" lately about the end of humanity at the "hands" of AI. And much of the commentary has been from a curious bunch, the leaders of the major frontier labs. Dario Amodei talks about "pacing" (we'll get into this in much more detail later) and it all feels like "stop me before I sin again." This is a curious essay coming from someone gunning for a huge IPO.

I do not think the end of humanity is nigh upon us. At least not via the devious connivings of generative language models and their agents. There is plenty to worry about, though. Craig Mundie, via Thomas Friedman in the New York Times, claims it is "too late" (to stop AI). This is, in part, not far from the mark. It makes the point that models that are highly capable of inflicting damage are already here. This is certainly true, especially in the "cyber class" models (Mythos et al.). Any model powering agents capable of finding novel zero-day vulnerabilities and turning them into exploits is fearsome indeed. The consequences are significant.

April 2026 will go down as some sort of bellwether combination event. The Trellix Cyberthreat Report April 2026 detailed the extraordinary calendar Q1 data: mean time-to-exploit fell from 23.2 days in 2025 to 1.3 days in Q1 2026. 69.7% were zero-days, exploited on or before the day of disclosure. During the same month, NIST was forced to change their "enrichment" activities for CVE submissions due to the huge increase in volume (one-third higher than the similar quarter in the year prior), following a 263% increase in the previous 5 years.

The other part of the combination was the release of Mythos preview on April 7. During the development phase, thousands of critical- and high-severity vulnerabilities were discovered. The "Glasswing" initiative that followed, a curated release to high-profile commercial and open-source developers, upped that number by an order of magnitude: over 10,000 new vulnerabilities were found. Before Mythos it was clear that the detection and patching regime had come under critical and unsustainable attack load. Mythos-class cyber models have now turbocharged the supply side of the attack chain.

So, it seems a bit curious that we are now worrying about the "end of humanity" via unspecified means when we were already suffering from a known and credible cyberthreat. Professionals well-versed in the art and practice of cybersecurity are well aware of the facts above. But there is an alarming lack of clarity in the current discourse. The most useful way to think about this is to distinguish models from systems.

A model is a carefully (and expensively) curated set of "weights" that allow these models to predict the "next token" with amazing alacrity. The power and impact of this is not necessary to repeat here. Initial training over a huge corpus of text, followed by alignment, reinforcement learning and other fine-tuning, produces a deployable model. The model itself is a (huge) file full of floating-point numbers. That file is literally useless without infrastructure surrounding it. Most of the general public has experienced this via various chatbots giving them advice, research, entertainment and even companionship.

The infrastructure that allows a model file full of weights to perform useful function is what I call the "system". A trained, deployable model uses a series of grounding prompts to direct its "purpose" and apply guardrails. Even simple text output can be dangerous, e.g. the recipes for biological, nuclear or other weapons of mass casualty. Or fueling mass disinformation and manipulation campaigns. An AI system, specifically one utilizing a large language model, will contain these "system prompts" and guardrails and make the model reachable via a series of APIs.

With these basic and high-level definitions, we should contemplate the juxtaposition of capability and execution. Or rather, ability to execute. The developers of even the earliest models wanted executability, this is now generally called tool calling. Once you give the model the ability to call a tool that does something (e.g. read email(s)), the connection has turned capability into action. A modern system prompt in an LLM system will often describe to the model its ability to create agents or an entity that can perform a particular task. One agent prompt may tell the model that a particular agent can assist the user with making travel arrangements or perform sophisticated fiscal analysis. Now we have turned potential (capability) into coordinated action in pursuit of a particular goal.

A key takeaway here should be that a model cannot independently take action in the world without facilities granted to it by the system in which it operates. A model is not "born" with free agency and the ability to independently execute. Human engineers build the systems that create the agency.

When I say above that "I find it curious" that leaders of the frontier model labs are wringing their hands, it is perhaps not so unexpected. Anthropic, OpenAI et. al. are in the business of making ever more capable models. That is their entire world. My opinion is they would consider themselves "model builders" and not "system builders". This leads to the general (but as we will see, somewhat unsatisfactory) classification of companies as "model companies" or "systems companies". Dario Amodei's "pacing" essay naturally has a clear orientation around models. In part he says:

"...building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas. To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this."

For those not well-versed in AI safety and security, Amodei is describing "building safety into the model". This is certainly valuable but this paragraph and the essay in its entirety shows a tunnel vision around "the only place to build safety is in the model." Later, Amodei even alludes to "AI system":

"I am most enthusiastic about pacing based on what a given frontier AI system can do, and how safe we observe it to be. For example, one possible scheme might be a series of “checkpoints”: if models have capability X, then they need to be accompanied by certifications of alignment properties Y and Z..."

He immediately pivots from system properties back to the model.

The central thesis in this post is you must also build safety into the system. It is a wonderful (and necessary) thing to reduce the propensity for harm, but everyone deploying AI systems has a responsibility to make such systems secure even when the models create harm, regardless of whether this is "probabilistic wanderings" or deliberate manipulation by adversarial actors.

In practical terms, this means that we need to use the best-aligned models in systems with state-of-the-art cybersecurity safeguards. This includes (a partial list):

• information labeling

• mediated tool access

• network egress controls

• credentials

• least privilege

• moderated persistence

• sandboxing and isolation

• monitoring

• incident response

• "stop" controls

And, yes:

• best aligned models

It is not as if Anthropic (or OpenAI et al.) don't understand these system security concepts. They do. But the fundamental nature of their tunnel vision is that they acknowledge and write frequently about security of the model (preventing model weights theft) and the safety of the model (training and alignment) but:

Their top-level frontier safety frameworks do not treat security from the model as a first-class safety category.

"We will try to make the models as safe as possible" (even if it means slowing down frontier development) does not address the third concern.

Earlier I spoke about "model companies" and "system companies". A leading company in the AI sphere, Microsoft, primarily integrates models from frontier lab companies (I will get to MAI in a bit). Microsoft has vast experience building enterprise-grade, secure systems powering Fortune 1000 companies across all significant industries. Microsoft, especially their security groups, well understand the third concern and build infrastructure around it. And while there are plenty of stories of challenges in even those systems, at least a company that primarily has system development and integration in its DNA knows better than to just trust the model alignment to protect their influential customers.

It is an interesting question to contemplate if there is a "balance of power" or a "competitive" analysis to be done about whether model companies or systems companies will predominate and where our security posture will end up being as time progresses. There does not seem to be evidence to date that model companies have adopted a system-centric stance in their dominant public safety frameworks, even when they themselves build systems incorporating their models. I have yet to see, for example, a frontier model lab advocate for inclusion of information flow control, labeling, tool mediation and (gasp) model mediation in their AI system security roadmap.

Microsoft itself, though, offers up an additional dynamic. A company as large and diverse as Microsoft is not solely a "systems company". Microsoft AI (MAI) led by Mustafa Suleyman (formerly of DeepMind) also produces models. A few days ago, MAI produced the MAI Code of Conduct and it reads very much like the Anthropic safety constitution. Although it discusses operator permissions, environmental boundaries, tool mediation, monitoring, least privilege, etc. these are expressed as behavior expected of the model rather than the system that surrounds it. In the constitution, The model will:

• act proportionately when using tools

• treat (tool) outputs as just another form of input

• operate within its defined purpose

• consider mitigations when invoking irreversible actions

• etc.

A systems view would ask the question: why are we asking the model to enforce those?

"Do not escalate your privileges" (model-centric)

versus:

"You cannot escalate your privileges" (security property).

The interesting cultural observation is not "model company" versus "systems company". Since we see this dichotomy inside a single company, the power dynamic is:

model-building organizational power versus system-building organizational power, including inside the same company.

Microsoft itself has systems discipline embodied in product, e.g. Agent365. And the FIDES integration into Microsoft Agent Framework. When Entra and Agent365 power an AI system, the identity infrastructure is effectively saying: "we survive a misaligned model because no matter what, it cannot escalate its privilege."

Satya Nadella, in introducing the MAI code of conduct, had a pithy 5-word summary: "Humans matter more than AI." The MAI code of conduct states that "[a model] may take actions that run against User intentions". Once that statement is made, the response is unavoidable: "don't make obedience to user intentions the security boundary."

The MAI code of conduct is a valuable and useful document. What we sorely need is unification and Microsoft/MAI provides the perfect example of this.

The glass half full view contemplates an industry-supported safety constitution initiative that addresses all three concerns noted earlier:

• security of the model

• safety of the model

• security from the model

In a way that is palatable to all the parties involved. The first step is to change the tenor and content of the conversation.