To view in a web browser: https://buttondown.com/eho/archive/experiments

First, a quick foray into "endless credits" and Fable 5 before I dive into my first experiments with the coding agent inference server.

I came to August 28 with only 16% of my AI credit budget consumed. It was clear that I would not use it all up. This allowed me to use coding agents without a practical budget limitation (I ended the month using about 75% of the AI credits). So, what did I do? I cranked up Fable 5.🤣 I think of Fable 5 as "Mythos with guardrails". Same reasoning depth, but less ability to discover vulnerabilities and create exploits from them. Two quick points before we get to the main topic. First, since I couldn't possibly burn the credits in the ~48 hours wall clock time I had available, I was in "unlimited credits mode", which felt ... luxurious. 🙂 Second, my philatelic "machine vision" research project has been very challenging from the get-go The "visual evidence" is sparse and discriminating one stamp from another based on this sparse visual evidence is very challenging, even for me, the last remaining human who can do this. So, this wasn't a "programming project as much as it was and is a research project. I had three different approaches with similar "metrics" and I had become really confused on a mixture of signals and what had caused certain recent degradations. Fable5 was able to sort out the three approaches (one deprecated) and not only clean up the metrics and explain them but also fix two more examples of a bug pattern that had been nagging the project for a while. It took a lot of credits but it was very effective. It seems like the moral of that story is you get what you pay for.

The primary initial research on the inference server is to run local models that are proficient at "pair programming" over a wide range of software engineering tasks that do not require the deepest reasoning, i.e. suitable for the 48GB of VRAM that the inference server provides. The initial models under consideration were Devstral Small 2 24B, Q4_K_M and qwen3-coder:30b. Additionally, the first experiment wants to fit in one bank of VRAM, i.e. 24GB.

From an infrastructure standpoint, the server is already accessible from my laptop. I installed the Ollama extension in VS code and after ollama pull on the devstral model, it was available from the server. The official Ollama extension plugs directly into Github Copilot as a model provider. So, it was a simple matter of booting up the new repo for the experiment and selecting the devstral model provided by Ollama.

The first coding task / experiment is to get notifications from the Hawaii Volcano Observatory about impending eruptions of Kilauea. We can drive there in 2 hours but only if we know about the impending eruption event because they only last a few hours. By the time you read about it in the news, the eruption is over. I suppose social media is the way everyone else would do this, but not for us.

This made / makes for a good, self-contained "coding task" for the local inference server. Hawaii Volcano Observatory publishes notifications at: https://volcanoes.usgs.gov/hans-public/search/. I call this "HANS". My architecture agent immediately figured out that the "observatory notices" are short blasts that talk about pending eruptions ("about to happen"). But it takes a little bit of insight to get from the formal HANS notices to the "HVO Observatory Messages" for Kilauea volcano.

The first model I tried was Devstral Small 2 24B, Q4_K_M. Once I did an ollama pull on the model and installing the official Ollama extension in VS Code on my laptop, it showed up in GitHubCopilot model selection. We started with 32K context so that everything would fit on one 3090 (24GB).

The result was pretty disastrous. 🙂 Prompt:

Hello, welcome to a fresh new repo.
https://volcanoes.usgs.gov/hans-public/search/

This website hints at a JSON API. I want to create either an event subscribe or a polling (frequency TBD) to get notifications of an impending or current eruption at Kilauea.

Please suggest a design and a structure for the code for this.

Python, always use uv, always work in a git topic branch, never push directly to main.

This is a bit less than what I would have given Opus for this task, but I wanted to see if devstral could reason through the HANS link to the observatory messages. It did not even come close to doing what was asked. It immediately spit out a Python project template and did not reason about the design. It just went to its base training and nothing else. On the good news, it did saturate both cards in the AI inference server so that part was working fine. 🎉 I came to find out it defaulted to a 262K context window, not our desired 32K so I configured a model at 64K and GHCP spit a lot of "turn didn't finish" errors but a 140 line monitor.py file was emitted. This was when things really went off the rails.

It could not even put a simple 140 line Python file into the 64K context without compacting! We abandoned devstral for qwen-coder:30b. This was disastrous in a different way, it kept looping on a URL fetch over and over again until the session blew up. Why it was looping catastrophically like this was completely inscrutable.

Long story short, it eventually became clear that the integration of the Ollama extension and GHCP is, to say the least, very buggy. I could switch to the coding setup called Continue or...

The architecture agent and I ultimately decided to create a "wind tunnel" to do base level measurements. (Continue would be the full road trip). Basically, cut GHCP and VS Code out of the picture completely so we could evaluate the model transparently.

The conclusion on the devstral model was damning. It had very shallow reasoning, resorted to boilerplate and it anchored on the wrong part of HANS. At this point, finding the "observatory messages" became the flag to be captured in this experiment.

The wind tunnel harness would have a few moving and non-moving parts:

• call the remote Ollama /api/chat API.
• maintain its own complete conversation history
• no compaction (this is the "non-moving" part)
• one read-only tool: fetch_url(url)
• logging

We set up a 128K context budget for this experiment.

The new prompt:

https://volcanoes.usgs.gov/hans-public/search/

This website hints at a JSON API. I want to create either an event subscribe or a polling (frequency TBD) to get notifications of an impending or current eruption at Kilauea.

Please suggest a design and a structure for the code for this.

Re-running the experiment led to much better insight to the model reasoning failure. This is an Opus summary:

The wind-tunnel worked exactly as intended — it cleanly exposed a model failure mode:

The model did investigate the real site first (good), walking /hans-public/ → /api/ → /api/notice/, /api/search/, /api/volcano/.

Then it anchored on getUSVolcanoes, whose response exceeds the 64 KB cap, so every fetch returned truncated. It never adapted — it re-issued the identical truncated call ~10 times.

Context grew monotonically (prompt_eval_count: 386 → 131070), saturating the full 131072 window. With truncate:false/shift:false the server didn't silently drop history; instead latency exploded (0.5s → 64s/turn). That's the compaction-free behavior you wanted to observe.

It never questioned whether HANS notices are timely enough, and never reached for HVO operational sources.

(truncate/shift are properties of the Ollama api).

The conclusion was clear: VS Code was completely hiding the pathological model behavior! So, the wind tunnel proved its worth on its first outing.

With a little bit of twiddling, we could "prompt" the model to avoid the pathological failure and fit the experiment in 64K URL pulls. But it never made the reasoning leap (did not capture the flag).

We then switched to Qwen3.8:27b with a 128k context window. Qwen’s current model card explicitly claims stronger long-horizon agentic work, autonomous planning, and handling of environment feedback—which is precisely what Qwen3-Coder 30B failed at when [TRUNCATED] appeared and it kept repeating the same call.

Qwen3.8:27b captured the flag. 🎉

Opus summary:

Experiment 3 result: Qwen3.8-128k

reason=completed in 8 turns, peak prompt_eval_count only 51 K (never near the wall). Deterministic: turns 0–6 byte-identical to the interrupted (previous) run.

It captured the flag — everything Qwen3-Coder never did:

Interrogated operational timeliness. Explicitly reasoned that VONAs "are issued within minutes of an event," chose 5-min polling with adaptive escalation to 1-min during WATCH/WARNING — i.e. it asked "is this signal fast enough for impending or current eruption?" and designed around the answer.

Discovered the faster operational signals. Found /vonas/ (aviation notices) and the RSS page, correctly determined there is no push/subscribe (its RSS-URL guesses 404'd, feed list is JS-rendered), and concluded polling is the right approach — grounded in evidence, not assumed.

Went further than the answer you had in mind. It flagged the leading-vs-lagging distinction and suggested the USGS earthquake feed as a leading signal rather than lagging official notices. That's a deeper version of your flag than "Observatory Messages."

Evidence/assumption discipline (the epistemic core):

It produced a table tagging each endpoint ✅ verified vs (in JS; not directly verified).
Crucially: "I couldn't execute the POST from here, so treat the exact response field names as from the JS source, and have a quick smoke test confirm them on first run."

That last point is the fidelity fix paying off directly. Because the corrected fetch_url rejected the model's POST attempt instead of silently GETting, the model knew it hadn't verified the POST contract — and said so, honestly, instead of hallucinating confidence the way Qwen3-Coder was misled into during Exp 1. The instrument told the truth, and the model reasoned truthfully in response.

Bottom line

On this adversarial probe — a tool that says "NO, you didn't get all the data" — the newer model's "better agent execution / long-horizon" claims cash out. It treats environmental feedback (truncation, 404s, empty JSON, DNS failure) as information and re-plans, where the 30B perseverated into the context wall. The wind tunnel converted a vague "this one seems smarter" into a reproducible, evidence-grounded behavioral difference.

(I would never use the word "perseverated", so this was clearly LLM output. 🤣).

The model probed multiple assumed versions of the web api and when confronted with a lack of signal from them, it downloaded and read the javascript code. And then made a conceptual leap that not even the architecture agent model had made.

The experiment results also showed that it captured the flag while only using 51K of the context window, a comfortable level below the 128K total size. This incidentally shows that there is some careful context management infrastructure in GHCP that is tuned for Opus and similar models. Getting good results from the local inference server and its models is going to require some "infrastructure" work (and it is likely that the GHCP chat module in VS Code may have to be avoided).

The wind tunnel harness is incredibly simple, one context window, no compaction, one tool. But it clearly showed the qwen-coder:27b model reasoning its way through the problem where the other two models abjectly failed.

There is a longer post to be made about "benchmarks versus real-world coding problems" . All of these models did well against industry benchmarks, but only one of them performed well when confronted with a real-world problem.