What You Need to Know: A solo developer released SHADOW-250M, a heavily quantized LLM that keeps recent context in fp16 while compressing older tokens to 1 bit on disk. Nvidia published a linear-mapping technique that transfers KV caches between model sizes without full re-prefill. Simon Willison shipped llm 0.32.1 and llm-openrouter 0.7 with new tool support and compatibility fixes. Builders should test the tiny model for offline retrieval tasks and the KV transfer method for multi-model agent pipelines.

DEPTH OVER BREADTH (news items)

Top Story

A developer released SHADOW-250M, a 250M-parameter model trained from scratch on 30B FineWeb tokens and quantized below 2 bits. The full deployment fits in 60 MB and runs at roughly 400 tokens per second on a laptop CPU with no GPU. Recent 2048 tokens stay in fp16 KV cache while older tokens compress to 1 bit on disk at 320 bytes each, supporting up to 100M tokens of history that the model was trained to retrieve. Language-modeling quality on held-out educational web text reached 3.15 nats cross-entropy and 23.3 perplexity, or 0.99 bits per byte. The vocabulary uses fixed 512-bit codes with no trained embedding parameters, scoring 0.619 Spearman on WordSim-353 versus 0.029 for random codes. Example greedy output on photosynthesis reads: “Photosynthesis is a process in which plants convert sunlight into chemical energy, which is then used to produce oxygen and other chemicals.” Example temperature-0.25 poetry output begins: “The waves had swept over, and they were crashing against each other like rocks on top of one another.” The full training kit, master weights for fine-tuning, test scripts, and demo are available at https://github.com/QLNI/SHADOW-250M-Instruct and https://huggingface.co/NODEMIND/SHADOW-250M. Source: reddit.com

Model Updates

SHADOW-250M: r/MachineLearning A 250M model trained on 30B tokens reaches 23.3 perplexity on unseen educational web text while fitting in 60 MB after sub-2-bit quantization. The system keeps the most recent 2048 tokens in fp16 KV cache and compresses everything older to 1 bit on disk at 320 bytes per token, enabling retrieval from up to 100M-token archives. A fixed 512-bit code vocabulary replaces learned embeddings and scores 0.619 Spearman correlation on WordSim-353. The repo includes training scripts, fine-tuning weights, and reproducible example outputs for photosynthesis explanations and poetry. Builders working on offline long-context retrieval should clone the repo and test archive-mode queries this week. The model was never trained to reason over the long archive, only to retrieve and answer from it, and the author notes it remains a 250M model so expect mistakes on open facts. Source: reddit.com

Agent & Tool Developments

Running Codex as a Headless Agent: Towards Data Science The post shows how to turn Codex from an interactive assistant into a programmable automation component that can be called directly from scripts or pipelines. No install command is provided, but the approach removes the chat interface so agents can invoke the model as a library function. The technique targets developers who already use Codex for code generation and want to embed it inside larger workflows without manual prompting. Limitation noted is that the model still requires the same API access and rate limits as the interactive version. The post walks through turning the model into a callable automation piece rather than a conversational partner. Source: towardsdatascience.com

llm 0.32.1 and llm-openrouter 0.7: Simon Willison llm 0.32.1 pins the OpenAI Python library to avoid a broken httpx dependency and prepares for an upcoming switch to httpx2. llm-openrouter 0.7 adds display of reasoning traces, adopts OpenRouter’s Responses API, and introduces three new server-side tools: Shell, WebFetch, and WebSearch. Both releases are free and open source; install with the usual pip commands. The updates make it easier to run agents that need external tool calls through OpenRouter models. The 0.7 plugin is now compatible with LLM 0.32 and can surface reasoning traces for any model available through OpenRouter. Source: simonwillison.net

Practical & Community

Hybrid collaborative filtering recommendation system: r/MachineLearning By-Its-Cover uses only CLIP embeddings for semantic search over book covers plus a two-tower neural collaborative-filtering model for personalized recommendations. The site currently holds a few thousand books and grows when users search; new titles are scraped asynchronously via the Hardcover API. Reciprocal Rank Fusion combines CLIP semantic results with GLiNER NER keyword search, and a Determinantal Point Process diversifies output. The full stack runs on AWS with Lambda, ECS, SQS, and Terraform; sign-up is required for personalized results that update every two hours. The author notes that no AI-generated code was used in the project. Source: reddit.com

Bayesian Guardrails for AI Decisions: Towards Data Science The tutorial explains how to measure prediction uncertainty so an AI system can defer decisions whose mistakes would be costly. It walks through adding a Bayesian layer that outputs both a prediction and a calibrated uncertainty estimate. No specific code snippet is given, but the post targets teams already running production models who need a practical deferral mechanism. The core tradeoff is extra inference cost for the uncertainty estimate versus reduced risk on high-stakes calls. The post is framed as Enterprise Document Intelligence volume 1 issue 7sexies. Source: towardsdatascience.com

Under the Hood: Cross-Model KV Cache Transfer

Everyone talks about swapping models mid-agent session as a simple routing decision. In practice the receiving model must normally re-run the entire prefill to rebuild its KV cache, which scales linearly with context length and model size. Nvidia’s approach observes that KV caches within the same model family are approximately linear structures, so a closed-form per-head ridge regression fitted on a few hundred calibration sequences can map source keys and values into the target model’s expected format. The mapper strips RoPE encodings first, then selects the most predictive source layers for each target layer, turning what used to be a full prefill into a few hundred milliseconds of linear algebra. On matched-KV pairs the technique retains 73–98 % of the target model’s standalone accuracy while running 2.7–25× faster than recomputing the cache; the 8 B to 70 B Llama leap still recovered 72.8 % accuracy. A single source layer recovered 56 % of target key variance and 32 % of value variance; combining multiple layers raised those figures to 79 % and 65 %. Two Ministral pairs needed a small nonlinear MLP fallback because the linear fit failed to extrapolate, showing the method’s current boundary. This builds on yesterday’s discussion of inference economics by showing a concrete way to cut the prefill tax in multi-model workflows. Use the linear mapper when you control both models and need low-latency handoff on long contexts; fall back to full prefill or a trained adapter only when accuracy on a specific pair drops below acceptable thresholds.

Things to Try This Week

• Clone https://github.com/QLNI/SHADOW-250M-Instruct and run the archive-mode retrieval demo on a laptop to see million-token context without a GPU.
• Add the new WebSearch and WebFetch tools from llm-openrouter 0.7 to an existing agent script to test server-side tool calling without extra API keys.
• Deploy the By-Its-Cover recommendation system locally and seed it with your own book-cover searches to watch the vector database grow.
• Prototype a two-model agent workflow using Nvidia’s linear KV mapper on any Qwen or Llama family pair to measure the latency savings on long sessions.

On the Horizon

• More quantized and compressed open-weight models are expected as developers chase 60 MB class deployments.
• Additional cross-model KV cache techniques will likely appear once the linear-mapping baseline is public.
• Simon Willison’s 0.33 release of llm is due soon with the httpx2 migration.
• Further agent-harness experiments will test whether models can absorb more of the scaffolding into weights.

💬 Reply to this email — Patrick reads every one.

Share: X · LinkedIn · WhatsApp

Forwarded this email? Subscribe here — it's free.