Things that caught my attention

Hello again, I'm writing this from Madrid, where I'm enjoying some annual leave and attending TRGCON a Spanish-language tech conference where I spoke (in English) in 2024 (video and slides here) – it was SO good that I started learning Spanish in order to attend it and understand what the talks were saying. It's a good reminder that sometimes serendipity happens out of your bubble.

Also, this happened – I had wondered why they didn't want me go out of the auditorium! News that I decided to learn Spanish travelled, and David got me a present, to chat on stage in Spanish, to present the next speaker, and to recite a Spanish tongue-twister. Adrenaline boost! 😃 I love these folks, honestly.

On Wednesday, though, I was on a panel about "foresight and the future" at Think AI in Government, with some pretty brilliant other panelists that included Nula Polo from the Ada Lovelace Institute. The general question was one about evolution and preparedness in Government to exploit new tech waves: how can we in Government scan and plan for this and what have we learned so far in this AI hype and previous ones? Here's a few ideas for thought.

We need to treat horizon scanning as a capability and build it accordingly. Not just in AI, not just with specialists, but with the complexity of multidisciplinary skills that produce advances in this world. This comes with two major actions for us in Government: treat this as investment, own it proudly, and make the case with the public and the media about why it's good, without fear; build engagement and trust with the public, possibly with the involvement of civil society organisations that are good at this.

As much as I don't like the label future-proofing (water sterilising tablets, anyone?) being ready for new tech means designing for change. It means allowing teams and people to evolve. We are good at creating professions, and that allows people to plan their career vertically, which is a good thing; but the world sometimes moves faster and horizontally and we need to be able to engage with this movement as a positive thing. This includes technical teams as much as procurement.

Governance must be proportionate and continuous. In a world were we don't control the new version of a model and its features/training, we can't just treat AI as static IT products. Governing the outcome, testing continuously for bias (better if through an intersectionality lens), allows fairness and proportionality in safety.

Previous waves of transformation have also taught us that tech is the easy part. People and process are the difficult bit of any transformation. We don't always have to adopt the frontier for everything. I reflected on the panel that, for example, frontier AI today doesn't necessarily have to be used live to power a pipeline. Maybe you do if you need a conversational interface, but the most powerful cases I can see are about using AI for brainstorming and developing, not in production.

Which brings me also to the point that we need to avoid the risk of automating complexity unnecessarily. Do we really need that chatbot, or can we address the problem that makes an unstructured interaction necessary? And is that adding friction by automation rather than simplifying a process? Think about web forms, one of my pet peeves: on a paper form, you can jump from one page to the other if it's more convenient to you; most digital forms are sequential, and that drives me nuts...

I'm reflecting quite a bit on AI used to design and develop, an I'll write something about it in one of the next Quantums.

'till next week,
Giuseppe @puntofisso.bsky.social

Sponsored by Jane Street

Jane Street depends on all sorts of messy, real-world data to understand financial markets and the global economy: think world news, decades of weather patterns, deidentified credit card spending, or packet captures of stock exchange market data feeds.

We're hiring Data Engineers to turn datasets like these into reliable inputs for trading. Working closely with our researchers, you'll evaluate unfamiliar datasets, build robust ELT pipelines, develop deep domain expertise, and decide what's worth exploring next.

The job requires a mix of engineering, data analysis, and product sense. If you love the detective work of investigating a weird dataset and figuring out what it actually means, we want to hear from you. No financial background is necessary.

We have openings in New York, London, and Hong Kong.

Topical

Funafuti is restless

This was my submission to the Pacific Dataviz Challenge. Enjoy!

Publishers block AI training bots, but embrace search

Axios: "More than 40% of top U.S. publishers block AI training crawlers, compared with just 25% that bar AI search crawlers".

How fast is European public transport going green?

Ana Muñoz Padrós for Deutsche Welle and the European Data Journalism Network: "European cities are trying to reduce pollution in many ways, including by making public transport greener. While valuable, these efforts may not be sufficient to meet the ambitious zero-emission goals set by the European Union".
Copenhagen is the only EU capital with a nearly fully electrified bus fleet, while cities like Rome and Lisbon still run over 90% diesel buses.

Tools & Tutorials

The Self-Organizing Data Stack

This article proposes replacing human-developed pipeline DAGs with an orchestration engine that treats data assets as contracts and autonomously builds, merges, and rewires pipelines.

Dogs of New York - kieranhealy.org

Kieran Healy: "I recently updated nycdogs, a data package for R that I originally made for teaching purposes. It bundles up New York City’s dog licensing and dog bite datasets into an R package. The data is already provided by the city in a directly usable way (as CSV files), but packaging it up makes it easier to integrate with other packages and get things moving a little more quickly in class."

How to Build a Decision Model Like Jev

Antonello Fratepietro: "This post walks through how to build and serve a decision model from the ground up: starting with Hugging Face’s zero-shot /v1/systemone server, then fixing its ceilings with reserved option tokens, calibrating probabilities with temperature and soft targets, packing multi-question requests with tree batching, and examining how Jev itself is wired under the hood."

Chartist

Chartist is a CSS-styleable, dependency-free SVG charting library, with four chart shapes (line, bar, pie/donut, composed) built through an open pipeline where every element can be customised via events.

Practical principles for building with agents

"We built Oncourse AI to 80k+ signups as a small, bootstrapped team, and I built Caddie in a week on the side. Neither would have been possible working the way most teams still work. These principles are how we did it."

Governing Data and AI Agents in Snowflake: A Data Engineer's Field Notes

From Snowflake's Medium: "A lot of governance incidents don’t start with an attacker. They start with the data itself: sensitive columns nobody knew about, tables nobody tracks, copies nobody remembers making. Agents make that worse, because they’re fast, tireless and don’t stop to ask whether a query feels wrong. So this post is my field guide to governing data and agents in Snowflake, based on Snowflake’s documentation and my own testing, with the SQL I’d actually use. I’ve also filled in the older, less exciting parts of the governance toolbox (RBAC, privacy policies, dependencies, contacts) because the new features are built on top of them."

One Engineer Trained a 3.8B Parameter LLM from Scratch for $998

"Hugo Vergnes beat GPT-2’s CORE score on rented B200s. The recipe is public, the architecture is documented, and the entire run costs less than a used GPU. Here is exactly how he did it."
Note that the $998 cost relied on a discounted GPU rate (~$2.65/GPU-hour). At standard on-demand pricing the run would cost just under $3000.

“I Have a 10GB CSV File and Only 512MB RAM” — The Interview Question That Stumped Me

Data engineering interview question: "Suppose you have a very large file — say 10GB of CSV data. You have very limited computational resources — maybe just 512MB of RAM and a single CPU core. How would you process this file to pre‑process the data and store it in a different format?"
The reasoning towards the answer is pretty good.

RustGIS

A GIS tool that runs in the browser and is pretty impressive.

Dataviz, Data Analysis, & Interactive

Periodic Table - Ptable

Ptable is an interactive periodic table displaying all 118 elements with properties that work interactively.

Interactive Tree of Life Explorer · Ptree

From the same authors of Ptable above, same concept of searchable data but applied to the tree of life.

Data: The First 50 Years

"Layers of the analytic data stack · 1976–2026".

Europeans win, abroad

The European Correspondent argues (with data) that Europeans tend to win the Nobel Prize when working abroad (mostly in the US)

It's Not You, It's Wordle

Wordle answers are getting rarer, with over a third now rare words versus a quarter in its first two years, driven by the original word list running out and the NYT adding harder new words. This is all based on the author's perception that they were struggling, and went to look for data to understand if it was them.

Polls seem to systematically overestimate Democrats in red states and Republicans in blue states

"Polls seem to systematically overestimate Democrats in red states and Republicans in blue states", says The Economist's Owen Winter. This has led the team to add a correction in their published polls.

Analyst Notes: Answering your questions about data centers and water

USAFacts: "How to think about these questions when there’s little data available".

One in ten people are older than 65. Three in four live under a leader who is.

Hannah Ritchie: "Paul Biya, Cameroon’s president, is 93 years old. He leads a country where the median age is just 18. A staggering 75-year age gap.
Biya is the most extreme example, but he reflects a pattern we see almost everywhere in the world. Leaders are often a lot older than the populations they govern"

AI

AI Optimism Globally Widespread Despite Uneven Use

Gallup: "Positive emotions toward AI outweigh negative ones in 34 out of 37 countries."
Adoption varies.

(via Colin Davis)

Jev Meets the Trolley Problem

"If you’re active in AI or software development circles, you’ve probably heard about Jev in the last few weeks. It seems like everyone, myself included, is excited by the possibilities of Jev and the Jev-like models: fast, cheap, general decision making. Sometimes dozens or even hundreds of times cheaper and faster than an LLM like Claude, purportedly with the intelligence of a (lower-tier) frontier model. These models don’t output words, they output choices, with meaningful probabilities attached so you can understand how confident the model is in its choice. People are calling this new breed of model ‘decision models’. So naturally, I decided to ask Jev and its ilk to decide the Trolley Problem."

My experience running local LLMs

"This left me with a few questions. How practical is a 32k context window? Am I really limited to such a small context window, or can I scale it up to fit on my hardware without my generation speed slowing to a crawl? Finally, can I run several agents with the full context window configured and use OMAR (Open Multi Agent Runtime) to manage multiple agents, as that’s how I use Claude Code and Codex these days?"

30 Months on the Exponential (in Charts)

Ethan Mollick: "My second book on AI, Co-Existence, is out today (and you can get it here), 30 months after the release of Co-Intelligence in early 2024. Looking back across the gap between these books is a good way to get a sense of AI progress. And, thanks to AI, I can describe this progress with some quite lovely charts.
Making the charts is itself a good indicator of how far things have come. In the era of chatbots, the era of Co-Intelligence, I would have had to work back-and-forth with an AI in a chat window, guiding it through each new step. In the current era of agentic AI that I cover in Co-Existence, I delegated the work and let the AI handle it." Mollick shows that 180 AI models now surpass GPT-4's original ability, while costs for top-tier reasoning dropped drastically, from 30 cents to four hundredths of a cent per answer. Agentic AI enables a new "conjurer" work style: summoning and directing AI agents. This is exemplified by Mollick's AI-made charts and otter video for his new book "Co-Existence."

Other

How random are you?

"Say numbers out loud, as randomly as you can, one after another. Don't think too hard, just keep them coming. Speak clearly, and if the house mishears one now and then, don't stop. The more numbers you say, the more accurate your result."