486: quantum of sollazzo

#486: quantum of sollazzo – 27 September 2022

The data newsletter by @puntofisso.

Hello, regular readers and welcome new ones :) This is Quantum of Sollazzo, the newsletter about all things data. I am Giuseppe Sollazzo, or @puntofisso. I’ve been sending this newsletter since 2012 to be a summary of all the articles with or about data that captured my attention over the previous week. The newsletter is and will always (well, for as long as I can keep going!) be free, but you’re welcome to become a friend via the links below.

Every now and then I include a six-question interview with an inspiring data person. This week, I speak with Robin Linacre, who is a data scientist at the UK Ministry of Justice, and the developer of their big data linkage platform Splink (which was covered in issue 485 of Quantum).

The most clicked link last week was mySociety’s initiative on climate data. Really topical.

And a final apology – a few of you reported that the sponsored link was pointing to the wrong website. Or vice versa, in fact the link was right but… I had forgotten to update the text. Sorry!

I’m away most of next week, so see you in two weeks’ time, Giuseppe @puntofisso

Six questions to...

Robin Linacre

Robin is a Data Scientist at the Ministry of Justice (UK).

What is your daily data work like and what tools do you use?

I build automated data pipelines to link some of MoJ's largest administrative datasets. This involves training probabilistic record linkage models, applying these models, and then running the graph analytics needed to produce final data products. These pipelines are dockerized, scheduled by Apache Airflow, and run using Apache Spark. I also spend a lot of time building the record linkage software itself, which is an open source Python library available on PyPi.

Tell me about a data project that you're proud of...

Splink is a Python library for probabilistic record linkage at scale that is in use in academia, government and the private sector, and has been downloaded 3.5 million times. The biggest linkage it’s been used for was of over 700 million records. I'm particularly proud of its 'polyglot' ability to run against multiple SQL backends (duckdb, Spark, Athena), enabling it to be used on a wide variety of data platforms.

I think its success has been mostly down to single mindedness/stubbornness. Seemingly every big organisation has a record linkage problem, so we knew there was demand for this kind of thing. But it's taken multiple failed attempts over several years to get something that is now starting to feel right and gaining widespread adoption.

...and a data project that someone else did and you're jealous of.

It's not an exaggeration to say my career has been shaped by a few exceptional open source data projects and tools. I got my first job as a data scientist by showing off tools I’d made in d3.js, an enduringly popular data visualisation library. Later, I helped develop our Analytical Platform, which made open source tools such Jupyter and R Studio available to all our analysts. And finally, the dirty secret of why Splink is so fast is that we ‘delegate’ all the hard computations to open source SQL engines which have been heavily optimised. I’d particularly highlight DuckDB - one of our backends - which is astonishingly fast, and has so many applications across data science workflows.

It’s mind boggling how much value these open source libraries must have to the global economy - especially when the driving force behind these projects is often just a handful of people.

If I say "dataset", you think of...

One or more parquet files hosted at a predictable URL with CORS enabled (please!!!).

Give someone new to data a tip or lesson you wish you'd learned earlier.

Every organisation has a data mess. It’s far too easy to fall into the trap of trying to ‘fix the mess once and for all’ by designing a perfect system. This is a mirage. Focus instead on your customers and ensuring the data is good enough to underpin evidence based decisions. Deliver early to find out what’s valuable, and iterate. Ensure your processes are federated rather than centralised so that many teams can work autonomously towards these goals.

Data is or data are...

Is!

Topical

How Britain has changed since Elizabeth II was crowned in 1953

“Fewer children, fewer coal miners, less cabbage.“

Royal family’s new line of succession after Queen’s death

Brilliant visualization by The Guardian.

NEW: income inequality in US & UK is so wide that…

An eye-opening Twitter thread by John Burn-Murdoch.

Elezioni, l’astensionismo vince solo quando cambia la Repubblica

Six Questions graduate Riccardo Sapority takes a look at abstention rates in Italian elections. Here is a translation into English.

Explore the world like a local with Babbel
Going on vacation is great, but exploring the world like a local is even better. Not speaking the language is no longer an excuse: with Babbel, you can learn to speak a new language in just three weeks. The language learning platform offers 10-minute lessons designed by real language experts, focused on conversational skills in 14 languages. Subscribers also have access to accent-perfecting speech recognition technology, live online classes, games, and more to help you build confidence before you board your flight. Sign up today to get up to 55 percent off your subscription.

Tools & Tutorials

A few lessons learned from tracking The Queue

My friend Steph Gray helped set up the Queue tracker.
From proof of concept to launch was about 3 days. The map itself was built as a single script PHP application using a flatfile database to store the lat/lon co-ordinates of the end of the queue, and a KML file of the planned maximum queue path. A simple management app with a big purple button designed to run on a phone would be used by the marshall at the end of the queue to use the HTML geolocation API built into modern browsers to get their position, preview it on a map to check, and then update the map.

What are capital gains? How are they taxed?

“About two-thirds of individual capital gains subject to tax are from selling corporate stock; the remainder is from property sales.” A good explainer by USAFacts.

Mapping wind data with R

I’m not into this sort of things… until I see a tutorial like this. By Milos Popovic.
“While I encountered many interesting tutorials […], they all used satellite imagery to show wind movement in the form of heatmap rather than vectors. So, I decided to give it a try and make a different wind map.
On my voyage, I was lucky to find package rWind that brings wind data from Global Forecast System to your drive. This fascinating library allows you to donwload wind data for any place on the planet in just a few minutes.“

Linear regression

Yet another brilliant interactive tutorial by MLU.

Deep Generative Model

A Jupyter notebook to play with Stable Diffusion.

ClimateMachine.jl

“The Climate Machine is a new Earth system model that leverages recent advances in the computational and data sciences to learn directly from a wealth of Earth observations from space and the ground. The Climate Machine will harness more data than ever before, providing a new level of accuracy to predictions of droughts, heat waves, and rainfall extremes.“

Data thinking

Deciding if a data leadership role is something you actually want to do

“When I was recently on the market, I also considered a few positions with Head Of titles and eventually ended up accepting one. As I’m sure you can imagine, the responsibilities and scope of those Head Of positions varied tremendously, ranging from an individual contributor doing developer relations to senior leadership roles in established organizations.“

Dataviz, Data Analysis, & Interactive

Is There a Mega Warehouse Near You?

“Explore the impact of e-commerce in every community – including yours”

‘Hope’ is out, ‘fight’ is in: Does tweeting divide Congress, or simply echo its divisions?

“USA TODAY analyzed more than 2.8 million tweets posted by members of Congress. The language has become more divided and emotional.“

Who worked at home before the pandemic?

Chris Loader takes another look at data for “work-at-home patterns from the ABS census 2016 for the six largest Australian cities, with some deeper dives for Melbourne and Sydney”.
“I dare suggest that if someone lived in an SA1 that intersected with their regular workplace Destination Zone, it’s pretty likely that they ordinarily worked at home.“

In the climate crisis, vulnerable countries bear the least responsibility

“Climate change is happening and affects all of us. It is really scary and we’re already starting to see its consequences everywhere. But not every country is truly facing an equal risk from climate change. Not every region will see the same changes in local weather, or be affected by rising sea levels to the same degree. And not every country has the same capability to prepare and protect itself from the dangers it will face.
A third very important difference to consider is climate responsibility.“

The oldest government in history

“America’s gerontocracy is disconnecting Congress from the rest of the country, an Insider data analysis reveals”. It uses data and code from this GitHub repository.

Tracking Fiona’s Latest Path

A nice visual tracker by Bloomberg.

quantum of sollazzo is supported by ProofRed’s excellent proofreading. If you need high-quality copy editing or proofreading, head to http://proofred.co.uk. Oh, they also make really good explainer videos.

Sponsors* casperdcl and iterative.ai Jeff Wilson Fay Simcock Naomi Penfold

[*] this is for all $5+/months Github sponsors. If you are one of those and don’t appear here, please e-mail me