The Sandia Report: numbers and struggle
In school, I was taught that there are two kinds of people: math people and non-math people. The math people are smart and the non-math people aren't. The main lesson I took away from my math classes was that I was a non-math, non-smart person. Then in college I discovered that I had a knack for logic and studied philosophy of mathematics, including set theory and mathematical logic, and started to work through the intense layers of ideology and bullshit in math education had left like sediment in my psyche.
After studying marxism more carefully, keeping my experience with math in mind, I saw that numbers have an intimidating power. They shut people down. They bore people. They're abstract and rigid in certain ways, and they run the world, but also deep down numbers are just language. Math is a formal language, yes, but it's as slippery and prone to/formed by struggle as natural languages like the one you're reading right now.
My favorite passage in Marx's Capital, Vol. 1 is the chapter on the struggle over the working day, which is a great example: workers' hours, pay, and benefits are the result of class antagonism. The numbers are literally caught up in the fight. But it can feel like these numbers are set in stone, determined by a market that has naturally and immutably set the price of labor and benefits at whatever the innocent and well-meaning bosses have stipulated. But they're not.
The intimidating power of quantification, combined with its underlying discursive contingency (for those that know how to wield it), has made math an effective tool in pushing agendas for centuries. Indeed, the power of numbers in education now is unquestioned. Everything must be data-driven, for example, because data don't lie.
Oh, but they do. This post is about one of the most maddening and exemplary cases of mathematics' ideological potency in educational policy.
A Nation at Risk
Maybe you've heard about the infamous A Nation at Risk report, published in 1983. It was a watershed moment for US education. After a landslide victory consolidating a new and vicious rightwing hegemony, the Reagan Administration set its sites on America's schools.
Until that point, there had been twenty years of incisive civil rights struggle, mostly in federal courts, to get schools to integrate. A big question in education policy wasn't necessarily whether schools were succeeding or failing in some decontextualized or standardized or traditional way, but whether they were equal or unequal, fair or unfair, unjust or unjust, segregated or integrated.
Reagan's administration changed all that, exemplified by this bombshell report. The thesis of the document is that American schools are failing, threatening the position of America's power in the world. The nation was at risk its authors argued: if students in our schools don't perform well, then how can America be the most powerful country in the world?
The report was based largely on a calculation of average student performance on the SAT. There was a big scary graph at the center of the propaganda: a downward sloping line showing how average scores on the test had been sinking. The authors argued this downward trend was the symbol and cause of America's sinking power overall: a graph of the loss of American greatness.
Deployed in the conservative froth of the 1980s, A Nation at Risk reset the terms of national educational policy debate that we still have today. The question became how to make schools excellent rather than equal, how to hold school people (teachers, boards of education, superintendents, parents, policymakers) accountable for their decisions, and a skepticism that the solution was spending more money. The high-stakes standardized tests that students have to take today have their policy origin in the discursive din this report caused.
Most conversations about schools start from a presumption, stated or unstated, that our schools are "failing." A Nation at Risk solidified this presumption and went on to justify all kinds of irrational, punitive, and miseducative practices to get schools to 'succeed'.
This is a well-known story in education policy. What's less well-known is that the numbers behind A Nation at Risk were deep, profound, statistical bullshit.
The Sandia Report
Sandia National Laboratories, under the direction of the Secretary of Energy James Watkins, raised its eyebrow at A Nation at Risk. A handful of researchers at the think-tank took a second look at what the report was claiming. Ten years after A Nation at Risk was released, these Sandia researchers published a report in response: "Perspectives on American Education: An Annotated Briefing."
The report found that, when it came to SAT scores, the reality was the opposite of what A Nation at Risk said. Scores were actually "steady or improving." After a couple decades of successful civil right struggle, "[d]eclining average SAT results underscore that a more diverse mix of students is taking the test." Of course, researchers admitted that "average performance of minority and urban students remains low," but this was "despite improvements over the last 20 years." Read their conclusions now is scandalizing:
Our investigation of the SAT data revealed that the much publicized "decline" in average SAT scores misrepresents the true story about student SAT performance. Although it is true that the average SAT score has been declining since the sixties, the reason for the decline is not decreasing student performance. We found that the decline arises from the fact that more students in the bottom half of the class are taking the SAT today than in years past.
So what was actually happening was that more and more diverse students were taking the SAT and doing better and better on it. The Sandia Report, as it became known, demonstrated that when you broke down the average decline in SAT scores by subgroups (like racial demographics) every subgroup showed steady performance or improvement.
But how could it be that the overall average of SAT scores went down while all the subgroups had gone up up? The Sandia authors attributed this quirk of the SAT data to a statistical phenomenon called Simpson's paradox, which "shows that an average can change in a direction opposite from all subgroups if the proportion of the total represented by the subgroups changes."
So the right wingers had manipulated the numbers to took like the nation was at risk. But it actually wasn't. In fact, the nation was doing well! At least when it came to SAT scores, there had been increases in the number of students taking it and their scores had improved. The whole narrative of America's failing schools and thus declining power relied on a statistical mirage exploited by right wing zealots, which was then seared into the national imagination and survives as an ideology plaguing public education to this day.
We should note that the ideology was racist. It characterized a more diverse and accessible student performance on the SATs, a dynamic in which non-white populations were gaining ground, as failing. But beyond that racism, the ideology of failing schools, accountability, and standardization was mobilized to attack on teachers' unions, educational expenditure writ large, and the entire public schools system. Rather than a statistical finding that led to action points, the move to average those SAT scores and used Simpson's paradox to launder an already existing set of rightwing demands.
And the history was written with A Nation at Risk on top. We don't talk about the Sandia Report today in the same breath. The two reports should be taught side by side, rather than ceding the historical ground to the rightwing agitprop. The more general lesson though is that numbers and quantitative methods are ideological, caught up in the wider class struggle, and socialists have to engage with them properly as we organize for a better world.