Trump appointed Marty Makary as head of the FDA. Marty Makary is one of the peripheral D-list conservative baddies of the COVID era, kind of a Trump administration gadfly who came out hard against vaccine mandates especially. Here’s how PBS describes him:
Makary gained prominence on Fox News and other conservative outlets for his contrarian views during the COVID-19 pandemic. He questioned the need for masking and, though not opposed to the COVID-19 vaccine, had concerns about booster vaccinations in young children. He was part of a vocal group of physicians calling for greater emphasis on herd immunity to stop the virus, or the idea that mass infections would quickly lead to population-level protection.
The Centers for Disease Control and Prevention estimated that COVID-19 vaccinations prevented more than 686,000 U.S. deaths in 2020 and 2021 alone. While children faced much lower rates of hospitalization and death from the virus, medical societies including the American Academy of Pediatrics concluded that vaccinations significantly reduced severe disease in the age group.
Before surfing the COVID waves to minor Fox news celebrity, Makary was most famous as the first author of a 2016 “study” purporting to show that medical errors are “the third leading cause of death in the US.”
What is in Makary’s paper? I put “study” in scare quotes above because it is not, in fact, a study. The authors (Makary and someone named Michael Daniel) neither collected nor analyzed any data. They did not systematically review any literature, either quantitatively (pooling the data from each selected paper and analyzing it as in a meta-analysis) or qualitatively (as in a narrative review of the existing literature, its strengths or weaknesses). None of that shit. The paper is a two page — literally, two page — overview of four other papers on medical error.
One of the papers is a Health Grades safety/quality study from 2004 of 37,000,000 inpatient hospital admissions among people enrolled in Medicare from 2000-2002. One is a Department of Health and Human Services report on the incidence of adverse events in US hospitals among admissions of Medicare beneficiaries (only 838 of them this time) in 2008. One is a study from Health Affairs including 795 patient admissions to “three tertiary care hospitals” in 2004. The last looks at 2341 admissions to “10 hospitals in North Carolina” over the years 2002-2007.
From each of these papers, Makary and Daniel derive — as a percentage of admissions — an “adverse event rate,” a “lethal adverse event rate,” “% of events deemed preventable,” the “number of deaths due to a preventable adverse event,” “% of admissions with a preventable lethal adverse event.” Finally, for each of these papers, they report an “extrapolation” to 2013 hospital admissions data; they also pool the percentage of admissions with a preventable lethal event and the extrapolations to 2013 hospital admissions into a single “point estimate” for the years 2000-2008.
Oy vey. To start, how were these papers selected? Makary and Daniel provide no information about the methodology they used to select papers. For formal systematic reviews, whether qualitative narrative reviews or quantitative meta-analyses, authors typically devote considerable space of their Methods sections to explaining their criteria for including or excluding papers in the review. This is done to make sure that, for example, the outcome of all the papers is the same, that the study designs used in the papers to be compared are comparable, that the study populations don’t differ systematically, and so on. There is none of this in Makary and Daniel’s paper. We also have no idea how comprehensive their review of the literature is — why these four papers? Are there others? It’s not specified.
Which brings me to the second point. Are all four of the papers Makary and Daniel use actually comparable? A cursory glance would suggest they are not. They all study wildly different populations. Two of the studies (HHS and Health Grades) use only admissions among Medicare beneficiaries, while others just use any inpatient admissions. Medicare beneficiaries tend to be older and sicker than the general population of admissions, which might have something to do with their likelihood of experiencing medical errors in general (older and sicker people usually have a lot more contact with the medical system) and of death. The other two papers (Health Affairs and North Carolina) use just inpatient admissions — not specifically among Medicare beneficiaries — but again from two different contexts. The Health Affairs paper uses admissions from “three tertiary care hospitals,” the North Carolina study uses admissions from “ten North Carolina hospitals.” Tertiary care essentially means specialist (i.e., more intensive) care — heart surgeries and things of that nature. Are all ten of the North Carolina hospitals tertiary care centers? It is not specified in Makary and Daniel’s paper. Here’s what the original paper says:
All acute care North Carolina hospitals listed in the American Hospital Association (AHA) database except those providing exclusively pediatric, rehabilitation, or psychiatric care were eligible for selection for the study. These hospitals were stratified according to the AHA's definition of the facility as small, medium, or large; urban or rural; and teaching or nonteaching. The number of hospitals that underwent randomization for inclusion in each stratum reflected the proportion of national discharges from that type of hospital. If an invited hospital declined to participate, another closely matched hospital was randomly invited to participate in its stead.
Ok, so we’ve already got a problem. The admissions that Makary and Daniel are attempting to compare to make their argument are not comparable. Medicare beneficiaries are a systematically different population than all people admitted to tertiary care, who are again a systematically different population than a random sample of people taken from a random sample of hospitals in one state.
And it gets worse! The authors purport to compare adverse events and lethal adverse events due to medical error. Do all four of the papers they cite define these things in the same way? They do not. Here’s how the HHS Medicare report defines the construct:
To establish an estimated adverse event incidence rate, we included events on the NQF [National Quality Forum] and the HAC [NQF list of Serious Reportable Events or the Medicare list of hospital-acquired conditions] lists and events resulting in the most serious harm as defined by a patient harm index (prolonged hospital stay, permanent harm, life-sustaining intervention, or death).
…
The National Coordinating Council for Medication Errors Reporting and Prevention (NCC MERP) Index for Categorizing Errors can be used to classify adverse events by level of patient harm. The NCC MERP Index was initially developed to categorize the effect of medication errors and considers whether the occurrences had an effect on the patients and, if so, how harmful they were.
Rather than using a list, the North Carolina study relied on a “trigger tool” (also used in the Health Affairs study) that is used in conjunction with medical record review:
Internal and external review teams independently conducted two-stage reviews of the same records in each hospital. Within each team, a primary reviewer conducted a review of each record using the trigger tool, which consists of 52 triggers, or clues, in patient records that indicate the possibility of medically induced harm.
They later looked at the NCC MERP to categorize the severity of the harm.
Here’s what the Health Grades study did:
To identify the patient safety incident rates for every hospital in the country, HealthGrades applied AHRQ’s Patient Safety Indicator software to three years of Medicare data (2000-2002) to identify incidences of patient safety events by PSI.
I’m no expert in this, but it seems like the large studies of Medicare beneficiaries, HHS and Health Grades, used — appropriately, because reviewing 37 million medical records is not feasible, and because it’s not possible to do medical record review on administrative data — a list-based screening tool to pick out incidence of adverse events, while the smaller Health Affairs and North Carolina studies used the “trigger tool” in conjunction with medical record review. Full disclosure, I didn’t review the Health Affairs study closely, because it is too much of a pain in the dick to get access to it. But even without looking at it closely, my point still stands.
Is this a joke? Is Marty Makary a secret Foucauldian? Has he been reading the économie des conventions theorists to pull a big Sokal-style goof about representation, encoding, and how science in the vein of the Classical episteme creates classes of equivalence between non-equivalent objects? No, of course not. He’s a clinician play-acting at doing “research” because some number of publications is now an important part of credentializing in medical training.
Returning to the paper, the third issue is — how are these authors extrapolating their findings from each of these papers to the US population? Of course, they don’t say, providing no details about whatever method they used. My guess is that they pulled up some estimate of total US hospital admissions in 2013 and multiplied the “adverse event rate” or “lethal adverse event rate” they created by that number. Again, I will stress, the four studies they use are conducted in different populations and use different methods to determine the rate(s) of adverse events. This isn’t even apples to oranges, it’s more like… apples to wild geese.
Fourth, and finally, how are the “point estimates” they report generated? These point estimates supposedly pool together all the different estimates from the four papers (and the four extrapolations they made based on those) into one figure. Once again, they don’t provide any information about how they did this. My strong guess is that they took some kind of weighted average. (They don’t even have the statistical chops to calculate confidence intervals around these.) I think my guess is right because — notice that the overall point estimate for the percentage of admissions with a preventable lethal adverse event is 0.71, and the overall point estimate extrapolated to 2013 admissions is 251,454. These are the exact same numbers as for the Health Grades study. This is the one that had 37,000,000 admissions, compared to fewer than 3000 admissions in the other three studies. If you take a weighted average of all four of these numbers, the study with three orders of magnitude more observations will swamp the others. How informative is this, really? What was the point of this exercise?
Makary and Daniel would probably say that the point of the exercise was to draw attention to the need for more and better data. I guess this study does do that, albeit indirectly, by being really shitty. There is, in reference to this article, a great Rapid Response on the BMJ’s website by Kaveh Shojania which I will quote from at length:
First, the estimate fails the plausibility test. Of around 2.5M deaths in the US each year, approximately 700,000 occur in hospital.[2] We – and many clinicians and researchers - find it very hard to believe that one in 10 of all US deaths, or a third of inpatient deaths (the 251,454 estimated by Makary and Daniel) result from “medical error”.
Second, the authors of the article do not provide any sort of formal methodology. Their estimate seems to rely on extrapolating preventable death rates from those reported in other studies. They then place the estimate derived from these heterogeneous studies in a “ranking” of causes of death in the US to make their argument that it is the third leading cause. These two steps are both precarious. The four studies on which they appear to base their estimate on use different methodologies and wildly varying definitions that Makary and Daniel collapse into their vividly-titled construct of “preventable lethal adverse event”. It is not clear how the “point estimates” they derive were calculated, but it is notable that the denominators across the studies are not comparable and no confidence intervals are reported.
Makary and Daniel argue for some kind of process (probably medical record review) to add a field to death certificates indicating whether a preventable medical error contributed to the death. In the real-life implementations of this that I am familiar with (maternal mortality review committees), this involves extensive time and labor, not to mention access to medical records. Administrative data, like that in Medicare and Medicaid, hospital discharge databases, and other billing databases, is not medical record data and does not have the same fields as a medical record. So it probably wouldn’t even be possible to apply Makary and Daniel’s vague propositions to some of the major health databases in the country, like those of Medicare. After nodding to the practical considerations involved in doing medical record review, Shojania continues:
As it turns out, this approach [that Makary and Daniel are arguing for] has been implemented in research settings on at least three occasions.[4-6] In all of these studies, the authors sampled deaths from multiple institutions and asked trained reviewers to look over the cases to identify possible quality of care problems and to make a judgment about the preventability of death. In all three studies, reviewers estimated that around 3% to 5% of deaths were ‘probably preventable’ (a greater than 50% chance that optimal care would have prevented death). The largest and most recent of these studies[5] reported that trained medical reviewers judged 3.6% of deaths to have at least a 50% probability of avoidability. Applying this rate of preventability to the total number of hospital deaths in the US each year produces an estimate of about 25,200 deaths annually that are potentially avoidable among hospitalized patients in the US—roughly 10-fold lower than the estimate advanced by Makary and Daniel.
TL;DR, Makary’s most famous piece of research output is basically just made-up garbage. What is interesting about this?
For one, this guy is gonna be in charge of the FDA now. Might we want someone with a bit better track record of evaluating and producing clinical research to be in charge of food safety regulations and which medical devices get approved? For another, consider the takeaway of this “study,” and how it was extensively reported on in the media: medical error is the third leading cause of death in the US. This paints biomedicine and its practitioners in a terrible, terrible light that feeds into the conspiratorial mistrust that the MAHA movement is nurturing. Doctors and medical expertise will kill you — because they won’t tell you the TRUTH about SEED OILS! Because they’re all phony experts on the take! Because (if you’re Emily Oster and apparently believe this) they’re FEARMONGERING ABOUT THE RISK OF RAW MILK!
Of course, doctors and the health care system are pretty fucking terrible. This is one of the most vexing and interesting things about the MAHA movement: it braids together the bullshit with the non-bullshit in a way that is very hard to disentangle. To just about any outside observer, it all looks like… well, hair. But it’s not surprising that someone who makes bogus, overblown claims like this (that scientific journals like BMJ are all too willing to publish) is going to have an important role in RFK Jr.’s HHS. As I’ve said before and still suspect, this is paving the way not to bring a holistic and preventive approach into the heart of health policy in the US but rather to crack open new markets for alternative and woo-woo treatments and the soi-disant health entrepreneurs, like Calley Means, who peddle them. Undermining trust in the health care system as such — whether it deserves to be undermined or not, as it certainly does to some extent — is a crucial component of this.
In the afterword of Burnout, Hannah Proctor quotes from the poet Anna Mendelssohn:
There was no nourishment for us in the world we were born to, and the initial fuel we had found in each other was burning out. It was a lonely time. And I blamed the State for our alienation — though by that time I was numb from the harm we do to each other.
It hit me that our political activities arose out of despair — that I had never really believed that the revolution was possible anyway, let alone inevitable. Everything out there was all too big, too complicated, to take on; too anarchic to make any logical sense of. I’d slowly progressed from world politics to city (London) politics to local politics, till finally I was left with the smallest unit — myself.
COVID was a political defeat, as I have written about before, on several levels, no matter what a person’s actually expressed politics are. It crushed us, it made everything about life worse, it alienated us from ourselves and each other, it shattered trust in anything, let alone any institution or any type of collective activity (besides MLM scams, which are a form of collective activity). It forced a retreat into our wounded individual subjectivities. When this happens, New Age cults and alternative healing are always there, speaking at the level of the self, ready to scoop desperate and hurting people up. It will be interesting to see how this plays out on an institutional level with the MAHA takeover of health authorities, but it probably won’t be fun. I am not optimistic.