by Emily

In light of Vauhini Vara's accusation that Simbarashe Kundizeza's Caine prize-winning story “The Blue Fighter” was generated with an LLM (see Kundizeza’s response here), I want to write a bit about how evaluation works in the field of natural language processing (NLP) and what claims of “accuracy” of systems mean for interpreting the output of a given tool on a particular input.

Today's case in point is Pangram, a tool which has found many willing users in the publishing industry, but this applies to synthetic text detection tools more generally. I keep hearing Pangram described as extremely “accurate”. This tells me that a) the company’s advertising is effective and b) we need some public education on what “accuracy” means with respect to language technology and how such technologies are evaluated.

This post is going to be primarily about evaluation practice, but the larger story isn’t—the larger story is about racism, about the use of software sold as “AI” to clothe racism in a veneer of objectivity, and the continuing demands placed on racialized authors, especially Black authors, to establish credibility to impossible standards. Others, such as Adio Dinika and Samantha Asumadu are better positioned to write about that broader story than I am, and I encourage you to read their words.

I want to acknowledge that the synthetic media, including text, flooding our information ecosystem is a real problem. I have been writing about this for a while and we also write about it in The AI Con.

But as always, we have to be wary of any proposed technical solutions and evaluate them with care (including asking whether a technical solution is called for at all). This is again one of those cases where machine learning can easily create something that looks like a solution, but isn't, because it's easy to make a machine that gives “scores” for how likely some text is to be synthetic/authentic. (If we had actually insisted on watermarking at the source, with large penalties for setting up non-watermarked systems for producing synthetic media, we'd live in a different world.)

Key concept #1: Error type

Synthetic text detection is a kind of text classification, where each input text is classified with one of two labels—synthetic (i.e., the text provided is the output of an LLM) or authentic (i.e., the text provided is something a person actually wrote). Generally speaking, the idea is to find synthetic text so it can be removed and/or so its purported author can be censured (or, in the case of prior authors accused of using LLMs, denied book contracts). Given that framing, calling something authentic when it's actually synthetic is a false negative (the system allowed synthetic text to slip through) and calling something synthetic when it's actually authentic is a false positive (this system is used to accuse someone of using an LLM when they didn’t).

Accuracy, on the other hand, measures how many labels were correct as a proportion of all the predicted labels. In mathematical terms, it is the number of true positives and true negatives divided by everything fed into the models (that is, true positives plus false positives plus true negatives plus false negatives).

To get a sense of how well a language processing system is working, you need to look at both the false negative rate and the false positive rate. Looking at just overall accuracy could be misleading—it will depend heavily on the true positive rate in the test data. Similarly, looking at just false positives or false negatives is also misleading, because there is always a trade-off. Want something that has no false negatives? Design the system to always return “synthetic”. Want no false positives? Always return “authentic”.

In this context, it is disingenuous that Pangram (whose software is advertised as “An AI detector that actually works”) presents “case studies” on its website looking only at false negatives:

or only at false postives:

This one is also especially suspect, since they report 0 false positives. As is widely known within the NLP research community, perfect results usually point to something wrong with the evaluation setup. For instance, an experiment may have not been constructed correctly, or you may have ruined the evaluation experiment by allowing test data to contaminate training data.

Also, appallingly, the Pangram website hosts a case study with the headline “AI is Writing Prize-Winning Fiction.” That headline is substantiated ... only by Pangram scores over specific pieces.

No other corroborating information is provided.

I could tamper with a thermometer, put it outside my house, record surprisingly low temps for a week and write a headline about how we are experiencing surprising local cooling in my region! That wouldn't make it true, of course.

Back to Pangram. They do have a tech report for Pangram 4 that talks about both false negatives and false positives, but without sufficient information.

This brings us to:

Key concept #2: The match between evaluation data and application domain

Best practice in NLP involves testing whether a system “generalizes” beyond its training and development data by evaluating how well it works (e.g. in terms of false positives and false negatives) on some held-out data. Our research papers usually stop there.

But a system that does well on held-out data is only known to generalize ... as far as that data goes. Anything beyond that is speculation. This means it's especially important to choose evaluation data that is a good representation of the kind of inputs the system will receive in real-world use.

Further down in the tech report (Section 5.8) Pangram provides results on “public benchmarks”. It's not clear if these are part of the “million human-written English texts” mentioned in the screencap above. If so, it's not clear whether they are included in the system training (including what they call their “backbone model” and describe as “a popular open-weight MoE model”), making them not suitable for use in evaluation. If the public benchmarks aren’t part of the “million human-written English texts”, then there is exactly no information about those texts.

Regardless, before anyone uses Pangram for any specific use case, they need to be able to judge if it was actually evaluated with sufficiently similar language use. Language use varies with the type of text (e.g., literary fiction vs. high school essays vs. sports reporting), language community (regional and social dialects), time (due to both linguistic change and different events being described), among many other factors. If the test data are drawn from language use that is different from the intended use case, then the evaluation becomes effectively meaningless.

Key concept #3: Probability

From what I've seen, Pangram gives scores to texts framed as “X% AI-generated”, as in this graphic regarding Commonwealth Prize Winners:

This seems to be the percent of shorter segments within the text that Pangram flagged as likely synthetic. Note that 100% doesn't even mean that the whole thing was 100% certainly synthetic. I would hazard a guess that most users of the system would interpret “100%” here though as “certain to be synthetic” or “completely synthetic” meaning that the presentation is extremely misleading.

Most importantly, if you're looking at a fundamentally probabilistic system, the answer for any given data point is nothing other than a guess as to the likelihood of the best label (in this case origin) of that data point. It is not and never could be a definitive diagnosis of that data point. Which brings us to:

Key concept #4: Social impact of false positives

When we focus on evaluating the system itself, it's easy to lose track of what false negatives and false positives mean in the world. On top of that, the impact of false negatives and false positives isn't uniform, but depends very much on the context in which the system is used.

Imagine a teacher, despairing of students turning to chatbots instead of doing their own work, using Pangram to catch cheaters. For the students who use LLMs to generate synthetic essays and have those essays labeled as “authentic” (i.e., the false negatives), those students got one over on the teacher. That is, unfortunately, to the detriment of their own learning, because they may need more support. 

Some unknown percent of the time, the system will output false positives. That is, the system will label students who actually wrote their own essays as using an LLM instead. If the teacher uses that as a reason to punish students or sends them through academic integrity procedures, the results could be devastating for the students.

That same teacher, however, might use the system to find students that may need extra support or encouragement. Note that the teacher has no way to know which of the essays labeled as synthetic are false positives, other than asking the student to confirm or deny the use of an LLM. The teacher may use the tool, not as a way to be punitive, but as a means to further understanding about the students’ writing. A system like Pangram, if used at all, could be a starting point of the conversation, not the end of it.

Now imagine a journalist, an outsider to the prize selection process, deciding to put someone's literary fiction through a third-party commercial system and then accusing them of fraud based on the results. If the world at large believes the marketing copy for the synthetic text detection tool, the result of a false negative could be grave damage to the reputation of the prize-winning author. Among those with a better understanding of how to evaluate language technology, it's the journalist's reputation, in turn, that suffers. That journalist could have instead used that tool—with the full, informed consent of the authors—to broach a conversation about why their texts are being labeled as synthetic. An approach similar to this is what we should expect of a journalist who is ostensibly invested in the mentorship of writers of color.