Will the LLM decide you need an ambulance?

By: Emily M. Bender and Decca Muldowney

In a new dispatch from the front lines of AI Hell, we learned this week that the city of Seattle is using “AI” technology to listen to people’s 911 calls and provide triage suggestions to the dispatcher without the caller’s knowledge. 

New reporting from the Seattle Times shows that, for the last two years, the city has been using tech from a Danish company called Corti to process calls alongside the dispatcher and provide input on whether they deserve a rapid response, e.g. from an ambulance, or whether they should be rerouted to a nurse line for less immediate care. After questions from the newspaper, the Seattle Fire Department confirmed this system, which they are calling “AI”, had been implemented without any public review and without disclosing its use to callers. 

We are, of course, horrified by this. Emily has been working on the societal impacts of language technology since late 2016, and her nightmare example since pretty early on has been using automatic transcription in emergency services systems. Now we learn it’s happening in the city where she lives — and that (per the Seattle Times), Seattle and Corti have been working together since 2019. 

So why is this particular use of speech technology a nightmare scenario for a computational linguist? Because automatic processing of speech (whether for transcription or detecting any other info in the speech signal) does not work equally well for different people. If you haven't evaluated the system on all of the language varieties (different regional accents, different racial or ethnic accents, different second language accents, different life stages, different disabilities) among the people potentially calling in, you can't be sure it'll function as required. But you can be sure it will disproportionately impact anyone whose language is outside the mainstream, i.e. anyone who is already suffering through stigmatization of their speech patterns.

And there are further issues. Speech processing systems also tend to work best when people are relatively calm, which is ... not what usually happens when you're calling 911.

The way Seattle and Corti are (apparently) using this tech at least sidesteps a second concern: automatic transcription systems are likely to do worse with unusual words, e.g. street names. What's happening here, according to the Seattle Times, is that Seattle routes some calls to a nurse helpline if they are deemed less urgent, and the automated system is processing the call and providing suggestions to the 911 operator. The Seattle Fire Department said the existence of the “AI” wasn’t disclosed to callers, but even if it had been, this sure as shit isn't a context in which people could meaningfully consent.

We’re grateful to the Seattle Times for exposing this and hope that the Mayor of Seattle will fix this, stat. We need transparency — not just of the fact of usage of this kind of tech but also of the decision making processes here — and ideally, the end of this contract.

At Mystery AI Hype Theater 3000, we’ve been covering the impacts of “AI” in healthcare, and the hype around LLMs, for years. We’re also on the beat of the disparate impact of tech failures, and how those patterns routinely and unsurprisingly reproduce established patterns of oppression. Here are some episodes to get you up to speed:

• In Linguists Versus 'AI' Speech Analysis, linguistics professor Nicole Holliday helps us unpack the massive hype around using LLM technology to summarize video meetings, arguing it is sparkling bossware with little insight into how we talk — but a large dollop of racial bias. [Livestream, Podcast, Transcript]

• In Beware the Robo-Therapist, UC Berkeley historian of medicine and technology Hannah Zeavin tells us why the datafication and automation of mental health services are an injustice that will disproportionately affect the already vulnerable. [Livestream, Podcast, Transcript]

• In Med-PaLM or Facepalm? A Second Opinion on LLMs in Healthcare, Stanford professor of biomedical data science Roxana Daneshjou discusses Google and other companies' aspirations to be part of the healthcare system, and the inherently two-tiered system that might emerge if LLMs are brought into the diagnostic process. [Livestream, Podcast, Transcript]

• In Chatbots Aren't Nurses, we talk to registered nurse and nursing care advocate Michelle Mahon about why generative AI falls far, far short of the work nurses do. [Livestream, Podcast, Transcript]

• Finally in A Bad Case of Hype-itis, Alex and Emily take a good look at ChatGPT Health, and argue that that cure for an expensive and inaccessible health care system is not AI hype. [Livestream, Podcast, Transcript]