Episode 28: LLMs Are Not Human Subjects
Episode 28: LLMs Are Not Human Subjects
In this episode, we put on our social scientist hats and take on the churn of research papers suggesting that LLMs could be used to replace human labor in social science research—or even human subjects, getting into why these writings are essentially calls to fabricate data.
We take apart four pieces:
ChatGPT outperforms crowd workers for text-annotation tasks in the Proceedings of the National Academy of Sciences. Plus the the responses by MTurkers themselves (Beware the Hype: ChatGPT Didn't Replace Human Data Annotators) and media coverage at Vice (ChatGPT Can Replace the Underpaid Workers Who Train AI, Researchers Say).
Political Analysis: Out of One, Many: Using Language Models to Simulate Human Samples
Behavioral Research Methods: Can large language models help augment English psycholinguistic datasets?
Information Systems Journal: Editorial: The ethics of using generative AI for qualitative data analysis
Plus, Fresh AI Hell about a fake Willy Wonka fairyland, fake quotes attributed to Emily in a newspaper, and election misinformation via deepfakes of Black voters with former President Trump.
We've crossed 50,000 downloads!
Thanks to all of you who have downloaded, shared, tweeted and tooted about us. Not bad for an accidental podcast forged from a Twitch stream! Looking to the next 50,000 with you all!
You can download the podcast at any place you download fine podcasts. Find us and all our past episodes on PeerTube, and wherever you get your podcasts! You can watch and comment on the show while it’s happening LIVE on our Twitch stream.