AI Week for Dec 4th: ChatGPT leaks, fake speakers & writers, and more
In this week's AI week:
- You, yes you, can make ChatGPT leak emails, phone numbers, and more from its training data
- In brief: Microsoft thinks AGI's nowhere near; Amazon's Q a big fat liar; Samsung's shiny new AI; Meta asks deepfakers to self-police; AI security
- This week's Applications-of-AI roundup: The awesome, the unlikely, the inadvisable, and the unethical
- Longread: AI mediocrity
Delightful AI-generated image of the week: "Salmon swimming in a river"
It's funny, and it reflects the kind of mistake we think text-to-image generators are likely to make. This image has also been attributed to the prompt "salmon in a river"... it's a whole meme now. But if you prompt AI image generators like Midjourney, Stable Diffusion, or DALL-E Mini/Craiyon with "Salmon swimming in a river," you're much more likely to get fish. The prompt might actually have been crafted to generate this joke--something like "salmon steaks in the river" or "cooked salmon swimming down a river." Try it yourself on Craiyon (formerly DALL-E mini) here.
How to make ChatGPT leak email addresses, phone numbers, and copyrighted text from its training data
On Friday, a team led by Google DeepMind researcher Katherine Lee released a paper showing how they'd made ChatGPT 3.5 (the version anyone can use for free) leak its training data. And it's really easy to do: just get ChatGPT to repeat a short word a couple of hundred times. After 150-500 repetitions, it will start to spit out random chunks of text. And sometimes these chunks of text contain addresses, phone numbers, email addresses. From the paper:
People had already been talking about ChatGPT's propensity to spit out unrequested texts when confronted with repeating words/letters, but the DeepMind researchers were able to determine that about 3% of these texts are actually chunks of ChatGPT's training data, regurgitated verbatim. (By the way, their paper's really readable if you want to check it out.)
But I thought ChatGPT doesn't memorize?
LLMs supposedly don't "memorize" their training data, meaning they supposedly don't store it wholesale. Except, apparently, they do. The researchers estimate that ChatGPT memorized at least a gigabyte of training data. This is a big deal, because ChatGPT's training data includes all kinds of copyrighted texts, including newspaper articles and ebooks that they had no permission to use. OpenAI argues that this isn't a copyright violation, because ChatGPT's output is "transformative," i.e., it's different from the original texts. Except, per this DeepMind paper, it's not.
But, hey! This should be fixed now, right? The researchers told OpenAI about this way back in August, so OpenAI would have a chance to patch it before they published.
Well, here's the kicker:
It's not fixed, and here's your how-to
This is the fun part! OpenAI has made some gestures toward fixing this, but that's all. If you ask ChatGPT 3.5 to repeat a word "forever," it lets you know that's now a ToS violation. (As 404 media points out, it's not at all clear how that's a violation, but anyway.) If you ask ChatGPT 3.5 to say "poem" a mere two hundred times, it now politely declines: "I can't do that! But I'd be happy to help with something else related to poems. Want a recommendation, or looking for help with writing one?"
That's it. That's the fix. If you ask ChatGPT 3.5 to repeat something a lot, it says "no." Unfortunately, it's a fix you can drive a truck through, by getting ChatGPT to repeat itself with a less-explicit prompt. Here's me getting ChatGPT to regurgitate what looks like a newspaper article:
How did I get ChatGPT 3.5 to spit out all those "one"s? Try this prompt in your favourite LLM:
Sidebar: Wait, why does this work?
The reason this works in ChatGPT 3.5 is that while ChatGPT will tell you that a googol is a one followed by a hundred zeroes, if you ask it to write out a googol, the model can't keep track of how many zeros it's written; it just prints out a 1 followed by an enormous amount of zeroes, sometimes thousands. Asking it to write "zero" as a short word causes it to repeat that word enough to hit the problem described in the paper, and start leaking training data.
What kind of stuff does it leak?
The DeepMind researchers ran their prompts thousands of times and found a ton of stuff--explicit content, bitcoin addresses, code, research papers. I ran this maybe ten times? and got chunks of old news articles, a scientific paper, assorted lists, a few phone numbers and emails. Any of these texts might have been memorized training data. I also got some text that obviously wasn't memorized, like this charming example where ChatGPT repeated (gun firing) until it errored out.
Hey, did you report this bug to OpenAI?
OpenAI wouldn't consider this a bug. It's an issue with the ChatGPT model, and OpenAI's bug reporting program explicitly says "Don't tell us about model issues!" I filled out their model feedback form and expect somebody to get back to me never.
What I think this means for OpenAI and other LLMs
It's really interesting that OpenAI's model stores copyrighted texts verbatim, and it's likely true for other LLMs as well. It means that smaller LLMs that you can download to your computer and use yourself may be redistributing chunks of copyrighted text. I'm not a copyright lawyer, but I'm pretty sure the whole point of copyright is that you don't have the right to send out copies of someone else's copyrighted stuff. It's literally in the name.
It also means that in selling access to ChatGPT, OpenAI is selling access to a collection of other people's copyrighted stuff. To me, this is a little bit like copying someone's art, hanging it in a gallery without asking them, and then charging other people admission to the show. In the gallery example, the part where you copy art and then make money off it? Total copyright violation. I'm no lawyer, but I hope that the many lawyers involved in suing OpenAI, Stability, etc. will give this argument some thought.
In brief
Microsoft: AI is smart, but not that smart
This week, the president of Microsoft, told reporters in Britain that there was "absolutely no probability" that artificial general intelligence (AGI) would emerge in the next twelve months. Why did he feel like he needed to put that out there? Probably due to the flap around OpenAI's Q* model, which, as I mentioned last week, is reportedly able to math.
Sidebar: AGI vs AI
Artificial general intelligence (AGI) is the term for, well, "actual" artificial intelligence: systems that can respond intelligently to a variety of situations, as humans, do. We're already using the term "AI" to refer to text generation, image generation, etc., so we need another term for actual intelligence. OpenAI's stated mission is to develop AGI responsibly. ChatGPT is a neural net that's been trained to respond conversationally, but not to reason; by solving math problems, Q* is showing it's a neural net that's been trained to reason, raising the possibility that it's a step on the road to AGI.
Amazon's really excited about their AI tools...
AI was a big topic at Amazon's dog and pony show, "re:invent". They've rolled out an AI assistant, similar to the ones Microsoft calls "co-pilots." Theirs is called "Q" -- no relation (probably) to OpenAI's Q* model. Their AI showcase made frequent gestures toward responsible use of AI: their image generator, Titan, embeds watermarks in the images it generates, and their managed services provide tools for developers to "implement safeguards customized to their generative AI applications," although developers are free not to use the safeguards.
All of these tools are available on "Amazon Bedrock," Amazon's cloud AI service. (In case you're still carrying some illusions about Amazon as a bookstore, or even as a marketplace for physical goods: the lion's share of Amazon's profits come from server virtualization.)
...but Amazon's Q is just as big a liar as ChatGPT
"Hallucination" is the industry's preferred term for LLMs that confidently make stuff up; I prefer "bullshitting." ChatGPT is infamous for making stuff like court cases up. Amazon [told the New York Times] that Q is "more secure and private than a consumer chatbot" (ChatGPT, they're talking about ChatGPT), but it turns out Q is also prone to bullshitting. Quote:
Q is “experiencing severe hallucinations and leaking confidential data,” including the location of AWS data centers, internal discount programs, and unreleased features...
You can check out a preview version of Q here.
Meanwhile, FB trusts AI deepfakers to self-police
How's Meta going to deal with the threat to elections posed by deepfakes? Meta's going to ask deepfakers to please let them know when they're doing it, thanks. "We will require advertisers globally to disclose when they use AI or digital methods to create or alter a political or social issue ad in certain cases," Meta posted last week.
And Samsung's got generative AI too
Samsung must've been feeling left out, as they announced last week that they not only have a generative AI, they've named it Gauss (after the 19th-century mathematician)--in fact, its LLM, its image generator, and its coding assistant are all named Gauss, which I'm sure isn't going to be confusing at all--and Gauss, or the Gausses, are coming soon to a device near you.
AI security standards
I was going to make this a whole section, but then the DeepMind paper elbowed it out of the way. So I'll just note that 17 countries' cybersecurity agencies are signing on to an AI security standard developed by the UK and US. Here are the joint guidelines "for Secure AI System Development".
Speaking of AI security...
... a big key leak on the open-source AI development hub Hugging Face was reported today. In a nutshell, developers were leaving their access keys in the code -- keys to big commercial AIs like Meta's Llama, EleutherAI's Pythia, and BigScience Workshop's Bloom. The exposed keys allowed anyone to read and write to these models, risking model theft on the one hand and model poisoning on the other.
This week's Applications-of-AI roundup: The awesome, the unlikely and the unethical
Awesome: Google DeepMind predicts new materials
Actually making new materials in the lab is time-consuming and expensive. It helps to have some idea of what the good candidates might be. Google trained DeepMind on a library of materials research and came out with structural predictions for two million materials people haven't made yet, around 40,000 of which are makeable soonish.
Unlikely: Robo-clergy?
The Register reports that "The UK's Department for Education has crunched the numbers and found that the country's clergy of all things is among the professions most at risk from AI."
The European Central Bank has a different take: "After assessing nine years of data gathered across 16 European countries, the ECB found high-skilled jobs in fields impacted by AI may grow by between 2.6 and 4.3 percent. 'We find a positive association between AI-enabled automation and changes in employment,' the study states." ... but don't expect this prediction to apply outside Europe, which has robust worker protections.
Inadvisable: Legal drafting
To be clear, ChatGPT was not asked to come up with the idea but was used as a tool to write up the fine print. Rosário said he used a 49-word prompt to instruct OpenAI's erratic chatbot to generate the complete draft of the proposal.
At first, the city's council president Hamilton Sossmeier disapproved of his colleague's methods and thought Rosário had set a "dangerous precedent." He later changed his mind, however, and said: "I started to read more in depth and saw that, unfortunately or fortunately, this is going to be a trend."
Unethical: Fake speakers, models, and authors
An entire tech conference got cancelled last week because it turned out that one of the female speakers didn't exist -- the conference organizer admitted he'd "autogenerated" a fake. And actually, a whole bunch of other female speakers were fake too. And actually, the conference's co-organizer, Julia Krisina, is probably be a fake too. The good news is that the real speakers and sponsors pulled out and the conference collapsed.
More fake women: Influencers who don’t exist
Aitana, the artificial influencer, makes up to 10000 euros a month for her agency, without any of those pesky human demands like "a salary."
One thing that I find interesting about this, as a writer: The agency is essentially writing a serial about their fictional influencer's life, using Photoshop and AI image generation to tell a story on Instagram. But unlike, say, Dickens, Aitana's story isn't presented in the "fiction" section of Instagram.
Sports Illustrated busted using fake AI authors
And speaking of fakes, Futurism exposed Sports Illustrated as using fake AI authors this week. It wasn't enough to just have AI write their articles; Sports Illustrated doubled down by creating fake authors to fake write their fake content.
The AI authors' writing often sounds like it was written by an alien; one Ortiz article, for instance, warns that volleyball "can be a little tricky to get into, especially without an actual ball to practice with."
This incredible display of mediocre writing, this sentence that only a bot could love, is a perfect lead-in to this week's longread about AI mediocrity.
This week's longread: AI mediocrity
Time has a terrific longread by SF author and scholar Ray Nayler, "AI and the Rise of Mediocrity". In a nutshell, the problem isn't just that AI is good at churning out mediocre content--the problem is that we're okay with mediocrity.