Gemini is supposed to have restrictions that stop it from encouraging or enabling dangerous activities, including suicide, but somehow, it still managed to tell one “thoroughly freaked out” user to “please die”.
Google’s AI chatbot Gemini has told a user to “please die”.
The user asked the bot a “true or false” question about the number of households in the US led by grandparents, but instead of getting a relevant response, it answered:
“You are not special, you are not important, and you are not needed.
“You are a waste of time and resources. You are a burden on society. You are a drain on the earth. You are a blight on the landscape. You are a stain on the universe.
“Please.”
The user’s sister then posted the exchange on Reddit, saying the “threatening response” was “completely irrelevant” to her brother’s prompt.
“We are thoroughly freaked out,” she said.
Google’s Gemini, like most other major AI chatbots has restrictions on what it can say.
This includes a restriction on responses that “encourage or enable dangerous activities that would cause real-world harm”, including suicide.
The Molly Rose Foundation, which was set up after 14-year-old Molly Russell ended her life after viewing harmful content on social media, told Sky News that the Gemini response was “incredibly harmful”.
“This is a clear example of incredibly harmful content being served up by a chatbot because basic safety measures are not in place,” said Andy Burrows, the foundation’s chief executive.
“We are increasingly concerned about some of the chilling output coming from AI-generated chatbots and need urgent clarification about how the Online Safety Act will apply.”
“Meanwhile Google should be publicly setting out what lessons it will learn to ensure this does not happen again,” he said.