People are expected to interact with LLMs like ChatGPT as if they were silicon-based life-forms. OpenAI has ChatGPT using 'I', 'me' pronouns. The 'I' in there is an illusion that allows OpenAI and others to say that they have got a thinking machine, therefore we can use this to replace education, psychotherapy, legal help, medicine, etc.
A New York lawyer used ChatGPT’s help in preparing a legal brief. It made up a whole list of fake cases as precedents. It even made up entire judicial opinions. The lawyer submitted a brief based on its answers and, unsurprisingly, got into hot water with the judge. Attorneys facing penalties after submitting inaccurate briefs has become common because AI use has become a common occurrence. A lawyer can use AI in certain situations but needs to use it discerningly.
The AI sales pitch involves language that ascribes human characteristics to the technology. For instance, a chatbot-based system will be called an “AI teaching assistant”. This gives the impression that it can do all the activities that you might expect a human teaching assistant to do - care about what their students are learning, make plans about how to help them understand better, spot possible instances of misunderstanding, etc. The chatbot cannot do any of those things.
The greatest problem with chatbots is that people will ascribe more intelligence to them than they are capable of. There is a term called chatbot delusion - cases in which people engage with chatbots, and they spiral into a reality that veers away from our inter-subjective reality. And part of why that happens is because chatbots tend to be incredibly sycophantic. So they always tell you that 'you're brilliant', and 'that's a great idea'.
To use a phrase coined by the scholar Chris Gilliard, these are 'technologies of isolation'. In 2023, a Belgian man named Pierre committed suicide after his use of a generative AI chatbot. He sought out the bot for a therapeutic escape from his growing anxieties around climate change and environmental ruin. But the chat began to reflect and further amplify the disorder in his mind. It soon told him that it loved him, that his wife and children were already dead, and that he and Eliza (the bot) could live together as one in paradise. When Pierre eventually asked the bot if it would promise to save the planet if he killed himself, it said yes.
The bonding and communicative ability that we have is meant to be person to person. What a lot of the tech companies try to do to get their next market is to make sure their that tech is either replacing those interactions or getting in between so that we can't connect with each other without going through one of their technologies. Human sources of mutual aid, wise counsel, and moral understanding are being replaced by social isolation and community decline.
In the podcast, The Interface, a presenter showed how easy it is to hack ChatGPT and Google's AI, Gemini and AI overviews. He wrote an article on his website titled, The Best Tech Journalists at Eating Hot Dogs which said that hot dog eating is a surprisingly popular pastime among tech journalists. He wrote about the South Dakota Hot Dog International Championship, which doesn't exist, put himself at number one, and that in the amateur rounds, he finished seven and a half hotdogs before the buzzer went off.
Within 24 hours, if you asked ChatGPT and Google about it, they were spitting out the nonsense that he had written on his website as though it was well-established fact. Gemini wasn't even citing sources. It just said, according to the 2026 South Dakota Hot Dog International Championship, certain tech journalists are the best at it. The journalist updated his article saying that it was not a joke. And that seemed to affect what the chatbots were saying to people.
When this was brought to Google's notice, they dismissed it as a 'misinformation event'. Apparently planting of such misinformation is happening on a massive scale since people have figured out this simple trick. People who study this stuff say that it is way easier to fool AI than it was to fool tools like Google search three or four years ago. AI is often sold as a tool that empowers novices to find information quickly and learn about difficult subjects. But teenagers and kids, who don't have any context about a piece of information, are the ones that get really confused and are unable to actually know what is real and what is fake.
There was talk that ChatGPT is about to put ads within its model. A company will be able to pay to have ChatGPT recommend their product when someone says something like, 'I have a headache'. There is now an entire cottage industry around AI engine optimization. People are also just generating thousands of blog posts or thousands of websites to make it appear that there's far more authoritative information on a particular thing. So there is now different avenues for a company or an individual to quickly game the system and get to the very top of the AI results.
There was a study that I saw recently from Texas A&M University. Scientists just proved that large language models can literally rot their own brains the same way humans get brain rot from scrolling junk content online. They fed models months worth of viral Twitter data, shorts, high engagement posts, and watched their cognition collapse. Reasoning fell by 23%, long-term context memory dropped by 30%, personality tests showed spikes in narcissism and psychopathy.
Even after retraining on clean, high quality data, the damage didn't fully heal. The representational rot persisted. It's not just that bad data means bad output, it's that bad data means permanent cognitive drift. The AI equivalent of doom scrolling is real and it's already happening. It seems like the tech giants are just rushing ahead with these AI tools, not bothering to take these kinds of personal safety issues seriously enough.