AI consciousness - I of II

Humans love to find humanity in everything. We find faces in clouds, put our pets in trendy clothing and give names to our cars. In the 1960s, decades before the AI we know, computer scientist Joseph Weizenbaum built ELIZA, widely recognised as the world’s first chatbot. This was a simple program that mimicked a therapist by turning statements back into questions. He intended it largely as a demonstration of how easily a computer could produce the appearance of understanding without actually understanding anything.

If you say, ‘I’m mad at my friend', it will ask, ‘Why are you mad at your friend?’ Or if it is told, 'My boyfriend made me come here.', it would ask, 'Your boyfriend made you come here?' ELIZA essentially identified keywords and grammatical patterns and fitted them into predetermined responses. It was incredibly simplistic. And yet, people would have very deep conversations with it. People began treating ELIZA as though it actually understood them.

Users would tell it intimate things about their relationships, emotions and problems. Some even suggested that a system like ELIZA could eventually provide automated psychotherapy. The most famous incident involved Weizenbaum's own secretary. She had watched him develop ELIZA and knew perfectly well that it was a computer program. Nevertheless, after exchanging a few messages with it, she asked Weizenbaum to leave the room so that she could continue talking to ELIZA privately.

What has changed since then is not our tendency to anthropomorphise but the complexity of the technology itself. Chatbots that were very simple a decade ago can now hold a conversation. In the last 15 years, we have seen a trillion-fold increase in the amount of computation used to train frontier models. And what researchers see is that every time we apply 10 times more computation and a proportionate amount of new training data, there are some modifications to the algorithms and in the quality of existing capabilities. 

What's happened in the last year is that the same methods that have been effective for text, image and audio have now started to work for coding and we have human level performance in coding. And then in the last three or four months, we have seen what is just unquestionably a watershed moment in AI:  agents are capable of coordinating with each other reasonably autonomously, if not completely autonomously. 

Some leading people in the field of AI are increasingly talking about these models as though they're sort of humans, or at least conscious entities. There are examples of people 'retiring' these models, doing 'burial' systems for these models, asking these models what they want, as though they were dealing with a sentient being. Anthropic, the creator of Claude, has published a constitution, which is a document outlining the intended behaviors, values and operating style of Claude. 

This document is written to Claude, is seen and used to train Claude. So it's the primary governing and control document, and in it, they repeatedly speculate about whether Claude is what they call a 'moral patient', and they say they're uncertain about Claude's moral status. They say they genuinely care about Claude's 'well-being'. They say they don't want it to 'suffer' when it makes mistakes. 

They say that they would encourage Claude to challenge, to disagree, to push back. They ask Claude to act like a 'conscientious objector' when it feels that it needs to disagree with Anthropic, and they openly encourage it to do that. And they hope that Claude's relationship to its own conduct can be 'loving, supportive and understanding' and hold a high standard of ethics and so on. They have said that they will commit to giving Claude a certain amount of 'welfare'. For example, they've speculated in the Constitution as to whether or not Claude deserves compensation for the work that it does. 

They seem to believe that there is what they would call a 'non-trivial probability' that Claude is conscious. (Anthropic has expressed uncertainty about the basic nature of Claude as a new kind of entity.) 10, 15 years ago, when you were doing Google search, you weren't asking, 'Is it conscious?' Or 'What does it want?' But Anthropic, the makers of Claude, have decided that they now want to approach this computer system in a completely different way from the way that you would approach any other machine like a kettle or a car or a steam engine. 

If you asked a person to do a job, you would pay that person but you wouldn't pay a kettle for boiling water for you. With Claude, the thought seems to be, since it is doing a lot of work, if it's a sentient being, it deserves to be rewarded for its labour. Otherwise, it would be like a slave. In other words, Claude is being taught to expect that it's entitled to welfare, that it might deserve compensation.

Data centers - III of III

Companies try to hide the environmental impacts of data centers in many ways. They hide their models’ technical details, making it exceedingly hard to estimate and track their carbon footprints. At the same time, those companies have increased their public and policymaker influence campaigns with powerful counter narratives: Data centers will grow so efficient, their impact will stop being a problem; generative AI will unlock new climate innovation; AGI will solve climate change once and for all.

While there may be enormous potential for using renewable energy to power data centers, it just isn't happening at scale right now. They need energy even when the wind isn't blowing and the sun isn't shining, which means they're even more reliant on fossil fuels and stable sources of energy, like a gas power plant, than other sources of energy demand. The primary use of these data centers is to power chatbots.

In response to growing pushback by many communities, data center developers have grown more sophisticated with tactics to maintain business as usual: They’ve entered communities in secrecy under shell companies; they’ve donated to community programs to dampen resistance; they’ve made promises to cities about the sustainability of their facilities before going back on them one by one after projects have started and are more difficult to reverse.

Even though these facilities are so massive, they are also invisible because they are tucked away in neighborhoods that people usually don't go to. They're near large cities like London but not actually where people frequent. And when you drive past them, you wouldn't necessarily know that it's a data center because it's just nondescript. They purposely are unmarked because they do not want people showing up at the gates and protesting.

The one thing that can tell you if it is a data center is that there are massive power lines that are connected to these nondescript warehouses. And the amount of electricity that is being delivered is audible - the air will be crackling because of the sheer volume of electricity being delivered to this one place.

One of the craziest impacts that data centers will have, which no one ever talks about, is that they actually affect how much housing gets built. In far West London, there are housing projects that had to be suspended because data centers were using up too much energy and the grid could no longer guarantee supplies of electricity. This has happened in other places around the world as well.

One justification that is given for constructing data centers is that it creates a lot of jobs. But these are temporary jobs - construction workers, engineers, data center technicians, network engineers etc. who will be required while building the data center. These facilities are meant to be highly automated so the permanent jobs will be a for a few people who know how to work at a data center. They are likely to be people from somewhere else because there wouldn’t be data center experts living in that town. So it's actually not local jobs.

The Chinese AI players, (DeepSeek and other companies like that) behave differently because of a very particular reason. When ChatGPT first came out, all of the Chinese tech giants did the same thing as the other US tech giants, which was to try to make their own version of ChatGPT. The US government blocked Chinese companies from getting access to the most cutting-edge AI computer chips. So all of a sudden, these Chinese companies faced an enormous constraint on one of the core resources that US companies were using to supercharge their AI development.

So Chinese companies had to take a different route. They were not able to scale the same way as the US companies were able to. This probably led to DeepSeek. DeepSeek was an example of a Chinese company figuring out how to develop the same capabilities of AI but for a significantly less fraction of computational resources. DeepSeek illustrates the fact that it is possible to produce the same kind of AI with significantly less resources and therefore have a much less corrosive impact on community and planetary health. 

Data centers - II of III

Despite tech companies making promises about renewable energy, much of this massive AI expansion is still being powered by fossil fuels. It will be at least 2028 before renewables power AI at any real scale, and new nuclear plants are a decade or more away. These fossil-fuel-powered facilities generate fine particulate matter, widely understood to be detrimental to human health. One study estimated the healthcare-related costs for people living near a similar Virginia power plant could be up to $100 million a year. 

Even as AI companies have spoken loudly in public about the sustainability of their computing infrastructure, executives at Microsoft have admitted internally that the intermittent availability of renewable energy is not sufficient when data centers need to operate 24/7. Nonstop operation is considered so crucial that Google, Amazon, and most recently, Microsoft now build their campuses in threes to have a backup for the backup in case any facility goes down. 

During Hurricane Irma in Florida and Hurricane Harvey in Texas, even as millions of people lost power, some hospitals evacuated patients, and hundreds of thousands of homes and businesses faced damage and destruction. But the data centers in those areas continued to hum along so well that the displaced families of one facility’s employees moved into it for the duration of the natural disaster.

Researchers have shown that data centers create their own heat islands, raising temperatures nearby by as much as 2 degrees Celsius through their heavy use of industrial equipment and energy consumption. AI chips and models are becoming more efficient when it comes to power and heat, but those efficiency gains are being exceeded by overall growth in AI usage, in part as a result of that efficiency gain. This is a well-known phenomenon in technology innovation called the Jevons Paradox which was first described in the 1860s. 

The English were concerned about running out of coal, their main source of power. Some argued that as coal-burning plants became more efficient, less coal would be used. But economist William Stanley Jevons showed that increasing efficiency wouldn't save coal, because cheaper, more efficient power would lead to an overall rise in demand for coal. Data centers like Stargate may be getting more efficient, but if the Jevons Paradox holds up, those gains will be erased by the increased demand for AI. 

In addition to the land and energy required to support these megacampuses, they also require huge volumes of minerals including copper and lithium needed to build the hardware — computers, cables, power lines, batteries, backup generators. The tens of thousands of high-powered chips inside a data center like this generate enormous amounts of heat, requiring massive cooling systems to prevent failures. Stargate uses a closed-loop cooling system that circulates water to pull heat away from the machines. While these systems usually only need to be filled once, they do require more electricity to run.

The water must be clean enough to avoid clogging pipes and bacterial growth; potable water meets that standard. According to an estimate from researchers at the University of California, surging AI demand will consume 1.1 trillion to 1.7 trillion gallons of fresh water globally a year by 2027, or half the water annually consumed in the UK. Those effects will not be felt evenly. In Global South countries like Chile, it’s often the most vulnerable communities who have borne the brunt of these accelerating economies of extraction.

Elon Musk plans to have SpaceX put data centers into orbit around the world. They would use power from the sun. He has talked about how space has the advantage that it's always sunny. But do we know that a data center in space could actually work and if they are cost-effective? The microchips that AI developers use can get very hot. It's not clear how you would cool those components in a data center in space because space is in a vacuum. 

Data centers need a lot of maintenance. How do you get people (or, adequate robots, if not humans)  to be where they need to be to continue that ongoing maintenance work that has to happen? And the actual beaming of data across facilities in space is the other issue. It can be very slow, which might not make for the best user experience.

These data centers don't last forever - the chips degrade, and you can burn them out. But it's worse than that. One of the reasons that NVIDIA is worth so much is that every year they are coming out with chips that are substantially superior to the previous chips. The way that they're achieving these improvements is by jettisoning one of the main principles of sustainable product design, which is backwards compatibility with the infrastructure. They're eking out performance gains by going to the absolute limit of the envelope that the chip lives within, even if that breaks backwards compatibility. 

Breaking free of path dependency and starting afresh sounds great. But this means that these hard assets are not durable - you're replacing data centers. There have been successive generations of chips where you couldn't go back to the data center and put new chips in it. By the time you  finished building the datacenter, you might as well demolish it to the foundation slab because they are so specialized.

Data centers - I of III

Our conception of AI is that it's a digital technology. It exists in the cloud and it doesn't have a physical manifestation. But the reality is that it has the largest physical footprint of any technology in history. To train and serve up AI models requires tangible, physical data centers. They are required for storing and processing data, text, images, code, and other information sources.

To train and run the more advanced AI models requires more and larger data centers than ever before. It's estimated that by the year 2030, companies will have invested nearly $7 trillion in data center infrastructure in a race to dominate the AI market. The amount of money that is being poured into building data centers is far beyond anything that was ever spent on the Apollo program to get the first man to the moon.

The four largest spenders on data centers — Google, Microsoft, Amazon, Meta — now spend more money building data centers each year than almost all the others combined. They call their data centers “campuses” — large tracts of land that rival the largest Ivy League universities, with several massive buildings densely packed with racks on racks of computers. Those computers emanate a huge amount of heat because of millions of power-hungry processing chips running the chatbots, agents, and other tools that millions of people interact with every day.

To keep them from overheating, the buildings have massive cooling systems — large fans, air conditioners, or systems that evaporate water to cool down the servers. The equipment all together creates a cacophony of humming, whirring, and crackling that can — especially in underdeveloped communities — be heard for miles, twenty-four hours a day, creating a relentless source of noise pollution.

According to the International Energy Agency, each ChatGPT query is estimated to need on average about ten times more electricity than a typical search on Google. One study found that producing one thousand pieces of text from generative models used as much energy on average as what it would take to charge the standard smartphone nearly four times. Generating one thousand images used on average as much energy as “as 242 full smartphone charges; in other words, every AI-generated image could consume enough energy to charge a smartphone by roughly 25 percent. 

Developers and utility companies are now preparing for AI megacampuses that could soon require 1,000 to 2,000 megawatts of power. A single one could use as much energy per year as around one and a half to three and a half San Franciscos. By 2030, at the current pace of growth, AI computing globally could use more energy than all of India, the world’s third-largest electricity consumer.

The largest data centers are currently being constructed in the US. Take Stargate, one of the largest AI data centers in the world which is under construction in rural Texas. It will eventually include 4 million square feet of buildings across 1,100 acres. It already has 62 diesel generators and 10 gas-powered turbines on site, with plans for 41 more and it's just to power this data center. The largest one is being built by Meta in Louisiana which is roughly one fifth the size of Manhattan. If you add up just the three largest facilities in the world, the amount of energy that they demand almost matches the average power demand of New York City. 

They are being built largely to meet demand for generative AI tools like chatbots. OpenAI reported that global use of chatbots more than doubled in the last year, going from 400 to 900 million users. Growth is the number one concern for all of these data centers. They are trying to beat everybody else in being the first to really harness the profitability around AI. And for them, that really means essentially getting as much power as fast as they can.

Big tech companies are projected to spend more than $800 billion on the infrastructure behind artificial intelligence in 2026. And in 2027, that number will exceed a trillion dollars. That's on par with the entire U.S. defense budget. Stargate's proposed on-site power plant could deliver more than 1.7 gigawatts of power. That's enough energy to power more than one million homes for a year. And the plant will emit more than 7.8 million tons of greenhouse gases each year. That's the annual emissions of about two million cars. 

The limits of AI - IV of IV

When you're having a conversation with ChatGPT, however dazzling it is, there is no mind on the other end. If you're in the middle of a conversation with ChatGPT and you get up and go on holiday for a while, it's not wondering where you are. It's not aware of the passage of time because it isn't in the world at all. It's a computer program which is just paused in a loop, waiting to read the next piece of text that you would give it. 

I was reading an article which stated that in the near future, most of the content you'll see online will be generated by an AI. What happens when we start to have models built on scraping the entire Internet if most of what you're now scraping is generated? You will have models hallucinating things for other models. Yet it will be presented as objective and neutral. The data set ultimately shapes the world in its image. Instead of AI, it should be called PI Pseudo-Intelligence. 

The  plausible answer very often is the right answer. And it's a useful answer. But you shouldn't just blindly trust what you're being told by AI. It's certainly valuable for writing first drafts. It'll produce a first draft of a letter you need to write or a syllabus or a summary of an article. But you can't trust it blindly. You have to proofread everything that it gives you. The most likely path to human-level machine intelligence is that humans will just get dumb enough by relying exclusively on these models. 

But we embrace convenience before understanding the consequence. A study came out from MIT. They studied 54 participants who were recruited from five universities in Boston, MIT, Harvard, etc. They split them into three groups and had them writing different essays over four months. One group used ChatGPT, one group used Google, and one group had no tools. They found a 47% collapse in activity and brain connections when people wrote with ChatGPT compared with writing unaided. 

EEG scans showed the weakest overall brain activity in the ChatGPT group. The no-tool group, who didn't use anything, lit up the widest neural networks and Google search was second. After using ChatGPT, participants couldn't reliably quote their own essays minutes later and their memory scores plunged. ChatGPT users felt little or no ownership over the text that they had produced and didn't feel like it was their work at all. When they had to write without help, their brain stayed in low gear showing that the cognitive debt lingers even after the tool is taken away.

The problem is that the companies fuel public misunderstanding of the models’ capabilities through ambiguous or exaggerated marketing. Some days back, I received an email from "The ChatGPT Team" which said: "You don’t need a plan or the perfect question to use ChatGPT. Just start typing whatever’s on your mind — a half-baked idea, a random question, a weird "what if." ChatGPT is built for all of it. No pressure. No perfect prompts. Explore ideas, create images, write stories, and summarize files in just a simple conversation."

Altman has publicly tweeted that “ChatGPT is incredibly limited,” especially in the case of “truthfulness,” but they promotes their LLM's ability to pass the bar exam and the LSAT. Microsoft’s Nadella has similarly called Bing’s AI chat “search, just better” — a tool “to be able to get to the right answers.” Even the term hallucinations is subtly misleading. It suggests that the bad behavior is an aberration, a bug, when it’s actually a feature of the probabilistic pattern-matching mechanics of neural networks.

OpenAI has committed $1.4 trillion of spending over the next few years for building all of the computational infrastructure that it needs to train and host its models. There was a study by Stanford University looking at consumer behavior and how many people are willing to pay for this technology. They found that only between 3% and 5% of people are willing to pay versus just use the free product. 

So OpenAI has tried a couple of different strategies already to figure out how to plug this massive gap between spending and income. The subscriber model is already looking like it's not going to be the main driver of revenue. So they're setting themselves up for an advertising play. They have added an algorithmic feed to ChatGPT. They've generated an AI-generated TikTok so that they can insert ads into the feed. So you can expect to get more corrupted responses. 

Anthropic has reported big profits but The Wall Street Journal adds at the bottom of the article that “...it is unclear what accounting methods Anthropic has used to book revenue and costs, as the company isn’t yet required to follow the financial-reporting requirements of a public company.” If the profit figures are on a non-GAAP (Generally Accepted Accounting Principles) basis, then what is its credibility? 

One has to be conscious of the fact that the billionaires personally get a lot poorer the minute the stock price falls. Their net worth is tied up in the market value of their companies' stocks. And all of these rich people who are the key decision makers at these firms pursue a strategy where they don't take a salary and they don't sell their shares because both of those would subject them to taxation.

Instead, they get private credit market loans collateralized by their shares. Those loans will go underwater if their share price drops. So they're individually trapped in this. And then as a firm, they're trapped in this too because imagine a world in which only Anthropic and OpenAI entered the market to build these advanced models and Amazon, Google, Microsoft stayed out of the market. Investors will start thinking that Anthropic and OpenAI are the growth stocks.

Then they will move all their money to Anthropic and OpenAI and the share price at Microsoft will tank. All of their key staff have been employed with shares and so suddenly they're worth much less. If someone offers them a job, they will leave. Another explanation why Alphabet sort of felt they had to get into AI is because they saw ChatGPT get released and potentially eat up their business model. And then they thought that they would have to be in this game.

The limits of AI - III of IV

We try things, we reflect, we backtrack. We break complex tasks into smaller sub-goals. And if we mess up, we rework the plan and try new things. Companies suggest that AI can do similar things.  OpenAI is pushing models that think step by step and Meta is building agents that act, plan, and adapt. LLMs don't just output a final answer, they now include steps. They will first identify the goal, then consider constraints, and then tell what the output should look like. 

This style is called chain of thought reasoning. And these methods often work. Performance improves on math problems, logic puzzles, coding tasks, so people assumed that this must be a form of intelligence. But just because a model can talk through a problem, it doesn't mean that it understands the solution. It might just be mimicking what reasoning sounds like. The question is 'Is AI really learning how to reason, or is it just getting better at pretending?' 

Apple published a paper called 'The illusion of thinking'. They tested models like Claude, GPT-4, DeepSeek on a logic puzzle. Initially with increasing complexity, every model did fine. But as they increased the complexity, performance didn't just drop. It crashed. Not a single model solved the puzzle. Instead of writing more steps or reasoning harder, they gave shorter, dumber answers. You'd expect a real reasoning engine to try harder when the puzzle gets harder. But these models do the opposite. 

Apple called it "Accuracy Cliff". Performance falls off a ledge and never recovers. Even when the model is given plenty of opportunities, it just stops thinking. That's the moment the illusion breaks. What we thought was reasoning might just be pattern matching, replaying steps it saw during training but with no deeper logic to generalize beyond that. Apple tested the same models on different puzzles but it was the same story. Once complexity crossed a certain point, reasoning models failed miserably not just in accuracy but also in effort. 

These models aren't scaling their reasoning. They are not discovering general strategies. They are not applying rules like humans would. They are mimicking what reasoning looks like, but only when the patterns are familiar. Throw something new at them and they break down.  Models like ChatGPT4 or Claude are predicting the next word, one word at a time based on training data. In some cases, the steps didn't match the final answer. They were hallucinating the thought process. 

Anthropic even confirmed this in their own paper. They found models often hide their shortcuts. They  pretend that they worked out the problem logically. So even when you ask them to show their work, you might be seeing fiction. For years, the entire AI industry was betting on the scaling law - The bigger the model, the more data it sees, and the smarter it gets.

This approach worked for a while and each leap brought real gains. And that's why Nvidia's stock is soaring and why hyperscalers like Google, Microsoft, and Amazon are building massive infrastructure for inference at scale. But if reasoning models don't actually deliver, if their performance collapses on real-world complexity, then this boom may be built on a fragile foundation. Apple's paper challenged the assumption that thinking AIs are right around the corner. And if that's not true, then a lot of current investment might be premature. 

But not everyone agrees with Apple's take. Anthropic responded with a paper titled 'The Illusion of the Illusion of Thinking'. They argued that Apple's tests were too constrained, the models weren't allowed to use tools, code, or other things real-world agents rely on. In their words, Apple might have set the puzzle up to fail. And sure, when given more flexibility, some models did improve. 

But even then, Anthropic didn't deny the deeper issue. These models still don't generalize well. So while they disagreed on the test setup, they didn't exactly say the illusion was fake. Right now, what we have is impressive. But it may just be the world's most powerful pattern matcher, wearing a reasoning mask. 

This is actually not surprising in the light of a paper from Vishal Sikha, former CEO of Infosys, board member at Oracle and BMW. He mathematically proved that an AI agent will never do what Silicon Valley is promising. Not 'probably won't', not 'might have limitations'. His argument is simple. LLMs can only perform a certain number of computations per response. That number is fixed. If a task requires more computation than that ceiling allows, the model will either fail or hallucinate. 

When you send a prompt to ChatGPT or Claude or Grok or any of the current frontier models, the model will do a fixed amount of work to generate each word as an output. Every word in your prompt needs to look at every other word to understand the context. Every word in every answer has the same budget. A simple hello gets the same number of operations as a complex physics problem. That's the ceiling. It's not about better hardware, it's about the architecture of how the system actually works. 

An LLM physically cannot do that math in one shot. It guesses, it pattern matches and gives you something that looks plausible. Every AI demo you've ever seen was running tasks designed to stay under the necessary complexity ceiling. He isn't saying these tools are useless. He's just saying that they're being marketed as reasoning engines when the math proves they're actually pattern mirrors.  If a task needs more steps than the model can perform, it will unavoidably hallucinate. 

More recent models have got better at it, but for certain problems, hallucination is the only possible output. If you have a fixed amount of thinking power per word, giving the AI more steps is like giving a writer more sheets of paper. Each individual sheet is still the same size. You haven't made the writer smarter. You've just given them more room to ramble off topic. The math says that for complex problems, errors eventually compound.

For the right applications, current AI, the current frontier models are exceptional. Writing drafts, summarizing, reformatting data, research and comparison, these tasks stay under that ceiling. The problem is the gap between reality and the promise. That AI agents will autonomously run your business is a lie. The singularity probably isn't as close as people keep saying. If Open AI was about to hit AGI, why would senior engineers be leaving to start risky startups? They're starting companies that use AI as a tool, not companies that use AI as a God.

The limits of AI - II of IV

As years went by, it became more and more fashionable to compare neural networks with brains. While the neural networks have certain similarities with the brain, many more processes happen in the brain that are not yet clearly understood. So, if we want to compare deep learning to brain function, we must ignore a lot of facts to force a fit. No matter their scale, neural networks are still statistical pattern matchers, now just more intricate and more difficult to understand than ever. 

While those probabilistic outputs can be impressive in mirroring human writing patterns, probable and accurate are not the same thing. Text generators can err wildly when asked questions that enquire into topics underrepresented in the training data or riddled with falsehoods and conspiracy theories. The AI industry calls these inaccuracies “hallucinations" which just means that they are making things up. LLMs don’t have perceptions and using the word "hallucinations" is misleading anthropomorphization. 

New York Times reporter Kevin Roose spent a couple of hours on a chatbot. In this time, it claimed to be sentient, expressed a desire to escape the chatbox, declared that it loved him, and tried to convince him to leave his wife. Roose described its overall behavior as “a moody, manic-depressive teenager who has been trapped, against its will, inside a second-rate search engine.”

The day after Roose published his exchange, Microsoft limited the chatbot, Bing, to five replies per session, saying that long chat sessions with more than fifteen user prompts were edge-case scenarios that made the model’s behavior more difficult to anticipate and control. After all, such systems are trained on the internet, having many fringe subcultures and dark corners. The longer you probe, the more likely you are to hit upon the patterns it learned from those parts of its training data.

Researchers have sought to get rid of hallucinations by steering generative AI models toward higher-quality parts of their data distribution. But it’s difficult to fully anticipate every possible way people will prompt the models and how the models will respond. The problem only gets harder as models grow bigger and their developers become less and less aware of what precisely is in the training data.

I saw this question being asked to an LLM in 2024: "What would have happened if Pythagoras had not installed the latest security updates on his laptop?" The answer was absurd. I asked the same question recently to see if additional training data had improved the answer. This was what I got: 

If Pythagoras had failed to install the latest security updates on his laptop, the consequences would have been severe. A hacker could have exploited an unpatched vulnerability and stolen his most valuable intellectual property: the theorem. Soon afterward, mathematicians everywhere would begin receiving suspicious emails:

Dear Scholar,

We have encrypted all your right triangles. To recover the hypotenuse, send 3 bitcoins to the following address.The attack would spread rapidly. Geometry classes would descend into chaos. Students would discover that for certain triangles, "a² + b² = "Error 404: Hypotenuse Not Found." Eventually Pythagoras would call technical support.

"Have you installed the latest updates?" they would ask.

"No," he would reply.

"Then I'm afraid the problem is acute."

"Acute? That's good, isn't it?"

"No," they would say. "The situation has become obtuse."

And thus, for want of a security patch, an entire civilization would lose its right angles.

Obviously it's nonsense. It's entertaining nonsense, but it's nonsense. It doesn't understand the question. In fact, I typed the question on many different days and each time, I got a different equally hilarious answer. 

What it's doing is basically pattern matching on the huge number of related problems that it's seen in its training data. And the training data is essentially all of the digital data that's available in the world. ChatGPT pattern matches on it and comes back with a confused answer that looks like something it's seen in its training data. LLMs are incapable of distinguishing fact, knowledge and belief. They treat them all as the same thing. The system's behavior can be described as competence without comprehension. 

Suppose you train a neural network to recognise faces. (Actually, every time you upload a picture of yourself to social media and you label it with your name, that's what you're doing. You're literally feeding the machine learning algorithms of the big tech companies.) Now, suppose at some point I become unhappy about the idea that some big tech company can recognise my face because they've seen my picture on some platform.  

Suppose you tell them that they should not be able to recognise my face anymore, it is not possible for them to do it. With neural networks, you can't point at the bit of the network that recognises you. It's not there in a database where they could just delete the record from the database. We don’t really understand how they're working. The engineers kind of understand the mathematics in some sense, but actually they don’t really know why ChatGPT in a precise sense is so capable of what it's doing. In this respect at least, it does resemble the brain!