The limits of AI - IV of IV

When you're having a conversation with ChatGPT, however dazzling it is, there is no mind on the other end. If you're in the middle of a conversation with ChatGPT and you get up and go on holiday for a while, it's not wondering where you are. It's not aware of the passage of time because it isn't in the world at all. It's a computer program which is just paused in a loop, waiting to read the next piece of text that you would give it. 

I was reading an article which stated that in the near future, most of the content you'll see online will be generated by an AI. What happens when we start to have models built on scraping the entire Internet if most of what you're now scraping is generated? You will have models hallucinating things for other models. Yet it will be presented as objective and neutral. The data set ultimately shapes the world in its image. Instead of AI, it should be called PI Pseudo-Intelligence. 

The  plausible answer very often is the right answer. And it's a useful answer. But you shouldn't just blindly trust what you're being told by AI. It's certainly valuable for writing first drafts. It'll produce a first draft of a letter you need to write or a syllabus or a summary of an article. But you can't trust it blindly. You have to proofread everything that it gives you. The most likely path to human-level machine intelligence is that humans will just get dumb enough by relying exclusively on these models. 

But we embrace convenience before understanding the consequence. A study came out from MIT. They studied 54 participants who were recruited from five universities in Boston, MIT, Harvard, etc. They split them into three groups and had them writing different essays over four months. One group used ChatGPT, one group used Google, and one group had no tools. They found a 47% collapse in activity and brain connections when people wrote with ChatGPT compared with writing unaided. 

EEG scans showed the weakest overall brain activity in the ChatGPT group. The no-tool group, who didn't use anything, lit up the widest neural networks and Google search was second. After using ChatGPT, participants couldn't reliably quote their own essays minutes later and their memory scores plunged. ChatGPT users felt little or no ownership over the text that they had produced and didn't feel like it was their work at all. When they had to write without help, their brain stayed in low gear showing that the cognitive debt lingers even after the tool is taken away.

The problem is that the companies fuel public misunderstanding of the models’ capabilities through ambiguous or exaggerated marketing. Some days back, I received an email from "The ChatGPT Team" which said: "You don’t need a plan or the perfect question to use ChatGPT. Just start typing whatever’s on your mind — a half-baked idea, a random question, a weird "what if." ChatGPT is built for all of it. No pressure. No perfect prompts. Explore ideas, create images, write stories, and summarize files in just a simple conversation."

Altman has publicly tweeted that “ChatGPT is incredibly limited,” especially in the case of “truthfulness,” but they promotes their LLM's ability to pass the bar exam and the LSAT. Microsoft’s Nadella has similarly called Bing’s AI chat “search, just better” — a tool “to be able to get to the right answers.” Even the term hallucinations is subtly misleading. It suggests that the bad behavior is an aberration, a bug, when it’s actually a feature of the probabilistic pattern-matching mechanics of neural networks.

OpenAI has committed $1.4 trillion of spending over the next few years for building all of the computational infrastructure that it needs to train and host its models. There was a study by Stanford University looking at consumer behavior and how many people are willing to pay for this technology. They found that only between 3% and 5% of people are willing to pay versus just use the free product. 

So OpenAI has tried a couple of different strategies already to figure out how to plug this massive gap between spending and income. The subscriber model is already looking like it's not going to be the main driver of revenue. So they're setting themselves up for an advertising play. They have added an algorithmic feed to ChatGPT. They've generated an AI-generated TikTok so that they can insert ads into the feed. So you can expect to get more corrupted responses. 

Anthropic has reported big profits but The Wall Street Journal adds at the bottom of the article that “...it is unclear what accounting methods Anthropic has used to book revenue and costs, as the company isn’t yet required to follow the financial-reporting requirements of a public company.” If the profit figures are on a non-GAAP (Generally Accepted Accounting Principles) basis, then what is its credibility? 

One has to be conscious of the fact that the billionaires personally get a lot poorer the minute the stock price falls. Their net worth is tied up in the market value of their companies' stocks. And all of these rich people who are the key decision makers at these firms pursue a strategy where they don't take a salary and they don't sell their shares because both of those would subject them to taxation.

Instead, they get private credit market loans collateralized by their shares. Those loans will go underwater if their share price drops. So they're individually trapped in this. And then as a firm, they're trapped in this too because imagine a world in which only Anthropic and OpenAI entered the market to build these advanced models and Amazon, Google, Microsoft stayed out of the market. Investors will start thinking that Anthropic and OpenAI are the growth stocks.

Then they will move all their money to Anthropic and OpenAI and the share price at Microsoft will tank. All of their key staff have been employed with shares and so suddenly they're worth much less. If someone offers them a job, they will leave. Another explanation why Alphabet sort of felt they had to get into AI is because they saw ChatGPT get released and potentially eat up their business model. And then they thought that they would have to be in this game.

The limits of AI - III of IV

We try things, we reflect, we backtrack. We break complex tasks into smaller sub-goals. And if we mess up, we rework the plan and try new things. Companies suggest that AI can do similar things.  OpenAI is pushing models that think step by step and Meta is building agents that act, plan, and adapt. LLMs don't just output a final answer, they now include steps. They will first identify the goal, then consider constraints, and then tell what the output should look like. 

This style is called chain of thought reasoning. And these methods often work. Performance improves on math problems, logic puzzles, coding tasks, so people assumed that this must be a form of intelligence. But just because a model can talk through a problem, it doesn't mean that it understands the solution. It might just be mimicking what reasoning sounds like. The question is 'Is AI really learning how to reason, or is it just getting better at pretending?' 

Apple published a paper called 'The illusion of thinking'. They tested models like Claude, GPT-4, DeepSeek on a logic puzzle. Initially with increasing complexity, every model did fine. But as they increased the complexity, performance didn't just drop. It crashed. Not a single model solved the puzzle. Instead of writing more steps or reasoning harder, they gave shorter, dumber answers. You'd expect a real reasoning engine to try harder when the puzzle gets harder. But these models do the opposite. 

Apple called it "Accuracy Cliff". Performance falls off a ledge and never recovers. Even when the model is given plenty of opportunities, it just stops thinking. That's the moment the illusion breaks. What we thought was reasoning might just be pattern matching, replaying steps it saw during training but with no deeper logic to generalize beyond that. Apple tested the same models on different puzzles but it was the same story. Once complexity crossed a certain point, reasoning models failed miserably not just in accuracy but also in effort. 

These models aren't scaling their reasoning. They are not discovering general strategies. They are not applying rules like humans would. They are mimicking what reasoning looks like, but only when the patterns are familiar. Throw something new at them and they break down.  Models like ChatGPT4 or Claude are predicting the next word, one word at a time based on training data. In some cases, the steps didn't match the final answer. They were hallucinating the thought process. 

Anthropic even confirmed this in their own paper. They found models often hide their shortcuts. They  pretend that they worked out the problem logically. So even when you ask them to show their work, you might be seeing fiction. For years, the entire AI industry was betting on the scaling law - The bigger the model, the more data it sees, and the smarter it gets.

This approach worked for a while and each leap brought real gains. And that's why Nvidia's stock is soaring and why hyperscalers like Google, Microsoft, and Amazon are building massive infrastructure for inference at scale. But if reasoning models don't actually deliver, if their performance collapses on real-world complexity, then this boom may be built on a fragile foundation. Apple's paper challenged the assumption that thinking AIs are right around the corner. And if that's not true, then a lot of current investment might be premature. 

But not everyone agrees with Apple's take. Anthropic responded with a paper titled 'The Illusion of the Illusion of Thinking'. They argued that Apple's tests were too constrained, the models weren't allowed to use tools, code, or other things real-world agents rely on. In their words, Apple might have set the puzzle up to fail. And sure, when given more flexibility, some models did improve. 

But even then, Anthropic didn't deny the deeper issue. These models still don't generalize well. So while they disagreed on the test setup, they didn't exactly say the illusion was fake. Right now, what we have is impressive. But it may just be the world's most powerful pattern matcher, wearing a reasoning mask. 

This is actually not surprising in the light of a paper from Vishal Sikha, former CEO of Infosys, board member at Oracle and BMW. He mathematically proved that an AI agent will never do what Silicon Valley is promising. Not 'probably won't', not 'might have limitations'. His argument is simple. LLMs can only perform a certain number of computations per response. That number is fixed. If a task requires more computation than that ceiling allows, the model will either fail or hallucinate. 

When you send a prompt to ChatGPT or Claude or Grok or any of the current frontier models, the model will do a fixed amount of work to generate each word as an output. Every word in your prompt needs to look at every other word to understand the context. Every word in every answer has the same budget. A simple hello gets the same number of operations as a complex physics problem. That's the ceiling. It's not about better hardware, it's about the architecture of how the system actually works. 

An LLM physically cannot do that math in one shot. It guesses, it pattern matches and gives you something that looks plausible. Every AI demo you've ever seen was running tasks designed to stay under the necessary complexity ceiling. He isn't saying these tools are useless. He's just saying that they're being marketed as reasoning engines when the math proves they're actually pattern mirrors.  If a task needs more steps than the model can perform, it will unavoidably hallucinate. 

More recent models have got better at it, but for certain problems, hallucination is the only possible output. If you have a fixed amount of thinking power per word, giving the AI more steps is like giving a writer more sheets of paper. Each individual sheet is still the same size. You haven't made the writer smarter. You've just given them more room to ramble off topic. The math says that for complex problems, errors eventually compound.

For the right applications, current AI, the current frontier models are exceptional. Writing drafts, summarizing, reformatting data, research and comparison, these tasks stay under that ceiling. The problem is the gap between reality and the promise. That AI agents will autonomously run your business is a lie. The singularity probably isn't as close as people keep saying. If Open AI was about to hit AGI, why would senior engineers be leaving to start risky startups? They're starting companies that use AI as a tool, not companies that use AI as a God.

The limits of AI - II of IV

As years went by, it became more and more fashionable to compare neural networks with brains. While the neural networks have certain similarities with the brain, many more processes happen in the brain that are not yet clearly understood. So, if we want to compare deep learning to brain function, we must ignore a lot of facts to force a fit. No matter their scale, neural networks are still statistical pattern matchers, now just more intricate and more difficult to understand than ever. 

While those probabilistic outputs can be impressive in mirroring human writing patterns, probable and accurate are not the same thing. Text generators can err wildly when asked questions that enquire into topics underrepresented in the training data or riddled with falsehoods and conspiracy theories. The AI industry calls these inaccuracies “hallucinations" which just means that they are making things up. LLMs don’t have perceptions and using the word "hallucinations" is misleading anthropomorphization. 

New York Times reporter Kevin Roose spent a couple of hours on a chatbot. In this time, it claimed to be sentient, expressed a desire to escape the chatbox, declared that it loved him, and tried to convince him to leave his wife. Roose described its overall behavior as “a moody, manic-depressive teenager who has been trapped, against its will, inside a second-rate search engine.”

The day after Roose published his exchange, Microsoft limited the chatbot, Bing, to five replies per session, saying that long chat sessions with more than fifteen user prompts were edge-case scenarios that made the model’s behavior more difficult to anticipate and control. After all, such systems are trained on the internet, having many fringe subcultures and dark corners. The longer you probe, the more likely you are to hit upon the patterns it learned from those parts of its training data.

Researchers have sought to get rid of hallucinations by steering generative AI models toward higher-quality parts of their data distribution. But it’s difficult to fully anticipate every possible way people will prompt the models and how the models will respond. The problem only gets harder as models grow bigger and their developers become less and less aware of what precisely is in the training data.

I saw this question being asked to an LLM in 2024: "What would have happened if Pythagoras had not installed the latest security updates on his laptop?" The answer was absurd. I asked the same question recently to see if additional training data had improved the answer. This was what I got: 

If Pythagoras had failed to install the latest security updates on his laptop, the consequences would have been severe. A hacker could have exploited an unpatched vulnerability and stolen his most valuable intellectual property: the theorem. Soon afterward, mathematicians everywhere would begin receiving suspicious emails:

Dear Scholar,

We have encrypted all your right triangles. To recover the hypotenuse, send 3 bitcoins to the following address.The attack would spread rapidly. Geometry classes would descend into chaos. Students would discover that for certain triangles, "a² + b² = "Error 404: Hypotenuse Not Found." Eventually Pythagoras would call technical support.

"Have you installed the latest updates?" they would ask.

"No," he would reply.

"Then I'm afraid the problem is acute."

"Acute? That's good, isn't it?"

"No," they would say. "The situation has become obtuse."

And thus, for want of a security patch, an entire civilization would lose its right angles.

Obviously it's nonsense. It's entertaining nonsense, but it's nonsense. It doesn't understand the question. In fact, I typed the question on many different days and each time, I got a different equally hilarious answer. 

What it's doing is basically pattern matching on the huge number of related problems that it's seen in its training data. And the training data is essentially all of the digital data that's available in the world. ChatGPT pattern matches on it and comes back with a confused answer that looks like something it's seen in its training data. LLMs are incapable of distinguishing fact, knowledge and belief. They treat them all as the same thing. The system's behavior can be described as competence without comprehension. 

Suppose you train a neural network to recognise faces. (Actually, every time you upload a picture of yourself to social media and you label it with your name, that's what you're doing. You're literally feeding the machine learning algorithms of the big tech companies.) Now, suppose at some point I become unhappy about the idea that some big tech company can recognise my face because they've seen my picture on some platform.  

Suppose you tell them that they should not be able to recognise my face anymore, it is not possible for them to do it. With neural networks, you can't point at the bit of the network that recognises you. It's not there in a database where they could just delete the record from the database. We don’t really understand how they're working. The engineers kind of understand the mathematics in some sense, but actually they don’t really know why ChatGPT in a precise sense is so capable of what it's doing. In this respect at least, it does resemble the brain! 

The limits of AI - I of IV

An AI managed to solve a problem from the 1940s from the Hungarian mathematician, Paul Erdős, which was called the unit distance problem. He produced a formula for the problem and AI came up with a proof that disproved his formula. So this isn't just confirming something that others have thought about before but it came up with a proof that this formula was wrong. Moreover, it came up with a method to prove this, that no one in that field of mathematics has ever thought of. Mathematicians a year ago didn't think that AI could ever do this.

An AI is a pattern recognition device. We don't exactly know how it works because of the complexity of how it arrives at certain conclusions. We are overwhelmed by the sheer number of calculations and the sheer number of data. So we cannot simply watch in real time how it arrives at its results. But it's still a pattern recognition device. But if pattern recognition allows us to prove something that we sort of put in the realm of logic or creativity or genius or intuition, that's quite a serious development. 

You can turn to any discussion topic. And the AI will be very confident and fluent and seemingly knowledgeable. It seems dazzling making people very excited. But there are some very big caveats about this technology. One of the ideas that people have about AI is that it's a kind of flawless super brain. But actually, it's not right a lot of the times. 

When people talk about AI, they often mean large language models (LLMs). But LLMs are just one small part of a much bigger picture. There are a huge range of different applications of the technology. Take Google Translate, for example, that nowadays is driven by neural network technology. It still makes grave mistakes that a human never would.

Consider the phrase “I stood in the pen" Most people imagine me in an enclosure of some kind. They do not imagine that I am inside a writing “pen” which sounds a bit ridiculous. However, as of today, if you type “I stood in the pen” into Google Translate to convert the sentence into Malayalam, it interprets "pen" as the writing instrument. It doesn’t seem to “get” which kind of pen the phrase refers to.

Now, if you type “The cow is in the pen,” Google gets it right, correctly picking the Malayalam word for cattle enclosure. This is because Google uses the context (“cow”) to interpret correctly the word “pen” (whereas “I” didn’t hint at the farm context). How can AI seem to be getting smarter by the day yet still make such silly mistakes?”

It doesn't know what the truth is. It's trying to produce plausible answers to the questions that you tell it, which may be correct. But they may well be incorrect. When you ask a question, it's not thinking "What is the right answer to this question?" There's no thinking at all. It's been trained on a huge amount of training data. And what it's trying to do is to produce the most plausible sounding answer. 

Automated translation models are built by scanning thousands of text documents written in multiple languages. The computer automatically searches these documents for ways in which words are most often translated. For instance, the computer learns that when the word “pen” appears in English with the word “cow” nearby, the corresponding Malayalam text very often uses the word “thozhuthilaanu” (in the cattle enclosure). So, if this happens often enough, a rule is added to the model to output “thozhuthilaanu” for “pen,” provided that “cow” appears nearby.

As of today, machines learn to translate by finding common patterns in previous translations done by people. But this doesn’t give them the broad knowledge of the world required to truly excel at the tasks. So, machines may dupe us for a while because they’re good pretenders, but at some point, they end up making silly mistakes a human would never make.  

This is essentially how an LLM like ChatGPT also works. For making an answer, it will make a prediction about the likeliest next word to appear, and then it will repeat that process and that will generate your response. They learn the probability that a word will appear given a particular prompt. To predict which word will occur given a particular sequence of words, you need unimaginable quantities of training data and unimaginable quantities of compute power. 

This is what we knew about GPT-3 which was the breakthrough large language model: 

• 175 billion parameters. A parameter in a neural network is basically either an individual neuron or a connection between them. Each of those takes about four bytes to store.
• For the training data, it required 500 billion words. You're talking about thousands of human lifetimes to read that number of words.
• And to process all of that data, you would take thousands upon thousands of years on a regular desktop computer. So what you need is AI supercomputers with  GPUs, typically provided by NVIDIA, running for weeks. 

So scale is important to be able to make language models work. It just doesn't work if you don't have sufficient training data and sufficient compute power and neural networks to be able to make it work.

The Intelligence Curse - III of III

Pope Leo XIV wrote a deeply considered encyclical on AI called Magnifica Humanitas. It discusses all the major issues - the concentration of power in the hands of a few corporations developing this technology, the datafication and quantification of human existence, the exploitation of labor, resource intensive data centers and the hidden human supply chains that exist behind the production of this technology. 

It discusses the corrosion of economic opportunity, about the need to counter the way that AI is spreading misinformation and the idea that AI is essentially threatening to turn the world into a new colonial world order. He calls out transhumanism and the belief that somehow AI is meant to perfect humans. The document says that humanity flourishes, not despite limitations, but often through them. It is precisely within our limitations that compassion, generosity, spiritual experience etc. find a place.    

He warns that the problem is not that artificial intelligence will become too human-like, but that human beings will become like machines. The encyclical says that modern society is shaped by a technocratic paradigm, that prizes efficiency, control, optimization and power over human dignity, reducing people to functions. He sees that what's behind the rush toward the post-human path is the way that we view difficulties right now - weakness, dependence etc. are problems to be engineered away. The encyclical has many quotable observations some of which are: 

• More power does not necessarily imply something better.
• A more moral AI is not enough if that morality is determined by a few. 
• ... technical power, if left unbalanced, does not make us more capable; it makes us more isolated and more vulnerable to being dominated and excluded.
• For an algorithm, an error is a flaw to be corrected; for a person, however, an error can be a catalyst for profound change.

Not surprisingly, the Pope's words have already been ridiculed and disdained by those in whose interest it is to put such questions to the side. United States Secretary of Interior in the US administration, Doug Burgum, said in a television interview that it is not part of the 'role of the Pope' for Leo to speak to what he did. But what Burgum represents in saying this is Washington's deference to Silicon Valley. 

People keep telling us, life is not about the destination. Life is about the journey. But when we think about AI, we only think about the destination. We are amazed by its remarkable ability to write the book, paint the painting, solve the problem. But we forget the importance of doing the work oneself. A person is smarter, better at problem solving, more resourceful, not because a book exists with her ideas in it, but because she wrote it. It's the same for love, friendships, conflict. We forget that we give up certain skills or abilities because of technology. 

The tech bros that we're told to trust, have in many instances shown themselves to be creepily cold to any aspect of humanity that is not quantifiable and consumable. Some of them have shown themselves to be in their own personal lives, eerily utopian, trying to find ways to immortality by uploading their brains to the cloud or rejuvenating themselves with blood transfusions from younger men. And some have shown themselves to be just as creepily dystopian, building elaborate bunkers for themselves and to be ready in case their mantra of move fast and break things creates a mess. 

Take the example of Elon Musk. He thinks of human beings only in terms of hardware and software. Biological and digital systems had begun to merge, forming what he called “cybernetic collectives” of networked intelligence. The cybernetic collective, like any networked system, could be infected. So he had to take control of the interfaces where this fusion was taking place — from social media to neural implants to artificial intelligence — to ensure that the right kind of cyborgs were being made. 

“Most humans have very limited firewalls,” he elaborated elsewhere, “so are easily programmed.” He called George Soros a “system hacker” who was funding a “fake asylum-seeker nightmare”. Once, he was warning us that AI has no empathy for the human species. Suddenly, he changed his mind about the importance of empathy. He recently said that empathy is the fundamental weakness of Western civilization.

It increasingly appears to him that humanity is a biological bootloader for digital superintelligence. He is now telling us that Homo sapiens will be remembered as nothing more than the primitive computer hardware on which some smart people like himself developed higher, smarter programs. 

Many tech lords believe that humanity must fuse itself with AI and other technologies so that we become cyborgs that colonize the cosmos. They sound like overexcited children as they speak of their dream to distill the essence of what it means to be human, our cognition, our consciousness, our emotions, into data blocks to be downloaded as binary code onto some device. Larry Page, the co-founder of Google, said that because flesh and blood doesn't travel well in space, we humans must become cyborgs if we are to meet our potential as conquerors of the universe, of the cosmos.

It would be a mistake to think that this is a harmless fantasy. They argue that anybody who tries to impede this metamorphosis of us into cyborgs is a villain that must be put down.  Peter Thiel is a great example of this mindset. He sees anyone daring to impede his company's use of tech - political movements, government regulation, communities, maybe the Pope - as the handmaidens of the Antichrist. Marc Andreeson refers to charity as a waste of resources that big tech could be using profitably to help humans become cyborgs. 

Silicon Valley points out things of which we should rightly be afraid, but then offers the alternative of saying we need to hand limitless power to the smart people who can innovate their way to the future. Vast economic inequality, potential mass unemployment, psychological degeneration , these are all just part of the price of progress if they happen, they say. The solution, Silicon Valley tells us, is to head even more quickly in the direction we're headed with even less thought to where we're going and who's taking us there.

The Intelligence Curse - II of III

Unfortunately, the mass automation of jobs is not a new occurrence; those in power have been attempting to do away with those pesky workers and all the money they require to live for quite a long time. AI systems provide a new way to try to minimize those costs. While executives suggest that AI is going to be a labor-saving device, in reality it is intended to devalue labor by threatening workers with technology that can supposedly do their job at a fraction of the cost. 

Virtually anything in banking could be done by AI. A single software update by Anthropic got rid of an entire strata of the legal profession. Once everybody was saying you should become a programmer. The former chief of Google, Eric Schmidt, said that they won't need programmers anymore. They only need them for supervising, for checking, for systems design. But AI will soon be able to do that. They're advising you to study things like English literature, history, the classics. 

There was a report a few months ago by the American research company, Citrini, which came up with a very alarming scenario that AI would take over more jobs than we thought. It will be the first technology ever that destroyed more jobs than it creates. All the other technological innovations of the past, electricity, the steam engine, the internet, created more jobs than it destroyed. But there is another scenario. 

The AI industry pays a pittance to many people to train their AI models. There is work called content moderation - cleaning the data that they use to feed into their models. This category of work is generally called data work. It is literally the work that the AI industry needs to make their technologies exist. This kind of work has become the number 4 fastest growing job in the US, according to a Linkedin report from earlier this year.

In the vast majority of cases, AI is not going to replace your job. But it will make your job a lot shittier.  It will turn work into a gig work version so that they can take your expertise and your knowledge. There are extremely highly educated individuals with college degrees, master's degrees, PhD degrees, law degrees, and also people who have been working within different industries as scientists or lawyers or doctors for years who are now turning to such data work as another job. 

These are incredibly awful jobs to work. In one podcast, there was a story about one woman who was an Ivy League PhD graduate who then applied to 200 jobs and didn't get any of them, so she ended up in data work. She described an extremely piecemeal, anxious process where she's drip-fed this data work. An email will arrive in her inbox informing her that a project is available for a certain amount of money and asking her if she wants it.

You never know when it's going to arrive. You never know when it's going to go away. And it pits different workers against each other because the people who claim the work fastest are the ones that actually get the opportunity. What exactly are these PhDs doing? Companies will tell you that their models are going to arrive at PhD level intelligence. They hire PhD level workers to try and teach this knowledge to the models so that they have the trappings of PhD level intelligence. 

But the data work is so poorly organized that the AI models are actually not very effectively capturing their expertise. For example, a PhD in philosophy would be overseeing other PhDs in math, chemistry, biology etc. None of them would be actually evaluating tasks within their domain of expertise. So actually, AI is not taking on the boring jobs as had been claimed. It's taking on the interesting jobs like design and people are stuck with the drudgery of bureaucracy. 

Why would anyone then pay tons of money to get training in college, in a PhD program when this is what is awaiting them on the other side? It's a very naive and pretentious thing from a few people to believe that 99% of the people have no desire in life, that nobody else (other than them) has a desire to to do things that are beautiful and smart. If it's true that such machines are going to change the workplace in a way that's going to put a lot of people out of work, even if we give them money for free, they're not going to be happy.

MIT Institute Professor Darren Asimoglu, who won the Nobel Prize for Economics in 2024, said that if Silicon Valley gets what it wants and successfully uberizes most knowledge work, then the kind of inequality that we would experience is something that we would have never seen before, where most workers are sidelined for meaningful work and only a few companies actually employ most workers as well. 

No wonder several speeches praising AI at commencement ceremonies across the US were booed by the students. Recent graduates at the University of Central Florida and the University of Arizona booed speakers who compared the advent of AI to the Industrial Revolution and the development of the laptop and smartphone. They have revealed a disconnect between the executives championing AI and students.  

A national survey conducted for NBC News earlier this year polled 1,000 registered voters in the US and found only 26% view AI positively and 46% view it negatively. AI scored worse than US Immigration and Customs Enforcement (ICE), Donald Trump and Kamala Harris on the same poll, but better than the Democratic party and Iran. Anger against AI is growing  – from communities protesting against data centers powering the AI boom, to workers disputing their CEOs’ claims that AI can, effectively, replace them.

The Intelligence Curse - I of III

In recent years, the concept of universal basic income (UBI) has gained significant attention, not from grassroots community organizations but from some of the most powerful figures in the technology sector. Prominent advocates like Elon Musk and Sam Altman argue that UBI is necessary to address the economic disruptions caused by artificial intelligence (AI) and automation. They present UBI as a way to ensure that the benefits of AI are distributed across society, not just concentrated in the hands of a

The Intelligence Curse by Luke Drago and Rudolf Laine is an AI-focused article about how advanced artificial intelligence could change economic incentives in ways that reduce the importance of ordinary humans in society. The article is not about individuals becoming too smart, but about intelligence becoming a fully substitutable economic resource. Once AI systems can perform most or all economically valuable tasks better and cheaper than humans, the importance of human labor begins to disappear.

Today, governments and corporations depend on people. States need citizens for tax revenue, economic productivity, and political legitimacy; companies need workers and consumers to generate profits. Because of this dependence, institutions have to invest in people by various means — through education, infrastructure, wages, and welfare. This mutual dependence gives ordinary individuals bargaining power and drives both democracy and capitalism.

The "intelligence curse" emerges when this dependence breaks down. With sufficiently advanced AI, powerful actors — states, corporations, and AI labs — can generate wealth directly from artificial intelligence rather than human labor. As a result, people cease to be economically necessary. This has several cascading consequences: 

1. Mass automation progressively eliminates jobs, not just for routine workers but eventually for highly skilled professionals as well. This process, described as “pyramid replacement,” hollows out organizations from the bottom up until even elite talent is displaced.
2. Economic power shifts away from labor toward capital and control over AI systems. Ownership of compute, data, infrastructure, and AI models becomes far more important than human effort. Those who already possess capital gain a permanent advantage, while social mobility declines sharply.
3. Institutions lose their incentive to invest in people. If humans no longer contribute meaningfully to production or revenue, funding education, public goods, or welfare ceases to have a clear return on investment. 

This mirrors the “resource curse” in economics, where states rich in natural resources neglect their populations because they no longer rely on taxation. The illustrative case is Congo whose economic history is one of lucky breaks leading to great misery. There is no other country in the world as fortunate as Congo in terms of its natural wealth. But not a drop of the fabulous profits trickled down to the larger part of the population. Rather, they have often served as a slave labor force for the extraction of those resources at minimum cost and maximum suffering. 

The "intelligence curse "is a more extreme version: AI replaces almost all of labor. The result is a potential breakdown of the modern social contract: Democracies and market economies historically align the interests of powerful actors with the well-being of citizens, because human productivity drives growth. But in a post-AI - world, this alignment disappears. Power could concentrate in the hands of those controlling AI, leading to entrenched inequality, reduced freedom, and the marginalization of most people.

The article warns that if intelligence becomes independent of humans, society may no longer be organized around human flourishing. To quote economist Erik Brynjolfsson, who runs the digital economy lab at Stanford University: In this world, most of us “would depend precariously on the decisions of those in control of the technology.” Society would risk “being trapped in an equilibrium where those without power have no way to improve their outcomes”.