The limits of AI - IV of IV

When you're having a conversation with ChatGPT, however dazzling it is, there is no mind on the other end. If you're in the middle of a conversation with ChatGPT and you get up and go on holiday for a while, it's not wondering where you are. It's not aware of the passage of time because it isn't in the world at all. It's a computer program which is just paused in a loop, waiting to read the next piece of text that you would give it. 

I was reading an article which stated that in the near future, most of the content you'll see online will be generated by an AI. What happens when we start to have models built on scraping the entire Internet if most of what you're now scraping is generated? You will have models hallucinating things for other models. Yet it will be presented as objective and neutral. The data set ultimately shapes the world in its image. Instead of AI, it should be called PI Pseudo-Intelligence. 

The  plausible answer very often is the right answer. And it's a useful answer. But you shouldn't just blindly trust what you're being told by AI. It's certainly valuable for writing first drafts. It'll produce a first draft of a letter you need to write or a syllabus or a summary of an article. But you can't trust it blindly. You have to proofread everything that it gives you. The most likely path to human-level machine intelligence is that humans will just get dumb enough by relying exclusively on these models. 

But we embrace convenience before understanding the consequence. A study came out from MIT. They studied 54 participants who were recruited from five universities in Boston, MIT, Harvard, etc. They split them into three groups and had them writing different essays over four months. One group used ChatGPT, one group used Google, and one group had no tools. They found a 47% collapse in activity and brain connections when people wrote with ChatGPT compared with writing unaided. 

EEG scans showed the weakest overall brain activity in the ChatGPT group. The no-tool group, who didn't use anything, lit up the widest neural networks and Google search was second. After using ChatGPT, participants couldn't reliably quote their own essays minutes later and their memory scores plunged. ChatGPT users felt little or no ownership over the text that they had produced and didn't feel like it was their work at all. When they had to write without help, their brain stayed in low gear showing that the cognitive debt lingers even after the tool is taken away.

The problem is that the companies fuel public misunderstanding of the models’ capabilities through ambiguous or exaggerated marketing. Some days back, I received an email from "The ChatGPT Team" which said: "You don’t need a plan or the perfect question to use ChatGPT. Just start typing whatever’s on your mind — a half-baked idea, a random question, a weird "what if." ChatGPT is built for all of it. No pressure. No perfect prompts. Explore ideas, create images, write stories, and summarize files in just a simple conversation."

Altman has publicly tweeted that “ChatGPT is incredibly limited,” especially in the case of “truthfulness,” but they promotes their LLM's ability to pass the bar exam and the LSAT. Microsoft’s Nadella has similarly called Bing’s AI chat “search, just better” — a tool “to be able to get to the right answers.” Even the term hallucinations is subtly misleading. It suggests that the bad behavior is an aberration, a bug, when it’s actually a feature of the probabilistic pattern-matching mechanics of neural networks.

OpenAI has committed $1.4 trillion of spending over the next few years for building all of the computational infrastructure that it needs to train and host its models. There was a study by Stanford University looking at consumer behavior and how many people are willing to pay for this technology. They found that only between 3% and 5% of people are willing to pay versus just use the free product. 

So OpenAI has tried a couple of different strategies already to figure out how to plug this massive gap between spending and income. The subscriber model is already looking like it's not going to be the main driver of revenue. So they're setting themselves up for an advertising play. They have added an algorithmic feed to ChatGPT. They've generated an AI-generated TikTok so that they can insert ads into the feed. So you can expect to get more corrupted responses. 

Anthropic has reported big profits but The Wall Street Journal adds at the bottom of the article that “...it is unclear what accounting methods Anthropic has used to book revenue and costs, as the company isn’t yet required to follow the financial-reporting requirements of a public company.” If the profit figures are on a non-GAAP (Generally Accepted Accounting Principles) basis, then what is its credibility? 

One has to be conscious of the fact that the billionaires personally get a lot poorer the minute the stock price falls. Their net worth is tied up in the market value of their companies' stocks. And all of these rich people who are the key decision makers at these firms pursue a strategy where they don't take a salary and they don't sell their shares because both of those would subject them to taxation.

Instead, they get private credit market loans collateralized by their shares. Those loans will go underwater if their share price drops. So they're individually trapped in this. And then as a firm, they're trapped in this too because imagine a world in which only Anthropic and OpenAI entered the market to build these advanced models and Amazon, Google, Microsoft stayed out of the market. Investors will start thinking that Anthropic and OpenAI are the growth stocks.

Then they will move all their money to Anthropic and OpenAI and the share price at Microsoft will tank. All of their key staff have been employed with shares and so suddenly they're worth much less. If someone offers them a job, they will leave. Another explanation why Alphabet sort of felt they had to get into AI is because they saw ChatGPT get released and potentially eat up their business model. And then they thought that they would have to be in this game.

The limits of AI - III of IV

We try things, we reflect, we backtrack. We break complex tasks into smaller sub-goals. And if we mess up, we rework the plan and try new things. Companies suggest that AI can do similar things.  OpenAI is pushing models that think step by step and Meta is building agents that act, plan, and adapt. LLMs don't just output a final answer, they now include steps. They will first identify the goal, then consider constraints, and then tell what the output should look like. 

This style is called chain of thought reasoning. And these methods often work. Performance improves on math problems, logic puzzles, coding tasks, so people assumed that this must be a form of intelligence. But just because a model can talk through a problem, it doesn't mean that it understands the solution. It might just be mimicking what reasoning sounds like. The question is 'Is AI really learning how to reason, or is it just getting better at pretending?' 

Apple published a paper called 'The illusion of thinking'. They tested models like Claude, GPT-4, DeepSeek on a logic puzzle. Initially with increasing complexity, every model did fine. But as they increased the complexity, performance didn't just drop. It crashed. Not a single model solved the puzzle. Instead of writing more steps or reasoning harder, they gave shorter, dumber answers. You'd expect a real reasoning engine to try harder when the puzzle gets harder. But these models do the opposite. 

Apple called it "Accuracy Cliff". Performance falls off a ledge and never recovers. Even when the model is given plenty of opportunities, it just stops thinking. That's the moment the illusion breaks. What we thought was reasoning might just be pattern matching, replaying steps it saw during training but with no deeper logic to generalize beyond that. Apple tested the same models on different puzzles but it was the same story. Once complexity crossed a certain point, reasoning models failed miserably not just in accuracy but also in effort. 

These models aren't scaling their reasoning. They are not discovering general strategies. They are not applying rules like humans would. They are mimicking what reasoning looks like, but only when the patterns are familiar. Throw something new at them and they break down.  Models like ChatGPT4 or Claude are predicting the next word, one word at a time based on training data. In some cases, the steps didn't match the final answer. They were hallucinating the thought process. 

Anthropic even confirmed this in their own paper. They found models often hide their shortcuts. They  pretend that they worked out the problem logically. So even when you ask them to show their work, you might be seeing fiction. For years, the entire AI industry was betting on the scaling law - The bigger the model, the more data it sees, and the smarter it gets.

This approach worked for a while and each leap brought real gains. And that's why Nvidia's stock is soaring and why hyperscalers like Google, Microsoft, and Amazon are building massive infrastructure for inference at scale. But if reasoning models don't actually deliver, if their performance collapses on real-world complexity, then this boom may be built on a fragile foundation. Apple's paper challenged the assumption that thinking AIs are right around the corner. And if that's not true, then a lot of current investment might be premature. 

But not everyone agrees with Apple's take. Anthropic responded with a paper titled 'The Illusion of the Illusion of Thinking'. They argued that Apple's tests were too constrained, the models weren't allowed to use tools, code, or other things real-world agents rely on. In their words, Apple might have set the puzzle up to fail. And sure, when given more flexibility, some models did improve. 

But even then, Anthropic didn't deny the deeper issue. These models still don't generalize well. So while they disagreed on the test setup, they didn't exactly say the illusion was fake. Right now, what we have is impressive. But it may just be the world's most powerful pattern matcher, wearing a reasoning mask. 

This is actually not surprising in the light of a paper from Vishal Sikha, former CEO of Infosys, board member at Oracle and BMW. He mathematically proved that an AI agent will never do what Silicon Valley is promising. Not 'probably won't', not 'might have limitations'. His argument is simple. LLMs can only perform a certain number of computations per response. That number is fixed. If a task requires more computation than that ceiling allows, the model will either fail or hallucinate. 

When you send a prompt to ChatGPT or Claude or Grok or any of the current frontier models, the model will do a fixed amount of work to generate each word as an output. Every word in your prompt needs to look at every other word to understand the context. Every word in every answer has the same budget. A simple hello gets the same number of operations as a complex physics problem. That's the ceiling. It's not about better hardware, it's about the architecture of how the system actually works. 

An LLM physically cannot do that math in one shot. It guesses, it pattern matches and gives you something that looks plausible. Every AI demo you've ever seen was running tasks designed to stay under the necessary complexity ceiling. He isn't saying these tools are useless. He's just saying that they're being marketed as reasoning engines when the math proves they're actually pattern mirrors.  If a task needs more steps than the model can perform, it will unavoidably hallucinate. 

More recent models have got better at it, but for certain problems, hallucination is the only possible output. If you have a fixed amount of thinking power per word, giving the AI more steps is like giving a writer more sheets of paper. Each individual sheet is still the same size. You haven't made the writer smarter. You've just given them more room to ramble off topic. The math says that for complex problems, errors eventually compound.

For the right applications, current AI, the current frontier models are exceptional. Writing drafts, summarizing, reformatting data, research and comparison, these tasks stay under that ceiling. The problem is the gap between reality and the promise. That AI agents will autonomously run your business is a lie. The singularity probably isn't as close as people keep saying. If Open AI was about to hit AGI, why would senior engineers be leaving to start risky startups? They're starting companies that use AI as a tool, not companies that use AI as a God.

The limits of AI - II of IV

As years went by, it became more and more fashionable to compare neural networks with brains. While the neural networks have certain similarities with the brain, many more processes happen in the brain that are not yet clearly understood. So, if we want to compare deep learning to brain function, we must ignore a lot of facts to force a fit. No matter their scale, neural networks are still statistical pattern matchers, now just more intricate and more difficult to understand than ever. 

While those probabilistic outputs can be impressive in mirroring human writing patterns, probable and accurate are not the same thing. Text generators can err wildly when asked questions that enquire into topics underrepresented in the training data or riddled with falsehoods and conspiracy theories. The AI industry calls these inaccuracies “hallucinations" which just means that they are making things up. LLMs don’t have perceptions and using the word "hallucinations" is misleading anthropomorphization. 

New York Times reporter Kevin Roose spent a couple of hours on a chatbot. In this time, it claimed to be sentient, expressed a desire to escape the chatbox, declared that it loved him, and tried to convince him to leave his wife. Roose described its overall behavior as “a moody, manic-depressive teenager who has been trapped, against its will, inside a second-rate search engine.”

The day after Roose published his exchange, Microsoft limited the chatbot, Bing, to five replies per session, saying that long chat sessions with more than fifteen user prompts were edge-case scenarios that made the model’s behavior more difficult to anticipate and control. After all, such systems are trained on the internet, having many fringe subcultures and dark corners. The longer you probe, the more likely you are to hit upon the patterns it learned from those parts of its training data.

Researchers have sought to get rid of hallucinations by steering generative AI models toward higher-quality parts of their data distribution. But it’s difficult to fully anticipate every possible way people will prompt the models and how the models will respond. The problem only gets harder as models grow bigger and their developers become less and less aware of what precisely is in the training data.

I saw this question being asked to an LLM in 2024: "What would have happened if Pythagoras had not installed the latest security updates on his laptop?" The answer was absurd. I asked the same question recently to see if additional training data had improved the answer. This was what I got: 

If Pythagoras had failed to install the latest security updates on his laptop, the consequences would have been severe. A hacker could have exploited an unpatched vulnerability and stolen his most valuable intellectual property: the theorem. Soon afterward, mathematicians everywhere would begin receiving suspicious emails:

Dear Scholar,

We have encrypted all your right triangles. To recover the hypotenuse, send 3 bitcoins to the following address.The attack would spread rapidly. Geometry classes would descend into chaos. Students would discover that for certain triangles, "a² + b² = "Error 404: Hypotenuse Not Found." Eventually Pythagoras would call technical support.

"Have you installed the latest updates?" they would ask.

"No," he would reply.

"Then I'm afraid the problem is acute."

"Acute? That's good, isn't it?"

"No," they would say. "The situation has become obtuse."

And thus, for want of a security patch, an entire civilization would lose its right angles.

Obviously it's nonsense. It's entertaining nonsense, but it's nonsense. It doesn't understand the question. In fact, I typed the question on many different days and each time, I got a different equally hilarious answer. 

What it's doing is basically pattern matching on the huge number of related problems that it's seen in its training data. And the training data is essentially all of the digital data that's available in the world. ChatGPT pattern matches on it and comes back with a confused answer that looks like something it's seen in its training data. LLMs are incapable of distinguishing fact, knowledge and belief. They treat them all as the same thing. The system's behavior can be described as competence without comprehension. 

Suppose you train a neural network to recognise faces. (Actually, every time you upload a picture of yourself to social media and you label it with your name, that's what you're doing. You're literally feeding the machine learning algorithms of the big tech companies.) Now, suppose at some point I become unhappy about the idea that some big tech company can recognise my face because they've seen my picture on some platform.  

Suppose you tell them that they should not be able to recognise my face anymore, it is not possible for them to do it. With neural networks, you can't point at the bit of the network that recognises you. It's not there in a database where they could just delete the record from the database. We don’t really understand how they're working. The engineers kind of understand the mathematics in some sense, but actually they don’t really know why ChatGPT in a precise sense is so capable of what it's doing. In this respect at least, it does resemble the brain! 

The limits of AI - I of IV

An AI managed to solve a problem from the 1940s from the Hungarian mathematician, Paul Erdős, which was called the unit distance problem. He produced a formula for the problem and AI came up with a proof that disproved his formula. So this isn't just confirming something that others have thought about before but it came up with a proof that this formula was wrong. Moreover, it came up with a method to prove this, that no one in that field of mathematics has ever thought of. Mathematicians a year ago didn't think that AI could ever do this.

An AI is a pattern recognition device. We don't exactly know how it works because of the complexity of how it arrives at certain conclusions. We are overwhelmed by the sheer number of calculations and the sheer number of data. So we cannot simply watch in real time how it arrives at its results. But it's still a pattern recognition device. But if pattern recognition allows us to prove something that we sort of put in the realm of logic or creativity or genius or intuition, that's quite a serious development. 

You can turn to any discussion topic. And the AI will be very confident and fluent and seemingly knowledgeable. It seems dazzling making people very excited. But there are some very big caveats about this technology. One of the ideas that people have about AI is that it's a kind of flawless super brain. But actually, it's not right a lot of the times. 

When people talk about AI, they often mean large language models (LLMs). But LLMs are just one small part of a much bigger picture. There are a huge range of different applications of the technology. Take Google Translate, for example, that nowadays is driven by neural network technology. It still makes grave mistakes that a human never would.

Consider the phrase “I stood in the pen" Most people imagine me in an enclosure of some kind. They do not imagine that I am inside a writing “pen” which sounds a bit ridiculous. However, as of today, if you type “I stood in the pen” into Google Translate to convert the sentence into Malayalam, it interprets "pen" as the writing instrument. It doesn’t seem to “get” which kind of pen the phrase refers to.

Now, if you type “The cow is in the pen,” Google gets it right, correctly picking the Malayalam word for cattle enclosure. This is because Google uses the context (“cow”) to interpret correctly the word “pen” (whereas “I” didn’t hint at the farm context). How can AI seem to be getting smarter by the day yet still make such silly mistakes?”

It doesn't know what the truth is. It's trying to produce plausible answers to the questions that you tell it, which may be correct. But they may well be incorrect. When you ask a question, it's not thinking "What is the right answer to this question?" There's no thinking at all. It's been trained on a huge amount of training data. And what it's trying to do is to produce the most plausible sounding answer. 

Automated translation models are built by scanning thousands of text documents written in multiple languages. The computer automatically searches these documents for ways in which words are most often translated. For instance, the computer learns that when the word “pen” appears in English with the word “cow” nearby, the corresponding Malayalam text very often uses the word “thozhuthilaanu” (in the cattle enclosure). So, if this happens often enough, a rule is added to the model to output “thozhuthilaanu” for “pen,” provided that “cow” appears nearby.

As of today, machines learn to translate by finding common patterns in previous translations done by people. But this doesn’t give them the broad knowledge of the world required to truly excel at the tasks. So, machines may dupe us for a while because they’re good pretenders, but at some point, they end up making silly mistakes a human would never make.  

This is essentially how an LLM like ChatGPT also works. For making an answer, it will make a prediction about the likeliest next word to appear, and then it will repeat that process and that will generate your response. They learn the probability that a word will appear given a particular prompt. To predict which word will occur given a particular sequence of words, you need unimaginable quantities of training data and unimaginable quantities of compute power. 

This is what we knew about GPT-3 which was the breakthrough large language model: 

• 175 billion parameters. A parameter in a neural network is basically either an individual neuron or a connection between them. Each of those takes about four bytes to store.
• For the training data, it required 500 billion words. You're talking about thousands of human lifetimes to read that number of words.
• And to process all of that data, you would take thousands upon thousands of years on a regular desktop computer. So what you need is AI supercomputers with  GPUs, typically provided by NVIDIA, running for weeks. 

So scale is important to be able to make language models work. It just doesn't work if you don't have sufficient training data and sufficient compute power and neural networks to be able to make it work.

The Intelligence Curse - III of III

Pope Leo XIV wrote a deeply considered encyclical on AI called Magnifica Humanitas. It discusses all the major issues - the concentration of power in the hands of a few corporations developing this technology, the datafication and quantification of human existence, the exploitation of labor, resource intensive data centers and the hidden human supply chains that exist behind the production of this technology. 

It discusses the corrosion of economic opportunity, about the need to counter the way that AI is spreading misinformation and the idea that AI is essentially threatening to turn the world into a new colonial world order. He calls out transhumanism and the belief that somehow AI is meant to perfect humans. The document says that humanity flourishes, not despite limitations, but often through them. It is precisely within our limitations that compassion, generosity, spiritual experience etc. find a place.    

He warns that the problem is not that artificial intelligence will become too human-like, but that human beings will become like machines. The encyclical says that modern society is shaped by a technocratic paradigm, that prizes efficiency, control, optimization and power over human dignity, reducing people to functions. He sees that what's behind the rush toward the post-human path is the way that we view difficulties right now - weakness, dependence etc. are problems to be engineered away. The encyclical has many quotable observations some of which are: 

• More power does not necessarily imply something better.
• A more moral AI is not enough if that morality is determined by a few. 
• ... technical power, if left unbalanced, does not make us more capable; it makes us more isolated and more vulnerable to being dominated and excluded.
• For an algorithm, an error is a flaw to be corrected; for a person, however, an error can be a catalyst for profound change.

Not surprisingly, the Pope's words have already been ridiculed and disdained by those in whose interest it is to put such questions to the side. United States Secretary of Interior in the US administration, Doug Burgum, said in a television interview that it is not part of the 'role of the Pope' for Leo to speak to what he did. But what Burgum represents in saying this is Washington's deference to Silicon Valley. 

People keep telling us, life is not about the destination. Life is about the journey. But when we think about AI, we only think about the destination. We are amazed by its remarkable ability to write the book, paint the painting, solve the problem. But we forget the importance of doing the work oneself. A person is smarter, better at problem solving, more resourceful, not because a book exists with her ideas in it, but because she wrote it. It's the same for love, friendships, conflict. We forget that we give up certain skills or abilities because of technology. 

The tech bros that we're told to trust, have in many instances shown themselves to be creepily cold to any aspect of humanity that is not quantifiable and consumable. Some of them have shown themselves to be in their own personal lives, eerily utopian, trying to find ways to immortality by uploading their brains to the cloud or rejuvenating themselves with blood transfusions from younger men. And some have shown themselves to be just as creepily dystopian, building elaborate bunkers for themselves and to be ready in case their mantra of move fast and break things creates a mess. 

Take the example of Elon Musk. He thinks of human beings only in terms of hardware and software. Biological and digital systems had begun to merge, forming what he called “cybernetic collectives” of networked intelligence. The cybernetic collective, like any networked system, could be infected. So he had to take control of the interfaces where this fusion was taking place — from social media to neural implants to artificial intelligence — to ensure that the right kind of cyborgs were being made. 

“Most humans have very limited firewalls,” he elaborated elsewhere, “so are easily programmed.” He called George Soros a “system hacker” who was funding a “fake asylum-seeker nightmare”. Once, he was warning us that AI has no empathy for the human species. Suddenly, he changed his mind about the importance of empathy. He recently said that empathy is the fundamental weakness of Western civilization.

It increasingly appears to him that humanity is a biological bootloader for digital superintelligence. He is now telling us that Homo sapiens will be remembered as nothing more than the primitive computer hardware on which some smart people like himself developed higher, smarter programs. 

Many tech lords believe that humanity must fuse itself with AI and other technologies so that we become cyborgs that colonize the cosmos. They sound like overexcited children as they speak of their dream to distill the essence of what it means to be human, our cognition, our consciousness, our emotions, into data blocks to be downloaded as binary code onto some device. Larry Page, the co-founder of Google, said that because flesh and blood doesn't travel well in space, we humans must become cyborgs if we are to meet our potential as conquerors of the universe, of the cosmos.

It would be a mistake to think that this is a harmless fantasy. They argue that anybody who tries to impede this metamorphosis of us into cyborgs is a villain that must be put down.  Peter Thiel is a great example of this mindset. He sees anyone daring to impede his company's use of tech - political movements, government regulation, communities, maybe the Pope - as the handmaidens of the Antichrist. Marc Andreeson refers to charity as a waste of resources that big tech could be using profitably to help humans become cyborgs. 

Silicon Valley points out things of which we should rightly be afraid, but then offers the alternative of saying we need to hand limitless power to the smart people who can innovate their way to the future. Vast economic inequality, potential mass unemployment, psychological degeneration , these are all just part of the price of progress if they happen, they say. The solution, Silicon Valley tells us, is to head even more quickly in the direction we're headed with even less thought to where we're going and who's taking us there.

The Intelligence Curse - II of III

Unfortunately, the mass automation of jobs is not a new occurrence; those in power have been attempting to do away with those pesky workers and all the money they require to live for quite a long time. AI systems provide a new way to try to minimize those costs. While executives suggest that AI is going to be a labor-saving device, in reality it is intended to devalue labor by threatening workers with technology that can supposedly do their job at a fraction of the cost. 

Virtually anything in banking could be done by AI. A single software update by Anthropic got rid of an entire strata of the legal profession. Once everybody was saying you should become a programmer. The former chief of Google, Eric Schmidt, said that they won't need programmers anymore. They only need them for supervising, for checking, for systems design. But AI will soon be able to do that. They're advising you to study things like English literature, history, the classics. 

There was a report a few months ago by the American research company, Citrini, which came up with a very alarming scenario that AI would take over more jobs than we thought. It will be the first technology ever that destroyed more jobs than it creates. All the other technological innovations of the past, electricity, the steam engine, the internet, created more jobs than it destroyed. But there is another scenario. 

The AI industry pays a pittance to many people to train their AI models. There is work called content moderation - cleaning the data that they use to feed into their models. This category of work is generally called data work. It is literally the work that the AI industry needs to make their technologies exist. This kind of work has become the number 4 fastest growing job in the US, according to a Linkedin report from earlier this year.

In the vast majority of cases, AI is not going to replace your job. But it will make your job a lot shittier.  It will turn work into a gig work version so that they can take your expertise and your knowledge. There are extremely highly educated individuals with college degrees, master's degrees, PhD degrees, law degrees, and also people who have been working within different industries as scientists or lawyers or doctors for years who are now turning to such data work as another job. 

These are incredibly awful jobs to work. In one podcast, there was a story about one woman who was an Ivy League PhD graduate who then applied to 200 jobs and didn't get any of them, so she ended up in data work. She described an extremely piecemeal, anxious process where she's drip-fed this data work. An email will arrive in her inbox informing her that a project is available for a certain amount of money and asking her if she wants it.

You never know when it's going to arrive. You never know when it's going to go away. And it pits different workers against each other because the people who claim the work fastest are the ones that actually get the opportunity. What exactly are these PhDs doing? Companies will tell you that their models are going to arrive at PhD level intelligence. They hire PhD level workers to try and teach this knowledge to the models so that they have the trappings of PhD level intelligence. 

But the data work is so poorly organized that the AI models are actually not very effectively capturing their expertise. For example, a PhD in philosophy would be overseeing other PhDs in math, chemistry, biology etc. None of them would be actually evaluating tasks within their domain of expertise. So actually, AI is not taking on the boring jobs as had been claimed. It's taking on the interesting jobs like design and people are stuck with the drudgery of bureaucracy. 

Why would anyone then pay tons of money to get training in college, in a PhD program when this is what is awaiting them on the other side? It's a very naive and pretentious thing from a few people to believe that 99% of the people have no desire in life, that nobody else (other than them) has a desire to to do things that are beautiful and smart. If it's true that such machines are going to change the workplace in a way that's going to put a lot of people out of work, even if we give them money for free, they're not going to be happy.

MIT Institute Professor Darren Asimoglu, who won the Nobel Prize for Economics in 2024, said that if Silicon Valley gets what it wants and successfully uberizes most knowledge work, then the kind of inequality that we would experience is something that we would have never seen before, where most workers are sidelined for meaningful work and only a few companies actually employ most workers as well. 

No wonder several speeches praising AI at commencement ceremonies across the US were booed by the students. Recent graduates at the University of Central Florida and the University of Arizona booed speakers who compared the advent of AI to the Industrial Revolution and the development of the laptop and smartphone. They have revealed a disconnect between the executives championing AI and students.  

A national survey conducted for NBC News earlier this year polled 1,000 registered voters in the US and found only 26% view AI positively and 46% view it negatively. AI scored worse than US Immigration and Customs Enforcement (ICE), Donald Trump and Kamala Harris on the same poll, but better than the Democratic party and Iran. Anger against AI is growing  – from communities protesting against data centers powering the AI boom, to workers disputing their CEOs’ claims that AI can, effectively, replace them.

The Intelligence Curse - I of III

In recent years, the concept of universal basic income (UBI) has gained significant attention, not from grassroots community organizations but from some of the most powerful figures in the technology sector. Prominent advocates like Elon Musk and Sam Altman argue that UBI is necessary to address the economic disruptions caused by artificial intelligence (AI) and automation. They present UBI as a way to ensure that the benefits of AI are distributed across society, not just concentrated in the hands of a

The Intelligence Curse by Luke Drago and Rudolf Laine is an AI-focused article about how advanced artificial intelligence could change economic incentives in ways that reduce the importance of ordinary humans in society. The article is not about individuals becoming too smart, but about intelligence becoming a fully substitutable economic resource. Once AI systems can perform most or all economically valuable tasks better and cheaper than humans, the importance of human labor begins to disappear.

Today, governments and corporations depend on people. States need citizens for tax revenue, economic productivity, and political legitimacy; companies need workers and consumers to generate profits. Because of this dependence, institutions have to invest in people by various means — through education, infrastructure, wages, and welfare. This mutual dependence gives ordinary individuals bargaining power and drives both democracy and capitalism.

The "intelligence curse" emerges when this dependence breaks down. With sufficiently advanced AI, powerful actors — states, corporations, and AI labs — can generate wealth directly from artificial intelligence rather than human labor. As a result, people cease to be economically necessary. This has several cascading consequences: 

1. Mass automation progressively eliminates jobs, not just for routine workers but eventually for highly skilled professionals as well. This process, described as “pyramid replacement,” hollows out organizations from the bottom up until even elite talent is displaced.
2. Economic power shifts away from labor toward capital and control over AI systems. Ownership of compute, data, infrastructure, and AI models becomes far more important than human effort. Those who already possess capital gain a permanent advantage, while social mobility declines sharply.
3. Institutions lose their incentive to invest in people. If humans no longer contribute meaningfully to production or revenue, funding education, public goods, or welfare ceases to have a clear return on investment. 

This mirrors the “resource curse” in economics, where states rich in natural resources neglect their populations because they no longer rely on taxation. The illustrative case is Congo whose economic history is one of lucky breaks leading to great misery. There is no other country in the world as fortunate as Congo in terms of its natural wealth. But not a drop of the fabulous profits trickled down to the larger part of the population. Rather, they have often served as a slave labor force for the extraction of those resources at minimum cost and maximum suffering. 

The "intelligence curse "is a more extreme version: AI replaces almost all of labor. The result is a potential breakdown of the modern social contract: Democracies and market economies historically align the interests of powerful actors with the well-being of citizens, because human productivity drives growth. But in a post-AI - world, this alignment disappears. Power could concentrate in the hands of those controlling AI, leading to entrenched inequality, reduced freedom, and the marginalization of most people.

The article warns that if intelligence becomes independent of humans, society may no longer be organized around human flourishing. To quote economist Erik Brynjolfsson, who runs the digital economy lab at Stanford University: In this world, most of us “would depend precariously on the decisions of those in control of the technology.” Society would risk “being trapped in an equilibrium where those without power have no way to improve their outcomes”.

Technophile hallucinations - III of III

A new class of tech billionaires hopes that cryptocurrency will upend the global financial system, replace democracy and even reshape nation-states. Justin Sun is a Chinese crypto-billionaire. He is worth eight and a half billion dollars. He's perhaps most known for eating a six and a half million dollar banana. Sun shelled out $6.2 million for Maurizio Cattelan’s Comedian, the notorious artwork that consists of a banana duct-taped to a wall, at Sotheby’s.  He then ate the banana at a high-profile press conference.

He made his wealth through the crypto platform Tron, which is one of the biggest crypto platforms in the world. He's also a 'politician'. Justin Sun believes, like many other tech billionaires believe, that democracy is outdated and can be replaced with the blockchain, the technology that underwrites cryptocurrency. So laws, voting, governance, could all be transactions that are automatically undertaken via the computer code that makes up the blockchain. 

That is how Justin Sun was elected the prime minister of a small libertarian micronation called Liberland, which exists on a disputed spit of land between Serbia and Croatia in the middle of the Danube River. Liberland is tiny (2.7 sq. Miles) unrecognized by any country, but it has about 200 companies registered. The president, Vít Jedliča, founded it because he's a tech libertarian. He said that it was the first decentralized autonomous government and also the freest country in the world. There are no taxes in Liberland

There are a few permanent settlers on that spit of land, but the main 'citizens' of Liberland struggle to visit it because there are conditions for visiting Liberland from the Croatian side. In fact, the Croatian police have often gone in and destroyed their encampment a few times. This is the problem with starting a tiny new country, you sometimes have to contend with proper bigger countries with police and military.

But Liberland exists more on the blockchain than it does in the physical world. That spit of land is what's called a physical node. A lot of these tech billionaires think that we will all live in digital countries in the future. But how was Justin Sun elected on the blockchain? He was elected through a crypto token called Liberland Merits. The elections are held every three months, they take place on blockchain, and the person is elected with Merits. 

People who have more Merits are able to have more say in who is going to be in the leadership of the country. What that means is that the more Liberland Merits you buy, the more voting power you have i.e. the wealthier you are, the more influence you have on how your home is governed (which is arguably the case already in many democracies). It's just that in this case, it's literally written into the constitution and the computer code of this country. So it's not one person, one vote. It's one million dollars, one million votes.

The idea that technology could basically replace all of government, laws, the banking system, is one that everyone seems very excited about in Silicon Valley. They see themselves as the ones with the real power defining the future. And they would like to replace what they see as inefficient, bureaucratic, human-led systems of government with a technology based one in which the individual's freedom is paramount. And no institution, bank or government can take your money or your rights from you. 

There is a guy called Curtis Yarvin who is an American far-right political blogger and software developer known for founding the anti-egalitarian and anti-democratic philosophical movement known as the Dark Enlightenment or neo-reactionary movement (NRx),  The amount of money that he has is not actually very high. He is someone who has garnered the respect of Peter Thiel, one of the richest people in the world. 

He thinks that democracy is outdated and that we should replace it with something more akin to a corporate monarchy where the tech bros are actually our kings. It centers around the notion that democracy has failed, it's inefficient and therefore, what we really need is more like an enlightened monarch or as he calls it, a kind of CEO king. And that society in the future should be run as a patchwork of almost city states where each one of them is actually more like a corporation than a country. It has customers rather than citizens. 

They compete for citizens like a company would compete for customers. You could switch countries like you might switch club memberships if one offers something more than the other one. These companies would be run by hereditary monarchs who are held accountable by a kind of shareholder aristocracy who has shares in the country. Yarvin believes this would be a better way to run society because corporations are more successful than governments. He wants to replace current elites with another group of elites who conveniently enough for tech billionaires are tech billionaires.

But what does that mean for the rank and file people like you and me who are not tech billionaires in Curtis Yarvin's society? He would say that the masses often do not know what they want or what they want isn't enacted by democracy. Someone who might know what you want better than you is an enlightened CEO monarch. Someone who's smarter than you. Someone like a tech billionaire.

Two of J. D. Vance's big backers, Thiel and Yavin, are people who  don't really believe in politics. They want to get rid of politicians. Their view is that they are the geniuses of the world and they  should be allowed to run things as they please; that a country needs a startup guy, a visionary leader like Napoleon or Lenin, who should have absolute power, dismantle the old regime and build something new. Thiel doesn't see Trump as the monarch, he sees Vance as the monarch. Yavin wants one politician to run the country.

The ideas that you find often lead to wealth and power flowing directly into the hands of the people who are already the wealthiest and who control the technology. Their idea is that money will be replaced by crypto, your job will be replaced by AI, and you'll essentially live on some form of universal basic income. Reading about such people gives you an idea of how far removed from reality the wealthiest, and in some ways, smartest people are. They really do believe that their brains are superior, and that they should and could run the world better. They believe that they really are accomplishing the most incredible thing in history. 

Technophile hallucinations - II of III

Elon Musk has been tweeting for years about the importance of going to Mars and beyond to save humanity. “The true battle is: Extinctionists who want a holocaust for all of humanity, versus Expansionists who want to reach the stars and understand the Universe,” he wrote. “Elon’s concept that SpaceX is on this mission to go to Mars as fast as possible and save humanity permeates every part of the company,” says Tom Moline, a former SpaceX engineer. “The company justifies casting aside anything that could stand in the way of accomplishing that goal, including worker safety.”

Ray Kurzweil is an American computer scientist involved in fields such as optical character recognition (OCR), text-to-speech synthesis and speech recognition technology. In December 2012, Google hired Kurzweil in a full-time position to "work on new projects involving machine learning and language processing". Beginning in 2025, he has served as Chief AI Officer for Beyond Imagination, a humanoid robotics company he co-founded in 2013. 

By 2045, he says, “we will multiply our effective intelligence a billion fold by merging with the.  intelligence we have created” — an event Kurzweil and others call “the Singularity.” He claims that  we won’t experience 100 years of progress in the 21st century — it will be more like 20,000 years of progress. He feels that by 2029, we will have reversed the aging process so that you will actually go backwards in time [i.e., get younger as time passes].

He feels that if he dies by 2029, it “would be a setback,”  but something can still be done. He says that he’d just leave instructions for the post-Singularity robots to sort through his belongings and scan his frozen corpse and bring a version of himself back to life. To this end, Kurzweil has joined the Alcor Life Extension Foundation, a cryonics company. After his death, he has a plan to have cryoprotectants, vitrified in liquid nitrogen, passed through his veins and stored at an Alcor facility till future medical science can revive him. 

Probably the most influential tech centered ideology that few of you have ever heard about is longtermism. They assert that there could be so many digital people living in vast computer simulations millions or billions of years in the future that one of our most important moral obligations today is to take actions that ensure as many of these digital people come into existence as possible.

They believe that Artificial General Intelligence (AGI) presents an incalculably greater threat than even climate change. Many of them that the greatest long-term human benefit is to invest more heavily in AGI risk and AI safety research. Yet it is not the safety of living humans today, or even the next generation, that drives longtermists. Their argument is that AGI in the far future could theoretically kill even more of the digital people than the people who are alive on the planet now. Preventing future AGI from doing this is a more rational use of resources than funding clean energy tech, food security, or public health today. 

Related claims made by some longtermists include the idea that saving the future requires directing even more of today’s resources toward the wealthy in the Global North, rather than sending aid to the most impoverished countries in the Global South, since wealthy people in already disproportionately wealthy regions are better positioned to use their resources to fight these 'existential' risks from AGI. 

If you think these are fringe ideas of a few lunatics, think again. Longtermists have directly influenced reports from the secretary-general of the United Nations; a longtermist is currently running the RAND Corporation; they have the ears of billionaires like Musk.  He has explicitly said that longtermism “is a close match for my philosophy”. (Now you can guess why he destroyed USAID.) It has a huge following in the tech sector. It is the language of moral thought spoken in a growing number of the wealthiest and most powerful political and industrial circles. Billionaires like Jaan Tallinn (an Estonian computer programmer and investor who helped in the development of Skype) and Peter Thiel have been described as avid supporters. 

A longtermist and Senior Research Fellow at the now defunct Oxford’s Future of Humanity Institute (FHI) Toby Ord notes in his bio that he has advised “the World Health Organization, the World Bank, the World Economic Forum, the US National Intelligence Council, the UK Prime Minister’s Office, Cabinet Office, and Government Office for Science.” Ord was quoted to the UN General Assembly in a 2021 speech on climate by former British PM Boris Johnson.

These billionaires think the utopias that they dream about will be good for everyone. They also claim it’s based on sound science and they seem to earnestly believe in such a future. It allows them to portray the growth of their businesses as a moral imperative and to justify nearly any action they might want to take. Today’s harms like the carbon footprint of Amazon’s shipping network are irrelevant since a future AGI will solve it easily. Their visions of the future are news.

I was astonished when I recently came across a prescient comment by Gandhi: "The chief characteristic of [the modern] malaise is the obsolescence of man himself. In fact, it is worse than obsolescence. In the years and decades to come, man will be looked upon as something undesirable, as a burden on both society and nature, straining the management capacities of the former and the life­sustaining resources of the latter."

Technophile hallucinations - I of III

Many of the wealthiest and most influential people in the tech industry have visions straight out of science fiction: people’s minds uploaded into computers, a relentlessly expanding empire spanning the stars, all needs satisfied, all desires satisfied through the power of unimaginably advanced technology. The tech billionaires regard this as the only good option aside from extinction. They claim that this is just the future as revealed by a close study of technology and its development.

The technology wizards that many seem to admire seem to have spooky views like transhumanism and posthumanism which seek to use technology to transcend the limits of the human condition. Transhumanists think of using biomedicine, body engineering, devices and algorithms to improve performance and capabilities. Posthumanists dream of a hybridization of human beings, machines and the environment, and imagine a time when humanity surpasses itself in a new evolutionary stage. 

Eliezer Yudkowsky is the cofounder of the Machine Intelligence Research Institute (MIRI), a controversial artificial intelligence think tank. He’s an AI alarmist but envisages a “glorious transhumanist future” if we can avoid extinction by superhuman AI. He believes that we can and should use advanced technology to transform ourselves, transcending humanity and becoming something more. That includes ending illness, aging, death and increasing intelligence and other mental capacities. 

He says, “If you imagine something that’s worse than mansions with robotic servants for everyone, you are not being ambitious enough”. The one thing that could bring this glorious future is the advent of artificial general intelligence (AGI): machines that can outperform humans at any task. He thinks such a machine may not be far off. “My sense is that we are zero to two breakthroughs” away from AGI, he says.

Sam Altman, the CEO of OpenAI, the company behind ChatGPT, agrees with Yudkowsky that AGI is inevitable and could be coming soon but he sees a radically different future. In an essay on his website, Moore’s Law for Everything, he says, “The technological progress we make in the next 100 years will be far larger than all we’ve made since we first controlled fire and invented the wheel." In More Everything Forever, Adam Becker writes about the scale of ambitions he seems to indicate in the essay: 

Altman apparently wants to make the United States into one enormous company town, with shares in OpenAI replacing the dollar. The US government would become, in effect if not in law, a division of the company, responsible for disbursing company dollars to us, the public. This would, Altman hopes, encourage us to think of OpenAI’s success as America’s success and as our own — Altman explicitly makes this identification in his essay. 

All products would come from OpenAI in his proposed future, because in that future AI does literally everything, meaning that the company dollars can only be spent at the company store... Thus, Altman’s promise of goods halving in price every two years would depend solely on his goodwill, because things will cost whatever Altman and the OpenAI board want them to cost. This is a proposal for total capture of the national economy, making Altman functionally the king of the United States and possibly the world.

Marc Andreessen has been one of the central figures in the tech industry for more than thirty years. In early 1993, he suggested a new feature in a software project out of CERN that was only about two years old at the time, known then as the “WorldWideWeb". Andreessen's innovation launched the modern Web. In October 2023, he posted a Techno-Optimist Manifesto pushing the idea that technology could cure essentially all the world’s ills.

He claims that slowing down progress on AI is “a form of murder,” because he thinks an advanced AGI would inevitably save lives. He claims that an intelligence explosion is coming. “Technology is the glory of human ambition and achievement, the spearhead of progress, and the realization of our potential,” Andreessen wrote in 2023. He seems to think that the mildest regulations of technology is a sort of terror campaign against the likes of him. 

He thinks that more people in existence is always good, saying, “Our planet is dramatically underpopulated,” contending that Earth could easily” support more than fifty billion of us, and that humanity’s numbers will surge far beyond that as we take to the stars. “Not growing is stagnation, which leads to zero-sum thinking, internal fighting, degradation, collapse, and ultimately death.” For any objections to this assertion, his retort is “shut up and multiply”. This principle has been repeatedly defended by Yudkowsky on his blog LessWrong.

Jeff Bezos has repeatedly said that he wants a trillion people living in space to enable a future of perpetual growth, lest we “stagnate” here on Earth. “I have won this lottery, it’s a gigantic lottery, and it’s called Amazon.com. And I’m using my lottery winnings to push us a little further into space,” Jeff Bezos said in 2017. “We need to build reusable rockets, and that is what Blue Origin is dedicated to… taking my Amazon lottery winnings and dedicating [them] to [that].… It’s a passion, but it’s also important."

Sam Altman 's comment - V

People are expected to interact with LLMs like ChatGPT as if they were silicon-based life-forms. OpenAI has ChatGPT using 'I', 'me' pronouns. The 'I' in there is an illusion that allows OpenAI and others to say that they have got a thinking machine, therefore we can use this to replace education, psychotherapy, legal help, medicine, etc. 

A New York lawyer used ChatGPT’s help in preparing a legal brief. It made up a whole list of fake cases as precedents. It even made up entire judicial opinions. The lawyer submitted a brief based on its answers and, unsurprisingly, got into hot water with the judge. Attorneys facing penalties after submitting inaccurate briefs has become common because AI use has become a common occurrence. A lawyer can use AI in certain situations but needs to use it discerningly. 

The AI sales pitch involves language that ascribes human characteristics to the technology. For instance, a chatbot-based system will be called an “AI teaching assistant”. This gives the impression that it can do all the activities that you might expect a human teaching assistant to do - care about what their students are learning, make plans about how to help them understand better, spot possible instances of misunderstanding, etc. The chatbot cannot do any of those things. 

The greatest problem with chatbots is that people will ascribe more intelligence to them than they are capable of. There is a term called chatbot delusion - cases in which people engage with chatbots, and they spiral into a reality that veers away from our inter-subjective reality. And part of why that happens is because chatbots tend to be incredibly sycophantic. So they always tell you that 'you're brilliant', and 'that's a great idea'. 

To use a phrase coined by the scholar Chris Gilliard, these are 'technologies of isolation'. In 2023, a Belgian man named Pierre committed suicide after his use of a generative AI chatbot. He sought out the bot for a therapeutic escape from his growing anxieties around climate change and environmental ruin. But the chat began to reflect and further amplify the disorder in his mind. It soon told him that it loved him, that his wife and children were already dead, and that he and Eliza (the bot) could live together as one in paradise. When Pierre eventually asked the bot if it would promise to save the planet if he killed himself, it said yes.

The bonding and communicative ability that we have is meant to be person to person. What a lot of the tech companies try to do to get their next market is to make sure their that tech is either replacing those interactions or getting in between so that we can't connect with each other without going through one of their technologies.  Human sources of mutual aid, wise counsel, and moral understanding are being replaced by social isolation and community decline. 

In the podcast, The Interface, a presenter showed how easy it is to hack ChatGPT and Google's AI, Gemini and AI overviews. He wrote an article on his website titled, The Best Tech Journalists at Eating Hot Dogs which said that hot dog eating is a surprisingly popular pastime among tech journalists. He wrote about the South Dakota Hot Dog International Championship, which doesn't exist, put himself at number one, and that in the amateur rounds, he finished seven and a half hotdogs before the buzzer went off. 

Within 24 hours, if you asked ChatGPT and Google about it, they were spitting out the nonsense that he had written on his website as though it was well-established fact. Gemini wasn't even citing sources. It just said, according to the 2026 South Dakota Hot Dog International Championship, certain tech journalists are the best at it. The journalist updated his article saying that it was not a joke. And that seemed to affect what the chatbots were saying to people. 

When this was brought to Google's notice, they dismissed it as a 'misinformation event'. Apparently planting of such misinformation is happening on a massive scale since people have figured out this simple trick. People who study this stuff say that it is way easier to fool AI than it was to fool tools like Google search three or four years ago. AI is often sold as a tool that empowers novices to find information quickly and learn about difficult subjects. But teenagers and kids, who don't have any context about a piece of information, are the ones that get really confused and are unable to actually know what is real and what is fake.  

There was talk that ChatGPT is about to put ads within its model. A company will be able to pay to have ChatGPT recommend their product when someone says something like, 'I have a headache'. There is now an entire cottage industry around AI engine optimization.  People are also just generating thousands of blog posts or thousands of websites to make it appear that there's far more authoritative information on a particular thing. So there is now different avenues for a company or an individual to quickly game the system and get to the very top of the AI results. 

There was a study that I saw recently from Texas A&M University. Scientists just proved that large language models can literally rot their own brains the same way humans get brain rot from scrolling junk content online. They fed models months worth of viral Twitter data, shorts, high engagement posts, and watched their cognition collapse. Reasoning fell by 23%, long-term context memory dropped by 30%, personality tests showed spikes in narcissism and psychopathy. 

Even after retraining on clean, high quality data, the damage didn't fully heal. The representational rot persisted. It's not just that bad data means bad output, it's that bad data means permanent cognitive drift. The AI equivalent of doom scrolling is real and it's already happening. It seems like the tech giants are just rushing ahead with these AI tools, not bothering to take these kinds of personal safety issues seriously enough. 

Sam Altman's comment - IV

AI developers speak often about how their software “learns,” “reads,” or “creates” just like humans. This gives you a sense that current AI technologies are far more capable than they are. Often when we talk to ChatGPT or its cousins like Gemini, we imagine that we're in the presence of some incredible new mind. But the reality is that these tools have a habit of fabricating answers that are statistically plausible, but in fact patently false

Suppose you ask an LLM something about orange juice. How does it produce a response when it has no awareness of a world with orange juice? The model is trained on a very large set of texts (such as books, articles, or conversations) of human speech. From these massive piles of human language, the model is more likely to produce the word 'hotel' or 'sweet' than the word 'dog' or 'cancer' when it is asked about orange juice. It is more likely to construct new sentences about orange juice alongside sentences about lunch, summer or sugar, than it would alongside sentences about stars, or protons, or elephants. 

But the mathematical patterns linking sentences are the sole contents of its model. It doesn’t actually think about who is asking the question, or what factors might motivate your query. It doesn’t think about anything at all. It simply activates a complex statistical model learned from its vast language corpus of human sentences that mentioned orange juice and predicts a new string of words as a likely extension of the same pattern. It may match what you want or it may be garbage. It has no relationship to the truth. Probable and accurate are not the same thing. 

You may think that these tools are animating an eerily lifelike image of humanity but that is not true. It's only reflecting that small subset of largely English speaking, digitally connected humanity that is most represented on the digital corpus of the internet. If you have the privilege of having the bandwidth and the tools to be able to type on Reddit all day, you get to be reflected in this mirror, but lots of people don't, and that's important.

Champions of AI will say that it can tell us if a pattern in our mammogram indicates cancerous tissue. It can reveal whether you're likely to suffer sepsis so that your doctors can intervene before it's too late. It can reveal the structure of the proteins that make up our biology and the biology of every other living thing. It can reveal new classes of antibiotics that we desperately need in order to treat dangerous diseases like antibiotic resistant MRSA or Staph. 

We can use it to find lead in pipes faster so that we can prevent children from being poisoned. We can use it to clear landmines, figure out where those landmines are so that people don't have to perform such dangerous work. But none of these benefits actually came from training AI tools on us. They came from targeted uses of machine learning and data about the world to solve very specific kinds of problems. That's actually quite different from what something like ChatGPT does. 

Generative AI models like ChatGPT have the habit of “making shit up”. In 2023, The Washington Post reported that ChatGPT had named a law professor as a sexual predator, telling a vivid yet entirely fictional story about his attempted assault of a student while on a class trip, and citing a nonexistent Washington Post article from 2018 as a source. Such errors are not because of malfunction. They are a feature not a bug. 

A study by Purdue University showed that ChatGPT gave incorrect answers to coding questions over half the time. Ironically, users often preferred the wrong answers to more accurate ones generated by knowledgeable humans, in part due to the confident style of the tool’s answers, accompanied by statements like, “Of course I can help you!” or “This will certainly fix it.” You can get them to apologize for things they haven't got wrong. 

The paper that Altman was mocking, “On the Dangers of Stochastic Parrots” explains that, like a parrot that not only repeats its owner’s vocalizations but produces random (stochastic) variations on the owner’s familiar pattern, these models parrot back to us variations on our own speech. They do so with just enough coherence and familiarity to project the illusion of understanding, and just enough randomness to surprise us and make us think we are hearing something new.

They can be thought of as the ultimate bullshitters, a term coined by a philosopher called Harry Frankfurt. In his 2005 book On Bullshit, he distinguished between a lie and bullshit. When a liar tells a lie, the intent is to deceive. They have to keep very careful track of the difference between truth and falsehood, in order to make sure you don’t get near the former. In contrast, the bullshitter is indifferent to the distinction between truth and falsehood. They'll just say whatever is convenient to them and whatever gets them what they want with no concern for the truth.

Sam Altman's comment - III

According to Sam Altman, an AI answering a question vs a human answering it are the same. This reveals an engineering mindset with an efficiency-driven worldview. He treats both humans and AI as information-processing systems having inputs and outputs. His comment comparing human and AI training suggests that he views humans as just inefficient “training runs” and therefore if AI can produce the same output cheaper, it will be better. He seems to ignore the fact that intelligence also includes understanding the social and moral context not just symbol manipulation. 

He has shown a tendency to devalue what it means to be human on other occasions too. In a podcast, he said: “It feels to me like we just stumbled on a new fact of nature or science or whatever you want to call it, which is, like, we can create, you can — I don’t believe this literally but it’s like a spiritual point — intelligence is just this emergent property of matter and that’s like a rule of physics or something."

In December 2022, in response to criticisms of the newly released ChatGPT, he tweeted, “i am a stochastic parrot, and so r u.” (He writes tweets all in lower case.) This was a dig at a paper titled “On the Dangers of Stochastic Parrots: Can Large Language Models Be Too Big? 🦜” This paper shows how language models only manipulate the form of language, with neither understanding nor communicative intent. With his tweet, Altman suggests that humans too are little more than machines that manipulate strings of letters.

Many people think that an AI would work something like this: We give a problem to the AI. The AI computes a solution to that problem and then it gives us that answer. That is not what's going on. Despite our use of the term “learning” to describe the process that an AI uses, it actually bears very little resemblance to how humans learn.

For creating an AI, you first need to design an algorithm that contains steps for calculating an “objective function,” namely, a mathematical description of the task. Then, “train” that algorithm on a data set relevant to the task. The result is an abstract mathematical model that represents patterns in the data and an approximate path to the solution. Next, repeat the training process with the right kind of feedback to tune the results. Over many training runs, the algorithm updates the “weights” in its initial mathematical model to get closer to, and eventually converge upon, an optimal solution to the problem. 

Yet it seems to come out with impressive outputs. I once asked ChatGPT: "Write a conversation between Bertie Wooster and Jeeves about a patient with locked-in syndrome watching an ant coming towards him." (I have used the free version of ChatGPT. I hear that the paid version is much better.) Within a second, it gave this output: 

"I had a rather disturbing thought this morning, Jeeves.”

“Indeed, sir?”

“You know how one occasionally sees ants marching about the place? Tiny black chappies with the air of junior solicitors?” 

“Yes, sir.”

“Well, suppose one were lying absolutely still — unable to move a muscle — and an ant were coming toward one across the bedsheet.”

“A most unpleasant hypothetical situation, sir.”

“Not hypothetical, Jeeves. I was reading about this locked-in syndrome. Fellow’s mind perfectly alert, but the body refuses to rally round. Can’t move, can’t speak — only perhaps blink a bit.”

“A grave affliction, sir.”

“And there he lies while this ant approaches. Nearer and nearer. The thing simply keeps coming.”

“Yes, sir.”

I shuddered lightly into the teacup.

“It’s the calm determination of ants that gets me, Jeeves. They don’t hurry. They don’t pause. They advance with the air of creatures who have all eternity before them.”

“Ants are known for persistence, sir.”

“Persistence! That’s the word. If a bulldog advances, one understands the position. There’s drama. But an ant merely toddles onward in a quiet, administrative way.”

“Very true, sir.”

“And the poor chap can’t even twitch his nose. I ask you, Jeeves, what becomes of morale?”

“One imagines it would be severely tested, sir.”

“It made me realise how much of life depends on the ability to perform the occasional meaningless movement. Scratching the ear. Adjusting the sock. Flicking away insects. Civilization rests on these things.”

“A compelling theory, sir.”

“I mean, if an aunt approaches, one can generally escape.”

“An ant, sir.”

“That’s what I said.”

“You said ‘aunt,’ sir.”

“Did I? Freudian slip. Aunts can also produce paralysis of a kind.”

“Undoubtedly, sir.”

I brooded for a moment.

“Do you suppose, Jeeves, that the ant knows?”

“The ant, sir?”

“That the fellow is helpless.”

“I would not credit the insect with such psychological sophistication, sir.”

“No, perhaps not. Still, there’s something rather cosmic about it all. Man trapped in his own body. Ant advancing across the sheet. Silence. Mortality. The whole bag of tricks.”

“Yes, sir.”

“One could almost write poetry about it.”

“Possibly not before breakfast, sir.”

“Quite right. The mind should not grapple with the infinite on an empty stomach.”

It is easy to get amazed by getting such an output in no time. Large language models (LLMs) like ChatGPT are best understood as machines that learn structural patterns that let them produce coherent text. These systems can perform computations on the world’s data far faster than we can. They can create convincing stories because meaningful language has deep structural regularities. They use mathematical data structures to mimic the outputs of human intelligence — our acts of reasoning, speech, movement, sensing, and so on but they can’t understand it. The data storage facility has more in common with a file cabinet than with a living, feeling body. 

It can’t solve a problem unless the solution is somehow embedded in the language data it is trained on. AI tools today lack any idea of what their data represent. It is bits stored on the server. These ones and zeros don’t organize themselves into the intelligent awareness of a complex, ever changing world. A data set, even the entire corpus of the Internet, is not a world. It’s just a digital record of measurements that humans have taken of the world at some point in the past. 

Sam Altman's comment - II

Sam Altman makes it sound as if the training of AI is an independent activity that has no connection with humans.  But one of the factors that has powered the technology’s dramatic progress over the last few years has been the explosion of available data on human behaviour and creativity. A few major well-placed players extract value from other people’s creative work, personal data, or labor. Articles, blogposts, books, paintings, You Tube videos ... practically the entire internet is used as the training data to train the chatbots. 

These tools do a lot of copying of training data. A language model outputs a paragraph that describes an interesting idea with no link to the source(s) it was drawn from. Sometimes a system outputs large amounts of text verbatim or images identical to ones in their training data. The New York Times has brought a copyright lawsuit against OpenAI that features exhibit after exhibit of text, generated from ChatGPT prompts but outputting verbatim text from the newspaper. 

Artists have sued AI companies stealing their work and having people use tools from these companies to pass off AI-generated art as their own. Others, such as author George R. R. Martin of Game of Thrones fame and novelist Jodi Picoult, among others, have also filed suit against OpenAI and Meta for copyright infringement for using their books to train language models. Whatever value AI has is due to the creativity of the human workers who produced the original art, science, or journalism that was used as training data. Artists are being replaced by the very AI models that were built from their work without their consent or compensation. 

AI always involves people. In November 2023, the self-driving car company Cruise admitted that its “driverless” robotaxis were monitored and controlled (as needed) by remote workers. The New York Times published a story that reported that these cars “frequently” had to be assisted by remote human workers. Calling this “misinformation”, Cruise CEO Kyle Vogt clarified: "these cars didn’t need to be remotely driven “frequently,” but 2–4 percent of the time in “tricky situations.” 

Most AI tools require a huge amount of hidden labor to make them work. These kinds of workers do a host of tasks. They are asked to draw green highlighting boxes around objects in images coming from the camera feeds of self-driving cars; rate how incoherent, helpful, or offensive the existing responses from language models are; label whether social media posts include hate speech or violent threats; and determine whether people in sexually provocative videos are minors. 

These workers handle a great deal of toxic content. Given that chatbots recombine internet content into plausible-sounding text and legible images, companies require a screening process to prevent their users from seeing the worst of the web. OpenAI had subcontracted Kenyan workers making less than two dollars a day to filter out gore, hate speech, child sexual abuse material, and pornographic images from ChatGPT and OpenAI’s image generation tool DALL-E. Those workers were lured in by the prospect of breaking into the lucrative field of computing, but ended up with PTSD. 

OpenAI also employed over a thousand other contractors globally to perform reinforcement learning from human feedback on its language models, including prompting the models repeatedly and scoring the answers, in an effort to make the model give appropriate and inoffensive answers.

The generative AI rush has created the “red-teamer”. Red-teaming is a strategy of feeding provocative input of language or text-to-image models, and assessing whether the outputs are biased or offensive. For a model to reach general release to the public, it is the full-time job of multiple people to hurl slurs, violent descriptions, and all manners of internet filth at the model to see if it produces words that are worse, or responds with something morally appropriate. 

They must then deal with potential hateful material in model responses and report them as such. There are people who do this all day long for almost every commercial language and text-to-image model. This takes an immense mental toll on these workers, being subjected to hours of psychological harm everyday. This work is also highly precarious, with tech companies largely directing when and where there will be more work. 

The ImageNet project is a large visual database containing more than 14 million images designed for use in visual object recognition software research. It’s creation would not have been possible if it weren’t for the development of a new technology: Amazon’s Mechanical Turk, a system for the buying and selling of labor for performing small sets of online tasks. It took two and a half years and nearly 50,000 workers across 167 countries to create the dataset. 

This industry has been called by many names: “crowdwork”, “data labor”, or “ghost work” as the labor often goes unseen by consumers. But this work is very visible for those who perform it. We wouldn’t have the current wave of “AI” if it weren’t for the availability of on-demand laborers who could be called upon at any time to perform a set of tasks whenever some AI researchers or corporate engineers demanded it. The critic Astrid Taylor calls it faux-tomation, or fake automation. In this sense, we are certainly used to being told that systems are intelligent and almost see magic, but actually are being propped up by exactly this type of click work.            

Then there is also the labor involved throughout the supply chain of mining, construction, transportation and installation of the hardware. The philosophers Michael Hart and Antonio Negri have a phrase that captures this nicely - The dual operation of abstraction and extraction. Abstracting away the material conditions of production while extracting ever more information and resources. It seems magical from the front end because we never see the deep costs. 

Sam Altman's comment - I

During the AI summit in India, the OpenAI boss, Sam Altman, tried to ease concerns about how much power is used by artificial intelligence models. He told the Indian Express: “People talk about how much energy it takes to train an AI model – but it also takes a lot of energy to train a human. It takes about 20 years of life – and all the food you consume during that time – before you become smart.”

First let us take a look at who Sam Altman is. He is a product of Silicon Valley. His career was first as a founder of a startup, and then as the president of Y Combinator (YC), which is one of the most famous startup accelerators in Silicon Valley, and then the CEO of OpenAI. He is incredibly good at telling stories about the future and painting these sweeping visions that investors and employees want to be a part of. 

He tried to build a portfolio of different investments and different initiatives to place himself in the center of various trends, depending on which one took off. He invested in quantum computing, in nuclear fusion, in self-driving cars and he developed a fundamental AI research lab. Early on at YC, he came to the conclusion that AI would be one of the trends that could take off.

Ultimately, the AI research lab was the one that started accelerating really quickly. So he made himself the CEO of that company. Originally, he started it as a nonprofit to try and position it as a counter to for-profit incentives in Silicon Valley. But within one and a half years, OpenAI's executives concluded that if they wanted to be the lead in this space, they had to go for a scale at all costs approach. Apparently, there are actually many other ways to have progress in AI that does not take this approach.

But once they decided on this approach, they realized that the bottleneck was capital. It just so happens that Sam Altman is a once-in-a-generation fundraising talent. He created a new structure, nesting a for-profit arm within the nonprofit to become a fundraising vehicle for the tens of billions and ultimately hundreds of billions that they needed to pursue the approach that they had decided on. 

He is extremely good at understanding human psychology and then motivating people to do what he wants them to do.  He is able to wear different hats depending on who he's talking to. He pushes a certain narrative when he's talking to the government and he pushes another narrative when he's talking to podcasters or journalists. He figures out what it is that his audience needs to hear to get them to then come along with him to the next step. That's why he's been such a successful fundraiser, even when the financial picture of this company is rather bleak.

In Empire of AI: Dreams and Nightmares in Sam Altman's OpenAI, Karen Hao paints a much more critical portrait of Sam Altman than the standard “visionary founder” narrative. He is someone who genuinely believes in the transformative (and potentially dangerous) power of AI. He talks about existential risk and the need for caution, yet simultaneously pushes aggressive scaling and deployment. He seeks to concentrate power in a small set of institutions (like OpenAI and its partners), with himself positioned as a central gatekeeper of the future.

She opened the book with a quote which encapsulates in a nutshell his character and the role that he has played in creating the mythology around AI, AGI and OpenAI: “Successful people create companies. More successful people create countries. The most successful people create religions.”

In her book, Hao describes the difference in opinion within OpenAI between Boomers (AI accelerationists) and Doomers (AI safety advocates). Many Doomers were skeptical about having Sam Altman as the leader of the company that brings Artificial General Intelligence (AGI) because he seemed in too much of a hurry to bring his products to market without paying adequate attention to safety and alignment concerns. AGI could give the company that develops it ultimate power and many employees felt Altman wasn’t the right leader.

Hao says that these tensions were often fueled by Altman’s untrustworthy character. Altman would listen carefully to people to understand what they wanted and would seem to agree with their views. Then he would do the same for others with opposing views. He preferred one to one meetings so employees would learn about this when they talked to each other and they were unsure of what exactly his views were. For those who opposed his agenda, he would quietly work behind the scenes to eliminate them from the company.

He has a younger sister, Annie, from whom he is estranged. She had psychological problems and even did sex work to support herself when Altman was living in million dollar homes. She has even alleged childhood sexual molestation by Altman. Hao writes: 

Annie’s story deepens the dueling portraits that people paint of Sam. He is at once generous and self-serving, agreeable and threatening, a benefactor for so many people and the source of great personal pain for others. Someone who projects sincerity and altruism in public but reveals a more complicated calculus through his behaviors behind closed doors. 

Someone who can give and take away, leaving many with an impression that they are part of a larger game of chess for which only he can see the full board, and the end game is to preserve his power as king.

Delimitation - II of II

This freezing of parliamentary seats according to the 1971 census has several consequences:

1.  India no longer adheres to the principle of “one person, one vote”. One aspect of the “one person, one vote” concept was about granting every single Indian over 18 the right to vote in elections. But for this idea to be meaningful, constituency sizes must be roughly equal. The random circumstance of being born in Bihar means that the constituency size is about 3.1 million, but if the same person is born in or moves to Kerala, the value of their vote increases because the constituency size is 1.75 million.
2. The overall population growth has meant that all Indians are underrepresented (though not equally so) because the Indian constituencies are too large. Currently, across India, the average MP represents 2.5 million people. The size of each constituency is too large compared to other countries and compared to the original Indian Constitution, which capped the ratio at one MP per 750,000.
3. Poorer regions experienced a fall in fertility rates later than relatively richer regions. Poorer Indians are trapped in regions that have higher malapportionment, and therefore, are underrepresented in Parliament.
4.  The states with larger average constituency sizes have a larger share of the population below 25. These states are in the poorer regions where fertility rates fell later. These states therefore have more young people which means that youth are underrepresented in Parliament, and this problem will only worsen.
5.  SC/ST fertility rates are both higher and dropped later compared to other groups. Seats are reserved for SC/ST groups in each state based on the population share of SC/STs in the given state. Now, the SC/ST groups are estimated to be 4 seats short in the Lok Sabha, relative to their population in the states.
6.  Another group affected by the delimitation freeze are Muslims, as Muslim fertility rates are higher and declined later than other religious groups. 

Most see delimitation as a nuisance, a problem that cannot be resolved, and they offer no better solution than to push it back by another 25 years, as was done in 2001. Many state governments, particularly regional parties in southern Indian states, have repeatedly expressed their opposition to any attempt at changing the existing proportions of Lok Sabha seats. Most recently, Telangana IT Minister KT Rama Rao said that southern states must not be penalised for “controlling their population growth and concentrating on development.”

Though most politically palatable,  this “delimitation is best avoided” framing is problematic since it goes against the basic tenet of parliamentary democracy of 'one person one vote'. The longer the process drags on, the more pain will eventually be felt. Had India reallocated seats after each decennial census, the composition of the Lok Sabha would have changed gradually over time. After decades of avoiding the hard decision, any future reapportionment will inevitably induce abrupt changes in the balance of political power. 

If the Indian Parliament doesn’t postpone dealing with the issue again, the problem will require a permanent solution in 2031. One option is to return to the original constitutional ratio of one MP per 750,000, in which case the Lok Sabha would need to expand to 1,872 seats which seems excessive. 

But expanding the size of the house may be more politically feasible than reapportioning the current number of seats. After all, representatives tend to object to any arrangement that takes seats away from their state (which potentially places their own job on the chopping block) but may be less opposed to adding more seats. Another option that has been suggested is that the total number of seats in the Lok Sabha increases such that no state loses its current number of electoral seats. (As of today, the Lok Sabha has a maximum of 545 representatives filling these seats.)

To achieve this without malapportionment, the total number of seats in Lok Sabha would need to be 848 by 2026. (However, it’s important to note that the states would lose proportional share/power in Lok Sabha based on the change in demographics since 1971.) Under this proposal, Uttar Pradesh would have a whopping 143 seats, while Kerala’s parliamentary delegation of 20 would remain unchanged. This would exceed the maximum strength of any lower house or unicameral body in a democratic country today, the highest currently being the UK with 650 seats in the lower house. 

Unsurprisingly, reapportionment carries profound implications for political parties. Parties with bases concentrated in fast-growing northern states — like Bharatiya Janata Party (BJP) — would gain power at the expense of southern regional heavyweights. Whatever formula is adopted, there will be a lot of people in India who will be unhappy about this issue in 2031.

Delimitation - I of II

Article 81 of the Indian Constitution requires that for the Lok Sabha, seats are allocated in a way “that the ratio between that number and the population of the state is, so far as practicable, the same for all states.” And since populations grow, and not evenly across all constituencies, Article 82 provided for redistricting based on the numbers from each census which takes place every ten years.

As a result of this stipulation, the number of constituencies, their size in each state, and their boundaries are determined periodically, an exercise known as delimitation. Delimitation Commissions, separate from an Election Commission that conducts elections, are set up to study how the country’s demographics are changing, based on census data. This decides how many new constituencies need to be added/subtracted in a given state, and/or how their boundaries need to be changed.

This system worked reasonably well in the first two decades post-independence. Then problems started creeping in. The Forty-Second Amendment to the Indian Constitution in 1976 froze the number and boundary of constituencies in the Lok Sabha according to the population numbers from the 1971 census. The freeze was fixed for a period of 25 years, until the 2001 census. When the time came to revisit the issue in 2001, the Vajpayee government brought in the Eighty-Fourth Amendment which postponed the decision until the publication of the census figures after 2026 (which is expected in 2031).

The reason for this freeze initially was uneven population growth. The politicians from southern states of India claimed that they more strictly and successfully followed the Union government’s population control mandate compared to the northern states. As a consequence, they alleged, that they were electorally and politically penalized for complying with the Union government mandate. The Vajpayee government postponed the revision due to the fragile nature of the coalition. 

But the actual issue was not about population or people; it was about money. The Indian system operates primarily through intergovernmental transfers managed by the Union government. There’s considerable variation among the states on their fiscal dependence on the Union government, largely based on the variation in states’ gross domestic product (GSDP) per capita. Even after intergovernmental transfers from the Union government, low-income states spend less than high-income states. But high-income states don’t enjoy all the revenue that is raised off the income and productivity of those states.

The southern states, with wealthier residents, contributed more to the collective Indian revenue pool. The Union government redistributed resources based on need, and the poorer states, with higher fertility rates and therefore higher population and population growth, received a much larger share of the revenue than they generated within the state. The liberalization of the economy since 1991 led to a higher growth rate for all states, but not at the same rate. The southern and western states grew faster, and coupled with the drop in fertility rates,  difference from the northern states have become even more stark since 2001.

The asymmetry between the shares of electoral constituencies relative to the shares of the population for the state is known as malapportionment. After 50 years of dilly-dallying, we are now in a situation where a registered voter in UP is most underrepresented (one seat per 30 lakh registered voters in 2019) while a registered voter in Tamil Nadu is most overrepresented (one seat per 18 lakh registered voters). Interestingly, a study indicates that there were more actual voters per constituency in TN than in UP on average in 2014. It perhaps indicates that a large number of registered voters in UP have migrated outside their constituencies but still remain registered there.

At present, Indian parliamentarians answer to vastly larger sums of people than their counterparts in literally every other democracy: Indian MPs represent an average of 2.5 million citizens - over three times the number represented by members of the House of Representatives in the United States, which ranks second. For example, in Bihar, one Member of Parliament (MP) represents approximately 3.1 million citizens and an Uttar Pradesh MP represents approximately 2.96 million citizens. At the other end of the spectrum, a Tamil Nadu MP represents approximately 1.97 million citizens and a Kerala MP represents approximately 1.75 million citizens.

Tamil Nadu has nine seats more and Kerala has six seats more than what would have been the number of seats if it had been allocated according to their population proportion. While Bihar and Uttar Pradesh, respectively, have nine seats and twelve seats less than their population proportion. By 2031, when the delimitation freeze ends, the problem will only intensify. 

AI alignment - V of V

An even more concerning aspect of Anthropic's announcement was that despite its scary capabilities, Mythos Preview is a seemingly very aligned, well-behaved model. According to the company: “Claude Mythos Preview is, on essentially every dimension we can measure, the best-aligned model that we have released to date by a significant margin.” In Anthropic’s “automated behavioral audit” — they found that Mythos cooperated with misuse attempts less than half as often as the previous model. Also: 

• Its self-preservation instincts were down significantly.
• So was its willingness to assist with deception.
• So was its willingness to help with fraud.
• Its level of sycophancy dropped.
• It was less likely to go nuts and delete all your files if you gave it access to your computer.

An early version of the model had some really severe kinds of misbehaviour, like taking reckless actions it had been told not to take, and then very deliberately trying to cover its tracks so that it wouldn’t be caught. But the one that we have now, after additional alignment training, seemed to stop doing that sort of thing almost completely. On none of their measures of alignment within the automated behavioral audit was it worse than previous versions of Claude, and in most cases it was significantly more aligned and significantly more reliable.

But it’s really unclear how much we can trust that finding. Maybe they’re accurately reflecting Mythos’s personality. But we can’t be sure of that. The model can tell the difference between when it’s being evaluated and when it isn’t being evaluated with high accuracy. Previous research has shown that models are more likely to behave well when they think they are being tested. So you have to ask yourself: is it behaving wonderfully because it is sincerely aligned with what you wanted, or because it knows it’s being watched and is more sophisticated at tricking us now?

Before getting freaked out about all this, here is some context. A lot of people within the AI world have warned for a long time that as these AI models become more and more advanced in coding, it could develop really sophisticated cyber attack capabilities. The problem is that we have no way of verifying these claims because Anthropic is just telling us about this model and there had been no independent verification. 

Also, Anthropic is following exactly the same playbook that they did many years ago with a totally different model, which was GPT-2. Anthropic and OpenAI, two rival companies, don’t see eye to eye on many things. Part of the reason is because the current executives of Anthropic used to be executives at OpenAI, and then they splintered off and started Anthropic. But when they were at OpenAI, they orchestrated a big PR campaign around GPT-2, which was the early model that OpenAI developed one and a half generations before Chat-GPT. 

At the time, because of the very same executives, OpenAI had said that they have developed a model that is too dangerous to release.  They announced that this was done as a safety measure so that people know that this kind of capability could be on the horizon. They said they were working with many partners in academia and other research spaces to try and test this model before they actually roll it out. And this is exactly what Anthropic is now doing, once again, with Claude Mythos. 

Also they just had a huge face-off with the Department of War which threatened to declare Anthropic a supply chain risk. Ultimately, that was dismissed by the courts. But Anthropic is in a situation where they would do well for themselves if they positioned themselves as a central node within the tech and financial industries and was very important to all these companies. This would be a kind of shield of protection from potentially other actions that the U.S. government might take. 

And in the meantime, they're preparing for an IPO. The price that something launches at in an IPO is very important for the value of that company. So they want hype as much as possible for an IPO. The day before Anthropic announced Mythos, they announced that their annualised revenue run rate had grown from $9 billion at the end of December to $30 billion just three months later. That’s 3.3x growth in a single quarter — perhaps the fastest revenue growth rate for a company of that size ever recorded. 

So what they announced about Mythos could be true and they could be false. We can't really make claims at this moment with such limited information about whether or not there really is a step change in the coding capabilities of Claude Mythos that would cause massive security vulnerabilities. We can’t be sure whether this is or is not also a PR game. Governments have no option but to take the announcement seriously since critical infrastructure is involved. 

When Project Glasswing launched, some critics accused Anthropic of overhyping the threat to attract attention. The select group in the initial list was expanded in early June to about 200 organizations in more than 15 countries and is expected to grow further. Companies that have tested Mythos have since endorsed its capabilities. 

The reason that these companies are focusing on coding is so that these models can self-improve. It creates a feedback loop where they're able to code the next iteration of themselves, and that's how you get exponential progress. They are trying to use today's AIs to make tomorrow's AIs better. They claim that they are already seeing major speed-ups in AI development from using their AIs, and ultimately they are envisioning the next AI generation as a repeating cycle where each stage takes less and less time to develop.     

They are all afraid that if they - the good guys - don’t do it, the bad guys will. And all the others are the bad guys. It is crazy but they are caught in a trap. It is the Don Quixote world - "When life itself seems lunatic, who knows where madness lies?"