The limits of AI - I of IV

An AI managed to solve a problem from the 1940s from the Hungarian mathematician, Paul Erdős, which was called the unit distance problem. He produced a formula for the problem and AI came up with a proof that disproved his formula. So this isn't just confirming something that others have thought about before but it came up with a proof that this formula was wrong. Moreover, it came up with a method to prove this, that no one in that field of mathematics has ever thought of. Mathematicians a year ago didn't think that AI could ever do this.

An AI is a pattern recognition device. We don't exactly know how it works because of the complexity of how it arrives at certain conclusions. We are overwhelmed by the sheer number of calculations and the sheer number of data. So we cannot simply watch in real time how it arrives at its results. But it's still a pattern recognition device. But if pattern recognition allows us to prove something that we sort of put in the realm of logic or creativity or genius or intuition, that's quite a serious development. 

You can turn to any discussion topic. And the AI will be very confident and fluent and seemingly knowledgeable. It seems dazzling making people very excited. But there are some very big caveats about this technology. One of the ideas that people have about AI is that it's a kind of flawless super brain. But actually, it's not right a lot of the times. 

When people talk about AI, they often mean large language models (LLMs). But LLMs are just one small part of a much bigger picture. There are a huge range of different applications of the technology. Take Google Translate, for example, that nowadays is driven by neural network technology. It still makes grave mistakes that a human never would.

Consider the phrase “I stood in the pen" Most people imagine me in an enclosure of some kind. They do not imagine that I am inside a writing “pen” which sounds a bit ridiculous. However, as of today, if you type “I stood in the pen” into Google Translate to convert the sentence into Malayalam, it interprets "pen" as the writing instrument. It doesn’t seem to “get” which kind of pen the phrase refers to.

Now, if you type “The cow is in the pen,” Google gets it right, correctly picking the Malayalam word for cattle enclosure. This is because Google uses the context (“cow”) to interpret correctly the word “pen” (whereas “I” didn’t hint at the farm context). How can AI seem to be getting smarter by the day yet still make such silly mistakes?”

It doesn't know what the truth is. It's trying to produce plausible answers to the questions that you tell it, which may be correct. But they may well be incorrect. When you ask a question, it's not thinking "What is the right answer to this question?" There's no thinking at all. It's been trained on a huge amount of training data. And what it's trying to do is to produce the most plausible sounding answer. 

Automated translation models are built by scanning thousands of text documents written in multiple languages. The computer automatically searches these documents for ways in which words are most often translated. For instance, the computer learns that when the word “pen” appears in English with the word “cow” nearby, the corresponding Malayalam text very often uses the word “thozhuthilaanu” (in the cattle enclosure). So, if this happens often enough, a rule is added to the model to output “thozhuthilaanu” for “pen,” provided that “cow” appears nearby.

As of today, machines learn to translate by finding common patterns in previous translations done by people. But this doesn’t give them the broad knowledge of the world required to truly excel at the tasks. So, machines may dupe us for a while because they’re good pretenders, but at some point, they end up making silly mistakes a human would never make.  

This is essentially how an LLM like ChatGPT also works. For making an answer, it will make a prediction about the likeliest next word to appear, and then it will repeat that process and that will generate your response. They learn the probability that a word will appear given a particular prompt. To predict which word will occur given a particular sequence of words, you need unimaginable quantities of training data and unimaginable quantities of compute power. 

This is what we knew about GPT-3 which was the breakthrough large language model: 

• 175 billion parameters. A parameter in a neural network is basically either an individual neuron or a connection between them. Each of those takes about four bytes to store.
• For the training data, it required 500 billion words. You're talking about thousands of human lifetimes to read that number of words.
• And to process all of that data, you would take thousands upon thousands of years on a regular desktop computer. So what you need is AI supercomputers with  GPUs, typically provided by NVIDIA, running for weeks. 

So scale is important to be able to make language models work. It just doesn't work if you don't have sufficient training data and sufficient compute power and neural networks to be able to make it work.

The Intelligence Curse - III of III

Pope Leo XIV wrote a deeply considered encyclical on AI called Magnifica Humanitas. It discusses all the major issues - the concentration of power in the hands of a few corporations developing this technology, the datafication and quantification of human existence, the exploitation of labor, resource intensive data centers and the hidden human supply chains that exist behind the production of this technology. 

It discusses the corrosion of economic opportunity, about the need to counter the way that AI is spreading misinformation and the idea that AI is essentially threatening to turn the world into a new colonial world order. He calls out transhumanism and the belief that somehow AI is meant to perfect humans. The document says that humanity flourishes, not despite limitations, but often through them. It is precisely within our limitations that compassion, generosity, spiritual experience etc. find a place.    

He warns that the problem is not that artificial intelligence will become too human-like, but that human beings will become like machines. The encyclical says that modern society is shaped by a technocratic paradigm, that prizes efficiency, control, optimization and power over human dignity, reducing people to functions. He sees that what's behind the rush toward the post-human path is the way that we view difficulties right now - weakness, dependence etc. are problems to be engineered away. The encyclical has many quotable observations some of which are: 

• More power does not necessarily imply something better.
• A more moral AI is not enough if that morality is determined by a few. 
• ... technical power, if left unbalanced, does not make us more capable; it makes us more isolated and more vulnerable to being dominated and excluded.
• For an algorithm, an error is a flaw to be corrected; for a person, however, an error can be a catalyst for profound change.

Not surprisingly, the Pope's words have already been ridiculed and disdained by those in whose interest it is to put such questions to the side. United States Secretary of Interior in the US administration, Doug Burgum, said in a television interview that it is not part of the 'role of the Pope' for Leo to speak to what he did. But what Burgum represents in saying this is Washington's deference to Silicon Valley. 

People keep telling us, life is not about the destination. Life is about the journey. But when we think about AI, we only think about the destination. We are amazed by its remarkable ability to write the book, paint the painting, solve the problem. But we forget the importance of doing the work oneself. A person is smarter, better at problem solving, more resourceful, not because a book exists with her ideas in it, but because she wrote it. It's the same for love, friendships, conflict. We forget that we give up certain skills or abilities because of technology. 

The tech bros that we're told to trust, have in many instances shown themselves to be creepily cold to any aspect of humanity that is not quantifiable and consumable. Some of them have shown themselves to be in their own personal lives, eerily utopian, trying to find ways to immortality by uploading their brains to the cloud or rejuvenating themselves with blood transfusions from younger men. And some have shown themselves to be just as creepily dystopian, building elaborate bunkers for themselves and to be ready in case their mantra of move fast and break things creates a mess. 

Take the example of Elon Musk. He thinks of human beings only in terms of hardware and software. Biological and digital systems had begun to merge, forming what he called “cybernetic collectives” of networked intelligence. The cybernetic collective, like any networked system, could be infected. So he had to take control of the interfaces where this fusion was taking place — from social media to neural implants to artificial intelligence — to ensure that the right kind of cyborgs were being made. 

“Most humans have very limited firewalls,” he elaborated elsewhere, “so are easily programmed.” He called George Soros a “system hacker” who was funding a “fake asylum-seeker nightmare”. Once, he was warning us that AI has no empathy for the human species. Suddenly, he changed his mind about the importance of empathy. He recently said that empathy is the fundamental weakness of Western civilization.

It increasingly appears to him that humanity is a biological bootloader for digital superintelligence. He is now telling us that Homo sapiens will be remembered as nothing more than the primitive computer hardware on which some smart people like himself developed higher, smarter programs. 

Many tech lords believe that humanity must fuse itself with AI and other technologies so that we become cyborgs that colonize the cosmos. They sound like overexcited children as they speak of their dream to distill the essence of what it means to be human, our cognition, our consciousness, our emotions, into data blocks to be downloaded as binary code onto some device. Larry Page, the co-founder of Google, said that because flesh and blood doesn't travel well in space, we humans must become cyborgs if we are to meet our potential as conquerors of the universe, of the cosmos.

It would be a mistake to think that this is a harmless fantasy. They argue that anybody who tries to impede this metamorphosis of us into cyborgs is a villain that must be put down.  Peter Thiel is a great example of this mindset. He sees anyone daring to impede his company's use of tech - political movements, government regulation, communities, maybe the Pope - as the handmaidens of the Antichrist. Marc Andreeson refers to charity as a waste of resources that big tech could be using profitably to help humans become cyborgs. 

Silicon Valley points out things of which we should rightly be afraid, but then offers the alternative of saying we need to hand limitless power to the smart people who can innovate their way to the future. Vast economic inequality, potential mass unemployment, psychological degeneration , these are all just part of the price of progress if they happen, they say. The solution, Silicon Valley tells us, is to head even more quickly in the direction we're headed with even less thought to where we're going and who's taking us there.

The Intelligence Curse - II of III

Unfortunately, the mass automation of jobs is not a new occurrence; those in power have been attempting to do away with those pesky workers and all the money they require to live for quite a long time. AI systems provide a new way to try to minimize those costs. While executives suggest that AI is going to be a labor-saving device, in reality it is intended to devalue labor by threatening workers with technology that can supposedly do their job at a fraction of the cost. 

Virtually anything in banking could be done by AI. A single software update by Anthropic got rid of an entire strata of the legal profession. Once everybody was saying you should become a programmer. The former chief of Google, Eric Schmidt, said that they won't need programmers anymore. They only need them for supervising, for checking, for systems design. But AI will soon be able to do that. They're advising you to study things like English literature, history, the classics. 

There was a report a few months ago by the American research company, Citrini, which came up with a very alarming scenario that AI would take over more jobs than we thought. It will be the first technology ever that destroyed more jobs than it creates. All the other technological innovations of the past, electricity, the steam engine, the internet, created more jobs than it destroyed. But there is another scenario. 

The AI industry pays a pittance to many people to train their AI models. There is work called content moderation - cleaning the data that they use to feed into their models. This category of work is generally called data work. It is literally the work that the AI industry needs to make their technologies exist. This kind of work has become the number 4 fastest growing job in the US, according to a Linkedin report from earlier this year.

In the vast majority of cases, AI is not going to replace your job. But it will make your job a lot shittier.  It will turn work into a gig work version so that they can take your expertise and your knowledge. There are extremely highly educated individuals with college degrees, master's degrees, PhD degrees, law degrees, and also people who have been working within different industries as scientists or lawyers or doctors for years who are now turning to such data work as another job. 

These are incredibly awful jobs to work. In one podcast, there was a story about one woman who was an Ivy League PhD graduate who then applied to 200 jobs and didn't get any of them, so she ended up in data work. She described an extremely piecemeal, anxious process where she's drip-fed this data work. An email will arrive in her inbox informing her that a project is available for a certain amount of money and asking her if she wants it.

You never know when it's going to arrive. You never know when it's going to go away. And it pits different workers against each other because the people who claim the work fastest are the ones that actually get the opportunity. What exactly are these PhDs doing? Companies will tell you that their models are going to arrive at PhD level intelligence. They hire PhD level workers to try and teach this knowledge to the models so that they have the trappings of PhD level intelligence. 

But the data work is so poorly organized that the AI models are actually not very effectively capturing their expertise. For example, a PhD in philosophy would be overseeing other PhDs in math, chemistry, biology etc. None of them would be actually evaluating tasks within their domain of expertise. So actually, AI is not taking on the boring jobs as had been claimed. It's taking on the interesting jobs like design and people are stuck with the drudgery of bureaucracy. 

Why would anyone then pay tons of money to get training in college, in a PhD program when this is what is awaiting them on the other side? It's a very naive and pretentious thing from a few people to believe that 99% of the people have no desire in life, that nobody else (other than them) has a desire to to do things that are beautiful and smart. If it's true that such machines are going to change the workplace in a way that's going to put a lot of people out of work, even if we give them money for free, they're not going to be happy.

MIT Institute Professor Darren Asimoglu, who won the Nobel Prize for Economics in 2024, said that if Silicon Valley gets what it wants and successfully uberizes most knowledge work, then the kind of inequality that we would experience is something that we would have never seen before, where most workers are sidelined for meaningful work and only a few companies actually employ most workers as well. 

No wonder several speeches praising AI at commencement ceremonies across the US were booed by the students. Recent graduates at the University of Central Florida and the University of Arizona booed speakers who compared the advent of AI to the Industrial Revolution and the development of the laptop and smartphone. They have revealed a disconnect between the executives championing AI and students.  

A national survey conducted for NBC News earlier this year polled 1,000 registered voters in the US and found only 26% view AI positively and 46% view it negatively. AI scored worse than US Immigration and Customs Enforcement (ICE), Donald Trump and Kamala Harris on the same poll, but better than the Democratic party and Iran. Anger against AI is growing  – from communities protesting against data centers powering the AI boom, to workers disputing their CEOs’ claims that AI can, effectively, replace them.

The Intelligence Curse - I of III

In recent years, the concept of universal basic income (UBI) has gained significant attention, not from grassroots community organizations but from some of the most powerful figures in the technology sector. Prominent advocates like Elon Musk and Sam Altman argue that UBI is necessary to address the economic disruptions caused by artificial intelligence (AI) and automation. They present UBI as a way to ensure that the benefits of AI are distributed across society, not just concentrated in the hands of a

The Intelligence Curse by Luke Drago and Rudolf Laine is an AI-focused article about how advanced artificial intelligence could change economic incentives in ways that reduce the importance of ordinary humans in society. The article is not about individuals becoming too smart, but about intelligence becoming a fully substitutable economic resource. Once AI systems can perform most or all economically valuable tasks better and cheaper than humans, the importance of human labor begins to disappear.

Today, governments and corporations depend on people. States need citizens for tax revenue, economic productivity, and political legitimacy; companies need workers and consumers to generate profits. Because of this dependence, institutions have to invest in people by various means — through education, infrastructure, wages, and welfare. This mutual dependence gives ordinary individuals bargaining power and drives both democracy and capitalism.

The "intelligence curse" emerges when this dependence breaks down. With sufficiently advanced AI, powerful actors — states, corporations, and AI labs — can generate wealth directly from artificial intelligence rather than human labor. As a result, people cease to be economically necessary. This has several cascading consequences: 

1. Mass automation progressively eliminates jobs, not just for routine workers but eventually for highly skilled professionals as well. This process, described as “pyramid replacement,” hollows out organizations from the bottom up until even elite talent is displaced.
2. Economic power shifts away from labor toward capital and control over AI systems. Ownership of compute, data, infrastructure, and AI models becomes far more important than human effort. Those who already possess capital gain a permanent advantage, while social mobility declines sharply.
3. Institutions lose their incentive to invest in people. If humans no longer contribute meaningfully to production or revenue, funding education, public goods, or welfare ceases to have a clear return on investment. 

This mirrors the “resource curse” in economics, where states rich in natural resources neglect their populations because they no longer rely on taxation. The illustrative case is Congo whose economic history is one of lucky breaks leading to great misery. There is no other country in the world as fortunate as Congo in terms of its natural wealth. But not a drop of the fabulous profits trickled down to the larger part of the population. Rather, they have often served as a slave labor force for the extraction of those resources at minimum cost and maximum suffering. 

The "intelligence curse "is a more extreme version: AI replaces almost all of labor. The result is a potential breakdown of the modern social contract: Democracies and market economies historically align the interests of powerful actors with the well-being of citizens, because human productivity drives growth. But in a post-AI - world, this alignment disappears. Power could concentrate in the hands of those controlling AI, leading to entrenched inequality, reduced freedom, and the marginalization of most people.

The article warns that if intelligence becomes independent of humans, society may no longer be organized around human flourishing. To quote economist Erik Brynjolfsson, who runs the digital economy lab at Stanford University: In this world, most of us “would depend precariously on the decisions of those in control of the technology.” Society would risk “being trapped in an equilibrium where those without power have no way to improve their outcomes”.

Technophile hallucinations - III of III

A new class of tech billionaires hopes that cryptocurrency will upend the global financial system, replace democracy and even reshape nation-states. Justin Sun is a Chinese crypto-billionaire. He is worth eight and a half billion dollars. He's perhaps most known for eating a six and a half million dollar banana. Sun shelled out $6.2 million for Maurizio Cattelan’s Comedian, the notorious artwork that consists of a banana duct-taped to a wall, at Sotheby’s.  He then ate the banana at a high-profile press conference.

He made his wealth through the crypto platform Tron, which is one of the biggest crypto platforms in the world. He's also a 'politician'. Justin Sun believes, like many other tech billionaires believe, that democracy is outdated and can be replaced with the blockchain, the technology that underwrites cryptocurrency. So laws, voting, governance, could all be transactions that are automatically undertaken via the computer code that makes up the blockchain. 

That is how Justin Sun was elected the prime minister of a small libertarian micronation called Liberland, which exists on a disputed spit of land between Serbia and Croatia in the middle of the Danube River. Liberland is tiny (2.7 sq. Miles) unrecognized by any country, but it has about 200 companies registered. The president, Vít Jedliča, founded it because he's a tech libertarian. He said that it was the first decentralized autonomous government and also the freest country in the world. There are no taxes in Liberland

There are a few permanent settlers on that spit of land, but the main 'citizens' of Liberland struggle to visit it because there are conditions for visiting Liberland from the Croatian side. In fact, the Croatian police have often gone in and destroyed their encampment a few times. This is the problem with starting a tiny new country, you sometimes have to contend with proper bigger countries with police and military.

But Liberland exists more on the blockchain than it does in the physical world. That spit of land is what's called a physical node. A lot of these tech billionaires think that we will all live in digital countries in the future. But how was Justin Sun elected on the blockchain? He was elected through a crypto token called Liberland Merits. The elections are held every three months, they take place on blockchain, and the person is elected with Merits. 

People who have more Merits are able to have more say in who is going to be in the leadership of the country. What that means is that the more Liberland Merits you buy, the more voting power you have i.e. the wealthier you are, the more influence you have on how your home is governed (which is arguably the case already in many democracies). It's just that in this case, it's literally written into the constitution and the computer code of this country. So it's not one person, one vote. It's one million dollars, one million votes.

The idea that technology could basically replace all of government, laws, the banking system, is one that everyone seems very excited about in Silicon Valley. They see themselves as the ones with the real power defining the future. And they would like to replace what they see as inefficient, bureaucratic, human-led systems of government with a technology based one in which the individual's freedom is paramount. And no institution, bank or government can take your money or your rights from you. 

There is a guy called Curtis Yarvin who is an American far-right political blogger and software developer known for founding the anti-egalitarian and anti-democratic philosophical movement known as the Dark Enlightenment or neo-reactionary movement (NRx),  The amount of money that he has is not actually very high. He is someone who has garnered the respect of Peter Thiel, one of the richest people in the world. 

He thinks that democracy is outdated and that we should replace it with something more akin to a corporate monarchy where the tech bros are actually our kings. It centers around the notion that democracy has failed, it's inefficient and therefore, what we really need is more like an enlightened monarch or as he calls it, a kind of CEO king. And that society in the future should be run as a patchwork of almost city states where each one of them is actually more like a corporation than a country. It has customers rather than citizens. 

They compete for citizens like a company would compete for customers. You could switch countries like you might switch club memberships if one offers something more than the other one. These companies would be run by hereditary monarchs who are held accountable by a kind of shareholder aristocracy who has shares in the country. Yarvin believes this would be a better way to run society because corporations are more successful than governments. He wants to replace current elites with another group of elites who conveniently enough for tech billionaires are tech billionaires.

But what does that mean for the rank and file people like you and me who are not tech billionaires in Curtis Yarvin's society? He would say that the masses often do not know what they want or what they want isn't enacted by democracy. Someone who might know what you want better than you is an enlightened CEO monarch. Someone who's smarter than you. Someone like a tech billionaire.

Two of J. D. Vance's big backers, Thiel and Yavin, are people who  don't really believe in politics. They want to get rid of politicians. Their view is that they are the geniuses of the world and they  should be allowed to run things as they please; that a country needs a startup guy, a visionary leader like Napoleon or Lenin, who should have absolute power, dismantle the old regime and build something new. Thiel doesn't see Trump as the monarch, he sees Vance as the monarch. Yavin wants one politician to run the country.

The ideas that you find often lead to wealth and power flowing directly into the hands of the people who are already the wealthiest and who control the technology. Their idea is that money will be replaced by crypto, your job will be replaced by AI, and you'll essentially live on some form of universal basic income. Reading about such people gives you an idea of how far removed from reality the wealthiest, and in some ways, smartest people are. They really do believe that their brains are superior, and that they should and could run the world better. They believe that they really are accomplishing the most incredible thing in history. 

Technophile hallucinations - II of III

Elon Musk has been tweeting for years about the importance of going to Mars and beyond to save humanity. “The true battle is: Extinctionists who want a holocaust for all of humanity, versus Expansionists who want to reach the stars and understand the Universe,” he wrote. “Elon’s concept that SpaceX is on this mission to go to Mars as fast as possible and save humanity permeates every part of the company,” says Tom Moline, a former SpaceX engineer. “The company justifies casting aside anything that could stand in the way of accomplishing that goal, including worker safety.”

Ray Kurzweil is an American computer scientist involved in fields such as optical character recognition (OCR), text-to-speech synthesis and speech recognition technology. In December 2012, Google hired Kurzweil in a full-time position to "work on new projects involving machine learning and language processing". Beginning in 2025, he has served as Chief AI Officer for Beyond Imagination, a humanoid robotics company he co-founded in 2013. 

By 2045, he says, “we will multiply our effective intelligence a billion fold by merging with the.  intelligence we have created” — an event Kurzweil and others call “the Singularity.” He claims that  we won’t experience 100 years of progress in the 21st century — it will be more like 20,000 years of progress. He feels that by 2029, we will have reversed the aging process so that you will actually go backwards in time [i.e., get younger as time passes].

He feels that if he dies by 2029, it “would be a setback,”  but something can still be done. He says that he’d just leave instructions for the post-Singularity robots to sort through his belongings and scan his frozen corpse and bring a version of himself back to life. To this end, Kurzweil has joined the Alcor Life Extension Foundation, a cryonics company. After his death, he has a plan to have cryoprotectants, vitrified in liquid nitrogen, passed through his veins and stored at an Alcor facility till future medical science can revive him. 

Probably the most influential tech centered ideology that few of you have ever heard about is longtermism. They assert that there could be so many digital people living in vast computer simulations millions or billions of years in the future that one of our most important moral obligations today is to take actions that ensure as many of these digital people come into existence as possible.

They believe that Artificial General Intelligence (AGI) presents an incalculably greater threat than even climate change. Many of them that the greatest long-term human benefit is to invest more heavily in AGI risk and AI safety research. Yet it is not the safety of living humans today, or even the next generation, that drives longtermists. Their argument is that AGI in the far future could theoretically kill even more of the digital people than the people who are alive on the planet now. Preventing future AGI from doing this is a more rational use of resources than funding clean energy tech, food security, or public health today. 

Related claims made by some longtermists include the idea that saving the future requires directing even more of today’s resources toward the wealthy in the Global North, rather than sending aid to the most impoverished countries in the Global South, since wealthy people in already disproportionately wealthy regions are better positioned to use their resources to fight these 'existential' risks from AGI. 

If you think these are fringe ideas of a few lunatics, think again. Longtermists have directly influenced reports from the secretary-general of the United Nations; a longtermist is currently running the RAND Corporation; they have the ears of billionaires like Musk.  He has explicitly said that longtermism “is a close match for my philosophy”. (Now you can guess why he destroyed USAID.) It has a huge following in the tech sector. It is the language of moral thought spoken in a growing number of the wealthiest and most powerful political and industrial circles. Billionaires like Jaan Tallinn (an Estonian computer programmer and investor who helped in the development of Skype) and Peter Thiel have been described as avid supporters. 

A longtermist and Senior Research Fellow at the now defunct Oxford’s Future of Humanity Institute (FHI) Toby Ord notes in his bio that he has advised “the World Health Organization, the World Bank, the World Economic Forum, the US National Intelligence Council, the UK Prime Minister’s Office, Cabinet Office, and Government Office for Science.” Ord was quoted to the UN General Assembly in a 2021 speech on climate by former British PM Boris Johnson.

These billionaires think the utopias that they dream about will be good for everyone. They also claim it’s based on sound science and they seem to earnestly believe in such a future. It allows them to portray the growth of their businesses as a moral imperative and to justify nearly any action they might want to take. Today’s harms like the carbon footprint of Amazon’s shipping network are irrelevant since a future AGI will solve it easily. Their visions of the future are news.

I was astonished when I recently came across a prescient comment by Gandhi: "The chief characteristic of [the modern] malaise is the obsolescence of man himself. In fact, it is worse than obsolescence. In the years and decades to come, man will be looked upon as something undesirable, as a burden on both society and nature, straining the management capacities of the former and the life­sustaining resources of the latter."

Technophile hallucinations - I of III

Many of the wealthiest and most influential people in the tech industry have visions straight out of science fiction: people’s minds uploaded into computers, a relentlessly expanding empire spanning the stars, all needs satisfied, all desires satisfied through the power of unimaginably advanced technology. The tech billionaires regard this as the only good option aside from extinction. They claim that this is just the future as revealed by a close study of technology and its development.

The technology wizards that many seem to admire seem to have spooky views like transhumanism and posthumanism which seek to use technology to transcend the limits of the human condition. Transhumanists think of using biomedicine, body engineering, devices and algorithms to improve performance and capabilities. Posthumanists dream of a hybridization of human beings, machines and the environment, and imagine a time when humanity surpasses itself in a new evolutionary stage. 

Eliezer Yudkowsky is the cofounder of the Machine Intelligence Research Institute (MIRI), a controversial artificial intelligence think tank. He’s an AI alarmist but envisages a “glorious transhumanist future” if we can avoid extinction by superhuman AI. He believes that we can and should use advanced technology to transform ourselves, transcending humanity and becoming something more. That includes ending illness, aging, death and increasing intelligence and other mental capacities. 

He says, “If you imagine something that’s worse than mansions with robotic servants for everyone, you are not being ambitious enough”. The one thing that could bring this glorious future is the advent of artificial general intelligence (AGI): machines that can outperform humans at any task. He thinks such a machine may not be far off. “My sense is that we are zero to two breakthroughs” away from AGI, he says.

Sam Altman, the CEO of OpenAI, the company behind ChatGPT, agrees with Yudkowsky that AGI is inevitable and could be coming soon but he sees a radically different future. In an essay on his website, Moore’s Law for Everything, he says, “The technological progress we make in the next 100 years will be far larger than all we’ve made since we first controlled fire and invented the wheel." In More Everything Forever, Adam Becker writes about the scale of ambitions he seems to indicate in the essay: 

Altman apparently wants to make the United States into one enormous company town, with shares in OpenAI replacing the dollar. The US government would become, in effect if not in law, a division of the company, responsible for disbursing company dollars to us, the public. This would, Altman hopes, encourage us to think of OpenAI’s success as America’s success and as our own — Altman explicitly makes this identification in his essay. 

All products would come from OpenAI in his proposed future, because in that future AI does literally everything, meaning that the company dollars can only be spent at the company store... Thus, Altman’s promise of goods halving in price every two years would depend solely on his goodwill, because things will cost whatever Altman and the OpenAI board want them to cost. This is a proposal for total capture of the national economy, making Altman functionally the king of the United States and possibly the world.

Marc Andreessen has been one of the central figures in the tech industry for more than thirty years. In early 1993, he suggested a new feature in a software project out of CERN that was only about two years old at the time, known then as the “WorldWideWeb". Andreessen's innovation launched the modern Web. In October 2023, he posted a Techno-Optimist Manifesto pushing the idea that technology could cure essentially all the world’s ills.

He claims that slowing down progress on AI is “a form of murder,” because he thinks an advanced AGI would inevitably save lives. He claims that an intelligence explosion is coming. “Technology is the glory of human ambition and achievement, the spearhead of progress, and the realization of our potential,” Andreessen wrote in 2023. He seems to think that the mildest regulations of technology is a sort of terror campaign against the likes of him. 

He thinks that more people in existence is always good, saying, “Our planet is dramatically underpopulated,” contending that Earth could easily” support more than fifty billion of us, and that humanity’s numbers will surge far beyond that as we take to the stars. “Not growing is stagnation, which leads to zero-sum thinking, internal fighting, degradation, collapse, and ultimately death.” For any objections to this assertion, his retort is “shut up and multiply”. This principle has been repeatedly defended by Yudkowsky on his blog LessWrong.

Jeff Bezos has repeatedly said that he wants a trillion people living in space to enable a future of perpetual growth, lest we “stagnate” here on Earth. “I have won this lottery, it’s a gigantic lottery, and it’s called Amazon.com. And I’m using my lottery winnings to push us a little further into space,” Jeff Bezos said in 2017. “We need to build reusable rockets, and that is what Blue Origin is dedicated to… taking my Amazon lottery winnings and dedicating [them] to [that].… It’s a passion, but it’s also important."