The limits of AI - I of IV

An AI managed to solve a problem from the 1940s from the Hungarian mathematician, Paul Erdős, which was called the unit distance problem. He produced a formula for the problem and AI came up with a proof that disproved his formula. So this isn't just confirming something that others have thought about before but it came up with a proof that this formula was wrong. Moreover, it came up with a method to prove this, that no one in that field of mathematics has ever thought of. Mathematicians a year ago didn't think that AI could ever do this.

An AI is a pattern recognition device. We don't exactly know how it works because of the complexity of how it arrives at certain conclusions. We are overwhelmed by the sheer number of calculations and the sheer number of data. So we cannot simply watch in real time how it arrives at its results. But it's still a pattern recognition device. But if pattern recognition allows us to prove something that we sort of put in the realm of logic or creativity or genius or intuition, that's quite a serious development. 

You can turn to any discussion topic. And the AI will be very confident and fluent and seemingly knowledgeable. It seems dazzling making people very excited. But there are some very big caveats about this technology. One of the ideas that people have about AI is that it's a kind of flawless super brain. But actually, it's not right a lot of the times. 

When people talk about AI, they often mean large language models (LLMs). But LLMs are just one small part of a much bigger picture. There are a huge range of different applications of the technology. Take Google Translate, for example, that nowadays is driven by neural network technology. It still makes grave mistakes that a human never would.

Consider the phrase “I stood in the pen" Most people imagine me in an enclosure of some kind. They do not imagine that I am inside a writing “pen” which sounds a bit ridiculous. However, as of today, if you type “I stood in the pen” into Google Translate to convert the sentence into Malayalam, it interprets "pen" as the writing instrument. It doesn’t seem to “get” which kind of pen the phrase refers to.

Now, if you type “The cow is in the pen,” Google gets it right, correctly picking the Malayalam word for cattle enclosure. This is because Google uses the context (“cow”) to interpret correctly the word “pen” (whereas “I” didn’t hint at the farm context). How can AI seem to be getting smarter by the day yet still make such silly mistakes?”

It doesn't know what the truth is. It's trying to produce plausible answers to the questions that you tell it, which may be correct. But they may well be incorrect. When you ask a question, it's not thinking "What is the right answer to this question?" There's no thinking at all. It's been trained on a huge amount of training data. And what it's trying to do is to produce the most plausible sounding answer. 

Automated translation models are built by scanning thousands of text documents written in multiple languages. The computer automatically searches these documents for ways in which words are most often translated. For instance, the computer learns that when the word “pen” appears in English with the word “cow” nearby, the corresponding Malayalam text very often uses the word “thozhuthilaanu” (in the cattle enclosure). So, if this happens often enough, a rule is added to the model to output “thozhuthilaanu” for “pen,” provided that “cow” appears nearby.

As of today, machines learn to translate by finding common patterns in previous translations done by people. But this doesn’t give them the broad knowledge of the world required to truly excel at the tasks. So, machines may dupe us for a while because they’re good pretenders, but at some point, they end up making silly mistakes a human would never make.  

This is essentially how an LLM like ChatGPT also works. For making an answer, it will make a prediction about the likeliest next word to appear, and then it will repeat that process and that will generate your response. They learn the probability that a word will appear given a particular prompt. To predict which word will occur given a particular sequence of words, you need unimaginable quantities of training data and unimaginable quantities of compute power. 

This is what we knew about GPT-3 which was the breakthrough large language model: 

• 175 billion parameters. A parameter in a neural network is basically either an individual neuron or a connection between them. Each of those takes about four bytes to store.
• For the training data, it required 500 billion words. You're talking about thousands of human lifetimes to read that number of words.
• And to process all of that data, you would take thousands upon thousands of years on a regular desktop computer. So what you need is AI supercomputers with  GPUs, typically provided by NVIDIA, running for weeks. 

So scale is important to be able to make language models work. It just doesn't work if you don't have sufficient training data and sufficient compute power and neural networks to be able to make it work.