We try things, we reflect, we backtrack. We break complex tasks into smaller sub-goals. And if we mess up, we rework the plan and try new things. Companies suggest that AI can do similar things. OpenAI is pushing models that think step by step and Meta is building agents that act, plan, and adapt. LLMs don't just output a final answer, they now include steps. They will first identify the goal, then consider constraints, and then tell what the output should look like.
This style is called chain of thought reasoning. And these methods often work. Performance improves on math problems, logic puzzles, coding tasks, so people assumed that this must be a form of intelligence. But just because a model can talk through a problem, it doesn't mean that it understands the solution. It might just be mimicking what reasoning sounds like. The question is 'Is AI really learning how to reason, or is it just getting better at pretending?'
Apple published a paper called 'The illusion of thinking'. They tested models like Claude, GPT-4, DeepSeek on a logic puzzle. Initially with increasing complexity, every model did fine. But as they increased the complexity, performance didn't just drop. It crashed. Not a single model solved the puzzle. Instead of writing more steps or reasoning harder, they gave shorter, dumber answers. You'd expect a real reasoning engine to try harder when the puzzle gets harder. But these models do the opposite.
Apple called it "Accuracy Cliff". Performance falls off a ledge and never recovers. Even when the model is given plenty of opportunities, it just stops thinking. That's the moment the illusion breaks. What we thought was reasoning might just be pattern matching, replaying steps it saw during training but with no deeper logic to generalize beyond that. Apple tested the same models on different puzzles but it was the same story. Once complexity crossed a certain point, reasoning models failed miserably not just in accuracy but also in effort.
These models aren't scaling their reasoning. They are not discovering general strategies. They are not applying rules like humans would. They are mimicking what reasoning looks like, but only when the patterns are familiar. Throw something new at them and they break down. Models like ChatGPT4 or Claude are predicting the next word, one word at a time based on training data. In some cases, the steps didn't match the final answer. They were hallucinating the thought process.
Anthropic even confirmed this in their own paper. They found models often hide their shortcuts. They pretend that they worked out the problem logically. So even when you ask them to show their work, you might be seeing fiction. For years, the entire AI industry was betting on the scaling law - The bigger the model, the more data it sees, and the smarter it gets.
This approach worked for a while and each leap brought real gains. And that's why Nvidia's stock is soaring and why hyperscalers like Google, Microsoft, and Amazon are building massive infrastructure for inference at scale. But if reasoning models don't actually deliver, if their performance collapses on real-world complexity, then this boom may be built on a fragile foundation. Apple's paper challenged the assumption that thinking AIs are right around the corner. And if that's not true, then a lot of current investment might be premature.
But not everyone agrees with Apple's take. Anthropic responded with a paper titled 'The Illusion of the Illusion of Thinking'. They argued that Apple's tests were too constrained, the models weren't allowed to use tools, code, or other things real-world agents rely on. In their words, Apple might have set the puzzle up to fail. And sure, when given more flexibility, some models did improve.
But even then, Anthropic didn't deny the deeper issue. These models still don't generalize well. So while they disagreed on the test setup, they didn't exactly say the illusion was fake. Right now, what we have is impressive. But it may just be the world's most powerful pattern matcher, wearing a reasoning mask.
This is actually not surprising in the light of a paper from Vishal Sikha, former CEO of Infosys, board member at Oracle and BMW. He mathematically proved that an AI agent will never do what Silicon Valley is promising. Not 'probably won't', not 'might have limitations'. His argument is simple. LLMs can only perform a certain number of computations per response. That number is fixed. If a task requires more computation than that ceiling allows, the model will either fail or hallucinate.
When you send a prompt to ChatGPT or Claude or Grok or any of the current frontier models, the model will do a fixed amount of work to generate each word as an output. Every word in your prompt needs to look at every other word to understand the context. Every word in every answer has the same budget. A simple hello gets the same number of operations as a complex physics problem. That's the ceiling. It's not about better hardware, it's about the architecture of how the system actually works.
An LLM physically cannot do that math in one shot. It guesses, it pattern matches and gives you something that looks plausible. Every AI demo you've ever seen was running tasks designed to stay under the necessary complexity ceiling. He isn't saying these tools are useless. He's just saying that they're being marketed as reasoning engines when the math proves they're actually pattern mirrors. If a task needs more steps than the model can perform, it will unavoidably hallucinate.
More recent models have got better at it, but for certain problems, hallucination is the only possible output. If you have a fixed amount of thinking power per word, giving the AI more steps is like giving a writer more sheets of paper. Each individual sheet is still the same size. You haven't made the writer smarter. You've just given them more room to ramble off topic. The math says that for complex problems, errors eventually compound.
For the right applications, current AI, the current frontier models are exceptional. Writing drafts, summarizing, reformatting data, research and comparison, these tasks stay under that ceiling. The problem is the gap between reality and the promise. That AI agents will autonomously run your business is a lie. The singularity probably isn't as close as people keep saying. If Open AI was about to hit AGI, why would senior engineers be leaving to start risky startups? They're starting companies that use AI as a tool, not companies that use AI as a God.
No comments:
Post a Comment