Wednesday, January 20, 2016

The illusion of skill - III

Pundits on TV are another who confidently predict future events that are essentially unpredictable. TV experts make it sound as if predicting the future is only slightly more messy than solving mathematical equations. And they rarely mention the word 'luck'.  As Kahneman says in his book, Thinking, Fast and Slow 'The idea that large historical events are determined by luck is profoundly shocking, although it is demonstrably true.' History is full of low predictability and large impact Black Swan events that are predictable only in retrospect. History looks more easy to explain when viewed backwards than it does when the events are actually happening, what Nassim Nicholas Taleb calls 'the narrative fallacy'. As Taleb says in The Black Swan:
Our inability to predict in environments subjected to the Black Swan, coupled with a general lack of the awareness of this state of affairs, means that certain professionals, while believing they are experts, are in fact not.  Based on their empirical record, they do not know more about their subject matter than the general population, but they are much better at narrating -  or, worse, at smoking you with complicated mathematical models. They are also more likely to wear a tie.
 Philip Tetlock, a psychologist at the University of California, Berkeley selected 284 people who made their living "commenting or offering advice on political and economic trends," including journalists, foreign policy specialists, economists and intelligence analysts, and began asking them to make predictions. Over a couple of decades, he asked them to rate the probability of outcomes of several questions: Would George Bush be re-elected? Would apartheid in South Africa end peacefully? Would Quebec secede from Canada? Would the dot-com bubble burst? Overall he had over 80,000 predictions.

How did the experts do? When it came to predicting the likelihood of an outcome, the vast majority performed worse than random chance i.e, they would have done better picking their answers blindly out of a hat. Liberals, moderates and conservatives were all equally ineffective. Most of the subjects had post-graduate degrees but they were mostly useless when it came to forecasting. Even in the region they had most knowledge of, the experts were not much better than non-specialists.

The main reason for the inaccuracy has to do with overconfidence. Because the experts were convinced that they were right, they tended to ignore all the evidence suggesting that they were wrong - they had an enhanced illusion of their skill. Those with the most knowledge were less reliable. This is because these experts were cocooned in their area of specialisation and tended to view the world through a narrow lens, what Nassim Nicholas Taleb calls 'the philistinism of the over-specialised scholar'. It is like the blind who touch different parts of an elephant and conclude that it is like a rope, a pillar, etc.

Tetlock also found that the experts were resistant to admitting error when it was pointed out to them, offering a number of excuses for their mistakes. The problem is that the over-specialised expert who can come up with the catchy one-liner is more likely to be invited to TV studios since  he is more interesting to listen to than the expert who uses a lot of 'ifs' and 'buts' even though the latter may be closer to the truth. The preferred expert will be the one who gives short, snappy answers to a screaming host who demands, 'India wants to know.' A safe rule of thumb to follow is to ignore the views of the experts who sound very confident about their forecasts. As Kahneman says in his book:
The first lesson is that errors of prediction are inevitable because the world is unpredictable.  The second is that high subjective confidence is not to be trusted as an indicator of accuracy (low confidence could be more informative).

P.S :  The Trouble With Experts

No comments:

Post a Comment