Palace of illusions

Sometime back, I read The Palace of illusions which is a novel based on the Mahabharata from Draupadi's point of view. Draupadi is the narrator of the story. It had raised the hackles of some Hindu groups which was the reason why it came to my notice and decided to read it - a minor illustration of the Streisand effect.

It was not difficult to find  passages that would have annoyed some groups. For eg., when her brother Dhrishtadhyumna's tutor says that the primary duty of  a woman is to support her father, brother, husband and sons, Draupadi tells her brother, 'And who decided that a woman's highest purpose was to support men? A man, I would wager! Myself, I plan on doing other things with my life.'

About fortune-tellers, Dhai Ma (who is the nurse of Draupadi, a character invented by the author) says, ' Fortune-tellers are always predicting weddings. They know that's what foolish girls want to hear. That's how they get fatter fees.'

The book is worth a read. I found it the most interesting version of whatever I have read of the Mahabharata. It humanises deified characters and gives them qualities that one relates to, for eg., the mother-in-law - daughter-in-law psychological tussles between Kunti and Draupadi or the steady deterioration of moral values as a war proceeds. Most interesting is the depiction of a soft corner that Draupadi always had for the most tragic hero of the Mahabharata, Karna.

This was something I had not come across earlier. The author got the idea from an incident described in a Bengali version of the Mahabharata. The incident itself does not form part of the book but is described by the author in this talk about the book.It happens after the Pandavas were exiled following Yudhishtira's loss in a game of dice.

While they were  traveling through a forest, they saw a tree laden with fruits from which they plucked one fruit. At this point Krishna appeared and told them that the tree belonged to  a great sage who had a bad temper. When he finds out that they had plucked a fruit from his tree there was no knowing what curse he might put on them - it might even mean the  death of the Pandavas. A frightened Draupadi asked him how to atone for the misdemeanor.

Krishna said that as atonement, each person should tell his or her deepest secret. Each of the Pandava brothers reveals his deepest secret and each time, the fruit rises part of  the way towards the tree. It was only a little distance away from the tree when it was Draupadi's turn but when she revealed her secret, it dropped to the ground. Krishna said that Draupadi had not revealed her deepest secret and asked her to try again. But the fruit did not rise - it was not her deepest secret. Finally she confesses, 'I always had a soft corner for Karna.'


I read that the author is working on a novel based on the Mahabharata  from Sita's point of view.It will definitely be part of my antilibrary.

Analysis of disasters

After every disaster - earthquakes, floods, industrial accidents, terrorist attacks...-there will be breathless coverage for days on end with lots of expert analysis. They quickly become rather tiring.

In The Black Swan, Nassim Nicholas Taleb suggests doing a thought experiment. Suppose there was a far-sighted manager who had implemented some years ago the safety measures that are now being discussed by the media. He would have been told that he was wasting scarce resources in superfluous areas; the scenario that he had painted would be dismissed as a figment of his imagination - it has never happened before. He would have got a bad annual review since his department had 'squandered resources on non-productive expenses'.

If he continues along his 'foolish' path, he may lose his job. He may be replaced by a 'dashy-pushy' (see note below) guy who has more 'Confidence in the future' (i.e. who ignores the possibility of Black Swans) who will be obsessive about buzzwords like efficiency/cash-flow/ bottom line etc. and reduce the 'unproductive expenses'. He will focus on 'leveraging intellectual capital and intangible assets' to create a 'knowledge-based' firm. (Add a few more buzzwords to impress CNBC) And suppose some of the 'unproductive expenses' had been retained and they had helped mitigate the effects of the disaster that would have happened some years after they had been implemented, the far-sighted manager who had risked his career over them would have been long forgotten.

Accidents often happen because of seemingly trivial faults and minor malfunctions that had been overlooked. Small faults like a tiny leak or a rusted bolt that had been routinely picked up earlier would now be missed by the fewer number of over-worked employees that had resulted from Downsizing/Layoffs/Ramping down an operation/Right sizing/Restructuring etc. to 'trim costs' in order to 'remain competitive'. (The 'flattening of the organizational pyramid' would be to take 'nimble advantage of market nuances'.) The mistake that caused the accident may be the final straw on the camel's back. Taleb writes in The Black Swan:

Who gets rewarded, the central banker who avoids a recession or the one who comes to 'correct' his predecessors' faults and happens to be there during some economic recovery? Who is more valuable, the politician who avoids a war or the one who starts a new one (and is lucky enough to win)?

[SNIP] 

...everybody knows that you need more prevention than treatment, but few reward acts of prevention.  We glorify those who left their names in history books at the expense of those contributors about whom our books are silent.  We humans are not just a superficial race (this may be curable to some extent); we are a very unfair one.

Hindsight bias, also known as the knew-it-all-along effect, is the inclination, after an event has occurred, to see the event as having been predictable, despite there having been little or no objective basis for predicting it. After an event, people often believe that they knew the outcome of the event before it actually happened. They often forget a dictum that one historian stated - what is now in the past was once in the future -and assume that a decision-maker at the time had the same information that they have now.

In his book, Thinking, Fast and Slow, Danil Kahneman give an example of hindsight bias. The day before the attacks on WTC in the US, the CIA got information that al-Qaida may be planning a major attack in the US. This information was given to the NSA rather than to President Bush. When this became known later, the executive editor of The Washington Post said, 'It seems to me elementary that if you've got the story that's going to dominate history you might as well go right to the president.' But the day before the attack, no one knew - or could have known - that the next day would 'dominate history'.

After a terrorist attack you will often be told that the suspect had been in some police record somewhere for some petty crime. The implication will be drawn that if there was better coordination between the different agencies, the person would have been caught then and the terrorist incident would not have happened. But there was no way for the police to know that he would plant a bomb in a bus a few months later.

Note: I came across the word 'dashy pushy' in an article in The Caravan magazine. It is a corrupted combination of two English words and is used in West Bengal:

By chopping the last three letters off “dashing,” and adding a “y” to ease its coupling with “pushy,” we get a new word. It denotes a go-getter with an unsubtly aggressive edge about him — a slightly pejorative term in its early days, but now one of approval, if not admiration. 

Reunion at NIT, Trichy

The NIT, Trichy batch of 1990 (to which I belong) held its reunion in December last year. I decided to go for the reunion since it gave me the chance to meet many friends who I had thought I will never see again. The same contingent that had traveled with me for the IIMA reunion accompanied me this time too - Jaya, Sujit, my brother-in-law and the nurse (but this time it was a different nurse).

We decided to first go to Pudukkottai and stay overnight at the house of Kamala, the person who had been my physiotherapist for 4 years soon after my stroke. She had subsequently got married and settled down in Pudukkottai. We thought we could kill two birds with one stone -  we could meet Kamala and her family and since NIT was only about an hour's drive away, we would reach the campus refreshed after a night's rest.

We reached the campus a little while before the group photo session was to start. It felt good to meet old friends after such a long time. Like at the IIMA reunion, I felt that my lack of speech was a blessing since it gave me time to recognise some people. They will introduce themselves, I will look at their faces...and yes, their faces did resemble the faces I remembered from 25 years ago.

Some folks were well on their way to becoming the sort of men that Ceasar liked to have about him. Remember that he liked men about him that are fat, having a dim opinion of the lean and hungry look of yond Cassius. He would have liked what he saw at the reunion.

After the photo session, there were some formal sessions after which everyone went to visit the hostels where we had stayed. I did not go since it would have taken me a lot of time time to get there. Instead I preferred to stay in the hall where the next program was to be held. I requested a friend to take Sujit and my brother-in-law to see the hostel where I had stayed. I thought of the ragging time when I had stayed in a hostel (this was in a different hostel from the one Sujit had visited).

At the time the common practice among seniors was for North Indians to rag South Indians and vice-versa. I was conversant with a North Indian and a South Indian language so I decided to try out an idea. Whenever North Indian seniors came, I always said that I was from Jamshedpur and spoke only in English/Hindi. At the slightest opportunity, I brought an Amitabh Bacchan movie or song into the conversation and then everything was quite pally.

Whenever South Indian seniors came, I always said that I was from Palakkad and spoke only in English/Malayalam. I also used the smattering of Tamil that I knew at the time. (There were some Tamil Brahmins in my village, Tamil films and songs are popular in the area and there is some similarity between Malayalam and Tamil languages so I knew some Tamil.) The result was that I hardly ever got ragged.

I used to be surprised that my trick was never found out. It helped that I was an unremarkable, low-profile guy who would not have figured in any conversations. There was also the fact that I did not have a readily identifiable accent.

With my batch mates

With my Mech. Engineering classmates

Sujit standing in front of the hostel in which I used to stay

The illusion of skill - III

Pundits on TV are another who confidently predict future events that are essentially unpredictable. TV experts make it sound as if predicting the future is only slightly more messy than solving mathematical equations. And they rarely mention the word 'luck'.  As Kahneman says in his book, Thinking, Fast and Slow 'The idea that large historical events are determined by luck is profoundly shocking, although it is demonstrably true.' History is full of low predictability and large impact Black Swan events that are predictable only in retrospect. History looks more easy to explain when viewed backwards than it does when the events are actually happening, what Nassim Nicholas Taleb calls 'the narrative fallacy'. As Taleb says in The Black Swan:

Our inability to predict in environments subjected to the Black Swan, coupled with a general lack of the awareness of this state of affairs, means that certain professionals, while believing they are experts, are in fact not.  Based on their empirical record, they do not know more about their subject matter than the general population, but they are much better at narrating -  or, worse, at smoking you with complicated mathematical models. They are also more likely to wear a tie.

 Philip Tetlock, a psychologist at the University of California, Berkeley selected 284 people who made their living "commenting or offering advice on political and economic trends," including journalists, foreign policy specialists, economists and intelligence analysts, and began asking them to make predictions. Over a couple of decades, he asked them to rate the probability of outcomes of several questions: Would George Bush be re-elected? Would apartheid in South Africa end peacefully? Would Quebec secede from Canada? Would the dot-com bubble burst? Overall he had over 80,000 predictions.

How did the experts do? When it came to predicting the likelihood of an outcome, the vast majority performed worse than random chance i.e, they would have done better picking their answers blindly out of a hat. Liberals, moderates and conservatives were all equally ineffective. Most of the subjects had post-graduate degrees but they were mostly useless when it came to forecasting. Even in the region they had most knowledge of, the experts were not much better than non-specialists.

The main reason for the inaccuracy has to do with overconfidence. Because the experts were convinced that they were right, they tended to ignore all the evidence suggesting that they were wrong - they had an enhanced illusion of their skill. Those with the most knowledge were less reliable. This is because these experts were cocooned in their area of specialisation and tended to view the world through a narrow lens, what Nassim Nicholas Taleb calls 'the philistinism of the over-specialised scholar'. It is like the blind who touch different parts of an elephant and conclude that it is like a rope, a pillar, etc.

Tetlock also found that the experts were resistant to admitting error when it was pointed out to them, offering a number of excuses for their mistakes. The problem is that the over-specialised expert who can come up with the catchy one-liner is more likely to be invited to TV studios since  he is more interesting to listen to than the expert who uses a lot of 'ifs' and 'buts' even though the latter may be closer to the truth. The preferred expert will be the one who gives short, snappy answers to a screaming host who demands, 'India wants to know.' A safe rule of thumb to follow is to ignore the views of the experts who sound very confident about their forecasts. As Kahneman says in his book:

The first lesson is that errors of prediction are inevitable because the world is unpredictable.  The second is that high subjective confidence is not to be trusted as an indicator of accuracy (low confidence could be more informative).

P.S :  The Trouble With Experts

The illusion of skill - II

Although professional investors are able to beat amateurs, Kahneman says that few, if any, have the skill to beat the market consistently. The logic for testing this is simple: if the rankings of the funds in any one year is entirely due to luck, then the year-to year correlation of their rankings should be zero. But if skill is involved, then the rankings will be more stable and therefore there will be year-to-year correlation, showing persistent achievement. This is similar to the rankings in any game, say tennis. If the rankings are random, then there will be no year-to-year correlation. But the top players are consistently ranked at the top so there will be year-to-year correlation of their rankings.

More than 50 years of research has shown that professional investors are playing a game of dice, not of skill. The year-to-year correlations are barely higher than zero. The traders think that they are making sensible, educated guesses but Kahneman says, 'In highly efficient markets, however, educated guesses are no more accurate than blind guesses.'

In Thinking, Fast and Slow, Kahneman relates a story of a speaking assignment that he had with a firm of investment advisers. He was given a spreadsheet having the investment data of 25 wealth advisers for 8 consecutive years which determined their year-end bonuses. He proceeded to calculate the correlation coefficients between the rankings in each pair of years giving a total of 28 correlation coefficients.If there was skill involved then there should be significant correlation between the year-to-year rankings of the advisers.

What Kahneman found was that the average of the 28 correlations was .01 which was statistically equivalent to zero. It meant that the wealth advisers were playing a game of chance, not of skill. No one in the firm including the advisers were aware of this reality. They all thought that they were doing a professional job. But even after being shown that the firm was rewarding luck not skill (a result that they couldn't deny because they had provided the data and could check the results) Kahneman was sure that it would be forgotten immediately.

He thinks that the key reason for the persistence of this illusion is because the traders are indeed using a type of skill. It requires a lot of hard work and training to be able to check economic forecasts , read P&L statements and balance sheets, study competition, etc. But is this skill enough to answer the key question: is all this information already incorporated into the stock price? Here traders switch to feelings and the data shows that it becomes a guessing game.It does not give one correct answer - except in retrospect, which is not very useful. But after all that hard work, traders become resistant to admitting this.

It is like what happens when I type. Sometimes, after typing a few sentences, I will feel that they are not quite fitting into the rest of the post. The logical thing to do would be to delete them. But after having spent all that time and effort in typing those sentences, I don't feel like seeing them disappear into the ether in less than a second. So I think of some excuse to retain them in the post and I generally succeed.

Kahneman adds:

...the illusions of validity and skill are supported by a powerful professional culture. We know that people can maintain an unshakable faith in any proposition, however absurd, when they are sustained by a community of like-minded believers. Given the professional culture of the financial community, it is not surprising that large numbers of individuals in that world believe themselves to be among the chosen few who can do what they believe others cannot.

If there is a discussion with Kahneman on CNBC, I would be a very interested listener,  although the chances of that happening are quite remote. You will not be happy if someone walks into the studio (an economics Nobel Prize winner at that) and says that the self-styled 'Masters of the Universe' are being paid because of their luck and not because of their skill. (But Kahneman seems a polite person so he may have some nice way of saying that they are fooling themselves.)

PS: Intuition - Marvels and Flaws. Daniel Kahneman, Nassim Taleb, Gillian Tett.

The illusion of skill - I

In every work of genius we recognize our own rejected thoughts: they come back to us with a certain alienated majesty. - Ralph Waldo Emerson

Daniel Kahneman won the Nobel Prize in economics even though he is a psychologist, not an economist. His two papers (along with his colleague Amos Tversky) - "Judgment under uncertainty: Heuristics and biases" and "Prospect theory: An analysis of decisions under risk" - had a big impact in diverse fields like economics, philosophy, military strategy, etc. When he was once asked to summarise his work in seven words, he said that five will do - 'Endlessly amused by peoples' minds.'

His book Thinking, Fast and Slow questions the peculiar assumption that most economists have that actors in the economy are rational and selfish. (If they were, I wouldn't have survived for 16 years in relative comfort.) It is useful to assume that humans are sometimes part of the mix. Assuming human rationality is like assuming a frictionless surface in introductory physics: it is fine for introducing some basic concepts before complications are introduced into the model for a closer approximation to reality. As Kahneman writes in his book while discussing prospect theory:

The standard concepts and results that undergraduates are taught are most easily explained by assuming that Econs do not make foolish mistakes.

[SNIP]

In some contexts, however,...the Humans described by prospect theory are guided by the immediate emotional impact of gains and losses, not by long-term prospects of wealth and global utility.

(In his book Nudge, the behavioural economist Richard Thaler says that economists and psychologists seem to be studying two different species: Econs and Humans. The Econs of economists 'can think like Albert Einstein, store as much memory as IBM's Big Blue, and exercises the will power of Mahatma Gandhi'. But real people or Humans studied by psychologists 'have trouble with long division if they don't have a calculator [esp. if you are American - Suresh], sometimes forget their spouse's birthday, and have a hangover on New Year's Day'.)

Kahneman is not impressed with the stock-picking skills displayed by investors. He says that an entire industry has been built on an illusion of skill. Many investors lose regularly by trading, 'an achievement that a dart-throwing chimp could not match'. He cites a study by a student of his, Terry Odean, a Finance professor at UC Berkely, who studied the trading records of 10,000 brokerage accounts of individual investors covering nearly 163,000 trades over a 7 yr. period.

Odean then chose those cases where an investor sold some stock and immediately bought another stock. This showed that the investor expected  the stock that he bought (most investors were men) to do better than the stock he sold. Odean followed the stocks for 1 year after the transaction and found that, on average, the stocks that were sold did better than the stocks that were bought by 3.2%/yr after taking into account the transaction costs. Thus for the majority of investors, it would have been better to do nothing. Kahneman writes:

...it is clear that for the large majority of individual investors, taking a shower and doing nothing would have been a better policy than implementing the ideas that came to their minds. Later research by Odean and his colleague Brad Barber supported this conclusion. In a paper titled "Trading Is Hazardous to Your Wealth," they showed that, on average, the most active traders had the poorest results, while the investors who traded the least earned the highest returns.  In another paper, titled "Boys Will Be Boys," they showed that men acted on their useless ideas significantly more often than women, and that as a result women achieved better investment results than men.

This reminds me of a game that was played by the students when I was at IIMA. The students had some virtual money which they could use to invest in stocks. Whoever ended with the maximum wealth at the end of a given period was the winner. The person who won the game was one who invested in one stock on the first day and didn't do anything else for the remainder of the period. The others, who bought and sold stocks using various strategies, were left behind.

PS: What is obvious is not always true and what is true is not always obvious. In Descartes' Error, Antonio Damasio illustrates why a person with only reason and no emotion struggles to make good decisions. Psychologists refer to emotions as 'lubricants of reason'.  Here is a talk by Damasio on human decision making.


PPS: A talk by Daniel Kahneman on A Psychological Perspective on Rationality

Education is not a panacea - III

There are gated communities all over India where the educated rich live cut off from the rest of the country and cribbing about everything that doesn't resemble Singapore. In Geek Nation, Angela Saini describes one such community in the making, Lavasa -  'a metropolis governed mainly by machines' being built in the middle of the Western Ghats, a region rich in bio-diversity and populated by a few tribal villages. It is a half-billion dollar project that is 'the biggest thing to happen to  the Western Ghats since the Cretaceous Period'.

It is a surreal place  having an American Diner with staff dancing to Elvis tunes, opulent villas, a state-of-art hospital that looks deserted, delicate fountains, a street that looks as if it was in Italy...It sounds as if the promoter has taken the most picturesque parts of Europe and built a collage in the middle of nowhere.The employees say that it 'will be a city that governs itself' using technology, that it can provide a role model for the rest of India.  I got a feeling similar to what Angela Saini had - a 'feeling as if I've arrived in Jurassic Park but the dinosaurs haven't escaped...yet.'

In the biography of Steve Jobs by Walter Issacson, the author says that Jobs often disappeared into a 'reality distortion field' which made him view the world in black and white terms with no shades of grey,an ability to convince himself and others about almost anything without any sense of proportion. Similarly many educated people seem to live in a reality distortion field.An article in The New Yorker about the Indian print media gives an idea of why this is so. The desired stereotype is also promoted by television serials.  Nehru's comment in The Discovery of India may not have been off the mark: "I have not discovered any special qualities in a literate or slightly educated person which would entitle his opinion to greater respect than that of a sturdy peasant..."

On average, the educated and uneducated don't seem to be very different when it comes to basic human values. Knowing more about protons or perfect markets doesn't seem to help in this regard. The decision to extend voting rights to everybody without putting any restrictions on the basis of educational qualification was perhaps the wisest thing that Nehru did. Most people were opposed to the idea of giving voting rights to large numbers of illiterate people. But Nehru over-ruled all objections and went ahead with his decision. And his instinct has been proved right in election after election over the decades.

As soon as Indira Gandhi held elections after the Emergency, she was promptly booted out. The Congress did well in the more literate states in the South who preferred to ignore the horrors of the Emergency. It was highly educated, successful people who were likely to overlook the excesses of the emergency and say that population needs to be controlled somehow. It is educated, rich people who are likely to say that a spell of military rule will bring much needed discipline. (I have heard this, I am not making this up.) Talk of short-sightedness!

Granted there are  problems of inducement and intimidation but unpopular governments have been shown the door at regular intervals. If buying votes was so easy, the ruling dispensation would have been able to hold on to power more easily. I have heard servants say that they will take the money offered by both the main political parties in Tamil Nadu and then vote for whoever they like! As Ramachandra  Guha says in India after Gandhi:

...the distance - intellectual or moral - between Jawaharlal Nehru and Indira Gandhi, or between B.R. Ambedkar and Mulayam Singh Yadav, is not necessarily greater than between, say, Abraham Lincoln and George W.Bush. It is in the nature of democracies, perhaps, that while visionaries are sometimes necessary to make them, once made they can be managed by mediocrities.  In India, the sapling was planted by the nation's founders, who lived long enough (and worked hard enough) to nurture it to adulthood. Those who came afterwards could disturb and degrade the tree of democracy but, try as they might, could not uproot or destroy it.

I remember seeing a video where it was stated that in the airport, the people in the queue for first class passengers look more agitated and prone to anger than the economy class passengers. I saw this video after my stroke so I couldn't check it for myself but it rings true. In India whichever party comes to power will have the majority of people voting against it. Every winning party claims that it has the mandate of the people which is far from the truth.Nehru at the height of his popularity got only 47% of the votes. So no government can risk moving too far away from the centre much to the chagrin of the better off sections of society, who seem impatient like the first class airline passengers.

Chetan Bhagat has written a book called Making India Awesome which I have not read. For all I know, I may  agree with most of its contents. My problem is with the title. In all probability the publisher would have thought (probably correctly) that a title that gives the impression of there being easy, clear-cut solutions to complex problems would result in better sales. It is similar to the BJP's penchant for coming up with MBA style mnemonics like 3 'C's, 4 'D's, ABCD etc.

A more humble title like 'Some Suggestions that May improve India's Prospects' may not sell as well. During sales training in Wipro, an advice was given which I thought was sensible: 'it is better to under promise and over deliver than to over promise and under deliver'. I am probably a misfit in a social ecosystem that encourages simplistic bombast. I heard a great line in a talk by Arun Shourie which illustrates the problem, 'Jo hyper-bole so nihal.'  As Nassim Nicholas Taleb says in Fooled by Randomness:

I do not dispute that arguments should be simplified to their maximum potential; but people often confuse complex ideas that cannot be simplified into a media-friendly statement as symptomatic of a confused mind.  MBAs learn the concept of clarity and simplicity - the five-minute-manager take on things.  The concept may apply to the business plan for a fertilizer plant, but not to highly probabilistic arguments - which is the reason I have anecdotal evidence in my business that MBAs tend to blow up in financial markets, as they are trained to simplify matters a couple of steps beyond their requirement. (I beg the MBA reader not to take offense; I am myself the unhappy holder of the degree.)