McDermott and Shleifer double-team Taleb and Kahneman

Not really. But I still enjoyed reading the following passage from Andrei Shleifer's review of Daniel Kahneman's (superb) Thinking, Fast and Slow:

The fourth assumption of Prospect Theory is quite important. [i.e. In assessing lotteries, individuals convert objective probabilities into decision weights that overweight low probability events and underweight high probability ones.] The evidence used to justify this assumption is the excessive weights people attach to highly unlikely but extreme events: they pay too much for lottery tickets, overpay for flight insurance at  the airport, or fret about accidents at nuclear power plants. Kahneman and Tversky use probability weighting heavily in their paper, adding several functional form assumptions (subcertainty, subadditivity) to explain various forms of the Allais paradox. In the book, Kahneman does not talk about these extra, assumptions, but without them Prospect Theory explains less.  

To me, the stable probability weighting function is problematic. Take low probability events. Some of the time, as in the cases of plane crashes or jackpot winnings, people put excessive weight on them, a phenomenon incorporated into Prospect Theory that Kahneman connects to the availability heuristic. Other times, as when investors buy AAA-rated mortgage-backed securities, they neglect low probability events, a phenomenon sometimes described as black swans (Taleb 2007). Whether we are in the probability weighting or the black swan world depends on the context: whether or not people recall and are focused on the low probability outcome. [Emphasis mine.]

This exactly the issue I was trying to point out here. Sometimes people greatly overweight the risks of low probability events (as suggested by Kaheman and Prospect Theory)... other times they completely underestimate them (as suggested by Taleb's black swan metaphor). As a result, we should be cautious in trying to make generalisable statements about human behaviour from either one of these theories alone.

You may also recall that -- for my temerity in pointing out this apparent tension between Kahneman and Taleb's theories -- I was labelled an "idiot" by none other than Taleb himself. As I coyly suggested in that second post, Taleb's affinity for labelling others as idiotic meant that I was at least likely to be in good company. I am sure of that now having read Shleifer's article.

It's not every day that you're called an idiot by Nassim Taleb

Or a "bloggist" for that matter.

Here and here.

To be fair, Taleb has charged that many minds superior to my own are beset by idiocy, so I'm in reasonable company. More seriously, he did at least tone down his bombast when I pointed out that he had misunderstood what I was asking.

The background is this post, where I wondered (quite respectful like!) what Taleb made of the research that shows people have a tendency to overestimate the likelihood of low-probability events if they were framed in highly dramatic terms. This seemed to run counter to a recurring theme in his writings, which is that people are blind to "black swans"... basically that they consistently underestimate low-prob, high impact events.

Taleb pointed me towards a short paper on "binary" (up vs down) versus "vanilla" (+500 vs +5,000,000 vs -5,000,000) outcomes, which was supposed to refute the relevance of such studies. However, I remain rather unconvinced. Consider the key figure in my previous post:

Perceived versus actual fatalities. Adapted from Lichtenstein et al. (1978).

As I wrote back then: What we see here is that people have a clear tendency to overstate -- by an order of several magnitudes -- the relative likelihood of death arising due to "unusual and sensational" causes (tornadoes, floods, etc). The opposite is true for more mundane causes of death like degenerative disease (diabetes, stomach cancer, etc).

Now, I certainly agree with Taleb that it is important to distinguish between between binary and continuous outcomes. Asking whether a stock will go up/down is a much less interesting (and less complex) question to ask than whether it will go up/down by a certain amount. You are clearly not comparing apples with apples if you say that a stock will go up by 5% or 500%. In short, binary and continuous ("vanilla") outcomes are incommensurable in terms of evaluating payoffs.

However, the studies that I linked to are interesting exactly because they are comparing the *same* outcome (i.e. death). It makes no sense to say that death by tornado equals five times death by stroke. They are obviously equivalent. The "payoff" is thus the same because the outcome is the same. Further, I'm not claiming that the insights from these particular studies are fully generalisable to all other low probability, high-impact outcomes (especially those in finance). Yet they do show that underestimation of black swan events is hardly a universal phenomenon either... In fact, people here are shown to rely on heuristics that lead them to a diametrically opposite conclusion! I was ultimately interested in hearing from Taleb whether he thinks these heuristics are efficient or not. I didn't get an answer unfortunately, so I guess we'll have to judge for ourselves.

A final observation is that I disagree with the paper's assertion that "binary is limited to probability". (In other words, that binary outcomes say nothing about the size of a payoff.) This is certainly true in many cases -- again, especially in finance -- but not always. In some instances, binary outcomes imply payoffs directly. The obvious example is the one that we have been discussing in this very post, i.e. death. Indeed, I would think that Taleb probably agrees with me, given that one of his favourite analogies is that of a turkey being fattened up in preparation for Thanksgiving.

What Taleb calls his "classical metaphor". A turkey on his way to becoming dinner. (Source)

With apologies to Monty Python, you might say that the prospect of becoming an ex-turkey implies a very obvious payoff indeed.

UPDATE: Andrei Shleifer agrees.

A question for Nassim Taleb fans

I read an interesting article last night, detailing a public exchange between Daniel Kahneman and Nassim Taleb.

[E]ach man was asked to write a biography of seven words or less. Taleb described himself as: “Convexity. Mental probabilistic heuristics approach to uncertainty.” Kahneman apparently pleaded with the moderator to only use five words, which were: “Endlessly amused by people’s minds.” Not surprisingly these two autobiographies are descriptive of the two men’s bodies of work. Much of the discussion at this event, however, was not about making decisions under uncertainty, but a sort of tit for tat, with Kahneman asking probing questions and making pointed observations of Taleb. Little of the Nobel laureate’s [i.e. Kahneman's] work was discussed.

It would seem that Kahneman had Taleb on the back foot at various times during the exchange, pointing out (among other things) that the latter's framing of situations suffered from a clear "anchoring" bias. 

The above article also reminded me of a lingering question that I have about Taleb's work -- not least of all because it relates to the type of research that made Kahneman famous (i.e. the limits of heuristics in the face of statistical problems). Having failed to get any responses to my query on Twitter, I'd like to try and flesh it out here.

Let me state up front that I have yet to read, in full, any of Taleb's books. (They are patiently waiting on my kindle.) However, I have read several chapters from them and, moreover, a number of the articles that Taleb has penned in different media outlets. For instance, this essay for Edge magazine which seems to nicely sum up his position. 

So, I'm reasonably confident that I know where Taleb is coming from. I should also say that I think some of his points are very well made. Such as the "inverse problem of rare events" -- basically, that it is incredibly difficult to gauge the impact of extremely rare events exactly because they occur so infrequently. We lack the very observations that are needed to build up a decent idea of the probability distribution of their associated impact. As Taleb explains in the Edge essay: "If small probability events carry large impacts, and (at the same time) these small probability events are more difficult to compute from past data itself, then: our empirical knowledge about the potential contribution -- or role -- of rare events (probability × consequence) is inversely proportional to their impact."[*]

My reading of Taleb also leads me to think that he that he more or less regards everyone as blind to "black swan" (low probability, high impact) events. If that is true, however, I'm wondering how he squares that notion with the consistent empirical finding that people tend to overestimate the likelihood of low probability, high impact events. (And vice versa for more common, low impact events.) Consider the following chart, for example, which was originally produced in a seminal study by Lichtenstein et al. (1978):

Relationship between judged frequency and actual number of fatalities per year for 41 causes of death.

What we see here is that people have a clear tendency to overstate -- by an order of several magnitudes -- the relative likelihood of death arising due to "unusual and sensational" causes (tornadoes, floods, etc). The opposite is true for more mundane causes of death like degenerative disease (diabetes, stomach cancer, etc).

Similarly, have a look at Table 2 (p. 19) in this follow-up study by the same authors, where various groups of people were asked to rank the relative risks of different technologies. We clearly see a incompatibility between the opinions of experts and those expressed by laymen. For example, nuclear power is perceived to be far more risky by members of the general public than by those familiar with the actual number of fatalities and diseases brought on by this technology.

Now, Taleb might respond by that saying these are the exactly the type of misleading comparisons that he is talking about! He could argue that the "actual" observed fatalities are not necessarily an accurate representation of the underlying risks. After all, a single major event could significantly alter the average number of deaths of any particular cause (e.g. nuclear meltdown)... 

Well, perhaps, but I'm not totally convinced. For one thing, that says very little about the flipside of this problem, which is the degree to which "normal" causes of death are underestimated -- both in absolute terms and relative to more sensational outcomes. Second, by now we have accumulated decent data on numerous low-probability events that have occurred (rare as they are), from the outbreak of plague to massive natural disasters. Third, even disregarding my previous points, it doesn't seem at all obvious to me that the public is guilty of consistently underplaying the role of black swan events. Indeed, if anything they appear to be using a heuristic which causes them to significantly overestimate the likelihood of rare events.... Perhaps as a way of adjusting for the -- unquantifiable? -- impact that these outcomes could have if they do occur?

To restate my question then to those of you that know Taleb better than myself: Does he ever integrate (or reconcile) his theory about the ignorance of black swan events with the empirical evidence that people consistently overestimate the likelihood of low probability, dramatic outcomes?

UPDATE: This post appears to invoked Taleb's ire in somewhat amusing fashion. See follow-up here.
UPDATE 2: Second follow-up and some big name support of my basic point here.

___

[*] This type of unquantifiable uncertainty happens to be a big area of research in the climate change literature. In particular, the 'dismal theorem' proposed by Marty Weitzman, whom I have mentioned numerous times before on this blog. See here for more.