The Right Coast

February 08, 2006
Political Experts
By Mike Rappaport

Interesting interview at TCS of Philip Tetlock, who has just published the book "Expert Political Judgment, How Good Is It? How Can We Know?" The book is "the result of years of research during which he worked with almost 300 experts known for commenting or offering advice on political and economic trends." Here are some interesting excerpts from the interview:

Tetlock: And one of the first things we discover is that there is a tendency for experts to claim to know more than they do about the future.

The second thing you notice is that some experts are much more prone to be over-confident than are other experts. And that's where we get into our classification of experts in terms of their styles of reasoning as either hedgehogs or foxes.

Schulz: So you divided your experts into camps using the famous Isaiah Berlin distinction that hedgehogs know one big idea and they confidently stick with it, applying it to any and all or various scenarios, and foxes are more flexible and skeptical. And what did you find in comparing those?

Tetlock: Well, the key finding is a bit on the complex side. It's that if you have a hedgehog style of reasoning and you have a very strong ideological commitment to a point of view and you're making long-term predictions, you're at serious risk of falling off a cliff in terms of your predicted accuracy. You pay a price for having the combination of a very theoretically focused cognitive style, a very strong theoretical set of commitments associated with that style, and looking at the more distant future where there is more opportunity for our biases to come into play.

One of the curious things about my experience in this study is that those experts who tend to be more doubtful that they could predict anything, were actually somewhat better at predicting. So modesty was actually a useful cue for accuracy in this context.

Schulz: And these folks would have been foxes or …

Tetlock: They're more likely to be foxes. Foxes are more likely to believe that there's a certain amount of uncertainty and indeterminacy in the world, that the world just oscillates sometimes in violent unpredictable ways.

[These findings raise] the interesting question of [why experts do so poorly]. And one possibility is that people aren't selecting experts primarily on the basis of the truth value of their pronouncements. People are selecting experts on other grounds and what might those other grounds be?

They're suggested in a lovely book Richard Posner wrote on public intellectuals. One is that the experts are providing what he called solidarity goods and the other is experts are providing entertainment goods. Now, a solidarity good is when it doesn't really matter whether the experts are right or wrong. The expert is affirming your values and the expert is doing a good job mashing the other side, affirming your side and making you feel good about your world view. So that would be a kind of a solidarity function for instance, better than a truth-seeking function. And an alternative function, a third function would be simple entertainment value.
A fascinating interview. Read the whole thing.

Update: It is interesting to compare the accuracy of political experts with those of scientific or medical experts. My sense of scientists is that they follow a strong norm of being cautious, something that would certainly help with the political experts. Medical experts, though, strike me as prone to many of the same biases as the political experts. Witness, for example, the most recent study debunking the fat is bad view of the 1980s. Now, what business did all of those people have making that mistake.

Finally, consider global warming scientists. My sense is that there is great pressure on scientists to agree with the "consenus" on global warming. Very little caution there. And hence my skepticism about is claims.