Guessing Games

Pundits and prognosticators make predictions all the time: about everything from elections, to sports, to global affairs. This week on Hidden Brain, we explore why they’re often wrong, and how we can all do it better.

The transcript below may be for an earlier version of this episode. Our transcripts are provided by various partners and may contain errors or deviate slightly from the audio.


This is HIDDEN BRAIN. I'm Shankar Vedantam.


VEDANTAM: We're surrounded by people who tell us they know what's going to happen in the future.


UNIDENTIFIED MAN #1: A lot of people have no idea that Trump is headed for a historic defeat.

UNIDENTIFIED ANNOUNCER #1: Tom Brady comes in. Let me tell you, this kid's nervous. He is nervous.

UNIDENTIFIED MAN #2: Bear Stearns is fine. Don't move your money from Bear. That's just being silly.

VEDANTAM: These predictions have a few things in common. The commentators have complete confidence in themselves. We, as the audience, love to hear them make a complicated world seem simple. And finally, no one ever pays a serious price for being wrong.


UNIDENTIFIED MAN #3: Donald Trump wins the presidency.

UNIDENTIFIED ANNOUNCER #2: Patriots win the SuperBowl. Brady has his fifth.

UNIDENTIFIED REPORTER: Bear Stearns in the bargain bin - sold to rival JPMorgan Chase for just $2 a share.

VEDANTAM: On today's show, our fascination with making predictions and why we may need a revolution in the way we make them.

PHILIP TETLOCK: If you play the game the way it really should be played - the forecasting game - you're much more subject to embarrassment.

VEDANTAM: We look at how some people actually are better than others at predicting what's going to happen in the future. Ironically, these aren't the people you usually find on television bloviating about what's going to happen next week. They're ordinary people who happen to know a very important secret - predicting the future isn't about being unusually smart or especially knowledgeable. It's about understanding the pitfalls in the way we think and practicing better habits of mind.

Phil Tetlock is a psychologist at the University of Pennsylvania. Over several decades, he's shown that the predictions of so-called experts are often no better than what he calls dart-throwing chimpanzees. After spending years criticizing forecasts and forecasters, Phil decided to look at people who were very good at predicting the future. In his book "Superforecasting: The Art And Science Of Prediction," Phil explores how we can learn from these people to become better forecasters ourselves.

Phil, welcome to HIDDEN BRAIN. So Phil, lots of people watch television at night, and millions of people feel like throwing things at their television set each evening as they listen to pundits and prognosticators explain the day's news and predict what's going to happen next. Of all the people in the country, you probably have more cause than most to hurl your coffee cup at the television set because starting in 1984, you conducted a study that analyzed the predictions of experts in various fields. What did you find?

TETLOCK: Well, we found that pundits didn't know as much about the future as they thought they did. But it might be useful, before we start throwing things at the poor pundits on the - at the - on the TV screen, to consider their predicament. They're under pressure to say something interesting, so they resort to interesting linguistic gambits.

They say things like, well, I think there's a distinct possibility that Putin's next move will be on Estonia. Now, that's a wonderful phrase - distinct possibility. It's wonderfully elastic because if Putin does move into Estonia, they can say hey, I told you there was a distinct possibility he was going to do that. And if he doesn't, they can say hey, I just said it was possible. So they're very well-positioned.

Now, if you play the game the way it really should be played - the forecasting game - and use actual probabilities, say you play it the way Nate Silver plays it, and you wind up with a, say, a 70 percent probability that Hillary will win the election a few days before the election in November '16, you're much more subject to embarrassment. If he said there's a distinct possibility that Hillary would win, he could - he would have been very safely covered. Because when you ask people to translate distinct possibility into numbers, it means anything from about 20 percent to about 80 percent.

VEDANTAM: I want to talk about how and why people often arrive at the wrong forecasts. In your book, you tell the story of a doctor in 1956 who went in to see a dermatologist because he saw spots on the back of his hand. Tell me the story of Archie Cochrane.

TETLOCK: Well, Archie Cochrane was one of the early pioneers of evidence-based medicine. He was a great believer in running experiments - randomized controlled trials. And he was very skeptical of his fellow physicians. He thought that they were systematically overconfident in much the same way that my data suggests that political pundits are systematically overconfident. So he was a skeptic by nature. Yet, when he did get this cancer diagnosis from an eminent specialist, he didn't wait for the biopsy results to come in. He just allowed them to go right into surgery immediately before the biopsy, which was a shame because it turned out that he did not have a malignancy.

We use that as an example of the bait and switch heuristic. So Archie Cochrane didn't know whether he had cancer, but he was looking in the eye of a very renowned specialist, who looked exactly like the sort of person who would know. So he - and he took a really hard question - do I have cancer? - and he substituted a somewhat easier question. And that is does this guy I'm talking to right here look like the sort of person who would know if I have cancer? And the answer to that was a resounding yes. So he took the answer to the easier question. He assumed it applied to the harder question, and he forged ahead and got massive surgery that proved to be unnecessary when they did - when the biopsy results came in.

VEDANTAM: Archie Cochrane didn't make that mistake again. He went on to have a very distinguished career where he developed a series of techniques to help doctors guard against the pitfalls of following their intuitions. He said, look, you might be very smart. You might know a lot. But there's something you have to do before you act on your intuitions. This did not make him popular with his fellow doctors.

TETLOCK: Right. And Cochrane now is honored, but I think during his lifetime, a lot of his colleagues saw him as a pain in the butt. He would run experiments that would repeatedly show that the intuitions of his colleagues were off-base. There was a debate, for example, among cardiologists about how long heart attack patients should stay in the hospital after the heart attack and whether the mortality would be higher if they went home or if they stayed in the hospital. And the conventional wisdom among cardiologists was that it was better for them to stay in the hospital.

But Cochrane said, look, we have to test our intuitions. People would be happier. It'd be less expensive and so forth if people were resting at home. And they thought that would be a terribly unethical experiment to run because it could could kill people - right? - if it showed that the people getting the home rest were actually dying at a higher rate. But Cochrane persisted. He wasn't the guy to take no for an answer. And they ran the experiment.

And he also had a mischievous streak, so he presented the results to those to his colleagues, and he flipped them around. He showed them a chart that had indicated that the heart attack patients who went home were dying at a higher rate, just as the conventional wisdom predicted. And the doctors were outraged. And they said, Archie, we told you. You're a fool. Look what you've done. You've caused great harm. You've got to stop this experiment immediately. And then he paused. And he mischievously smiled. And he said, oh, I got the data turned around. It turns out that the people who are going home are living longer.


VEDANTAM: The truth is making predictions is difficult, but many biases also get in the way of making accurate forecasts. When we make a prediction and it turns out wrong, most of us don't remember that we'd predicted something different. In fact, the hindsight bias prompts us to believe we'd gotten it right all along.

We also hail people who make predictions that turn out right, whether in the stock market or politics or sports. But that keeps us from seeing the role of luck. Many people who get the right answer are just lucky. And some people who guess wrong are just unlucky. Over time, the laws of probability mean luck can only take you so far.

One reliable way to check if someone's success at predictions is driven by skill or by luck is to have them make lots of predictions and see how they turn out over time. A few years ago, the federal government launched such an experiment. They conducted a forecasting tournament where thousands of people logged into computers to make thousands of predictions. As time passed, the forecasts could be checked for accuracy.

TETLOCK: It was a remarkable thing for the professional career government officials to do - to sponsor a project that had the potential to be embarrassing. We were one of five academic research teams that were competing to pull together the best methods of making probability estimates of events that national security professionals cared about.

VEDANTAM: What kind of questions were they asking?

TETLOCK: All over the map, quite literally, so there would be questions about violent clashes in the East or South China Sea. There would be questions about the Syrian civil war, about Russian-Ukrainian relations, about the Iranian nuclear program, Colombian narco traffickers - literally all over the map.

VEDANTAM: So you found, after some time, that people who were not necessarily trained experts in a specific domain were actually able to perform as well or maybe even better than people who were the experts.

TETLOCK: Yeah, that's essentially right. David Ignatius broke the story, I think, in 2013 in The Washington Post about how the best forecasters in the Good Judgment Project are outperforming professional intelligence analysts who had access to classified information and were working on an internal prediction tournament.

VEDANTAM: How could this be? Are these people smarter than the rest of us, more knowledgeable? When we come back, I'll talk to Phil about what makes a superforecaster a superforecaster. Stay with us.


VEDANTAM: If you were asked to pick someone to answer a difficult question about the economy or foreign affairs, you might turn to an Oxford-educated public intellectual who writes a column for a very important newspaper. You probably wouldn't turn to a retiree in Nebraska who spends his time bird-watching. But Phil Tetlock says, maybe you should.

TETLOCK: He was the opposite of Tom Friedman in the superforecasting book. Tom Friedman, of course, being an eminent New York Times columnist - very, very elegant writer, best-selling author, well-known for his explanations, but nobody has any idea how good a forecaster he is. And Bill Flack is an anonymous retired irrigation specialist working in Nebraska, working out of the public library right out of his home and doing a fabulous job making probability estimates in the Good Judgment Project in the intelligence community forecasting tournament.

VEDANTAM: So you helped discover Bill Flack. Was he part of your team that you entered in this forecasting tournament?

TETLOCK: He was, indeed. He was one of the very best. The very best forecasters in the tournament were called out each year - the top 2 percent - and we put them into elite teams called superforecasting teams. And we let them work together. And they did just a fabulous job.

VEDANTAM: Superforecasters, like Bill Flack, turn out to have some things in common. Tell me about the kinds of philosophies they have and the kinds of thinking styles that you seem to find in common among many of these superforecasters.

TETLOCK: I would say the most distinctive attribute of the superforecasters is their curiosity and their willingness to give an - the idea a try. And when I say the idea, I mean the idea that forecasting is a skill that can be cultivated and is worth cultivating because it doesn't matter how intelligent you are or how knowledgeable you are. If you believe that it's essentially impossible to get better at these kinds of tasks, you're never going to try, and it's never going to happen. It's as simple as that. So a necessary - a big necessary condition for moving forward is having the attitude that this is a skill that can be cultivated and is worth cultivating.

VEDANTAM: Superforecasters tend to gather information and update their beliefs in a very particular way. Phil Tetlock points to Aaron Brown, the chief risk officer of the hedge fund AQR.

TETLOCK: Before he was a big shot in finance, he was a big shot in the world of poker. He was a world-class poker player. And he's - we quote him as saying that, you know, you can tell the difference between a world-class poker player and a talented amateur because the world-class player knows the difference between a 60-40 bet and a 40-60 bet. Then he pauses and said, oh, maybe like 55-45, 45-55. Distinguishing more degrees of maybe is an important skill. Why is that?

Well, the very best forecasters are well-calibrated. So when they say events are 80 percent likely, those events happen about 80 percent of the time. When they say things are 90 percent likely, they happen about 90 percent of the time. So it makes a difference how frequently you update your forecasts. If you don't update your forecasts reasonably frequently, you're going to fall out of phase with events. And that means often making adjustments that are relatively small.

VEDANTAM: You suggest that forecasters should do something that doesn't seem to be very intuitive. Instead of looking at the particulars of an individual case, you say forecasters should zoom out and figure out how often something has happened historically.

TETLOCK: So Daniel Kahneman is probably one of the greatest psychologists of the last hundred years, and he calls that the outside view. And he says people rarely take the outside view when they do forecasting. They normally start from the inside, and they work out. But there's a big advantage to you, as a forecaster, from starting with the outside view and working in. Take another example. Let's say you're at a wedding, and you're sitting next to somebody who has the bad taste to ask you, how likely do you think it is this couple's going to stay married?

VEDANTAM: (Laughter).

TETLOCK: And you look at the person. And it's bad taste and all that. But you see how happy the couple is, and you can see it's a joyous occasion. You say, I can't imagine these people who are so happy together getting divorced. I think maybe a 5 percent chance they're going to get divorced.

Now, if you'd asked that question to a superforecaster, they'd say, well, let's see. Let's look at the sociodemographics of the couple. And let's see. What's the base rate of divorce within this sociodemographic group? Let's say it's 35 or 40 percent over the next 10 years. OK, I think there's about a 40 percent chance they'll get divorced in the next 10 years.

Now, that's not the end of the forecasting process. That's just the beginning. The real value, though, of doing it this way, of starting from the outside and working in, is it puts you in the ballpark of plausibility right away. Forty percent is a much more plausible number than 5 percent. Now, then you could start adjusting the 40 percent.

So if you discover things about the couple that suggest they really are deeply bonded to each other and they've known each other a long time and they really understand each other and they've done great things for each other, you're going to lower your probability. If you discover that the husband is a sociopathic philanderer, you're going to raise the probability. Those are those inside view sorts of pieces of data that would cause you to adjust. Or you might just see them having a small fight and say, well, OK, I'm going to move from 40 to 41 percent (laughter).

And that's one of the interesting things about superforecasters. They do a lot of updating in response to relatively small events. And most of the news, most of the time, is what statisticians would call low diagnosticity news. It doesn't change things dramatically. But it does change things a little bit. And appreciating how the news gradually builds up toward one conclusion or another is a very valuable skill.

VEDANTAM: I'm wondering if one reason many of us start with the inside view rather than the outside view is that just at an emotional level, that's how our minds think, that, you know, you see a couple. And you put yourself in the shoes of that couple. And you try and imagine what's happening in their lives. And we think in stories. And we imagine what life must be like for that couple. And we're trying to see how that story will turn out.

And we're trying to follow the narrative where it leads, rather than do this much more, you know, abstract remote process of saying, let me start with the rough estimate of how often something happens. There's something about that, in some ways, that requires us to step outside this narrative frame that we often use to understand the world.

TETLOCK: I couldn't put it better. I think that's right. We're quite readily seduced by stories. You know, another example might be - comes from research. And this is again related to Daniel Kahneman and his work on the conjunction fallacy. It's another source of error in forecasting. Let's say I ask you, how likely is it that in the next 10 years, there'll be a flood in North America that kills more than a thousand people and ask you to make an estimate on that?

Let's say I ask another person to make the estimate, how likely is it that there'll be a flood in California that will be caused by an earthquake cracking a dam leading to a massive outflow of water? Now, when I put the two questions together like that, it's pretty obvious that a flood anywhere in North America due to any cause has got to be more likely than a flood in California caused by an earthquake cracking a dam, right? The California event is obviously a sub-sub-subset of the more general North American flood thing.

But people don't see it that way. The California earthquake dam story is more like a - it's like a story. You can actually put it together in a more meaningful way. Whereas a flood anywhere in North America is kind of abstract and vague. So people can transport themselves into the world. And you can imagine it's like a movie, like a Hollywood movie playing out. And they can see it happening. Yes, I can see that happening. And it's...

VEDANTAM: In fact, I can see it as you're speaking right now, Phil.


TETLOCK: And that pumps up the probability, and that screws up your forecasting track record.


VEDANTAM: So again, think of forecasting as a skill that can be improved with practice. When you're making a prediction, start with the base rate - the outside-in view. Beware of the risks of storytelling. Finally, amateurs make three kinds of predictions - yes, no and maybe. Professionals have many gradations of maybe, and they attach specific probability estimates to their predictions, allowing them to go back and learn where they went wrong. But even if you do all these things, I asked Phil how you can be really sure that predictions that turn out correct are because of good technique.

You know, there's an old trick that they play in business schools to talk about the role of luck, where they, you know, they divide the class into pairs. And they say, you know, have a coin toss between each person. And then the winner of each of those coin tosses competes against another winner.

And after 12 rounds, there's one person who has - who is declared the winner. And, of course, that person you know, in a business sense, might seem to be extraordinarily good. But really, all that's happened is that they've happened to win 12 coin tosses in a row. They've just been very, very lucky. How do you distinguish between people who are lucky and people who are actually very good?

TETLOCK: Well, that is, indeed, the $64,000 question. And it comes up in finance, too. I mean, there are some finance professors out there who would argue that really famous superinvestors, like Warren Buffett or Ray Dalio and people like that, are, in some sense, like coins that come up heads 20 or 30 or 40 times in a row. We have a lot of people competing in financial markets, and they're making a lot of predictions over long stretches of time. So you're going to expect some streaks.

And when we get really streaky performance, we declare we found a genius. So the skeptics would say, well, Phil Tetlock is doing essentially the same thing here. He's annointing people who are essentially lucky. So we built in a lot of statistical checks. But, you know, you can never be 100 percent sure.

VEDANTAM: One of the other critiques of Phil's work is that the kinds of questions that superforecasters are answering are not really the questions that people want answered. Most of us are interested in the big questions. Who's going to win the current standoff between the United States and Russia? Superforecasters tend to answer much more narrow questions. Is Russia going to invade Ukraine in the next 18 months?

TETLOCK: I think that's a very fair criticism of the first generation of forecasting tournaments - that we put all of our effort into improving forecasting accuracy. Now, I'm not going to say the questions that people were trying to answer were trivial because they were by no means trivial. It's by no means whether Russia was going to invade the Ukraine or the Syrian civil war was going to drag out as horribly long as it has and so forth. I mean, the questions were really - about really important things.

But were they about the things that matter the most from a public policy perspective? Could we have made the questions more relevant to deep policy questions? I think the answer is yes. And, you know, science proceeds incrementally. You do one thing at a time. So what these first - the first generation of forecasting tournaments generated is that there is a skill that is worth cultivating and can be cultivated. OK, that's step one.

Now, I think the next generation of tournaments should be focusing on exactly what you're saying here. I think we should be focusing as much on the insightfulness of the questions as the accuracy of the answers.

VEDANTAM: I'm wondering if some questions just are too hard to forecast. If we were having this conversation in 1917 rather than 2017 and someone were tell you, you know, 40 percent of the country right now is in farming, and in 100 years from now, you know, 2 percent of the country would be farming, it would be reasonable to sort of conclude that, you know, a good chunk of the country would be out of work, that our unemployment rate would be, you know, 35, 40 percent. And that would be a perfectly reasonable conclusion to draw. And, of course, there just would be no way to fully understand how that would not have come to pass.

And I'm wondering, in some ways, you know, is the point of superforecasting to break questions down so that they're so simple, so that they're answering questions with accuracy but not necessarily questions that are of importance? You know, there's the old joke of the guy who loses a coin on the street. And he searches for the coin under the street lamp because that's where it's brightest, not where he actually dropped the coin.

And is that what we're doing with superforecasting? Is it possible that we're asking the kinds of questions that are the easiest to answer, the easiest to which we can attach these probability estimates, but the moment you go a little bit further out, we essentially are just guessing?

TETLOCK: Well, you've raised a host of interesting questions. And I would be the first to agree with you that forecasting tournaments can degenerate into games of trivial pursuit. And I'd also agree very much with your premise that forecasting accuracy falls virtually to chance, to the dart-throwing chimpanzee, when you try to forecast more than about 10 years out on geopolitical and geoeconomic questions of consequence.

So both of those things are true. There's a limit on the forecasting horizon. And it's true that forecasting tournaments, if you don't pick the questions carefully so that they add up to something important, they can become trivial. So while both are true, I don't think that they mean we shouldn't be doing forecasting tournaments. I think that we should be doing forecasting tournaments well.

VEDANTAM: I'm going to ask you one final question. And this is also, I think, a potential critique of superforecasting, but it comes in the form of a forecast that I'm going to make. The reason I think many of us make forecasts or look to prognosticators and pundits to make forecasts is that it gives us a feeling like we have a handle on the future. It gives us a sense of reassurance.

And this is why liberals like to watch the pundits on MSNBC and conservatives like to watch the pundits on Fox. You know, a more cautious style that sort of says, you know, the chance that Donald Trump is going to be impeached is, you know, 11.3 percent, or the chance that you're going to die from cancer is 65.3 percent, these estimates run up against a very powerful psychological impulse we have for certainty, that we actually want someone to hold our hand and tell us, you're not going to die. We don't want a probability estimate. We want actually an assurance that things are going to turn out the way we hope.

So here's my last question for you. If someone advises people to do something that runs against their emotional need for well-being and reassurance, I am going to forecast that that advice, however well-intentioned, however accurate, is likely not going to be followed by most people. What do you make of my forecast, Phil?

TETLOCK: (Laughter) Well, I think there's a lot of truth to what you say. I think people - when people think about the future, they have a lot of goals. And they want to affirm their loyalty to their ideological tribe. They want to feel good about their past commitments. They want to reinforce their preconceptions. So those are all social and psychological goals that people have when they do forecasting.

And forecasting tournaments are very unusual worlds. We create a world in which only one thing matters. It's pure accuracy. So it's somewhat analogous to the sort of world that's created in financial markets or London bookies or Las Vegas bookies. All that matters is the accuracy of the odds. I would say this. I would say people would be better off if they were more honest with themselves about the functions that their beliefs serve.

Do I believe this because it helps me get along with my friends or my boss, helps me fit in, helps me feel good about myself? Or do I believe this because it really is the best synthesis of the best available evidence? If you're playing in a forecasting tournament, it's only the latter thing that matters.

But you're right. When people sit down in their living room and they're watching their favorite pundits, they're cheering for their team. It's a different kind of psychology. They're playing a different kind of game. So all I'm saying is you're better off if you're honest with yourself about what game you're playing.

VEDANTAM: Psychologist Phil Tetlock is the author of "Superforecasting: The Art And Science Of Prediction." Phil, thank you for joining me today on HIDDEN BRAIN.

TETLOCK: My pleasure.


VEDANTAM: This week's episode was produced by Rhaina Cohen edited by Tara Boyle. Our team includes Jenny Schmidt, Maggie Penman, Renee Klahr and Parth Shah. Our unsung hero this week is Portia Robinson Migas (ph) from our marketing team. Portia helped us brainstorm many ideas in the early days of HIDDEN BRAIN and has remained a steadfast friend. Thanks, Portia. You can find us on Facebook, Twitter and Instagram and listen for my stories each week on your local public radio station. I'm Shankar Vedantam, and this is NPR.

(SOUNDBITE OF MUSIC) Transcript provided by NPR, Copyright NPR.


Subscribe to the Hidden Brain Podcast on your favorite podcast player so you never miss an episode.

google podcast subscribe
spotify podcast subscribe


Go behind the scenes, see what Shankar is reading and find more useful resources and links.