Philip E. Tetlock

Dan Gardner

The Art and Science of Prediction

Superforecasting

Philip Tetlock

From one of the world's most highly regarded social scientists, a transformative book on the habits of mind that lead to the best predictions Everyone would benefit from seeing further into the future, whether buying stocks, crafting policy, launching a new product, or simply planning the week's meals. Unfortunately, people tend to be terrible forecasters. As Wharton professor Philip Tetlock showed in a landmark 2005 study, even experts' predictions are only slightly better than chance. However, an important and underreported conclusion of that study was that some experts do have real foresight, and Tetlock has spent the past decade trying to figure out why. What makes some people so good? And can this talent be taught? In Superforecasting, Tetlock and coauthor Dan Gardner offer a masterwork on prediction, drawing on decades of research and the results of a massive, government-funded forecasting tournament. The Good Judgment Project involves tens of thousands of ordinary people--including a Brooklyn filmmaker, a retired pipe installer, and a former ballroom dancer--who set out to forecast global events. Some of the volunteers have turned out to be astonishingly good. They've beaten other benchmarks, competitors, and prediction markets. They've even beaten the collective judgment of intelligence analysts with access to classified information. They are "superforecasters." In this groundbreaking and accessible book, Tetlock and Gardner show us how we can learn from this elite group. Weaving together stories of forecasting successes (the raid on Osama bin Laden's compound) and failures (the Bay of Pigs) and interviews with a range of high-level decision makers, from David Petraeus to Robert Rubin, they show that good forecasting doesn't require powerful computers or arcane methods. It involves gathering evidence from a variety of sources, thinking probabilistically, working in teams, keeping score, and being willing to admit error and change course. Superforecasting offers the first demonstrably effective way to improve our ability to predict the future--whether in business, finance, politics, international affairs, or daily life--and is destined to become a modern classic.

matej yangwao

** spoiler alert **
           Intentional predictability is possible by learned effort. During constructing intuition guessing, never stop doubting you inner agreement.

>tablespoon of doubt.
>Estimating is what you do when you do not know.

>Mauboussin notes that slow regression is more often seen in activities dominated by skill, while faster regression is more associated with chance.

Single toss forecast is mostly based on hard luck. 

Learn to agregate forecasts based people who know at least very little and scrape various facts rather than others intuition who doesn't know much about field.


> In light of the preceding, it would be safe to say the following highlights the attributes of an average superforecaster:
• Cautious
• Humble
• Nondeterministic
• Actively open-minded
• Intelligent and knowledgeable, with a “need for cognition”
• Reflective
• Numerate
• Pragmatic
• Analytical
• Probabilistic
• Thoughtful updaters
• Good intuitive psychologists
• A growth mindset
• Grit

Andrew John Kinney

This book was recommended to me by my son, and I found myself nodding in agreement with almost all of it. Tetlock focuses on what makes a person capable of excellent forecasting, and the short version of his explanation is that one must have an open mind, work hard at digesting a lot of information, be numerate, and be willing to adjust ones views as new facts heave into view. As Lincoln said, one must "adopt new views as quickly as they are shown to be true views". 

Tetlock had some interesting / controversial things to say about the German Army in WW2; the German Army (the Wehrmacht) was, one must remember, inherited by the Nazis, not invented by them. The open-minded decision processes ingrained in the Wehrmacht, which pushed decision-making (and generals) down to the front lines, had a lot to do with their early successes. Similar approaches to problem solving are usually successful in forecasting and decision-making. Interestingly, both the US Army and to a greater degree, the Israelis, adopted the German war-fighting approach. Commanders told subordinates what to do, and what the larger goal was; they did not tell them how to accomplish the goal; having read a good deal about the early battles of the Wehrmacht, I know what Tetlock is saying is correct. The lesson from these kinds of tactics is, don't fall in love with a plan and be prepared to improvise, because you almost certainly will have to do so. Similarly, forecasters must constantly incorporate new information and adjust forecasts as appropriate.

I used to jokingly open staff meetings by saying, "Let's look at the numbers, because, you know, everybody likes the truth". But of course, the reality is that if you are doing poorly as reflected by your scoring - well, in that case, you may not in fact like the truth - but generally you can't get better unless you keep score. The IARPA project that Tetlock describes in detail attempts to dramatically improve forecasting by breaking down big problems into smaller components which can be calculated; this was a trick that Enrico Fermi used with humorous effect to astonish people. Fermi would give his students problems like, estimate the number of piano tuners in the City of Chicago, and then demonstrate how to break down the problem in such a way as to get a reasonable result. I had a wonderful math professor, Frank Giordano, who gave us these kinds of problems, and they were always fascinating to me. 

My only complaint is that the author repeated himself occasionally, but, on the other hand, in my case he was preaching to the converted. This was a very worthwhile book which emphasizes the need for testing claims with data. Fundamentally, Tetlock's recommendation is to use the scientific method in areas outside of science. There is no doubt that public discourse could use a strong infusion of facts right now to influence decision-making.

Jens Madsen

Never trust a pundit! Accesible read that tells the story of two project aimed at identifying how the intelligence community can improve its forecasting skill. These were the "Expert Political Judgment project" and the "Good Judgment Project" - the latter funded by IARPA (equivalent of DARPA for Intelligence research). Having worked professionally with weather forecasting, I am pleased to see more rigorous efforts to assess predictive skills (e.g. Brier scores) applied to this field. The book is very much about explaining the common traits of those who proved consistently succesful in abovementioned forecast competitions, i.e. the objective mindset, attention to nuance, and hard work needed to continuously reassess and qualify predictions.

Dana Kraft

There are some interesting parts of this book. The whole series of objectively measured forecasting challenges is cool, and i like that he dispels the idea that the best performers are just freaky smart. I appreciated his discussion at the end about Black Swans and whether that idea makes his work less valuable. My biggest disappointment was when he prescribed 'striking a balance' between two opposite things. In my head, that's the challenge of life in general, so simply saying that you have to strike the right balance is not particularly helpful.

Lance Willett

Longer review on my blog: https://sensible.blog/2015/12/27/supe...

Though I first thought "Destined to become a modern classic" from the book jacket description might be a bit overstating the book's impact, I did enjoy the reading and came away inspired and energized. The style is approachable and authentic.

"Beliefs are hypotheses to be tested — not treasures to be guarded."

Superforecasters are people in perpetual beta, updating their beliefs constantly based on new information they eagerly gather from a variety of sources. They are constant self-improvers, proving that predicting what's next (at least in the short term) can be learned, practiced, and improved.

In this book we learn: who they are, why they are good at what they do, and how anyone can mimic their approach to improve their thinking.

One favorite thread of mine was how the author addressed other impactful books like "Thinking Fast and Slow" from Daniel Kahneman and "Black Swan" / "Anti-fragile" by Nassim Taleb. As another book about meta-cognition (thinking about thinking) Tetlock weaves elements of those authors' works into his own while both agreeing and disagreeing with their philosophies and techniques.

(Updated July 2018)

Adam

I'm kind of interested in estimation and forecasting. This book focuses on how people make broad guesses based on information - with a focus on improving and refining forecasts as new information comes in. This book focuses on the traits that make great forecasters. The top most is an open, growth mindset - being open to change and looking towards what's next. The other major focus is adapting when new data comes in. These traits and more are explored through a number of forecasters -- some professional and some amateur.

Richard Hankins

Shannon Archer

NEW YORK TIMES BESTSELLER WINNER OF THE CMI MANAGEMENT FUTURES BOOK OF THE YEAR AWARD 'A manual for thinking clearly in an uncertain world. Read it.' Daniel Kahneman What if we could improve our ability to predict the future? Everything we do involves forecasts about how the future will unfold. Whether buying a new house or changing job, designing a new product or getting married, our decisions are governed by implicit predictions of how things are likely to turn out. The problem is, we're not very good at it. In a landmark, twenty-year study, Wharton professor Philip Tetlock showed that the average expert was only slightly better at predicting the future than a layperson using random guesswork. Tetlock's latest project – an unprecedented, government-funded forecasting tournament involving over a million individual predictions – has since shown that there are, however, some people with real, demonstrable foresight. These are ordinary people, from former ballroom dancers to retired computer programmers, who have an extraordinary ability to predict the future with a degree of accuracy 60% greater than average. They are superforecasters. In Superforecasting, Tetlock and his co-author Dan Gardner offer a fascinating insight into what we can learn from this elite group. They show the methods used by these superforecasters which enable them to outperform even professional intelligence analysts with access to classified data. And they offer practical advice on how we can all use these methods for our own benefit – whether in business, in international affairs, or in everyday life. 'The techniques and habits of mind set out in this book are a gift to anyone who has to think about what the future might bring. In other words, to everyone.' Economist 'A terrific piece of work that deserves to be widely read . . . Highly recommended.' Independent 'The best thing I have read on predictions . . . Superforecasting is an indispensable guide to this indispensable activity.' The Times

Superforecasting The Art and Science of Prediction

Reviews