The Problem of Prognostication

Reason.com today has an article titled “It’s Hard To Make Predictions, Especially About The Future” Basically, the thesis of the article is that “Experts” have a particularly bad track record with making predictions about the future, so much so that a random process (here called a monkey and a dart board) could actually perform better than Expert (meaning that the Expert could be MORE accurate by having a secret dart board monkey). The examples used in the article include 20+ year predictions about the fate of countries, the economy, the environment, etc. All big events that have a huge effect on the entire globe. The author of a book on this subject then uses behaviorial economics to explain why everyone tends to flock to these often erroneous experts.

As I feel that “experts” tend to be puffed up prognosticators who either make heavily hedged non-predictions, or predictions that are extremely difficult to counter or measure, I am inclined to agree with this article. However, a Bloomberg article today about the partisan predictability of the Supreme Court shows that there are areas where we can make predictions, and often do. If we are so bad at predictions, then we should get these small things wrong as well? I think that we can make predictions like Supreme Court outcomes while getting the big things wrong. One major difference between the two is that Supreme Court predictions are about what the outcome of a case will be this term, not a case 20 years from now. The further we get away from the event in time, the less likely we are to get it right as much can happen in between the prediction and the event. Another factor is that the big predictions tend to be about chaotic systems where small changes early on have a significant impact much later. Finally, and this is the major difference between big predictions and predicting the outcome of a case: A Supreme Court case has at most 9 variables (the justices, or 11 if you count the parties), whereas the big predictions have a near limitless number of variables. In this sense, the complex equations of the experts in economics are useless for predictions, but the unquantifiable shiftiness of human nature and bias makes them easier to predict. Finally, Black Swan Theory neatly explains why the end of the world predictions are wrong, while Supreme Court opinions are rather mundane and common in contrast.

However, we can learn more about the predictions from big events by looking at how the predictions of small events work. My work with FantasySCOTUS gives me too major insights into predictive models and markets: the more you embrace and allow uncertainty, the more accurate we are, and that what we call “experts” are not necessarily knowledgeable. In the article I coauthored with Josh Blackman and Adam Aft, we account for uncertainty by finding confidence intervals for each of our predictions, which lets us know how much to discount certain predictions by. By holding back on weak predictions and putting forward strong ones, we’re more or less detected where and when predictions are going to be accurate (although sometimes our users get things significantly wrong). In respect to experts, we tend to find that factors such as credentials (degrees or publications) or work directly in the field are not indicators of strong predictive ability. Sometimes the exceptionally smart non-lawyer does a better job of predicting the outcome of a case than the Constitutional litigator. One possible explanation is that credentials do not select for predictive capacity, so we are trying to fit experts into a role they are not suited for (although some experts are more than willing to oblige).

As for predictive modeling and mathematics, I have to be on guard to not become one of those occultist economist who uses their crystal balls to gaze into the future. Generally speaking, I always stick to basic level descriptive statistics. By limiting myself to these tools, I know better than to try to make the world fit my visions. Instead, I apply statistics as a tool to give expression to the traits of the predictions our many users have made. So far, it has done very well for understanding how people view the court, and how cases are likely to come out. In the old crowd versus experts debate, I don’t see why we can’t crowdsource to find the real experts, not merely those who got a degree and took some classes.

Advertisements
    • Bob Bonsall
    • July 7th, 2011

    I think you left out one key factor in predicting a Supreme Court decision as opposed to predicting “the future”: in any given case, there are essentially only two broad outcomes available (with theoretically an indeterminate number of variable permutations, as limited by the outlines given above), whereas “the future” has an open number of indeterminate number of variable permutations limited only by the mathematics of chaos theory, natural law, and real world events as they happen.

    “Why, Eckhardt, you oughta think about the future.” -Jack Napier, Batman, 1989

    • Steve
    • August 30th, 2011

    I quite agree with your analysis of the accuracy of predictions, but I think that making predictions can be useful if they are treated as thinking exercises and not as prognostications of the future. To the extent that you think before you declare what the future holds, that is a good thing. It is even better if someone reads your prediction and points out its errors and limitations. This gives you an opportunity to try again (assuming that you have the proper attitude of taking criticism seriously). You have to be careful about taking your views of the future too seriously, of course, but thinking about the future can, under appropriate circumstances, help you to ask better questions and, when the future occurs, to think about aspects that you might otherwise fail to focus on.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: