To all of the people complaining about mathematical forecasting in relation to the current election, I'm here to tell you something: don't be so naive.
I know, I get it - Nate Silver of FiveThirtyEight is an adult. He can defend himself. He's also a political writer - so before he started engaging in his craft he had to know attacks were coming. However, let's defend the concept of forecasting in general - and let's start by stating that models that forecast elections are infinitely better than the alternative - people rooting on their chosen team and calling it journalism.
Baseball Sabermetrics
Sabermetrics, or the heavy usage of statistical data to make baseball strategy and roster decisions (SABR stands for The Society for American Baseball Research) has been around since the middle of last century, even if it wasn't hugely popular until somewhat recently. You see, baseball, unlike contemporary data driven professions like financials and engineering, had always been a 'gut feel' profession (/game). Scouts of various vintages were the final word on the potential talents of a player, and they would rank the five tools they believed a player had - hitting for average, hitting for power, fielding ability, throwing ability, and base-running skill.
Scouting, of course, still exists today - but there is a second way to scout talent - data mining. You see, Sabermetrics has brought more than a renewed statistical focus to the game - it has literally rewritten which statistics are useful. Gone are the days when Batting Average was ranked higher than On Base Percentage. These are the days when people will talk about win shares and value over replacement player with a straight face - even if your childhood card collection doesn't have a single mention of those metrics. Welcome to the new normal.
Political Sabermetrics?
It stands to reason that success in baseball (where previously data driven methods were scorned) means data mining would be successful in other areas. Why not politics?
Why not politics, indeed? If simple metrics like the percentage of the time a runner makes it to a base (or home) based on their plate appearances could help predict the number of wins a team was capable of... winning, shouldn't there be some stat which political scientists could point to which predicted election winners? I'm not going to seek out every political model for you, but they have been based on objective numbers like GDP, stock returns and job growth, and also subjective maters such as the presence of a scandal or the mood of the electorate (whatever that means). These models have been around for a while and have varying degrees of success and failure. However, two 'new' (in politics) methods have entered the fray.
Market Predictions and Political Forecasters
When I mention those two categories, you probably think of FiveThirtyEight and InTrade, but there are plenty of other models out there - the Princeton Political Model and the University of Colorado Model are two more you may have heard something about. On the betting side, you've also got Betfair and the Iowa Election Market.
While the election markets use the wisdom of crowds spending their money to set predictions, the thread that ties all of the political forecasting models together is based on one phenomenon - polling. It also means that Nate Silver and his ilk directly compete with naive forecasters (that take a simple average or median of polls) like RealClearPolitics and Talking Points Memo.
And What's Wrong With Polling?
Art or Science?
Have you ever heard the phrase, "garbage in, garbage out?" Well, if there is any chance that polling is garbage (we'll get to that in a second), then, well, the forecasts are also garbage. You see, baseball (and things like finance and engineering) have the advantage here - there are well defined data points either measured with sophisticated equipment or by 'agreed upon' results. In baseball, for example, there are many nuances which allow stats to be defined precisely - everything from balks to wild pitches to infield fly rules to ground rule doubles. Politics, on the other hand, doesn't have that luxury - polls are, by definition, uncertain. Finance as well - the ticker and the current trading price are all that matters.
At its core, polling is a basic thing - call people up and ask them to pick between some options, or fill in the blank. That form of polling might work in a non-important poll, but consider that by simply calling people you might be missing people who aren't home at a certain time of day, or people that use a cell phone, or any other person that, for any reason, has some systematic reason for not picking up the phone (maybe they are paranoid?). I'm not being sarcastic here - the truth is, only about 9% of people contacted will even respond to phone polling nowadays.
Let's talk about weighting for a second. If you call 1,000 people and they are all Democrats... well, of course Democratic issues are going to look better in the poll under consideration. Weighting is simply bringing demographics back into alignment - so some responses to a poll will be 'worth' more than others. For example, a phone survey in which only 5% of participants were black would be weighted to bring black respondents in line with the black vote - variously estimated to be tracking to about twice that number.
Note that in the last paragraph I used the word 'variously'. Want to know why? We don't actually know what percentage of the electorate will be black, white, religious, atheist, left-handed, or any other demographic. We can poll people to try to figure it out - but at the end of the day, a second poll will just introduce more inaccuracy. The official word doesn't come until well after the election, and it usually differs from the exit polls released hours after the polls close.
"More Art Than Science"
I don't have a major issue with art, other than young adults taking out $200,000 in loans to study it. However, art creeps into science with polling, something described often as "more art than science". That's right - there is no central polling arbiter. Every single pollster, from Republican and Democratic pollsters down to the independents, all have their own models as to what the electorate will look like. That means they all weight their poll responses in a different way. And, however we like to think of art as original, there are incentives to weighting in a similar way to other national polls. In that way, it is theoretically possible that all of the polls are off in one direction, and that has happened before many times (you only have to look as far back as the Gore-Bush race, where Bush was overrated in polling).
The potential of weighting inaccuracy is compounded by statistical inaccuracy. Here's how this works... 1,000 people are polled, but many times that will vote (millions of people). So even though the 1,000 people might be a good sample of the population, there is a margin of error (often stated as +/- some number) and a confidence interval (often not stated, but usually around 95% - so 1/20 times a poll will be flat-out wrong, in that estimation).
Okay, You Convinced Me - Polling Isn't Canon. So What?
Ahh, but we've flipped the script. If Mitt Romney wins the election, instead of blaming the model - whether Nate Silver (who has never shown Romney above 41% to win), or RCP or any of the other sophisticated or naive models, blame the data. Actually, blame the data sources.
Lost in all of the anger at Nate Silver (funnily enough, all coming before the election!) is the fact that all of his models are based on something which is, at its core, more art that science. You see, come next January, if Mitt Romney is being sworn in, it's the polls which brought us to this point. If you look at polls of the states that matter (so-called swing states) like Ohio and Wisconsin, they generally show President Obama leading Mitt Romney by around 2-5 percentage points.
So before you sharpen your pitchforks and assassinate Nate Silver's internet celebrity, recognize that he can only be as good as the data he receives. That's why it's so hard to pick stocks, pick baseball players, build bridges, pick a mate, design a circuit or send a rocket to Mars - uncertainty. Unfortunately for Mr. Silver, uncertainty is particularly high in his profession - more-so than in the other things I mentioned (other than picking a mate, possibly).
So, again, I repeat: "garbage in, garbage out." See you on the other side! Oh, and hang out at DQYDJ for our live-blog of the election. We promise to bring or sarcastic A-game and one-up our 2010 effort.
