There’s a great podcast out on the 538 Politics Feed where Nate Silver and Micah Cohen look back at the 2016 election and how their coverage and forecast evolved.
They both acknowledge that one of their biggest mistakes was underestimating Donald Trump’s chances early in the Republican Primary. They acknowledge that that framed a lot of their coverage of Trump early on.
But wait. Isn’t 538 at data site? Isn’t it all about empirical data?
Yes it is.
But a key part of analytics, of probabilistic thinking and Bayesian probability is acknowledging your priors. Priors, basically, are what you think about a given situation before it happens. They’re what you expect to happen, or think will happen. Good analytical thinking will take these into account. In terms of the election, a lot of news organizations (including 538) had a prior of “There’s no WAY Donald Trump becomes the Republican nominee or gets elected.” If that’s your prior, that will influence how you view the empirical data.
This matters in sports, because what we expect to happen influences how we read analytics, data and statistics.
That’s especially true as the Buffalo Bills’ season starts to fall apart.
As has often been the case during the 17+-year playoff drought, one of the team’s biggest issues has been at quarterback. Tyrod Taylor’s been a lightning rod around the team, as captured beautifully in this excellent Mina Kimes profile:
He's also polarizing around the NFL, where he's become a Rorschach test for how we think about quarterbacks in 2017. Some coaches and experts are quick to rattle off his deficiencies: too short at 6-foot-1, too quick to leave the pocket, too limited as a passer. Other analysts see him as an underrated star, arguing that traditional "volume" stats like yardage and touchdowns (to say nothing of wins, the data point most loathed by the numerate set) fail to account for his unique skills.
Before Sunday’s dumpster fire of a loss to the Saints, the Bills ranked 17th in the league in defense in DVOA (the advanced metric that measures overall team performance) and 21st in offensive DVOA. Taylor was 18th in the league in QBR, 20th in the DYAR quarterback ranking.
In other words, Taylor’s not the reason the Bills are struggling, but he is not helping them rise above their station.
So why does this matter?
If you came into this season thinking that the Bills needed to find a franchise quarterback, that Tyrod Taylor isn’t the answer, then you look at the numbers this season and see evidence that supports that.
If you came into this season thinking that Tyrod Taylor was underrated, that the Bills had left him with no real weapons and that the defense is letting him down, then you look at the numbers this season and see evidence that supports that.
Empirical evidence does not exist in a vacuum.
The point is to acknowledge, and pay attention to, your priors.