Hockey Statistics Have An Objectivity Problem

Garrett Hohl posted two articles this week about the often highly contentious “debates” that regularly flare up in the hockey twittersphere and elsewhere around the so-called advanced stats. “These debates often come to an impasse. Sometimes they even deviate into ad hominem and red herrings,” Hohl writes.

That’s putting it mildly, but a better debate is presumably the goal for most of us even peripherally involved in hockey stats or hockey twitter. One supposes, although it’s never stated outright, that these two pieces were intended to work together towards that end.

The first part caused a firestorm on twitter. I suppose we will all disagree on why that was. Hohl has stated multiple times that he believes the backlash was triggered by misinterpretations of his point. I would maintain that it was caused by proper interpretations not only of what he wrote but also what he didn’t and still hasn’t.

The posts and Hohl’s responses both on twitter and on the blog indicate to me that he still doesn’t get what it is that people, myself included, are so upset about and why so many people consider the stats community as a whole to be arrogant and hostile. Or what it is that actually needs to change about the whole thing.

Part I is entitled “Why statistic-folks are sometimes assholes, justifiably.” This got quote tweeted a lot and Hohl sort of admits to regretting using this title, although he’s wrong that this is the sole basis for the criticism the piece received.

It is, in fact, a fairly accurate representation of the thrust of Hohl’s argument if not the tone of the piece. The whole that he constructs is a justification for dismissing a huge number of people as illegitimate because they have not passed through his credentialing.

In the piece, Hohl argues that “one would (and should) expect there to be “assholes” in every demographic,” so why should the onus for civil dialogue be only on the stats guys?

Why are the “analytics guys” the only ones needing to change their ways to make things better? Why is it that only one side is discussed to be less cordial than the other?

There are a number of responses to this, from “they aren’t” to “being better than an asshole is often a valid goal in and of itself” to “because there are very specific forms of asshole behavior that are endemic to the statistical community and the statistical community alone is responsible for changing those.”

Hohl posited a different answer: objective methodology looks like assholitude when it goes up against emotion even when it isn’t.

Whether fan, analyst, or whatever. The statistical side of debate is one of the scientific method. One’s own opinion is simply a hypothesis. The hypothesis is then tested against the scrutiny of evidence and we ask how likely something is true or not.

The response to the testing is then what the stats person states, not their own opinion on the matter.

The answers to our questions are not what we feel, but what everything we know currently suggests. The answer is not always fixed. As more information is added, the best answer supported by evidence can change. Still, what is the best answer at the time given what we currently know and can test is still the best given those things.

So, when we have a statistical argument versus a bar argument, you have a disconnection.

It is not the statistical person’s opinion versus the bar person’s opinion. In fact, prior to testing, the two may have had the same opinion. The statistical person has simply done the testing to see what current information already says is the most likely answer.

However, the bar person may feel as if they were being told their opinion matters less than the statistical person simply because it is different each time.

Unfortunately, that’s not only not what’s actually happening, it’s not how either human cognition or the scientific method work. In other words, not only are the statistics community not adhering to this ideal of the scientific method in their discourse, the scientific method is not a producer of objectivity devoid of the taint of emotion and opinion.

Lowenstein and Lerner, in 2003 wrote about the strides cognitive science was making in understanding the role of emotions in human decision making. “Research conducted within the last decade has shown that (1) even incidental affect—affect that is unrelated to the decision at hand—can have a significant impact on judgment and choice, that (2) emotional deficits whether innate or experimentally induced can degrade the quality of decision making, and that (3) incorporating affect in models of decision making can greatly increase their explanatory power.” [from R.J. Davidson, K. R. Scherer, & H. H. Goldsmith, Handbook of Affective Sciences, Oxford University Press, 2003, p. 619, emphasis added]

In other words, emotion is deeply involved in that process we like to call rationality. Thought processes do not operate outside of emotion, but rather hand in hand with it. This is why you get a frisson of excitement when you see something that is “right” and why you try your experiment again when you get a result that is “wrong.” Emotions are part of information processing. [p. 627]

And the scientific method is a form of information processing that values certain kinds of experiences over others. It is a knowledge technology. It produces new opinions—perhaps informed, perhaps not—not objective fact.

So yes, in fact, Hohl is telling the non-stats, non-scientific person that their opinion is less valid than the scientific stats opinion. He spends the previous three paragraphs building a justification for this.

A feeling is not knowledge, Hohl is saying. It is untested, it is unsupported, it is emotional. Feeling opinions are inherently unreliable and not scientific and thus, not authoritative.

This “scientific method = objective = truth” formulation is a way to say “I know. You feel. And Knowing trumps Feeling.” It is an epistemological proposition that posits one opinion as knowledge and the other as not-knowledge and then privileges its own knowledge.

This is a vastly troubling take on the nature of objectivity and the ways that knowledge functions as a gatekeeping tool. And it is a troubling picture of how this group of thinkers understand their relationship to knowledge, to the mechanism of its production, and to the power it gives them. It’s frankly rather insulting.

If the only legitimate facts—the only evidence that can be admitted—are those facts produced by a technology you are the gatekeeper of, then no one else’s opinion is valid unless and until they enter the inner sanctum.

That’s arrogance and it’s not the natural outcome of the use of a particularly theorized set of numbers as evidence. This idea has put the methodology in service to power, not the other way around.

This is what people were responding to, the thing that the title caught out: this assertion that privileges one group of people over others and naturalizes that privilege. Hohl’s description of his relationship to bar patrons was condescending and dismissive. People got that.

Part II, entitled “Why statistic-folks are sometimes assholes, Unjustifiably,” attempts to address some of the concerns about “the 10%” of stats people who behave badly, as opposed to the bar patrons who think badly, but because Hohl never leaves behind his assumption of the inherent superiority of his method, it falls far short of either identifying the problem or proposing much of a solution.

One of the main concerns is the extent to which bad behavior happens and the extent to which the community overlooks it and rejects attempts to bring it to light.

It’s not something quantifiable, but it really isn’t a fringe element that are indulging. It’s endemic. I’d put the percentage far higher than 10%–more like 40%–but even if it really is only 10% of writers or 10% of interactions, when they’re perpetuated by some of the loudest and most respected people in your group—well that changes the dynamic quite a bit.

Besides, when it happens to you, it doesn’t matter if 90% of the rest of the gang is fine. They not only stood by and let it happen, they praised the perpetrators. When you call out bad behavior only to get more of it heaped on top of you, it adds up.

Hohl makes a start at acknowledging that emotions are present in all human endeavors, but this isn’t an emotion vs thought problem. For one thing, Hohl falls short of understanding that emotion is not a thing separate from analysis and that humans simply cannot “be dismissive of our own emotions.” Not only is this impossible to do, if you did this, you could not think.

For another, abuse is abuse whether it’s done scientifically by good guys or not.

The concern isn’t merely failing to take other people’s feelings seriously, it’s dismissing other people’s ways of knowing and their understanding of the world. It’s coming into a conversation with the belief that your methodology has resulted in a superior way of uncovering reality and that no other way of knowing has any insights worth considering.

This is predicated on the supposition that you have knowledge while other people have feelings and that feelings will never lead to knowledge. It’s saying that you have removed something you have not and cannot remove, and thus have more of something that doesn’t really exist: objectivity. And that having more of the latter and less of the former through one and only one method is the ideal way to gain understanding.

Which is why this formulation of subjectivity and interpretation rings so hollow:

Numbers are objective. In hockey analysis they explicitly define the quantity of events that occur. However, our own interpretation of these numbers can have some subjectivity. This, in part, adds credence to the old “lies, damn lies, and statistics”.

The evidence may point to something, but an individual’s interpretation of the evidence may lead them the wrong way. No argument is ever had in a vacuum, devoid of personal bias.

Numbers aren’t objective in any useful way. Certainly, one can philosophize on the objective existence of the number three, but that perfect and unmanipulated number has no social or analytical meaning here. And it’s certainly not the number that we use to debate hockey with.

The numbers used in hockey analysis are far from objective numbers. They are chosen numbers, official numbers, moderated numbers. Not only have we chosen what to count, we have chosen how counting happens. Someone privileged a particular set of numbers over others for a particular reason and chose a particular set of number gatherers who work to fulfill that purpose. We decided what is meaningful to count and told specific people to count that in a specific way.

These are subjective numbers from the very beginning. They are numbers under judgment, the end result of a process of interpretation and decision-making. They are the product of human eyes, human hands, and, yes, human judgment, not existing in a pristine state.

That’s before anyone gets their hands on them and uses them to build corsi and fenwick and expected goals, all of which are subjective processing mechanisms. What you end up with is not Truth but proposition—a signpost not a destination.

The belief that you’ve found objectivity is what leads you to dismiss the thoughts and insights of outsiders. Outsiders feel, but you know.

And the dismissal of people as outsiders has led to a lot of the bad behavior that has turned so many people off and led to the reputation that the stats community has for being arrogant, rude, and dismissive. It’s not because people don’t like math. It’s because you have behaved in very particular ways to defend your gates.

I’ll end this with a warning. This attitude not only locks you into already worn out paths but it is even now keeping some very bright minds from engaging in this debate and informing your analysis. If this doesn’t change, hockey will lose out and analysis will not become as rich and insightful as it otherwise could be.

If you want a welcoming community, call out the perpetrators of hostility within your own ranks. Let them know when you think they have crossed a line. Put out the fires earlier and more frequently. In other words, don’t allow the behavior you claim to dislike to go unchallenged.

Consider the idea that concepts outside of current knowledge, derived from other than your own methods, may yield insights.

And recognize that the numbers you quote are not a vaccine against your subjectivity and biases but rather products of them.


Me: The Chicago Blackhawks logo is racist.

You: People should stop saying things are racist.

Me: But it is racist.

Black Hawk portraitIt is a caricature of something you pretend has something to do with a vaguely Native American past. It reduces living people to objects. It uses stereotypes, like war paint and tomahawks. It misuses sacred objects such as feathers. It doesn’t bear any resemblance to the person it is meant to refer to. 

And it encourages people to engage in racist behavior like wearing headdresses and painting their faces red.

Both goaltenders have used this to design offensive goalie masks that they wear without awareness of what they are doing or even willingness to learn.

Both appropriate sacred symbols for your entertainment, one an eagle-feather headdress and the other the petroglyphs on Newspaper Rock. Neither of those symbols have any relation to the tribe that Black Hawk represented. 

You: I know Blackhawks fans and they aren’t racist so this can’t be racist.



I don’t know where you got the notion that the only racism that exists is the kind that wears a KKK robe.

And I don’t know where you got the notion that if someone’s your friend, they can’t participate in racist practices.

This is the kind of racism that people do–daily–without even thinking about it. It doesn’t even occur to you to look at what you’re doing. You’re just so used to it.

It’s the kind of racism I experience all the time. The kind that allows you and your friends to pretend to a past in which white America’s was engaged in a fair battle with the noble savage, now tragically gone.

It allows you to pretend that you aren’t still taking Native lands for your oil and mining companies, that you aren’t poisoning Native water sources right now, that don’t have a legal system that considers your body more inviolate than mine.

It’s the kind of racism that allows your government to chip away at their treaty obligations and make Native religion and languages illegal. That allows you to remove Native children from their families without due process and to shame Native nations who try to protect their citizens.

It’s the kind of racism that allows you to think calling someone “Pocahontas” is not demeaning. Or that Pocahontas had a great love story rather than a life of captivity and rape. Or that your grandmother descended from Pocahontas and so you have the right to speak for all Native people.

It’s the kind of racism that says that you’ve never heard any Native objections to mascots before even as you shout down Native objections to mascots.

It’s the kind of racism that allows people to say to me that cannibalizing my culture is an honor.

It’s the kind of racism that allows you to say to me “I can’t support your religious rights because I read about this horrible case in India.”

And it’s the kind of racism that allows you to demand that I use another word to describe what you’re doing, that I soften the truth so that it doesn’t hurt you as much.

But you didn’t listen when we were nice. Now we will be louder.


It’s Tuesday afternoon. Last Saturday, three days ago, a man massacred 50 people at a gay nightclub in Orlando. I didn’t know any of them personally, but I grieve for them.

My words have been dammed up, closed up inside of me by a conviction that words are not enough. It took approximately thirty seconds for the typical American responses to kick in: mental illness, Islam, ISIS, radicalization.

It took about that long for people who politicized the existence of homosexual, bisexual and transgender people to claim we shouldn’t politicize their deaths when their deaths were so clearly political.

Of course what they mean by politicizing their deaths is actually holding them accountable for their words and actions. They don’t want anyone to look any further than the tissue paper excuses they have for this incident.

They don’t want anyone to see that what you say in this world, the hate you pour into this world, has real and tragic consequences. They spent all this time telling everyone, most especially themselves, that LGBTQIA are the danger. Now we’re all in danger of seeing that they are, in fact, innocent targets. So we have to erase their queerness.

They don’t want you to see that their gun culture is a culture of toxicity, a culture that worships the destruction guns can do. They want you to think that an AR-15 is needed to protect you from a burglar.

They don’t want you to understand that every time there’s a massacre like this gun sales go up. They want you to think there aren’t enough guns. There never will be, you know. Never.

They want you to think this was an Islamic thing, that the shooter was foreign, that the danger is foreign.

It’s not. Political violence is an American tradition. It happens frequently.

Not protesting. Not marching in the streets.

Real violence. With guns and bombs and knives and fists. With real blood and real death.

They want you to think that’s outsiders doing that. It’s not. It’s Americans acting with the full understanding of great swaths of American people.

If you won’t believe that, they want you to think it’s the left doing it. Not right now. There has been leftist violence in this country, but these days it’s mostly right-wingers acting on right-wing rhetoric.

Mostly, though, they want you to think there isn’t anything that can be done about it, that it’s the cost of living in a society. They want you to think that the fact that this doesn’t happen in any other country with a functional government is irrelevant.

This isn’t human nature. Mass murder should be an anomaly. We choose, actively choose, that it be common. We blame everyone else for our failings so that it can continue. We continue to elect the people who support it, who promote the conditions under which it can occur regularly.

The gun lobby, meanwhile, stands there smirking and counting their money.

The hate lobby is grateful that there’s an Islamic connection so that they can pin this on others. They’re about to elect a president on this lie.

And we mourn our dead. Over and over, we mourn our dead.

Words simply are not enough.

Kinds of Seeing

I tweeted this last night, but as Kris Martel so “helpfully” pointed out, I could have blogged it. I guess this way it sticks around for a while, for better or for worse.

Context: Andrei Vasilevskiy started for the Tampa Bay Lightning against the Arizona Coyotes. The Coyotes aren’t a great team (they have some interesting players but they’re not good as a team yet.) The Lightning were mostly able to control play until the third period. Then they started to sit back. Eventually Vasilevskiy let in a goal off a faceoff.

Continue reading

Why Women Should Matter to the NHL 

This post was written by Robyn Pennington (@robyn_jftc) as a guest contribution for Puckology. It represents both how deeply the NHL’s attitude towards women is hurting their customers and how personal the choice is to remain engaged with the league.  Welcome, Robyn.

Content warning: Domestic violence, sexual assault.

Why Women Should Matter to the NHL

I wrote this piece partly inspired by Sarah Connor’s amazing article at Stanley Cup of Chowder and partly in light of the NHL’s and Blackhawks’ lack of response to the Patrick Kane case.

Continue reading


I confess. I do not like math or statistics. This is why it’s a little uncomfortable to me that I have in any way become considered part of the “stats crowd.”

I don’t enjoy statistics. I don’t enjoy doing it and I don’t enjoy reading about it. I don’t enjoy thinking about it. This is not my bailiwick. I’d rather be learning something else. It is very hard to be creative in a field you find a chore. Continue reading

The Thing About Stamkos

The Boltosphere is all in a tizzy these days because Steven Stamkos, who’s in the final year of his last RFA contract, hasn’t signed an extension.

This is not the first time this has happened, of course. We all bear the scars of the last #Stammergeddon in 2011. When he signed in July. It is now the latter part of August. So it’s getting a little antsy in here.

Continue reading

On Women, Sportwriting, and “Just Keep Writing”

People like to give advice. People who have become successful at something love to share their wisdom about how to end up where they are. It’s a kind of humble brag: I did this (I’m special and worth listening to) and you can, too! (but not too special.) It’s also a way of saying that the actions a person took are more important than the context in which they took them.

Recently, a successful hockey writer gave some (as far as I know unsolicited) advice to women and people of color who wanted to become successful at–and get paid for–writing about hockey. It was well-meant but naive and more than a little arrogant. If you don’t like the rules at one site, start your own. Create great content and people will find you. Just do it. That kind of thing. Continue reading

Oh, Captain! What a Day This Could Have Been

Marty St. Louis retired yesterday. It ends not with a bang, but with a whimper, to use a cliche. Everyone knew it was coming. He’d become something of a liability on the ice recently and no offers appeared to materialize on July 1. So the next day, he retired.

When I first heard the news, I simply stopped. I didn’t know what to think or what I really felt. Sadness? Nostalgia? A little bit of schadenfreude–some “serves you right!” ? How about regret? Continue reading

Observations on Drafting Goaltenders

Over the past few years, I’ve been spending my summers immersed in NHL goaltending draft data. In light of the upcoming draft I’ve pulled together some general observations that fans of drafting teams might find interesting. All of this has been published before, mostly at Raw Charge or here at Puckology, although in one instance it was published in Justin Goldman’s latest book, Between Two Worlds. Links are provided below. Continue reading