I updated this today (June 1, 2017) after someone used something I wrote more than a year ago–out of context–to bash a goaltender who had done nothing more horrible than have two bad games. It was an example to me of the objectivity issue and how these numbers get used to hurt people. The reaction I got to this post drove me away from hockey for more than six months. Not much has changed since then in this respect. I’m not even sure I’m any better at dealing with it but I’m reluctant to give up the fight.
Fair warning, this is a long read. If anyone reads it, that’s great. Either way, I needed to say it.
Garrett Hohl posted two articles this week (Author’s note: now last fall.) about the often highly contentious “debates” that regularly flare up in the hockey twittersphere and elsewhere around the so-called advanced stats. The posts caused a firestorm on twitter
The posts and Hohl’s responses both on twitter and on the blog indicate to me that he still doesn’t get what it is that people, myself included, are so upset about and why so many people consider the stats community as a whole to be arrogant and hostile. Or what it is that actually needs to change about the whole thing.
Part I is entitled “Why statistic-folks are sometimes assholes, justifiably.” This got quote tweeted a lot and Hohl sort of admits to regretting using this title, although he’s wrong that this is the sole basis for the criticism the piece received.
It is, in fact, a fairly accurate representation of the thrust of Hohl’s argument if not the tone of the piece. The whole that he constructs is a justification for dismissing a huge number of people as illegitimate because he believes his way is superior.
In the piece, Hohl argues that “one would (and should) expect there to be “assholes” in every demographic,” so why should the onus for civil dialogue be only on the stats guys?
Why are the “analytics guys” the only ones needing to change their ways to make things better? Why is it that only one side is discussed to be less cordial than the other?
First, they aren’t. These complaints are the ones you hear most because they directly involve you. Second, being better than an asshole is often a valid goal in and of itself. But more importantly, there are very specific forms of asshole behavior that are endemic to the statistical community and the statistical community alone is responsible for changing those.
But Hohl posited a different answer: objective methodology looks like assholitude when it goes up against emotion even when it isn’t. Because apparently people outside the circle aren’t qualified to determine when they’re being dismissed or not.
Whether fan, analyst, or whatever. The statistical side of debate is one of the scientific method. One’s own opinion is simply a hypothesis. The hypothesis is then tested against the scrutiny of evidence and we ask how likely something is true or not.
The response to the testing is then what the stats person states, not their own opinion on the matter.
The answers to our questions are not what we feel, but what everything we know currently suggests. The answer is not always fixed. As more information is added, the best answer supported by evidence can change. Still, what is the best answer at the time given what we currently know and can test is still the best given those things.
So, when we have a statistical argument versus a bar argument, you have a disconnection.
It is not the statistical person’s opinion versus the bar person’s opinion. In fact, prior to testing, the two may have had the same opinion. The statistical person has simply done the testing to see what current information already says is the most likely answer.
However, the bar person may feel as if they were being told their opinion matters less than the statistical person simply because it is different each time.
Unfortunately, that’s not only not what’s actually happening, it’s not how either human cognition or the scientific method work. In other words, not only are the statistics community not adhering to this ideal of the scientific method in their discourse, the scientific method is not a producer of objectivity devoid of the taint of emotion and opinion.
The statistical work undertaken in hockey bears little resemblance to rigorous scientific or social scientific methodology. There are no controls over experiment design, no blind peer review. There’s virtually no attempt to rule out alternative explanations or check others’ methods. No one even masks their data before running these tests. Nothing passes through any kind of quality control before being released publicly.
Thus the battles over ideas, conclusions, and methods take place publicly, which makes them unbearably vicious. And there is no cultural or structural check on well-respected stats figures presenting statistics and conclusions that they know to be insupportable.
Hohl’s assertion that stats guys are arguing solely from objective evidence sans emotion or preconception rings hollow when stats tweeters marshal the very sort of cherry picked numbers they claim to oppose in their cause du jour. That this happens for years at a time, dozens of times a week, isn’t even blinked at.
To call these practices “scientific” stretches the term beyond recognition. To claim that they grant some kind of inherent objectivity is absurd.
Leaving aside the question of the actual value of objectivity — a question that science is consciously grappling with — it’s impossible to in Hohl’s words “be dismissive of our own emotions.” They are part and parcel of human thinking. We can work towards empiricism; we cannot remove affect and subjectivity. And we’ve known this for some time.
Lowenstein and Lerner, in 2003 wrote about the strides cognitive science was making in understanding the role of emotions in human decision making. “Research conducted within the last decade has shown that (1) even incidental affect—affect that is unrelated to the decision at hand—can have a significant impact on judgment and choice, that (2) emotional deficits whether innate or experimentally induced can degrade the quality of decision making, and that (3) incorporating affect in models of decision making can greatly increase their explanatory power.” [from R.J. Davidson, K. R. Scherer, & H. H. Goldsmith, Handbook of Affective Sciences, Oxford University Press, 2003, p. 619, emphasis added]
In other words, emotion is deeply involved in that process we like to call rationality. Thought processes do not operate outside of emotion, but rather hand in hand with it. This is why you get a frisson of excitement when you see something that is “right” and why you try your experiment again when you get a result that is “wrong.” Emotions are part of information processing. [p. 627]
And the scientific method is a form of information processing that values certain kinds of experiences over others. It is a knowledge technology. It produces new opinions—perhaps informed, perhaps not—not objective fact.
This is why all of those checks and balances on design, processing, and communication are so important. And why the lack of them in the hockey stats world leads to unjustifiable assholitude.
In other words, Hohl is telling the non-stats person that their opinion is less valid, useful, or worthwhile than his opinion based largely on the assertion of an authority that he cannot support.
A feeling is not knowledge, Hohl is saying. It is untested, it is unsupported, it is emotional. Feeling opinions are inherently unreliable and not scientific and thus, not authoritative.
His opinions, being free of those things, become objective and thus truthful. And, well as they say, truth hurts.
This “scientific method = objective = truth” formulation is a way to say “I know. You feel. And therefore I am right.” It is an epistemological proposition that presupposes that knowledge lies only in the place where he happens to stand.
This is a vastly troubling take on the nature of objectivity and the power that it gives. It’s a way of dismissing ideas out of hand rather than seeing other points of view as seeds of new insight. You can’t possibly be as smart as my numbers have made me. It’s frankly rather insulting.
This arrogance is not the natural outcome of the use of a particularly theorized set of numbers as evidence. It is a choice to put methodology in service to power, not the other way around.
This is what people were responding to, the thing that the title caught out: this assertion that privileges one group of people over others and naturalizes that privilege. Hohl’s description of his relationship to bar patrons was condescending and dismissive. People got that.
Part II, entitled “Why statistic-folks are sometimes assholes, Unjustifiably,” attempts to address some of the concerns about “the 10%” of stats people who behave badly, as opposed to the bar patrons who think badly, but because Hohl never leaves behind his assumption of the inherent superiority of his method, it falls far short of either identifying the problem or proposing much of a solution. If you’re an asshole because of the numbers, well, ¯(ツ)/¯.
One of the main concerns I have about this post is the extent to which bad behavior happens, as well as the extent to which the community overlooks it and rejects attempts to bring it to light.
It’s not something quantifiable, but it really isn’t a fringe element that are indulging. It’s endemic. I’d put the percentage far higher than 10%–more like 40%–but even if it really is only 10% of writers or 10% of interactions, when they’re perpetuated by some of the loudest and most respected people in your group—well that changes the dynamic quite a bit.
Besides, when it happens to you, it doesn’t matter if 90% of the rest of the gang is fine. They not only stood by and let it happen, they praised the perpetrators.
Hohl makes a start at acknowledging that emotions are present in all human endeavors, but this isn’t an emotion vs thought problem. It’s a behavior problem. Abuse is abuse whether it’s done scientifically by Good Guys or not.
The concern isn’t merely failing to take other people’s feelings seriously, it’s dismissing other people’s ways of knowing and their understanding of the world. It’s coming into a conversation with the belief that your methodology has resulted in a superior way of uncovering reality and that no other way of knowing has any insights worth considering. And then acting on that.
Which is why this formulation of subjectivity and interpretation rings so hollow:
Numbers are objective. In hockey analysis they explicitly define the quantity of events that occur. However, our own interpretation of these numbers can have some subjectivity. This, in part, adds credence to the old “lies, damn lies, and statistics”.
The evidence may point to something, but an individual’s interpretation of the evidence may lead them the wrong way. No argument is ever had in a vacuum, devoid of personal bias.
Numbers aren’t objective in any useful way. Certainly, one can philosophize on the objective existence of the number three, but that perfect and unmanipulated number has no social or analytical meaning here. And it’s certainly not the number that we use to debate hockey with.
The numbers used in hockey analysis are far from objective numbers. The RTSS that counts, among other things, shots, misses, and blocks is a subjective, judgement-laden system. Not only have we chosen what to count, we have chosen how counting happens: in real time, by people trained in particular ways, and under particular kinds of supervision. Many years ago, the NHL decided what is meaningful to count and told specific people to count that in a specific way.
This decision has consequences, whether or not it was a “good” decision. We ignore that to our detriment.
These are subjective numbers from the very beginning. They are numbers under judgment, the end result of a process of interpretation and decision-making. They are the product of human eyes, human hands, and, yes, human judgment, not existing in a pristine state.
That’s before anyone gets their hands on them and uses them to build corsi and fenwick and expected goals, all of which are subjective processing mechanisms. Not arbitrary; not meaningless. Subjective.
What you end up with is not Truth but proposition—a signpost not a destination. The numbers hockey stats work with are simply not imbued with natural meaning. And the choice to use these numbers, to only look for meaning in these places, has consequences, whether or not it is a defensible choice.
The belief that you’ve found objectivity is what leads you to dismiss the thoughts and insights of outsiders. Outsiders feel, but you know.
And the dismissal of people as outsiders has led to a lot of the bad behavior that has turned so many people off and led to the reputation that the stats community has for being arrogant, rude, and dismissive. It’s not because people don’t like math. It’s because you have behaved in very particular ways to defend your gates.
I’ll end this with a warning. This attitude not only locks you into already worn out paths but it is even now keeping some very bright minds from engaging in this debate and informing your analysis. If this doesn’t change, hockey will lose out and analysis will not become as rich and insightful as it otherwise could be.
If you want a welcoming community, call out the perpetrators of hostility within your own ranks. Let them know when you think they have crossed a line. Put out the fires earlier and more frequently. In other words, don’t allow the behavior you claim to dislike to go unchallenged.
Consider the idea that concepts outside of current knowledge, derived from other than your own methods, may yield insights.
And recognize that the numbers you quote are not a vaccine against your subjectivity and biases but rather products of them.
It is a caricature of something you pretend has something to do with a vaguely Native American past. It reduces living people to objects. It uses stereotypes, like war paint and tomahawks. It misuses sacred objects such as feathers. 
Puckology 
You must be logged in to post a comment.