Garrett Hohl posted two articles this week about the often highly contentious “debates” that regularly flare up in the hockey twittersphere and elsewhere around the so-called advanced stats. “These debates often come to an impasse. Sometimes they even deviate into ad hominem and red herrings,” Hohl writes.
That’s putting it mildly, but a better debate is presumably the goal for most of us even peripherally involved in hockey stats or hockey twitter. One supposes, although it’s never stated outright, that these two pieces were intended to work together towards that end.
The first part caused a firestorm on twitter. I suppose we will all disagree on why that was. Hohl has stated multiple times that he believes the backlash was triggered by misinterpretations of his point. I would maintain that it was caused by proper interpretations not only of what he wrote but also what he didn’t and still hasn’t.
The posts and Hohl’s responses both on twitter and on the blog indicate to me that he still doesn’t get what it is that people, myself included, are so upset about and why so many people consider the stats community as a whole to be arrogant and hostile. Or what it is that actually needs to change about the whole thing.
Part I is entitled “Why statistic-folks are sometimes assholes, justifiably.” This got quote tweeted a lot and Hohl sort of admits to regretting using this title, although he’s wrong that this is the sole basis for the criticism the piece received.
It is, in fact, a fairly accurate representation of the thrust of Hohl’s argument if not the tone of the piece. The whole that he constructs is a justification for dismissing a huge number of people as illegitimate because they have not passed through his credentialing.
In the piece, Hohl argues that “one would (and should) expect there to be “assholes” in every demographic,” so why should the onus for civil dialogue be only on the stats guys?
Why are the “analytics guys” the only ones needing to change their ways to make things better? Why is it that only one side is discussed to be less cordial than the other?
There are a number of responses to this, from “they aren’t” to “being better than an asshole is often a valid goal in and of itself” to “because there are very specific forms of asshole behavior that are endemic to the statistical community and the statistical community alone is responsible for changing those.”
Hohl posited a different answer: objective methodology looks like assholitude when it goes up against emotion even when it isn’t.
Whether fan, analyst, or whatever. The statistical side of debate is one of the scientific method. One’s own opinion is simply a hypothesis. The hypothesis is then tested against the scrutiny of evidence and we ask how likely something is true or not.
The response to the testing is then what the stats person states, not their own opinion on the matter.
The answers to our questions are not what we feel, but what everything we know currently suggests. The answer is not always fixed. As more information is added, the best answer supported by evidence can change. Still, what is the best answer at the time given what we currently know and can test is still the best given those things.
So, when we have a statistical argument versus a bar argument, you have a disconnection.
It is not the statistical person’s opinion versus the bar person’s opinion. In fact, prior to testing, the two may have had the same opinion. The statistical person has simply done the testing to see what current information already says is the most likely answer.
However, the bar person may feel as if they were being told their opinion matters less than the statistical person simply because it is different each time.
Unfortunately, that’s not only not what’s actually happening, it’s not how either human cognition or the scientific method work. In other words, not only are the statistics community not adhering to this ideal of the scientific method in their discourse, the scientific method is not a producer of objectivity devoid of the taint of emotion and opinion.
Lowenstein and Lerner, in 2003 wrote about the strides cognitive science was making in understanding the role of emotions in human decision making. “Research conducted within the last decade has shown that (1) even incidental affect—affect that is unrelated to the decision at hand—can have a significant impact on judgment and choice, that (2) emotional deficits whether innate or experimentally induced can degrade the quality of decision making, and that (3) incorporating affect in models of decision making can greatly increase their explanatory power.” [from R.J. Davidson, K. R. Scherer, & H. H. Goldsmith, Handbook of Affective Sciences, Oxford University Press, 2003, p. 619, emphasis added]
In other words, emotion is deeply involved in that process we like to call rationality. Thought processes do not operate outside of emotion, but rather hand in hand with it. This is why you get a frisson of excitement when you see something that is “right” and why you try your experiment again when you get a result that is “wrong.” Emotions are part of information processing. [p. 627]
And the scientific method is a form of information processing that values certain kinds of experiences over others. It is a knowledge technology. It produces new opinions—perhaps informed, perhaps not—not objective fact.
So yes, in fact, Hohl is telling the non-stats, non-scientific person that their opinion is less valid than the scientific stats opinion. He spends the previous three paragraphs building a justification for this.
A feeling is not knowledge, Hohl is saying. It is untested, it is unsupported, it is emotional. Feeling opinions are inherently unreliable and not scientific and thus, not authoritative.
This “scientific method = objective = truth” formulation is a way to say “I know. You feel. And Knowing trumps Feeling.” It is an epistemological proposition that posits one opinion as knowledge and the other as not-knowledge and then privileges its own knowledge.
This is a vastly troubling take on the nature of objectivity and the ways that knowledge functions as a gatekeeping tool. And it is a troubling picture of how this group of thinkers understand their relationship to knowledge, to the mechanism of its production, and to the power it gives them. It’s frankly rather insulting.
If the only legitimate facts—the only evidence that can be admitted—are those facts produced by a technology you are the gatekeeper of, then no one else’s opinion is valid unless and until they enter the inner sanctum.
That’s arrogance and it’s not the natural outcome of the use of a particularly theorized set of numbers as evidence. This idea has put the methodology in service to power, not the other way around.
This is what people were responding to, the thing that the title caught out: this assertion that privileges one group of people over others and naturalizes that privilege. Hohl’s description of his relationship to bar patrons was condescending and dismissive. People got that.
Part II, entitled “Why statistic-folks are sometimes assholes, Unjustifiably,” attempts to address some of the concerns about “the 10%” of stats people who behave badly, as opposed to the bar patrons who think badly, but because Hohl never leaves behind his assumption of the inherent superiority of his method, it falls far short of either identifying the problem or proposing much of a solution.
One of the main concerns is the extent to which bad behavior happens and the extent to which the community overlooks it and rejects attempts to bring it to light.
It’s not something quantifiable, but it really isn’t a fringe element that are indulging. It’s endemic. I’d put the percentage far higher than 10%–more like 40%–but even if it really is only 10% of writers or 10% of interactions, when they’re perpetuated by some of the loudest and most respected people in your group—well that changes the dynamic quite a bit.
Besides, when it happens to you, it doesn’t matter if 90% of the rest of the gang is fine. They not only stood by and let it happen, they praised the perpetrators. When you call out bad behavior only to get more of it heaped on top of you, it adds up.
Hohl makes a start at acknowledging that emotions are present in all human endeavors, but this isn’t an emotion vs thought problem. For one thing, Hohl falls short of understanding that emotion is not a thing separate from analysis and that humans simply cannot “be dismissive of our own emotions.” Not only is this impossible to do, if you did this, you could not think.
For another, abuse is abuse whether it’s done scientifically by good guys or not.
The concern isn’t merely failing to take other people’s feelings seriously, it’s dismissing other people’s ways of knowing and their understanding of the world. It’s coming into a conversation with the belief that your methodology has resulted in a superior way of uncovering reality and that no other way of knowing has any insights worth considering.
This is predicated on the supposition that you have knowledge while other people have feelings and that feelings will never lead to knowledge. It’s saying that you have removed something you have not and cannot remove, and thus have more of something that doesn’t really exist: objectivity. And that having more of the latter and less of the former through one and only one method is the ideal way to gain understanding.
Which is why this formulation of subjectivity and interpretation rings so hollow:
Numbers are objective. In hockey analysis they explicitly define the quantity of events that occur. However, our own interpretation of these numbers can have some subjectivity. This, in part, adds credence to the old “lies, damn lies, and statistics”.
The evidence may point to something, but an individual’s interpretation of the evidence may lead them the wrong way. No argument is ever had in a vacuum, devoid of personal bias.
Numbers aren’t objective in any useful way. Certainly, one can philosophize on the objective existence of the number three, but that perfect and unmanipulated number has no social or analytical meaning here. And it’s certainly not the number that we use to debate hockey with.
The numbers used in hockey analysis are far from objective numbers. They are chosen numbers, official numbers, moderated numbers. Not only have we chosen what to count, we have chosen how counting happens. Someone privileged a particular set of numbers over others for a particular reason and chose a particular set of number gatherers who work to fulfill that purpose. We decided what is meaningful to count and told specific people to count that in a specific way.
These are subjective numbers from the very beginning. They are numbers under judgment, the end result of a process of interpretation and decision-making. They are the product of human eyes, human hands, and, yes, human judgment, not existing in a pristine state.
That’s before anyone gets their hands on them and uses them to build corsi and fenwick and expected goals, all of which are subjective processing mechanisms. What you end up with is not Truth but proposition—a signpost not a destination.
The belief that you’ve found objectivity is what leads you to dismiss the thoughts and insights of outsiders. Outsiders feel, but you know.
And the dismissal of people as outsiders has led to a lot of the bad behavior that has turned so many people off and led to the reputation that the stats community has for being arrogant, rude, and dismissive. It’s not because people don’t like math. It’s because you have behaved in very particular ways to defend your gates.
I’ll end this with a warning. This attitude not only locks you into already worn out paths but it is even now keeping some very bright minds from engaging in this debate and informing your analysis. If this doesn’t change, hockey will lose out and analysis will not become as rich and insightful as it otherwise could be.
If you want a welcoming community, call out the perpetrators of hostility within your own ranks. Let them know when you think they have crossed a line. Put out the fires earlier and more frequently. In other words, don’t allow the behavior you claim to dislike to go unchallenged.
Consider the idea that concepts outside of current knowledge, derived from other than your own methods, may yield insights.
And recognize that the numbers you quote are not a vaccine against your subjectivity and biases but rather products of them.