Friday, October 27, 2017

The Search for Useful Metrics & Fair Discussion

Imagine that there are 10 high schools in a school district and nearly everyone suspects that one school (Acme High School, the one in the poorest neighborhood) has inferior football equipment compared to the rest of the schools. 

Concerned parent David Francis writes an op ed in the local paper stating that something has to change.  He lays out some pretty convincing evidence that Acme is underfunded when it comes to equipment.  He also points out that they have never even been to a state championship quarterfinals in football (which everyone agrees is true).  “For me,” David writes, “The ultimate metric of quality equipment is whether a team is ever represented in even the quarterfinal round of the state championship.”

Another parent, Mark Smith, responds in the comments that although everyone knows Acme has worse equipment, if we use state championship qualification to measure status or progress, we 1) won’t know whether the equipment is improving at all, and 2) are asking the kids to prove, by winning, that they are properly supported.  Can’t we find a different way to measure the equipment, or at least an effect of having good equipment that is closer to the cause (and therefore more reliable, with fewer confounding variables)?

Parents at Acme (and their supporters) can’t believe Mark doesn’t care about these kids getting better equipment!  Mark responds that he does care about the kids, and points out that the teams that make it to the state championship playoff level have 100 advantages other than equipment over Acme High.  Kids move to play at those schools.  The best coaches coach there.  They have every advantage you can think of that comes with caring deeply about winning at football and being in a position to help yourself do it.  Importantly, even with the exact same equipment, Acme will not be making the playoffs in the next 5 or 10 years. 

“So, you don’t think we should even improve the equipment?”  Mark didn’t say that.  In fact he said more than once that we should improve the equipment.  They accuse Mark of simply playing a semantics game.  What exact game and which semantics it is that got unfairly twisted, they never quite say.  David wrote that there is something important and we ought to track our progress, everyone agreed, and then Mark pointed out that the way David suggests we do so simply won’t work

Finally, David adds that “Well, whatever the total circumstances are that keep Acme High out of the state championships, we need to remedy them all.  Stop being a dick.”  But Mark points out that Acme spends its discretionary budget on afterschool programs and SAT training and that those programs might suffer if funds were redirected to football.  In other words, Acme has made different choices than schools that go all out for football championships.  Mark believes we can search for ways to improve the equipment without demanding a state championship.  


This is where we find ourselves whenever someone suggests that we measure our progress in creating a welcoming environment for women and girls by checking in with the Pro Tour and seeing how many women are succeeding there.  Most recently, this article offered Pro Tour participation as not just one thing to check in on, but as “the ultimate metric of success for all efforts meant to make Magic more welcoming.”  There isn’t a way to read that with any amount of fairness and not conclude that Daniel believes that’s where you look to investigate our progress.

But although we do need to measure our progress in inclusiveness, this metric doesn’t work.  The end result of looking there is not a significant increase in understanding what has happened over the last 1 year, 5 years, or 10 years.  

I wrote on twitter that:

The factors that I believe contribute to women not showing up on the Pro Tour are (in no order since I don’t know enough to come close to ranking anything): sexism and discrimination denying them opportunities and resources,  personal choices about how to spend their time and energy, personal choices about what a successful and healthy hobby is, the competitive advantage men have accrued over years of having fewer barriers and more interest (whether that interest is caused by a toxic community or not, the gap that accumulates is there), the fact that Wizards of the Coast has hired the most promising players and role models off of the Pro Tour into jobs that prohibit them from competing, and other factors. 

My relationship to this hobby was the most negative in my life when I was trying the hardest to qualify for the Pro Tour.  I put in hours and hours and got little tangible benefit in return.  Yes I made friends along the way, but that’s the kind of thing people say whenever they do something stupid for a long time and happen to make a friend out of someone else walking that same misguided path.  Was I financial rewarded?  Was I emotionally rewarded in a way that let me find balance in my life (eventually, yes, but not until over a decade of financially and emotionally draining choices).  I don’t want to fall into the familiar trap of measuring women’s progress using men’s goals.  And there is an overwhelming amount of evidence that men and women do not choose to spend their time the same way or pursue the same goals, for a multitude of reasons.  (This is an article about the research which I found informative on the topic:

If you go survey women at local game stores and try to measure inclusion that way, there will be confounding variables too (selection bias unless you somehow find all the ones that stopped showing up comes to mind), but does anyone doubt that that survey is closer to the source of what we are trying to measure?  There are countless numbers of ways to check in with how we’re doing as a community when it comes to inclusiveness.  Stop focusing on the one with the most confounding variables, the one that hasn’t moved whether things have gotten worse or better over the years.  If you want to measure how you’re doing on something, get close to the damn thing and measure it fairly.

Lastly, if we want to measure diversity on the Pro Tour, let's measure it.  In that case, we are close to what we intend to measure in the same way we can always claim "Acme High isn't very competitive in football" and make a case.  What gets us in trouble is making a claim of one type, and a measurement of a very different type.  The trouble we get into can be described as a severed feedback loop, total blindness, about our progress towards an important goal (and lack of clarity about what that goal is).  


  1. Well written and reasoned. I think a lot of the push back you are getting comes from people who do not believe (and in some cases think it's immoral to suggest) that women and men have different preferences. One interesting statistic on this front would be the proportion of magic players who are trans women. I suspect they are heavily overrepresented (certainly relative to cis women, probably relative to their population size). I searched briefly for the numbers but could not find them. If my suppositions are right this would be very strong evidence of different preferences being important, as it is difficult to imagine a likely culture that would be more deleterious to cis women playing than trans women. Of course it may be that trans women are still underrepresented relative to their average interest in MTG, even if they are overrepresented by population, so this obviously would not refute arguments re culture also having an impact.

    1. As I understand (and I could be wrong as far as stats are concerned), it seems that many/most transgender MTG players were MTG players before they decided to transition to becoming a woman (I remember Emma Handy having mentioned this fact about her, and I assume it is true for others in the same situation).

  2. The bottom line is moving an indicator metric as the sole point of arguing about an overall cultural issue instead of concentrating on a sociological approach is flawed. You are making the economist as quant mistake and touch on the real problems (which are cultural about pro tour acting a general gatekeeper effect for homogeneity above and beyond jus gender) and using it to discredit the conclusions through "whataboitism" creating the usual excuses for people to auto-support the status quo. And increase online harassment through the usual acts of "see this person says there aren't numbers" and happens to hit all the right markers of authority to be considered "more expert"

    There IS an academic approach - however this is not it. This brings up far more issues with Pro Tour than simply gender when you apply the actually appropriate academic approaches to it. Pro Tour is a single metric in a multi indicator tool for assessing accessibility for diverse participation.

  3. Honestly, I find this issue very frustrating, as a frequent critic of the sort of behaviors that are being called out here. The problem as I see it, is that this isn't really a problem with an individual (although as people largely exist on spectrums, of course some are people are going to be better/worse than others) at least to me this is a problem with a broader culture.

    To use the SSC lingo, I do think there are serious issues in Blue Tribe culture with this stuff (As well as the Red Tribe). I identify pretty strongly with what would be called the Grey Tribe (I.E. not Blue or Red), and while I think the Grey Tribe has issues of its own...this isn't one of them. It's not unique or new. It's been flooding out, to be sure, since the Weinstein stuff came out (again, not a surprise to the Grey Tribe, although maybe the flood of information coming out is).

    The truth is that I'm a strong critic of the Blue Tribe. I think there's big issues with sexism/racism/misogyny/abuse/etc, who largely justify it because the Red Tribe are worse (they probably are). But it's missing that there's a Grey tribe out here. Who has cookies. And individualism.

    I legitimately feel that the people who read that article, and nod their head with the strict identitarian perspective, are substantially more likely to act in those harmful ways than people who don't. All my experience points me in that direction. So even though I really do think there's a problem (I just don't think the bulk of it is a Magic problem, although certainly there's overlapping Venn circles)..there's nothing I can do as an individual to fix it, because IMO the problem isn't individuals, it's social norms in a tribe/subculture I'm not a part of. (Quite frankly, I'm often seen as some MAGA-hat wearing proto-Nazi).

    I also think there's structural issues in Magic that get ignored, especially financial barriers. Again, not unique or special, it's always been my experience that as people adopt the idea that the Blue Tribe's social/cultural dominance is the solution, that concern about structural issues gets pushed to the wayside.

    So yeah. It's not that I don't think that there's a problem. And I'd LOVE to change the status quo. (I'd love to burn our system of social/cultural standards to the ground and build it back up from scratch) And I can call out individuals. But individuals who have the social/cultural backing of a dominant tribal affiliation, when their behavior lies well within accepted group standards? That's difficult.

  4. I agree with your points, but some still see these kind of observations as problematic since they feel you don't sufficiently address the fact that even though women being present in the Pro Tour isn't necessarily the best way to declare success in inclusivity, it's important to state that it's the most promising means to achieving it. I think that's what people mean when they keep insisting you're "missing the point". I disagree with that personally, I see your argument as a way to improve discourse in a small way, and you certainly don't challenge the actual main goals of the original article, but simply amending a more forceful recognition of the fact that visibility of women in the Pro Tour is a fantastic means to an end and not necessarily the goal would improve people's reception to your view point. It may sound silly, but you can't argue against the fact that it would certainly be more positive if it's said more often.

  5. David and his friends make money from the state championship