Wednesday, June 24, 2015

38% of Claims on Tumblr are Unsubstantiated

“According to some research I can’t share with you, __________” can lead to a frightening amount of emphasis on a small set of studies that may be very flawed.  Beware the man of one study, and turn and run full speed from the man whose one study has not been examined carefully. 

When the qualifying phrase “According to some research…” is omitted and the conclusion, _________, is given as fact, things get even more frightening. 

Mark Rosewater started citing WotC market research of some kind, without revealing the methodology or definitions used, a few months back, and in another Tumblr reponse that went MTG-viral last week, he decided to drop the context and fully embrace his man-of-one-studydom.    

MaRo writes, “That’s what this conversation is about. Women make up 38% of Magic players yet this isn’t remotely reflected in in store play. Why? What factors are causing this to be so? And if it’s going to change, it requires those of us in the majority to stand up and say, ‘You know what? This isn’t right. We need to change this.’” [emphasis mine]

This is now clearly being used to drive discussion about policy, and by all indications it is about to drive actual policy (diversity initiatives).  And the figure 38% is now stated as fact.  By releasing just the conclusion in a way that cannot be intelligently scrutinized in detail, motivated by the fact that this statistic lines up with his narrative, MaRo is being careless at best and dishonest at worst.

Let’s go back to that original, more complete citation: “Our most recent market research shows the gender breakdown of male to female is 62% to 38%.” 

Is this 38% of people who know what Magic is, have ever played Magic, have ever purchased a Magic product, play Magic at least once a month, purchase Magic products for their own use at least twice a year, have opened an MTGO account, or something else entirely?  Before we get to the accuracy of their measurement, what are they even measuring? 

Once we know what is being measured, and how, I suspect we will better understand why the number is so high – shockingly high relative to the ratios we observe in tournament attendance.    

Why do I think they MaRo owes me the courtesy of showing his work?  MaRo is making a claim that contradicts popular wisdom based on an avalanche of lived and observed experience.  This popular wisdom isn't the end of the story, but it's upsetting to see people toss it aside so easily, without even asking MaRo to show his work.

MaRo is in some ways absolving himself of his responsibility when he pins it on us and implies that it's our responsibility to shape up out there and make a more welcoming environment for girls and women who want to play Magic.  "Hey, it's a healthy 38% when it leaves my nest, women love my work."  Out there in the tournament halls, we're doing work to improve the tournament environment, but we'll get more done measuring our progress using a reliable baseline rather than an inflated one.  Importantly, MaRo leaves us no choice but to take (a hypothetical) 10% female attendance as a sign that things are horribly broken when my intuition is that if we have 90/10 and everyone is having a good time, we've done great (not saying we're there today).  

But Matt...
a)      Isn’t even one or a few studies better than your anecdotal experience?
b)      You play tournaments, of course you think 38% is high, but more people play casually than in tournaments!

First of all, I hope MaRo doesn’t cite my experience on his Tumblr either, it may be horribly unrepresentative.  Even over 20 years a person's experiences can be biased, and selection pressure can operate such that a person never encounters a true cross-section of the population.  

We need to try to paint a composite picture from many data sources, each of which has to be unpacked and examined, not presented as a black box.  Terms have to be defined.  For example, how do we define "Magic player?"  Is it possible that changes in the gender ratio we observe as we change contexts are a function of implicitly swapping out the definition of "Magic player" to include only more deeply interested players?  Perhaps the odds of ever buying a single booster are 62:38 male, but that initial gap is present at every level of interest such that you're 62:38 to buy that next set of packs, then 62:38 again to visit an online retailer to get singels, etc. etc.  The cumulative impact on the ratio of men to women participating in, say, a Grand Prix might be 97:3 if these layers of selection have occurred.  If so, then we're back to asking what causes an interest-level gap, and we can't be as confident as MaRo is that it isn't primarily the game and which demographic it appeals to most.  More stats and more detail on how things were defined and measured will help us see what is actually going on, what is causing the gap, and ultimately whether guys being immature and rude at the local store is a deck chair or an iceberg relative to the titanic lack of diversity we face at that level.

And yes, I play in uber-competitive Magic tournaments.  But my brain doesn’t shut off when I leave the tournament hall.  Every week, I tell men and women I meet that I play Magic.  I can tell you that the odds I get a “me too” response based on the sex of the person I tell does not correspond to anything close to 62:38.  People tell me their kids play Magic and it’s not 62% sons and 38% daughters they speak of.  Again, I don’t offer my experience as an alternative to MaRo's stats, but as an explanation of why I'm asking for more than just his word.  I'm skeptical based on how I'd defined the terms, and I want to see how they defined the terms and gathered the data since that might explain their finding.

Wizards of the Coast, show me that my experience is not representative.  Please, show me.  But don’t just tell me.