Wednesday, February 21, 2018

That wasn't even the worst question...(GDS3)

Mark Rosewater revealed the answer key to the Great Designer Search 3's multiple choice test about a week ago and some interesting discussion has followed.

Even those of us who weren't involved in GDS3 directly took interest, most notably PVDDR:

Thus began a long Twitter discussion over whether that 4/4 flyer question was sloppily worded or downright incorrectly worded, or something else.  

I thought the question was so poorly worded that if the test actually mattered, and I was in charge, I would throw it out.  But today I'd like to point out that this wasn't even the worst offender on that same test!  If I could only throw out one question, I would throw this one out:

32. You've designed a card, and you want the Play Design team to like it. How should you choose your casting cost? 
a. Ask someone from the Play Design team to choose the casting cost.
b. Find a similar card as a basis for the casting cost and then make it one cheaper.
c. It doesn't matter; they'll fix the cost if you get it wrong.
d. Pick the strongest cost that isn't broken.
e. Use your intuition.
Here is Mark Rosewater's notes in the answer key:
Making Magic is a collaborative process. To best accomplish this, you need to learn to let people play to their strengths. Could I cost a card? Sure, but I won't do it as well as a play designer who was hired specifically for their ability to judge power level. By asking them first, I lessen the chance that playtesting gets affected because of poor costing. Also, if your goal is to get Play Design (or anyone, really) to sign off on something, it helps to involve them in the process.
I don't think MaRo's answer is the best or even the second best answer among the choices given.

The biggest problem with this answer is that asking another team for help is absolutely part of a healthy exchange of ideas and part of solving complex problems in a collaborative way, but it's a very dangerous hammer if even routine daily tasks such as picking a casting cost for a card you want spikes to like start to look like nails. 

"Who should write the first draft?" is an important question in many contexts.  "Who should review that first draft and help improve it?" is a separate and equally important concern.  But the answer to these challenges is certainly not to collapse them into one question or one step.  It's fine if Play Design has early and frequent input into the design of casting costs, and it's even fine if there is an exceptional circumstance in which they do get first crack at it (if the card has "Emrakul" in it's name for example, maybe we can shortcut right to Play Design input), but the question's only parameter here is that the designer wants Play Design to like the card.  That's way too broad a category of designs to be going out to a team whose primary job isn't design and asking them to do the initial design work.

One answer (not my favorite) that is still better than MaRo's top choice is "Pick the strongest cost that isn't broken."  This isn't an ideal starting point since you won't always be right about what is broken, you might not even have enough context to know what broken means in the world the card will be released into, and not every cost should be near the maximum power level anyway, but your best guess at strongest not-broken cost is a better starting point for Play Design to work with than no starting point at all ("you pick it"), especially if you just interpret "isn't broken" conservatively.  So if you're using this sparingly (which you better be - see above) then "strong but not broken" plus your intuition (hmm, what's that?) about where that line is, applied conservatively, leaves you at least actually attempting to do your job before handing it off to QA.

Because the other answers are all deeply flawed, "Use your intuition" is the best answer among those presented, even though it's a weird answer in the context of multiple choice.  You don't think your best designers can create casting costs that please the Play Design team within the boundaries of fair power level, even as a first draft that will get additional testing?  That's a depressingly low bar for design.  And if you're thinking, "'Intuition' is just too ill-defined to scale properly or be consistent designer to designer" or something like that, my response is, "If intuition wasn't involved in a big way, they'd be tweaking algorithms instead of interviewing humans for the design roles."  Let me get this straight, intuition is something we all know is a major asset your strongest designers have, but incoming designers are scored INCORRECT if they say they intend to use their intuition to design a card?

I"ll end with this: The Legal team (my team) where I work often reviews marketing materials before they are released, to check for claims that might not be well supported, IP issues like use of trade names & media, trade secret/confidentiality concerns about the level of detail in any description, among other issues.  If I'm interviewing a marketing candidate and I ask them, "How would you make sure that marketing descriptions of, say, certain security features are satisfactory to the Legal team?" the answer, "I'd use my intuition about what Legal is looking for, then have Legal review a draft before it goes out" is a pretty good answer (not quite as good as actually unpacking what the issues might be or how to find out, but pretty good).  On the other hand, "I'd ask Legal to draft the content for me" would be a horrible answer.  


  1. I agree wholeheartedly. Many of these questions were written to see if you could psychoanalyze Maro and the type of behaviors he asked people to write questions about.

    For the Paulo question they were thinking about a very small design rule related to french vanilla multi-color creatures and wanted to test our research skills and follow the rules. They explicitly didn't want us thinking about how rules might take us to a weird place and we might want other input.

    In your example they are looking for the extreme opposite where they are reminding you that you are an INTERN, and that you can't possibly make a card others will like without other external information.

    They are subtly reinforcing how important a group is that is less than a year old. Right..... A group who's job is to test cards to see if they are broken, a group that is made up of a bunch of spiky tournament players, what could they possibly like in a card? But we aren't supposed to have done our research on them unlike the Paulo question.

    I work in chemical development, and quality is always reviewing my work to make sure our products aren't broken. But never once have I considered "There is a big gap in my work, I am not even going to take a stab at it before I send it out for review"

    The real answer is probably "Do your work to figure out the cards place in the set, use your intuition to take a first stab, work with development to iterate the cost so that it fits with what everyone else is working on, make sure the card history file includes anything other might need. Finally introduce it and follow up with play design to work out any kinks in usability, respect their feedback and prepare to rework your card based on their experience."

    One thing I know is that with GB flying vigilance creature I was disappointed because I almost made it, after this question I am not so worried about what my score was.

  2. The flying vigilance question was fine dear lord. They gave you a limitation "make a creature with keywords that can't be monocolor" and you were not supposed to find ways of not following that limitation. A flying vigilance creature can be mono white so that rules out any Wx answer. a flying creature can't be black but a vigilance creature can't be black, a vigilance creature can be green but a flying creature can't be greend, therefore BG is the answer. Yes a flying vigilance creature is more than likely UW but that was not the question. If your boss gives you a limitation to work with you are not supposed to find ways to not follow it or "follow what feels right".

    1. To be clear as Paulo has pointed out, the question didn't say what you said it did. It used softer language that was supposed to be interpreted at a hard and fast constraint. Misinterpreting that is the fault of the question writer not the test taker.

  3. The problem with this question is that it is about internal processes. Why should an intern already know that? Also, this question has a rookie flaw of being about play design and the correct answer is the only one to use that term.

    Basically, this question is exactly the same as this:
    Building Services issues key cards needed to gain entry to the building. You arrive for work and realize you forgot your card. What do you do?
    A Go to Building Services to get a temporary card.
    B Go home and get your card.
    C Sneak in with a group of co workers.
    D Borrow a card from a co worker.

    The answer you should choose is A, but WOTC may have a zero tolerance policy so the actual answer is B. But why would an intern be expected to know this before starting?