Kevin Slavin, co-founder of Area/Code (now Zynga New York) and Sebastian Deterding, UX designer, academic and gamification expert, both agree that Saatchi & Satchi S’ new study, “Engagement Unleashed: Gamification for Business, Brands, and Loyalty“, is plain and simply sloppy. “In a world filled with sloppy thinking, this ‘Gamification’ deck… is the sloppiest I’ve seen in a while,” says Slavin on his Tumblr feed.
The oft-quoted finding from the study is that “Most Americans Want Gamification at Work” (something we used in our own headline). Although the number is certainly growing, the percentage of American’s that have heard of gamification is minimal. Even more basic, Slavin finds that Saatchi and Saatchi and research firm, Ipsos, do not fully understand the categories of games, “There are simple errors in taxonomy, including the fact that “Gamification” is not “games”, and “Social Games” are not what 99% of people are playing on their tablets.” Kevin’s post outlines the finding that although most people play games at work, this does not correspond to the number of people that are excited to implement gamification solutions for innovation and communication.
It may all seem like a game, but Slavin identifies one of the reasons how communication around their products has been difficult, and more generally why businesses focusing on serious games and gamification have been slow going in the past,
For the first five years of Area/Code, we were working with agencies and brands and companies to develop games that built new forms of engagement. That was very difficult, because basically, no one had any idea what we were talking about, and we had to spend a lot of our time educating clients as to what games did, and can do.
Deterding clarifies these details in his footnotes where he goes through many of the questions from the study with a critical eye. Saatchi & Saatchi S’s study may have some important information, but as Deterding and Slavin point out, sloppy research may do more harm than good.
Be sure to read both Kevin Slavin’s post and Sebastian Deterding’s additions for their full take.
Obviously, Kevin and Sebastian are trying to be controversial. There are definitely good things in the Saatchi study that we shouldn’t throw out.
>There are definitely good things in the Saatchi study that we shouldn’t throw out.
Great! Like what?
Might it be useful to specify in your response, that you are in the video endorsing Saatchi and the study?
Kevin:
True – I am in the Saatchi video, but I’d hardly say that my mere presence is an endorsement of the study. There are lots of people talking about the use of games/gamification for sustainability (the subject), and recording took place before I even knew there was a study.
But I’m not opposed to supporting the study, with caveats, actually – despite some of the issues. Here’s why:
1. More research on game behavior is good…even if it’s not gamification-related per se. The title is somewhat misleading, but the social game behavior stuff is interesting.
2. The sampling methodology is sound
3. The brand-fun connection among consumers is very interesting stuff…I’m often pointing out that the idea that things should be more fun/playful in general is *not* obvious to most strategists and marketers. Seeing consumer “pull” for those concepts is heartening.
4. The “time spent for cash prize” question is interesting, and the results are definitely interesting. I’d love to see this tied to empirical data, but if you imagine it as a baseline, it could be compelling.
And, sloppiness aside – what’s the actual risk of misinterpretation of the results?
-G
>True – I am in the Saatchi video, but I’d hardly say that my mere presence is an endorsement of the study. There are lots of people talking about the use of games/gamification for sustainability (the subject), and recording took place before I even knew there was a study.
Point taken.
On the study, however:
>1. More research on game behavior is good…even if it’s not gamification-related per se. The title is somewhat misleading, but the social game behavior stuff is interesting.
Yes. More research is good. That’s why bad research is especially bad: it brings a lot of dreck and noise into an area with little signal to begin with. The “social game behavior stuff” is just plain stupid, I contend. Asking whether people play games “to alleviate boredom” as if this is a meaningful question is disingenuous at worst, and valueless at best.
When I worked in agencies that knew how to do research, for a few years I reviewed studies on why people watched different types of television. Believe me, if those surveys looked like this one (“the trend of social gaming to fulfill pockets of boredom”) we would have been fired.
>2. The sampling methodology is sound
In that they asked 2004 people questions, yes. That the questions they asked them were leading, nonsensical, confusing and ill-informed makes the value of that sampling dubious. Asking nonsensical questions “correctly” doesn’t make them more valuable.
>3. The brand-fun connection among consumers is very interesting stuff…I’m often pointing out that the idea that things should be more fun/playful in general is *not* obvious to most strategists and marketers. Seeing consumer “pull” for those concepts is heartening.
I agree, it’s not obvious to the folks who deal with this. Which is, again, exactly why confusing and conflating language, ideas, technologies and behaviors is so dangerous to the industry. It leads a nascent conversation with bullshit. It alters the trajectory of a lot of talent and money, and that trajectory is into the mud that this study is made with.
>4. The “time spent for cash prize” question is interesting, and the results are definitely interesting. I’d love to see this tied to empirical data, but if you imagine it as a baseline, it could be compelling.
Sebastian Deterding covers this carefully, so I’ll quote his response to this question:“Q14. How long would you spend to play a clues-based challenge for the chance to win a prize of around $100?”
Writes Deterding: “Put otherwise, “For a game you don’t know, a prize you don’t know either, and an unknown situation you’re not in right now, please estimate how long you would play that game”. Oh questions about hypothetical future behavior, we all know how well you work. (The same goes for question 4a., which essentially translates into: “So even though you don’t know what we’re talking about with “social challenges”, how interested would you be in participating in one?”)
>And, sloppiness aside – what’s the actual risk of misinterpretation of the results?
The actual risk is that everyone’s breathless cheerleading enthusiasm for a concept they’ve been led to deeply misunderstand means wasting everyone’s time (clients, agencies, develops, and players) and a lot of money. Insofar as time and money are what entertainment industries are made of, this damages the industry.
In my ten years at agencies, I worked as an agency planner for four of them, and then ran Area/Code for five. We used to pay a *lot of money* for studies like these, because they are *valuable*. If they are valuable, then when they are bullshit, that value isn’t lowered — it’s inverted.
You can keep adding cheerleaders, or you can swap out the team with people who know how to play. The cheerleading doesn’t move the ball further down the field. From what I’ve read, winning is the main reason anyone does anything, ever.
Oh, and boredom. But just women, according to the study.
I personally think that a survey study where you basically ask employee “would you like to play at work or would you like more incentives” is hardly revealing at all.
Strong case studies over longer periods of time are only what will help us get a better idea on the true potential of gamification.
I like Sebastian Detering’s talk “meaningful play” when he presents gamification companies (bunchball, get glue etc..I really just think that maybe people are surfing on a trend but its not sustainable. How long have you ever played a game for in your life? most of the time not very long…or maybe there is lottery, card games and so on.
My point is how often do you need to review you game mechanics to maintain engagement and efficiency? Is it better to have simple game principales that change often? or more complexe designs that takes more time for users to master?
>My point is how often do you need to review you game mechanics to maintain engagement and efficiency?
Well… ask the people who made (or play!) chess or poker or baseball. Or think about whether any of those games would be more fun if you had to learn and master NEW rules every time you played them. Or are they are more fun because you are working to achieve mastery over something, this time a little further than last time?
>Is it better to have simple game principales that change often? or more complexe designs that takes more time for users to master?
Simple game mechanics can take forever for users to master. That’s one reason people will keep playing them, e.g., http://en.wikipedia.org/wiki/Computer_Go . “Simple” and “Complex” rules don’t equate to “simple” and “complex” experiences playing them.
But none of this is covered or even hinted at in Saatchi’s groundbreaking study on gamification. I think the section about “working in a socially responsible company for a salary reduction” must have displaced it.
First of all thank you for your answers!
I’ll probably be working for a company specialized in entreprise social networks. Thus i’m very interested to see how can such platforms be used to host game mechanics within a company.
Baseball and poker are never going to get bore people (even though i’m already bored by baseball!) that I agree they are perfectly designed. But their only goal is fun or making money.
How can you create simple game mechanics that are efficient within a workforce without damaging current processes? I mean at the moment entreprise social networks have a pretty clear goal: Improve communication within a network of people.
My question is where would you start? Employee collaboration? Discipline? Creativity?…
The Saachi study implies that since people “play more” games, at least electronic games, it means that gamification in the workplace would be more accepted and embraced in the workplace? Has gaming become more popular?
There is also one question that game up when reading your comment. Everyone has different games tastes meaning the biggest difficulty in designing a game is to get a maximum of people into it. How can you create game mechanics for a set group and achieve the same amount of engagement as if they all chose to play a specific game?
Jonas, thanks — those are all great questions.
The answers to those questions are complex and long-winded. But start by isolating for yourself what the differences are between game mechanics and games. A shopping cart has wheels and traffic flow, but you’re not driving when you use one, and so on and so on. With every question, think about whether you’re interrogating why people play the games they play, or whether you’re going after what makes the game mechanics of those games meaningful.
You see that it’s both simpler and more complex than the Saatchi study, and the conflation of terms and ideas is mud, in what might someday be a clear conversation.
In the meantime, two very good videos you might like, for *very* different reasons:
JP Rangaswami from Salesforce.com, gamification as the lipstick on the pig of work: http://bit.ly/jZkOJN
Frank Lantz (Area/Code’s co-founder and Chief Creative Officer) on Poker and Go: http://www.gdcvault.com/play/1014383/Life-and-Death-and-Middle
Thanks a lot for the great info. It was really enriching. The growth of Salesforce is impressive and the employee interactions they create equally, nice to here they are looking to innovate instead of following the traces of those ERP giants.
Do you by any chance trails or case studies in corporate environments?
Regards,
Jonas
[…] Gamification.co’s blog about both, the comments thread is well worth a read as Slavin dissects the issues further. […]
[…] Deterding For a little update, Kevin has expanded the dissection of his dislike in this comment: http://gamification.co/2011/06/15/recent-gamification-study-sloppy-according-to-experts/#comment-112… […]
[…] all I found was a collection of disgruntled bloggers writing about how studies were biased or sloppy. So take those purported promises with a grain of […]