A Teachable Moment

Actually the aspect of Sebastien’s review that I found most insightful, and the one that I think is most serious is the misuse of language.

Intrinsic/extrinsic motivation is one example, or the labelling of points as ‘game mechanics’ is another. There are many more where he calls out the fundamental reasoning and use of concepts (the co-opting of Bartle, for instance) and the over-reduction of some ideas to the point where they lose all sense of meaning. These are not small things that can be covered by errata, they are core to your whole argument Gabe.

What it comes down to is that writers like Sebastien and myself are increasingly convinced that the kinds of ideas that you and folks like yourself are putting forward contain a yawning gap of understanding over why games actually work, and thus your idea of ‘gamification’ is itself deeply flawed.

I think you owe it to readers to engage Sebastien’s review point-by-point and engage with his breakdown of the concepts in your book. He has definitely hit on a nerve with them (239 tweets of his post to 3 for this one) and deserves a detailed response.

Anything less merely validates what he’s saying.

Completely agree with Tadhg.
The objections raised by Sebastian are more than valid and the misuse of a lot of game design terms (one for all: game mechanic) is evident.
All the critics I’ve heard about Sebastian article can be reduced to remarks like “you’re envious”, “write your own book” and such. Not exactly a good way to carry on a useful conversation.
But probably there’s no intention to do it.

“t is spectacularly naive to suggest that research – by mere virtue of its publication – is somehow “the one truth”. Almost every piece of work in social science and psychology has significant methodological problems, and opinions about what works (and why) go in and out of fashion as quickly in academia as they do on the runways.”

This is exactly the same argument used by those who deny the science around climate change, global warming effects, etc. That “every piece of work has methodological problems and opinions…” But it is a convenient way to reframe dozens of studies as somehow just “opinions”. You know perfectly well that the science Sebastian is referring to is *fact*. Results happened. Over and over and over. Decade after decade. That there are opinions around the precise mechanism by which the results occur does not render the results themselves “opinions” or “fashion.” By your reasoning, man of the most well-used, time-tested medications should be discounted because we still do not understand the precise mechanism by which they produce the results. Conflating the “why it happened” with the “what happened” is a convenient, but wrong, understanding of science. This surprises me, given now often the word “science” appears in your marketing materials.

As for this one,
“I believe gamification critics share a common goal… Even if they lack practical business experience from which to conjecture.”

That is a lie, and you know it. One of your most outspoken critics is, for example, Kevin Slavin who sold his company to Zynga. That not only counts as “practical business experience” but puts him smack in the middle of the very domain you are claiming as your own. And as a reminder, I, too, come from a successful business and technical background, not academics. Buster Benson, who stated publicly that he supported every word of Sebastian’s review? Pretty sure he counts as not only a person with “practical business experience”, but you’ve referred to him in your own work. Framing your critics as somehow lacking relevance is spin, not fact. Even those who once supported you are surfacing as critics.

And I will yet again ask the question: why is the “thought leader” and “expert” in this space (I.e. You) NOT actually using gamification? You are, for example, telling the publishing industry that “gamification is the key to the future of publishing” yet in your own two books, you use NO gamification, and you have consistently failed to use gamification to increase engagement or sales around your own two books (books with actual sales numbers that, by most publishing standards, would count as business failures). He who pushes others to embrace and pay for a solution he himself has failed to use ought to spark far more skepticism. Not sure why others have somehow failed to notice that your case studies and success stories have come from others, not your own “practical business experience.”

You do appear to excel at traditional marketing techniques which, far as there is evidence, are the tools you apply. Still have not seen you use ANY gamification in your own work. That more people are not asking the question, “why is this guy telling us to use something he himself does not use?” is a surprise to me, though there is a long history of this phenomenon in the get-rich-quick and self-help-motivational-speaker space. As for your company being dedicated to the “science of engagement”, have you not asked yourself why you have STILL not been able to produce meaningful, measurable, useful engagement around your own books and websites?
(though I do believe the one gamification attempt you and your publisher made at building engagement around your book led to four retweets.)

It’s depressing to see this. It’s clear that Sebastian is a very bitter man. I pains me just to sense his anger in what he writes.

It’s never right to take someone’s work, someone’s creative gift to the world, and write a hateful note on it. You’d be better off just keeping it to yourself. We have too much hate in this world already! I would have thought game experts would be more focused on fun, pleasure and community than in anger and destruction.

In any case, this is typical of academics, unfortunately. It’s not surprising to me that Sebastian declined the offer to discuss this live, academics tend to love to talk a whole bunch but then don’t take action. I really liked Sebastian’s slides so it’s disappointing to see him act like this.

And in terms of the “research”? There’s also a lot of *research* that has shown that most papers published are biased, unregulated and tampered with to benefit someone (funders, publishers, researchers, etc.) Add that to the fact that science is periodically rendered obsolete and old “truths” replaced by new ones, I would urge everyone to take any piece of research as just another opinion, and RUN from anyone who claims to know an absolute truth.

Even if he has some good points, I wish he would have just written about them, instead of using them against someone’s words.

Very sad… 🙁

Oh, here’s another one.

Hi Gabe,

thanks for the response! I agree and hope we can all have a teachable moment in this :). So in that spirit:

First, some clarifications

“Sebastian is a vocal critic of the Gamification industry.”
No, I am not. I have stated time and again – during my previous talks, in the review, and in my response to O’Reilly’s reaction – that I see value in gamification done well, that I highly respect certain people and companies in the industry, and that my critique is not aimed at gamification as such, but at certain ways it gets presented and implemented. I understand it would be an easier story if this were just “pro vs. against gamification”, rather than “pro gamification, but against the specific rendition in Gamification by Design“, but that’s how it is.

“Unfortunately, he turned down our expenses-paid offer to come and share his unedited, critical opinion directly at GSummit, choosing that moment to level his criticism of my book and our widely proven methodology online instead.”
Indeed, you kindly offered me to join the Gamification Summit on the Summit’s expenses. As you will also remember, I replied to him that I would likely not be able to make it since I had at the time of his email already submitted a proposal for a gamification roundtable at the bi-annual DiGRA conference on game research, scheduled in parallel to the Gamification Summit. The proposal got accepted, hence I couldn’t make it to New York, which I did let you know.

“Despite our differences, and many suggestions to the contrary, I believe Gamification’s critics share a common goal with its advocates: to make the world a better (and more fun) place, … Perhaps my optimism is as naive as some of our critics’ personal advancement strategies, but I remain firmly committed to these goals and excited about the great future we’ll create together.”
This statement implies that the true motivation behind my review was “personal advancement”, not the actual concerns I stated here. But since, as I also already stated, my review likewise unintentionally slipped into the personal once (similarily trying to guess an underlying motivation), I’m happy to call it even with that :).

So moving on, your reply states that my critique falls in three categories: errata, plagiarism, and differences of opinion on theory and practice. Let’s look at those point-by-point.

1. Errata

It is interesting to put this as the first category of critique, as it implies that this is the most important part of the issues raised. In my review, I counted all in all two things which can be seen as errata: The misreading of the social engagement loop diagram, as mentioned by Zichermann, and a typo in the name James L. Halperin.

The remainder – and the overwhelming majority – of the review points out either underlying conceptual flaws, or systematic misrepresentations, and what I see as problematic with them – such as positively endorsing to obscure the actual value users get out of an interaction with a brand; using a misunderstood, obsolete typology of players to understand the motivation on non-game application users; or calling things “game mechanics” that have nothing “gamy” to them.

So I’m happy to submit the “James L. Halperin” typo to the official errata page, as you suggest (the engagement loop is already there), to free you up to respond to my substantive critique of said flaws if he wants to.

2. Plagiarism

I did not write the review with the intent of accusing you of plagiarism. The title of the review points chiefly at the writing and structure of the book, one that – to my eyes – displays a hasty collection of things mentioned in the gamification discourse without a coherent flow of argument underneath, putting incompatible bits side-by-side, like Csikszentmihalyi’s flow theory and reinforcement schemes. The main issue of my review are those conceptual flaws.

Furthermore, I did not accuse, but pointed out case-by-case, open to any reader to see for herself, where I saw no due credit given, which is a different thing than plagiarism, like for instance the unmarked verbatim copy of a piece of a blog post by James Halperin I pointed out. I concluded from the amount of these instances that the book is “lacking due credit to an extent that borders on plagiarism”.

As for the slides by Amy Jo Kim, which I gave as one example for such lacking credit, you state that both your slides

“(and Amy Jo’s) are based on substantial prior research. For example, the player types research comes from Richard Bartle and the stages of mastery work from Oppenheimer and Oppenheimer. One can also find inspiration for the viral loops from Penenberg (and myriad others) and game design action verbs from Lopes, Kuhnen et al – without taking anything away from Amy Jo.”

Indeed, Lopes and Kuhnen have written a nice text on “Game Design Cognition“, Dreyfus and Dreyfus have layed out a model of five stages of skill acquisition (I assume you mean Dreyfus and Dreyfus, not Oppenheimer and Oppenheimer – and I happily grant that this is an honest typo for sure ;-). And to your credit, you have already used Bartle’s player types in his 2009 book Game-Based Marketing, and your player journey exercise replaces Kim’s three stages of online community with Dreyfus & Dreyfus’ five stages, and also features some additional design exercises (like ‘find out what your Bartle type is’).

But I would ask you to kindly point me to the place where any of these people before Amy Jo Kim suggested to create a gamified design in a series of exercises entailing to first write the player story, then rank their top 5 actions from a list of verbs (matching almost all words on your list, but with no match with the game verbs Lopes and Kuhnen mention), then place those actions on the Bartle diagram, then place them on progression of levels of expertise – in that precise order?

And yes, one can find inspiration for talking about loops and virality from other sources. (Personally, I find the work of Kontagent on social game metrics informative, even beyond social games.) But again I would ask you to point to the place where Penenberg’s Viral Loop with its three categories of viral expansion loops and features of viral businesses (stackability, point of nondisplacement etc.), or any other person in the viral marketing space, has modeled a “social engagement loop” with that exact name and exactly four phases exactly named “visible progress/reward”, “motivating emotion”, “social call to action” and “player re-engagement” exactly in that order. The only way the model in Gamification by Design differs from Amy Jo Kim’s is that it removes the brackets around “social” in “(social) call to action” (see Kim’s presentation slide 140 and the book on p. 68).

Finally, you note that “Perhaps the choice of omnibus citations at the outset of the book and bibliography creates some ambiguity, but the current standard is not to burden the text with dissertation-style notes inline.”

That is a curious statement. I happen to own quite a couple of similar design/strategy books from O’Reilly – excellent reference as well as introductory material written for similar audiences, like Web 2.0: A Strategy Guide, The Art of Community, Designing Social Interfaces, Designing Web Interfaces, or Building Web Reputation Systems, as well similar books by other typical industry publishers, like New Riders, Rosenfeld Media, or Morgan Kaufman.

A standard that unites all these books is that they reference used sources and recommend further reading – with a hyperlink where possible –, on the page and/or at the end of a chapter. Even the most hardcore plain-English-actionable-introductory-business-focussed books like Web Design for ROI do so. In contrast, Gamification by Design does not, has no bibliography, as Zichermann claims (or I didn’t find it), and the acknowledgement section names three books all in all. Instead, Gamification by Design funnels readers repeatedly into your own site GamificationU.com.

As I stated in my review, the annoying thing here is not lack of credit, but self-linking, together with the additional work created for the reader to find the sources the book merely mentions in passing, which incidentally also makes it harder to cross-check what the sources say with what the books says (like cross-checking with Lopes and Kuhnen, Dreyfus and Dreyfus, or Penenberg above).

3. Differences of Opinion on Theory and Practice

The third point of contention you see is both a ‘naive overreliance’ of my views on research, and mere differences of opinion on what research says. Quote:

“It is spectacularly naive to suggest that research – by mere virtue of its publication – is somehow “the one truth”. Almost every piece of work in social science and psychology has significant methodological problems, and opinions about what works (and why) go in and out of fashion as quickly in academia as they do on the runways.”

I do not and did never claim that published research is “the one truth”, and I would not know of any researcher who does. As a pragmatist myself, nothing could be further from where I stand. As I said in another exchange we had on this, “The whole point of scientific constructs is to point to stable regularities in intersubjectively observable and reproducible data, and to define and explain them in a network of other constructs and relations to make them as robust, inflexible, noncontingent as possible.”

There is a difference between a contentious field in science, with good argument and data supporting both sides, and a field were all data and argument unilaterally point in one direction, and coincide with what practicioners have learned from experience. The importance of mastery, achievement, competence as a core component of what makes games “fun” is such a case. Again, I invite you to point to but one peer-reviewed empirical study that states that incentives and status, not mastery is at the core of what makes games fun.

There is a difference between “academics believe” and a scientific community – after a long process of debate, experiments and replications, counter-arguments, further experiments refuting those counter-arguments, meta-reviews statistically teasing out the effects found by all experiments in the field – finally settling on a certain body of knowledge as ‘well-established’. The research on the “hidden costs of extrinsic rewards” is such a case. Is that body of knowledge always incomplete and open to amendment? By the very definition of what science is. Is it also the most reliable, acid-tested piece of knowledge on the topic out there? It is. If push comes to shove and I was a business decision maker, would I want to base my business decision on that knowledge, or on someone’s “personal experience” with his clients that the product he sells ‘works’? You decide.

There is a difference between “going in and out of fashion … as they do on the runways” and scientific progress. The cognitive revolution, replacing behaviorism with cognitive psychology as the dominant paradigm in psychology, was a slow, 25-year process from the mid-1950s to the early 1980s, working its way from argument to argument, experiment to experiment. And today’s neurosciences or evolutionary psychology build on the work of cognitive psychology, they do not just randomly flip back or bounce to some previous opinion.

Finally, there is a difference between having different “opinions” and misrepresenting what the other side says and the evidence it brings to bear – i.e. Csikszentmihalyi, Lazzaro, or the research by Amabile, Dweck, Ryan and Deci summarized by Pink, all named in Gamification by Design. For example, in Zichermann’s presentation of said summarizing book by Dan Pink, Drive, he states that “While we agree that cash is not always motivational and in some cases is actually demotivational, other extrinsic rewards can be astoundingly motivational for players. For example, long-term social status rewards can be particularly effective at nurturing creativity and play.” (p. 27)

This misrepresents both that Pink and the research he references talk about extrinsic, tangible if-then rewards in general, not just money (also certificates, trophies, food, badges, etc.), and that the researchers Pink cites explicitly studied the effect of caring about “long-term social status rewards” for creativity, and found that it was detrimental both for the commercial success and the personal well-being of the artists they studied. You may hold a different opinion on why those studies found those results (for instance, “flawed methodology”), or why they don’t matter in gamification contexts, but you should at least present them correctly, so that the reader make make up her own opinion.

Theory versus practice?

Your reply casts a stark theory-vs-practice gap, with me, the “PhD student”, who apparently demands industry-atypical “dissertation-style notes”, “lack(s) practical business experience from which to conjecture”, and merely “believes” that “the techniques in the book work” or not, in contrast to “the doers” like yourself, who has written “a practical book for practical purposes”, “based on my experience with dozens of clients” and “interviews with hundreds of practitioners”.

This is a false contrast for at least three reasons:

1. Research holds enormous values for practical business decisions, and in my past presentations on the topic, I have always tried to demonstrate that connection to direct business cases. For instance, as I also noted in the review, research reliably shows that overall, men prefer competitive situations much more than women do. The startup Fanlib.com employed an early version of “gamification” by trying to motivate fan fiction authors to submit their work through competitions and sweepstakes, which stirred a huge backlash within the fan fiction community, and the startup quickly folded (see references here, here, and here). Now I don’t claim that there weren’t also many other things problematic with Fanlib.com, but had they known the research on competition, and the research on fan fiction showing that the overwhelming majority of writers are female, they might have done a smarter job. Why would anyone not have her business decisions informed both by practical experience and existing research? I agree from my own experience that business decision makers don’t have the time for a thick book and are in need for a good, actionable ‘introductory summary’. But what use is a good summary of flawed thinking?

2. As Kathy Sierra noted, my critique is not ‘one lone student’s personal opinion’. Numerous practicioners highly respected in the gamification field have publically supported my critique of the arguments presented in the book, or fashioned their own critique, such as Buster Benson or Tadhg Kelly. Jon Radoff, who spoke at the Gamification Summit, likewise publically spoke up multiple times against reducing what works in games and gamification as behaviorism.

3. As for myself, I have worked several years in the industry as a program manager and user experience designer, and continue working on projects increasingly involving the use of game design beyond games. Among them, I have helped the world’s second-largest online retailer optimise their check-out process, and worked with browser games on improving the conversion of their onboarding and virtual item sale processes – in user experience design, it doesn’t really get more ‘down to business’ than that. I happily invite everyone to ask my past employers and clients whether my work lacked practicality or business acumen.

In summary

To conclude, the argument my review makes is not a simple me “believing” that gamification ‘does not work’ versus your “experience with dozens of clients” that ‘it does’. As I stated both in the review and elsewhere, all things you mention have some relevance: learning by association is important, incentives work, and social status motivates. My argument is:

1. The book Gamification by Design is seriously underplaying and misrepresenting the existing research and business cases that incentives are of limited effectiveness and often come with unwanted side effects, for no apparent reason – businesses only benefit from becoming smart about these things.

2. The book is seriously overplaying the importance of social status and neglecting the complications and prerequisites for something becoming a meaningful conveyor of social status.

3. By presenting reinforcement, incentives and social status as “the core psychology of what makes games compelling”, the book is contradicting both all peer-reviewed empirical research on the core motivational appeal of video games and play, and all game designers I know, have read and spoken with, including the researchers and designers the book points to as important sources.

4. By doing so, the book is seriously underplaying the existing research and business cases on the elements of games and play beyond incentives and status that make them highly engaging, and have been shown to be crucial for engagement in workplaces and education alike – components like achievement and competence, autonomy, trust and safety, or meaning and purpose. Thus, it needlessly leaves on the table for its readership the huge motivational potential of games, play, and extending them beyond games.

To all of this, you may counter that his book is “focused not on games at all, but Gamification as a unique, emerging and hybridized discipline.”

Now I have nothing against studying or deploying customer loyalty and employee rewards programs, long practiced and promoted in marketing and human resources by bodies like the Enterprise Engagement Alliance – I just hope that the people doing it take notice of what psychology has learned in the past decades about human motivation. I have also nothing against you branding your own variant of this, and I welcome your co-founding of his own Engagement Alliance, as this indeed helps to clarify the space your are coming from and aspire to.

My issue is: Why, when your are indeed “not focused on games at all”, then claim that your book brings to businesses “the core psychology of what makes games compelling”, and call what your propose “game”-anything, namely “gamification”, and claim that it is “the process of game-thinking and game mechanics”? Why such forceful repackaging/misleading advertisement against the majority of both researchers and design practicioners? Why not inform readers about the complications and caveats involved? Why not be more careful in presenting what knowledge and experience is already out there? Why leave so much value for readers on the table?

I am happy to continue the discussion on these matters, as I agree it will be a teachable moment for all.

Let’s see Gabe address Deterding’s criticisms with equal attention to detail, cited sources, and clear language (not marketing language). I think that if Gabe continues to fail in this regard, then he will be exposed as just another marketer with a malevolent agenda.