Professionalism seems to be an amorphous concept in the UX profession. And why not, it seems anyone with an unembarassing resume can get a UX job doing the most unqualified work imaginable. Luckily, rarely does anyone outside of UX really know when someone is doing a bad job. This is a quiz I wrote tongue-in-cheek for the ACM/SIGCHI Conference. No one need agree with my answers, but they should be conversant in defending their answer.
Here is a shortened version of the original paper:
INTRODUCTION
In my 20+ experience in the User Experience/HCI Design field I have seen an incredible amount of wasted UX effort and pointless design work. Early in my career I heartily took part in such practices because it impressed the customer but did not serve the user nor the company hiring my work. The deep point was reached when a company implementing one of my most lauded designs, went broke trying to implement it. Since then, I have been sensitive to user experience designer charlatans. These are, surprisingly number in our profession.
I would like to propose a User Experience Designer’s Charlatan Test. This is a personal test meant as an introspective test the UX Practitioner can take to heart and test themselves. Not taking into account the power of those who lie with such ferocity they believe their own deceptions. For those of goodwill, taking this test with an honest introspection and perhaps show areas of weakness in your practice. No one is perfect but we can all do better.
Now the User Experience Designer’s Charlatan Test test:
1. Design is purely data determined
☐ Agree ☐ Disagree ☐ No Opinion
2. UX has an ROI and that is it’s sole reason to exist.
☐ Agree ☐ Disagree ☐ No Opinion
3. You know what a great User Experience is when you see it?
☐ Agree ☐ Disagree ☐ No Opinion
4. Do you weight positively a UX resume that shows a lot of experience in impressive UX titles?
☐ Yes ☐ No ☐ No Opinion
5. Do you consistently follow popular UX methods or tools?
☐ Yes ☐ No ☐ No Opinion
6. Can you judge a UX portfolio just by looking at it?
☐ Yes ☐ No ☐ No Opinion
7. Are you opposed to the formation of a universally accepted UX certifying agency?
☐ Yes ☐ No ☐ No Opinion
8. Can you name the aspects of the software you work on you should have the final authority over?
☐ Yes ☐ No ☐ No Opinion
9. Aesthetics drives the UX, consequently, someone with a good aesthetic sensibility can naturally create a great user experience? Top designers know the profession in their gut and between their ears.
☐ Agree ☐ Disagree ☐ No Opinion
10. Are agile and/or lean startup design methodologies?
☐ Yes ☐ No ☐ No Opinion
11. Is software development the most important aspect to creating successful digital products?
☐ Yes ☐ No ☐ No Opinion
12. Can you sum up what a User Experience does in a short succinct sentence or two?
☐ Yes ☐ No ☐ No Opinion
13. Can you name three books on cognitive psychology that you have read, relevant to user experience?
☐ Yes ☐ No ☐ No Opinion
14. When asked who your favorite designer is (outside of Steve Jobs or Jonathan Ive), do you name someone who is not a designer or a designer whom you cannot name a single lasting artifacts of digital design excellence?
☐ Yes ☐ No ☐ No Opinion
15. Do you believe it is ideal to report into Engineering or Product management?
☐ Yes ☐ No ☐ No Opinion
16. Do you believe that only designers can come up with good designs?
☐ Yes ☐ No ☐ No Opinion
17. Are you specialized into a single domain?
☐ Yes ☐ No ☐ No Opinion
18. Are you specialized into a specific set of design tools?
☐ Yes ☐ No ☐ No Opinion
19. No one really understands UX design. It is best if I can go off and just do it without interference from engineering or product management. Then just show them what a good job we can do.
☐ Agree ☐ Disagree ☐ No Opinion
20. When designing, do you show multiple and significantly alternative concepts?
☐ Yes ☐ No ☐ No Opinion
The correct answers
1. Design is data determined
DISAGREE - data can inform design but data cannot make a design decision. Well collected and analyzed data can point out a problem but there are infinite number of ways to address the problem for that a designer - at the end of the day is always trusting their gut, their gut feelings maybe supported by data and with the help of ux researchers that gut can be inspired or directed but not dictated to. Moreover poor quality data (bad sampling, asking the wrong questions, naive or inaccurate analysis) can have a detrimental effect on design. A leading UX researcher once said, tell me what you want to prove and I can design a research protocol to prove it. [1] (See: Measuring the User Experience by Tom Tullis)
2. UX has an ROI and that is it’s sole reason to exist?
DISAGREE - Software development is one of the most complex activities. It requires work from so many different backgrounds, domains and resources. Their success is inevitably a sum of their parts rather than the isolation of the whole. Are there instances where an improved UX appears to have had a positive effect on the success of a product? No doubt, but that can also be said of good product management, good engineering, none of which operate on an ROI so why should UX? UX has more vital role than improving a company’s bottom line it is what a company is really about to the end user.
With so many competing interests and stakeholders in the software or digital product process, trying to prove an ROI on just UX is dubious at best. Instead the argument should be that UX is an essential part of the product just like the code just like marketing and sales. [2]
3. You know what a great User Experience is when you see it.
DISAGREE - a user experience in intangible and abstract you cannot see this thing it is based on visuals, behavior and user perceptions. Without knowing those elements you cannot judge a UX full stop so get off your hobby-horses the UX award shows are mostly shams and temples to charlatanism.
4. Do you weight positively a UX resume that shows a lot of experience and impressive UX titles?
NO - it is possible to be a total failure at UX and inhabit all kinds of responsible UX positions for major and minor companies. Since very few people understand UX, the chances of any engineering driven company giving out meaningful UX titles are very low. Do your research and see who bestowed the title on them, was it simply a manager, a Vice President, who did they report to, that would tell you a lot more. As to a UX resume, the gobbeldy-gook in them should set expectations of what the prospect will be grilled on in an interview. Saying does not make it so, read your Descartes.
5. Do you consistently use popular UX methods or tools?
NO - Popular methods can never compete with timeless ones. There is a reason why they are timeless. There are all sorts of sketching techniques, user research ideas and entire design ideologies, like Lean Startup, which have no basis in reality. When looking for the right design tools, check out case studies and see how these tools work or do not work in real life. Case studies that are simply “we came, we saw, we conquered” without nay of the trade offs involves are useless. Our profession is nothing if not trade-offs and only the timeless design method will really get you to make the right ones.
6. Can you judge a UX portfolio just by looking at it?
NO - By looking at a job applicant’s portfolio you are reviewing the work of the visual design’s initial impression, which may not have been a primary consideration in the design at all; moreover, you have no idea:
what role the applicant may have played in that design
what their achievements were,
what the ambitions of the project were,
their inter-team dynamics,
their design rationale, etc.
I think most people in this room would be astonished by how many - even seasoned “Experienced” Charlatan UX Managers there are. I have observed many a hiring committee in many different companies and I have been appalled at the ease these charlatan glance at portfolios and judge quickly whether the candidate is a great candidate or not, even when the role was researcher, interaction designer, visual designer, or UX designer. The result has been the mediocre and arbitrary hiring practices that lead to mediocre teams, which further ruin our professional credibility.
7. Are you intrinsically opposed to the formation of a generally (State, National or Domain specific) accepted UX certifying agency?
NO - Except for trepidation on the quality of the certification process, why should you be afraid to show off your skills and prove you are a certifiable worthy designer? As Dan Rosenberg once observed, you need a license to practice architecture, why not also UX? Charlatans reign supreme when no one is minding the professional store and no objective information is gathered about a user’s competencies and competencies in which area in UX. All too often people are touting themselves UX one-man bands. This is unrealistic. Each designer has their specialty (IxD, VD, etc.) They should be certified in their specialties at the very least. However if you have your suspicions about how a board would be comprised and what they would use to analyze and certify applicants that is fair enough; if you know you profession so well get involved in the certifying committee. I don’t like the image of a certifying authority being like the Guild Meistersingers, observing just adherence to rules of the trde and not their creative application; however, I prefer that to the UX Charlatans who would rather think of our profession as a mystical exercise that defies objective evaluation, observation or accountability.
8. Can you name the aspects of the software you work on you should have the final authority over?
YES - You must want to be held accountable to the design and its implementation. Even if, in your current organization, you are not you should still be able to articulate the accountability as a desirable goal. Among the things you can mention are:
- Accountable for the design concepts
- Accountable for quality and effectiveness of all UX artifacts
- Partner in the Development process, not a service
- Empowered to log high priority bugs against bad UX implementations, which warrant high priority [3]
- Aesthetics drives the UX, consequently, someone with a good aesthetic sensibility can naturally create a great user experience? Top designers know the profession in their gut and between their ears.
DISAGREE aesthetics can be surprisingly of little relevance because they are mostly subjective, except when they are based on the user’s aesthetic sensibilities not the designers. Design rationales should rule the design decisions instead of pure matters of taste; there should be a reason/rationale for every ‘gut’ level design decision that the designer makes. They should be able to argue defend their designs with objective data and when disproven or in doubt should be man enough to change their minds. Great designers often have great instincts but great designers use more of their brain than just their instincts. Incredible guts is only half the story as the difference between guts and gall is a thin line, without objective arguments. [4]
10. Are agile or lean startup design methodologies?
No - they are software engineering methods. UX charlatans are constantly coming up with books, articles, and cults that mistake software engineering with design. The two have little to do with each other, except that they need to be scaled and coordinated with each other. If software development scaled to UX, instead of the other way around, there would be a lot less unsuccessful products. But that would assume there is a competent enough designers and developers to trust in a real iterative design and development method.
All too often charlatan UX’ers use agile or lean as an excuse for turning in shoddy work they can ‘fix later’. Agile is largely nothing more than incremental waterfalls, and lean startup is for people who only vaguely know what they are doing. Neither is an excuse for UX to turn in substandard work. I have never met a developer than can output code faster than a competent designer can paint pixels or generate of rapid prototype. [5]
11. Is software development the most important aspect to creating successful software products and services?
No - Software Development is not the most important factor in product development. There are others equally important all of which are essential parts of the digital solution: UX, Product, Business, etc. The digital solution process (software, services or digital products of any kind) starts well before development, with an idea. The process also ends with development taking a back seat to marketing. Moreover the product is most of the time a successful product because of the team approach not because of the software engineer as hero.
12. Can you sum up what a User Experience does in a short succinct sentence or two?
YES - If you cannot sum up what you do every day of your life for work then you have no idea what you are doing. There are many ways to define UX whatever way you pick it is a statement of your vision of the great user experience.
My particular favorite definition is: User Experience designers take products that force a user to think like a computer and turn them into products that mimic the way their users think.
The UX Charlatan has no idea what User Experience is, they resort to a mish-mash of visual models, abstract visualizations and a grab bag of jargon where the recipient is none the wiser, which is the hidden agenda of the charlatan: obfuscate so they can just do whatever they want and call it good.
13. Can you name three books on cognitive psychology that you have read, relevant to user experience??
YES - if you do not understand basic tenants of the human brain you have no business being in this profession. You do not need to be a cognitive psychologist, and in many cases I prefer if you weren’t; however, in the list of books below that you should have read I include some popular titles as well as scientific tomes, so there is really no excuse.
The Psychology of HCI by Stuart K. Card, Thomas P. Moran, Allen Newell
Designing with the Mind in Mind, by Jeff Johnson
Why we Make Mistakes, by Joseph T Hallinan
How We Decide, by Jonah Lehrer
The Design of Everyday Things (whose original title was The Psychology of Everyday Things) by Don Norman
Some designers with no inkling of what Cognitive Psychology is, claim you do not need to be a psychologist in order to design. I actually will go out on a limb and tell them they are wrong. So much of what we do as designers is based on psychology, Cognitive, Gestalt, and Social all play an important role whether we know it or not. So, if you know what you are doing instead of intuiting it you are, by definition, a more professional designer.
14. When asked who your favorite designer is (outside of Steve Jobs or Jonathan Ive), do you name someone who is not a designer or a designer whom you cannot name a single lasting artifacts of digital design excellence?
NO - I cannot tell you how many job interviews I have suffered when a potential candidate names as their favorite design either Donald Norman or Bill Buxton. Mr. Norman has never claimed to be a designer. As for Mr. Buxton when I ask the candidate what Mr. Buxton has designed that they love so much they cannot tell me. Other people great and small have also been named but, surprisingly, rarely is there a design heavyweights with proven track records - because most of us wouldn’t know one if we saw one, nor know how to recognize them or their outstanding products. [6]
15. Do you believe it is ideal to report into Engineering or Product management?
DISAGREE - reporting into design makes sense; no CDO (Chief Design Officer) is always proof design is not taken seriously. In your organization it may make pragmatic sense to align yourself with Product or Engineering depending who is strongest in your organization. However that is never ideal and it rarely ever makes sense, it makes our profession reliant on the kindness of strangers - and perhaps Blanche Du Bois, the Tennessee Williams heroine, is a good analogy for the state of the UX today. Not believing in a CDO is an excuse for weak design with no accountability, charlatans can always blame it on their non-UX bosses. [7]
16. Do you believe that only designers can come up with good designs?
DISAGREE - you impoverish your design thinking when you restrict it to ideas that only you or your UX colleagues thought of. A real designer does not fear great ideas from developers, product managers marketers, etc. Rather, the designer learns to embrace these ideas and learn from them.
17. Are you specialized into a single domain
NO - A designer who is specialized in just consumer electronics e-commerce, or mobile music apps, or enterprise analytics software or … fill in the blank are usually the most inflexible and unimaginative designers around. They tend to see things from a single perspective. That prespective being the mainstream of the domain and fail to see problems that others with a fresh pair of eyes can see. The UX Charlatan has, at best, mastered trial by fire a single domain’s challenges and earns by rote how to solve problems by what worked in the past, instead of analyzing the situation and coming up with a fresh new idea. My most engaging work was often my first foray into a domain, or the first foray back into a domain. In both cases I had fresh eyes able to learn and apply lessons from other domains. Another good example is how all the specialized designers of Erickson, Nokia, Motorola, etc., failed not only to produce the iPhone first but even come up with a credible answer to it, it took another non-phone company, Google, to do it.
18. Are you specialized into a specific set of tools
NO - A good designer is open to learn new skills and techniques, it often means adding new tools to their list of trusted tools of the trade. I am not against anyone having a favorite tool, but one should still be open to see if other tools might be more suitable for a given project. One should also be flexible enough that if required, they could design in new tools they have not used before.
The UX Charlatan is a one trick pony, like to a hammer all problems are a nail. [8]
19. No one really understands UX design. It is best if I can go off and just do it without interference from engineering or product management. Then just show them what a good job UX can do.
DISAGREE - The best way to impoverish design is through walling yourself off from your product development team, and, by far the worst offense of UX Charlatans not being transparent with your product teams. Walling yourself off means just going off and designing, getting feedback from stakeholders and then iterating in isolation of other team members. Lack of transparency means hiding the design process, your decision making rationale, and your design activities from stakeholders. Stakeholders should be enabled to manage and assist you better if you are up front with your process and planned activities. You should also be upfront of what deliverables or outcomes will result from these activities and their trade offs if you need to compromise This empowers everyone to come up with a workable and understandable UX plan. Lastly, by iterating in isolation, instead of in participatory, collaborative or interactive design sessions, you mask your lack of knowledge or skills at the expense of getting essential feedback from other stakeholders.
20. When designing do you show multiple significantly alternative concepts?
YES - Just presenting them with a single UX solution and getting their feedback is more an engineering than a design exercise. While appropriate for developers for designers, iteration must involve interactive collaboration as well as the use of multiple design concepts and synthesizing them into a single unified concept.
Quick Summary Results
In Summary here are the answers, add 5 points for every correct answer. Give yourself an honest score. Please do not tell anyone what it is. Unless it is added as a job interview exercise, this quiz only works as an introspective check of reality that only you need to enjoy. Moreover, if you are a charlatan chances are we know that already. But cheer up, as I said you are in good company in this profession. However given your results, does the looming possibility of a universally accepted UX certification board make your tremble or shout with delight.
Question Answer Correct
1 - DISAGREE
2 - DISAGREE
3 - DISAGREE
4 - NO
5 - NO
6 - NO
7 - NO
8 - YES
9 - DISAGREE
10 - NO
11 - NO
12 - YES
13 - YES
14 - NO
15 - DISAGREE
16 - DISAGREE
17 - NO
18 - NO
19 - DISAGREE
20 - YES
Conclusion
The purpose of this exam is an introspection for the benefit of our profession. I hope people take this test honestly in their hearts and let it serve as a guide for where they may wish to work on their profession in order to become less and less of a charlatan and more and more a professional.
For those who wish to hire UX professionals, I hope this quiz can also serve as a guide to ask questions that will lead you to a competent candidate.
Let me just repeat what this paper is not: this paper does not single out any single company or person, I mean to accuse us all and almost without exception. There is no one company, organization or designer more culpable than another. My observations cover the work of many colleagues in many companies for whom I have not worked for. So everyone should feel equally distressed.
ACKNOWLEDGMENTS
There are also many non-charlatans: designers from whom I have learned and inspired by. I am also blessed by many excellent UX Managers with whom it has been a privilege to work with and help me grow out of my own charlatanism. My professional persona is also a sum total of the amazing product managers, developers, marketers and occasional CEO it has been my honor to know. To all of them I am greatly indebted and without which I could not have made these, hopefully, pithy, wise, and (for those who can appreciate such things) humorous observations.
REFERENCES
- Tullis book Acrobat Reader 7, Be sure that the references sections text is Ragged Right, Not Justified. http://www.adobe.com/products/acrobat/.
- Rosenberg, Daniel, The myths of usability ROI, interactions - Volume 11 Issue 5, September + October 2004, Pages 22–29
- Arnowitz, Jonathan and Dykstra-Erickson, Elizabeth, interactions intro http://www.acm.org/class/how_to_use.html.
- Mackay, Wendy, Dykstra-Erickson, Elizabeth, and Arnowitz, Jonathan. Trianular of Design.
In Proc. CHI 2002, ACM Press (2002), 1–8. - Arnowitz, Jonathan Ride article.
- Norman, Design of Everyday Things, B.D. Making up titles for conference papers. Ext. Abstracts CHI 2000, ACM Press (2000), 1–2.
- FIND CDO reference, M. Guidelines for Bias-Free Writing.
Indiana University Press, Bloomington, IN, USA, 1995. - Prototyping book, P.T., Bouvin, N.O., Jehøj, H., and Mackinlay, J.D. Fluid Annotations in an Open World. Proc. Hypertext 2001, ACM Press (2001), 9–18.
- Schon, Donald A., The Design Processs in Varieties of Thinking, Routeldge, New York (1990)