User Testing

“If you spend time observing and talking with people who use your product or service, fantastic creative ideas start to appear. None of this happens in front of a whiteboard in the comfort of your office.” — Jake Cook^[1]

Jake Cook is describing the importance of empathic design, which depends on feedback gained via user testing. The general rule when designing a new or improved product, service, business model or experience is to test early and test often.

That’s because as Dana Patton explains, “everything that we think we know about our users are assumptions, and assumptions must always be tested.”^[2] Gathering early-stage feedback is helpful to inform the next phase of design decisions you make as you iterate toward the final version. “Get out of the office and go find your users,” Patton encourages, “Talk to them, listen to them, observe them, and understand how they feel. You’ll be happy that you did, and the work will be much better as a result.”

For that reason, making user testing an ongoing activity is important, so that you can develop an authentic understanding of how real people experience your offering — how they understand it, how they feel about it, and what they would recommend to improve it. By conducting quick and continuous assessments, starting during the very first week of your project, you’ll increase the likelihood that the innovation will lead to an increase in sales, conversions, web traffic, and customer loyalties (referrals and retention). As David Peter Simon advises, it’s good practice, once a week throughout your project, to test different kinds of prototypes of various components of your solution with friends, family, colleagues, target users, and any other random people who will volunteer to help.^[3]

Recruiting Testers

User testing involves selecting representative users from a segment of the population, and learning more about them, so that we can build better products and services for everyone, explains Dana Patton. Sounds good.

However depending on your timeframe or budget, and as part of the continuous cycle of testing your ideas frequently with as many people as possible, you may also find it useful to conduct some impromptu prototype testing with people you know. This kind of convenience sampling is an effective preliminary idea validation technique and it requires minimal overhead. By running a quick usability test with 3-5 people you’re acquainted with, and watching them try to get their heads around and navigate your prototype, you can learn a lot. Fast, easy, and inexpensive user research sessions involving people close to you can produce valuable informal feedback on your design project-in-progress.

Recruiting Online

Recruiting testers online works wonders for design research. In order to successfully recruit online testers who genuinely represent your future target users, you will first need to decide who exactly you most need to speak to. To this end, preparing user personas is helpful, as they will aid you to clearly define what kind of users you most need to understand, satisfy, and hear from.

One strategy to find relevant test users is to follow Amanda Stockwell’s advice and invest a small amount of upfront time to create a go-to panel of research participants for your design project. “Building a panel means you’ll always have a list of people who have expressed interest in being part of research,” Stockwell writes.^[4] When it comes time to run a test, you can just send them an email invitation.

Another way to recruit testers is to leverage social media communications channels. Select your platforms, post a brief intro to your study and a link to your survey or test event invitation. To encourage social media users to complete your survey, decide on a particular and very specific problem you want to solve, and ask simple things, such as “Which word do you like better?” or “Which icon do you like better?” Sarah Kahn suggests.^[5]

Using social media for quick and simple prototype tests can deliver feedback on your idea within minutes －Kahn calls this approach “a quick win.”  The fact is, Kahn reminds us, at many points in the innovation process quick, straightforward, brief, honest feedback is all that’s needed to move a design project forward.

When deciding between social networking platforms, consider the demographics of users on each channel. Some will skew young, or female, or professional－ so you need to know who you are looking for before developing your social recruiting strategy.

In-Person Testing

To get the most out of an in-person test session, here are some general things to consider.

Clarify your goals.

What do you want to learn? Identify your questions, concerns, areas of interest, and the purpose of the research.

Define the scope of the session.

Be very specific about exactly what you are testing. Rather than trying to test everything at once, plan out which elements of the product or service each round of testing will cover.

Write your opening script.

Gale Yang shares her opening script, which you may find helpful to modify for your testing:

“You’ve been invited to help us understand which parts of the product work for you and which are confusing. The [product/service] we’re testing is [insert 1-sentence description here]. Even though we’re calling this activity a test, you’re not being tested. The product is. There’s nothing you can do wrong. It’s not your fault if you can’t get something to work, and you won’t hurt anyone’s feelings if you say something bad about the product. It’s really important that you speak all of your thoughts aloud.” ^[6]

Decide on your questions or tasks. 

In usability testing, researchers ask users to complete activities (tasks) while using the product, service, or interface. For example: when testing a reservation app for a restaurant, you might task the user with “booking for lunch tomorrow at noon for a party of six.”

Consider logistics (location, date, duration).

When picking a location, look for a setting with low or no distractions. Spaces such as the shopping mall food court, or a library common area might be too loud and hectic at certain times of the day. If at all possible, do your testing where people live or work, so you can observe them engaging with your product in their natural environment, advises Steven Hoober.^[7]

Decide on how many sessions you will conduct on testing day and exactly how long each will last. Remember that test participants may arrive late (especially if you neglect to send a reminder notification with directions to the test location) so leave a buffer between sessions.

Assemble equipment.

Do you need a computer or internet connection? Do you plan to do video recording, and in that case do you need to think about lighting and a microphone? If you’re doing the test in the user’s environment, be certain that your prototype will work in the field. For example, for a physical model, it has to be portable.

Decide on metrics.

What specifically are you measuring? What will a successful outcome look like? Consider tracking completion rates, time on task, frequency of errors (critical and noncritical), total time spent confused－and subjective data such as user satisfaction, likes, dislikes, perceived ease of use and perceived success.

Focus on active listening.

When planning in-person tests, it’s important to keep your questions simple and straightforward, and listen more than you speak. Don’t share too much information about your offering, Steven Hoober cautions, but instead be patient and let participants find their own way. You want to speak as little as possible and let them do the talking. By listening three times more than you speak, you’ll be less likely to resort to a confirmation bias－the tendency to hear and focus only on the feedback that confirms what you already think about the project.

Graciously receive feedback.

Prepare yourself to listen, and be willing to pivot or otherwise adapt in response to the feedback you hear. The trick is to get very good at listening to your testers, try not to explain, to not get defensive or feel defeated if they don’t “get it” and to instead let the people you’re designing for direct you to a preferred solution. Sometimes you can’t predict what users will say or do. Have the flexibility to change course if you receive new information or something goes completely wrong during your test sessions. Remember, “the purpose of user testing with prototypes is not to impress users, but to learn from them,” advises Laura Busche.^[8] She continues, ”instead of wowing people with our product, the goal of testing with low-fidelity prototyping is to have users wow us with their insight.”

Testing via Secondary Research

But what about doing user research when you can’t directly interact with real people? “Ideally we’d always be doing our research in-person, in moderated sessions running interviews and tests, until we’d gotten everything just right,” admits Chris Myhill.^[9]  

But sometimes you don’t have the budget or the timeline to run user live tests, or conduct interviews — and in those cases there are other ways to gather user insights, explains Jon Peterson.^[10]

When you don’t have the option of first-hand contact with the people you’re designing for, you might start by going online to find products or services similar to yours, and look for the following:

• FAQ pages
• Blog comments
• App reviews
• Forum discussions
• Testimonials
• Social media comments and questions
• Amazon or Yelp reviews and ratings

This kind of online, secondary research is valuable and far better than not doing any user research at all, Chris Myhill and Jon Peterson agree. So even if your users are not directly accessible to you, do some investigations online to learn all you can about them, what they expect from, and how they engage with offerings similar to yours.

Final Thoughts

To sum up, in this chapter we’ve covered why user testing is a critical part of the design thinking method. User testing is an exercise in empathic design. Sometimes you can’t fully understand other people’s problems, pain points, and perspectives until you put a prototyped solution in front of them and observe their honest responses.

When you build a model of what you think your solution should look and feel like, and share it with others, you are learning by making, and you will likely end up with suggestions for how to improve, simplify, or pivot your solution.

At the end of the day, to quote researchers at the Nielsen Norman Group: “a product’s success depends on how users perceive it.”^[11] This means it’s essential that design thinkers subject ideas and models to thorough and varied testing, so we can better understand what users see, think, feel, and need to do, as we strive to create innovations that they will want, need, and love to use.

Attributions: material from the following open source texts was adapted and integrated into this chapter

“Design Thinking for Educators” IDEO. Circa 2013. CC BY-NC-SA 3.0 https://designthinkingforeducators.com/toolkit/

“Collective Action Toolkit” Frog Design. Circa 2016. CC-BY-NC-SA 4.0 https://www.frogdesign.com/wp-content/uploads/2016/03/CAT_2.0_English.pdf

“Design Thinking for 11th Graders” Bridget McGraw. Circa 2016. CC BY-NC-SA 3.0 http://www.open.edu/openlearn/science-maths-technology/engineering-andtechnology/design-and-innovation/design/design-thinking/content-section-0