Usability Testing: Moderated vs. Unmoderated
Usability research methods are often divided into two categories: moderated and unmoderated. There are pros and cons for each. Identifying key differences and situations when to use them will help in choosing the right approach for your project.
Have you been approached by your design team with a request to do usability testing? Maybe they are asking for an unmoderated, remote usability test to evaluate the end-to-end experience of a major redesign. Or they are asking for a moderated, in-person test that costs $30,000 and takes 6 weeks just to evaluate the effectiveness of a new icon. How do you know what usability is the most effective for the project? Is the method being proposed the most efficient use of your project’s limited time and budget, and still providing valid data that can be used to make design decisions?
Usability testing is a qualitative research method used in mobile app, website, and software development to evaluate if real users are able to use a design as intended. Usability answers the questions of why people do things, and where they have problems. Doing usability can allow your team to identify and fix problems with a design before it’s launched.
Moderated Usability Testing
Moderated usability testing is conducted with users in person. Typically, a trained facilitator sits down with an actual user and has them do authentic tasks using the design. It gathers the richest qualitative information about users and their experience. Started early in the design process, usability provides the design team with data about how users actually use the design and think about the product. It shows the design team what works and does not work from the user’s perspective, and allows the design team to make changes that lead to an intuitive, usable interface.
Steve Schang, a Usability Engineer and UX Designer with 18 years of experience, said in-person, moderated testing takes more time, but often is more appropriate with complex prototypes, in situations where you need rich qualitative data, or have concerns about protecting intellectual property.
Early in the design process, Steve recommends multiple rounds of “lo-fi” moderated testing, which involves fewer participants, shorter sessions, and informal reports. With a one-page summary of key findings, designers can get back to work quickly on the next iteration. Testing like this also has an additional advantage of being quick to set up and run – about a week. It is ideally suited to design processes like Agile and Google Design Sprints.
“Doing multiple rounds of testing lets you learn from your users, make design changes, and then validate those changes. At the end of the day, this user-centered approach results in a more intuitive and usable product.” he said.
A more formal, in-person moderated usability study with more participants and longer sessions is often helpful to evaluate final designs, test end-to-end experiences, or test cross-channel experiences (i.e. phone-to-web). This type of testing is more time consuming and includes a formal report to document the findings.
Unmoderated Usability Testing
Unmoderated testing, where users complete tasks on a prototype without a facilitator present, can also be valuable. These tests use web-based or app-based software to track user activity, and in some cases, record audio to capture verbal reactions and feedback. This method is less effective when an in-depth understanding of the user’s behavior is needed because the facilitator is not there to investigate unexpected activity or ask follow-up questions.
Steve said remote, unmoderated tests are great when you need quick answers to very focused questions, when you need a larger sample of participants, or when you need to test across a large geographic area.
“It can be done at any point in the design process,” he said. “I like to use it as an ad hoc testing tool. As questions come up and need quick answers, you can set up, recruit, field, and report out in a day or two, using an online tool like usertesting.com. If I tried to do this by bringing users into the lab, it would take a minimum of 7 days just to recruit the participants to come to the lab.”
While moderated sessions can last up to 90 minutes, unmoderated sessions are more effective if they are 10-15 minutes, Steve said. All tasks and questions must be carefully written into an unmoderated session. There’s no room for improvisation if the user does something unexpected or the prototype breaks. And technical requirements are greater for unmoderated tests, requiring a reliable prototype that’s publicly accessible on the Internet. These remote tests sometimes run into confidentiality and non-disclosure issues, unlike in-person moderated testing, where you retain full control of the prototype.
Moderated usability allows testing of complex designs, provides more flexibility during the sessions, and gathers much richer qualitative feedback:
Yield a rich understanding of users.
Explore unexpected user activity.
Allows facilitator to ask follow up questions.
Retain full control of prototypes.
Allow for longer sessions, up to 90 minutes.
Unmoderated, remote testing is good for very focused research questions, especially when testing users across a larger geographic area:
Ideal to answer short, focused research questions
Quick turnaround recruiting and testing.
Recruit participants from a wider geographic area.
Require reliable prototypes that can be accessed publicly from the Internet
Shorter sessions, around 10-15 minutes.
If you have questions or would like more information, contact the experienced usability professionals at midwoodusability.com.