Beyond the Usability Lab: Conducting Large-scale Online User Experience Studies

by William Albert, Thomas Tullis, Donna Tedesco

Overview

Usability testing and user experience research typically take place in a controlled lab with small groups. While this type of testing is essential to user experience design, more companies are also looking to test large sample sizes to be able compare data according to specific user populations and see how their experiences differ across user groups. But few usability professionals have experience in setting up these studies, analyzing the data, and presenting it in effective ways.  Online usability testing offers the solution by allowing testers to elicit feedback simultaneously from 1,000s of users. Beyond the Usability Lab offers tried and tested methodologies for conducting online usability studies. It gives practitioners the guidance they need to collect a wealth of data through cost-effective, efficient, and reliable practices. The reader will develop a solid understanding of the capabilities of online usability testing, when it’s appropriate to use and not use, and will learn about the various types of online usability testing techniques.

*The first guide for conducting large-scale user experience research using the internet *Presents how-to conduct online tests with 1000s of participants – from start to finish *Outlines essential tips for online studies to ensure cost-efficient and reliable results

Product Details

ISBN-13:	9780080953854
Publisher:	Elsevier Science
Publication date:	12/21/2009
Sold by:	Barnes & Noble
Format:	eBook
Pages:	328
File size:	6 MB

About the Author

Bill Albert is Director of the Design and Usability Center at Bentley University. Prior to joining Bentley, Bill was Director of User Experience at Fidelity Investments, Senior User Interface Researcher at Lycos, and Post-Doctoral Research Scientist at Nissan Cambridge Basic Research. Bill is an Adjunct Professor in Human Factors in Information Design at Bentley University and a frequent instructor at the International Usability Professional’s Association Annual Conference. Bill has published and presented his research at more than thirty national and international conferences. He is coauthor (with Tom Tullis) of Measuring the User Experience and Beyond the Usability Lab. He is on the editorial board for the Journal of Usability Studies.
Tom Tullis is Vice President of Usability and User Insight at Fidelity Investments and Adjunct Professor at Bentley University in the Human Factors in Information Design program. He joined Fidelity in 1993 and was instrumental in the development of the company’s usability department, including a state-of-the-art Usability Lab. Prior to joining Fidelity, he held positions at Canon Information Systems, McDonnell Douglas, Unisys Corporation, and Bell Laboratories. He and Fidelity’s usability team have been featured in a number of publications, including Newsweek , Business 2.0 , Money , The Boston Globe , The Wall Street Journal , and The New York Times.
Donna Tedesco is a Senior User Experience Specialist with over ten years of user research experience. She has published and presented at local, national and international conferences, and is co-author with Bill Albert and Tom Tullis of the book, "Beyond the Usability Lab: Conducting Large-Scale Online User Experience Studies." Donna received a BS in Engineering Psychology/Human Factors from Tufts University School of Engineering and a MS in Human Factors in Information Design from Bentley University.

Read an Excerpt

Beyond the Usability Lab

Conducting Large-scale Online User Experience Studies

---

By Bill Albert Tom Tullis Donna Tedesco

MORGAN KAUFMANN PUBLISHERS

Copyright © 2010 Elsevier Company
All right reserved.
ISBN: 978-0-08-095385-4

---

Chapter One

Introduction

What do we mean by "beyond the usability lab" and "large-scale online user experience studies?" The usability lab has been, and continues to be, a great resource for learning about the usability of some product based on a relatively small number of users. But Web technology now enables us to move beyond the lab and efficiently conduct user experience studies with a much larger sample of users—hundreds or even thousands. That's where the large scale comes in. Traditional lab studies with small numbers of users will usually let you detect the most obvious usability issues. Getting at the less obvious, but still significant issues, comparing multiple designs, or even getting at a user's likelihood to buy your product, takes a new method better suited to collecting data from many more users. Our shorthand name for this new method is online usability or user experience studies.

Until fairly recently, user researchers may have had to conduct multiple studies—often expensive and time-consuming—to understand the complete user experience. This may have resulted in not conducting the necessary user research, leading to bad decisions or missed opportunities, or adding significant time and costs to every project. Fortunately, in the last few years, online usability testing has emerged to meet this critical need. Through online usability testing you will be able to gain valuable insight into the entire user experience. You'll be able to understand who the users are, their intentions, actual behavior, emotional reactions, and their drivers of future behavior.

Through online usability studies you will be able to gain valuable insight into the user experience, such as:

* Is there a significant difference between two (or more) designs?

* What visual design characteristics do users prefer?

* How significant are the usability issues?

* Where are users likely to abandon a transaction?

* Is there a difference in how various user groups will use and react to the design?

Going beyond the lab also means tapping into users well beyond the geographic constraints of your lab. Online usability studies allow you to reach many users all over the world, without worrying about time zones and languages.

We're writing this book to show you the value of online usability studies and how to use them effectively. Over the years we've learned through trial and error. We want to help you avoid the pitfalls. We'll show you when to use (and not use) this method and how it complements other types of user and market research methodologies. We'll help you design and build your own online usability study or choose the best out-of-the-box solution. This book will expand your user research toolkit. We will also draw upon many examples throughout the book to illustrate key points. We hope you find this research method as useful as we do.

1.1 WHAT IS AN ONLINE USABILITY STUDY?

Before we go any further, we should give you some idea of how an online usability study works from the user's perspective. Jane receives an email invitation to participate in a study, similar to many invitations to online surveys she may have received in the past. She may be intrigued by the topic, or simply enticed by the incentive she will receive, so she clicks on the link in the email to take part in the study. When she begins the study, she is shown instructions about how to complete the study and is alerted to any technical requirements. Initially, Jane is asked a few questions that ensure she is eligible to participate, thus allowing the researcher to maintain an appropriate mix of participants. She is then asked a series of background questions about her experience in using different Web sites and perhaps some questions to help narrow down the set of tasks she will perform.

Next, Jane is asked to perform a series of tasks to evaluate a new area of a Web site. For example, she may be asked to find a breast cancer screening facility near her home. While Jane is interacting with the Web site, she is providing feedback about her experience and answering questions through a small browser window or toolbar that sits above the main browser window (see Figure 1.1). Jane's behavior on the site is being captured, such as the time it takes her to complete each task, the pages she is visiting, data she has entered, and even her mouse clicks. Once she is finished interacting with the Web site, she is asked a series of questions about her overall experience with the Web site, concluding with a few demographic questions. The final page Jane sees is a confirmation that she completed the study successfully and information about how she will receive her incentive.

As you can see, online usability studies are fairly straightforward from the participant's perspective. Depending on the tool you use, the participant may be required to download a plug-in or custom control so that clickstream data can be captured. Online usability studies are compatible with most browsers and operating systems. Participants can take part in the study anywhere they have an Internet connection. The study typically runs from 5 to 45 minutes and most involve some small incentive. The ease of participating in these studies is certainly a driving force in their popularity, from the perspective of both the researcher and the participant.

One question we often get asked is what's the difference between online usability studies and more traditional online surveys, such as the ones that pop up in annoying ways while surfing the Web? After all, we recruit participants in the same way, both techniques deal with large sample sizes, and they sometimes include some of the same questions. The key difference between online usability studies and more general online surveys revolves around active behavior. This means that the participant is interacting with a Web site, Web application, mobile device, or any product in some way. This interaction might be in the form of a series of tasks or simply exploring a site and giving feedback. The participant is not just looking at something, but performing some action, such as clicking on buttons, entering text, or making some type of decision. This is clearly different from most online surveys where the participant is reflecting on some past experience or conveying attitudes that have developed over time. In many market research surveys, the participant may be asked to recall some past behavior, such as whether they accomplished their last task the last time they visited the Web site or what they typically do on the Web site. While this is certainly important information to know, we don't believe this falls within the scope of online usability studies.

1.2 STRENGTHS AND LIMITATIONS OF ONLINE USABILITY TESTING

So, what are the strengths and limitations of online usability studies? What can this method do for you that other research methods can't? How can online usability studies be used to improve the user experience? These are all fair questions. But to be honest, it is not so much about the unique strengths or advantages of online usability studies, but rather the combination of advantages that is at the heart of this method. Every advantage of online usability studies can be achieved through some other user research method. However, online usability studies bring together a diverse set of strengths to make it a highly versatile user research tool.

Before fully appreciating the strengths and limitations of online usability studies, it's important to consider the types of data that are collected. After all, the foundation of online usability studies lies in the data it generates. Unlike some user research methods that specialize in collecting only one type of data about the user experience, online usability studies have the potential to tap into four distinct types of data at once (see Figure 1.2). Online usability studies are well suited for collecting both qualitative and quantitative data. Because data are collected from many participants interacting with a Web site simultaneously, it's relatively easy to gather quantitative data from a large sample size in a short time period. It's common to collect quantitative data on task success, completion times, click paths, satisfaction, and user demographics, to name but a few. Online usability studies also have the advantage of collecting qualitative data about the user experience. Participants are often encouraged, and sometimes required, to provide verbatim comments about their experience. For example, a participant might be asked to explain why a task was difficult to complete or to provide specific suggestions for improvement. Unlike most online surveys that also let study participants enter verbatim comments, participants in an online usability study are providing qualitative insights during their actual experience, or reflecting on an experience that just took place.

Online usability studies are ideal for collecting data about attitudes and behavior. This technique provides insight into not only how users think and feel about their experience, but what they actually did. For example, we might learn what they like and don't like, their overall satisfaction, what colors they prefer, their likelihood to use the Web site in the future, and other drivers of their future behavior. We can also learn about their actual behavior on the site. We can see where they went to accomplish each task (through click-stream analysis), whether they were successful in completing a task, how efficient they were in their tasks, and even what information they entered at different points along the way. You can think of virtually standing over the shoulder of a thousand participants, watching their every move, and tapping into their thoughts at the same time.

The following sections describe some distinct advantages and limitations of online usability studies or situations where they're particularly useful.

1.2.1 Comparing designs

Instead of flipping a coin to make an important design decision, we recommend letting the users decide, and preferably a lot of them. The best way to do this is through an online usability study. One of the common ways of using online usability testing is to collect data on more than one design alternative. Perhaps you think that tab-based navigation is more effective, whereas your business partner is convinced that left-side navigation links are the most effective. So, if you are able to build out two prototypes, it's easy to evaluate both design alternatives in an online usability study. It is very easy to fall into the trap of assuming you represent your users. In our experiences, we have been wrong on more than one occasion. It is hard to argue with hundreds or even thousands of users who are sending you a very clear message through data about design alternatives. We have found that online studies can enlighten us on the unanticipated implications of some of the most subtle design differences, such as a simple link label change. It's one key reason why online studies have become a regular part of our work and why we want to share these benefits with you. In some ways, online usability studies can be thought of as "a/b testing" before the product has launched.

1.2.2 measuring the user experience

Online usability studies are probably the most efficient way to collect detailed and scalable usability metrics. For example, you might be interested in identifying reliable patterns in user preferences, detecting performance differences in subtle design options, or reliably measuring the impact of task performance on satisfaction. Traditional lab usability testing is not well suited for these types of investigations because a large sample size is required to obtain reliable metrics. Achieving a larger sample size in traditional lab testing requires a significant amount of time (up to a few weeks) in the lab, adding up to significant costs and possibly project delays. An online usability study allows you to reach hundreds or even thousands of people in a short amount of time. Once the study has been created, you can usually obtain whatever sample size is needed with little or no extra effort for data collection. Of course, the cost may increase depending on your incentive, but data collection and analysis time may increase only marginally.

1.2.3 Finding the right participants

Recruiting participants who meet specific criteria can be a challenge in any user research study, especially if they are dispersed geographically. What do you do if you need to find 100 people who own ferrets, shop online for ferret food, and have never used your Web site? Online usability studies provide a solution by being able to recruit literally anyone who has access to the Internet. You don't need to worry about only finding people in your city that can come into the lab. Anyone can participate in the study when they are at home, on their lunch break at work, or even using a mobile device. You can also take advantage of very large user panels, composed of thousands of Web users throughout the world. Casting a large net for participants also means you can be picky about who participates in the study. Through the use of carefully selected screener questions and quotas, you'll be able to find the right type of participants and achieve the desired mix of demographics. We'll discuss strategies for writing effective screener questions in Chapter 3.

Online usability studies also work well when you need to evaluate your product in many countries. Running moderated studies with participants who are not fluent in the language(s) you speak may be a challenge. Fortunately, some vendor solutions allow you to translate your studies into different languages. At the very least, allowing the participant to read the questions at their own pace should help tremendously, even if it is not in their first language.

1.2.4 Focusing design improvements

An online usability study is one of the most effective ways to help product teams focus on the areas of a Web site that need the most improvement. Online usability studies help identify and prioritize areas where users are having the most difficulty in accomplishing their tasks. A quick glance at task success rates and completion times will often reveal which aspects of the design require attention and their relative priority. When a Web site is evaluated in a traditional usability lab study, you can easily identify usability issues and the underlying reasons why participants did not do well on certain tasks. However, you often don't know the magnitude of the issue. If two out of six participants experienced a particular problem, how does that translate into the larger population? Without the benefit of a much larger sample size it is problematic (at best) to predict how many will experience that issue once the design is launched. Online usability testing allows you to make much more accurate predictions of the impact of specific usability issues and helps focus your design resources.

It's easy for some to think of online usability studies as only useful in collecting a bunch of metrics. That's not true. This technique online allows you to collect a tremendous amount of qualitative feedback about the participant's experience. One type of qualitative data that we have found especially useful is verbatim comments about why certain tasks are difficult. For example, if a user gives a low score on ease of use for a specific task or abandons the task, they can be asked to provide their reasons for the low score or abandonment and possibly suggestions for improvement. Examining the verbatim comments provides guidance about the nature of the problem and potential design solutions. It may be helpful to categorize the suggestions for improvement to see what might work for the largest number of users. To be fair, the richness of qualitative feedback from online usability studies pales in comparison to more traditional usability testing where there is a direct interaction between the moderator and the participant.

1.2.5 Insight into users' real experience

In a typical lab usability study you're generally asking participants to perform a set of tasks. While this is certainly useful in identifying usability issues, it often falls short in understanding what the user's real experience might be like. For example, there might be some characteristics about a user's technical setup such as the size of their monitor, the browser they use, screen resolution, or system performance that all impact their experience. When they come into a lab setting, all that useful information is thrown out the window. Online usability studies collect data in the user's natural environment, warts and all. So, if the participant is using a lower screen resolution and the design you are testing has key information lower on the page, you're likely to see longer completion times and perhaps even lower success rates.

(Continues...)

---

Excerpted from Beyond the Usability Lab by Bill Albert Tom Tullis Donna Tedesco Copyright © 2010 by Elsevier Company. Excerpted by permission of MORGAN KAUFMANN PUBLISHERS. All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.

---

Table of Contents

1. Introduction to online usability methods
a. What is online usability, and how it differs from traditional usability methods
b. Examples of different types of online usability studies
c. Pros and cons of online and non-online methods
d. When to use (and not use) online methods
e. Combining online studies with lab testing

Chapter 1 provides an overview to online usability testing. Special attention will be paid to how it differs from traditional usability methods (including remote testing). There will be an in-depth discussion of the pros and cons of online testing, and when to use and not use online methods. We will provide real-world examples to highlight the value of this method. We will also discuss ways to complement traditional usability testing with online testing. Our intention is that the reader will be in the position to determine if an online usability study is appropriate for their organization.

2. Planning your study
a. Study goals
b. Budgets and timeline
c. Technology options
d. Participant recruiting and panels
e. Sample size
f. Panel options
g. Sampling strategy
h. Study duration
i. Participant compensation

Chapter 2 focuses on all the activities and decisions that need to be made place prior to actually putting the survey together. The first three activities (goals, budgets/timelines, and technology options) are all essential to accurately scope an online study. The next part of this chapter focuses on finding the right number of targeted participants. This includes a discussion of research panels, sample size determination, and sampling strategies. The chapter will conclude with a discussion of estimating study duration and participant compensation.

3. Designing your study
a. Introducing the survey
b. Screener questions
c. Starter questions
d. Constructing Tasks
e. Post-task questions and metrics
f. Post-session questions and metrics
g. Branching
h. Progress indicators and navigation
i. Speed traps
j. Question types

Chapter 3 is devoted to developing the study design. The first half of the chapter (topics a through g) are the various sections that are typically included in an online usability study. For each section, we will review best practices and common pitfalls. We want to give the reader the confidence for putting together an effective online study. The last part of this chapter (topics h through k) deal with common techniques that are used in various parts of a study. They include topics such as branching, navigation, speed traps, and question types.

4. Launching your study
a. Piloting and Validating
b. Timing the launch
c. Phased launches
d. Monitoring results

Chapter 4 deals with issues around launching an online study. This includes all the activities that happen after a study has been developed until the final data are available. This chapter discusses how to set up a pilot test and validate the study, timing a launch to maximize participation and quality results, and phased launches. The chapter concludes with a discussion on how to monitor results. This includes both participation rates as well as data quality.

5. Data preparation
a. Fraudulent participants
b. Consistency checks
c. Data reliability
d. Outliers
e. Recoding variables

Chapter 5 will help the reader prepare their data for the analysis stage. There some very important activities that need to take place prior to data analysis that must be done to ensure valid results. Topics in this chapter will include how to identify fraudulent participants, running consistency checks on the participant responses, and identifying outliers in the data that may need to be removed from the analysis. The chapter will conclude with a brief discussion of how to recode variables that will be most useful in the analysis stage.

6. Data analysis and presentation
a. Verbatim responses
b. Task-based metrics
c. Segmentation analysis
d. Post-session analysis
e. Behavioral data
f. Combining data
g. Identifying usability issues
h. Presentation tips

Chapter 6 covers all the information the reader will need to know about how to analyze and present data derived from an online study. Each section of this chapter covers one type of data that are typically captured in an online study. Verbatim analysis focuses on how to derive meaningful and reliable findings from open-ended responses. Task-based metrics include success, completion times, and ease of use ratings. Segmentation analysis includes ways to identify how distinct groups performed and reacted differently. Post session analysis involves looking at metrics such as SUS scores, overall satisfaction and expectations, and ease of use ratings. Behavioral data analysis includes metrics such as clicks paths, page views, and time spent on each page. Combining data from more than one metric is a very important step in analysis. Methods for identifying usability issues from all the data will be described and examples given. This chapter will be very practically oriented, giving step-by-step direction on how to perform each type of analysis. Many examples will demonstrate different ways to present the results.

7. Building your own online study
a. Approaches to creating your own online study
b. Presenting tasks and prototypes
c. Capturing task completion status
d. Capturing task time data
e. Capturing self-reported data
f. Examples

Chapter 7 shows readers how to create relatively simple online studies themselves. Approaches to presenting tasks and prototypes will be described, as will techniques for collecting task success, times, and various kinds of self-reported data, including rating scales, open-ended questions, and the System Usability Scale (SUS). While some examples of HTML and JavaScript will be shown, we will describe them in such a way that even someone new to those technologies could understand and use them. Complete examples will be shown that readers could easily adapt. Code samples will also be provided on a companion website.

8. Online solutions
a. Keynote
b. RelevantView
c. User Zoom
d. MindCanvas
e. Survey Monkey
f. Opinion Lab
g. ACSI
h. Others

Chapter 8 reviews the common online tools that can be used for running online testing. While the ?Do-It-Yourself? reader may want to use the techniques described in Chapter 7, others may want to use a commercial tool like those described in this chapter. Most of the chapter will be devoted to those tools that used most often to collect behavioral data such as Keynote, Relevant View, and User Zoom. There will also be a discussion of online tools that do not collect performance data such as Survey Monkey, ACSI, and Opinion Lab. Comparisons of the tools, including what kinds of data can be collected with each, will be included. The chapter will conclude with a brief discussion of other possible solutions such as agencies that specialize in online testing. Readers will also be referred to our companion website to keep up with updates and emerging software solutions.

9. Ten tips for a successful online study
a. Planning for metrics
b. Deciding on the right tool
c. Choosing the right participants
d. Writing clear tasks
e. Piloting your study
f. Checking data
g. Comparing to other data sources
h. Being creative with the data
i. Allow enough time for analysis
j. Presenting only the top line results

Chapter 9 provides a summary of some of the key points made throughout the book. This summary will be in the form of the top ten tips that someone should know when conducting their own online study. These tips will be very practical in nature.

Editorial Reviews

"If you need to reach a large audience using online tools, this book will guide you, inform you, and help you with practical tips from a team that has a wealth of practical and academic experience. They explain how to plan, organise, conduct, analyse, and present your study, and they aren’t afraid to tackle tricky topics like dealing with possibly fraudulent participants. Highly recommended."— Caroline Jarrett, Effortmark, Ltd, author of Forms that Work

"Beyond the Usability Lab will become the gold standard for how to conduct online usability studies. Albert, Tedesco and Tullis masterfully combine strategic design related issues with tactical and practical step-by-step instructions on how to implement an online usability study." — Jim Berling, Managing Director, Burke Institute

From the Publisher