Remote Research. Tony Tulathimutte

skeptical? Then you should check out our exhaustively documented study for Wikipedia, complete with full-session videos and highlight clips at http://usability.wikimedia.org/wiki/Usability_and_Experience_Study. It includes both lab and remote sessions with identical goals, so it’s a good comparative case study.

      The best way to see if a remote study is for you, however, is by getting your feet wet with a quick, painless pilot study, which we’ll walk you through in Chapter 2, “Doing a Pilot Test Right Now”.

      I want to cut costs for my user research study. Is using remote methods a good way to do that?

      Not really. You might save on costs related to travel expenses, renting a lab facility, or hiring a recruiting agency, but then there are the expenses of the specialized remote research tools and services you’ll need, and where the researcher’s time, participant incentives, and project timeline are concerned, nothing is much different. We cover the equipment requirements of a basic moderated study in Chapter 2, “Gearing Up: Physical Equipment”, and the costs of many remote tools and services in Chapter 8.

      Where can I get people to participate in my remote study?

      You can technically use any methods to recruit for a remote study that you’d use for an in-person study: email contacts, recruiting agencies, and even craigslist ads (blegh!) are still an option. However, in this book, we introduce a method called live recruiting, with which you intercept visitors to your Web site by using a pop-up form to get them to participate in your study right away. We strongly believe this approach is ideal for remote research because it allows you to do what we call “Time-Aware Research,” which we introduce in Chapter 1 and discuss in depth in Chapter 3.

      How on earth can you call this a book about research without way more academic references and doctoral degrees?

      This book is aimed at people who want to hear practical information about how remote research is done in the real world.

      We don’t claim absolute peer-reviewed scientific rigor, but we are sharing what’s given us the best results after almost a decade of remote research experience: 2,615 moderated users, 2,676 automated users, 234 projects, 19,120 project hours, and 89 clients to date. Simply put, this is just what works for us, and we think it will work for you, too. (And for what it’s worth, Nate and Tony both have cognitive sciences/human-computer interaction degrees: yes, we roll that deep.)

      Foreword

      Peter Merholz is a founding partner and president of Adaptive Path, an experience strategy and design firm

      (www.adaptivepath.com). He coauthored Subject To Change: Creating Great Products and Services for an Uncertain World, published by O’Reilly.

      You are holding in your hand a portal into the future. The approaches discussed in this book are the first steps toward a user research methodology suited to our technological context. In comparison, lab research will be increasingly seen as an archaic approach to understanding people, akin to phrenology or trepanning.

      You see, lab usability engineering was born of a simpler time. Files were stored on floppy disks, and the computer wasn’t connected to a network. People used computers for Calculation (basic math, spreadsheets, etc.), Creation (word processors, graphics programs), and Capture (data entry). Computers would have only one “program” running at a time. And someone could be expected to focus on a single task at hand for many minutes, if not hours on end.

      These days, our customers’ technological world is much more complex. The bulk of their time online is spent engaged in Consumption (browsing the Web, listening to music) and Communication (email, instant messaging, Twitter), though they still Calculate, Create, and Capture. They have multiple applications open and multiple windows within those applications. Thousands of files and emails fill their hard drives, and they’re managing multiple devices, including computers, mobile phones, iPods, and digital cameras. Technologies have driven users to a point of extreme distraction—recent research has shown that workers are interrupted an average of every 11 minutes.

      Although the world has changed, the methods of standard usability practice are essentially the same as were practiced in the early 1990s. We recruit participants who satisfy demographic requirements, invite them into a fluorescent-lit lab with an imposing mirror along one wall, ask them to use a computer they’ve never seen before, have them engage in a set of scripted tasks, and “think aloud” while doing so. These participants won’t have an IM window pop open unannounced, nor be able to click on the bookmark bar to see their Facebook page, nor be able to pull up the notes they saved in a draft in their email. When everything about the observation environment is so unnatural, how can we expect our findings to provide legitimate insights?

      This is where the genius of Nate Bolt and his crew come in. Since I’ve known them, they’ve experimented with technologies in an effort to capture the most authentic user research data. Whether it was the pioneering screen sharing and recording tool Ethnio (which fundamentally changed how I considered usability), or real-time uploading of video captured in a drive-along study for Volkswagen, their mission is to deliver the unvarnished truth.

      Now, it’s easy to get caught up in the tools. While remote research is enabled by nifty technologies, which should be of secondary interest at best, I know that for Nate and Tony, this research approach is about mindset. When you simply desire to understand people in the most direct ways, you come up with clever means of doing so. And sometimes, you can’t go “remote.” When I first met Nate, he crowed about how they kitted out a usability lab with hidden microphones and cameras so that the participant would be as comfortable as possible. Or when researching the use of multiplayer video games, he recreated a cozy living room set up in his office, with snacks and lounge chairs, to make gamers feel at home.

      And that’s how this book is a time machine. Only the Flying Spaghetti Monster knows what the future will bring. The only thing that’s certain is that it’s going to keep changing. To prepare for that future, you need the mindset represented in this book, in order to figure out which approaches will provide the best user research data given your circumstances. Like Bolt | Peters, don’t settle for standard practices; instead, play with new ideas that will deliver fresh insights.

      —Peter Merholz

      Chapter 1

      Why Remote Research?

       The Appeal of Lab Research

       Is Lab Research Dead?

       What’s Remote Research GoodFor?

       When to Go Remote

       Moderated vs. Automated

       When to Use Which Remote Method

       Chapter Summary

      In-person lab research used to be the only game in town, and as with most industry practices, its procedures were developed, refined, and standardized, and then became entrenched in the corporate