Remote Research. Tony Tulathimutte

the users to the eye-tracking device. Other studies might require you to attend to the participants’ physical movements, which may be difficult to capture with a stationary webcam. And then there are multiuser testing sessions, in which a single research moderator facilitates many participants at once; screen sharing is currently not well suited to sharing multiple desktops at once, though some tools (e.g., GoToMeeting) make it relatively painless to switch from one desktop to another. We want to emphasize, however, that for most studies, seeing the user at all is not actually important; we explain why in Chapters 5, “Moderating,” (see “Ain’t Nothing Wrong with Using the Phone”) and 10, “The Challenges of Remote Testing.”

      Although these situations are all compelling reasons to conduct in-person research, part of what we want to demonstrate in this book is that remote research is very broad and adaptable, and even if a study is conducted in a lab, elements of remote methods can be adapted and incorporated to enhance in-person research methods. We’ll get to that in Chapter9, “New Approaches to User Research.”

      Why I Went Remote

      by Brian Beaver

      Old habits die hard, but for any number of reasons—cost, convenience, international testing—a handful of former lab researchers have switched to remote methods and never looked back. Brian Beaver, award-winning creative director at Sony, explains why he decided to go with remote methods after many years of lab testing.

      On Going Remote

      I have quite a bit of experience, either organizing user research sessions or participating in them, especially at Sony. My research has been pretty varied and the outcomes are always interesting, but I’m a big fan of remote usability testing. It seems to give me the best bang for my buck.

      I’d read about it a few years back and had done some work with Adaptive Path when it had a focus on usability. Adaptive Path referred me to Nate [Bolt, coauthor of this book], and when he shared his remote approach with me, I knew we were in sync because a lot of the pain points and skeptical raised eyebrows around results we’d obtained in previous lab testing instantly diminished with remote usability testing.

      The pain points always involved recruiting. With a Web site you have such a diverse geographic base that it can be challenging to bring a core group of your users together in one location. Sony tends to be very protective of its customer information, and wouldn’t share it with a research company for the purposes of recruiting, so we’d have to take on that task ourselves, which was always painful.

      The raised eyebrows were always about participant motivation and validity of the recruiting process and methodology. There were always questions: How valid are these findings? Are these real users? But when you’re intercepting users who are on your Web site in the middle of performing a task, those questions evaporate.

      On Participating in Remote Research

      In the past we’d invite our business partners or stakeholders to the lab, but it was difficult to get them to take time out of their day to travel to the lab, and it was a big production. But if they can just bring their laptop to a conference room down the hall and just be there to listen in, it’s fantastic. You’d get the same advantages if you had everyone available to go to the lab testing, and the level of engagement is a lot greater. By having a lot of stakeholders in the room, you get more diverse viewpoints, and the interaction between us observers and the moderator tends to be lively—we chat throughout that whole interview. The ability to observe and discuss things as they come up and then immediately give feedback to the moderator is really powerful.

      Because we are involved in the research process, we’ve got our customers and the usability to consider on the one hand, and on the other hand we have a lot of business stakeholders who have strong opinions about how things should be done. So having everyone in the room watching the feedback and engaging with the process is really powerful. We were recently in the middle of a digital camera usability session and were asking the user to go through the features and content we have on the site, and the customer’s going through it and he’s like, “This all seems really impressive, but I really just want to know if it takes great pictures.” And you see this light bulb go off above the product marketing people’s heads. We’re so close to this that we have absolute myopia. It was a real eye-opening moment.

      On Benefiting from Remote Methods

      One study was about TVs and the TV shopping process. Sony has a broad line of TVs, somewhere around 9 to 10 different series, and each has a dozen size options, so you have a lot of choices. During the study there was an “Ah-ha!” moment, a phenomenon we haven’t seen before: people would often have half a dozen to a dozen sites already open when we contacted them, and they were seamlessly going between sites like Engadget, CNET, Sony, Circuit City, Best Buy, really taking advantage of browser tabs to cross-shop and gather information. We simply wouldn’t have gotten that insight from a lab environment because we wouldn’t have been intercepting people in their natural browsing environment; instead, they’d sit down, have the browser already open, and they’d go. So that behavior would have been completely missed.

      The outcome was that, knowing that customers are looking not only for customer reviews but trustworthy, third-party editorial content, we’re actively pursuing ways to bring that content into the SonyStyle site, so that from within the interface they can access that info, instead of relying on the multi-tab approach. In the past, if a product was awarded an editor’s choice, we would have put that on the page as a badge of honor, but I doubt that we would have ever actually included the editorial alongside the product, if it hadn’t been for this study.

      Advice for Those Considering Going Remote

      If we’re talking about remote testing for Web sites, from my perspective it’s really a nonchoice. Having the benefit of intercepting users that are already coming to your site in order to perform a task already puts you so far ahead of the game because the motivation is there, you’ve got them captive, and you just gain so many more insights compared to creating an artificial environment with artificial motives. So you know from the quality and granularity of the results you’re going to get, it’s so much richer. If given the choice, I’ll never go back to lab testing again. And there’s the cost savings. Clearly, overall, it’s a less costly proposition. You avoid all the travel costs. There’s always a dud user in every batch of lab participants, and the great thing with usability testing is, if you start talking to someone you want to cut loose, it’s no harm; you can move on to the next person, as the recruiting form is literally filling up before your eyes.

What’s Remote Research GoodFor?

      Again, most studies can successfully be done either in person or remotely, but, just as there are times when lab testing is more appropriate, there are also times when it makes more sense to use remote research methods.

      Time-Aware Research

      Remote research is more appropriate when you want to watch people performing real tasks, rather than tasks you assign to them. The soul of remote research is that it lets you conduct what we call Time-Aware Research (TAR). By now, UX researchers are familiar with the importance of understanding the usage context of an interface—the physical environment where people are normally using an interface. Remote research opens the door to conducting research that also happens at the moment in people’s real lives when they’re performing a task of interest. This is possible because of live recruiting (the subject of Chapter 3), a method that allows you to instantly recruit people who are right in the middle of performing the task you’re interested in, using anything from the Web to text messages. Time-awareness in research makes all the difference in user motivation: it means that users are personally invested in what they’re doing because they’re doing it for their own reasons, not because you’re directing them to; they would have done it whether or not they were in your study.

      Consider the difference between these two scenarios:

       You’ve been recruited for some sort of computer study. The moderator shows you this online map Web app you’ve never heard of and asks you to use it to find some random place you’ve never heard of. This task is a little tricky, but since you’re sitting in this quiet