Remote Research. Tony Tulathimutte

the users’ expectations about what will happen during the study and what kind of mindset they should have entering the study. The most important things to establish are that you want the participants to use the interface like they normally would:

      So let me tell you a little bit about what we’ll be doing today. Basically, I’m going to be getting your feedback on ACME.com. Your job today is going to be really easy. Basically, I want you to just be yourself and act as you naturally would on the Web site. Now and then, I may chime in to ask you what you’re thinking, but mostly I’d just like you to show me how you use the site. If there’s a point where you’d normally quit or stop using the Web site, just let us know.

      And let them know you’d also like them to think aloud while they’re on the site:

      There’s just one thing that I would like you to do differently, which is to “think aloud” as you use the ACME Web site. For example, if you’re reading something, read it aloud, and feel free to comment as you read—for example “that’s interesting” or “what in the world are they talking about?” Let me know what’s going through your mind while you do things, so I can understand exactly what you are thinking and doing (for example, “Now I’m going to try to use the search engine”). If you get to a point where you would naturally leave or stop using the Web site, let me know.

      If your interface is a design in progress, you should set those expectations early so that users don’t bother wasting time trying to figure out something that isn’t finished anyway. You should remind them, however, that there’s no need to modify their behavior just because it’s a prototype. Our experience is that most people can’t tell the difference between a black-and-white prototype and a real functioning application anyway.

      Also, keep in mind that some things might not be working today on this prototype, which is fine. If you run into something that doesn’t work, I’ll let you know, and you can tell me what you were trying to do.

      It’s also nice to set users at ease by reassuring them that you had nothing to do with the design of the interface, so they can be completely honest:

      And finally, I want to let you know that I had nothing to do with the design of the Web site. I’m just collecting feedback, so feel free to be candid with your thoughts. No need to worry about hurting my feelings or getting anyone in trouble. Does that sound good? Great! Then let’s move on to the Web site.

      Natural Tasks (15–25 Minutes)

      If you’re conducting time-aware research, for the most part you won’t have to explicitly assign tasks. For example, if you’re testing the login process for your Web site, you can contact people just at the moment they first arrive at your homepage, when you know they haven’t yet logged in. This is what we call a “natural task”—something users were going to do anyway, whether or not they were participating in a study.

Since natural tasks have a lot of variability, this section of the script should be extremely open-ended. The moderator’s comments should be geared toward encouraging the user to do what he was intending to do with the interface, without directing the user to perform any specific tasks[1]. This is often the most important part of the testing session, but it’s actually the shortest part of the script simply because you can’t predict in advance exactly what the users will do. You may include some typical prompt questions, for your own reference, such as thefollowing:

       So, tell me what you’re looking at.

       What’s going through your mind right now?

       What do you want to do from here?

       Where would you go from here?

       Which parts of this page seem most interesting to you?

       What kind of info are you looking for on this page?

       What stands out to you most?

       What were you expecting when you first came to this page?

       When did you decide to leave the site/exit the program?

       What do you think about the way everything’s laid out?

       What brought you to this page?

       Is there any more info on this page you’d like to see included that you don’t already see here?

      Predetermined Tasks (5–15 Minutes)

      Since natural tasks are unpredictable, they may not always cover every last issue you’re curious about, so you’ll want to make sure you’re prepared to address any tasks, questions, or parts of the interface required by the study goals. This will require you to write down all the important questions you may have, even though you won’t necessarily be asking them all. Ideally, you want to cover as many of those questions as possible in the natural tasks section. Even though this section will appear long in the guide, you won’t be asking every question; therefore, it shouldn’t actually take as much time as the natural tasks.

      Wrap-up and Debrief (3–5 Minutes)

      Concluding the session should be the quickest part of most studies. Participants should be explicitly directed to exit or uninstall any screen sharing tools and tracking software used during the session and also informed about how they will receive incentives for participating in the study. It’s good to give participants a way to contact the researcher, in case they come up with any other questions regarding the study. And don’t forget to say thank you!

      Well, Bill, that does it. Let me just verify the email address where you would like us to send your incentive. [Verify necessary contact info.] The Amazon Gift Certificate usually takes 1–2 weeks to arrive and will be delivered via email. Check your junk mail folder if it doesn’t show up, and if you don’t receive it 2 weeks from today, send me an email at [email protected].

      [Disable screen sharing.] I’ve removed the screen sharing plug-in, so no need to get rid of that yourself; it’s completely removed from your computer.

      Do you have any last questions for me? Thanks so much for talking to me today; it’s been really helpful for us. Have a good day!

      Preparing Observers for Testing

      If you have colleagues or clients who plan on observing the sessions, you’ll have to give them their marching orders. Observers need to be aware of times when the sessions will be happening, how to use the screen sharing tool, and how to speak directly with the moderator during the session (which is one of the perks of remote testing).

      A standard way to prepare observers is to send an email with a complete set of instructions one week before testing and then a follow-up reminder email the day before testing, with the same set of instructions appended. Our sample instruction email shown in the sidebar assumes you’re using Adobe Connect to do screen sharing and using live recruiting techniques, but we have instructions for other tools posted at our Remote Usability Web site (http://remoteusability.com).

      On the day of testing, observers should be at their computers during testing hours, probably with a set of headphones so that they don’t have to hold the telephone all day long. You should let the observers know that depending on the scheduling and the method of recruiting, there may be significant downtime between each session. If you’re using live recruiting methods (as described in the following chapter), the amount of time between sessions may not be predictable, depending on the number of recruits you’re able to obtain.

      One thing to note: if you’re using live recruiting techniques, you won’t know exactly when the testing sessions are going to be held, so your email will have to set observers’ expectations that there may be periods of downtime between sessions, so they should have other things at hand to keep busy with.

      Observer Instructions

      The following is a sample email you should send to observers at least a week in advance of testing. The main point of the email is to inform observers what they