Validating Product Ideas. Tomer Sharon

easy task, especially if you are on your own. If you work in a team, have everyone participate. To prepare for group classification, predetermine the categories by which you’ll classify the answers. Create this list with your team and invest a little time clarifying what goes under each category. For example, imagine your experience sampling question is “What frustrated you the last time you went grocery shopping?” Categories for answers for this question might be the following:

      • Location, refers to where the frustration took place, and the options are: home, way to store, in car, at parking lot, at store, and way from store.

      • Close people are the friends and family who might be involved in grocery shopping frustrations: spouse, parent, sibling, roommate, and child.

      • People at the store might be other customers, cashier, deli personnel, produce personnel, dairy personnel, and other service people.

      • Issue could be finding items, understanding costs, long lines, and shopping cart.

      During analysis, you are disassembling each qualitative answer into components that you will later reassemble quantitatively. For example, in the previous example, you might be able to say the frequency at which people were frustrated during grocery shopping was the result of a long line created by other customers in the parking lot. You’ll then be more informed when you try to uncover user needs.

STEP 7: Set participant expectations.

      Study participants will make or break an experience sampling study. When the study is run, you are not there to see what your participants do and correct it if needed. Therefore, take all the necessary steps to make sure that participants understand what is asked of them and that they provide useful answers.

      Tell participants in advance what’s expected of them. For example, if you expect them to answer five notifications per day for three days, tell them exactly that. Explain that fewer responses do not help you learn from them. Don’t give them an example of a good answer to the study question; otherwise, you’ll bias them, and you’ll just get answers similar to your example. Stay vague. Tell them a good answer is one that you can understand, that is detailed enough, and that does not include one or two words but at least a full sentence.

      Always double-check that the participants do not have a holiday or foreign travel planned, unless it’s the focus of the research. Let them know that it’s okay to miss an answer here and there if they’re busy and that they can answer a little bit later.

      Tell participants they will get an incentive if they meet your answer quota. If you can’t pay participants with money, be creative and think of other ways to provide an incentive for participation. For example, dedicate a wall in your office (or a page on your website if you don’t have an office) to study participants. Call it The Wall of Research Fame and print their names on it after they participate. Take a picture and share it with each participant. We humans love our names. Take advantage of it!

STEP 8: Launch a pilot, then the study, and monitor responses.

      To make sure that participants understand what’s required of them and to test your notification mechanism, schedule a practice test where you send one notification to participants and they answer once. Do that a couple of days before the actual study begins. This way, you’ll have some time to make changes in case something does not work as expected. It’s also a great opportunity to make sure that participants give you the answers you need with enough level of detail. Don’t skip this step, because you’ll regret it if you do.

      After you iterated the study details following the pilot-test and all of the preparations are complete, it is time to start the study. Here are the steps you need to take:

      1 Launch and track: Launch the study, start sending notifications, and immediately track answers. Evaluate each individual participant’s responses and make sure that each person is giving you what you need.

      2 Adjust participant behavior and clarify: Contact participants who do not give you what you need or do not respond at all. Ask them to allow you to help them. Troubleshoot their issues and refine their participation. For example, if people give you one-word answers, explain that you need to be sure you understand what they meant and ask them to provide longer sentences as answers. Or if participants say they did not receive any notification, figure out the reason.

      3 Pat participants on the back: Encourage participants who cooperate fully and provide clear, useful answers. Tell them you are very happy with their participation and the level of detail they provided in their answers. Be careful not to say that any specific answers they gave were great. Otherwise, you’ll bias them to think this is what you need, and they’ll keep giving you the same answers. This way, you’ll miss other useful answers.

      4 Thank participants: Be sure to thank participants, no matter what. Some people will stop participating if they think their effort is underappreciated.

STEP 9: Analyze data.

      Begin analysis when you’ve collected about five percent of the expected number of responses. This will help you fine-tune the rest of the analysis. Look at each of the answers you collected, and one by one, classify them into the categories you have predefined. If you work in a team, do the first chunk together. This way you’ll understand better how to classify answers in a consistent manner. For example, here’s an answer you might get:

       A slow cashier combined with an elderly person who was in front of me in the line caused me to be late to pick up my son from school.

      This answer could be classified as follows:

      1 Location: at the store

      2 Close people: N/A

      3 People at the store: other customers, cashier

      4 Issue: long lines

      Get the idea? This way, you disassemble each and every answer you collected into components you can count later.

      As you make progress with classification, you will realize that some categories need to change, split, merge, or be removed, and that new ones should be added. For example, you might find that under Location, it doesn’t make sense to have both in car and way to store, because they are redundant. Or you might find there are many answers that would benefit from creating a separate Cost category with values such as item, expensive, cheap, compare to other store, and so on.

      Eliminate answers that are just incomprehensible or irrelevant. Don’t assume you understand what they mean. Only refer to the text in the answer. If there’s doubt, there’s no doubt. For example, if your question was “What frustrated you the last time you went grocery shopping?” and you get the answer “Dogs,” don’t guess what the reason was or assume any type of category for classification. This is an incomprehensible answer you should ignore.

      If you classify data as a team, work together while you are all in the same room so that you can discuss things as you make progress. If this is not possible, catch up with team members once in a couple of hours (or at least daily) to do the following:

      • Spot-test how team members are classifying so that you ensure the whole team is being consistent. (It’s a pain to undo later with thousands of data points.)

      • Highlight and discuss any data that is hard to classify with existing categories.

      • Create new categories and then share them with the whole team as soon as possible. If you don’t do this, then team members start to invent their own inconsistent categories or just use existing categories that are inappropriate, which is harder to consolidate later.

      Work on one spreadsheet when you classify. This spreadsheet should include all individual