Statistics for HCI. Alan Dix

it’s not just you; it’s basic human psychology.

      We have two systems of thought¹: (i) subconscious reactions that are based on semiprobabilistic associations, and (ii) conscious thinking that likes to have one model of the world and is really bad at probability. This is why we need to use mathematics and other explicit techniques to help us deal with probabilities. Furthermore, statistics needs both this mathematics of probability and an appreciation of what it means in the real world. Understanding this means you don’t have to feel bad about finding stats hard, and also helps to suggest ways to make it easier.

1.1.1 IN TWO MINDS

      Skinner’s famous experiments with pigeons (Fig. 1.1) showed how certain kinds of learning could be studied in terms of associations between stimuli and rewards. If you present a reward enough times with the behaviour you want, the pigeon will learn to do it even when the original reward no longer happens. The learning is semi-probabilistic in the sense that if rewards are more common the learning is faster, or if rewards and penalties both happen at different frequencies, then you get a level of trade-off in the learning. At a cognitive level one can think of strengths of association being built up with rewards strengthening them and penalties inhibiting them.

Figure 1.1: Pigeon about to tap for food (source: https://archive.org/details/controllingbehaviorthroughreinforcement).

      This kind of learning is not quite a weighted sum of past experience: for example, negative experiences typically count more than positive ones, and once a pattern is established it takes a lot to shift it. However, it is not so far from a probability estimate. We humans share these subconscious learning processes with other animals. They are powerful and lead to very rapid reactions, but need very large numbers of exposures to similar situations to establish memories.

      Of course we are not just our subconscious! In addition, we have conscious thinking and reasoning, which enable us to learn from a single experience. Retrospectively we are able to retrieve a relevant past experience, compare it to what we are encountering now, and work out what to do based on it. This is very powerful, but unlike our more unconscious sea of overlapping memories and associations, our conscious mind is linear and is normally locked into a single model of the world. Because of that single model, this form of thinking is not so good at intuitively grasping probabilities, as is repeatedly evidenced by gambling behaviour and more broadly our assessment of risk.

      One experiment used four packs of cards with different penalties and rewards to see how quickly people could assess the difference [5]. The experiment included some patients with prefrontal brain damage, but we’ll just consider the non-patients. The subjects could choose cards from the different packs. Each pack had an initial reward attached to it, but when they turned over a card it might also have a penalty, “sorry, you’ve lost $500.” Some of the packs, those with the higher initial per-card reward, had more penalties, and the other packs had a better balance of rewards. After playing for a while most subjects realised that the packs were different and could tell which were better. The subjects were also wired up to a skin conductivity sensor as used in a lie detector. Well before they were able to say that some of the card packs were worse than the others, they showed a response on the sensor when they were about to turn over a card from the disadvantageous pack—that is subconsciously they knew it was likely to be a bad card.

      Because our conscious mind is not naturally good at dealing with probabilities we need to use the tool of mathematics to enable us to reason explicitly about them. For example, if the subjects in the experiment had kept a tally of good and bad cards, they would have seen, in the numbers, which packs were better.

1.1.2 MATHS AND MORE

      Some years ago, when I was first teaching statistics, I remember learning that statistics education was known to be particularly difficult. This is in part because it requires a combination of maths and real-world thinking.

      In statistics we use the explicit tallying of data and mathematical reasoning about probabilities to let us do quite complex reasoning from effects (measurements) back to causes (the real word phenomena that are being measured). So you do need to feel reasonably comfortable with this mathematics. However, even if you are a whizz at maths, if you can’t relate this back to understanding about the real world, you are also stuck. It is a bit like the applied maths problems where people get so lost in the maths that they forget the units: “the answer is 42”—but 42 what? 42 degrees, 42 metres, or 42 bananas?

      On the whole, those who are good at mathematics are not always good at relating their thinking back to the real world, and those of a more practical disposition are not always best at maths—no wonder statistics is hard!

      However, knowing this we can try to make things better.

      It is likely that the majority of readers of this book will have a stronger sense of the practical issues, so I will try to explain some of the concepts that are necessary, without getting deep into the mathematics of how they are calculated—leave that to the computer!

1.2 DO YOU NEED STATS AT ALL?

      The fact that you have opened this book suggests that you think you should learn something about statistics. However, maybe the majority of your work is qualitative, or you typically do small-scale studies and you wonder if it is sufficient to eyeball the raw data and make a judgement.

      Sometimes no statistics are necessary. Perhaps you have performed a small user trial and one user makes an error; you look at the circumstances and think “of course lots of users will have the same problem.” Your judgement is based purely on past experience and professional knowledge.

      However, suppose you have performed a survey comparing two alternative systems and asked users which system they prefer. The results are shown in Fig. 1.2. It is clear that System A is far more popular than System B. Or is it?

      Notice that the left hand scale has two notches, but no values. Let’s suppose first that the notches are at 1000 and 2000: the results of surveying 3000 people. This is obviously a clear result. However, if instead the notches were at 1 and 2, representing a survey of 3 users, you might not be so confident in the results. As you eyeball the data, you are performing some informal statistics.

Figure 1.2: User preferences comparing two systems.

      What if it were 10 to 20, or 5 to 10? How clear a result would that be? The job of statistics is precisely to help you with judgements such as these.

1.3 THE JOB OF STATISTICS –FROM THE REAL WORLD TO MEASUREMENT AND BACK AGAIN

      If you want to use statistics you will need to learn about t-tests and p-values, perhaps Bayesian statistics or Normal distributions, maybe a stats package such as SPSS or R. But why do this at all? What does statistics actually achieve?

      Fundamentally, statistics is about trying to learn dependable things about the real world based on measurements of it.

      However, what we mean by ‘real’ is itself a little complicated, from the actual users you have tested to the hypothetical idea of a ‘typical user’ of your system.

1.3.1 THE ‘REAL’ WORLD

      We’ll start with the real world, but what is that?

      the sample First of all, there is the actual data you have: results from an experiment, responses from a survey, or log data from a deployed application. This is the real world. The user you tested at 3 PM on a rainy day in March, after a slightly overfilling lunch, did make precisely