Planning and Executing Credible Experiments. Robert J. Moffat

will that solve the problem that led to this work?

      Almost always, trying to answer Q1 honestly and carefully puts the issue in a new light.

      Once attention is focused on what we want to accomplish instead of on what we want to do, we can admit the possible existence of other ways to accomplish the same objective. This often leads us to formulate a more important question than we first had in mind and to propose a quite different experiment.

It is a good idea to start by formulating several versions of what seems to be the motivating question, and then start critiquing them. The first versions will leave a lot unsaid, and a good “devil's advocate” approach will make some of them look downright naive.² Don't be reluctant to play devil's advocate – if the question allows a silly answer, the question is poorly formulated. Precision will soon start to emerge.

      When you have a few good candidates, the test for identifying a really good motivating question is, “Which of these questions, if it alone is answered, will justify the cost of this experiment?”

      There can be only one “top priority question.” During the planning of the experiment, when a trade‐off is required, the motivating question provides the decision criterion. The motivating question is the “mission statement” of the experiment.

Sometimes, the question keeps on being refined, even while the program is in motion.³ No harm in that.

4.4 Three Levels of Questions

      In terms of experiment planning, it helps to distinguish three levels of questions that might be raised during the development of an experiment plan:

       Level 1: What happens?

       Level 2: By what means does it happen?

       Level 3: What are the underlying physics?

      It is important to identify the level of the motivating question and ensure that all following decisions are aimed at progress at that same level. The different levels lead to different experiments for a given situation.

      For example, consider a program aimed at studying the effects of high free‐stream turbulence u' on the value of the heat‐transfer coefficient h on a flat plate.

      The Level 1 question would be: “How does the heat‐transfer coefficient h vary with free‐stream turbulence u', all other factors remaining fixed?” This question would be answered by a series of experiments in which h was measured for different values of u' for some specified set of conditions.

      A Level 2 question would be: “What changes in the structure of the boundary layer were responsible for the changes in h?” This question would be answered by measurements of the velocity and temperature distribution within the boundary layer, seeking a correlation between high h and some recognizable change in the boundary layer structure.

      A Level 3 question would be: “How have the momentum and energy transports within the boundary layer been altered by the turbulence?” This question would be answered by measurements of the mixing length and turbulent Prandtl number or the turbulence intensity and dissipation length scale within the boundary layer.

      Note that these different levels require entirely different types of measurements, so it is important to know which level is most important to the client, the ultimate customer.

      There is an implicit presumption in modern fluid mechanics and heat transfer that low‐order events can be predicted from high order, i.e. that knowledge of the turbulence transport mechanisms will allow calculation of the boundary layer structure, and that knowledge of the boundary layer structure will allow prediction of h at the surface.

      This hierarchical assumption leads some experimenters to immediately start work at Level 2 or Level 3, seeking an elegant and powerful answer to the Level 1 question. In general, however, it seems to work best if one directly addresses the Level 1 question first over the entire range of the desired conditions before attempting to shift up to Level 2 and Level 3.

It is particularly important not to inadvertently mix levels in the same experiment plan. One may pursue two, or even three, levels of questions within one series of tests, and that is frequently done to save testing time, but one should remain aware that each level requires different data and that the different datasets are addressing separate objectives.

4.5 Strong Inference

      A particularly powerful approach to selection of the “right” question is described by John R. Platt, professor of physics and biophysics at the University of Chicago (1953), as “strong inference.” The essential features of strong inference are (1) the formulation of more than one alternative hypothesis concerning the major question, and (2) the execution of a set of experiments that have the possibility of disproving each hypothesis. The power of the method lies in the fact that, once a hypothesis has been disproved, then no further work need be done along that line. Note that if an experiment simply supports a hypothesis, not much progress has really been made, because the very next experiment may disprove it. To use strong inference, one must assemble systems of hypotheses that fully enclose the main question so that at least one hypothesis must finally survive its experimental test.

      Platt lists the following steps:

      1 Devise alternative hypotheses.

      2 Devise a crucial experiment (or several of them) with alternative possible outcomes, each of which will as nearly as possible exclude one or more of the hypotheses.

      3 Carry out the experiments so as to get clean results.

      4 Recycle the procedure, making subhypotheses or sequential hypotheses to refine the possibilities.

      Identification of the motivating question corresponds to the step devising alternative hypotheses. The specific experimental objective is, then, to test these hypotheses by experiment.

      Platt's contribution is his reminder to seek disproof for hypotheses, rather than support, as a more economic strategy.

4.6 Agree on the Form of an Acceptable Answer

      Once the question has been determined, the next step is to agree on the form of an acceptable answer. A clear agreement on the form of an acceptable answer helps ensure that both parties have the same question in mind.

      Thinking about the form of an acceptable answer may, in fact, lead to refining the question, especially if you play “devil's advocate” and look for “silly” answers. If “silly” answers are not precluded by the form of the question, then the question needs to be refined.

      The question “What is the effectiveness of this heat exchanger at its design conditions?” needs only a single number for an answer: “The effectiveness is 0.86.”

Another example, using instrument calibration, has more latitude. If the calibration question was: “Does this instrument meet its accuracy specification?” then a satisfactory answer would be a simple “Yes” or “No.” If, however, the question was “How do the readings of this instrument compare with the true values, over its range?” then an acceptable answer would require a table of values, or a chart, listing the indicated value corresponding to each true value.

      Note that a simple request to “Calibrate this instrument” does not establish which of these two answers would be acceptable and does not qualify as a desirable motivating question.

4.7 Specify the Allowable Uncertainty

      The third point that should be dealt with is uncertainty. How much uncertainty can be tolerated in the answer to the