The Pocket Guide to Critical Appraisal. Iain K. Crombie

9, ‘Appraising Randomised Controlled Trials’, and Chapter 11, ‘Appraising Systematic Reviews’. A new chapter for the critical appraisal of cohort studies that evaluate interventions has been added. This research design is now being frequently used in medical research (the separate chapter on conventional cohort studies has been retained). All the checklists distinguish between critical and important sources of bias and they evaluate risk of bias separately from value.

      Two new chapters at the end of the book provide methods for synthesising the findings of critical appraisal. One provides a method for summarising risk of bias. The second explores the important concept of certainty of evidence and describes how it is assessed. A further new chapter reviews the wide range of factors that determine the value of research findings to society as a whole. This second edition provides a much‐needed update to the guidance on critical appraisal.

CHAPTER 1 Introduction to Critical Appraisal

      Critical appraisal is the process of assessing the strengths and weaknesses of published studies. It involves a structured and rigorous evaluation of all the stages of the research, from design to analysis and interpretation. All studies have imperfections, so the question is not if there are flaws, but whether they are likely to be important. Critical appraisal is a method of systematically examining research studies to determine the worth of their findings.

The aims of critical appraisal

Critical appraisal assesses two issues: bias in study design or conduct, and the value that the findings have for clinical practice. Bias occurs when systematic errors distort the results of a study. For example, a clinical trial should provide a good estimate of the benefit of a treatment, but weaknesses in design or conduct could produce a misleading estimate. There are many types of bias and these are explored through a series of questions that examine the main deficiencies in research.

      The value of findings is the extent to which they will provide benefits for patients or the general population. Many factors contribute to the value of research findings. For a clinical trial, this would include the size of the treatment benefit, the importance of the outcome measure to patients, and whether the patients to be treated in a clinical setting are similar to the participants in the study. As with bias, there are sets of questions that assess value.

Three stages of critical appraisal

      The critical appraisal of a paper is conducted in three stages: identify the research design, select the appropriate appraisal tool for that study, and apply the tool to assess the bias and the value of the research. This section introduces these stages, with subsequent chapters explaining in detail how to carry them out.

      Identify the research design

      Research studies can ask different types of questions. Some may be concerned with the effectiveness of treatments; others may investigate the likely prognosis of an illness. The research design should be matched to the research question. To evaluate treatment effectiveness a randomised controlled trial would be used, whereas prognosis would be investigated by a cohort study. The other designs are surveys, case–control studies, cohort studies that test interventions, and systematic reviews. Chapter 3 of this book provides guidance on identifying the correct design.

      Select the appropriate appraisal tool

Critical appraisal tools have been developed to enable a forensic evaluation of research studies. They provide lists of questions that probe all aspects of the methods used and the results obtained. Judgements can then be made about the bias and value of the research. This book presents appraisal tools for the six most common research designs in medical research (Chapters 6–11).

      A key feature of the questions on bias is that they distinguish between critical and important sources of bias. For each research design there are a few crucial steps which, if conducted poorly, are very likely to lead to misleading findings. Flaws in the other steps are less likely to cause serious bias. Only when several of these flaws occur are serious concerns raised.

      Value, the usefulness or worth of a finding to individuals and to society, is more difficult to assess. Assessing the potential for improving health is usually straightforward, but often some pieces of information, such as the importance of the findings to patients, are not available. Decisions about value can use information from several sources and require judgement to balance many complex issues.

      Apply the tool to assess the bias and the value of the research

      The critical appraisal questions can only be answered by identifying key pieces of information from relevant sections of a paper. Chapter 2 provides a simple method for extracting the key pieces of information from a research paper. Careful reflection on what the questions reveal enables an assessment of the quality of the research study.

      The appraisal questions presented in this book are structured to lead to one of three answers: yes, no, or not enough information to decide. Selecting the appropriate option is not a simple tick box process. Critical appraisal involves pausing to think about the implications of each limitation, and whether it could seriously affect the interpretation of the study's findings. Sometimes there will be insufficient information to answer a question. As decisions in medicine affect patients, it is usually better to be cautious in drawing conclusions.

      In summary, critical appraisal assesses the bias and the value of research studies. By identifying the research design, the appropriate appraisal tool is selected. It asks probing questions about the design of the study and the way it was conducted, analysed, and interpreted. Chapter 2 describes a simple method for extracting the information from published papers to facilitate critical appraisal.

CHAPTER 2 Do Not Read the Paper

      The most important piece of advice when appraising a scientific paper is: do not read the paper. Scientific papers are complicated, information dense, and full of technical jargon – they cannot be read like a newspaper article. They are written for experts and prioritise technical correctness over ease of understanding. The research design is often complex, and the statistical analysis sophisticated. Papers are written with the assumption that the reader is familiar with the scientific terms and the methodologies used. Reading every word from the Introduction to the Discussion may lead to confusion.

A better approach is to interrogate the paper, asking questions to make it reveal the information about study design and conduct. Unlike reading, which is a passive process, critical appraisal searches out the key pieces of information about a study. Interrogation puts the reader in charge, directing questions at different sections of the paper, identifying and checking the information in the paper. This approach provides a quicker appraisal of a paper because it focuses on the important features of research studies. It probes not just what was done, but how well it was done. Sometimes the authors of papers try to gloss over weaknesses in their study. Interrogation will identify flaws that reading may miss. It provides all the information needed to evaluate the risk of bias and value of the results.

The initial interrogation

      The initial interrogation involves delving into some sections of the paper to get a feel for what the study is about. The aim is to start constructing a mental map of the study, not to engage in a serious appraisal. The main benefit of the map is that it allows you to slot subsequent