Stat-Spotting. Joel Best

ection>

      

      STAT-SPOTTING

      STAT-SPOTTING

      A FIELD GUIDE TO IDENTIFYING DUBIOUS DATA

      UPDATED AND EXPANDED

      JOEL BEST

      University of California PressBerkeleyLos AngelesLondon

      University of California Press, one of the most distinguished university presses in the United States, enriches lives around the world by advancing scholarship in the humanities, social sciences, and natural sciences. Its activities are supported by the UC Press Foundation and by philanthropic contributions from individuals and institutions. For more information, visit www.ucpress.edu.

      University of California Press

      Berkeley and Los Angeles, California

      University of California Press, Ltd.

      London, England

      © 2008, 2013 by The Regents of the University of California

      First paperback printing 2013

      ISBN 978-0-520-27998-8

      eISBN 9780520957077

      “Crystal-Clear Risk” on p. 23 is from “Party, Play—and Pay: Multiple Partners, Unprotected Sex, and Crystal Meth,” Newsweek, February 28, 2005, 39. © 2005 Newsweek, Inc. All rights reserved. Used by permission and protected by the Copyright Laws of the United States. The printing, copying, redistribution, or retransmission of the Material without express written permission is prohibited.

      Earlier discussions of some of the examples used in this book appeared in “Birds—Dead and Deadly: Why Numeracy Needs to Address Social Construction,” Numeracy 1, no. 1 (2008): article 6, and in the “Out of Context” feature in issues of Contexts magazine published between spring 2005 and fall 2007 (volumes 4–6).

      The Library of Congress has catalogued an earlier edition as follows:

      Library of Congress Cataloging-in-Publication Data

      Best, Joel.

      Stat-spotting : a field guide to identifying dubious data / Joel Best.

      p. cm.

      Includes bibliographical references and index.

      ISBN 978-0-520-25746-7 (cloth : alk. paper)

      1. Sociology—Statistical methods. 2. Social problems—Statistical methods. I. Title.

      HM535.B4772008

      301.072’7—dc222008017175

      Designer: Nola Burger

      Text: 11/15 Granjon

      Display: Akzidenz Grotesk and Akzidenz Grotesk Condensed

      Manufactured in the United States of America

      21 20 19 18 17 16 15 14 13

      10 9 8 7 6 5 4 3 2 1

      The paper used in this publication meets the minimum requirements of ANSI/NISO Z39.48-1992 (R 2002) (Permanence of Paper).

      The statistics you don’t compile never lie.

      – STEPHEN COLBERT

      CONTENTS

      PREFACE TO THE 2013 EDITION

      PART 1GETTING STARTED

      A.SPOTTING QUESTIONABLE NUMBERS

      B.BACKGROUND

       B.1Statistical Benchmarks

       B.2Severity and Frequency

      PART 2VARIETIES OF DUBIOUS DATA

      C.BLUNDERS

       C.1The Slippery Decimal Point

       C.2Botched Translations

       C.3Misleading Graphs

       C.4Careless Calculations

      D.SOURCES: WHO COUNTED–AND WHY?

       D.1Big Round Numbers

       D.2Hyperbole

       D.3Shocking Claims

       D.4Naming the Problem

      E.DEFINITIONS: WHAT DID THEY COUNT?

       E.1Broad Definitions

       E.2Expanding Definitions

       E.3Changing Definitions

       E.4The Uncounted

      F.MEASUREMENTS: HOW DID THEY COUNT?

       F.1Creating Measures

       F.2Odd Units of Analysis

       F.3Loaded Questions

       F.4Raising the Bar

       F.5Technical Measures

      G.PACKAGING: WHAT ARE THEY TELLING US?

       G.1Impressive Formats

       G.2Misleading Samples

       G.3Convenient Time Frames

       G.4Peculiar Percentages

       G.5Selective Comparisons

       G.6Statistical Milestones

       G.7Averages

       G.8Epidemics

       G.9Correlations

       G.10Discoveries

      H.RHETORIC: WHAT DO THEY WANT US TO THINK?

       H.1Using Short-Term Turnover to Measure Long-Term Problems

       H.2Sudden Turns for the Worse

       H.3Designating Myths

       H.4Rhetorical Flourishes

      I.DEBATES: WHAT IF THEY DISAGREE?

       I.1Causality Debates

       I.2Equality Debates

       I.3Policy Debates

      PART 3STAT-SPOTTING ON YOUR OWN

      J.SUMMARY: COMMON SIGNS OF DUBIOUS DATA

      K.BETTER DATA: SOME CHARACTERISTICS

      L.AFTERWORD: IF YOU HAD NO IDEA THINGS WERE THAT BAD, THEY PROBABLY AREN’T

      M.SUGGESTIONS FOR THOSE WHO WANT TO CONTINUE STAT-SPOTTING

      ACKNOWLEDGMENTS

      NOTES

      INDEX

      PREFACE TO THE 2013 EDITION

      I was introduced to thinking about bad statistics when I read Darrell Huff’s How to Lie with Statistics as a first-year college student. By the time I checked out the library’s copy, that book was more than ten years old, and the volume showed a lot of wear. Huff’s examples