The Failure of Risk Management. Douglas W. Hubbard

decision analysis. As mentioned in chapter 1, risk analysis is only part of decision analysis. We will be spending a lot more time discussing these approaches.

      Other methods under the umbrella of “preference theory” were originally created as derivatives of the previously mentioned expected utility theory, but instead of trading off risk and return, they purport to mathematically assist in the trade-offs of multiple different objectives. Variously named but similar methods include multi-attribute utility theory (MAUT), multi-criteria decision-making (MCDM), and analytic hierarchy process (AHP). They claim more mathematical validity than simple weighted scores but ultimately rely on statements of preferences, not forecasts or estimates, of experts. In the case of AHP, a more sophisticated method is used to determine whether the expert judgments are at least internally consistent. As with the other methods listed so far, these have been used on lots of decision analysis problems that might not strictly be risk assessments, but they are included here because they have been used to evaluate decisions according to their risks.

      Whatever the chosen method may be, it should be used to inform specific actions. Many of those actions will involve choices regarding whether and how to mitigate risk in some way. You may decide to invest in new cybersecurity controls, keep tighter control over your supply chain, diversify production processes, increase the number of auditors, require new training, and so on. If they were free you would do them all. If all risk mitigation options were equally costly and equally effective, you could do them in any random order you like. But neither of those is the case. You will have more risks than you can realistically control for and the bang for the buck will vary widely. You will have to prioritize and make choices.

If these methods were used for no more than assessing corporate art for the reception area or where to have the company picnic, then the urgency of this evaluation would not be nearly as high. But, as I have already pointed out, these methods are being used for many of the biggest and riskiest decisions in the corporate world and government. Fortunately, some of these can be modified to produce an approach that can be shown to be a significant improvement on the baseline condition of expert intuition alone. Instead of improving on expert intuition, some apparently add error to expert intuition. Until this gets sorted out, improvements in risk management will not be possible.

NOTES

      1 1. “Fall Guys: Risk Management in the Front Line,” Economist Intelligence Unit, 2010, https://advisory.kpmg.us/content/dam/advisory/en/pdfs/risk-assurance/risk-management-front-line.pdf; “Best Practice in Risk Management: A Function Comes of Age,” Economist Intelligence Unit, 2007, http://graphics.eiu.com/files/ad_pdfs/eiu_Risk_Management.pdf.

      2 2. “Global Risk Management Survey 2017,” Aon Corporation, 2017; “Global Enterprise Risk Management Survey,” Aon Corporation, 2010; “Global Risk Management Survey 2007,” Aon Corporation, 2007, https://www.aon.com/getmedia/d95563c6-a3b8-4ff1-bb45-0ed511c78f72/2017-Global-Risk-Management-Survey-Report-rev-120318.aspx.

      3 3. “Executive Perspectives on Top Risks for 2018,” Protiviti & NC State Poole College of Management, 2018; “2007 U.S. Risk Barometer: Survey of C-Level Executives with the Nation's Largest Companies,” Protiviti, 2007, https://www.protiviti.com/sites/default/files/united_states/insights/nc-state-protiviti-survey-top-risks-2018.pdf.

CHAPTER 3 How Do We Know What Works?

      Leaders get out in front and stay there by raising the standards by which they judge themselves—and by which they are willing to be judged.

      —FREDRICK SMITH, CEO, FEDEX

      The first principle is that you must not fool yourself, and you are the easiest person to fool.

      —RICHARD P. FEYNMAN, NOBEL PRIZE–WINNING PHYSICIST

      According to some risk management surveys, organizations are very often satisfied with their risk assessment and risk management methods. For example, a survey by the major consulting firm Deloitte in 2012 found that 72 percent of organizations rate themselves as “extremely effective” or “very effective” at managing risks (up slightly from 66 percent in 2010). In other words, a majority believe their risk management is working. But, as the quote by Feynman above tells us, we are easy to fool.

A harder question to answer is, “What is the evidence for the belief that it works?” For any firm that hasn't asked that question before, it should be an immediate priority. If the firm can't answer that question, then it has no reason to believe that efforts to manage risks are working or, for that matter, are even focusing on the right risks. The standard must be some objective measure that could be verified by other stakeholders in the organization or outside auditors.

      Most (69 percent according to the HDR/KPMG survey) don't even attempt to measure whether risk management is working. Of those who say they do measure risk, most (63 percent) are merely using a survey of staff with questions such as, “How would you rate the effectiveness of risk management?” It may not be obvious now, but there are ways to measure risk management objectively even though such measurements are uncommon.

      This chapter will describe the difficulties in conducting measurements of risk management and some solutions for overcoming them. But first, to highlight the importance of measuring risk management, let's look at one example involving the health and safety of large numbers of people.

ANECDOTE: THE RISK OF OUTSOURCING DRUG MANUFACTURING

      In 2007, I was asked to speak at a conference organized by the Consumer Health Products Association (a pharmaceutical industry association). The event organizers were specifically interested in my contrarian views on common risk management methods. After my keynote, I was asked by the event organizers to attend another session on a new risk management method for outsourcing drug manufacturing and provide my comments to the audience. They thought it would be interesting if I could start a conversation by offering an on-the-spot evaluation of the new method.

      To control costs, this large pharmaceutical manufacturer was more frequently outsourcing certain batch processes to China. Virtually all of this manufacturer's competition were doing the same. But although the costs were significantly lower, they had a concern that batches from China might have additional quality control issues over and above those of batches manufactured here in the United States. These concerns were entirely justified.

Earlier that year there had already been several widely publicized product safety incidents with goods produced in China. In June, there was a toxin found in toothpaste and lead found in toys produced in China. Then there was tainted pet food that killed as many as 4,000 pets. There was even the disturbing case of “Aqua Dots,” the children's craft-beads that stuck together to make different designs. The coating of these beads could metabolize in the stomach to produce gamma-hydroxybutyrate—the chemical used in date-rape drugs.

So, clearly, assessing the risk of outsourcing was a major area of interest at the conference, and the room was at capacity. The presenter—a very respected chemical engineer—began to describe a risk assessment method based on a subjective weighted score.¹ In it, several “risk