Impact Evaluation of Transport Interventions. David A. Raitzer

      Yasuyuki Sawada

      Chief Economist and Director General

      Economic Research and Regional Cooperation Department

      Asian Development Bank

       Acknowledgments

      This publication reflects the contributions of many individuals within and outside of the Asian Development Bank (ADB). It has been produced under the overall guidance of Edimon Ginting, Deputy Director General of the Economic Research and Regional Cooperation Department and Rana Hasan, Director of the Economic Analysis and Operational Support Division. The publication has also benefited from overall orientation by ADB’s interdepartmental Impact Evaluation Committee chaired by Bernard Woods, Director of the Results Management and Aid Effectiveness Division. David Raitzer, Economist, Economic Analysis and Operational Support Division; Nina Blöndal, independent consultant; and Jasmin Sibal, Economics Officer, Economic Analysis and Operational Support Division, authored the contents.

      Sakiko Tanaka provided valuable insights and contributions to early versions of the report. Independent consultant Howard White also provided valuable suggestions. Additional inputs and contributions were provided by ADB consultants Daryll Naval, Denise Encarnacion, and Reneli Gloria. Administrative support has been provided by Lilibeth Poot, Jindra Samson, Amanda Mamon, Gee Ann Burac, Rica Calaluan, Roslyn Perez, and Glennie Amoranto.

      Valuable peer reviews have been provided by Ari Kalliokoski, Shinichiro Nagao, Anders Pettersson, and Francesco Tornieri. Their input has helped to improve the content.

      Copy editing was performed by Tuesday Soriano and layout was performed by Joe Mark Ganaban. This publication is produced under regional technical assistance 0012 for Developing Impact Evaluation Methodologies, Approaches and Capacities in Selected Developing Member Countries.

       Key Messages

      There are important untested assumptions within traditional economic models of transport. For example, ex ante economic analysis models often mechanistically project that road improvements lead to increased vehicular speed, more productive time use, and reduced transport costs. Ex post analyses are usually based on before–after comparisons that assume that all changes over time are due to the road projects and not other trends or factors. This means that assumptions about how the roads are used and how users behave are often not directly tested. Understanding these behavioral aspects is critical to making transport investments more effective.

      Impact evaluation can bring new evidence to transport design choices. The approach can test basic assumptions about effects of transport on travel speeds, transport costs, time savings, employment, prices, productivity, and welfare. It can also test new ways of doing transport interventions better and shed light on the role of complementary interventions, such as contracting arrangements, maintenance schemes, road safety initiatives, and incentives to shift to more sustainable modes of transportation. It also can test transport policies and effects on pollution and other environmental outcomes.

      Transport investments have been subject to a growing number of impact evaluations. Transport has been a major sector for official development assistance. Yet, the evidence on the causal effects of transport sector interventions is relatively limited, compared with the social sectors. This is particularly true for sustainable transport investments, which often seek to change transport modes from private vehicles to public transit, make traffic behavior more efficient, or reduce excess travel. At the same time, a number of new and innovative impact evaluations have emerged in recent years, and the body of studies is growing rapidly. This review identifies 91 transport impact evaluations in developing countries, of which more than 65% have been published after 2012.

      Impact evaluations to date have found effects on a range of outcomes. These include effects on income, poverty, employment, education, health, gender disparities, land prices, firm productivity, migration, and much more. Both significant expected effects have been found, as well as unanticipated outcomes. A striking pattern is that effects are variable across studies and contexts, and that impacts are more heterogenous than transport programming often assumes. The evidence offered is limited to specific contexts, periods, and types of interventions, and much of the evidence on these outcomes pertains to rural roads. More studies are needed to better understand factors conditioning effects observed.

      Impact evaluations have focused increasingly on transport policies, in addition to infrastructure. A body of recent work has emerged, which evaluates a broader set of interventions, such as vehicle usage restrictions, safety interventions, and toll pricing on outcomes including driving behavior, congestion, local air pollution, and health. This new generation of impact evaluation often utilizes “big data,” multiple data sources, and automated data collection to capture new variables in innovative ways.

      Impact evaluation of transport has special challenges. The nature of transport is inherently challenging for impact evaluation. Not only are there problems of nonrandom placement and large facilities with few treated units, but there are also other unique problems. Transport conditions the effects of location, so it has inherently heterogenous impacts. It can also condition location and cause populations of interest to shift. At the same time, a number of methods show promise for transport sector applications, which have not been used frequently. These include regression discontinuity designs, encouragement designs, and in some contexts, synthetic controls.

      Randomized evaluations can offer evidence on a broader set of transport interventions. Although large transport infrastructure often cannot be randomly assigned, encouragement designs can introduce random variation in the use of transport facilities. These designs have untapped potential to enable testing of an expanded array of assumptions and interventions, especially through pricing encouragements.

      “Open data” can enable new impact evaluation possibilities. High frequency, geospatial, and “crowdsourced” data are increasingly available, and many of these data sets are relevant to transport. For example, remote sensing can now detect road network development and pollution, and ride-sharing applications can enable vehicle speed monitoring in real time. Many recent impact evaluations have started to use automatically collected data from traffic detection systems, air pollution monitors, public transit monitors, and even license plate recognition technology. These developments expand the number of impact evaluation possibilities.

      There are important transport evidence gaps that future studies can address. Impact evaluation has given relatively little coverage to major areas of investment, such as urban and sustainable transportation, major transport corridors, and efficiency enhancing measures. Effects via access to public services and prices have also had little investigation. Even where evidence exists, it is often limited to a few countries and programs. With new methods and increasing openness of geospatial data, there is scope for generating more innovative impact studies.

       1. Importance of Impact Evaluation for the Transport Sector

      The existing body of research on the causal effect of transport sector interventions is limited, but growing. To date, the effects of transport sector projects have mostly been modeled using mechanistic and/or unverified assumptions about how people and markets behave (ADB 2013a, Lacono and Levinson 2008). For example, project economic analysis typically uses baseline traffic counts and engineering specification of road improvements to mechanically project changes in travel speeds, costs, and time savings, without considering how drivers actually behave, or how external trends condition effects. Broader economic modeling, such as the use of computable general equilibrium models, is usually rooted in strong assumptions that markets operate in a perfectly competitive manner without frictions or transaction costs, under assumed elasticities of substitution, and often with assumed shocks from interventions.

      Impact evaluation, on the other hand, empirically measures the causal effects (and the statistical