Learning in Development. Olivier Serrat

interest in evaluations. In March 2005, ADB became the first member of the Evaluation Cooperation Group (ECG)¹⁰ of the multilateral development banks to formally adopt guidelines¹¹ to avoid conflict of interest in independent evaluations, which specify conditions under which OED management, staff, and consultants must recuse themselves.

      OED has also strengthened the process used to formulate its work program. Previously, only annual work programs were prepared, but starting in 2005, a 3-year rolling framework was developed.¹² Wide-ranging consultations took place for the preparation of the 2006–2008 work program. Discussions on institutional priorities for evaluations were held with the DEC and all vice-presidents. Inputs were also sought from ADB’s operations departments as in the past. The evaluation program has become more strategic and more integrated, with one evaluation feeding into another. For instance, evaluations of the effectiveness of ADB’s technical assistance (TA), microcredit operations, energy policy, governance and anticorruption policies, approaches to partnering and harmonization, policy-based lending, adoption of managing for development results, safeguard policies, and accountability mechanism were programmed to feed into reviews of these by ADB’s Management.

      The influence of evaluations on ADB’s operations, policies, and strategies was assessed in the 2006 Annual Evaluation Review.¹³ Their influence was evident in (i) the sector priorities for ADB’s operations under the new model for selectivity and focus set out in the second medium-term strategy, (ii) ADB Management’s agreement to implement an action program to improve portfolio performance in response to the DEC’s recommendation following its discussion of the 2005 Annual Report on Loan and Technical Assistance Portfolio Performance,¹⁴ and (iii) changes in new country strategies and programs directly related to lessons and recommendations from previous country assistance program evaluations.¹⁵

      Through its oversight of OED, the DEC is now helping to ensure that actions are taken on lessons and recommendations that it considers to be of high priority and that ADB’s Management has accepted. In its annual report to the Board in 2005, the DEC included a specific, candid assessment of the system for taking actions. The DEC considered that there was room for ADB to become a full-fledged learning organization using evaluation results more systematically. The DEC reported on actions taken by OED and by ADB’s Management on its recommendations:

(i)	Traditionally, the intended end-users were consulted during and near the completion of an evaluation. Nowadays, OED reviews the scope and methodology with the main stakeholders, particularly ADB’s operations departments, before starting.
(ii)	OED has introduced the new step of discussing draft findings and recommendations from evaluation with ADB’s operations departments and, in the case of country assistance program evaluations, with the government before the evaluations are finalized. The objectives are to ensure that those who will be responsible for implementing the recommendations understand them, to find out which are acceptable and feasible, and to build early commitment.
(iii)	The actions that ADB’s Management commits to in its formal responses to evaluation reports have generally been more specific and time bound.

      To improve consistency and quality, guidelines for the evaluation of public sector projects, program loans, TA, and country assistance programs were issued in 2005. Guidelines for evaluating private sector operations were completed in 2007.¹⁶ The new modus operandi has brought institutional advantages, but it also raises the question of how OED itself should be evaluated. To address this issue, the ECG has begun to examine the feasibility of establishing a peer review process of the evaluation function in its members.

       Evaluating Operations Performance

There is a long feedback loop from approval to completion of an operation, when development results can be assessed. It takes 7 years or more to implement projects or programs after Board approval and for completion and evaluation reports to be prepared. Thus, virtually all projects or programs that were approved in the mid-1990s were in various stages of implementation at the turn of the millennium. Indeed, many are still under implementation. The completion and evaluation reports for most of the projects and programs approved in the 1990s were prepared between 2000 and 2005. Analysis of project success was undertaken based on the year of approval. Figure 1 shows that the success rates of Asian Development Fund (ADF)-funded projects approved before the 1990s tended to be considerably lower than those funded by ordinary capital resources (OCR),¹⁷ and average ADF ratings were much more volatile. In the 1990s, however, the OCR and ADF ratings converged, and both improved. Given the development challenges faced by ADF-eligible countries, this is a positive finding. The long-term trend of the ratings suggests that the design and implementation of ADB-assisted projects have progressively improved, whether funded by OCR or by ADF. This indicates successful, gradual institutional learning from lessons of past operations. This improvement in project outcomes needs to be continuously monitored to ensure that the trend is sustainable.

Figure 1: Trends in Project Ratings by Year of Approval and Source of Funding^a (Combined Completion and Evaluation Results)

      ^a Success rate for ADF and OCR based on 3-year moving average.

      ADF = Asian Development Fund, OCR = ordinary capital resources.

      Source: OED.

While project performance has varied significantly across countries, it has not varied significantly by country classification (i.e., degree of eligibility for ADF funding).¹⁸ The analysis by country classification in Table 1 demonstrates improvement from the 1970s to the 1990s across all country groupings, though patterns of progress differ. By the 1990s, the success rate of each group with access to ADF had gradually increased, and countries in Group A (with the exception of Nepal) were performing better than those in Groups B1 and B2.

Table 1: Project Success by Country Classification

      Source: OED.

      The averages by classification mask important country differences. Within all groups, there are outliers in both directions. The country variation holds true for source of funding. For some OCR borrowers, projects are more likely to be successful than in ADF-eligible countries, but ADB’s portfolios perform better in some countries that have access to ADF than in some OCR borrowers. In Group B2, the People’s Republic of China stood out as one of ADB’s best-performing portfolios, achieving an 82% success rate in the 1990s. In contrast, Papua New Guinea had a 22% success rate in the 1990s, the weakest portfolio performance in ADB. In Group B1, the success rates of the portfolios in Bangladesh (84%) and Viet Nam (91%) significantly exceeded both group and ADB averages for projects approved in the 1990s. In contrast, only 57% of projects approved for Pakistan in the 1990s were rated as successful. In Group A, the portfolio success rates of Bhutan, Cambodia, Kyrgyz Republic, Lao People’s Democratic Republic, Maldives, Mongolia, and Solomon Islands exceeded Group A and ADB averages in the 1990s. On the other hand, the success rates for Kiribati, Nepal, Samoa, and Vanuatu were below average for countries in Group A.

      There is a school of thought that country ownership, which is a key factor for project success, increases with a country’s share of the financing of a project. This hypothesis was tested by comparing project success rates with the percentage of project costs financed by the executing agency after controlling for sector differences. No significant statistical relationship was found, nor was there any clear pattern to suggest that project