Improving the World Bank’s Development Effectiveness What Does Evaluation Show? 2005 The World Bank Washington, D.C. WORLD BANK OPERATIONS EVALUATION DEPARTMENT http://www.worldbank.org/oed © 2005 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 Telephone 202-473-1000 Internet www.worldbank.org E-mail [email protected] All rights reserved Manufactured in the United States of America The findings, interpretations, and conclusions expressed here are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. The World Bank cannot guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in the work do not imply on the part of the World Bank any judgment of the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The World Bank encourages dissemination of its work and will normally grant permission promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400, fax 978-750-4470, www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, World Bank, 1818 H Street NW, Washington, DC 20433, USA, fax 202-522-2422, e-mail [email protected]. Cover photo: Uzbekistan family by Curt Carnemark for the World Bank Photo Library. ISBN-10: 0-8213-6461-8 ISBN-13: 978-0-8213-6461-1 e-ISBN 0-8213-6462-6 DOI: 10.1596/978-0-8213-6461-1 Library of Congress Cataloging-in-Publication Data has been applied for. Ainsworth, Martha, 1955– Committing to results : improving the effectiveness of HIV/AIDS assistance: an OED evaluation of the World Bank’s assistance for HIV/AIDS control / Martha Ainsworth, Denise A. Vaillancourt, Judith Hahn Gaubatz. p. cm. — (Operations evaluation studies) Includes bibliographical references. ISBN-13: 978-0-8213-6388-1 ISBN-10: 0-8213-6388-3 1. Economic assistance—Developing countries—Evaluation. 2. AIDS (Disease)—Economic aspects—Developing countries. 3. HIV infections—Economic aspects—Developing countries. 4. AIDS (Diseasse)—Developing countries—Prevention. 5. HIV infections—Developing countries—Prevention. 6. World Bank. I. Vaillancourt, Denise. II. Hahn Gaubatz, Judith. III. Title. IV. World Bank operations evaluation study. HC60.A4575 2005 362.196'9792'0091726—dc22 2005052329 Printed on Recycled Paper World Bank InfoShop E-mail: [email protected] Telephone: 202-458-5454 Facsimile: 202-522-1500 Operations Evaluation Department Knowledge Programs and Evaluation Capacity Development (OEDKE) E-mail: [email protected] Telephone: 202-458-4497 Facsimile: 202-522-3125 iii v Executive Summary ix Acronyms and Abbreviations 1 1 Progress on Poverty and Improved Living Conditions 7 2 The Bank’s Record in Supporting Development 7 Project Performance Ratings Show Improvements over the Past 10 Years 9 Country Assistance Evaluations 14 Two Special Categories of Country Programs 14 Post-Conflict Countries 14 Four Lessons Learned 16 Middle-Income Countries 17 Global Programs 21 3 Crosscutting Lessons from Sector and Thematic Evaluations 27 4 Looking Forward 29 Annexes 31 A: Management Comments 43 B: Report to the Board from the Committee on Development Effectiveness (CODE) 47 Endnotes 49 References Figures 1 1.1 Poverty Reduction Shows Large Regional Variations (1981–2001) 2 1.2 Only One-Third of Developing Countries Are Converging 4 1.3 Income Diverged But Social Indicators Converged Overall 5 1.4 Policy and Institutional Improvements 5 1.5 Democracy Advanced and Conflict Declined in the Past Decade 8 2.1 The Outcomes of Bank-Supported Projects Have Been Improving Contents 8 2.2 Project Outcomes Improved in All But One Region 10 2.3 The Majority of Sectors Show Improvement 11 2.4 Almost One-Third of Country Programs Show Unsatisfactory Outcomes 14 2.5 Post-Conflict Countries Received Very Different Levels of Bank Support per Capita 16 2.6 Bank Commitment and Disbursement as a Ratio to Total Budget Are Declining Tables 9 2.1 Country and Project Outcome Ratings 12 2.2 Sectoral and Project Ratings for Fiscal Year 2001–03 CAEs 19 2.3 Only a Few Global Programs Address Global Public Goods IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? iv v Executive Summary Our task is not to fix the blame for the past—but to help fix the course for the future. —John F. Kennedy The pace of change in the overall performance of the developing world has not altered markedly over the past 20 years. The number of people living in extreme poverty declined from 1.5 billion in 1980 (40 percent of population), to 1.2 billion in 1990 (28 percent of population), to 1.1 billion in 2001 (21 percent of population). Growth per capita has followed much the same profile. In the 1980s, only about two-thirds of developing countries showed positive per capita income growth, and this percentage remains unchanged. Life expectancy and literacy indicators show overall improvements, but some Regions show worrisome trends. There has been slow and steady progress in overall development outcomes during the period, but the speed and scale of change remain static. These averages, of course, mask huge differences across Regions, with very worrisome increases in poverty and continued low growth in Sub-Saharan Africa. Since the mid-1990s, the World Bank has dramatically altered its direction to emphasize poverty eradication as the institution’s main mis sion. Many programs and policies have been revised, with the objective of making the Bank more effective in its support of the global fight against poverty. The following key changes have been frequently highlighted in internal communications and external commentary on the Bank as strategic moves designed to improve the Bank’s development effectiveness: • Eradication of poverty as the Bank’s central mission statement • Shift to country focus combined with decentralization of staff • Expansion of global programs to address global challenges in partnerships • Enhanced participatory and community-based project approaches • Increased attention to governance and institutional change • Greater country ownership and donor alignment through Since the mid-1990s, the World Bank has dramatically altered its direction to emphasize poverty reduction as the institution’s main mission. Poverty Reduction Strategy Papers (PRSPs) and other instruments •Greater focus on results and performance •New approaches to deal with fragile states— for example, low-income countries under stress (LICUS) • Enhanced safeguard policies. Some of these initiatives are now well established, while others, such as the strengthened results focus, were introduced more recently and are still evolving. How have these initiatives improved the Bank’s development effectiveness? This paper uses recent evaluations to assess the development effectiveness of the World Bank and how it could be improved. It covers three questions: 1. What is the measurable progress in improved living conditions in the developing world over the last 10–15 years, and how does it compare with the previous decade? 2. How effective has the World Bank been in helping countries improve their living conditions through its various interventions and programs? 3. Finally, what broad lessons emerge from OED evaluations on improving the Bank’s development effectiveness? The Bank’s development effectiveness can be measured at the project or individual program level, at the country level, and by looking at its global programs. There has been a steady improvement over the past decade in the ratings of the outcomes of Bank projects. Outcome ratings have increased from around 65 percent satisfactory in the mid-1990s to over 75 percent satisfactory in the past four years. This is indeed impressive progress, and it should be continued. But it does not necessarily indicate improved development impact at the country level. OED’s Country Assistance Evaluation (CAE) ratings, which take a long-term perspective and can better assess the impacts of World Bank support to countries, show a satisfactory outcome in about two-thirds of the Bank’s country programs. They also show that even when project ratings in a sector are high, sectoral outcomes may not be satisfactory. With one-third of country programs rated unsatisfactory, there is substantial room for improvement in the Bank’s development effectiveness through more coherent country programs tailored to country circumstances, as well as through further improvements in specific projects and programs. The PRSP initiative offers the low-income countries a framework under which both country ownership can be encouraged and donor interests can be incorporated. This direction could also improve the outcomes of the Bank’s country assistance programs. The Bank is struggling to improve its effectiveness at two ends of the development spectrum—in LICUS countries where state capacity has collapsed or weakened considerably, and in middle-income countries (MICs) that have much greater access to other sources of capital. Greater focus and selectivity are also needed in the Bank’s global programs to enhance their poverty impact and their benefits to developing countries. The Bank also must focus on its costs of doing business, which have risen significantly over the last 10 years. For every dollar of its administrative budget the Bank disbursed $13 in fiscal year 1995, but only $9 in fiscal 2005, after a brief increase in fiscal 1998–99 during the Asian crisis. Bank commitments have also fallen from $16 per dollar of administrative budget in fiscal 1995 to $11 in fiscal 2005. The average size of loans has fallen from around $90 million in the mid1990s to around $80 million in the last two years.1 IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? vi There has been a steady improvement over the past decade in the ratings of the outcomes of Bank projects, but almost onethird of country programs are rated unsatisfactory. The PRSP initiative offers promise, but the Bank is struggling to improve its effectiveness at two ends of the development spectrum—LICUS and MIC countries. Lending has stagnated but the budget has continued to grow. Lending has dropped sharply to middle-income countries with access to cheaper sources of finance. The Bank argues that some of the increased cost, such as safeguards or new communitybased approaches, has helped improve its projects. Some of the increased cost is due to expansion of knowledge activities, and some to the financing of global programs. But smaller loans do not square with the Bank’s objective of scaling up its poverty-reducing activities. It must also address why lending is declining at the Bank and not in other Multilateral Development Banks (MDBs), and how effective all its non-lending activities—the knowledge bank—are for supporting development. Knowledge and lending should complement each other and not be seen as substitutes. OED findings and evaluations at the project, sectoral and thematic, and country and global levels suggest that the following nine directions can improve the Bank’s development effectiveness: • Understand and analyze comprehensively, but act far more selectively. • Emphasize institutional development and capacity building even more strongly. • Re-think areas of punctured optimism, such as growth, private provision of infrastructure, and turn-around situations. • Tailor programs and projects to the circumstances of each country, and adapt strategies to the local political economy. • Lend mainly to countries with improving policies and institutions, but find ways to deal with poor, misgoverned states. • Introduce greater flexibility into programs with well-managed MICs. • Make better use of the Bank’s knowledge and technical assistance. • Improve monitoring and evaluation for results, and start measuring what is important. • Improve coordination within the Bank and across the Bank Group. This review of the Bank’s development effectiveness points to a number of opportunities for further improvements. Some of these will require changes in business processes, such as new ways of dealing with LICUS and MICs and further improvements with the PRSPs. Some will require changes in the organization, to improve coordination within the Bank and across the Bank Group to set the right incentives; consolidate the Bank’s networks toward the two pillars—the investment climate and social inclusion; more focused use of knowledge to address country needs, less mechanistic application of safeguards; and a review of the matrix structure, which leads to multiplication of tasks and works against selectivity. Still others will require strategic choices for the institution, such as how to define its role more selectively, more focused on its core competencies in the global war on poverty, rather than trying to cover every aspect of development. A greater emphasis on growth is needed for lasting poverty reduction. The Bank has transformed itself significantly in the past 10 years, and should be ready for further adjustments to current climate of rapid change. Greater selectivity, more flexibility, and improved efficiency within its chosen areas of intervention are needed going forward if a global institution such as the Bank is to remain useful and relevant and show concrete results in a fastchanging world. EXECUTIVE SUMMARY vii Greater emphasis on growth is vital for lasting poverty reduction. Greater selectivity and a more hard-nosed focus on results are needed. ix ACRONYMS AND ABBREVIATIONS AAA Analytical and advisory activities AFR Africa Region APL Adaptable Program Loan ARDE Annual Review of Development Effectiveness CAE Country Assistance Evaluation CAS Country Assistance Strategy CDD Community-driven development CEPF Critical Ecosystems Partnership Fund CGAP Consultative Group to Assist the Poor CGIAR Consultative Group on International Agricultural Research CPIA Country Policy and Institutional Assessment (ratings) EAP East Asia and the Pacific Region ECA Europe and Central Asia Region ED Education EMT Energy and mining ENV Environment EP Economic Policy ESMAP Energy Sector Management Assistance Program ESW Economic and sector work FIRST Financial Sector Reform & Strengthening Initiative FSAP Financial Sector Assessment Program FSP Financial sector FY Fiscal year GAVI Global Alliance for Vaccines and Immunization GDN Global Development Network GDP Gross domestic product GEF Global Environment Facility GIF Global Integrated Pest Management Facility GPP Global programs and partnerships GWP Global water partnerships HIPC Heavily Indebted Poor Countries HNP Health, nutrition, and population IBRD International Bank for Reconstruction and Development (World Bank) IDA International Development Association IDB Inter-American Development Bank IF Integrated Framework for Trade-Related Technical Assistance IFC International Finance Corporation IFI International financial institution infoDev Information for Development Program LAC Latin America and the Caribbean Region LDC Least-developed country LICUS Low-income countries under stress LIL Learning and Innovation Loan M&E Monitoring and evaluation MDB Multilateral development bank MDG Millennium Development Goal MIC Middle-income country MIGA Multilateral Investment Guarantee Agency MLF Multilateral Fund for the Implementation of the Montreal Protocol MNA Middle East and North Africa Region ODA Official development assistance OED Operations Evaluation Department (World Bank) PPIAF Public-Private Infrastructure Advisory Facility PREM Poverty Reduction and Economic Management Network PRS Poverty reduction strategy PRSP Poverty Reduction Strategy Paper PSD Private sector development PSDD Private Sector Development Department PSG Public sector governance PSM Public sector management RBM Roll Back Malaria RDV Rural sector SAR South Asia Region SP Social protection SSA Sub-Saharan Africa SWAp Sectorwide approach TDR Special Program for Research and Training in Tropical Diseases TR Transport TSS Transitional Support Strategy UCW Understanding Children’s Work UD Urban development UNAIDS Joint United Nations Program on HIV/AIDS USAID United States Agency for International Development WSP Water and Sanitation Program WSS Water supply and sanitation IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? x 1 Progress on Poverty and Improved Living Conditions The global fight against poverty shows progress—but at a very slow pace and with huge variations across Regions. The number of people in the developing world living in extreme poverty declined from 1.5 billion in 1980 (40 percent), to 1.2 billion in 1990 (28 percent), and to 1.1 billion in 2001 (21 percent). This progression could be construed as a slow-down in the rate of improvement (figure 1.1). 1 1 Figure 1.1: Poverty Reduction Shows Large Regional Variations (1981–2001) 0 10 20 30 40 50 60 70 East Asia & Pacific, excluding China China Eastern Europe & Central Asia Latin America & Caribbean Middle East & North Africa South Asia Sub-Saharan Africa All LDCs Percentage of population living on less than US$1 per day 1981 1990 2001 Source:2004 Annual Review of Development Effectiveness (OED 2005a). Note:LDCs = least-developed countries. 2 IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? Figure 1.2: Only One-Third of Developing Countries Are Converging Source:SIMA–February 2005. GDP PC observations for 143 developing countries between 1990 and 2003. –9.0 –6.0 –3.0 0.0 3.0 6.0 9.0 $100 $1,000 $10,000 $100,000 Income GDP per capita growth 2.1% Long-run growth in U.S. GDP per capita Much of the decline took place in East Asia— and most spectacularly in China. In South Asia, which has the largest population of poor people, the number of poor people has begun to decline after almost a half-century of increase.1 Poverty has increased sharply in the Africa and Eastern Europe and Central Asia Regions and has shown no visible improvements in Latin America or the Middle East. Overall, some 50 percent of developing countries showed an increase in the number of people below the poverty line of US$1 per day during the 1990s. In the past decade there has not been a measurable acceleration in the eradication of poverty in the developing world. This picture could change as Asia, with a combined popula tion of almost 3 billion people—half the world’s population—continues to reduce poverty in the coming decade, and to reduce it in a more dramatic manner. But Sub-Saharan Africa would remain a Region of concern in the war on poverty. All the evidence points to a very close correlation between poverty reduction and growth in income per capita. But growth remains as elusive as it was in the 1980s. The most dramatic reduction in poverty has occurred in countries with very rapid growth in income per capita. Poverty reduction is almost a mirror image of this growth. But in more than two-thirds of developing countries, income per capita is growing more slowly than in the developed world. The picture looks even less promising if one looks back to 1980: from 1980 to 2003, more than three-quarters of the developing countries have grown more slowly than the developed world. Convergence of income, except for Asia and some accession countries in Eastern Europe, is only a distant possibility. Poverty has increased sharply in the Africa and Eastern Europe and Central Asia Regions. Broad social indicators show more widespread convergence with the developed world, with the exception of Sub-Saharan Africa. Moreover, almost one-third of the developing countries have shown no increase in GDP per capita since 1980. Once again, the growth picture in Asia looks encouraging, but SubSaharan Africa remains mired in slow growth. In contrast to income and poverty indicators, broad social indicators, such as life expectancy and literacy, show more widespread convergence with those in the developed world, with the exception of Sub-Saharan Africa (figure 1.3). In every other Region there is convergence toward the developed country figures for basic social indicators such as life expectancy and literacy. But, again, SubSaharan Africa stands as an outlier on life expectancy, partly because of the widespread incidence of AIDS in the sub-continent. Institutional quality and governance, which effect people’s lives, especially those of the poor, show some improvements (figure 1.4), but there has been deterioration in several components, including corruption and political instability. There is a perception that other indicators, such as conflict and democratization, are deteriorating—but evidence does not support this proposition. In fact, indicators of conflict show an improvement (less conflict), although they remain at high levels for the poorest countries. Conflict has not declined in the poorest quartile of countries. This supports the proposition that conflict and poverty seem to feed off each other. Indicators of democracy, which improved sharply in the late 1980s and early 1990s, continued to show steady but slower improvements during the past few years (see figure 1.5). To summarize, over the past decade, the record of development is very mixed. Poverty has declined, but not at an accelerated pace, except in East Asia. Growth remains as elusive as it was in the 1980s. Social indicators are improving and show convergence, except on life expectancy in SubSaharan Africa, largely due to HIV-AIDS. Other factors that affect people’s lives— such as governance and conflict—show some improvements, but again the poorest countries do not show progress. The drive for more democratic government, which increased dramatically in the 1980s and early 1990s, continued to show small improvements. There are reasons for hope—but also reasons for concern, especially in Sub-Saharan Africa. PROGRESS ON POVERTY REDUCTION AND IMPROVED LIVING CONDITIONS 3 Institutional quality and governance, which effect people’s lives, especially those of the poor, show some improvements. Conflict has not declined in the poorest quartile of countries. Conflict and poverty feed off each other. 4 IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? Figure 1.3: Income Diverged But Social Indicators Converged Overall (Regional as percentage of developed world) GDP per capita (constant 1995 US$) Life expectancy at birth (total years) Adult literacy (people ages 15 and above) 0 5 10 15 20 25 30 1980 1990 2000 Regions Percentage Percentage Percentage 50 60 70 80 90 100 1980 1990 2000 Regions 0 20 40 60 80 100 1980 1990 2000 Regions EAP ECA LAC MNA SAR SSA Source:SIMA–April 2005. Developed world = high income, OECD countries. Note:EAP = East Asia and the Pacific; ECA - Europe and Central Asia; LAC = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia; SSA = Sub-Saharan Africa. PROGRESS ON POVERTY REDUCTION AND IMPROVED LIVING CONDITIONS 5 Figure 1.4: Policy and Institutional Improvements Figure 1.5: Democracy Advanced and Conflict Declined in the Past Decade 2.8 2.9 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 Economic management Structural policies Policies for social inclusion/equity Public sector management and institutions CPIA ranking 2003 1999 Source:2003 Annual Review of Development Effectiveness(OED 2004a). Note:CPIA = Country Policy an Institutional Assessment. 100 80 60 40 20 0 1,000 800 600 400 200 0 1950 1960 1970 1980 1990 2000 1950 1960 1970 1980 1990 2000 Trends by armed conflict type, 1946–2004 Episodes of major armed conflict Units of democracy and autocracy Global democracy and autocracy, 1946–2004 Inter-state war Revolutionary war Ethnic war Democracy Autocracy Source: Center for Systemic Peace. Note: The figure on the right sums Polity IV scores of institutional authority for democracy and autocracy for each independent state for each year. Each country is given annual scores (10-point scales) on each of two basic types of regime authority. Although the two types are opposing, many countries exhibit mixed authority traits (that is, they have middling values on each scale). The figure shows global changes in total “units of democracy” in contrast to total “units of autocracy” in the global system. 7 The Bank’s Record in Supporting Development Given the World Bank’s enhanced profile in the fight against poverty in the past decade, how effective has it been in helping countries eradicate poverty? Critics of the Bank from one side argue that much of what the Bank does is immiserization. They say that countries that have followed advice from the World Bank—as in Sub-Saharan Africa—have not shown much progress. Where success has been achieved in the fight against poverty, as in Asia, it is largely the result of home-grown policies and programs, which often go against the advice offered by international agencies. Critics from another side argue that by working largely with ineffective and corrupt governments, the Bank not only has very little to show for its efforts, but has also helped prop up these governments and perpetuated poverty. To answer these criticisms, it is useful to look objectively at the Bank’s record over the past decade. In this section we look at the Bank’s performance at various levels of aggregation—at the project level, at the country level, and at the global level. What results are attributable to support from the Bank? Which programs and projects supported by the Bank have shown good results? What processes and initiatives show promise to build on for the future? Project Performance Ratings Show Improvements over the Past 10 Years1 Outcome Since fiscal year 2000, project performance as measured by outcome has shown continued improvement, except in fiscal 2003: outcome ratings dropped from 78 percent to 75 percent satisfactory between fiscal 2002 and fiscal 2003, as illustrated in figure 2.1. A partial cohort for fiscal 2004 shows a resumption in the improvement in outcome ratings. This represents a recovery from the fiscal 2003 drop, and may be indicative of a resumption of the upward trend 2 2 Project performance ratings show improvements over the past 10 years, and project outcomes improved in all but one Region. in project performance that started in fiscal 2000. Regional Performance Figure 2.2 presents the percentage of satisfactory project outcomes, weighted by disbursement, for the fiscal 2000–04 (partial) cohort compared with the fiscal 1995–99 cohort. The East Asia and South Asia Regions are the top performers for the fiscal 2000–04 (partial) cohort, exceeding the Bank average of 80 percent. This is not surprising, given the overall gains in growth and poverty reduction in those Regions. The Latin America Region, which was the best 8 IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? Figure 2.1: The Outcomes of Bank-Supported Projects Have Been Improving 20 40 60 80 100 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004* Exit fiscal year Percentage satisfactory outcome By project Weighted by disbursement Source:World Bank database, 2005. Note:2004* Partial (dashed line). Figure 2.2: Project Outcomes Improved in All But One Region 40 50 60 70 80 90 100 AFR ECA LAC Bankwide MNA SAR EAP Percent satisfactory outcome (weighted by disbursement) FY95–99 FY00–04* Distribution of disbursements by Region (FY95–04*) MNA 6% LAC 26% ECA 16% EAP 24% AFR 13% SAR 15% Source: World Bank database, 2005. Note:2004* Partial. EAP = East Asia and Pacific, SAR = South Asia, MNA = Middle East and North Africa, LAC = Latin America and Caribbean, ECA = Europe and Central Asia, AFR = SubSaharan Africa. The majority of the sectors show improvement in project outcomes. THE BANK’S RECORD IN SUPPORTING DEVELOPMENT 9 performer of the fiscal 1995–99 cohort, is the only Region that declined in performance for the fiscal 2000–04 (partial) cohort, partly reflecting the overall performance in the Region. Sectoral Performance Compared with fiscal 1995–99 exits, the outcome performance weighted by disbursement for the fiscal 2000–04 (partial) exits improved for 7 of 13 sector boards.2 Figure 2.3 presents the sector boards’ outcomes in order of their performance. The biggest gains in outcome ratings were in the water supply and sanitation and financial sectors. The greatest declines in performance were in the social protection and economic policy sectors. Outcomes for economic policy, energy and mining, environment, and water supply and sanitation remain below the Bankwide average for fiscal 2000–04 (partial). The overall improvement in lending outcome indicators was not created by a shift in the composition of lending toward adjustment loans—both adjustment and investment lending show improved indicators. While the improving trends in project performance are noteworthy and creditable, they offer no ground for complacency. They do not always show impacts and final results. Impact evaluations offer one solution to assessing better the results of Bank support and are now being initiated more vigorously by the Bank. But they also offer only a partial picture, and in any case would cover only a part of the Bank’s activities.3 Country Assistance Evaluations In a third of the country programs evaluated for the period 1990-2003, aggregate project outcomes were satisfactory, but the overall country assistance programs were unsuccessful (table 2.1). Evaluation at the country level yields a more complete picture of the outcome of the Bank’s assistance programs than do evaluations of individual programs or projects. Evaluation needs to capture critical dimensions of country assistance programs, such as whether project interventions are addressing key development constraints. The Country Assistance Evaluation (CAE) is a comprehensive evaluation of the Bank’s program in a country that includes both projects and analytical and advisory services. Since CAEs assess the overall Bank strategy, including the size, sectoral composition, and type of lending, the country assistance outcome may thus be unsatisfactory if, for example, there are critical omissions in the Bank’s overall assistance strategy, even if the outcomes of individual projects are rated satisfactory. The Bank can have a satisfactory program, even in countries that are not doing well. Figure 2.4 presents the outcome ratings for 21 countries that were classified into three groups: good performers, transition countries, and modest and poor performers. GDP per capita grew rapidly in the first group of six good-performing countries (Chile, China, Dominican Republic, India, Tunisia, and Vietnam) from 1990 to 2003, ranging from 3.1 percent per annum in Tunisia to 8.1 percent in China. This led to substantial reductions in the Country project Country outcome ratings portfolio outcome Satisfactory Unsatisfactory Satisfactory 53 33 Unsatisfactory 7 7 Source:OED databases and staff estimates. Table 2.1: Country and Project Outcome Ratings (percent) In a third of country programs evaluated for the period 1990-2003, aggregate project outcomes were satisfactory, but the overall country assistance programs were unsuccessful. The Bank can have a satisfactory program, even in countries that are not doing well. 10 IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? Figure 2.3: The Majority of Sectors Show Improvement 40 50 60 70 80 90 100 EMT WSS Bankwide ED RDV FSP PSG TR Percent satisfactory outcome (weighted by disbursements) Improvers Disbursements by Sector Board (FY95–04*) ENV 2% HNP 5% RDV 12% TR 11% WS 4% PSD 3% UD 4% SP 5% PSG 6% FSP 12% EP 14% ED 8% EMT ED = Education EP = Economic policy EMT = Energy and mining ENV = Environment FSP = Financial sector HNP = Health, nutrition, and population PSD = Private sector development PSG = Public sector governance RDV = Rural sector SP = Social protection TR = Transport UD = Urban development WSS = Water supply and sanitation 14% 40 50 60 70 80 90 100 EP ENV HNP UD SP Bankwide PSD Percent satisfactory outcome (weighted by disbursements) FY95–99 FY00–04* Decliners Source:World Bank database, 2005. Note:2004* Partial. The sector classification provides a single rating for the entire project rather than rating individual project components that cover particular sectors or themes. The sector board classification applies to the whole project and enables outcomes to be matched to it. THE BANK’S RECORD IN SUPPORTING DEVELOPMENT 11 Figure 2.4: Almost One-Third of Country Programs Show Unsatisfactory Outcomes 0 1 2 3 4 5 6 Chile China Tunisia Vietnam Dominican Rep. Armenia Lithuania Mongolia Bulgaria Russian Fed. Croatia Moldova Brazil Jordan Guatemala Peru Lesotho Morocco Zambia Zimbabwe India Good performers Transition countries Modest and poor performersHighly satisfactory Highly unsatisfactory Moderately satisfactory Moderately unsatisfactory Satisfactory Unsatisfactory Source:2004 Annual Review of Development Effectiveness(OED 2005a), Appendix C, table C.1. percentage of the population living below the poverty line (income poverty), from 35 percent in 1990 to 16 percent in 2001. But even in good performers, the outcome of the Bank’s program could be less than fully satisfactory because of a few ill-chosen interventions, as in India and the Dominican Republic. The seven countries in the second group experienced negative growth in the early stages of their transition, starting in 1990. This was followed by a slow recovery, starting in 1993 in Armenia, Croatia, and Mongolia; 1994 in Lithuania; 1997 in Bulgaria; 1998 in Russia; and 1999 in Moldova. All the transition countries experienced substantial increases in poverty during their periods of recession, as well as considerable increases in inequality and deteriorating social indicators. While rapid economic growth has brought about a decline in income poverty in these countries in recent times, social indicators have lagged. The Bank’s programs were more successful in Armenia and Lithuania than in Croatia and Moldova. The third group, of eight modest and poorperforming countries (Brazil, Guatemala, Jordan, Lesotho, Morocco, Peru, Zambia, and Zimbabwe), experienced either negative or low growth in GDP per capita for the period 1990–2003 and showed no improvements in income poverty. But even in this category of countries, the Bank’s program could have successful outcomes, as in Brazil, if interventions were chosen carefully. In Brazil the Bank was successful in supporting fiscal reforms and in regional programs in Northeast Brazil, which lags behind other parts of the country. There is no obvious correlation between The Bank has had successful outcomes in health and education, but efforts in private sector development, public sector management, and rural development have been less successful overall. successful country assistance ratings and decentralization of Bank teams, although strong field presence is cited as a contributory factor to program success. Field presence, pari passu, should improve country knowledge and adaptation to local conditions. Greater field presence is also welcomed by most Bank clients. But a successful country assistance strategy requires relevance and the astute use of the Bank’s skills in a specific set of country issues where Bank support is needed. The Bank has had successful outcomes in health and education. But in private sector development, public sector management, and rural development, the Bank’s efforts have been less successful overall (table 2.2). These results are also largely reflected in aggregate project ratings for the same set of countries (table 2.2). In private sector development (PSD), rural development, and public sector management (PSM), despite the high proportion of satisfactory outcomes at the project level, the Bank’s program has had poor outcomes in the sector as a whole because: • Reforms in these sectors seem to face stronger opposition from vested interests and potential losers. • Institutional capacity constraints take time to resolve and many projects in these sectors attempt to improve upon the legal, institutional, and regulatory framework, but to implement change, inertia and adverse incentives in the bureaucracies of many countries must be overcome. • These sectors also suffer larger adverse effects from exogenous events and macroeconomic shocks. Improving outcomes in all sectors would imply focusing more on measuring and supporting results-based indicators. What lessons can we draw for improving the formulation and implementation of the Bank’s country assistance strategies? First, a key component of successful country programs is that they are tailored to the country context, and a sound understanding of the political economy of reform is essential. Domestic politics and vested interests largely determine the pace and content of reform. An insufficient understanding of the political economy of 12 IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? CAE sector outcome ratings (% satisfactory)a Project outcomes Satisfactory Unsatisfactory Mixed Not assessed (% satisfactory)b Social protection 12 24 20 44 37.9 Rural 12 36 28 24 87.2 Private sector development 12 24 44 20 60.1 Environment 16 16 24 44 25.5 Financial sector 24 16 32 28 81.6 Public sector 24 32 28 16 99.9 Infrastructure 36 16 24 24 87.7 Health 40 20 8 32 86.5 Education 48 8 4 40 100.0 Source: World Bank data and OED staff estimates. a. These columns are ratings by CAE task managers and a desk review of CAEs of the outcomes by sectors. In most cases, explicit ratings by sectors were not included in the CAEs, and thus these ratings should be treated as indicative only. b. This is the percentage of satisfactory project outcomes by sector (commitment value) for the countries where CAEs were completed during FY01–03. Table 2.2: Sectoral and Project Ratings for Fiscal Year 2001–03 CAEs Successful country programs are tailored to the country context, and a sound understanding of the political economy of reform is essential. reform led the Bank to push reforms that stood little chance of success. More active dialogue with parliaments, local governments, and stakeholders enhances the Bank’s understanding of political economy considerations. Second, country knowledge is strongly associated with success. It is well understood that project interventions are more successful when they are based on in-depth analytical work. The same finding conveys to the success or failure of country programs. The OED CAE Retrospective (OED 2005b) found that in more than two-thirds of successful country programs, the Bank’s analytical work was timely. Analytical and advisory activities (AAA) can also be an effective vehicle for engaging governments in policy dialogue given to dissemination. In many cases, dissemination has been inadequate. Poor dissemination can also be a feature of participatory analytical work, as findings still may not be widely shared outside the government ministries that collaborated in the analytical work with the Bank. Third, technical assistance and investment loans can help promote institutional development and capacity building. But the sustainability of benefits hinges on the incorporation of these operations within a broader macro stabilization and reform program. Linking technical assistance and investment loans with policy reforms supported by adjustment loans also improves the probability of success. Fourth, adjustment lending can be successful, especially when combined with a strong government commitment to macro stabilization and structural reform. But adjustment lending in the absence of sustained progress on stabilization and reform saddles the country with debt and weakens the incentive to reform. The Bank needs to resist pressures to persist with adjustment lending in the absence of government commitment to, and a satisfactory track record in, implementing reforms. There are, however, few cases when the Bank resisted such pressures. In many countries, giving in to the pressure to lend produced a number of unsatisfactory outcomes. More important, it weakened the incentive to reform. Finally, there are several specific actions that would serve to improve the outcomes of Bank assistance programs: • Carry out more robust risk analysisto carefully assess borrower implementation capacity and commitment to reform. This work should be informed by the analysis done by others and by the feedback obtained through wider dialogue with stakeholders. • Reduce the level of planned assistance when faced with clear evidence of policy slippage. The Bank needs to set clear and meaningful triggers for its assistance programs and, when confronted with slippage, cut back its level of assistance. • Lend more prudently in turnaround situations. This is especially important in the presence of long-standing issues of implementation failure. Initial levels of assistance should be prudent and calibrated to measurable outcomes and to meeting concrete benchmarks. The outcomes of Bank lending are better when lending goes up the ladder with the reform program, rather than ahead of it. In summary, optimistic projections or expectations accompanied by inadequate risk analysis often weaken the performance of country strategies. Programs should be based on the most likely—not the best possible— forecasted outcomes. And, finally, country THE BANK’S RECORD IN SUPPORTING DEVELOPMENT 13 The Bank needs to resist pressures to persist with adjustment lending in the absence of government commitment to, and a satisfactory track record in, implementing reforms. The Bank is struggling with its model for development effectiveness in LICUS countries and MICs. strategies need to be flexible, not rigid and narrow, with only one path to follow. Two Special Categories of Country Programs There are at least two categories of countries where the Bank is struggling with its model for development effectiveness: low-income countries under stress and middle-income countries. OED is beginning work on LICUS countries, and it is too early to draw any lessons for that group. But OED has reviewed its CAEs for a subset of post-conflict countries (some of these are also LICUS) and has an evaluation of LICUS under way. Post-Conflict Countries Since 1980, over 50 countries have experienced significant periods of conflict, with severe and lasting effects through destruction of physical assets, disruption in trade links, displacement of people, and loss of income and life (including genocide). The World Bank is engaged in 38 post-conflict countries and areas. A fifth of Bank commitments were devoted to those countries in fiscal 2004. When Bank commitments to Bosnia and Herzegovina are compared with the Bank’s response in other post-conflict countries, the exceptional nature of the Bosnia and Herzegovina case stands out starkly (figure 2.5). Four Lessons Learned Early Engagement in Post-Conflict Situations The need for early engagement is strongly indicated by Bank experience. Time is of the essence in post-conflict situations. Often, there are windows of opportunity for significant progress, but these opportunities can quickly pass. In general, the Bank has responded quickly and flexibly in initiating contact and preparing or participating in the preparation of strategy and funding packages. Many factors constrain the Bank’s early engagement in post-conflict situations, including the presence of “stateless societies.” Outstanding debt and arrears to the Bank constitute an additional impediment to 14 IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? Figure 2.5: Post-Conflict Countries Received Very Different Levels of Bank Support per Capita 0 5 10 15 20 25 30 35 40 45 50 Bosnia and Herzegovina (1996–99) Timor Leste (2000–03) West Bank and Gaza (1994–97) Rwanda (1994–97) Eritrea (1994–97) Cambodia (1992–95) Per capita annual average Source:World Bank Business database. Early engagement and a strong and continuous field presence improve performance. the Bank’s early engagement. Since access to resources from the Bank and other international financial institutions (IFIs) is important for most post-conflict countries’ medium- and longer-term development, planning financial normalization is crucial. Strong and Continuous Field Presence The importance of a strong and continuous field presence is the second lesson to emerge from Bank experience. Post-conflict situations are complicated and involve a multitude of players and organizations. Field presence is essential to monitor Bank programs and projects, to maintain coordination with other donors, and to respond quickly and flexibly to changes. Bank field missions need to be strengthened to meet post-conflict roles. An expansion of field presence and the devolution of authority to managers in the field has proved to be a critical aspect of Bank programs in the West Bank and Gaza and in Bosnia and Herzegovina, and should be a primary consideration for other countries where local dynamics are similarly complicated and in flux. In contrast, the absence of a strong field presence reduced Bank effectiveness in a number of reconstruction programs. The lack of Bank representation in Haiti reduced the ability of the Bank to work with key donors, such as the Inter-American Development Bank (IDB) and the U.S. Agency for International Development (USAID), which were more decentralized than the Bank, as well as with government and civil society. And in Rwanda, the Bank’s working relationship was mixed in the aftermath of the genocide because a fully staffed Bank country office was not in place until two years later. Adapting Bank Services and Products to Post-Conflict Situations The uncertainties inherent in post-conflict situations suggest that a different approach is required, as it is not always feasible for the Bank to define an overall Country Assistance Strategy (CAS) a priori. Instead, an opportunistic approach is needed that builds on what is feasible. In countries where conditions do not warrant a full CAS, the Bank has recently adapted this approach and has prepared Transitional Support Strategies (TSS) for nine countries, including Afghanistan (2003), Angola (2003), Burundi (2002), Democratic Republic of Congo (2004), and Eritrea (2000), consistent with the Bank’s post-conflict policy. In a post-conflict setting, in particular, where there is a considerable uncertainty, assistance programs should not be overly ambitious and their objectives should include capacity building. The Rwanda CAE recommends simple project design and allocation of sufficient Bank resources to ensure close project supervision. The recommendation is based on the CAE finding that project lending regularly failed for two reasons: overly ambitious design and inadequate supervision, given the country need. Effective Aid Coordination and Partnerships with Other Donors Effective aid coordination in a reconstruction program improves Bank responsiveness to the realities of post-conflict situations. At the request of international donors, the Bank has played a key role in coordinating international aid in the West Bank and Gaza, Bosnia and Herzegovina, and other transitional situations through its Holst Fund (US$270 million pledged by 26 donors) and other mechanisms. In contrast, had the Bank played a more formal role in the negotiations leading up to the El Salvador Peace Accords of January 1992, a clearer, more realistic picture of the costs of its provisions might have emerged earlier. Partnerships with the U.N. and other donor agencies need to be forged as soon as possible, because they play a critical role in delivering aid and services in post-conflict situations. However, THE BANK’S RECORD IN SUPPORTING DEVELOPMENT 15 In a post-conflict setting, assistance programs should include capacity building and not be overly ambitious. In a reconstruction program, effective aid coordination and partnerships with other donors are vital. high-level agreement should be reached early on the respective roles of the main players, and especially on who will have the lead role in each sector. Middle-Income Countries The Bank has played a useful role in MICs, but it has struggled to be relevant in these countries. For a variety of reasons, the Bank has seen its role in the MICs diminish in recent years, and as a result, IBRD (International Bank for Reconstruction and Development) lending has stagnated. The stagnation in IBRD lending to MICs and the increase in the costs of doing business with the Bank have raised questions about its overall efficiency. Disbursements per dollar of administrative budget have declined from $13 in fiscal 1995 to $9 in fiscal 2005—except for a brief period in fiscal 1998 and fiscal 1999 due to fast-disbursing loans to help countries deal with the Asian crisis. Commitments as a ratio of the administrative budget also show a sharp decline (figure 2.6). The average size of Bank loans has declined since the mid-1990s from around $90 million in the mid-1990s to around $80 million in the last two years.4 The scaling-up of interventions is not occurring and internal incentives—including the matrix structure— have led to proliferation of activities. Recent OED evaluations of MICs show that the Bank continues to have an important role in MICs, but that it needs to focus on the following issues. Helping the MICs Through Their Remaining Development Challenges The Bank has played a positive role in helping MICs through crisis—both as a contra-cyclical financier and in helping countries introduce better regulatory and institutional structures and mitigate the social costs of crisis. A more systematic and articulated role for the Bank in these crisis situations would be useful. The Bank played an important role in a number of countries, such as Turkey and Tunisia, but appeared to lack coordination in some crisis situations, as in Argentina and Indonesia. Many 16 IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? Figure 2.6: Bank Commitment and Disbursement as a Ratio to Total Budget Are Declining 0.0 5.0 10.0 15.0 20.0 25.0 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 Fiscal year Ratio to budget Disbursement Commitment Source:World Bank database. Datasets for administrative budget, all project commitments, and disbursements by fiscal year for active and inactive projects–July 2005. The stagnation in IBRD lending to MICs and increased costs of doing business with the Bank raise questions about its overall efficiency. MICs continue to find Bank support on a range of development issues—such as the environment, pension reform, banking and energy regulation, social assistance, and subnational institution building—very useful. Dealing with Regional Inequality and Pockets of Poverty With a few exceptions,5 the MICs do not have large-scale problems of extreme poverty. Instead, they typically have pockets of poverty and identifiable regions that are less developed and poorer than other areas of the country. Where the Bank has consciously focused on these issues, it has played a constructive role, as in Brazil. There is greater demand from MICs for Bank support in their less-developed regions than in better-off areas. Ironically, the Bank’s sharper poverty focus has made its role in some MICs less relevant, as they do not identify themselves as poor countries and are therefore reluctant to engage with the Bank. Transferring Knowledge and Ideas Most MICs still regard the Bank as a key source of knowledge and ideas and as a neutral source of policy advice. But several MICs have expressed a desire for the Bank to provide less generic knowledge and more help in finding solutions to their specific problems—more custom and country fit. Many MICs with deeper local capacity would also like Bank teams to work more closely in partnership with local institutions. Working with MICs also helps the Bank gain ideas, which can be transferred to low-income countries. Costs of Doing Business with the Bank MIC lending has declined at the World Bank, although not at the regional multilateral development banks (MDBs). This has raised the issue of the increased costs—both financial and nonfinancial—of doing business with the Bank. Enhanced safeguard and fiduciary policies are meant to benefit countries that borrow, but some of these policies may be perceived as an external imposition, or be applied in a rigid manner, without a full understanding of how they may work in a particular context. With MICs’ ability to access other funding sources, countries that are not convinced of the benefits of such Bank policies can proceed with projects where these policies have become an issue using other funds. As with all other policy matters, countries must be convinced of the utility of safeguards, not have them imposed by outsiders. MICs’ Global Role The MICs’ increasingly important and expanding role on a range of global development issues makes them key players in both the achievement of global collective action and in enhancing the links between the Bank’s country and global programs. The larger MICs, such as Brazil, China, and India, are increasingly vocal regarding a wide range of global institutions. The Bank needs to develop a strategy to ensure that the voice of these MICs influences Bank global programs and is internalized within country programs. Global Programs Global Programs Are an Important and Growing Line of Business for the World Bank The Bank manages by far the largest stock of trust funds among international organizations—$7.1 billion at the end of fiscal 2004 (not including Heavily Indebted Poor Country [HIPC] and International Finance Corporation [IFC] trust funds). Sixty-four percent of this sum was committed to global and Regional programs. OED has completed a series of evaluations of these programs over the past 4 years and has presented the following findings to improve their effectiveness. THE BANK’S RECORD IN SUPPORTING DEVELOPMENT 17 Dealing with regional inequality and pockets of poverty is a key agenda in MICs. There is a strong demand in MICs for the Bank’s knowledge and ideas. The Bank should give more importance to the MICs’ global role. The Bank’s strategy for global programs is poorly defined. The Bank has lacked, but clearly needs, a global strategy that is developed in conjunction with its key partners and draws on the capacity that exists in its central vice presidencies, network anchors, and Regions to do so. The strategy needs to address the coherence, or lack thereof, between global expectations (particularly in the donor community) and the needs of developing countries. At its center, the global strategy needs a clear focus on sustainable, poverty-reducing growth in the Bank’s client countries; on global policy issues that prevent such growth; and on mobilizing incremental, unrestricted funding to address global issues that are high-priority for developing countries. Such a strategy will not simply emerge from improved selectivity or oversight of individual global programs—it must be worked out. Furthermore, strengthening oversight in the absence of an overall strategy risks micro-managing the global program portfolio. The current criteria for Bank involvement in a global program are too general. This is largely because the criteria are broad and difficult to apply precisely. For example, the first criterion— “an international consensus that global action is required,” which all programs claim as their reason for being—provides no basis for selectivity, because the concept of international consensus is amorphous and loosely applied. While largely supplydriven, most Bank-supported global public goods programs (Multilateral Fund for the Implementation of the Montreal Protocol [MLF], Global Environment Facility [GEF], ProCarbFund, Critical Ecosystems Partnership Fund [CEPF], Consultative Group on International Agriculture Research [CGIAR], Special Program for Research and Training in Tropical Diseases [TDR], Joint United Nations Program on HIV/AIDS [UNAIDS], Stop TB, Roll Back Malaria, Global Forum for Health Research, and Global Alliance for Vaccines and Immunization [GAVI] Global Research Funding) largely meet the four Development Committee criteria for selectivity. Most global programs also largely meet the approval and eligibility criteria for Bank involvement. CGIAR does not meet the arm’s-length criterion; the Bank did not involve developing country stakeholders in CEPF’s establishment or its global-level governance; and the Bank did not do a thorough analysis of the expected level of Bank resources required for the health programs, or of how to implement and manage this new commitment. It is time to move from “letting a thousand flowers bloom” to assessing which programs deserve continuing Bank support, and which do not. Their Strengths Only a few global programs deliver global public goods (table 2.3). Global public goods programs (CGIAR, TDR, MLF, parts of GEF, and even some new global-health programs) rate well in their impacts on reducing poverty or on focusing on the policy, institutional, infrastructure, or technological constraints faced by developing countries in achieving sustainable economic growth. Adding value on the ground in client countries is typically a joint product of global and country-level activities. For example, CGIAR, like TDR, has demonstrated impressive poverty-reducing impacts, in part because the Bank, donors, and some governments made complementary investments at the country level. However, as country-level investments have shrunk, donors have tried to compensate by encouraging CGIAR to move downstream. They have offered restricted funding tied to research programs that demonstrate immediate impacts to push CGIAR toward more national- and local-level applied and adaptive work, and away from its core competency of research. Global programs have revealed major gaps in investment. Evidence indicates that investments in health research have substantial 18 IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? The Bank’s strategy for global programs is poorly defined, and criteria for Bank involvement are too general. Only a few global programs deliver global public goods. poverty-reducing impacts. The current global policy and aid environment has huge investment gaps at the global level in the provision of global health research, as well as gaps in complementary investments at the country level. Health research, like agricultural research, is a long-term activity that is unlikely to be addressed by the private sector on the scale needed. Global programs have also revealed gaps in global public policy. Several global programs highlight global public-policy gaps—often involving industrial-country policies in trade, aid, finance, and intellectual-property rights— that affect developing countries. Few programs regard it as within their mandate to address these policy gaps. If changing the international ground rules is the objective of the programs, and if advocacy is the means to achieve it, then the programs should be assessed on their ability to deliver changed policies or a changed global environment from the perspective of the poor. Their Weaknesses The voices of developing countries, as well as those of the Bank’s operational Regions, are inadequately represented in the international consensus. The case studies of corporate advocacy programs show that including developing country voices at the concept stage enhances program ownership, makes the organizational design more effective, and increases program impacts. Based on the evidence OED has provided so far, management has acknowledged the need to strengthen the role of developing countries and the Bank’s operational Regions in global programs. Governance is weak in several programs. While pure shareholder models of program governance are being replaced by stakeholder models, programs are THE BANK’S RECORD IN SUPPORTING DEVELOPMENT 19 Fostering Financing Financing Financing country-level country-level Strengthening Number global country-level approaches, technical country-level of investments investments standards assistancea capacity programs Delivering CGIAR (1972), GEF (1991), UNAIDS (1996), 0 global public TDR (1975) MLF (1991), RBM (1998), goods ProCarbFund (2000), Global Forum (1997), CEPF (2000) Stop TB (1999)c Delivering PostConFund (1998), CGAP (1995), WSP (1978), InfoDev (1995), 16 national GAVI (1999)b GIF (1996), ESMAP (1982), World Links (1998), public goods GWP (1997), IF (1997), GDN (1999), UCW (2000) PPIAF 1999), FSAP (1999), Cities Alliance (1999) FIRST (2002) Number of programs 2 6 8 5 5 26 Source: OED Global Programs Evaluation (OED 2004c). Note:Each program is classified by OED according to only one category, corresponding to its primary activity. Programs are listed chronologically by start date within each category. a. .With the intent of stimulating public or private investments in the sector. b. .The GAVI Vaccine Fund also finances research and development of new vaccines and promotes strategies to address the constraints to R&D investment. c.. Stop TB also has a small drug facility that is financing country-level investments in the form of drugs for the treatment of tuberculosis. Table 2.3: Only a Few Global Programs Address Global Public Goods The voices of both developing countries and the Bank’s operational Regions are inadequately represented in the international consensus. Global programs have revealed major gaps in investment and global public policy. still struggling to balance legitimacy and accountability for results with efficiency in achieving them. The permanent members of the programs’ governing bodies, who tend typically to be the major international organizations and donors, have greater de facto responsibility, relative to the rotating members, to ensure that programs are successful. But such responsibility and accountability are rarely clearly articulated. Lack of effective governance and management constitutes a risk for the Bank. Global programs have increased overall aid very little. At the aggregate level, the global programs reviewed have added little new money to official development assistance. Exceptions include funds from private sources for the Prototype Carbon Fund, from the Gates Foundation for health, and small amounts from pharmaceutical companies through new public-private partnerships for drug and vaccine development. Given the opportunity cost of ODA funds, the Bank’s involvement in programs with important goals, but little demonstrated value, should be reconsidered. In some cases, too close an association with the Bank has hampered mobilization of other funds for these programs. The Bank deploys its comparative advantages more at the global level than at the country level. The international consensus on the existence of a global problem is usually strong; consensus on what collective action is required is often weak. Many global programs are implicitly (sometimes explicitly) established to promote consensus, to “harmonize” donor approaches to specific problems, to delineate donor comparative advantages in addressing those problems, and to give the donors specialized knowledge they can use to deal with the problems. Capacity building in the recipient countries is secondary in such projects. There is very weak connection in strategy or implementation between the Bank’s global and country programs. 20 IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? Governance is weak in several programs. Global programs have increased overall aid very little. 21 Crosscutting Lessons from Sector and Thematic Evaluations Nine crosscutting lessons emerge from recent evaluations—including the sector and thematic studies—that suggest directions to further improve the Bank’s development effectiveness. Specific sectoral lessons are also available, but are not highlighted here. 1. Understand and analyze comprehensively, but act more selectively • A comprehensive reform strategy is needed in various sectors to produce results in a specific area. In real life, no developing country has the capacity to implement all the needed reforms at the same time. • The weak capacity of central and regional agencies makes it difficult to implement government-wide results-orientation. • The Bank and clients need to be more selective about the poverty indicators they choose to monitor under the PRSP. • Carefully define governance requirements for lending in countries with poor governance. Be selective. • Global programs have weak selection criteria. Several programs do not address a global public-good need, and poverty eradication, the Bank’s main mission, does not factor into program selection. Some, such as CGIAR, do meet these criteria, but many new ones do not. 2. Emphasize institutional development and capacity building even more strongly, but sequencing and prioritization are necessary. • Institutional quality is especially important for public sector reforms and private sector development. Major institutional aims include the creation of competitive systems, appropriate incentives for key stakeholders, appropriate regulatory and supervisory frameworks, transparency, and accountability. • The record of attempts to unbundle and privatize infrastructure has been poor (especially in electricity and water), unless it was accompanied by strong institutional and regulatory frameworks. Much the same is true of financial sector reforms: strong prudential regulations and supervisory systems are required for 3 3 Selectivity is vital. No developing country has the capacity to implement all the needed reforms. success. The record of privatizing poorly run government corporations has been very mixed, and the record of voucher privatization has been the worst of all. Rushing to privatize without adequate regulatory, prudential, and incentive systems is a recipe for failure and for serious political and social consequences. • In social sectors such as education and health, the required reforms are not privatization, so much as better service delivery. This, in turn, requires institutional changes to empower users of services, and thus change the incentives for service providers. The institutional changes need to be tailored to individual country settings. •Some promising approaches are threatened by unsustainability arising from an absence of strong institutions when a project ends. Social funds, environmental management, forestry, and natural disaster management are examples. Lack of sustainability is an issue in CDD, urban reforms, and service delivery in general. • Decentralization can create permanent local government institutions that are close to the grassroots and can improve accountability and service delivery. However, political decentralization (creation of local councils) needs to be accompanied by administrative decentralization (right of local councils to hire and discipline staff) and fiscal decentralization (giving local governments tax powers and a share of central government revenues through a revenuesharing formula). This will give decentralized institutions the wherewithal to deliver results. •The Bank has not yet established a knowledge base and guidance for capacity building work comparable to that of its other main work. • Strong institutions are clearly important for rapid growth, but high growth has sometimes been experienced in countries with weak institutions. The Bank needs to do more careful work on the link between institutions and growth and poverty reduction. • It is time to start assessing the impact of the Bank’s work on governance and to provide more specific solutions. 3. Tailor programs to the circumstances of each country and adapt strategies to the local political economy. • While the Bank produces many toolkits and papers disseminating best practices in various sectors, experience shows that what may be best practice in one setting may fare poorly in another. The Bank needs an inventory of good practices, but these should be adapted to the local context. Pilot projects in different regions of a country can test the waters and yield lessons for modifications before scaling up. • Privatization is a good example of an area where processes and procedures were often not adapted to the local context. It succeeded where political commitment and institutions were strong, but not elsewhere. Evaluations reveal a lack of custom-fitting in public sector reform, private sector development, infrastructure, social sectors, and fragile states. • PRSPs are supposed to be designed by countries, to ensure strong ownership. Yet Washington has to sign off on these, which means clients tend to see them as ways of accessing cash. This conflicts with the goal of ownership. • Lending to governments is inescapably political. The best technocratic designs can wither in infertile political soil. OED evaluations show that the Bank needs to pay much more attention to the political economy of countries, assessing the political risks, costs, and benefits as well as the eco 22 IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? The Bank needs to do more careful work on the link between institutions and growth and poverty reduction. The Bank needs an inventory of good practices, but these should be adapted to the local context. Custom-fitting to the political economy is as important as customfitting to the real economy. nomic implications. Custom-fitting to the political economy is as important as custom-fitting to the real economy. • Persuade rather than prescribe. Country ownership of programs is key for success, and conditionalities are no substitute. Yet Bank staff still harbor—and Bank documents still include—the assumption that the Bank has high leverage in aid-dependent countries. This is an illusion: leverage produces lip service, not country ownership. • Local political champions are essential for pushing through reforms. The Bank needs to piggyback on the efforts of local champions rather than press for change through conditionalities. But local champions are not enough: their work needs to be buttressed by new institutions that change the balance of power in the political economy (such as elected local governments in place of a monolithic central government). • Where there are no local champions, and the political economy resists reform, the Bank should reduce lending, and instead engage with a wide variety of stakeholders to create a consensus for reform. This can take time, effort, and patience. • Consensus building for reforms within a country can be slow and gradual, but creates a better basis for reform than radicalism imposed from outside. Often gradualists have achieved better results than radical reformers. Experience shows that the timing, sequencing, and selection of reforms are crucial, and these may be better determined by a country’s political dynamics than by outside advice. 4. Rethink areas of punctured optimism. • The 10-year record shows progress in many areas, but also repeated setbacks arising from persistent over-optimism. • Privatization and structural adjustment have been plagued by over-optimism about political commitment and institutional capacity. • The record shows persistent over-optimism about debt sustainability, export growth, and GNP growth in HIPCs and other fragile states. Over-optimism also characterizes efforts to rehabilitate persons displaced by projects. • The Bank needs to go back to the drawing board to deal with some of these issues. More research and pilot projects may throw light on alternative approaches. • Learning and Innovation Loans (LILs) and Adaptable Program Loans (APLs) designed to accommodate experimentation and innovation as implementation proceeds or to test new ideas have proven much less successful than standard projects. • Provide support in turnaround situations or in countries where the Bank engages after a long absence in a carefully selected manner. Avoid rushing in with large volumes of lending. 5. Lend mainly to countries with improving policies and institutions, but find ways to deal with poor, misgoverned states. • Bank research shows that aid works well in countries with good policies and institutions, and tends to fail elsewhere. With this understanding, clear performance criteria for lending has become an important principle: the Bank seeks to lend more to countries with a higher and improving CPIA ratings. • Lending in post-conflict situations has no clear and systematic criteria and is not linked to poverty. The poor countries in East Africa received much less Bank assistance per capita than less-poor countries in Europe and Central Asia or the Middle East and North Africa. • Several evaluations highlight the risks and moral hazard of lending to fragile states with poor governance. Nonlending strate CROSSCUTTING LESSONS FROM SECTOR AND THEMATIC EVALUATIONS 23 Consensus building can be gradual, but creates a better basis for reform then radicalism imposed from outside. The record shows persistent over-optimism on privatization, debt sustainability, and growth. gies constitute an alternative approach. These include early and continued engagement, building a consensus for reform among a wide range of stakeholders, creating enclaves of improved governance and development through social funds that bypass the normal state structure, adapting procedures and processes for the difficult conditions of fragile states, and improving aid harmonization. • But nonlending strategies do not deal with the moral dilemma of withdrawing assistance from desperately poor, misgoverned countries. One proposed solution is to shift from soft loans to outright grants for weak states. Such a shift, which may blur the line between humanitarian and development assistance, is proposed under IDA 14, but will be evaluated after some time has elapsed. • Standards for lending to HIPCs have been diluted. Many HIPCs have yet to demonstrate the ability to put in place frameworks for reform, which is raising concerns about outcomes and track records. This risks a race to the bottom. 6. Introduce greater flexibility and innovation in its dealing with MICs. • The Bank needs to provide less international best practice and more custom-fit policy advice. • The Bank needs to develop new and innovative instruments for subnational financing. • Old-style Bank lending does not mesh with fiscal constraints in the country, or even its institutional framework. • We have a very good knowledge of local conditions and the local market, but the great asset of the Bank is having people from different places, with knowledge of international and cross-country experiences. • The costs of doing business with the Bank are high and its procedures are very bureaucratic. More intelligent application of safeguards and fiduciary requirements is needed. 7. Make better use of the Bank’s knowledge and technical assistance. • The Bank has the largest operational experience and research output among development agencies. Since the mid-1990s, it has sought to evolve into a “knowledge bank.” OED evaluations suggest that the Bank’s knowledge initiative was timely and appropriate, but was not linked tightly to core lending and nonlending processes. This has limited its impact. • The Bank must establish itself as a preeminent center of excellence in key policy and institutional areas. It needs to be recognized as an authority on good practices, their adaptation to country circumstances, and ways to replace templates with a menu of options bearing different risks. • Bank staff have a wide array of specialist skills, and a huge reservoir of intimate knowledge of the political economy in different countries. Yet this is not used widely enough for custom-fitting. • Such intimate knowledge should be brought more openly into Country Assistance Strategies. A challenge is to share this knowledge of political economies effectively among all Bank staff and with other donors. • OED evaluations highlight the need for the Bank to research and build knowledge on how to deal with fragile states, where traditional approaches have failed widely. • Evaluations suggest that Bank research needs to establish more clearly the links between institutions and growth. It also needs to improve our understanding of when and why reforms occur, and why the timing and sequencing varies so much in different countries. 24 IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? Safeguards and fiduciary requirements are applied mechanically. Bank knowledge and lending activities are not well aligned, and few efforts are made to track the Bank’s impact on poverty. 8. Improve monitoring and evaluation (M&E). • A persistent theme running through several sectors is the inadequacy of M&E. While strong M&E adds to costs, it is essential to throw light on the impact of projects and programs and to suggest remedial steps. • Evaluations in the health and education sectors note that M&E is so weak that little is known of outcomes. In consequence, Bank staff tend to measure performance by inputs rather than by outputs or outcomes. • In the urban sector, few efforts have been made to track the Bank’s impact on poverty, so successes here have not received the publicity they deserve. • At completion, almost 40 percent of lines of credit had no information on repayment rates of Bank-funded subloans. • Environmental projects require Geographical Information Systems to track ecological trends. • Where pilots constitute the first stage of projects, participatory M&E by stakeholders can provide valuable feedback that greatly improves project design. • Data on Millennium Development Goals (MDGs) is not available in a large number of countries. • Despite some advances, M&E activities are still mainly donor-driven and -funded, and not well integrated into normal government operations. 9. Improve coordination within and across the Bank Group. • Links between the global and country programs of the Bank are very weak. • Coordination between the Bank, the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA) on investment climate issues needs improvement. • There are unexploited synergies between the agendas for private sector development and public sector reform. • Pension reform requires close coordination with financial sector work, but pension experts lack access to information. • Closer coordination is needed between the Poverty Reduction and Economic Management Network (PREM) and the Private Sector Development Department (PSDD) on investment climate issues. • Both the Bank and other donors tend to be organized by sector, fostering institutional “silos” that inhibit cross-sector dialogue and planning. CROSSCUTTING LESSONS FROM SECTOR AND THEMATIC EVALUATIONS 25 There are enormous opportunities to further improve coordination within and across the Bank group. 27 Looking Forward The fight against poverty has not yet turned the corner, despite all the effort. The past decade saw some progress on poverty eradication—but at a pace too slow to make a significant dent in global poverty. Better progress was made in Asia, which has the largest share of the world’s poor, but significant increases in poverty in other parts of the world, especially in Sub-Saharan Africa, remain very worrisome. Poverty has also risen in Europe and Central Asia, although this may be a more transitory phenomenon, especially in Eastern Europe. Latin America has also not seen great progress on growth and poverty reduction and is shifting away from the liberal market model it embraced enthusiastically two decades ago. A renewed emphasis on growth is vital for poverty reduction. The World Bank has been dramatically altered over the past decade, with the objective of making it more effective in supporting the global fight to eradicate poverty. It has revamped its over-arching mission and reorganized internally to scale up the focus of its activities from individual projects to country and global programs. Ironically, during this period the success rate of its projects has improved, but its country programs still show considerable scope for improvement. Almost a third of the Bank’s country programs—where overall impact on broader objectives such as growth, poverty reduction, and environmental sustainability are better measured—remain unsuccessful. Its global programs, when focused on global public goods such as agricultural research, show positive results, but others are less useful. The links between global and country programs also remain weak. The Bank is still struggling to find a viable model for working with poor, misgoverned states, and its rising costs of doing business have reduced its attractiveness to the middleincome countries, and this has led to a shrunken role for the Bank in global development finance. This review of the Bank’s development effectiveness points to a number of opportunities for further improvements. Some of these will require changes in business processes, such as new ways of dealing with LICUS and MICs, and 4 4 A renewed emphasis on growth is vital for poverty reduction. further improvements with PRSPs, as well as less mechanistic application of safeguards. Some will require changes in the organization, such as the need to find and put in place organizational structures to improve coordination within the Bank and across the Bank Group to set the right incentives; a consolidation of the Bank’s networks toward the two pillars, the investment climate and social inclusion; and a review of the matrix structure, which leads to multiplication of tasks and works against selectivity. Still others will require strategic choices for the institution, such as how to define its role more selectively, rather than trying to cover every aspect of development. The Bank has transformed itself significantly in the past 10 years, but in the current climate of rapid change, it should be ready for further adjustments. 28 IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? Greater selectivity, more flexibility, and better efficiency—a more hardnosed focus on concrete results—are needed. ANNEXES 31 Introduction Management welcomes this opportunity to comment on the recent synthesis of evaluation findings by the Operations Evaluation Department (OED). The report raises challenging but wellknown questions of both organizational and development effectiveness. Perhaps inevitably, given the nature of the review in trying to highlight areas for further work, it does not document the major achievements in recent years in terms of development in general or of the Bank’s development support. It does not report on the encouraging evidence of strong overall developing country growth and poverty reduction, nor does it highlight the centrality of the country and country ownership to recent successful growth and poverty reduction, the Bank’s leadership in helping to align external assistance around country-owned programs, or the Bank’s success in channeling funds to well-performing countries that can effectively use the resources provided. It downplays the substantial improvement in the development outcomes of Bank-supported operations and better management of Bank program delivery, both of which management saw as necessary before moving, as it has, to further scale up the impact of the Bank’s support and strengthen its results focus. For example, the report fails to mention that while in 1995, one out of three Bank-funded operations yielded unsatisfactory results, today, that ratio is down to one out of five.1 On a portfolio in which projects costing some $40 billion are completed each year, this means that over the last 10 years about $5 billion of projects per year have moved from unsatisfactory to satisfactory status and are delivering on results. Similarly, the OED report does not mention the Bank’s increasing lending commitments and deeper engagement with the neediest countries. Management considers that the Bank’s record in working to improve living conditions in borrower countries has been much solider than OED reports, and it cites evidence in this note. Coverage This note is organized around the three questions OED’s synthesis covers: • What is the measurable progress in improved living conditions in the developing world over the last 10–15 years, and how does it compare with the previous decade? • How effective has the World Bank been in helping countries improve their living conditions through its various interventions and programs? • Finally, what broad lessons emerge from OED evaluations on improving the Bank’s development effectiveness? Progress in Living Conditions OED finds that “the pace of change in the overall performance of the developing world has not altered markedly over the past 20 years.” By painting the picture in this way, OED opens the door for critics to use this review to undermine support for development assistance at a time when every dollar of development assistance is very much needed—and is more likely than any time in recent history to be used effectively. As OED’s Annual Review of Development Effectiveness(ARDE)2 recognized last year, there is a real trend in better policy performance, and that performance is reflected in better growth outcomes in developing countries. On average since 1990, and especially in recent years, developing countries have grown faster in per capita terms than high-income countries. Over the past four ANNEX A: MANAGEMENT COMMENTS years, high-income countries’ income per capita has grown at an average rate of 0.9 percent per year, that of middle-income countries at 2.6 percent, and that of low-income countries at 3.2 percent (see figure A.1). This means that increasing numbers and proportions of people are living in countries with positive growth rates (see table A.1). In Sub-Saharan Africa, a Region of special focus for the development community, per capita income grew at an annual rate of 1.2 percent between 2000–2003 (figure A.2), and 15 countries have had a real per capita growth rate above 2 percent per year since 1995. Median Growth Rates OED’s report also states that “in the 1980s, only about 2/3 of developing countries showed positive per capita income growth, and this percentage remains unchanged.” It is true that since 1980, the median proportion of low- and middle-income countries with positive annual rates of growth in GDP per capita has been 64 percent. But that proportion has risen from a median value of 59 percent between 1980 and 1990 to 72 percent in the period 1991 through 2003 (see figure A.3). Outlook The overall outlook for a decline in income poverty remains promising for the next decade. If current projections for growth hold, worldwide, the poverty headcount index will fall from IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? 32 Figure A.1: GDP per Capita Growth (average annual, %) 1.2 0.3 2.0 2.6 1.2 3.1 1.5 1.7 1.4 2.9 2.6 2.4 3.2 2.6 0.9 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 Low income Middle income Percent High income 1980–85 1985–90 1990–95 1995–2000 2000–03 Number of countries Share of people (percent) Direction 1980s 2002–03 1980s 2002–03 Positive 58 119 78 90 Negative 53 22 22 10 Table A.1: Direction of per Capita Growth Rates Source:Development Economics Department. Source:Development Economics Department. 28 percent in 1990 to 10 percent in 2015, and the number of people living on less than $1 a day will fall from 1.22 billion to 622 million.3 These achievements will largely reflect successes in China and India, which contain most of the world’s poor people, but where income growth has accelerated since 1990 and has remained high. In Europe and Central Asia, where the poverty rate is relatively low, the sharp drop in incomes of the early 1990s has been reversed. ANNEX A: MANAGEMENT COMMENTS 33 Figure A.2: Average Annual GDP per Capita Growth (percent), by Region 6.1 0.0 –0.5 –1.1 6.9 –1.0 1.6 3.6 4.4 –1.2 1.6 1.2 3.3 –1.8 1.0 –0.1 3.6 6.2 –2.0 0.0 2.0 4.0 6.0 8.0 East Asia & Pacific Europe & Central Asia Latin America & Caribbean Middle East & North Africa South Asia Percent Sub-Saharan Africa 1980–90 1990–2000 2000–03 Figure A.3: Proportion of Countries with Positive Annual per Capita Income Growth Rates 0 25 50 75 100 1980 1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 Low-income growth > 0 Middle-income growth > 0 High-income growth > 0 Percent Source:Development Economics Department. Source:Development Economics Department. Social Indicators OED notes that “broad social indicators, such as life expectancy and literacy, show more widespread convergence with those of the developed world, with the exception of Sub-Saharan Africa.” Management would add two additional indicators—under-5 mortality and primary education completion—on which developing countries have also shown encouraging signs of convergence with developed countries. As figure A.4 shows, since 1980 developing countries have consistently reduced the mortality rate of children under age 5 (although it is clear that countries in Sub-Saharan Africa in particular have a long way to go). And as figure A.5 shows, middle-income countries and the countries in the East Asia and Pacific and Latin America and Caribbean Regions are close to universal primary education. Although South Asia and Sub-Saharan Africa have the lowest percentages of school-age children completing primary education, they have made good progress in the last decade to record levels. IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? 34 Figure A.4: Under-Five Mortality Rate in Developing Countries (deaths per thouand) 0 50 100 150 200 250 High income Low income Middle income EAP ECA LAC MNA SAR SSA 1980 1990 1995 2000 2003 0 20 40 60 80 100 Low income Middle income EAP ECA LAC Percent MNA SAR SSA 1990 2002 Figure A.5: Primary Completion Rate (percent), by Region Source:Development Economics Department. Source:Development Economics Department. Continuing Efforts While management is pleased to point out that progress in growth and poverty reduction has been positive in recent years, it is not saying that all is well. The entire development community is united behind the Millennium Development Goals (MDGs). Achieving them will require that developing countries further improve their policy performance, but it will also require more—and even more effective—development assistance. The Bank will continue striving to improve its effectiveness (the subject of the next section); but OED should not ignore its own evidence of the Bank’s effectiveness in recent years, or the Bank’s work to scale up the impact of its assistance by harmonizing its assistance with that of other donors and aligning that assistance with country goals; Bank leadership on results at the country, Bank, and donor community levels; and the work to modernize and simplify Bank operational tools for greater effectiveness. World Bank’s Effectiveness in Helping Countries Improve Living Conditions Management notes that it is difficult for any development institution to truly measure its impact on improving living conditions, since many elements—some of which are not well understood—affect growth and poverty levels.4 As last year’s ARDE pointed out, the Bank has done well in directing its financial support to countries with better policies, which can best use these resources for growth and poverty reduction.5 At the same time, it has also stepped up its support to low-income countries under stress (LICUSs), with the goal of helping them achieve consensus on policy direction and over time improve their policy and institutional performance and gain greater access to financing from IDA and other sources. With regard to the Bank’s effectiveness, management has some concerns about both the methodology OED used to reach its conclusions, and the conclusions themselves. Country Program Ratings OED finds that with one-third of country programs rated unsatisfactory, there is substantial room for improving the Bank’s development effectiveness. Management agrees that development effectiveness at the country level could be further improved, and it has taken a number of steps in that direction. However, management would like to note certain problems with the methodology OED uses. First, OED split the ratings of some of the countries it cites—notably Russia, where OED rated the outcome since 1999 as satisfactory, while rating the 1992 to 1998 period as unsatisfactory. A more fundamental point is that, in spite of long discussions, management and OED have never agreed on the country program rating methodology. A major reason for these disagreements is that, before resultsbased Country Assistance Strategies (CASs) were introduced, country programs were not designed to be evaluated. OED has long rated the achievement of objectives, but objectives and strategies to achieve them change from one CAS to the next in response to changing country circumstances, and it has never been clear to management which objectives OED rated.6 Management has also identified some inconsistencies in the rating system across Country Assistance Evaluations (CAEs). Having expressed this concern, management would note that we are now working constructively with OED to establish a clear understanding on the basis for ratings in the context of results-based CASs, and management expects a higher level of convergence of views in the future. Caveats Management would also like to point out three things a reader should bear in mind in interpreting the country program ratings OED reports. • OED’s country program rating is not a rating of a country’s progress in living standards. OED evaluates a country assistance program on how well it meets its particular objectives, which are typically a subset of the client’s development objectives. Thus the outcome of the Bank’s assistance could be fully satisfactory, yet the country’s economy could be deteriorating. ANNEX A: MANAGEMENT COMMENTS 35 • OED’s rating of a country program is not a rating of the Bank’s performance. OED’s evaluation methodology distinguishes between the performance of a country program and that of the Bank. Therefore, having one-third unsatisfactory outcomes in country programs does not translate into an unsatisfactory Bank performance in one-third of country programs. A country program rating in OED’s methodology is determined by the jointimpact of four agents: the client, the Bank, partners and other stakeholders, and exogenous factors. Thus Bank performance (which OED does not rate) can be good even when OED rates outcome as unsatisfactory. • Divergence between country outcome ratings and outcome ratings of Bank projects is not significant. OED finds that while project outcome ratings are 75 percent satisfactory, outcome ratings are satisfactory in two-thirds (67 percent) of country evaluations. Management would like to note that OED has rated country-level outcomes in four countries that had a heavy concentration of Bank operations in FY04. Country-level outcomes were rated as moderately satisfactory in India and satisfactory/highly satisfactory in Brazil, China, and Vietnam (figure 2.4 in OED’s synthesis report). OED rated project outcomes in these same countries as moderately satisfactory or above. Growth for Poverty Reduction OED’s analysis of growth in this report and the findings of its ARDE 2004 suggest that the Bank has given insufficient emphasis to growth for poverty reduction.7 OED cites countries’ poverty reduction strategies (PRSs) as evidence. While it is true that early PRSs in low-income countries were less focused on broader growth issues than on social services (although, as OED emphasizes, better social services do contribute to growth), more recently low-income countries have placed greater emphasis on sustainable growth in developing their PRSs. Management also wants to make the point that, while the Bank supports countries in producing PRSs, they are not Bank documents. Centrality of Growth In the mid-1990s, while the IMF worked with clients on the macroeconomic front, the Bank took a leading role in helping structure the privatization of utilities and promote private participation in other infrastructure as a way to promote growth, given the hige infrastructure gaps and the shortage of public funds (notably development assistance). In the late 1990s, the Bank was clearer in emphasizing that while privatization could contribute to growth in many countries, it needed to respond to local conditions; the key was not necessarily privatization but private sector development. The Bank also emphasized that macroeconomic stabilization, while fundamental, could not in itself create thriving economic growth. Entrepreneurs needed to be free to set up new firms—which meant appropriate business regulation. Thus, the Bank broadened its assistance to helping countries level the playing field for responsible business development. The joint World Bank–IFC Doing Business project has produced objective measures of the cost of common business procedures in 145 countries. Similarly, the online Investment Climate Survey database provides a rich set of data based on responses from over 27,000 firms. Both the Doing Business Reports and Investment Climate Surveys have helped the Bank Group to develop clear, locally tailored strategies for supporting countries in undertaking country-owned reform (an OED recommendation) and have created powerful domestic constituencies for removing obstacles to private sector–led growth. The agenda-setting World Development Report 20058 pulled together the state of the art on investment climate reforms. At the same time, the broader development community and the Bank have all recognized that there cannot be a long-term growth strategy that does not rely on human development. While a growth-oriented strategy is critical for reducing poverty, long-term growth will fail if human development is ignored. Cost of Doing the Bank’s Business OED recommends that the Bank focus on its cost of doing business, pointing out that for every dollar of its administrative budget, the IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? 36 Bank disbursed $13 in FY95 but only $9 in FY05. It is unusual, to say the least, for an evaluation unit to imply that the Bank should measure effectiveness in this way. It is clear that if lending were the only measure of Bank contributions to development outcomes—without concern for efficiency and effectiveness or debt sustainability—the Bank could easily expand lending. However, it has been management’s view that the key focus of Bank work should be on development impact, not lending. In view of this, heavy emphasis has been placed over the last five years on core analytic work and improving project quality. Having established a solid base for increased lending, the Bank has expanded its lending over the past three years: new commitments in FY05 of $22.3 billion are 10 percent higher than new commitments in FY04, and the highest since the post-financial crises period of FY00–01 (figure A.6).9 Moreover, it has increased lending without sacrificing quality. There have been noteworthy increases in the quality and results of Bank products and services, and a significant increase in the Bank’s work program, particularly in LICUSs, all against a backdrop of a flat budget in real terms during FY95–04, signaling rising productivity. Before OED attempts to use the ratio of lending to Bank budget as a meaningful indicator, it should also take into account the costs of this involvement in LICUSs, the Bank’s stepped-up engagement in middle-income countries (MICs), and the Bank’s expanding leadership in global programs—all of which are discussed in great detail below. • Composition of Lending Commitments. An important point that OED does not note is that IDA commitments have increased by 60 percent between FY95 and FY04, growing at an average of 16 percent a year during the FY0005 period. IDA countries are the poorest in the world and are key to achieving the Millennium Development Goals. IDA resources have financed projects and programs that have helped build and sustain the policies, institutions, infrastructure, and human capital needed for equitable and environmentally sustainable development in low-income countries. IDA allocations are performance-based. Better policy performance of IDA countries—as documented by OED in last year’s AROE—has permitted this increase in IDA commitments and disbursements. Since IDA credits and grants are generally smaller than IBRD loans and are more ANNEX A: MANAGEMENT COMMENTS 37 Figure A.6: IBRD/IDA Commitments and Disbursements ($m), FY95–05 10,000 15,000 20,000 25,000 30,000 FY95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05 IBRD/IDA disbursements IBRD/IDA commitments Source: SAP/World Bank data. Note: FY05 includes guarantees. expensive for each dollar lent (because of many factors, including smaller average country size and greater capacity constraints), the increase in IDA commitments and disbursements has contributed to reducing commitments and disbursements per dollar of administrative budget. • Results of Bank Lending. Results have improved a great deal over the past 10 years: in FY95, 67 percent of projects exiting the portfolio were rated satisfactory for development outcomes, compared with 78 percent in FY04 (table A.2). If, as is expected, this trend continues through the rest of FY05, the Bank will reach its target of 80 percent satisfactory outcomes for the first time in a decade. As mentioned earlier, this means that each year projects costing about $5 billion are delivering substantial results just because of the attention to quality. Evaluated commitments in FY04 also reached a record high of 85 percent satisfactory, meaning that almost $1 billion more in commitments achieved their planned development outcomes in FY04. • Delivery and Results of Economic and Sector Work. OED itself has often made the point that lending alone is not sufficient for devel opment impact; it must be complemented by analytic and advisory services. Reinforcing this point, Bank operations are now supported by more, stronger, better-monitored, higher-quality economic and sector work (ESW) programs, which also contribute directly to supporting country outcomes. ESW deliveries have been ramped up: 734 pieces of ESW were delivered in FY04 compared to 368 in FY95 (figure A.7). By the end of FY04, most active IDA borrowers were covered with up-to-date core diagnostic reports (Poverty Assessments, Country Economic Memoranda, Public Expenditure Reviews, Country Procurement Assessment Reports, and Country Financial Accountability Assessments). This increase in ESW deliveries did not come at the expense of quality; in fact, the quality of individual pieces of ESW improved substantially from 73 percent satisfactory or better in FY98-99 to over 90 percent today. The likely impact (on client policies and institutional reforms, on Bank thinking, and on the wider development community) of ESW also went up from 66 percent in FY98–99 to 86 percent in FY02 (table A.2). OED’s synthesis reaffirms the importance of ESW, particularly in MICs: “Most MICs still regard the Bank as a IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? 38 Factor FY96 FY05 Country-owned PRSPs None 46 full/52 interim Country Assistance Strategies (CASs) 1 38 (FY04) ESW deliveries 336 734 (FY04) Encouraging a comprehensive development framework None In 50+ countries Country directors in the field None 73 percent Post-conflict lending and advice 15 countries 40 countries and territories Civil society involvement <50 percent 72 percent Debt relief operations None 27 countries (receiving debt relief), 18 countries (completion point) Project preparation time 24 months 14 months Quality at entry 78 percent 85 percent (FY03) Quality of supervision 63 percent (FY97) 90 percent (FY03-04) Quality of ESW 73 percent (FY98-99) 90+ percent (FY02) Satisfactory project outcomes 69 percent 78 percent (FY04) Table A.2: Examples of the Changing Bank, FY96–FY05 key source of knowledge and ideas and as a neutral source of policy advice” (page 17). • Client Ownership. The Bank has increasingly emphasized building client ownership by supporting countries in their formulation of country development strategies. The cumulative number of country-owned PRSPs completed has increased from zero in FY96 to 56 in FY05. The number of CASs completed increased from 1 in FY96 to 38 in FY04, and these CASs are now set up to support country-owned goals. • Alignment, Harmonization, and Results. The Bank has taken a leading role in working with other donors to align donor support around country priorities, harmonize donor requirements to lower costs to client countries, and put the focus on results. While this work entails up-front budgetary costs, it will pay growth and poverty reduction dividends as this agenda is taken to the country level and implemented widely. Executive directors have repeatedly noted that they expect the Bank to play a leading role in ensuring progress. The Bank is currently working with other development agencies to establish baseline data, set targets, and monitor progress with regard to this important agenda.10 Approach to MIC Clients and LICUSs The OED report identifies two areas in which it charges that the Bank is “struggling with its model for development effectiveness”—MICs and LICUSs. Management acknowledges that these groups of countries do present special challenges, but would note it has worked diligently to develop approaches that help the Bank better meet the needs of these countries. MICs. Lending to MICs is up in FY05, and there is a strong loan preparation pipeline. These trends result in large part from a number of changes the Bank has made in its approach to MIC clients. • It is increasing its use of well-performing country systems in special fiduciary areas (notably financial management). In September 2004, the Board endorsed the idea of pilot-testing the use of country environmental and social safeguard systems, and it is now considering pro ANNEX A: MANAGEMENT COMMENTS 39 Figure A.7: ESW Deliveries, FY95–04 0 100 200 300 400 500 600 700 800 FY95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 Source: SAP/World Bank data. posals. Following a review of the experience with implementing the pilot projects, proposals for wider adoption of the approach would be considered. While the pilot is not limited to one set of countries, many MICs are expected to benefit from it. • It is expanding the use of sectorwide approaches (SWAps), which until now have focused mainly on social sectors, to also include infrastructure. • It is offering MIC clients more choices, especially by promoting the use of a range of IBRD financial products. • The Bank Group is working to better exploit synergies in supporting development efforts in MICs: a Bank Group framework has been developed for lending to financial intermediates, including municipal funds; and a working group has been established to outline a proposal for a Bank Group facility for subsovereign lending. • Recent CAS documents for MICs—notably El Salvador, India, Kazakhstan, Mexico, and Slovakia—have customized support to country circumstances and facilitated a more flexible and rapid response to lending demand.11 Approach to LICUS. Until recently, the Bank was able to do very little in countries troubled by conflict or other serious problems, where policy performance was too poor to allow significant lending. But the Board has repeatedly stressed that all member countries deserve some level of support from the Bank; thus Bank resources are now going to country teams for work in countries emerging from conflict and in LICUSs. Several developments illustrate the dimensions of this work. • According to the FY06 Bank budget document, budgetary costs to support 13 of these countries/country groupings have risen from $12 million in FY95 to $53 million in FY04, with appropriately limited flow-through to lending commitments. • An important institutional change for engaging more effectively with LICUSs has been delinking administrative budgets for ESW and technical assistance from lending volumes, in recognition of the importance of maintaining analytical and capacity-building work in these countries, even while lending is low. Bank budget allocated for ESW to LICUSs increased by 85 percent between FY02–04, allowing the Bank to rebuild its knowledge base in LICUS, increasing readiness for fuller engagement, and enhancing the effectiveness of development interventions in the more stable LICUSs. • The Bank has taken a leadership role in bringing donors together around a common platform for support to these countries. At the Senior Level Forum for Development Effectiveness in Fragile States, held in London on January 13–14, 2005, participants agreed to draft and field-test a set of principles for support, and the Bank has been instrumental in producing these draft principles. They are now available and are being applied in LICUSs, notably in conflict-affected countries.12 In addition, the Transitional Results Matrix,13 a planning tool that the Bank helped develop and put into operation, integrates the political, economic, security, and social dimensions of recovery; allows for close donor coordination around government-agreed priorities; permits flexibility; and supports the monitoring of progress. Global Programs The findings of OED’s global program review underscore the Bank’s important role in the area of global activities—for example, HIV/AIDS programs, the HIPC Debt Initiative, observance of standards and codes, and efforts to fight money laundering and combat the financing of terrorism—and the need for more systematic attention to the management of global programs. Management took into account OED’s recommendations in preparing a report on a strategic framework for global programs. Each vice presidential unit is now explicitly accountable for strengthening the links of global programs and partnerships (GPPs) to Bank country- and Region-based development strategies; periodically reviewing and adjusting global programs to ensure that they focus on getting results; ensuring stable IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? 40 funding for key GPPs and planning exit strategies ahead of declines in funding; strengthening the Bank’s oversight of its GPPs to reduce reputational risks to the Bank; and ensuring a properly functioning control environment for programs that have in-house secretariats or are funded by the Development Grant Facility. New business processes will allow the Bank to track its GPPs from their start through to evaluation and impact assessment. These areas build on the actions management previously took in response to OED’s Phase I Report on Global Programs: for example, establishing a GPP Council as the senior management committee overseeing the strategic direction and operational policies for Bank involvement in GPPs,14 and establishing a new GPP Group led by a director reporting to the Vice President, Concessional Finance and Global Partnerships. Continuing Efforts While management believes the Bank has been—and is increasingly—effective in helping to improve the living standards of people in client countries, it never stops working to increase that effectiveness. It is currently working with the Board to reform the Bank’s instruments, policies, and processes to be more responsive to clients. Development policy lending now emphasizes country ownership of the reform program, embraces consultation and participation of stakeholders, and requires greater systematic attention to the possible effects of policy changes on poverty and the environment. The Bank is also reforming its tools for investment lending to increase development impact. It has changed some of its policies to support the new ways of doing business: for example, it has updated the policy on expenditure eligibility to make it easier for the Bank to finance a broad array of its expenditures that borrowers need to make to implement agreed operations; it has updated its procurement policies and is working on its disbursement policies; it has streamlined its approach to project auditing; and it has clarified its policy on participation in pooled financing to support SWAps. With regard to processes, the Bank has introduced changes in documents and procedures to allow projects to be processed more quickly, reducing average project processing time from 18 months to 14 months. OED Recommendations OED’s findings and evaluations at the project, sectoral and thematic, and country and global levels lead it to make the following nine recommendations to improve the Bank’s development effectiveness: • Understand and analyze comprehensively, but act far more selectively. • Emphasize institutional development and capacity building even more strongly. • Rethink areas of punctured optimism, such as growth, private provision of infrastructure, turn-around situations. • Tailor programs and projects to the circumstances of each country, and adapt strategies to the local political economy. • Lend mainly to countries with improving policies and institutions, but find ways to deal with poor misgoverned states. • Introduce greater flexibility into programs with well-managed middle-income countries. • Make better use of the Bank’s knowledge and technical assistance. • Improve monitoring and evaluation for results and start measuring what is important. • Improve coordination within the Bank and across the Bank Group. Management Response Management has little difficulty with any of these recommendations. They all represent good advice. Moreover, they are areas in which management has already done a great deal of work—with evidence of results. For example, results-based CASs are designed to analyze broadly but pick selectively a subset of country outcome goals for support, emphasize governance and capacity building, and tailor programs to country conditions; and flexibility has been a key element of recent country programs in support of MICs. In addition, coordination across the World Bank Group is improving, notably because of joint CASs, joint work around support for subnational entities, and support for guarantees. Management fully ANNEX A: MANAGEMENT COMMENTS 41 intends to continue in these directions. However, management does not entirely agree with one OED recommendation—rethinking Bank support in turn-around situations. Successful turn-arounds have extremely high payoffs in terms of growth and poverty reduction. If Bank support is crucial to a turnaround, management believes that the Bank should take the risk. The key is to be open and clear about the nature of the risk internally and in discussions with executive directors. (Tanzania is a good example of a country where, by taking a risk in the 1990s, the Bank contributed to a successful turn-around.) Focus on Results In management’s view, the key to addressing all of these recommendations is the Bank’s enhanced focus on results, which builds on earlier successes in turning around operational quality and delivery. The results agenda started with a focus on results-based CASs and is increasingly moving to operations and ESW; the emphasis this year is on key sectoral and thematic areas, and a longer-term goal is the results focus of global programs. The results focus is an integral contributor to achieving selectivity in country programs, helping countries build appropriate capacity, tailoring programs to each country’s circumstances, providing flexibility to middle-income countries, using knowledge and technical assistance better, and improving measuring, monitoring, and evaluation. Management will report to executive directors later in FY06 on progress on the enhanced results focus and its goals going forward. Lessons Learned However, the principal lesson to be drawn from the OED report is that the Bank’s work has made a significant difference to developing countries, and its efforts to further increase its impact are on the right track. Management is confident that, as it continues pursuing the initiatives it has begun, the Bank will continue to increase its effectiveness in helping countries reduce poverty and enhance growth. IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? 42 43 On May 16, 2005, the Committee on Development Effectiveness met to discuss the OED report Improving the World Bank’s Development Effectiveness: What Does Evaluation Show? The Draft Management Comments to the OED report were prepared and circulated after the meeting. OED Report. The report provides a broad overview of the development context in which the development effectiveness of the Bank is being assessed by providing a summary of the record in developing countries of poverty reduction, growth, policy and institutional measures, and security. OED evaluation findings at different levels of aggregation at the project, sectoral and thematic, and country and global levels are provided. The report suggests that the following nine directions can improve the Bank’s development effectiveness: • Understand and analyze comprehensive, but act far more selectively. • Emphasize institutional development and capacity building even more strongly. • Re-think areas of punctured optimism, such as growth, private provision of infrastructure, turn-around situations. • Tailor programs and projects to the circumstances of each country, and adapt strategies to the local political economy. • Lend mainly to countries with improving policies and institutions, while finding ways to deal with poorly governed states. • Introduce greater flexibility into programs with well-managed MICs. • Make better use of the Bank’s knowledge and technical assistance. • Improve monitoring and evaluation for results, and start measuring what is important. • Improve coordination within the Bank and across the Bank group. This review of the Bank’s development effectiveness points to a number of opportunities for further improvements that will require changes in the business process, changes in the organization, and strategic choices for the institution, and it highlights the decline in the ratio of lending to the Bank’s administrative budget in recent years (with perhaps some reversal this year), and the perceived high costs of doing business with the Bank. Management comments. Since this paper is not a new evaluation and attempts to summarize lessons from recent evaluations, management felt that it has already responded to the OED findings and recommendations. Management recognized that more progress could have been made in dealing with poverty, but thought that the tonality of the OED paper might be too pessimistic. While agreeing on the importance of growth for development and poverty reduction, management commented that it is remarkable that developing countries have significantly surpassed the industrial countries’ growth ratio in the past 6 to 7 years. Regarding the disconnect between project and country outcomes, management noticed that OED evaluated CASs that were not conceived as evaluable instruments, which will be the case for the results-based CASs. Overall conclusions. Members welcomed the opportunity to discuss the paper, an earlier version of which had served as a platform for some reflection at the last CODE Retreat. They commended OED for preparing a useful synthesis focusing on a number of key questions that emerged by distilling the lessons ANNEX B: REPORT TO THE BOARD FROM THE COMMITTEE ON DEVELOPMENT EFFECTIVENESS (CODE) from recent evaluations. One member wondered how to operationalize these lessons. Speakers found the paper well-written and candid, and praised its format and length, which could serve as a model for other CODE papers in the future. Given the extensive scope of the paper despite its brevity, the comments were also wide-ranging and covered substantive findings, methodological issues, as well as implications for both for Bank strategy and CODE’s own work. A few issues which resonated with several participants were the following: (i) decline in lending volumes even as the administrative budget has continued to grow; (ii) non-financial costs of doing business with the Bank; (iii) disconnect between project performance and country outcomes; (iv) need for a systematic review of policy advice on design and sequencing of reforms; and (v) simplification and modernization of lending. Overall, most speakers welcomed the emphasis on growth, which they saw as a necessary condition for poverty reduction. Among the specific issues raised by the Committee were: Portfolio size. Some members expressed concern about the decline in the Bank’s lending volumes even as the administrative budget has continued to grow, which one member suggested could be a sign of increased bureaucracy. They agreed with OED that the increased cost is due to the rising share of knowledge activities and other non-lending activities such as global programs. A few members added that the rise in administrative budget could be explained by duplication of activities created by the Bank’s decentralization, high cost of staff travel, and insufficient outsourcing. Management informed that intra-regional staff travel has increased with decentralization, while outsourcing-type activities were already in place in the Chennai back-office. Some members encouraged the Bank to reverse the decline of lending to MICs, and to ensure more efficient use of resources in knowledge activities including delivering more “just-in-time” analytic and advisory assistance (AAA). A member noted that lending to MICs by regional development banks had not declined as it had in the World Bank. Management informed that there is this year an increase in lending to the MICs. Other members added that transitional economies, LICUS, and small states also need the Bank’s engagement, and felt that the Bank needs to improve the staff incentive structure and management of risky projects. Cost of doing business. Some members perceived that higher non-financial costs of doing business were making the Bank less competitive for many borrowing countries. One of them stressed the importance of consultation and participatory processes in Bank projects but cautioned against too much deference to nongovernmental organizations (NGOs) that may contribute to undermining the authority of democratically elected governments. Disconnect between project and country outcomes. Several members noted the disconnect in several countries between aggregate project ratings and country program outcomes. Some suggestions were made (i) to review the efficacy of the Bank’s development model rather than just outcomes of individual programs and projects; (ii) to focus on sustainability of country outcomes, including financing of recurrent costs; (iii) to consider the impact of external factors on programs and projects; and (iv) for OED to address lessons learned also from unsuccessful outcomes of assistance strategies as indicated by CAE ratings (which do not report on Bank performance). A member suggested that a synthesis of the findings of each year’s CAEs be added to the ARDE. OED responded that the CAE retrospective, which looked at 25 CAEs from FY0l to FY04, collected many lessons learned from a diverse group of cases such as conflict-affected countries, small states, transitional economies, and countries affected by external shocks. Poverty reduction and development strategies. Members concurred that there is strong correlation between economic growth and reduction of poverty. One of them felt that the analysis of the Bank’s development effectiveness should factor in the Bank’s weight (e.g., share of external flows) in assisting a client country relative to other players. Another member felt that in some cases the CAEs IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? 44 showed positive outcomes, while the growth performance of the countries had not been positive, which questions the development effectiveness of the assistance. OED noted that there is high correlation between positive country growth performance and satisfactory outcomes of Bank programs. OED clarified that these outcomes were measured against the objectives set in the country programs. Many speakers felt that a systematic review was needed of policy advice on the design and sequencing of reforms, and the expected interaction between public and private sectors, especially in areas where results are mixed such as privatization, or fiscal decentralization. Simplification and modernization of lending. A member asked for more simplification of lending and enhanced use of country systems. It was also suggested that the Bank should be selective in its assistance and exit from programs where the Bank’s development effectiveness is marginal. Management noted that new initiatives such as use of country systems could contribute to reduction of costs. Country programs. Many members agreed that the Bank’s programs should be tailored to the specific circumstances of the countries including their political economy. They highlighted the importance of assisting countries to build capacity for economic planning and policy formulation, improve governance, strengthen environmental and social institutions, and align the results framework of the CAS and lending operations. OED commented that some programs aligned to Washington Consensus– type of reforms had not had the desired effect, perhaps because they had not considered sufficiently the political economy of the countries. Management added that three main issues—governance, capacity, and the country‘s results management framework— are addressed at each new CAS preparation. ANNEX B: REPORT TO THE BOARD FROM THE COMMITTEE ON DEVELOPMENT EFFECTIVENESS 45 Chander Mohan Vasudev, Chairman 47 Executive Summary 1. The average size of investment loans dropped from US$80 million in the mid-1990’s to about US$65 million in the last two years. Chapter 1 1. The increase in poverty head-count probably stretches back to the beginning of the 20th century. Chapter 2 1. This discussion is based on the 2004 Annual Review of Development Effectiveness: The Bank’s Contribution to Poverty Reduction. 2. The analysis excludes the global information and communications technology, poverty reduction, and social development because OED has evaluated very few of their projects. 3. OED has carried out two detailed impact evaluations in education in Ghana (OED 2004b) and health and nutrition in Bangladesh (OED 2005c) and is conducting another on rural poverty in India. OED is also reviewing all the impact evaluations under way at the Bank. 4. The size of investment loans declined from US$80 million to US$65 million over the same period. 5. China being an obvious one. Annex A 1. OED notes that the first subsection of Chapter 2 (page 7) is entitled “Project Performance Ratings Show Improvements over the Past 10 Years,” and that this section shows clearly that project ratings have increased (see, for example, figure 2.2). 2. 2003 Annual Review of Development Effectiveness—The Effectiveness of Bank Support for Policy Reform (Washington, D.C.: The World Bank, 2004). 3. Global Monitoring Report 2005—Millennium Development Goals: From Consensus to Momentum (Washington, D.C.: The World Bank, 2005). 4. The Global Monitoring Report 2005drew on the most recent literature on growth to conclude that there is no minimum set of reforms required to spur growth, or a larger set that is sufficient to sustain it. The relationships among growth and policies, aid, shocks, the external environment, and other factors are complex. 5. 2003 Annual Review of Development Effectiveness, op. cit. 6. OED notes that before commencing work on a CAE, an Approach Paper (AP) is prepared that clearly sets out the objectives against which the country assistance program will be evaluated. Each AP is sent to OPCS and the Country Department for comment prior to its circulation to CODE for approval. Therefore, it is unclear to OED what is meant by the statement that “it has never been clear to management which objectives OED rated.” 7. See 2004 Annual Review of Development Effectiveness: The Bank’s Contributions to Poverty Reduction (R2005-0084, IDA/R2005-0061), April 12, 2005, available at http://www.worldbank.org/oed/arde/2004/. 8. World Development Report 2005: A Better Investment Climate for Everyone (Washington, D.C.: World Bank and Oxford University Press, 2004). 9. The years fiscal 1998–99 were extraordinary years for the World Bank, given the East Asian Crisis and support to Argentina. 10. Paris Declaration on Aid Effectiveness: Ownership, Harmonisation, Alignment, Results and Mutual Accountability (DC2005-0002), April 1, 2005. 11. Enhancing World Bank Support to Middle Income Countries—Management Action Plan: Progress Memorandum(Washington, D.C.: The World Bank, 2005). ENDNOTES 12. Principles for Good International Engagement in Fragile States(Paris: Organisation for Economic Cooperation and Development, 2005). 13. Search “Transitional Results Matrix” at www.worldbank.org. 14. Update on Management of Global Programs and Partnerships (Washington, D.C.: The World Bank, 2003). Two managing directors chair the Council, and its members are vice-presidents from Networks, Regions, and corporate areas. IMPROVING THE WORLD BANK’S DEVELOPMENT EFFECTIVENESS: WHAT DOES EVALUATION SHOW? 48 49 OED (World Bank Operations Evaluation Department). 2005a. 2004 Annual Review of Development Effectiveness: The World Bank’s Contribution to Poverty Reduction. OED Study Series. Washington, D.C.: World Bank. ———. 2005b. Country Assistance Evaluation Retrospective: An OED Self-Evaluation. OED Study Series. Washington, D.C.: World Bank. ———. 2005c. Maintaining Momentum to 2015? An Impact Evaluation of Interventions to Improve Maternal and Child Health and Nutrition in Bangladesh. OED Study Series. Washington, D.C.: World Bank. ———. 2004a. 2003 Annual Review of Development Effectiveness: The Effectiveness of Bank Support for Policy Reform. OED Study Series. Washington, D.C.: World Bank. ———. 2004b. Books, Buildings, and Learning Outcomes: An Impact Evaluation of World Bank Support to Basic Education in Ghana. OED Study Series. Washington, D.C.: World Bank. ———. 2004c. Addressing the Challenges of Globalization: An Independent Evaluation of the World Bank’s Approach to Global Programs. OED Study Series. Washington, D.C.: World Bank. OECD (Organisation for Economic Co-operation and Development). 2005. “Principles for Good International Engagement in Fragile States.” Report DCD(2005)8/REV2. Paris. OECD/DAC. 2005. “Paris Declaration on Aid Effectiveness: Ownership, Harmonisation, Alignment, Results and Mutual Accountability.” Paris. World Bank. 2005a. Global Monitoring Report 2005—Millennium Development Goals: From Consensus to Momentum. Washington, D.C. ———. 2005b. “Enhancing World Bank Support to Middle Income Countries—Management Action Plan: Progress Memorandum.” Report No. 31469. Washington, D.C. ———. 2004. World Development Report 2005: A Better Investment Climate for Everyone. Washington, D.C.: World Bank and Oxford University Press. REFERENCES