Profiles

Emerging evidence suggests that social protection programmes can have a positive role in delaying marriage for girls. But the pathways and design features by which programmes may influence child marriage outcomes remain unknown. This mixed-methods study explores whether and how the Productive Safety Net Program (PSNP) in Ethiopia, given its national reach and potential to address poverty, can also affect child marriage practice. It draws on descriptive quantitative and qualitative data from an ongoing impact evaluation of the Integrated Safety Net Program (ISNP) pilot in the Amhara region. It finds that PSNP, through an economic channel, is effective in reducing financial pressures on families to marry off girls and in improving girls’ education opportunities. Income-strengthening measures must, however, be accompanied by complementary efforts – including girls’ empowerment, awareness-raising and legal measures – to transform deep-rooted social and gender norms and attitudes that perpetuate the harmful practice of child marriage.

This rapid review seeks to inform initial and long-term public policy responses to the COVID-19 pandemic by assessing evidence on past economic policy and social protection responses to health and economic crises and their effects on children and families. The review focuses on virus outbreaks/emergencies, economic crises and natural disasters which, similar to the COVID-19 pandemic, were rapid in onset, had wide-ranging geographical reach, and resulted in disruption of social services and economic sectors without affecting governance systems. Lessons are also drawn from the HIV/AIDS pandemic due to its impact on adult mortality rates and surviving children.

Ethiopia is one of the world’s largest recipients of donor funds for development and emergency interventions. As such, its targeting of social protection has received substantial attention. In particular, concerns have been raised that political connections could play a role in determining the selection of beneficiaries. With the introduction in 2005 of the Productive Safety Net Programme (PSNP), Ethiopia implemented various policies aimed at increasing transparency in the targeting of social protection. This case study compares targeting before and during the implementation of PSNP, and shows improvements in targeting for both public works and emergency aid in relation to the dimensions of poverty, food security and political connections. Most notably, political connections are no longer found to determine the receipt of benefits during the implementation of PSNP.

Social protection in Ethiopia is primarily allocated through community-based targeting. The few studies that have analysed the efficacy of aid targeting in Ethiopia have revealed targeting biases in regard to demography, geography and political affiliations. With the introduction in Ethiopia in 2005 of the Productive Safety Net Programme (PSNP), a major social protection programme, various administrative guidelines were introduced (and subsequently periodically revised) with the aim of improving targeting. This paper uses data from the last two rounds of the Ethiopian Rural Household Survey to investigate whether PSNP implementation resulted in changes in both targeting determinants and amount received for public works (a component of PSNP) and emergency aid between 2004 and 2009 in 11 rural villages. In general, public works appear to have been allocated on the basis of observable poverty-related characteristics, and emergency aid according to household demographics. In addition, the results suggest that, for both public works and emergency aid beneficiaries, political connections were significant in determining the receipt of aid in 2004 but that this was no longer the case by 2009, indicating an improvement in the channeling of social protection to its intended target groups. However, a household’s experience of recent shocks was found to bear no relationship to receipt of support, which suggests that a more flexible and shock-responsive implementation could improve targeting for transitory needs.

There is increasing interest in understanding if social protection has the ability to foster social cohesion, particularly between refugees and host communities. Using an experimental evaluation of transfers, including cash, food and food vouchers to Colombian refugees and poor Ecuadorians in urban and peri-urban areas we examine if transfers resulted in changes in social cohesion measures. The evaluation was a cluster-randomized control trial examining a short-term programme implemented over six months by the World Food Programme. We examine six aggregate dimensions of social cohesion, derived from 33 individual indicators, in addition to an overall index of social cohesion. Overall results suggest that the programme contributed to integration of Colombians in the hosting community through increases in personal agency, attitudes accepting diversity, confidence in institutions, and social participation. However, while having no impact for the Ecuadorian population. There were no negative impacts of the programme on indicators or domains analysed. Although we are not able to specifically identify mechanisms, we hypothesize that these impacts are driven by joint targeting, messaging around social inclusion and through interaction between nationalities at mandated monthly nutrition trainings.

How do we know if a programme made a difference? The answer to this question is not as straightforward as it seems, because we never know what would have happened without the programme. This concept is referred to as the ‘missing counterfactual’ (or simply 'the counterfactual' since, by definition, a counterfactual is missing). Impact evaluation is the science of estimating the missing counterfactual; getting it right is the necessary first step in any evidence-based approach to policy design. The science of impact evaluation was the subject of a two-week technical training workshop organized jointly by the Transfer Project and the African Economic Research Consortium (AERC) in Nairobi, Kenya from June 24 to July 4 2019. The training attracted 24 participants (15 women) from 22 African countries who were selected by the AERC from over 350 applicants. Participants were primarily university lecturers working in the fields of applied and agricultural economics. The workshop is part of a larger collaboration between the Transfer Project and AERC focused on promoting evidence-informed decision-making in sub-Saharan Africa funded by the Hewlett Foundation. The workshop content was structured around this core question (“How do we know if a programme made a difference?”) and the related fact that unless we can invent a machine to take us back in time, we can never observe the outcomes both with and without the programme. The job of the evaluator is to estimate the “missing counterfactual” using the tools of research design and statistics. 'Greatest Hits' During the first week of training, participants were taught the ‘greatest hits’ of estimating a counterfactual, beginning with randomized designs and followed by non-randomized designs, such as propensity score matching, discontinuity design, instrumental variables, and natural experiments. They also learned how to add ‘design elements’ to strengthen their studies - for example through baselines or repeated follow-up observations - and to estimate sample sizes in both simple and complex sampling scenarios. Theoretical lectures were complemented with hands-on computer-based demonstrations and case studies from Transfer Project impact evaluations in Ghana, Malawi and Zambia. In the second week, participants were given a hypothetical “Request for Proposals” for an impact evaluation and worked in groups to develop a response using the tools they had learned throughout the course. Final presentations included proposal designs for a new impact evaluation of the Ghanaian LEAP cash transfer, an evaluation of the Kenya NICHE programme, and an evaluation of the Government of China/UNICEF joint pilot conditional cash transfer programme for child nutrition. These are all actual programmes undergoing or seeking to undergo an evaluation—implementers should feel free to contact the study teams for their full proposals! Impact Evaluation Training with AERC: Ghana's LEAP Programme Technical Research Proposal from The Transfer Project It was clear the participants had absorbed the course contents when, by the end of the second week, we noticed an increasing number of jokes and statements using evaluation language. Patrice from Congo suggested that we didn’t know if we could replicate the success of the course again because he and the other participants were “self-selected”; Isabelle from Cote d’Ivoire suggested we use ‘exact matching’ to figure out who should stand next to whom for the group photo; and everyone supported the idea that the order of group presentations should be randomly assigned... If nothing else, at least we have increased the number of nerdy evaluation jokes that are likely to be cracked across sub-Saharan Africa. In the closing session, Gustavo Angeles from the University of North Carolina at Chapel Hill told participants that the workshop itself was an intervention, and he was expecting a steep upward slope in the intervention group relative to the counterfactual. AERC Executive Director Dr. Njuguna Ndung’u encouraged participants to make the most of the training by using impact evaluation techniques to make a difference in their home countries. While increasingly used to inform policymakers on the effects of policies, impact evaluation methods are not always included in curricula of undergraduate economics degrees, and even when they are, the focus is merely theoretical with few methods covered. There is increasing demand for comprehensive courses on impact evaluation methods, which has led to the proliferation of several intensive workshops worldwide. In developing countries, however, the supply of such courses has not yet met the demand. The Transfer Project is beginning to close this gap with its capacity building work, but it is clear from the large number of applications for this training that there is a large appetite in the region for more information on this technical topic.   Assess your own knowledge of impact evaluation techniques—take our ‘Who Wants to be a Millionaire?’ quiz and see whether you would benefit from this course! Ashu Handa, University of North Carolina: Ashu is Lawrence I. Gilbert Distinguished Professor in UNC’s Department of Public Policy. Elsa Valli, UNICEF Innocenti: Elsa is a Research Analyst with UNICEF’s Office of Research - Innocenti in Florence. Gustavo Angeles, University of North Carolina: Gustavo is the Senior Evaluation Advisor for MEASURE Evaluation & Associate Professor at UNC. All three authors contribute to the work of the Transfer Project research and learning collaborative.