close× Call Us
close×
Title Letting Communities Take Charge: A Randomized Controlled Trial on Sustaining Schools in Afghanistan
Post date 08/17/2018
C1 Background and Explanation of Rationale

This pre-analysis plan lays out the objectives for the analysis of Phase II Endline data from the Assessment of Learning and Social Effects of Community Based Education (ALSE) study in Afghanistan. For general background on the ALSE project, refer to previous reports, available at www.alseproject.com.

ALSE Phase II tests a strategy for sustaining access and learning gains from community-based education (CBE) classes after the withdrawal of non-governmental organizations (NGOs) who were responsible for setting up the classes. CBE classes are often started by non-governmental organizations (NGOs) and in Afghanistan most CBE classes are administered by NGOs. However, NGOs operate on program cycles that are dictated, largely, by funding cycles. As such, it is a near inevitability that NGO administration of a CBE class will have to come to an end at some point. When that happens, the question arises as to how one might sustain access to education for the children attending CBE classes.

Intervention
The ALSE team worked with the MOE, other government actors, and implementing NGOs to develop a “sustainability model” that may increase the chances that NGO-initiated CBE classes could continue to function within the village under overall MOE administration and co-management of village-level community institutions after the departure of NGOs. This sustainability model is the intervention tested in ALSE Phase II.

The sustainability model takes advantage of the decentralized community governance structure created as part of Afghanistan’s National Solidarity Program (NSP), evaluated by Beath, Christia, and Enikolopov (2013). The NSP organized village-level community development councils, consisting of a small number of appointed and elected villagers. Under NSP, the community development councils were tasked with devising village development plans that would be used to guide the application of village block grants. These community development councils have remained active through to the present (see the ALSE Phase II Baseline report for relevant findings). The ALSE “sustainability model” charges an education-focused subcommittee of the village community development councils to assume administrative responsibility of the CBE class after the NGO’s departure. This administrative responsibility included recruiting, paying, and monitoring the teacher, facilitation of acquisition of space, school supplies, and other material needs, and attending to other problems that arose in relation to the class. ALSE provided management capacity training to the community development councils and subcommittee to take on these tasks. With the village-level community development councils members capacitated to manage the CBE class, all that is needed will be a line of financing to the community development councils to cover teacher salaries and material costs, as well as occasional central oversight. ALSE provided financial support, which included teacher salary and classroom supplies. The distribution of teacher salaries and central oversight were conducted jointly by ALSE and district MOE offices. If successful, the model is intended to be taken up fully by the MOE. The model accords with the Citizen’s Charter Initiative, undertaken at the level of the Afghan Presidency, and which proposes to use the NSP institutional infrastructure to improve basic service delivery.

The primary goal for our analysis is to assess whether the sustainability model might be deemed successful in making it highly likely that classes will continue (sustainability effects), while achieving attendance and learning outcomes at a level that meets a benchmark set by NGOs (access and learning effects). This assessment will be based primarily on sustainability, access, and learning effects. It will also explore gender- and social institution-related (e.g., ethnicity-related) moderator effects. Finally, assessments will be drawn on whether or/and to what extent this sustainability model exudes “added value” in terms of effects on local and national government legitimacy (governance effects). A secondary motivation for the analysis is to assess mechanisms that contribute to effects that we observe.

In addition to being assigned to the community management (that is, village-level institutions’ management) versus extended NGO management treatment conditions, communities have also been randomly assigned to receive two types of enhancements to the ways in which the CBE program was implemented during Phase I. This included a “teacher recruitment” enhancement. For this enhancement, the treatment condition was that teachers were recruited using a MOE requirement of education credentials for teachers. The control condition was the standard recruitment practice of the implementing NGOs, which did not require teachers to possess the MOE-required educational credentials. The rationale for this enhancement was to see if hiring such “qualified” teachers would help sustainability by making it easier for the MoE to take over management responsibility for the school. An assessment of the effects of this teacher recruitment treatment on Phase I attendance and learning outcomes is given in Burde, Middleton, and Samii (2016). That study found that requiring teachers to meet the MOE’s education credentials had a modestly positive effect on attendance (0.050 percentage points, p = 0.072) and learning (0.097sd, p = 0.058). That said, the main interest with this variation was to improve sustainability, which is the focus of the current study.

The second enhancement was a “community engagement” enhancement. In the treatment condition, communities were given placards with Qur’anic messages about education and an adult reading program was established. Both of these programs were intended to increase parents’ enthusiasm for education generally and also to contribute to their motivation and capacity to help their children learn. The control condition was the NGOs’ default programming, which did not include such extensive efforts to engage parents. The rationale for this enhancement was that it could improve children’s learning by creating a more supportive environment. It could also contribute to communities’ willingness to make efforts needed to promote sustainability. Burde, Middleton, and Samii (2016) found that this variation had negligible effects on attendance (0.002, p = 0.475) and learning (-0.060sd, p = 0.862). The current study will assess whether this variation contributes to sustainability.

C2 What are the hypotheses to be tested?

Research Questions
We aim to inform a discussion of the merits of a sustainability model that involves local governance and community institutions as a way to sustain gains from NGO-initiated CBE. We do not propose to establish standards at this time by which we will definitely declare the model a “success” or “failure.” Rather, we propose that the evidence be used in a more deliberative process to determine whether the sustainability model offers a reasonable solution to challenges in maintaining access and learning in communities that have hosted NGO-initiated CBE.

Primary Research Questions
We have four primary research questions, as follows:

1. Do access or learning suffer from switching to the sustainability model rather than continuing NGO administration?

We are interested to know whether switching from NGO administration to the sustainability model results in any substantial losses in terms of access and learning. Some degree of loss may be tolerable, to the extent that there are other compensating benefits in terms of institutionalizing access to primary education for the long term, reducing costs, or helping to boost government legitimacy.

2. Does the sustainability model substantially improve stakeholders’ confidence in institutionalizing access to primary education into the future?

As discussed above, while the access and learning may suffer to some degree from switching to the sustainability model, this would be acceptable if the sustainability model helps to institutionalize access to primary education into the future. Thus, we want to measure the extent to which the sustainability model promotes a sense that the schools will be sustained into the future as compared to expectations formed under NGO administration. Indeed, the precariousness of NGO-initiated CBE classes under status quo arrangements, as discovered by Burde, Middleton, and Samii (2015), was the primary motivation for this study.

3. Does the sustainability model promote local or national government legitimacy relative to the legitimacy effects under NGO administration?

In previous work, Burde, Middleton, and Samii (2018) document that NGO administered CBE has the effect of increasing trust of participating communities in NGOs, community institutions, and national government. The latter two points are of special interest here. Burde, Middleton, and Samii (2016) documented that the strongest effect was for trust of NGOs, reflecting, it would seem, people’s appreciation of the NGOs role in significantly improving access to primary education. Here, we are interested to see if the sustainability model induces: i) an even higher level of trust in community institutions and national government as compared to NGO administered CBE and ii) a transfer of such sentiments of trust to the national government, given that under the sustainability model the national government is jointly responsible for the oversight of the CBE system. Of course, these effects likely depend on whether the quality of access and learning meets the standards set by NGOs.

4. Can the sustainability model operate at costs that are substantially less than NGO administration?

Finally, the MOE and Citizen’s Charter Initiative have indicated that they cannot sustain the gains from NGO-administered CBE if doing so requires spending the same amount that NGOs spend per child. Burde, Middleton, and Samii (2016) estimated the annual operating cost under NGO administration to be approximately $150 per eligible child. (Note that from Phase I we estimated that about 50% of eligible children took up the opportunity to attend CBE, with another 30% attending other government schools and 20% not attending either. Then,$150 per eligible child implies about $150/0.50 =$300 per child attending CBE. Estimating costs per eligible child is generally preferable, because it keeps the denominator fixed. See Dhaliwal, Duflo, Glennerster, and Tulloch, 2012, for a relevant discussion.) In this study, we assess costs under the sustainability model to compare.

Secondary Research Questions
We also have secondary research questions that pertain to moderator effects and causal mechanisms.

1. Does the sustainability model generate effects that are biased against girls or particular ethnic groups relative to the NGO benchmark?

Even if we were to find that the sustainability model performed well by the standards set under the primary research questions, it would be important to know whether effects differ by important social categories. For reasons of vulnerability, under-representation, and political sensitivity, we focus on whether effects are moderated by gender or ethnicity.

2. Do the access/learning, sustainability, or legitimacy effects depend on the ways that community institutions are activated under the sustainability model relative to NGO administration?

The sustainability model is based on a principle of joint responsibility between the community and national government institutions. As part of the implementation, communities assigned to operate through the sustainability model received management capacity training. The question is whether this training was adequate and whether the transfer of authority to community institutions inspired greater community engagement in the administration of the schools. We will look at whether indicators of management quality and community engagement plausibly mediate the access/learning, sustainability, or legitimacy effects. These indicators include whether sustainability model communities differ from NGO administration communities in the amount and manner of funding for classes, levels of community satisfaction and engagement, or teacher motivation and performance. We will also conduct an analysis that examines how performance under the sustainability model varies with levels of pre-training administrative capacity. We assessed such administrative capacity prior to the launching the training for the sustainability model intervention. A descriptive analysis of the results of the administrative capacity assessment is given in Burde, Middleton, and Samii (2017). For the current study, if the training were adequate, we should find that our measure of pre-existing administrative capacity does not have a substantial moderating effect. We will also examine whether the “community engagement” treatment variation introduced in Phase I moderates the effectiveness of the sustainability model relative to the NGO benchmark.

3. Does sustainability improve under enhanced variations of the NGO-administered CBE class, in particular, enhancements focusing on teacher recruitment and community engagement?

During the Phase I period, communities were randomly assigned to receive enhancements focused on teacher recruitment and community engagement. The teacher recruitment enhancement constrained the NGO to recruit teachers who had educational qualifications that would allow them to be hired into the MOE system. The status quo was that NGOs would recruit teachers who had requisite literacy and numeracy and that were well integrated in the village, but otherwise may not have had adequate formal education to qualify for being hired by the MOE. The community engagement enhancement used Qur’anic messaging, adult reading groups, and a community library to try to increase parents’ engagement with their children’s learning. Communities were randomly assigned to received either one or both of these enhancements. Our interest is to see whether these enhancements then translate into better sustainability outcomes.

C3 How will these hypotheses be tested? *

Research Design
Treatment Assignment
The design of ALSE Phase II analysis builds on the randomization done prior to ALSE Phase I. In 2014, candidate villages were randomly assigned to treatment conditions using a restricted randomization, whereby a treatment assignment profile was randomly selected from the set of treatment assignment profiles that satisfied a covariate balance criterion. The details of this assignment procedure are given in the technical appendix of the ALSE Phase I endline report (Burde, Middleton, and Samii 2016). At that time, communities were randomly assigned to one of two different intervention conditions that would not be implemented until Phase II. The conditions were either the “sustainability model” treatment condition that involved handing over the village school management responsibility to community institutions at the start of Phase II, or a control condition that involved continued NGO management through the duration of Phase II. The teacher recruitment and community engagement enhancements were similarly randomly assigned, and these were done in a way that perfectly cross-cut each other and the sustainability model assignment. As such, the overall design is a three-way full factorial design. Table 1 (see attached pre-analysis plan) shows the distribution of communities over the treatment conditions.

Data Collection
Our outcome data come from face-to-face surveys of village heads and samples of household heads in the study villages as well as literacy and numeracy learning assessments conducted with children from the sampled households. Sample sizes with respect to villages, households, and children appear in Table 1. The surveys were conducted between October and December 2017 toward the end or just after the conclusion of an academic year during which either the sustainability model or extended NGO administration were in effect. D3 Systems/ACSOR managed the enumeration teams that conducted the face-to-face interviews. They were trained on our data collection instruments and procedures. Their work was subject to independent monitoring and audits in randomly selected villages by teams of ALSE project associates.

The respondent sampling procedure was developed to account for the fact that written registers of households were not available, and so household enumeration and selection had to occur in the field. To select respondents, first village heads were identified in each village and the enumeration teams obtained permission to conduct the survey. The village heads were also interviewed. Then, households were enumerated by having enumeration team members start at the central mosque or equivalent and radiate outward, enumerating and listing domiciles until they enumerated 70 households. Then, every other household on the list was selected for interview for a target of 35 households. Multiple attempts were made to contact households if no one was present at the first attempt. If a selected household refused or had no one present after these multiple attempts, this would be documented and the next household on the list would be selected as a replacement. Heads of household or the most senior person available was chosen for an interview in the selected household. In addition, the survey team collected data on rosters of other household members and all children. Children aged 6-13 were also given the verbal and mathematical learning assessments. Finally, the survey team interviewed the teacher in each of the village schools. The household response rate for the survey (using the AAPOR Response Rate 3 criterion) was 95.1%; for children on the learning assessments it was 78.4%; for community leaders it was 98.3%; for teachers it was 98.6%. For more details and discussion of deviations from sampling protocol, see Frank (2018).

Outcome Variables
The survey gathered data on outcome variables that we classify into 12 themes. The first five themes are our main outcomes of interest:
(i) access and attendance
(ii) learning,
(iii) sustainability,
(iv) civic engagement and village institution legitimacy
(v) national government legitimacy
The next seven themes track “mechanism” outcomes that we intend to use to understand any differences between the treatments:
(vi) barriers to school access
(vii) the amount spent per eligible student
(viii) ways that funds are applied in communities
(ix) community engagement with managing education
(x) community satisfaction with education services
(xi) teacher motivation, and
(xii) teacher performance.
Table 2, panels 1-12 (see attached pre-analysis plan), display the indicators measured for each of these themes, the data source, the type of variable created, and then the nature of the hypotheses to be tested on each. (Our strategies for hypothesis testing are described below.)

Moderator Variables
We also have data on a set of moderator variables that we plan to use in assessing conditions that contribute to the functioning of the sustainability model. In some cases what interests us are moderator effects for the attendance (Table 2 - Panel 1) and learning outcomes (that is, verbal learning assessments, math learning assessments, and school attendance, as shown in Table 2 - Panels 1 and 2), and in other cases for sustainability outcomes (as shown in Table 2 - Panel 3). The relevant moderator variables and effects of interest include the following:

(i) Capacity of the administrating organizations’ (NGO vs community institutions) moderator effect on attendance, learning, and sustainability,

(ii) Teacher qualifications effect on attendance, learning, and sustainability,

(iii) Levels of community engagement moderator effect on attendance, learning, and sustainability,

(iv) The quality of NGO services prior to handover moderator effect on attendance, learning, and sustainability,

(v) Various barriers to accessing school among households moderator effect on attendance, learning, and sustainability,

(vi) Village population size moderator effect on attendance, learning and sustainability;

(vii) HH economic conditions, which define both status and also incentives for educational attainment, moderator effect on attendance, learning and sustainability,

(viii) Gender moderator effect on attendance and learning,

(ix) Children’s ages moderator effect on attendance and learning, and

(x) Social institutions and ethnicity moderator effect on attendance, learning, and sustainability.

Table 3 (see attached pre-analysis plan) lists the indicators associated with these moderators along with details on how they will be used.

The equivalency hypotheses are motivated by a desire to check whether the sustainability model operates in a way that unintentionally disadvantages certain types of households relative to NGO administration. For example, for gender, we want to be sure that under the community administration of the sustainability model that there is no disadvantage for girls. Another example is community management capacity: here, we want to ensure that the training that was done prior to handover to community institutions was adequate so as to ensure that all communities were equally ready to take on the school management responsibilities. (Of course, had there been no such training, we would pose a non-equivalency hypothesis, because one would presume that community’s ability to manage the school would be affected by such capacities.)

The non-equivalency hypotheses are based on factors that one would expect to contribute to the performance of community institutions---for example, the intervention that sought to ensure teachers with relevant qualifications were hired so as to facilitate the sustainabilty of the class.

Estimation and Inference
In the following section we discuss estimation and inference for both the equivalency and non-equivalency hypothesis that are indicated in Table 2, adjustments for multiple comparisons, and estimation and inference for moderator effects.

For discussion of estimating average treatment effects, inference for average treatment effects, multiple comparisions for average treatment effects, and moderator effects, please refer to the attched pre-analysis plan.

Application of methods to research questions

Primary Research Questions
1. Do access or learning suffer from switching to the sustainability model rather than continuing NGO administration?
This will be assessed through the non-inferiority analysis of access/attendance and learning assessment outcomes. Reasons for differences or similarities in access/attendance or learning will be examined through non-inferiority and treatment effects analysis (as indicated in Table 2) for the community engagement in managing education, parental involvement, community satisfaction with education services, teacher motivation, and teacher performance outcomes.

2. Does the sustainability model substantially improve stakeholders’ confidence in institutionalizing access to primary education into the future?
This will be assessed through our estimates and inference for treatment effects on the sustainability outcomes.

3. Does the sustainability model promote local or national government legitimacy relative to the legitimacy effects under NGO administration?
This will be assessed through our estimates and inference for treatment effects on the civic engagement and village institution legitimacy and the national government legitimacy outcomes.

4. Can the sustainability model operate at costs that are substantially less than NGO administration?
This will be assessed through the costing exercise listed as the “amount spent per student” outcome theme (Table 1 - Panel 7) as well as treatment effects for the ways that funds are applied in communities outcome theme.

Secondary Research Questions
1. Does the sustainability model generate effects that are biased against girls or particular ethnic groups relative to the NGO benchmark?
This will be assessed through our moderator effects analysis with the gender and social institutions moderator themes.

2. Do the access/learning, sustainability, or legitimacy effects depend on the ways that community institutions are activated under the sustainability model relative to NGO administration?
This will be assessed through the moderator effects analysis with the administering organization capacity, community engagement, and the quality of services under NGO administration moderator themes.

3. Does sustainability improve under enhanced variations of the NGO-administered CBE class, in particular, enhancements focusing on teacher recruitment and community engagement?
This will be assessed through the moderator effects analysis with both the moderator effect estimated from the first indicator in the community engagement moderator theme as well as the teacher qualifications moderator theme.

4. Does access/learning, sustainability, or legitimacy depend on other incentives or constraints that communities face for education?
This will be assessed through moderator effects analysis with barriers to education, village demographics, access to economic opportunities, and age of children moderator themes.

C4 Country Afghanistan
C5 Scale (# of Units) 4620 households in 102 clusters of villages
C6 Was a power analysis conducted prior to data collection? Yes
C7 Has this research received Insitutional Review Board (IRB) or ethics committee approval? Yes
C8 IRB Number NYU IRB-FY2016-733
C9 Date of IRB Approval 5/10/2016
C10 Will the intervention be implemented by the researcher or a third party? There are two arms, one is an arm that involves the “sustainability model”, and this is implemented jointly by the research team and the Afghan Ministry of Education, and other other “NGO administration” is administered by the NGOs CARE and CRS.
C11 Did any of the research team receive remuneration from the implementing agency for taking part in this research? No
C12 If relevant, is there an advance agreement with the implementation group that all results can be published? not provided by authors
C13 JEL Classification(s) O22, I25