Enormous progress has been made in reducing child mortality and disability over the last two decades in low- and middle-income countries (LMIC), and childhood vaccinations have played an important part in this success story (Bhutta et al., 2013). They represent one of the most cost-effective health technologies, in that they prevent mortality and disability at relatively low cost (Bärnighausen et al., 2014). Yet, despite the well-documented evidence and consistent investment in national immunisation programmes, the WHO estimates that globally 25 million infants were not fully vaccinated in 2021 (World Health Organisation, 2022). More than 60% of these children live in 10 countries: Angola, Brazil, the Democratic Republic of the Congo, Ethiopia, India, Indonesia, Myanmar, Nigeria, Pakistan and the Philippines. Not since 2009 has the number of children who are unvaccinated been so high (World Health Organisation, 2022). Understanding how to increase the uptake of vaccines is especially pressing following COVID-19, which not only interrupted routine vaccination services, but also highlighted the need to better understand the determinants of vaccine uptake. Indeed, strategies that are successful in improving vaccine coverage for one disease may prove effective in raising acceptance of vaccines to protect against future outbreaks of COVID-19 or other emerging diseases.

The setting for our study is Uttar Pradesh, one of the most populous and poorest states in India, and in which 70% of children aged 12 to 23 months are fully vaccinated against common childhood disease (International Institute for Population Sciences (IIPS) and ICF., 2021). While this represents a marked improvement over the past five years, it is clear that the widespread availability of free immunisation services in public facilities has been insufficient to guarantee high coverage and the benefits of herd immunity in the population. Consistent with this picture is a growing body of evidence that suggests demand-side factors, including poor parental knowledge, distrust, time costs and procrastination, are important barriers to vaccination uptake (Larson et al., 2014; Mills et al., 2005). This, in turn, has prompted investigation of light-touch behavioural interventions that target parents with health information messages, cash or in-kind incentives.

Various systematic reviews and meta-analyses (Shea et al 2009; Johri et al., 2015; Oyo-Ita et al., 2016), as well as more recent studies (Banerjee et al., 2021; Gibson et al., 2017), have evaluated strategies for increasing coverage of childhood vaccinations. The most recent review included 14 studies evaluating a range of interventions such as health education, monetary incentives, home visits and supportive supervision (Oyo-Ita et al., 2016). Some of these interventions were found to be effective in the short term – health education interventions, for example, improved immunisation coverage by 68% – although the quality of evidence varied. Despite this body of literature, the evidence is limited on two important questions.

First, are the effects of demand-side interventions sustained over time? It may be the case that the initial effects of an intervention are attenuated over time, if the intervention merely brings forward vaccinations that would have happened anyway. Under such a scenario, any health benefits of vaccination would be temporary, and so if estimates of the relative cost-effectiveness of the intervention are based on immediate impacts of uptake, then this would overstate the cost-effectiveness of the intervention. An additional perspective on the question of sustainability concerns the persistence of behaviour change in response to temporary or one-off interventions (Celhay et al., 2019). Behaviour change of this nature can be thought of as habit formation, which has obvious relevance for smoking cessation and exercise interventions (Charness & Gneezy, 2009; Volpp et al., 2009). In the context of childhood immunisation, an intervention that led to a sustained change in parental behaviour could deliver benefits – in terms of vaccine uptake – to children who were not born at the time of intervention. Such evidence from follow-up of additional children would serve to increase the cost-effectiveness of an intervention.

Second, who benefits from the intervention? Evidence on heterogenous treatment effects has a number of uses. It can add to existing knowledge and a priori reasoning to help inform policymakers as to who should be targeted by the intervention to maximise uptake, and in the context of immunisation, reach the herd immunity threshold. It can be informative as to which groups the effects of the intervention are most persistent for. It can provide insights into potential inequities, for example by indicating how widely the benefits of an intervention are felt. It can potentially shed light on the mechanisms through which the intervention worked and may be informative as to which groups the intervention has or does not have persistent effects for. Finally, it can offer policymakers insights on how to adapt the intervention or what other forms of intervention may be needed in tandem with demand-side strategies to increase effectiveness in certain subgroups. For example, if the intervention is shown not to work for remote households far from public health facilities, additional strategies such as community outreach or immunisation camps may be needed. While distinct, the two questions are of keen interest to policymakers seeking to implement interventions at scale. Interventions whose effects are short-lived or undermined by the passage of time are of limited value to public health officials. Evidence on what drives variation in intervention effects can provide valuable information on how a public health programme should be designed and delivered outside the confines of a research project.

This paper addresses these questions in the context of a brief health education intervention that was undertaken in rural Uttar Pradesh, India from 2015 to 2016 (Powell-Jackson et al., 2018). The setting has general appeal; it is relevant to low-income countries in which immunisation services are free at the point of use, yet immunisation uptake is low. The intervention provided the mothers of unvaccinated or incompletely vaccinated children aged 0 to 36 months with health information on the benefits of vaccination through home visits. It was implemented as an individually randomised controlled trial and outcomes were measured seven months after the information was given. The intervention led to a large immediate increase in vaccination uptake. Coverage of three doses of diphtheria–pertussis–tetanus vaccine (DPT3) was 28% in the control group and 43% in the intervention group (risk difference of 15 percentage points, p<0.001), and coverage of measles vaccination was 42% in the control group and 64% in the intervention group (risk difference of 22 percentage points, p<0.001). The cost per disability-adjusted life year averted of providing information was US$186, implying that the intervention was highly cost-effective.

We use new data and analytical methods to assess the sustainability and the heterogeneity of effects of the educational intervention. Fieldworkers returned to the study participants approximately 30 months after the intervention, and measured outcomes amongst the original sample of children (hereon referred to as the index children) and younger siblings that were not yet born at the time of intervention. Levels of attrition over the follow-up period were low: 93% of the 722 study participants who were randomised completed follow-up at 30 months, and for this subsample baseline characteristics remained well balanced between the randomised groups.

To study heterogeneity, we estimate individual treatment effects using causal forests (Wager & Athey, 2018), an ensemble machine learning approach that is becoming increasingly popular. The Causal Forest approach is a non-parametric method that builds on causal trees (Athey & Imbens, 2016), which recursively splits individuals into groups with a rule tailored towards the estimation of heterogeneous treatment effects. The method allows for high dimensional interactions between covariates while avoiding overfitting by repeatedly estimating causal trees from random subsets of the data, using the remainder of the data to predict effects, and then averaging the predictions to obtain an overall predicted outcome for each individual under each treatment state (Wager & Athey, 2018). The difference between the two predictions is then the individual level-effect estimate. By considering variation in these estimated individual-level effects with respect to covariates, we can characterise the groups that benefit most/least from the intervention. Alternatively, the individual-level effect estimates for pre-specified subgroups can be aggregated to obtain subgroup effect estimates. Subgroup analyses have traditionally been seen as controversial. For good reason, there is much scepticism of ex post analysis of subgroups because of the risk that results are selectively reported owing to their statistical significance. Studies typically have substantially less statistical power to estimate subgroup effects than overall effects, and must fully acknowledge any lack of precision in reporting and interpreting subgroup-level effects (Burke et al., 2015). To guard against data mining, there are established norms around trial registration and pre-specification of subgroup analyses. But relying solely on information about those subgroups that are prespecified risks discarding potentially valuable information, which may be useful to help target interventions, but also to inform priorities for future research. Moreover, there may be genuine uncertainty regarding what factors may influence effectiveness. When interest is around hypothesis generating, existing approaches may be excessively conservative, and fail to raise new hypotheses. Machine learning allows the researcher to stay neutral as to the source of heterogeneity (i.e. the effect modifiers) and discover patterns in the data by searching over high-dimensional functions of covariates. Such a machine learning method can complement the approach of a priori specification of a limited set of outcome models or subgroups of interest, and the extent to which particular subgroups have been specified a priori, and the use of a theory or intuition to inform the likely direction of effects for particular subgroups is important in interpreting the strength of recommendations for policy-making and further research.1 Effects can still be aggregated for a limited number of pre-specified subgroups to test pre-specified hypotheses (e.g. by mothers perception of vaccine efficacy), and also to build on ‘theory’ or ‘intuition’ for further subgroups that were not pre-specified (e.g. age and vaccination history), in a way that is useful for helping target interventions and future research priorities. The use of ‘honest’ estimation, where the same data are never used for estimation and sample splitting helps to protect against false discovery and yields confidence intervals with correct coverage (Athey & Imbens, 2016). Nonetheless, studies should be careful in the strength of the policy recommendations and further research recommendations that are made from the results of subgroup analyses according to whether the subgroups are pre-specified, and the extent to which they are predicted by theory or prior reasoning.

We report three key findings. First, the large initial effect of the intervention on vaccination outcomes was maintained at 30 months follow-up. The magnitudes of the effect in absolute terms were similar to those previously reported at seven months follow-up, suggesting that the intervention did not simply bring forward vaccinations that would have happened anyway. In this sense, the effects of the brief intervention were sustained. We are unable to make firm conclusions as to the persistence of parental behaviour change because the confidence intervals on the effects of the intervention on vaccination uptake of younger siblings are wide. The effect on uptake of DPT3, for example, was 9 percentage points amongst younger siblings (compared with 32.5% in the control group) but confidence intervals ranged from -2.2 to 20.3 percentage points.

Second, estimates of individual level treatment effects show that the majority of participants benefited from the intervention. For DPT3 vaccination at 30 months follow-up, individual treatment effects ranged from 1.4 to 28.6 percentage points, with a statistically significant effect observed for 56.4% of the participants. For measles vaccination at 30 months follow-up, individual treatment effects ranged from 16.3 to 44.5 percentage points, with a statistically significant effect observed for all the participants, after excluding those children that had already received the measles vaccine at baseline. These findings therefore suggest that the intervention did not cause harm by reducing the chances of children being vaccinated, as would be expected with an intervention of this nature.

Third, we examined whether the heterogeneity was associated with baseline characteristics. When looking at DPT3 uptake, individuals that benefitted most (top 25% of effects) from the intervention tended to be older, had received previous vaccinations in the schedule (e.g. the first and second doses of DPT), were more likely to be located closer to a public rather than private health facility, and had mothers that demonstrated less knowledge about the causes, symptoms and prevention methods for tetanus compared to those that benefitted least (bottom 25% of effects). Age and the receipt of the first and second doses of DPT were also strongly associated with larger effects on measles vaccine uptake, while mothers’ knowledge regarding tetanus or proximity to a public health facility did not explain variation in treatment effects.

In a broad sense, our study contributes to the literature on demand-side interventions for immunisation uptake in low- and middle-income countries, providing novel insights on the persistence and heterogeneity of effects (Banerjee et al., 2020, 2021; Gibson et al., 2017; Johri et al., 2015). There is a small literature on whether temporary incentives can lead to healthy habit formation, such as smoking cessation and exercise (Charness & Gneezy, 2009; Volpp et al., 2009), and a rich body of theoretical work on how to maintain behaviour change (Kwasnicka et al., 2016). There is, however, a need for more evidence on whether one-off health education interventions can lead to sustained changes in behaviour in the uptake of health care technologies. We find little evidence that the intervention caused harm by reducing the chances of children being vaccinated in contrast to another study in India, where a combination of small incentives, reminders and persuasion was found to reduce immunisation rates in some villages, possibly because the interventions crowded out existing intrinsic motivation of parents to vaccinate their children (Chernozhukov et al., 2018).

The paper is structured as follows: Section 2 provides background information on the study setting, and the information intervention. Section 3 discusses the original experimental design and the data. Section 4 describes the econometric methods used. Section 5 presents the results and Section 6 discusses the findings with respect to the limitations of the study and the broader literature.