Is the intended treated group actually treated?

One problem with all projects in which the target population receives support for a certain action is dead weight loss (DWL). DWL is defined as the situation in which the target population would have chosen to be treated even without financial support. In this study, this would imply that pregnant women would have attended a health clinic regardless of the support given. The DWL could potentially be affected by the eligibility criteria for the voucher. The criteria to be eligible for the voucher scheme was that a pregnant woman scored 12 or lower on the poverty grading tool used. To obtain some information about DWL, a survey question was formulated as follows: ‘Would you have visited a health care clinic regardless of the support?’ Of the beneficiaries, 86 per cent indicated that they would still have attended the health facility to access services during their pregnancy regardless of whether they had a voucher. The corresponding number for the non-beneficiary group is very similar, 83 per cent. This result has two implications. Firstly, the voucher programme, for the majority of mothers, mainly represented a transfer from private consumption to public consumption. Secondly, the targeted group for the support, pregnant women who, due to financial restrictions, could not have attended health care clinics, were not reached to a large extent. The latter result could be either due to the project failing to identify the target population with high precision or the target group being too broadly defined. In fact, retrospective analysis of the poverty score shows that, overall, only 32 per cent of the beneficiaries were poor, i.e. scoring 8 or lower on the poverty grading tool, while the remaining 68 per cent were classified as medium or higher on the poverty scale. The eastern region was more affected as only 29 per cent of the selected beneficiaries were deemed poor according to the poverty grading tool compared to 33 per cent in the western region. The targeting of pregnant women who were not poor opposes the very objective of the project and consequently reduces its impact. It is also apparent that about 4 per cent of the project beneficiaries were ineligible. These had scored over 12 marks and yet they benefitted from the project. This is because, in some cases, poverty assessment was not undertaken at the mothers’ homes although many of the poverty parameters considered could only be assessed there, such as the mothers’ sanitation facilities, water source and shelter. Interviews with the mothers revealed that, when the vouchers were scarce, some mothers looked for the community-based distributors and purchased the voucher without proper poverty assessment on their premises, rendering the assessment ineffective.

To summarize, the finding is that the targeted group of pregnant and vulnerable women who, due to financial restrictions, could not afford to visit a health care clinic – could have been reached to a larger extent. This could for example have been achieved by having a lower eligibility level on the poverty grading tool. Since vouchers were scarce in many regions a better targeting could possibly have been achieved by only making pregnant women that scored 8 or lower on the poverty grading tool eligible for the voucher scheme. The fact that a higher level on the poverty grading tool was chosen for eligibility have most likely had an influence the impact of the project. Especially so since there is a non-ignorable number of treatment group members (voucher) who, in the absence of a voucher, still would have attended a health care clinic. The result reported in the following section should therefore be viewed as a lower limit regarding the impact size.

Treatment effect on the treated

The objective of this study was to assess the impact of using a voucher on the beneficiaries of the URHVP. The outcome measure used was the survival of babies during pregnancy and birth by estimating the average difference in the probability of survival between beneficiaries and non-beneficiaries. To measure the effect of the voucher project, we made use of very rich and detailed data from a survey collected especially for this purpose. We applied a matching strategy to compare beneficiaries with non-beneficiaries and to estimate the effect of using the voucher.

A brief look at the raw data on the outcome indicator, namely the survival of the baby during pregnancy and birth, shows that 97 per cent of the babies of beneficiaries survived compared with 93 per cent of similar non-beneficiaries.Footnote 8 This preliminary analysis indicates that beneficiaries were better off since their babies had a higher chance of surviving. However, these groups could differ in various ways that could influence their participation in the voucher programme. To estimate the effect, we applied our matching strategy described above. We performed a PSM that matched the beneficiary group with the non-beneficiary group based on women’s poverty score and the following variables: age, educational level, religious denomination, marital status, number of pregnancies, district, average household expenditure and year of pregnancy. The result from this main analysis is presented in Table 4 below.

Table 4 Effect of having participated in the voucher programme, ATT

The result shows a positive impact of the voucher project, with an average treatment effect of 0.054, which means that babies of the beneficiaries have a 5.4% point higher probability of survival during pregnancy and birth. The infant mortality rate for non-beneficiaries in the target group of poor and vulnerable women is 8.2 per cent. A reduction of 5.4% points means that, due to the project, the infant mortality rate in the targeted group fell to less than 3 per cent, that is, a reduction of 65 per cent. This also means that the infant mortality rate in the targeted group was somewhat lower than the average infant mortality rate in Uganda, which is 3.4 per cent.

Differences in outcomes between the western and eastern regions

Interviews held with the project management at the Ministry of Health (MoH) and MSU showed that the eastern region had been identified at the mid-term review as having a lower likelihood of obtaining good outcomes than the western region. For example, the mid-term review mission indicated that there was a lower uptake of vouchers in the eastern region and fewer health units providing comprehensive obstetric care. It was also noted that the western region had a previous voucher scheme called Child Plus, dealing with family planning issues, which might have improved the chances of good implementation of the studied voucher scheme. However, interviews with MSU indicated that the structure and numbers of the project staff within the two regional offices were the same. In addition, the work plans provided showed the same allocation of resources (time, manpower and money) for monitoring and evaluation, training and mentorship for both regions. As noted earlier, this is also mentioned by the World Bank in a mid-term review. The review notes especially the poor management of referrals in Eastern Uganda, and the conclusion is that there was a risk of not achieving the project outcome related to the handling of pregnancy complications in the eastern region.

Despite this prior knowledge, no measures were placed within the design of the project to avert this imbalance, which makes it even more interesting to estimate the effect of the voucher project separately for the western and eastern regions. The result from this exercise is presented in Table 5.

Table 5 Effect of having participated in the voucher programme for the different regions, ATT. Standard errors within parentheses

A comparison of the impacts of the project in the western and eastern districts indicates that babies of beneficiaries in the western region have a 7.6% point higher probability of survival during pregnancy and birth than those of non-beneficiaries. However, for the eastern region, there is no evidence of a significant difference in the probability of survival of the babies of beneficiaries and non-beneficiaries.

It is likely that the indicated problems in the implementation of the project in the eastern region influenced the efficiency of the voucher scheme. It is also worth noting the strong and significant positive effect of almost 8 per cent in the western region, suggesting a large difference in the possibility of a child surviving pregnancy and delivery if the mother was a beneficiary of the studied voucher project.

Sensitivity analysis

A sensitivity analysis was conducted to establish the stability of the estimated effect of the voucher project. To determine whether our results are stable, we estimated the effect on infant mortality using only the first pregnancy during the studied period between 2016 and 2019. This eliminated the risk that some mothers who gave birth multiple times during the studied time frame influenced the results. However, we could only include 1,418 births instead of the 1,881 in our complete sample. The result of this sensitivity analysis is provided in Table 6.

Table 6 Effect of having participated in the voucher programme: only the first birth in the studied period 2016–2019, ATT

The conclusion from Table 7 is that there is still a positive and statistically significant effect on the survival of babies born to mothers who participated in the voucher project. The probability of survival is 3.5% points higher for babies of participating mothers than for babies of non-participating mothers.

We also estimated several different selection models. In the first basic model (Model 1), we only included the poverty score to match beneficiaries and non-beneficiaries. The logic is that much of information about the mothers is given by the poverty rating score.

We then increased the number of control variables to observe whether our estimated effect, that is, the difference in the survival of babies between beneficiaries and non-beneficiaries, changes when different variables are added. The results of this analysis are presented in Table 7.Footnote 9

Table 7 Effect of having participated in the voucher programme on infant mortality. Different model specifications, ATT. Standard errors within parentheses

As can be seen from Table 7, the results are positive and significant in all the estimated models. The estimated survival rate of babies of beneficiaries is between 4 per cent and 7 per cent higher than that of babies of comparable mothers who did not use a voucher. It can, for example, be noted that the estimated effect of using a voucher on the survival of the child when we only control for the poverty grading score is around 5 per cent and statistically significant at the 1 per cent level. This is a very similar result to our main result, presented in Table 5. It is likely that the poverty score captures the differences between beneficiaries and non-beneficiaries to a large extent and that additional variables do not add much information. The main point is that our results are very stable and show a positive and statistically significant effect for all the estimated models.

We can also compare the result from the matched analysis with the estimated effect from a crude Probit model without matching but with the same explanatory variables. The Probit model produces an estimate of the treatment of 0.044 with a standard error of 0.01. This is similar to the result in the first columns in Table 7 above. The matched result changes from the crude estimate, albeit not by much, as household expenditures are introduced into the selection model. This is reassuring since large differences would indicate that selection might have been present in the distribution of the vouchers. If the vouchers had been randomly distributed, the crude Probit and the matched results would have been similar, which we are not too far from now.

Cost efficiency of the voucher project

According to UNICEF, 23 per cent of children in Uganda live in households that are below the poverty line [22]. In this section, we present a rough calculation of the cost efficiency of introducing a programme such as the studied voucher project nationwide in Uganda. Such a project would target only poor households. For this calculation, we assume that the number of births for women in poverty are the same as the number for the rest of the women in Uganda. In 2019, there were approximately 1.65 million births in Uganda.Footnote 10 This means that approximately 379,500 children (23 per cent * 1.65 million births) were born in households that were below the poverty line and would therefore be eligible for a nationwide voucher scheme.

Furthermore, we assume the cost of the services provided by the voucher project to be around US$60.Footnote 11 This cost does not include the cost of surgical deliveries by caesarean section, which is approximately US$130. The total yearly cost of the of the introduction of a nationwide voucher style project would be around US$22.8 million (US$60 * 379,500 births). Our calculated infant mortality rate for non-beneficiaries is 8.2 per cent, which is a reasonable number compared with the infant mortality rate of around 3.4 per cent in Uganda considering that the focus in this study is on poor and vulnerable women. Without a voucher project, we therefore expect that around 31,100 (379,500 * 0.082) babies will not survive birth. If the voucher was to be implemented for every woman in this group, we would instead expect that 10,600 (379,500 * 0.028) babies would not survive birth. Our results from this study therefore indicate that an intervention that would give a health voucher to all poor pregnant women in Uganda has the potential to enable around 20,500 (31,000–10,600) more children to survive birth. Considering our estimated cost of such an intervention, this implies an estimated cost of around US$1,100 per child surviving birth.