記住我
The Zindagi Mehfooz (safe life; ZM) electronic immunization registry (EIR) is a comprehensive suite of digital health interventions that aims to improve equitable access, timeliness, and coverage of child immunizations through a smartphone-based app for vaccinators, web-based dashboards for supervisors and managers, text message alerts and reminders for caregivers, and a call center. It has been implemented at scale in Sindh Province, Pakistan.
Electronic immunization registries (EIRs) are digitized case-based immunization record–keeping systems designed to improve the collection, analysis, and use of immunization program data to improve the efficiency, equity, and coverage of immunization programs []. A 2015 systematic review [] of aggregate immunization information systems in high-income settings highlighted the main pathways through which such systems improve immunization outcomes, such as coverage, although only 1 study in Australia measured such an association. The benefits of aggregate systems align with more recent reviews and studies of EIRs that record and track data for individual children and generate aggregate data for immunization program monitoring and planning []. These benefits include client reminders, client vaccination status, missed vaccinations, and vaccination program management and accountability [,]. Improved access to better data can support evidence-based decisions, planning, and operations for the immunization program, which, in turn, will lead to better delivery of immunization services []. When EIR data are triangulated with other primary health care services, recorded births, and population denominators, EIRs can enable population-level analysis to identify areas where the number of children vaccinated is lower than the target population [,]. In this way, EIRs can improve the identification and reach of zero-dose children (ie, children who have not received a diphtheria-pertussis-tetanus [DPT] 1 vaccine) and provide targeted service improvements and extended outreach services to those areas and communities. EIRs can also address demand-side challenges by sending automated reminders to caregivers, providing access to educational messages and service-related updates and improving the quality of services at vaccination centers [].
Because EIRs are intended to support immunization programs with improved data-driven decision-making, the evaluation of data use is also important []. Drivers of EIR use include organizational factors (supervision, support, resources, facility type, location, and volume of clients), technical factors (infrastructure, electricity, and internet connectivity), and behavioral factors (training, capacity, and motivation of system users) []. A recent study [] using EIR data to evaluate the timeliness of vaccination systematically found mixed results, with the conclusion that the overall reliance of the study on system-level data was insufficient to evaluate its impact on immunization outcomes. A review of the evidence for EIRs reveals a moderate number of process evaluations, few pilot studies, and no large-scale effectiveness studies of EIRs [].
The ZM-EIR comprises complementary digital health interventions that satisfy and go beyond the conventional definition of an EIR. The ZM-EIR aims to strengthen Expanded Program for Immunization (EPI) service delivery and the broader health system by addressing both supply- and demand-side barriers that contribute to suboptimal use of immunization services. On the supply side, it is designed to enable health workers to track the immunization history for individual children and assess catch-up immunization schedules, support motivation among health workers by reducing the excessive burden of paperwork and demonstrating the value that quality data can contribute to their workflow [], and reduce the occurrence of poor reporting, management, supervision, and monitoring of vaccinators. On the demand side, the ZM-EIR is designed to overcome challenges associated with low uptake of immunization services due to a lack of awareness and motivation among parents, as well as the inability to remember vaccine appointments.
Pakistan represents 1 of the 6 countries in the world with the highest concentration of zero-dose children. In 2012-2013, only about half (58.3%) of children aged 12-23 months in Pakistan received all basic vaccinations [], according to the Pakistan Demographic and Health Survey 2012-2013, with marked geographic variation across the country. Sindh Province recorded full immunization coverage of 35% [] in 2014, according to the Multiple Indicator Cluster Survey (MICS) conducted in 2014. Sindh Province is home to 47.9 million people and comprised 28 districts during the time frame covered by the evaluation, with around 1500 fixed immunization centers that conduct outreach activities, as well as over 200 private clinics []. The provincial capital, Karachi, is Pakistan’s largest capital city and economic hub representing a large urban population. Most childhood vaccines in the district are delivered through public sector immunization centers or outreach activities managed by the national EPI [].
As of August 2022, the ZM-EIR was scaled to cover all of Sindh Province, including 2 new districts that were not covered during the evaluation period. The ZM-EIR has also been introduced in other areas of Pakistan, including the Federal Capital (Islamabad), Khyber Pakhtunkhwa Province, and the Gilgit-Baltistan region, and has been used by over 5000 vaccinators at public and private immunization clinics to enroll over 7 million children.
This paper aimed to summarize findings from a mixed methods evaluation of ZM-EIR implementation in Sindh Province. Retrospective evaluations of program implementation are difficult to design and carry out due to the lack of counterfactuals and recall biases. Habicht et al [] proposed “adequacy” as the minimum level of evidence in support of a program’s impact part of an evaluation. We used a range of data sources to support the hypothesis that the ZM-EIR intervention may have contributed to the outcomes achieved by the immunization program. We showed that ZM intervention activities were implemented adequately, accepted (provider and caregiver perceptions of the ZM-EIR), and adopted (vaccinator use of the ZM-EIR) and may have contributed to improved coverage (changes in antigen coverage) and timeliness.
The ZM-EIR includes the following functionalities:
Web interface for data visualizationMobile-based data entry and access for vaccinatorsGeospatial data for vaccination events2-way interactive SMS reminders for parents/caregiversCall center to answer queries of parents/caregiversOffline mode to work in areas of poor connectivityChild registry for enrolling newborns or never-vaccinated childrenImmunization decision support system (iDSS) to guide vaccinators on routine and catch-up immunizationsDefaulter reports for vaccinatorsGamified videos for training vaccinatorsArtificial intelligence (AI)-based predictive analytics to identify children highly likely to drop outAI-based chatbot to address caregiver queries and concerns in real timeGlobal System for Mobile Communications (GSM)-based tracking of vaccinators during work hoursQuick response (QR) code–based tracking for unique identification of children and womenThese functionalities center around a mobile phone–based registry with a suite of complementary interventions. More details and underlying research that have contributed to the development of the intervention over the past decade have been published elsewhere [-]. See for a graphical overview of the ZM-EIR intervention. Although the intervention can capture the immunization of pregnant women, this research focused on its use for routine immunization of children.
Figure 1. Overview of the ZM system. EIR: electronic immunization registry; EPI: Expanded Program for Immunization; iDSS: immunization decision support system; QR: quick response; ZM: Zindagi Mehfooz (safe life). The ZM mobile Android app is used to capture data for each point of contact between the immunization service, caregivers, and their children. At the time of first enrollment, the child’s data are entered on the paper EPI card and into the ZM-EIR. A unique QR code is printed and affixed to the home-based paper record and linked to the electronic system. During follow-up visits, the child’s ID is retrieved by scanning the QR code, giving the vaccinator access to the child’s vaccine schedule. For each vaccination received, the data are entered into the ZM app and linked to the server. All data entered into the ZM app are collated on a central web-based dashboard, which supervisors can use to monitor progress, identify defaulters, and take corrective action to achieve targets. The decision support system (DSS) uses a built-in algorithm to automatically calculate which vaccine is to be administered based on the child’s date of birth and immunization history. This minimizes the chance of missed immunization and enables the vaccinators to provide age-appropriate and timely immunization. In addition to recording vaccination activities within the ZM intervention, vaccinators were required to document their work in paper-based registers that were then manually tabulated and submitted as part of monthly reporting activities.
To remind caregivers of when they need to next visit the clinic, up to 3 automatic personalized SMS reminders are sent 1 day before, on the day of, and 6 days after the scheduled immunization date (if the child did not make the scheduled appointment). A child is considered to have completed the full course of immunizations after the second dose of the measles vaccine.
In addition to the inclusion of standard EIR features (as illustrated in ) [], the ZM-EIR includes AI-based predictive analytics to identify children more likely to drop out; automated SMS reminders; a call center; and an AI chatbot for caregiver questions, concerns, and complaints. The suite of interventions also includes geospatial data for vaccination events, geospatial data to track each child’s immunization progress, real-time tracking of vaccinators with Geographic Information System (GIS) tracking to improve monitoring and accountability, simultaneous supervision of multiple teams, monitoring of attendance and mobility of field staff, and gamified videos and training for vaccinators.
Table 1. Comparison of ZMa and ideal EIRb characteristics and functionalities.FeatureZM-EIRIdeal requirements of an EIR []Registration and searchThe ZM-EIR is a registry to enroll all newborns or never-vaccinated children with a unique ID/QRccode assigned to each woman and child; a web interface allows data visualization, mobile-based data entry, and access for vaccinators.Enrollment at birth, client management, unique and unequivocal IDPatient recordsRegistration includes all relevant demographic and location data of the patient and facility with geolocation, barcoded vaccine vials linked with vaccine records, and an offline mode for areas with low mobile/data coverage (not linked to other health areas).Individual demographic data, vaccine event data, nonroutine (campaign and outreach) vaccine events, and linkages to other health areasVaccination monitoring and follow-upThe iDSSd guides vaccinators on routine and catch-up immunizations and provides defaulter reports for vaccinators and interactive SMS reminders for parents/caregivers.Clinical decision support, identification of undervaccinated children/missed vaccination, and reminder and recall messagesRegistryVaccinators are registered within the system and assigned to a health facility registered to a district management unit.Health facility registration and managementStockThe vaccine stock is tracked through the number and type of vaccinations given by vaccinator, facility, and district.Stock managementData and reportingGeospatial analysis maps pockets of undervaccinated children with exact locations to enable follow-up with households refusing vaccination. There is a call center for complaints and adverse events following immunization (AEFI) reporting.Data aggregation at different geographic and administrative levels, adverse event reportingOther system requirementsThe ZM-EIR follows industry standards for the data security and privacy platform interoperable withDistrict Health Information System 2 (DHIS2) and the government’s EPIemanagement information system (MIS) deployed at full scale in all districts of Sindh Province by over 3096 vaccinators at 1694 public and private sector immunization clinics to enroll 3.8 million children and 1.3 million women, with 31 million immunization visits recorded (as of 2021).Data exchange and interoperability, offline capability, alignment with international standards, data privacy and security, scalability and capacity, usabilityaZM: Zindagi Mehfooz (safe life).
bEIR: electronic immunization registry.
eQR: quick response.
diDSS: immunization decision support system.
eEPI: Expanded Program for Immunization.
Study DesignThe pre-post mixed methods study design measured changes in the proportion of fully immunized children aged 12-23 months from 2014 to 2019. Fully immunized children are those who have received the complete recommended schedule of 5 antigens by 23 months of age. The pentavalent (Penta) vaccine refers to a DPT-Hib-HepB (where Hib-HepB refers to the Haemophilus b–hepatitis B conjugate) combination in this context. Additional aims and objectives of the study were assessed through qualitative research (observation and in-depth interviews), secondary data analysis (MICS from 2014 and 2018-2019), and comparison between districts and health facilities with high, moderate, and low ZM-EIR compliance (based on data from the platform or app of log-ins and entries of child encounter information). The results are presented later based on the study aim and objective. Experimental or quasi-experimental study designs were not feasible as the ZM-EIR was implemented throughout Sindh Province and a suitable comparator was not available.
Patient and Public InvolvementThere was no patient or public involvement in the design or implementation of the study. Caregivers, health workers, and health administrators were included in the qualitative research component as key informants after they provided informed consent.
Ethical ConsiderationsThe study received Institutional Review Board (IRB) approval from the IRD (approval number IRD_IRB_2021_11_001) and was registered with the ISRCTN registry (#23078223). Informed consent was obtained from all key stakeholder groups prior to interviews.
Data Sources and AnalysisQuantitative Data CollectionThe 2 sources of quantitative data were the MICS conducted across the state of Sindh, Pakistan, by the United Nations Children’s Fund (UNICEF) from 2014 and 2018-2019 and the ZM registry database. Details of the MICS are provided on the UNICEF website. The indicators across the study aims and objectives are listed in .
Table 2. Indicators included in quantitative analyses.IndicatorDefinitionNumeratorDenominatorData sourceComplianceThe vaccinator updates/uploads 1 record per vaccination day.Number of days the vaccinator logged in to the ZMa app and uploaded/updated at least 1 child recordNumber of vaccination days for that health center/vaccination centerZM databaseZM registry coverageN/AbTotal number of children receiving any immunization reported in the ZMEstimated monthly target population of children in Sindh Province derived by the EPIc in close collaboration with the ZM teamZM child records databaseZM registry useThe proportion of vaccinators trained to use ZM upload details for at least 1 client per day in Sindh Province from 2017 to 2019.Number of days the vaccinator logged in to the ZM app and uploaded/updated at least 1 child recordNumber of vaccination days for that health center/vaccination centerZM vaccination staff databaseZM dashboard useThis is the number of unique user (manager/supervisor) log-ins to the dashboard.Number of log-ins per district per yearN/AZM management staff usage of dashboard derived from the ZM databaseFully immunizedIf the child has received the BCGd vaccine at birth and OPVe 1-3, Pentaf 1-3 vaccines, PCVg 1-3, and the measles vaccine 1 at 12-23 months, the child will be considered fully immunized.Number of children 12-23 months old who received the BCG vaccine at birth, OPV 1-3, Penta 1-3 vaccines, PCV 1-3, and the measles vaccine 1Sample of children 12-23 months old as part of the survey in Sindh ProvinceMICSh 2014, 2018-2019Zero dosesA child aged 6-23 months has not received the first dose of the DPTi 1/Penta 1 vaccine.Number of children who did not receive the DPT 1/Penta 1 vaccineSample of children 12-23 months old as part of the survey in Sindh ProvinceMICS 2014, 2018-2019Timely immunizationA child has received the dose of antigen at the recommended age specified in the EPI schedule.Number of children 12-23 months old who received the BCG vaccine at birth, OPV 1-3, Penta 1-3 vaccines, PCV 1-3, and the measles 1 vaccine as per the timeliness criteria of the Pakistan/Sindh Province EPISample of children 12-23 months old as part of the survey in Sindh ProvinceMICS 2014, 2018-2019aZM: Zindagi Mehfooz (safe life).
bN/A: not applicable.
cEPI: Expanded Program for Immunization.
dBCG: Bacillus Calmette-Guérin.
eOPV: oral polio vaccine.
fPenta: pentavalent.
gPCV: pneumococcal conjugate vaccine.
hMICS: Multiple Indicator Cluster Survey.
iDPT: diphtheria-pertussis-tetanus.
The ZM platform uses multiple sources to register children, including birth cohort registration, the recording of pregnancies and births, the EPI’s estimated targets, and the date of enrollment. The analysis reflected the vaccine-wise coverage for the age group of 12-23 months. The registry-based coverage was calculated using the enrollment numbers for each calendar year birth cohort as the numerator and the annual targets provided by the EPI (ascertained population based on modeling using birth rates and recent census figures). Furthermore, the target population of children was derived based on the enrollment of children at the time of birth through a birth cohort registration and an EPI schedule–based continuum-of-care mechanism. The numerators were based on aggregate numbers for the different antigens and did not distinguish the age of the child for the receipt of vaccination, which is reflective of immunization data from administrative sources.
We analyzed MICS data using the design considerations for weighting, adjustment of clustering at the level of primary sample units, and strata for survey errors according to guidelines provided by the MICS. We presented survey design–adjusted estimates of vaccine coverage with 95% CIs.
Timeliness was calculated according to the following recommended age schedule. It is worth noting that timeliness for the Penta 3 vaccine does not correct for any minimal interval with previous doses, which makes being untimely almost normal if Penta 1 vaccination is already late ().
Table 3. Timeliness schedule of vaccination.Antigen and doseMinimum ageRecommended ageTimely schedule (days)BCGaBirth0 weeks/0 days0-28Pentab 1/OPVc 1/PCVd 16 weeks6 weeks/42 days39-70Penta 3/OPV 3/PCV 314 weeks14 weeks/98 days95-126Measles 16 months9 months/274 days180-302aBCG: Bacillus Calmette-Guérin
bPenta: pentavalent.
cOPV: oral polio vaccine.
dPCV: pneumococcal conjugate vaccine.
Qualitative Data CollectionQualitative data collection was conducted in 3 districts in Sindh Province, selected for the high, medium, and low immunization coverage rates according to the analysis of ZM-EIR system data. Direct observations of immunization services were conducted for 1 day per facility in 6 health facilities and 3 district offices. Within each district, 2 health facilities were selected based on high and low ZM-EIR compliance. At each health facility, 2 vaccinators and 2 caregivers (1 male and 1 female) who were attending the facility for their child’s immunization were interviewed. Selection was based on their availability and consent to participate during field visit days. For each district, the managing supervisor and the district field coordinators and EPI managers were also interviewed to capture views of the leadership, along with vaccinators and clients. A structured discussion guide was used to conduct the interviews, led by a researcher with experience in qualitative methods. Qualitative data were collected from caregivers of children <2 years old (n=12), vaccinators (n=12), supervisors (n=3), district-level EPI managers (n=7), provincial EPI managers (n=2), and ZM-EIR implementation and data teams (n=4). The interviews were audio-recorded and transcribed for qualitative analysis. Direct observations of immunization services were also conducted for 1 day per facility in the 6 health facilities and 3 district offices.
As guided by the Consolidated Criteria for Reporting Qualitative Research (COREQ) checklist, the research team comprised a Pakistani female principal investigator and health system researchers trained in qualitative methods, with master’s degrees in public health and over 10 years of experience. There was no previous relationship with the study participants. District field supervisors facilitated contact with health facilities, and health facilities facilitated contact with caregivers. The sampling was purposive to ensure that all key stakeholder groups engaged in using or having data captured within the ZM-EIR: EPI managers, district health officers, EPI supervisors, vaccinators, and caregivers were included in the sample. Interviews were conducted face to face either in offices/places of work or in health facilities. There were no refusals, and no nonparticipants were present at the time of the interviews. Content analysis was conducted using a framework developed through the content captured within the in-depth interviews. Interview guides were developed and tested prior to data collection. There were no repeat interviews. Saturation was achieved through the structured sampling and interviews conducted. Interviews were approximately 30 minutes in length. They were audio-recorded and field notes captured. Microsoft Excel was used to organize data by theme for ease in the synthesis of findings and reporting. Thematic analysis was conducted using grounded theory through a 2-step process: (1) iterative debriefs conducted concurrently with data collection and (2) coding of transcripts and generation of subthemes and themes. A theoretical framework based on these themes and subthemes was then developed in line with the research study objectives and used to analyze and present findings.
Vaccination staff training on the ZM-EIR started in quarter 3 of 2017 as part of a district-wide planning with the Sindh Provincial Department of Health. Data on the proportion of vaccination staff trained on the system from the total were not available. However, all the 1307 staff members nominated for training were trained across 52 batches between September and December 2017. Training batch sizes ranged from 12-40 participants per training session.
The Sindh Provincial EPI defines compliance with the ZM-EIR system by vaccinators as 1 upload or update for at least 1 client per vaccination day. Compliance estimates include instances in which a vaccinator may be present at the health facility but no child is brought in for vaccination. Findings suggested that across 27 districts that were active during the time frame covered by the study, compliance rates ranged from 46% to 85%. The average compliance increased from 40% in 2018 to 57% in 2019. Data are available in .
Trends in compliance were assessed based on demographic characteristics of vaccinators. Compliance and usage during 2017-2019 were slightly higher among urban vaccination staff compared to their rural counterparts. Compliance among female vaccinators was higher compared to males, even though the proportion of females in the vaccination staff was only 15%. There was a gradual increase in compliance in all age-groups during 2017 to 2019. Younger age groups (less than 40 years old) showed improvement from 40% in 2017 to 49% in 2019, with the highest proportional increase observed for those aged 26-30 years (49%) and those aged 18-25 years (47%). A more modest increase was found for vaccinators in higher age groups, with the 51-55-year group showing only a 28% increase.
Stakeholder Perceptions of the ZM-EIRStakeholder perceptions of the ZM-EIR were assessed for each set of functions within the ZM-EIR suite of digital health interventions, including the perceived benefits and challenges of use to improve immunization service delivery (). Overall, all stakeholder groups interviewed had a favorable impression of the ZM-EIR and the associated suite of interventions. Provincial managers felt that the real-time access to vaccination data facilitates longitudinal monitoring of vaccination coverage over time and across geographic areas. They further noted that expansion of the ZM-EIR would allow this to occur across the country as it would enable the provinces to see and evaluate each other’s coverage. The ZM-EIR was thought to provide a blueprint of what a national repository of EPI records could look like and further be a mechanism for improving supply monitoring and fraud prevention. As the primary end users of the ZM-EIR, vaccinators expressed their satisfaction with ZM-EIR features, specifically the use of automatic lists generated of those who may have missed a scheduled vaccine dose or may not have received any vaccines (defaulter and zero-dose lists).
Table 4. Key informant general perceptions of the ZM-EIRa.Key informantExample quotationsCaregiver“...It is very helpful. Otherwise, we are busy with our work and forget about it. When we receive a message, then we know we have to get the child vaccinated soon.”aZM-EIR: Zindagi Mehfooz (safe life) electronic immunization registry.
Registry CoverageFrom 2017 to 2019, the ZM-EIR immunization registry enrolled nearly 5.4 million children and 1.9 million married women of reproductive age (15-49 years).
Gender-Related Barriers to Registry UptakeGender was identified to play a significant role in ZM-EIR uptake by caregivers owing to its embeddedness within the local, rural contexts. There is often a clear distinction between a caregiver’s interaction with male vaccinators and that with female vaccinators. One key informant emphasized the need for female health workers who can bond with female caregivers (who are usually more responsible for taking care of their children than male caregivers are) to resolve any resistance or anxieties regarding vaccinations to further expand immunization efforts. A female caregiver shared:
The female health worker can come inside the house, and we [caregiver] know her, so it is good that a female health worker comes. If a man comes, then it is quite difficult.The provincial managers shared that the ZM-EIR intervention provides defaulter and zero-dose lists, which helps them meet their target and improve performance. It enables those who supervise to encourage the vaccinators to bridge the gaps in the coverage of immunization targets. The data are viewed as reliable due to the feature allowing the system to operate in both offline and online modalities, and the use of the platform is not dependent on the availability of internet services. These data are then communicated to other responsible representatives at the district and provincial levels. Among the stakeholders who can access the data reports and dashboards, the role of supervisors has been seen to be relatively limited compared to others. The supervisor’s overall responsibilities include monitoring and providing support to vaccinators, checking the vaccination stock, and planning and conducting field visits. For their limited use of the ZM-EIR app to monitor the activities of vaccinators in their respective district areas, it was stated that newer and younger supervisors can use the app comfortably, while older supervisors find the app difficult and less engaging. Coordinators and supervisors also suggested that the latter be given access to a separate app to monitor vaccinators so that they do not have to rely on district managers for the data.
Observation of Data Entry and UseAmong the health facilities observed, half were clean, organized, and relatively well maintained. The other half were seen as unclean, “cluttered,” and “with broken furniture.” The number of rooms in a health facility ranges from 2 to 7. All the health facilities provide vaccination and other health services. Most of the health facilities have 2 vaccinators present. Of these, 1 (50%) in each facility uses the ZM-EIR for registration and defaulter lists. In all 3 (50%) facilities, it was reported by vaccinators that when the caregivers would bring their children for vaccination, they did not use the ZM-EIR for registration in real time, but they recorded information in the paper-based register system and later entered it into the electronic app. Only 4 (67%) of the 6 facilities had electricity, 1 (25%) being backed by a solar power system. Network connectivity was limited in all facilities, which also hindered the use of the ZM-EIR in real time. All district offices observed had 1 supervisor on-site with access to a computer/laptop in the office, which enabled access to web-based ZM-EIR dashboards. The supervisors reported engaging in daily monitoring of the vaccinators, whereby they tracked the vaccinators’ attendance, location, and registration. No other use of the ZM-EIR was reported or observed ().
Table 5. Key informant views of ZM-EIRa data use by key stakeholder group.Key stakeholder groupExample quotationsDistrict field coordinators and managers“When we have their attendance in the morning, the supervisors call right then to ask why this vaccinator isn’t marked present, tell their timing, and all. So, we talk to them daily.”aZM-EIR: Zindagi Mehfooz (safe life) electronic immunization registry.
bUC: Union Council.
Vaccination Coverage and TimelinessThe vaccine-wise coverage across this evaluation’s time frame of consideration (2017-2019) reflected that since its inception for scale-up in the province and phase-wise implementation in all districts, increases were observed for all vaccines (the EPI in Sindh Province had measles 1 as the last vaccine as per the child vaccination schedule till 2018. Subsequently, measles 2 was added as part of the EPI schedule; however, given the evaluation’s considered time frame of 2017-2019, the quantitative analyses focused on measles 1). Findings suggested that improvements in coverage were achieved for BCG (+10.1%), Penta 1 (+8.8%), Penta 3 (+7.4%), and measles 1 (+5.5%) vaccines. Vaccination timeliness was assessed by antigen by comparing the proportion of children 12-23 months old immunized across MICSs in 2014 and 2018-2019. Findings suggested that improvements in timeliness were achieved for BCG (+10.0%), Penta 1 (+3.1%), Penta 3 (+2.0%), and measles 1 (+0.7%) vaccines. However, only the improvement in timeliness for the BCG vaccine was statistically significant ().
Table 6. Vaccination coverage and timeliness in Sindh Province between 2017-2018 and 2014 (MICSa).VaccinationPrevalence (95% CI)Increase (%)aMICS: Multiple Indicator Cluster Survey.
bBCG: Bacillus Calmette-Guérin.
cPenta: pentavalent.
Additional analyses were conducted to compare increases in immunization coverage in Sindh Province, along with Punjab Province, which implemented a different EIR as part of its EPI. The analyses indicated increases in immunization coverage, making the increases in immunization coverage and timeliness difficult to attribute to the ZM intervention ( and ).
Table 7. Vaccination coverage in Sindh and Punjab Provinces between 2017-2018 and 2014 (MICSa).VaccinationSindh ProvincePunjab ProvinceaMICS: Multiple Indicator Cluster Survey.
bBCG: Bacillus Calmette-Guérin.
cPenta: pentavalent.
Table 8. Key informant views of the relationship of the ZM-EIRa to immunization coverage and equity.Key informantExample quotationImmunization program performance‘…In what [performance] category are [] UCsb, that is, whether [they] are in red, green, blue, or yellow…every UC-wise, every vaccinator-wise, every town-wise, every district-wise, we are getting all the data.”Immunization coverage“Because of ZM, the EPIc office has found out that we have 114 vaccinators. If only 67 vaccinators out of them are working, they ask your district where the rest of your vaccinators are. They ask us to enroll them, assign them a field, and send them to work. So now, I have over a hundred vaccinators enrolled, and they work themselves after returning. Coverage will thus ultimately increase.”Immunization equity“In outreach, more female children are being vaccinated because they find it easy that they can get the immunization service at their doorstep…we can see [in defaulters list] that more females might be lagging behind; as a result of this, there is catch-up, and we do see better coverage among the female population.”Immunization timeliness“…Three SMS reminders are sent so it is not that you only get one SMS reminder for the child’s appointment day…there are these three different time points as a constant nudge to the caregiver to bring the child for immunization at the time that he is required to show up.”aZM-EIR: Zindagi Mehfooz (safe life) electronic immunization registry.
bUC: Union Council.
cEPI: Expanded Program for Immunization.
From a ground-level perspective, vaccinators shared that the selection of local people for authority positions in rural governments has increased the trust of the people in the government, leading local caregivers to trust health authorities and their immunization efforts. In general, the vaccinators also believe that there has been an increase in awareness, coupled with an increase in diseases, which has had a concurrent effect on improved vaccination rates among rural communities.
Managers shared that the timeliness of immunization may be due to the DSS built within the app. Although SMS reminders as a ZM-EIR feature are more caregiver oriented, the DSS within the app integrated as an explicit function is more vaccinator oriented. In instances where vaccinators have not had substantive training, they are likely to get confused about which vaccine to give a child at a specific time.
Zero DosesThe prevalence of zero-dose children aged 6-23 months, defined as not receiving at least 1 dose of the Penta vaccine, went down from 36.3% to 28.5% (at 95% CI) between 2014 and 2019. Relative decreases in Punjab Province make attribution to the ZM intervention difficult to ascertain.
This external evaluation of the ZM-EIR and the associated suite of digital health interventions for childhood immunization in Sindh Province, Pakistan, highlights important insights and recommendations across stakeholder groups related to the benefits of an EIR, namely the availability and use of data through alerts and reminders to caregivers, identification of defaulters, supervision of vaccinators, and overall management of vaccination program activities [].
Qualitative findings support this assertion; indicating that overall, the stakeholders interviewed had a favorable impression of the ZM-EIR suite of interventions and the perceived benefits of ZM defaulter lists on improved immunization coverage, as well as decreased percentage of zero-dose children. Several suggestions and areas for improvement were made by key informants to address some of the challenges experienced by users of the ZM-EIR.
First, gender dynamics, gender segregation (male vaccinators/female caregivers), education, literacy, and cultural norms impacted access and use by vaccinators and the ability to make direct contact with caregivers. Although their numbers are lower, female vaccinators have higher compliance with the ZM-EIR system than their male counterparts. Power, trust, and relationships were noted to present challenges between vaccinators and clients of the opposite gender. The implementation of the ZM-EIR brings to the fore the deep-rooted gender inequities in society that need to be addressed through relevant policy measures to leverage the full potential of the intervention. In addition, gender analyses at the start of future ZM-EIR intervention planning may also help identify strategies to help overcome this gap. These deeply embedded gendered preferences related to vaccination experiences reinforce the need to incorporate cultural and gender considerations when designing digital health interventions.
Second, we noted reliance on double record keeping, which respondents recommended phasing out to reduce the data entry burden (double requirement to complete both paper and mobile data entry) for field vaccinators and facility staff. Paper-based reporting is a requirement for health program audits. This has led to prioritization of paper reporting over the consistent use of the ZM-EIR for reporting. Our study found the perception that ZM-EIR uptake is hindered by physical record keeping, with a recommendation to transition to paperless reporting.
Although changes in immunization coverage across all antigens, timeliness for the BCG vaccine, and the reduction in the number of zero-dose children cannot be attributed wholly to the ZM-EIR program, the study findings are a promising indication that EIR data availability and use can support improvements in child health outcomes, such as vaccination, possibly through better tracking of clients and availability of data. Small pilot studies in countries such as Vietnam and Bangladesh with immunization programs rolling out EIRs report improvements in key immunization program milestones, such as timeliness of immunization (vaccinations delivered on time according to established schedule recommendations), a reduction in vaccination schedule dropouts, and improved ability for vaccinators to identify and follow-up with defaulters [,-]. Similar to this evaluation, health workers using EIR systems report that they use the data to identify defaulters, coverage disparities, and vaccine stock levels and that they have more confidence to take action based on these data analyses []. However, this is the first known study of this degree of comprehensiveness, scale, and rigor to evaluate the availability and use of data of an EIR and the potential for associated effects on immunization outcomes.
Limitations of the StudyThe study was a retrospective one, with a small sample of qualitative interviews, and information bias due to recall cannot be excluded. Stakeholders may also have expressed a positive view of the program due to reasons of social desirability. The MICS estimates show similar changes in Pakistan’s Punjab Province, which was not a target of the ZM intervention. But since immunization has been at the forefront of public health issues in Pakistan, similar programs were likely in place in Punjab Province. These create challenges in the attribution of effects to programs such as ZM. ZM did not act in isolation but would have been greatly affected by the presence of other health system–strengthening activities, including improvements in supply chain logistics, training and supervision of providers, and demand generation activities targeting immunizations. The increase in immunization coverage may also be attributed to more structural factors (eg, public trust in the government).
ConclusionThis study contributes to the evidence base of the added value of increased data availability and use through the implementation of EIRs in low- and middle-income countries as part of broader EPI interventions. It aligns with key benefits identified in other studies of immunization information systems and EIRs, namely alerts and reminders, missed vaccinations, supervision and oversight of vaccinators, and management and accountability. Study methods included a retrospective analysis of system-generated data, qualitative stakeholder interviews, and assessment of potential contributions to improvements in immunization coverage and timeliness and reduction in the number of zero-dose children. Findings suggest that the ZM-EIR is a promising technology platform that may have contributed to improvements in the coverage and timeliness of BCG vaccination, as well as the reduction in the number of zero-dose children as part of a broader package of immunization interventions in Sindh Province, Pakistan, from 2017 to 2019.
The external evaluation of the Zindagi Mehfooz (safe life) electronic immunization registry was led by HealthEnabled, in collaboration with a team of local and global researchers with the lead research group Gallup Pakistan, who supported the primary data collection for this study. We especially acknowledge inputs from Vijay Kumar Dharma, Sundus Iftikhar, and Bilal Gilani. This evaluation was funded by Gavi, the Vaccine Alliance.
A comprehensive report of all quantitative and qualitative data analyses conducted as part of the study has been included as .
MTS, DAS, and SC contributed to the research design and implementation and the review of study findings. RS provided a high-level review and inputs in the design, implementation, and review of study findings. The funders played an oversight role in the study design, data collection, analysis, interpretation, and writing of the paper. The authors/researchers in Pakistan had full access to all the data in the study, while those outside Pakistan had access to only postanalysis data in the study. All authors had final responsibility for the decision to submit for publication.
The core study team was independent of funders. MTS, DAS, and SC are employed by the Interactive Research and Development, the implementer of the Zindagi Mehfooz (safe life) electronic immunization registry (ZM-EIR). RS is from Gavi, which funded the implementation and evaluation of the ZM-EIR.
Edited by A Mavragani; submitted 15.09.23; peer-reviewed by T Ryman, MC Danovaro-Holliday; comments to author 16.10.23; revised version received 24.11.23; accepted 20.08.24; published 11.10.24.
©Patricia Mechael, Sara Gilani, Ahsan Ahmad, Amnesty LeFevre, Diwakar Mohan, Asra Memon, Mubarak Taighoon Shah, Danya Arif Siddiqi, Subhash Chandir, Riswana Soundardjee. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 11.10.2024.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
留言 (0)