For this analysis, we used a data set of temporally and geographically concurrent household and health provider data collected in 2016 in the Savanes region of Cote d'Ivoire and used in a previous assessment of effective coverage estimation methods [7]. We used data on care-seeking collected from a sample of households and data on service readiness and process quality collected from a census of health care providers. We generated estimates of effective coverage for both sick child care and labor and delivery care. We used exact-match linking based on the reported provider utilized as our gold-standard estimate of effective coverage. We employed two ecological linking methods (1) linking a subject to the closest provider within the specified provider category and (2) assigning a subject the average quality score of all providers with the source of care category within a defined administrative unit. We then repeated the ecological linking analysis using a sample of facilities (drawn from our facility census) to simulate the effective coverage estimates that would be produced using standard SPA sampling methods. Comparing the census-based estimates to the sampled estimates by ecological linking method allows us to isolate facility sampling's effect on linked effective coverage estimates.

Household survey

Household data were collected via an independently implemented MICS. The MICS collects data on services related to reproductive, maternal, neonatal, and child health. In Cote d'Ivoire, the 2016 MICS was sampled to be representative at the regional level [8]. For labor and delivery, women with a live birth in the last two years were asked about where they gave birth, classified by sector and facility level. For sick child care, women with a child under the age of five years with diarrhea, fever, or suspected acute respiratory infection (ARI) in the two weeks preceding the survey were asked about whether care was sought for the sick child and the source of that care, classified by sector and provider type. Additional questions were added to the standard survey asking respondents to specify the provider or facility they utilized for birth and/or sick child care. This information allowed us to link each subject to the individual health facility, outlet, or community-based provider from which care was sought. The central coordinates of each household cluster, as provided by the MICS, was used to approximate the location of sampled households.

Healthcare provider assessment

In parallel with the MICS, we conducted a health provider assessment in the Savanes region. The assessment included all public, private, NGO, and religious first-level and referral facility-based providers, pharmacies, and pharmacy depots in the region. All CHWs serving the 44 sampled MICS household clusters in the region were also included in the assessment. We adapted the facility assessment tool from the SPA. The assessment collected data on infrastructure and commodities through a facility inventory, asked health workers about training and supervision, and assessed provision of care through direct observation of sick child consultations and health worker report of the interventions delivered during the last childbirth attended in the preceding 12 months. We used these data to generate estimates of facility inputs (structural quality) and quality (process quality) to provide labor and delivery and sick child care. We defined structural quality as the infrastructure, equipment, commodities, and human resources necessary to manage a given health issue. We consider process quality to be the correct assessment and management of patient needs by the health provider, gauged through observation of provision of care. The location of each facility, or CHW’s work primary site, was captured at the time of the survey.

We received ethical approval for the health facility assessment from the Johns Hopkins School of Public Health institutional review board (#00006896) and the Cote d’Ivoire Comité National d’Ethique de la Recherche (#025/MSLS/CNER-kp).

Effective coverage estimation

We estimated the effective coverage of (1) sick child care and (2) labor and delivery (L&D) care. We estimated effective coverage by assigning each individual from the household survey the structural and process quality score of their linked source of care based on data from the health provider assessment. Applying the effective coverage cascades proposed by the Countdown to 2030 [9], a global collaboration for tracking progress towards the SDGs for women, children, and adolescents (countdown2030.org), we calculated input-adjusted coverage (care-seeking adjusted for facility structural quality) and quality-adjusted coverage (care-seeking adjusted for facility process quality). We implemented the linking analyses using the same methods outlined in the original effective coverage analysis by Munos and colleagues, including definitions of provider structural and process quality [7]. We used three linking methods in this analysis:

1.

Exact-match linking: Each woman or sick child was linked to the specific provider they reported utilizing—collected via the additional questions in the MICS asking respondents to name their source of care.

2.

Euclidean distance linking: Each woman or sick child was linked to the closest provider (as the crow flies) within the reported provider category (defined by managing authority and level of care).

3.

Administrative unit linking: Each woman or sick child was assigned the aggregate score of all health providers with the administrative unit (district) and reported provider category, weighting for provider caseload.

We consider exact-match linking based on the specific, stated provider as the most precise method for linking a subject and source of care. We have used this measure as our "truth" comparator for gauging bias introduced by ecological linking and facility sampling. A previous analysis using this dataset compared both the Euclidean distance and administrative aggregate methods against the exact-match method. It found both approaches produced similar estimates of effective coverage in this setting when restricting by reported provider category and weighting for caseload [7]. We chose to examine these two ecological linking methods as they are commonly used approaches and performed well compared to the exact-match method when implemented using a provider census [7].

To reflect the type of data available from existing health facility assessments, we limited our analysis to use only health facility data. While we had data on both pharmacies and community-based health workers, standard SPA or SARA facility assessments do not include these sources of care. Exclusion of these sources had a minor effect on estimates of coverage in our prior analysis as both pharmacies and CHWs accounted for a small proportion of care-seeking for child illness (Fig. 1) and were not a source of care for labor and delivery [7].

Fig. 1

Source of sick child and delivery care reported in household survey

In some cases, individuals could not be linked to a provider. Using the exact-match linking method, an individual would not be linked to their source of care if the respondent (1) could not recall the specific source of care, (2) the specified provider reported they did not provide the service the respondent stated they received, or (3) the provider assessment did not include the reported source of care. Despite using a census, a few providers that were not identified (primarily small private sector providers) for inclusion in the facility census or were outside of the study area. Using the ecological linking method, an individual would not be linked if there was no provider from the reported source of care category within the defined administrative area included in the assessment. In all cases, the subjects were assigned an average score for the management authority and level of facility they reported utilizing if they could not be linked.

While women could only give birth in a single facility, care for a sick child could be sought from multiple locations over the duration of the illness episode. While uncommon, children that sought care from more than one health provider were assigned the average input and quality scores of the providers they utilized. All GPS-related analyses, including measuring distance between points, were conducted in ArcGIS 10.8 (Esri, Redlands, CA, USA). All statistical analyses were conducted in Stata 16 (StataCorp, College Station, TX, USA).

Sampling

We replicated the sampling methods used by the SPA as closely as possible. In line with standard SPA sampling methodology, we sampled among first-level facilities and maintained a census of referral facilities. Assuming the SPA was conservatively powered to estimate an indicator with ± 10% precision at the regional level, we estimated 65 facilities would need to be sampled. We also considered larger samples of 90 (precision ± 7.5%) and 130 (precision ± 5%) facilities. For each sample size, we drew 20 samples of facilities at random. We applied our administrative and closest provider linking analysis to generate effective coverage estimates for each sampled dataset.

In our dataset, there was limited variability in quality within facility categories. This limited the potential generalizability of our simulation results. Therefore, we ran two additional sensitivity analyses using simulated quality scores to assess the effect of sampling in settings with greater diversity in quality scores.

Quality simulation 1: random facility quality

In the first simulation, we maintained the data on facility type, facility location, and household care-seeking behavior. However, each facility was assigned a structural quality and process quality score completely at random (possible scores ranged from 0 to 1). Assigning a score at random was intended to increase the variability in provider scores within each provider category, increasing the opportunity that sampling could result in an individual being linked to facility with a substantially different score than their true source of care. The rest of the sampling and effective coverage estimation methods using both exact-match and ecological linking were implemented in the same manner as in the above section.

Quality simulation 2: preferential care-seeking from higher quality facilities

As in the previous simulation, we maintained the data on facility type, facility location, and household care-seeking behavior. However, each facility was assigned a structural quality and process quality score designed to simulate preferential care-seeking in favor of higher-quality facilities within a provider category. The rest of the sampling and effective coverage estimation methods were implemented in the same manner as in the above section. We assigned facilities that were utilized more frequently, based on the household survey data, a higher quality scores than those that were utilized less frequently or not at all. Facilities more than 10 km from a sampled MICS household cluster, and therefore unlikely to have been utilized by the MICS sample due to distance, had their score increased or decreased based on their performance relative to the median provider category score. The selected cut-offs and degree of score inflation/deflation were selected to (1) produce a plausible but sufficient spread in scores between high and low performing providers and (2) balance the number of providers whose scores were increased or decreased accounting for distance to prevent inherently biasing an ecological linking method.

Rules for altering facility quality scores from original Cote d'Ivoire data:

If a facility was utilized by one or more (sick child) or two or more respondents (labor and delivery): increased facility quality by 15 percentage points

If a facility was utilized by fewer than one (sick child) or two respondents (L&D) AND within 10 km of a sampled household cluster: decreased facility quality by 15 percentage points

If a facility was utilized by fewer than one (sick child) or two respondents (L&D) AND more than 10 km from a sampled household cluster:

o

If facility score > the category median → increased facility quality by 15 percentage points

p

If facility score < category median → decreased facility quality by 15 percentage points

Analysis

Our primary analysis compared estimates of input-adjusted and quality-adjusted effective coverage against (1) our best estimate of true effective coverage derived by exact-match linking and (2) the best estimate of effective coverage derived through ecological linking with a census of health providers. Comparing estimates from sampled datasets against the exact-match estimates quantifies the total bias introduced by both ecological linking and sampling providers. Comparing estimates from sampled datasets against ecological estimates using a census of providers isolates the bias introduced by facility sampling within an ecological linking method. Our analysis considers whether individual estimates derived through each approach differed significantly from either the exact-match estimate or census-derived ecological estimate. We also considered the distribution of the estimates to characterize the variability in estimates produced through each approach over 20 samples. Confidence intervals around the effective coverage estimates were derived from the sampling design and response rate of the household survey and treats the linked provider scores as an extension of the household survey dataset, ignoring error in the health facility assessment.