Maternal health is the health of woman during her gestation, delivery and after delivery through providing prenatal care, intra-partum and post-partum care, respectively [1]. Maternal mortality is still high and a global public health problem with Maternal Mortality Ratio (MMR) of 211 per 100,000 live births. Sub-Saharan African (SSA) countries account two-thirds (40%) of the global MMR [2, 3]. Maternal mortality was also a public health problem in Ethiopia with a MMR of 412 per 100,000 live births at the end of millennium development goals (MDGs) [4]. The direct obstetric causes including hemorrhage, obstructed labor, ruptured uterus, pregnancy induced hypertension, puerperal sepsis and unsafe abortion accounted 70% of the total maternal deaths in Ethiopia [4, 5]. Majority of maternal deaths were avoidable through intervening an intensive management of complications during pregnancy, labor and delivery, and postpartum [6]. About 15% of mothers’ experienced severe obstetric complications and needs comprehensive emergency obstetric and newborn care (CEmONC) services in Ethiopia [7].

The Ethiopian government set a 5 years health sector transformation plan (HSTP) in 2015/16 to reduce the MMR to 199 per 100,000 live births and avert such high maternal deaths in 2020 [8, 9]. The World Health Organization (WHO) tried to design different policy and program to realize the global MMR lower than 70 per 100,000 live births by the end of 2030 [4, 10]. An CEmONC was one of the strategies to achieve this goal and it has a package of service necessary to manage the direct obstetric complications [11]. It comprises the seven fundamentals of basic emergency obstetric and newborn care (BEmONC) interventions, including parenteral antibiotics, uterotonic drugs, parenteral anticonvulsants, manual removal of placenta, removal of retained products of conception, assisted vaginal delivery, and resuscitation of newborn care plus blood transfusion, caesarian delivery and neonatal resuscitations [11]. CEmONC service was first launched in 1992 and had an explicitly organized guideline for the purpose of monitoring the availability and practice of the service [12]. A minimum of one CEmONC facility had planned to establish in every four BEmONC facilities and for every 500,000 population to prevent pregnancy related complications during pregnancy, labor and delivery, and postpartum period [13].

Appropriate healthcare service was very crucial to prevent obstetric complications during delivery [7]. CEmONC helps to curb maternal and neonatal mortality related with delay in receiving appropriate care at health facilities [14,15,16]. Although CEmONC program was implemented for a long period, only few evidences were available on its implementation status and the barriers and successes on the implementation of the program in Ethiopia. Therefore, this evaluation aims to assess the implementation status of comprehensive emergency obstetric and new born care program at University of Gondar comprehensive specialized hospital (UoGCSH) in terms of resources availability, compliance of health care providers and acceptability of the program services.

Evaluation methodsEvaluation settings

The Evaluation was conducted at University of Gondar Comprehensive Specialized Hospital (UoGCSH) located in Central Gondar zone of Amhara National Regional State, Northwest Ethiopia. It is located 780 kms away from Addis Ababa (capital city of Ethiopia) and 180 kms from Bahirdar (capital city of Amhara region). The hospital established in 1954 as a public health college and training center and currently it is one of the referral and teaching hospital served for more than 7 million people in the region. The hospital had a total 66 beds in obstetric wards (9 labor beds and 57 maternity beds), two operational theaters, three maternity wards, three antenatal ward OPD, one labor ward with six delivery coach, one emergency ward and 253 Gynecology and / or obstetric staffs.

Evaluation objectives

The study aimed to assess the availability of required resources for the CEmONC program. Also, the level of compliance of health care providers to the national guideline during CEmONC program service delivery and the level of CEmONC client satisfaction were determined, and factors associated with CEmONC client satisfaction level were identified.

Evaluation approach and dimensions

A formative evaluation approach with a single case study design was employed to obtain the detail and explorative reports on program implementation from 01 to 30 April 2021. The quantitative and qualitative data were collected simultaneously, analyzed separately, and mixed during interpretations of the findings. This evaluation assessed the implementation status of CEmONC program, using availability, compliance and acceptability dimensions based on the interests of stakeholders. Availability is access to health care in relation with the volume and types of services, and adequacy of resources (physical and human) used to provide CEmONC services [17]. Compliance is the degree to which CEmONC services is being implemented in UoGCSH by health care providers according to the national standards and clinical protocols [17]. Acceptability determines the level of clients (mothers who have received CEmONC service) perception about CEmONC services [17]. The focus of the evaluation was process theory which included input, activity and output (immediate result of program activities) as indicated in the program logic model (Fig. 1).

Fig. 1

Logic model for comprehensive emergency obstetric care in UoGCSH, 2021

Sample size and sampling procedure

Availability of necessary medical equipment, supplies and medicines, and human resource and infrastructure were inspected by principal investigators by using resource inventory checklist.

Observation was conducted in delivery ward to assess the healthcare providers’ adherence to the guideline and WHO surgical safety standards. Data for compliance were collected until information saturation and based on recommendations of three or more consecutive observations to assess the compliance of one healthcare provider as per the national guideline and/or WHO standard [18]. The observation session represented the whole activity of the team and enables us to assess and judged the strength and weakness of health care providers as a whole or individual. The compliance was also assessed by using document review and the sample size was calculated by using single population proportion formula considering 50% proportion (p), 1.96 confidence interval, 5% margin of error and 10% the clients’ charts may not found. The calculated sample size was 423, but we used finite correction formula since the source population was below 10,000. We used 320 as our final sample and reviewed the maternal charts from 01 January to 01 February 2021. Purposive sampling technique was used to select the key informants based on seniority, level of education and position. The sample size for acceptability dimension was determined by using single population formula considering 79.4% patients satisfied by the service [19], 95% confidence level, 0.05 margin of error and 5% non-response rate. Our sample size for acceptability dimension was 265.

Data collection tools and procedures

Semi-structured questionnaire, interview guide, resource inventory, observation and data extraction checklist were prepared through reviewing of literatures [16, 18, 20]. Indicators were also developed from the national CEmONC implementation guideline, WHO surgical safety, and other related evaluations [21, 22]. Observation checklist was used to assess the healthcare provider-client interactions and providers’ adherence to CEmONC national standards, including the interpersonal interaction, ways of provider’s history taking, information transfer and other components of CEmONC as per the national standards offered to CEmONC clients. Resource inventory checklist was used to assess the existence of the required resources used for CEmONC service. This tool contains infrastructures, human resources, laboratory diagnostic tests, essential drugs, and medical equipment/supplies for CEmONC services. Interview guide was prepared for key informants, such as chief executive officers (CEOs), clinical director, midwifery head, health care providers, maternity service coordinator and Gondar town MCH coordinator and focused to assess availability and compliance. Data extraction checklist tool was prepared to review the specific procedure of diagnostic and type of signal functions of CEmONC services from clients’ chart. To find the clients’ chart, we took the medical record number from the service register. Semi-structured interviewer administered questionnaire contained the background characteristics, reproductive history, accessibility and acceptability of the service was used to assess the acceptability of the service by the clients. The questionnaire and interview guideline were translated in to the local language (Amharic) and then translated back to English language to ensure consistency.

Drug store, labor and delivery ward, laboratory room, operation theater room and the whole physical working environment via the resource inventory checklist was assessed to evaluate the availability of resource needed for CEmONC service. Besides observation, the availability of resources was also checked through interviewing the head of MCH (maternal and child health) ward and chief executive manager of the hospital. Exit interview was carried out for mothers who gave childbirths to assess the acceptability of the services. Direct observation was conducted at delivery service and operation theater room after the principal evaluator got an informed consent from the managers. Two MSc midwife students wore sterilized operational gowns and conducted a hidden observation after we got informed consent from the coordinators. A total of nine key informant interviews (KIIs) were conducted by principal evaluator after getting informed consent from interviewees. Key informants were interviewed using the interview guide and probing was done following their response to receive more information. Tape recorder and taking field notes were required during the interviewing process. The interview was taken about 20–30 min for each KIIs.

Data quality control

Two days training was given for data collectors about the basic techniques of data collections. Pre-test was also conducted on 13 participants at Poli health center, a neighboring affiliated health center where Caesarean delivery and blood transfusion service had provided and necessary modification was made. Observation, data extraction and resource inventory checklists were pretested and amendments were made. The questionnaire was checked for its completeness on the daily basis by the principal evaluator and supervisors during data collections. The principal evaluators transcribe the voice of the respondents in to text to analyze and check the consistence of the information with the initials. Key informant interviews (KIIs) were conducted to explore their experiences on how they deliver CEmONC services in relation with the WHO CEmONC guideline until saturation of information. Observation of healthcare providers conducted caesarian delivery in the operation theatre room to see the operation team adherence to WHO surgical safety standards by using checklists [20]. The healthcare providers did not know who observed them and it was done by two masters of Science in midwifery students who had clinical attachment in the hospital after principal evaluator had got permission from the hospital manager.

Data management and analysis

The quantitative data were cleaned and checked for completeness, consistency and coded by the trained supervisor and principal evaluator. Data were entered into Epi data version 4.4.1 Software and exported to SPSS version 20 for analysis. Availability, compliance and acceptability dimensions for the implementation status of CEmONC program were evaluated and judged as very good, good, fair and poor. Acceptability of CEmONC service was measured by a five point Likert scales (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree) and classified as acceptable if the score was above or equal to the value calculated from the demarcation threshold formula \(\left(\frac-\mathbf}\right)+\mathrm\) score [16]. Binary logistic regression model was fitted to identify the factors associated with acceptability of the services. Variables with p-value less than 0.25 during bi-variable analysis were the candidates for multivariable logistic regression and in multivariable logistic regression those variables having p-value < 0.05 were considered as the significant predictors for acceptability of the services. The tape-recorded qualitative data were transcribed and translated into text format and finally thematic analysis was done.

Judgment matrix

The weight of each dimension of CEmONC program was determined by the agreement of the stakeholders based on the degree of relevance. Value was given for each dimension proportionally according to their level of importance considered by the stakeholders. The score of each dimension was aggregated to decide the level of performance of CEmONC program based on the predetermine judgment criteria. Indicators’ weight is the weight given by stakeholders before the evaluation for each selected indicator, and indicator scores were calculated using the formula (Indicator score = \(\frac}}\)) [17]. The weighted values of availability, compliance and acceptability were 25%, 40% and 35%, respectively. The judgment parameter for each dimension and the overall program implementation were also categorized as poor, fair, good and very good with the corresponding judgment values of less than 60%, 60–74.9%, 75–84.9%, and more than 85%.