A realist evaluation is a theory-building and refinement process that proceeds in iterative cycles that start and end with theory formulation.17 The process involves developing an initial programme theory (IPT) that is then refined using empirical evidence. An IPT can be elicited using various approaches such as expert consultations, document reviews or literature reviews.17 The PT developed through the realist review will act as the IPT for this realist evaluation.9 Thus, the evaluation represents a move from a global, that is, MWH implementation in LMICs, to a country-level, that is, the Mozambican context. Figure 1 provides an overview of our adaptation of the realist evaluation research process to this study.17 20 21

Realist building and refinement cycle. Adapted from Pawson et al,17 Van Belle20 and Marchal et al.21 CMOCs, context–mechanism–outcome configurations; LMICs, low-income and middle-income countries; MWH, maternity waiting home.

Realist evaluation is methods-neutral, so evaluators can choose the study design and method(s) best suited to refine the IPT.17 We chose a comparative embedded case study design. The comparative case study approach entails identifying key dimensions of variations and then finding cases that vary from each other as much as possible.22 This approach will enable us to test the IPT across multiple MWH–facility birth interventions.23 The comparative aspect is critical to this evaluation because the MWHs built by the Mozambique-Canada Maternal Health project boast of modern masonry infrastructure and amenities, including running water, beds, linen mosquito nets, storage cabinets, flush toilets, shower facilities, and indoor and outdoor kitchens. These MWHs are significantly different from most other MWHs in Inhambane province, which lack basic amenities and are old and dilapidated masonry buildings or built from precarious local building materials such as reeds.12 The comparative approach was proposed by working group members to produce a robust and balanced refined theory.

Embedded case study designs involve multiple units or objects of analysis24 and can cater to an exploration of the complexities and intricacies of both top–down and bottom–up implementation of the MWH–facility birth intervention in Inhambane, including the relationship between the supply-side and the demand-side actors (see in figure 2). The approach will enable us to test the IPT, which describes what works in relation to the engagement of all MWH–facility birth intervention stakeholders, including health system sectors (across national, provincial, district and health facility levels), community leaders and pregnant women and their families.25

Structure of MWH implementation based on information in the Mozambican MWH strategy and the experience of the research team. MWH, maternity waiting home.

MWHs in Mozambique are implemented next to two of the four levels of health facilities—district and rural hospitals, which are the first level of referral care (level II facilities); urban health centres in districts without hospitals, the first level of referral care in those districts (level I type I facilities); and rural health centres which serve the most peripheral communities and offer essential primary care services (level I type II facilities).11 The MWHs built by the Mozambique-Canada Maternal Health project are next to the level I and II facilities, with both types of level I facilities represented. To understand differences in how MWHs affiliated with these different levels of care operate, we have included three MWHs built by the project and their affiliated health facilities (see online supplemental appendix 1). These three MWH–facility birth interventions have each been matched with another MWH–facility birth intervention in the province based on the criteria of maximum variation in the built environment (see online supplemental appendix 1). The six interventions are in four different districts and their day-to-day running and management are in the hands of the healthcare system, that is, provincial, district and facility-level authorities. Therefore, the comparative approach is also expected to provide insight into how variations in internal and external contexts and intervention resources across the participating MWH–facility birth interventions influence users’ engagement with the intervention and associated outcomes. We will also explore contexts–mechanisms and outcomes associated with the uptake of MWHs in the project’s partner communities compared with communities that have not been exposed to resources provided by the project.

Realist evaluation does not discriminate between methods as the goal is to collect data that can enable researchers to test the IPT rigorously.18 Testing an IPT entails refining it based on an assessment of outcomes (un)achieved in the Mozambican context and the underlying contexts and mechanisms. This includes the generation of fresh hypotheses through the discovery of unanticipated mechanisms and behavioural responses. The contexts, mechanisms and outcomes (CMO) framework provides a good overview of the types of data that should be collected, namely, outcome, mechanism and context level data.

In this evaluation, we will begin by systematically profiling each participating MWH–facility birth intervention in terms of stakeholders involved in implementation and their roles, the resources and opportunities the intervention provides to women and families, the quality of MWH–facility birth services provided, the existence of influential contextual factors according to the IPT as well as the level of achievement of implementation outcomes as per the IPT. The profiling exercise will be achieved by identifying and analysing secondary data, including MWH and maternity ward registers, and internal statistics and reports from provincial and district health authorities and the Mozambique-Canada Maternal Health project. We will supplement secondary data analysis with overt observations of the MWH–facility birth environment.26 While we are aware of the variation in the built environments, facility profiling will help to illuminate any other variations in how MWH–facility birth care is provided at the six facilities that the evaluation should take into consideration. Interviews will uncover additional variations and nuances in MWH–facility birth resources, as well as in the internal and external environments, which may not be detected by the other approaches.

MWH–facility birth profiling will lay the foundation for testing and refining the IPT through tailored realist interviews and focus group discussions, suitable approaches for developing theory.27 28 These qualitative approaches enable the elucidation of the reasoning of different stakeholders about their behaviours in response to opportunities provided by the intervention or lack thereof and the role of contextual factors.17 29 30 They will support the examination of relationships between CMO gleaned from different data sources. Collecting data from a group of interviewees in a natural setting, where trust and comfort have already been established, is often more effective and successful than gathering data from a group specifically convened for research purposes.31 Therefore, we will use focus groups to collect data from stakeholders who work together, such as members of a facility co-management committee, and one-to-one interviews with all other stakeholders. Individual in-depth interviews will last approximately 80 min and focus groups will last approximately 2 hours.

The respondents for interviews and focus groups will be purposively and theoretically sampled among supply-side and demand-side stakeholder groups, as presented in figure 2.32 33 Supply-side stakeholders can be grouped into three categories: community leaders/gatekeepers, health system actors (policymakers and administrators at national, provincial, district and facility levels, including health providers) and external donors. The main category of stakeholders on the demand-side is women and their families, but this category can be further divided into four groups: MWH users; non-users; male partners of MWH users and non-users; and companions of MWH users and non-users. Women who used an MWH can be grouped into those who stayed until they gave birth and those who left the MWH before giving birth. For each study site, study participants will be sampled from among these stakeholder groups (see figure 2). Our approach to purposive sampling will involve asking ‘who’ is the ideal research participant and ‘why’ and ‘how’ their knowledge and experience can help to confirm, falsify or refine our IPT.22 29 31 32 We will use theoretical sampling, a well-known sampling strategy in grounded theory, which entails seeking additional data sources based on insight gained from initial data analysis.33 This method involves following where the data have led to expand and refine the evolving theory during the analytical process.33

Research participants will be identified with the support of members of a working group set up to support the research (refer to the section on Patient and Public Involvement). As primary key informants, working group members will take part in pilot interviews aimed at generating baseline data. We anticipate that being interviewed will enable them to gain a first-hand understanding of the planned interviews in terms of the level of sensitivity of the questions, their understandability, and the burden and time required to participate in the research. In turn, this will enable them to contribute significantly to the conduct of the research. They will also take part in a brainstorming session aimed at identifying specific key informants at all levels. For example, we envision that individuals with overall responsibility for maternal and newborn care will be primary key informants at national, provincial, district and non-governmental organisation (NGO) levels. Similarly, community leaders and structures (eg, community health committees and facility co-management committees) involved in maternal health promotion activities, especially those with direct involvement in MWH implementation, will be identified. Through a snowball approach, individuals identified as key informants will be asked to suggest other eligible participants.34 35 Facility representatives in the working group will gather data from MWH users and companions interested in the research after explaining its aims and objectives. The information will include the name of pregnant woman, the name of companion, the age of pregnant woman, the reason for using the MWH, the duration of stay and address (name of community). Information will also be collected about women who did not use an MWH despite having been recommended to do so or those who did not use an MWH but who experienced birth complications. The collated master list of eligible participants will enable purposive sampling of individuals to approach for consent and participation.

Sample size in realist evaluation is determined by the rigour and relevance of the collected data for theory testing.29 30 Rigour and relevance hinge on an interplay between the quality of participants, that is, the extent to which they possess rich information and are able to openly engage in the interview, and the knowledge and interviewing skills of the interviewers. It is impossible to predict the exact sample size required to attain a satisfactory level of rigour and relevance. However, at minimum, we expect to need 66 interviews with participants from the women and family category and 18 focus groups with community-level implementers (11 and 3 at each MWH–facility birth intervention, respectively). The women and family category will consist of MWH users and non-users, male partners of MWH users and non-users and companions of MWH users and non-users. We also expect to interview 32 health system stakeholders (12 providers, 8 district-level officials, 4 provincial-level officials and 4 national-level officials). Lastly, we expect to interview eight external donor informants, such as representatives from civil society and NGOs.

The IPT informed and structured the development of initial interview questions aimed at IPT refinement. These interview questions will be tailored to each participating MWH–facility birth intervention using insights gleaned from the profiling exercise. Additionally, we will work closely with data collectors and working group members to refine research questions for enhanced understanding by each participant group. With their assistance, we will modify interview guides for implementers, MWH users and non-users (refer to the sample of the initial interview guide for implementers in online supplemental appendix 2online supplemental appendix 2). The interview focus will vary depending on each participant’s areas of expertise. For example, interviews with women and family members will aim to test theories related to uptake at their level. In contrast, interviews with implementers will primarily assess theories related to health system-level adoption, with the possibility of addressing user uptake theories if time and interest permit. Realist interviewers present their IPT to interviewees, who are expected to engage with them, refuting, corroborating and adding depth to the theory.17 29 30 This approach is contrary to what is recommended for interpretive research approaches, where the interviewee is seen as the expert, and the interviewer remains neutral by concealing any knowledge they may have on the interview topic.30 Working group members expressed concern that participants from the women and family category are likely to confirm the interviewer’s theories due to inexperience in expressing opinions. However, the approach has been used in maternal health research within other rural African country contexts, which gives us confidence that the approach is viable, especially if it is done in a culturally sensitive manner.36 The African Evaluation Principles advocate for using culturally appropriate approaches to data collection and recommend storytelling as particularly suitable for the African context. Therefore, storytelling will be an integral part of data collection in this evaluation. Users will be asked to share the story of their pregnancy, MWH utilisation and birth experiences to get them talking. Similarly, implementers will be asked to share their stories about how they came to be involved in maternal health promotion and MWH–facility birth implementation work. Where feasible, context-appropriate vignettes will be developed based on facility profiling data and ongoing data analysis to help convey the theories to participants.37

We will conduct interviews with external donors and national, provincial and district-level implementers through various means such as in-person, over the phone or online, depending on the convenience of the interviewees and their geographical accessibility. Health provider interviews will be conducted in their homes to minimise disruption to their work and create a more comfortable and open environment for sharing stories, thoughts and ideas. Similarly, interviews with participants from the women and family category will be conducted in their homes, with a consideration to avoid scheduling interviews within the first 3 weeks postpartum to allow women adequate time for recovery and adjustment. Community leaders will be interviewed at locations of their choice.

To ensure high-quality data collection, we will recruit and train local data collectors. While we may not find experienced realist interviewers, data collectors must, at a minimum, possess experience in qualitative interviewing. We will provide comprehensive training in realist methodology, including instruction in the IPT. This training will encompass interactive sessions, role-playing exercises and pilot interviews conducted with working group members, during and after which they will be encouraged to propose revisions to the research questions. Due to the complexity of the theories being tested, the questions in the interview guide will primarily serve as conversation openers. Therefore, the training will primarily focus on equipping data collectors with the ability to ask insightful follow-up questions in line with the realist approach. Data collection will be organised by one facility at a time, with the research coordinator reviewing transcripts to offer feedback and support to data collectors, thereby enhancing their realist interviewing skills. Additionally, the research coordinator will conduct interviews with national-level and NGO stakeholders proficient in English.

Language and translation approaches can also affect the quality of data, and consequently the rigour and validity.38 Interviews will be conducted in Portuguese, Xitswa or Chope and translated into English for analysis. Translation will occur concurrently with transcription, where interviews in the two local languages will first be translated into Portuguese and then into English. Interviewers will transcribe their interviews for accuracy. To ensure accuracy, a project employee will compare the translated transcripts with the audio, verifying content and meaning. Due to the high cost of human translation, the artificial intelligence translation service, DeepL Pro will be used for English translation. To maintain quality, a team member proficient in both languages will review the audio alongside the transcript, making essential corrections in terms of meaning and grammar. This process will obviate the need for backtranslation. Additionally, data collectors will assist in clarifying meaning when required.

In realist research, the CMOC is the main heuristic for data analysis.18 The analysis will involve extracting CMOCs, coding them to the IPT and refining the IPT as indicated by the evidence. Data extraction entails reading each transcript line-by-line and extracting passages that contain explanatory accounts, either implicitly or explicitly. Based on each extraction, ‘if …then … because’ statements are formulated describing what activated outcomes, how, for whom and in which contexts. The pair of extracted passages and ‘if-then-because’ statements is then coded to the IPT. Developing if-then-because statements involves retroduction, a form of reasoning where a researcher asks ‘why’ about the evidence, the theories and the causes of events.39 Retroduction will be used to link evidence regarding CMO, and this will start already during the interviews. Interviewees will be taught to reason in this way through questions that will require them to think about affective and cognitive responses shaping their engagement with the MWH–facility birth intervention resources and the conditions influencing their responses.

In this evaluation, extractions will be coded to the nine CMOCs making up the IPT as appropriate and used to refine the IPT. CMOCs that will not code onto the existing IPT will be synthesised into new transferable CMOCs as indicated by the data. We will stop the rounds of data collection and preliminary analysis and start to synthesise emerging theories when we achieve adequate rigour and relevance, including an explanation of intended/unintended, expected/unexpected and positive/negative outcomes. Once this is completed, we will generate a comprehensive model and set of specific and transferable recommendations.

We will conduct a comparative analysis by analysing data from intervention and comparator MWHs independently and comparing the resulting refined PTs. Considering the diverse group of stakeholders participating in the evaluation, we expect a range of perspectives and, potentially, contrasting theories concerning the contexts and mechanisms that either facilitate or hinder successful adoption and scaling. If contrasting theories emerge, we will employ a theory-based stakeholder evaluation approach, separately refining theories for various stakeholder groups instead of a unified theory encompassing all stakeholder perspectives.40

The analysis will be performed simultaneously with data collection so that each interview will explore explanatory hunches gleaned from previous interviews. This approach of incremental refinement will help to achieve relevance and rigour, as insights from preceding interviews will guide the selection of appropriate research participants for subsequent interviews and support the cumulative refinement of the IPT. The research coordinator will work with a team of analysts to perform CMOC extractions from interview transcripts and code and refine the IPT using NVivo 12.41 Emerging refinements will be discussed in regular meetings with the principal investigator, data collectors and working group members, including patient public advisors. The ultimate refined PT will be shared with research participants, including those in the women and family category who contributed actively during interviews, to gather their final thoughts and input.