To develop and validate a reproductive health literacy scale for refugee women, we used the following steps: identification of the needed domains and items, and assessment of content validity, face validity, and inter-item reliability [13].

Identification of the domains and items

We conducted a literature review of existing reproductive health literacy scales that met the Healthy People 2030 definition to identify the scales that measured the following domains of health literacy: general health literacy, digital health literacy, and reproductive health literacy [12].

Search criteria for scales in PubMed and the Health Literacy Tool Shed included: (a) addressed health literacy according to the Healthy People 2030 definition, (b) covered the area targeted in the RHL trainings and (c) were validated in diverse cultural and linguistic groups. In PubMed, keywords included “health literacy,” “reproductive health,” “scales OR questionnaires,” “digital health OR electronic health OR eHealth”, and “refugee.” In the Health Literacy Tool Shed, the following filters were applied: “health literacy,” “digital health,” “reproductive health,” and “cancer” [14]. The identified scales were evaluated by the research team and ReproNet content experts (international medical graduates) according to the following criteria:

addresses the content topics of the RHL Training (cervical cancer, family planning, maternal health/postpartum care)

is aligned with the Healthy People 2030 health literacy definition to assess an individual’s ability to find, understand, and use health information

reports robust parametrics

has been piloted with low-income, multicultural populations. Of particular interest were scales that had been tested in Arabic, Dari, and/or Pashto.

Upon review of the scales, we selected the following scales.

General health literacy

For the domain of general health literacy, we selected the European Health Literacy Survey Questionnaire 6 (HLS-EU-Q6). The HLS-EU-Q6 is a shortened version of the European Health Literacy Survey Questionnaire 47 (HLS-EU-Q47) that has been widely used in a variety of countries and was translated and validated in several languages including Persian, Slovenian, Bulgarian, Dutch, German, Greek, Polish, Spanish, Indonesian, Kazakh, Russian, Malay, Myanmar/Burmese, Mandarin, and Vietnamese [15,16,17,18]. The HLS-EU-Q6 correlates strongly with the 47- question version (0.896) and is a reliable (α = 0.803) measure of health literacy [19]. The 16-question version of the questionnaire (HLS-EU-Q16) has also been administered in Dari, Pashto, and Arabic [20, 21]. We opted to use the Q6 because it could be completed in a shorter duration, while still giving us an accurate measure of general health literacy.

Digital health literacy

For the domain of digital health literacy, we identified the e-Health Literacy Scale (eHEALS). The eHEALS scale assesses the ability to find, understand and use electronic health information. Psychometric testing indicated an alpha coefficient for the 8-item scale was 0.88 and factor loadings for each item ranged from 0.60 to 0.84 [22]. This scale has been translated and validated in Arabic while still maintaining strong parametric (α = 0.92), and it has been used in Arabic with migrant populations from Syria and Iraq [20].

Reproductive health literacy

The search for scales measuring reproductive health literacy domains provided mixed results. To assess cervical cancer health literacy, we found the 24-item Cervical Cancer Literacy Assessment Tool (C-CLAT) [23]. This tool has a strong factor loading of each item and has been validated in Arabic [23]. For maternal health/postpartum care, we selected five items from a postpartum health literacy scale that was developed and piloted with Arabic immigrants in Southern California (questions 17, 20, 21, 22, 24) [24]. The scale that most closely measured family planning and sexual health literacy was the 40-item Iranian Adult Sexual Health Literacy Assessment Standard Questionnaire (SHELA) [25]. We used two items question 5, “I can obtain information on various methods of pregnancy prevention from various sources” and question 30, “As soon as I realize a sexual problem or disorder, I can find out where or to whom I should go” from this questionnaire [25]. We developed item 23 in our survey, “I know when symptoms after giving birth are so severe that I should see a doctor,” to assess an additional aspect of postpartum health that was not covered in the previous scales.

In order to assess the impact of reproductive health knowledge on changes in health literacy from ReproNet training sessions, we included 3–4 questions per session topic, including questions about cervical cancer, family planning, and maternal health/postpartum care.

Scale formatting

In the next step we reviewed the response options of the scale. Formatting of responses for HLS-EU-Q6 was kept the same as it was when it was validated, offering the choices “very difficult (=1),” “difficult (=2),” “easy (=3),” and “very easy (=4).” Response options for eHEALS were the same as they were in the study with Arabic speaking migrants, including the choices “strongly disagree (=1),” “disagree (=2),” “agree (=3),” and “strongly agree (=4)” [20]. We deleted the option “don’t know” that was used in the original verbal interview. A 4-point Likert scale was used for the remaining reproductive health literacy questions.

A total of 30 survey items were finalized in English and then translated into Dari, Arabic and Pashto by bilingual ReproNet scholars including trained medical interpreters. In case items were already translated and validated in Arabic or Dari, bilingual and bicultural subject matter experts reviewed the translation for understandability and appropriate use of medical terms. The complete survey was then piloted with bilingual ReproNet volunteers and refugee women for understandability and accuracy.

Content validity

Content validity of questions was established by review of 11 ReproNet steering committee members, which represent refugee community members, social service and health providers serving refugee communities, refugee health scholars, and subject matter experts on medical accuracy and cultural appropriateness. The surveys were shared and commented on in group e-mails and consensus on the final wording was obtained at steering committee meetings.

Face validity

Face validity was conducted by having refugee women from the community evaluate the items. For each translated version of the scale, one woman who was comfortable reading and writing in that language and English reviewed the scale and was asked to judge the items based on understandability and appropriateness. Items that everyone agreed to be culturally appropriate, sensitive, and easy to understand were kept. Based on feedback from the ReproNet subject matter experts and community members, we modified several questions on the reproductive health literacy scale to assess a person’s ability to understand and apply knowledge. Question 1 from C-CLAT was modified from “Cervical cancer is preventable” to “I know what I can do to prevent cervical cancer”. Question 10 from C-CLAT was modified from “When detected early, cervical cancer can be cured” to “I understand what can be done if I have an abnormal cervical cancer test.” We originally planned to use subscales from the SHELA. However, upon review by the subject matter experts the wording of the Farsi/Dari items was very complicated and not easily understood in translations to English, Pashto, or Arabic and we chose only two items as described above and designed one new item.

Survey administration

This study was approved by the University of California, Irvine IRB board. Surveys were administered at training sessions in-person and online. In-person sessions were held at Sacramento Public Library branches. Participants were given a hardcopy of the pre-test survey including demographics questions. Those who were preliterate received help from a bilingual ReproNet volunteer to complete the survey. After the session, survey responses were entered into REDCap by ReproNet research staff. Online sessions were offered via Zoom to pre-registered attendees. Participants received a link to the pre-test REDCap survey which they completed at home, and they were able to request help in completing the survey if needed. For their time, in-person training participants received $20 when they completed the pre-test and online participants $30 gift card if they completed pre- and post-tests.

Statistical analysis

All statistical analyses were performed using SAS 9.4 (SAS Institute, Cary, NC). Statistical significance was set at p < 0.05, using two-tailed tests. We used ANOVA to compare the difference in total mean for each domain of the survey.