WHAT IS ALREADY KNOWN ON THIS TOPIC

Patient experience influences health outcomes in various acute conditions commonly seen in the ED.

Patient-reported experience measures (PREMs) are validated questionnaires that aim to reliably assess patient experience. No questionnaire has yet been validated to assess the ED experiences of older adults ≥65 specifically.

We previously developed an 82-item PREM for this age group, which required item reduction and initial validation by psychometric evaluation.

WHAT THIS STUDY ADDS?

We prospectively enrolled ED patients ≥65 to take the 82-item PREM-65 immediately after discharge and then at 7–10 days.

A 25-item version of PREM ED 65 resulted from psychometric evaluation, demonstrating good psychometric properties, including test–retest reliability.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

The PREM-ED 65 enables the assessment of older adults’ perspectives on relational care, the physical environment, information provision and pain assessment within the ED. As such, PREM-ED 65 may enable the identification of vulnerabilities and discrete areas where quality improvement may be required. While further work is needed to ensure applicability across cultures and special populations, PREM-ED 65 may facilitate broader evaluation and benchmarking of emergency services from a patient perspective.

Background

Patient experience is an internationally recognised dimension of healthcare quality, positively associated with outcomes for various acute conditions and increased adherence to ED discharge instructions.1–3 Recent studies have sought to conceptualise patient experience in the ED, emphasising the importance of patient-provider communication, relational care and an optimised ED environment.4–6 These studies provide a theoretical foundation for understanding patients’ experiences that may inform the development of interventions to improve patient experience in the ED. The International Federation of Emergency Medicine recommends routine assessment of ED patient experience, which may be accomplished using Patient Reported Experience Measures (PREMs).7 PREMs are questionnaires usually developed with patients and other relevant stakeholders that provide a standardised method for capturing patients’ views of their care.8 As such, PREMs may indicate where improvement is needed and provide a means of monitoring service performance over time. PREMs supplement patient-reported outcome measures (PROMs), which assess patients’ health status, and satisfaction surveys, which measure the extent to which care meets patients’ expectations.

Past limitations of ED PREMs and PROMs include inconsistent validity and reliability, although recent instruments aimed at the general ED population are promising.9 10 However, few PREMs exist to measure the experiences of specific ED user groups, including older adults, who comprise an increasing proportion of attendances in many settings.11 In response to the ageing global population, there is burgeoning interest in the ED care of older adults and increasing recognition of this group’s often more complex, holistic needs.12 Current evidence emphasises older adults’ vulnerability when attending the ED, demonstrating an increased likelihood of severe pain, increased length of ED stay, need for hospital admission and subsequent in-hospital mortality compared with the general population.13–15

We have previously developed a new English-language instrument—PREM-ED 65—to enable valid and reliable measurement of older adults’ ED experiences. (online supplemental appendix 1) The 82-item draft version has been derived using meta-synthesis, qualitative research, stakeholder prioritisation of candidate items and cognitive interviews to inform design and usability (Figure 1).6 16–18

Figure 1

Flow diagram summarising the development process for PREM-ED 65. PREM-ED 65, Patient Reported Experience Measure. ED,emergency department.

This study aims to provide a psychometric evaluation of the 82-item draft version of PREM-ED 65. The objectives are to (1) provide descriptive statistics, including per-item proportions of responses, (2) eliminate draft items with inadequate response characteristics using hierarchical item reduction (HIR), (3) confirm the structural validity of the finalised item set using exploratory factor analysis (EFA), (4) assess the reliability (internal consistency) of measurement scales, (5) assess test–retest reliability and (6) criterion validity of PREM-ED 65.

MethodsStudy design

This is a cross-sectional survey study. Practical recommendations for reporting scale development and the Consensus-based Standards for selecting health Measurement Instruments (COSMIN) guidelines have been followed.19 20 A multimodal design meant participants could complete either a paper-based or electronic version of PREM-ED 65.

Study setting

Adults over 65 discharged from the ED between 0600 and 2100 hours were recruited at the end of their visit. Data collection occurred between May and August 2021.

Inclusion and exclusion criteria

Eligible participants were those assessed, treated and approved for discharge to their domicile by the attending clinician. Exclusion criteria included patients who were hospitalised, were too unwell to participate, had suspected or confirmed SARS-CoV-2 infection, lacked the mental capacity to provide informed consent (as assessed by the attending clinician), were in police/prison custody, or did not speak English.

Procedures

A collaborative research design was devised by the lead researcher (BG), who contacted EDs hosting medical students studying a 1-year intercalated Bachelor of Science (BSc) programme in Urgent & Emergency Care (University of Plymouth, UK). Following local approval, students were invited to participate voluntarily as site investigators. Students were provided with training, and a registered clinician provided oversight. Student responsibilities included screening, participant recruitment, informed written consent, data collection and entry. Students received a certificate and were named collaborators. No additional incentives were provided.

Sampling and recruitment

Opportunistic sampling, guided by a sampling matrix, aimed to maximise recruitment while accommodating students’ clinical commitments. Based on COSMIN criteria, a total sample of 400—600 respondents (ie, 5–7 responses per PREM item) was targeted.19

All sites were mailed study materials, including a tablet computer, to allow completion of the electronic version of PREM-ED 65, created using SurveyMonkey.

Survey administration

Participants could complete either the paper-based or electronic version of the draft instrument. In addition to the 82 experience items, participants self-reported gender, age, ethnicity, mode of arrival, reason for attendance (ie, injury/ illness) and length of ED stay. All were invited to complete a postal or online retest survey 7—10 days later.

Measurement scales

Responses to each of the 82 draft items were measured using a 4-point Likert scale, from least favourable opinion (1, ‘strongly disagree’) to most favourable (4, ‘strongly agree’). An ‘I cannot answer’ option was included for all items, and a ‘not applicable’ option was included for items regarding cultural needs, sensory impairment and disability.

Data collection and cleaning

Data from electronic surveys were captured via SurveyMonkey (San Mateo, CA). Students transcribed paper-based responses into a secure online form. The study team downloaded and analysed the results using SPSS Statistics for Windows V25 (IBM, 2017). Records that were blank, spurious or lacking a unique participant identification number were excluded.

Data analysisPlanned analysis

Analysis of PREM-ED 65 data consisted of six elements that were (1) descriptive statistics, (2) HIR, (3) EFA, (4) assessment of internal consistency, (5) test–retest reliability and (6) criterion validity.

Descriptive statistics

Site and participant characteristics were analysed. Per-item descriptive analysis included the Shapiro-Wilk test of normality, proportions of responses, central tendency (average rating assigned to each item) and spread.

Hierarchical item reduction

HIR was conducted based on established principles to eliminate irrelevant or non-contributing items.21 22 Prospective criteria for HIR consisted of (a) respondent non-engagement (>20% responses ‘I cannot answer’ or ‘not applicable’ or blank); (b) presence of floor or ceiling effects (>50% responses ‘strongly agree’ or ‘strongly disagree’); (c) presence of statistically significant differences in the distribution of responses (differential validity) based on gender, age or reason for attendance and (d) high interitem correlation (>0.7). Items were excluded if any one criterion was met (figure 2).

Figure 2

Funnel schematic demonstrating hierarchical item reduction strategy for PREM-ED 65. PREM-ED 65, Patient Reported Experience Measure. ED,emergency department.

Exploratory factor analysis

The need to minimise survey length, simplify subsequent analyses and facilitate the identification of discrete areas for quality improvement using PREM-ED 65 was recognised. Thus, EFA was applied to identify patterns among draft items by grouping them into fewer variables known as factors.23

First, responses that were blank or marked as ‘I cannot answer’ or ‘not applicable’ had to be handled. Following assessment using Little’s test, these data were treated as missing completely at random. Single imputation using the expectation-maximisation (E-M) algorithm was performed using SPSS Missing Value Analysis.

The EFA then consisted of four steps. The first step was to exclude items measuring identical constructs (inter-item correlation >0.8). The second step was identifying and excluding items unlikely to be explained by the analysis (communality <0.2). The third step confirmed the suitability of the remaining data to undergo EFA using the Kaiser-Meyer-Olkin (KMO) Measure of Sampling Adequacy and Bartlett’s Test of Sphericity. Finally, factor extraction using principal axis factoring with promax rotation was performed to identify patterns of correlation, revealing the underlying factors. Prospectively agreed quality criteria for factor extraction, based on established criteria,24 stipulated that (1) only factors demonstrating sufficient variance, guided by Kaiser’s criterion (ie, Eigenvalue >1) and scree plot inflexion, were retained, (2) a minimum correlation between any item and its associated factor (factor loading) was >0.4, (3) post-EFA communality for each item was >0.4 and (4) maximum correlation between any item and another factor (cross-loading) was <0.4. Therefore, factors with an Eigenvalue <1 were disregarded, and items not meeting the other criteria were removed stepwise until a solution was identified.

Internal consistency

Internal consistency describes how well items within a scale correlate, providing a measure of reliability. The internal consistency of each extracted factor was considered satisfactory if Cronbach’s Alpha (α) was >0.6, good if α>0.7 or excellent if α ≥0.8.

Test–retest reliability

The intra-class correlation coefficient (ICC) between initial and case-matched retest survey responses was calculated for individual items, scales and the overall instrument. Test–retest reliability was considered adequate with an ICC between 0.5 and 0.75, or good if >0.75.

Criterion validity

The ‘friends and families test question’ (FFTQ) is widely used to assess patient experience in the UK, consisting of the following question:

I would recommend this A&E department to my friends or family if they were in a similar situation. (FFTQ

Responses for finalised items were compared with the FFTQ using Spearman’s correlation coefficient (r). An r >0.4—0.69 was considered fair, and r >0.7 was considered strong.

Patient and public involvement

The study proposal was presented to the Sheffield Emergency Care Forum in December 2017. Interviews with older adults informed draft item creation between September 2018 and April 2019.16 Patients and members of the public, health professionals and third-sector representatives assessed and prioritised draft items during December 2019.18 Cognitive interviews with seven stakeholder representatives, including three older adults, informed readability and design during October 2020.

ResultsCharacteristics of the study sites

Eighteen EDs participated, involving 23 students. Data collection was undertaken exclusively by medical students in 13 EDs, shared between students and clinicians in two EDs, and conducted by clinicians in three EDs.

A median of 20 patients was recruited per site (range 5—98 patients; IQR 10.75) (table 1A).

Table 1

Study characteristics

Characteristics of the participants

Of 525 ED patients recruited, 510 completed initial surveys were returned (97.1%). The median respondent age was 76 years (range 65–100 years); there was an approximately equal gender distribution (women 50.7%). Most participants self-reported their ethnicity as white (88.6%). Participants’ most typical mode of arrival was private transportation; 72% attended the ED without an accompanying person (table 1B).

Per item proportions of responses

Data completeness was 95.7% (4.3% missing item responses). Responses were non-normally distributed across all items (Shapiro-Wilk p<0.01) and demonstrated a left skew (average skewness −0.54) reflecting a high proportion of positive ratings (average proportion of ‘agree’ or ‘strongly agree’ responses across all items=73.2% (range 2.2%–94.9%; IQR 41.7%)). Negative ratings of experience were infrequent (median proportion of negative (‘strongly disagree’ or ’disagree’) responses=3.1% per item (range 0.2%–44.3%; IQR 5.5%)). The median proportion of ‘I cannot answer’ or ‘Not applicable’ responses was 13.7% per item (range 0.4%–90.8%; IQR 34.8%)(online supplemental appendix 2).

Table 2A,B present the items with the highest proportion of positive and negative responses, respectively.

Table 2

Items with the highest proportions of positive and negative responses

Hierarchical item reduction

Using the predetermined criteria, 33 out of 82 items (40.2%) were excluded for non-engagement (online supplemental appendix 3A). Five items (6.1%) demonstrated ceiling effects (online supplemental appendix 3B). No floor effects were observed.

The remaining 32 items were assessed for differential validity based on the proportions of responses by age group, gender and mode of arrival. No gender-specific differences occurred. Two items were excluded due to differences by age group. These were ‘I had enough access to pillows and blankets’ (‘agree/strongly agree’ 44.0% <85 years vs 58.4% >85 years (p=0.04)) and ‘I was involved in decisions about my care’ (‘agree/strongly agree’ 68.5% <85 years vs 58.4% >85 years (p=0.04)). A further item was excluded because of differences in responses based on the mode of arrival (‘staff told me if they were unsure about anything’)(‘agree/strongly agree’ responses 62.2% ambulance vs 44.2% all other transport (p=0.01)) (online supplemental appendix 3C).

Interitem correlation >0.8 occurred for six groups of items. The conceptual similarity of these items was confirmed, and a further 12 items were excluded (online supplemental appendix 3D)

In summary, 53 out of 82 items (64.6%) were excluded using HIR.

Exploratory factor analysis

The remaining 29 items underwent EFA. Little’s Missing Completely at Random (MCAR) test was non-significant (p=0.825), confirming the suitability of E-M for imputation of missing values.

Pairwise correlation coefficients between items were all lower than 0.8. One item (‘Staff cared about my wellbeing’) had an initial communality <0.2 and was excluded. The KMO Measure of Sampling Adequacy and Bartlett’s Test of sphericity were both satisfactory (0.932 and p<0.05, respectively).

Exploratory factor analysis identified four factors with an Eigenvalue >1. Stepwise refinement of the structure excluded a single item with cross-loading >0.4 (‘I was asked how I would like to be addressed’) and two items with a communality of <0.4 (‘I could easily read name badges’ and ‘the temperature was about right’).

A subsequent factor structure comprising 25 items distributed across four factors was established, explaining 65.4% of the total variance. The scree plot supported the retention of all four factors (online supplemental appendix 4). All items demonstrated loadings >0.4 onto a single factor; average communality was 0.6. Patterns of loading of items indicate that the scales measure (1) relational care, (2) the ED environment, (3) staying informed and (4) pain assessment. Internal consistency of each of the scales was good–excellent (Cronbach’s alpha 0.944—0.786; mean 0.870) (online supplemental appendix 5).

Test–retest reliability

One hundred and sixty one retest surveys were issued, of which 52 were returned within 7—10 days (32.3%). Missing data (16.7%) were also imputed using E-M (online supplemental appendix 6). The overall ICC for the 25 items and four subscales was 0.670 (95% CI 0.632 to 0.704), indicating good test–retest reliability (online supplemental appendix 7).

Criterion validity

The average correlation between PREM-ED 65 items and the FFTQ was fair (r=0.397). The strongest positive correlation was for the item ‘A&E met my expectations’ (r=0.592) (online supplemental appendix 8).

Finalised instrument

The finalised, 25-item version of PREM-ED 65 retains user-friendly design features established with stakeholders, and the concise length is appropriate for routine administration to ED patients. This version may be reproduced and copied under the Creative Commons Licence (figure 3; printable version—online supplemental appendix 9).

Figure 3

Finalised PREM-ED 65 Survey Pro-forma. PREM-ED 65, Patient Reported Experience Measure.

Discussion

This study aimed to provide initial psychometric validation for PREM-ED 65.

The resulting 25-item PREM-ED 65 captures essential aspects of the care experience for older adults discharged from the ED, including relational care with providers, the ED environment, information provision and pain assessment. The resulting structure meets predetermined psychometric and interpretability criteria and is intended to follow COSMIN standards for content validity, structural validity and internal consistency.19

While participants mainly reported a positive ED experience, several items with higher proportions of negative responses related to waiting experience (eg, I was given updates while waiting), physical comfort (eg, the seat or bed was comfortable) and provision of refreshments (‘eg, I was able to get food or drink if needed’). These findings may indicate areas where real-world quality improvement may be indicated and beneficial.

Common barriers to adopting patient-reported measures in the ED, including timing of administration, development of relevant measures and accommodating differing patient priorities,25 have been addressed throughout the development of PREM-ED 65. The high test–retest reliability suggests that PREM-ED 65 is suitable for administration after an ED attendance or up to 10 days postattendance.

Eliminating items with low engagement is intended to facilitate ease of survey administration and limit respondent fatigue, acquiescence bias and satisficing, all of which may adversely affect response rate and instrument reliability.26 HIR excluded items with low response rates relating to the reception area, presence of disability, sensory disturbance, non-pain symptomatology and interpretation requirements. Excluding items with significant differential validity aims to ensure that survey results are not inadvertently biased towards a certain subpopulation; in the case of PREM-ED 65, such items were limited to providing blankets/pillows, shared decision-making and communication of uncertainty.

While eliminating unnecessary items is essential for creating PREMs that are user-friendly for the general patient population, exclusion may not diminish their potential value for select subgroups. We advocate the development of supplementary scales for future iterations of PREM-ED 65 to ensure the experiences of minority, under-represented, and very old/ frail adult populations are captured. This is especially pertinent as implicit bias within healthcare may negatively impact patient experience, health outcomes and patient safety.27 For example, the significant differential validity observed for ‘I was involved in decisions about my care’ may indicate that the very old adults are less involved in shared decision-making.

In keeping with similar literature, our findings revealed only a fair correlation with the FFTQ, suggesting that PREM-ED 65 provides a more comprehensive measure of patient experience, extending beyond the likelihood of recommending care.28

Limitations

Although opportunistic sampling precluded the calculation of a precise capture rate, using the sampling matrix resulted in a representative distribution of participants. Characteristics of the study sample generally align with UK ED attendance patterns. The lower than expected proportion of ambulance arrivals was likely due to excluding high-acuity and admitted patients. Although we did not include patients with known or suspected SARS-CoV-2, pandemic precautions restricted the presence of accompanying persons and may have broadly impacted waiting times and ambulance conveyance rates.

While the age-related inclusion criterion for PREM-ED 65 has been adopted for simplicity, chronological definitions of older age are imperfect due to heterogeneity of ageing.29 Further work is required to ensure PREM-ED 65 captures the needs of very old adults, those living with frailty or other long-term conditions. Validation among older adults hospitalised from the ED should be a priority, as this group may be especially vulnerable to shortcomings in care processes. Different approaches may be required to ensure perceptions of experience among older adults with cognitive impairment are represented.

While PREM-ED 65 demonstrates promising psychometric properties, confirmatory factor analysis with additional data is desirable to confirm the stability of the four measurement scales. Separate cross-cultural adaptation will ensure applicability to patients from minority ethnic and non-English-speaking backgrounds. Future studies may also explore the effect of characteristics such as education, health literacy and socioeconomic status on ED patient experiences. Finally, while comparing PREM-ED 65 scores to the FFTQ provides a preliminary test of criterion validity, the FFTQ does not represent a ‘gold standard’ test of patient experience with limitations relating to question construct, comprehension and differential validity.30 Therefore, a comparative analysis of PREM-ED 65 with other ED-specific PREMs or satisfaction surveys is desirable.

Conclusion

The 25-item PREM-ED 65 is suitable for administration to older adults up to 10 days following ED discharge. Findings may be used to assess the quality of ED services from older adults’ perspectives, identify areas for impactful quality improvement and monitor service quality over time. Future priorities should include validating PREM-ED 65 in those hospitalised from the ED and developing supplementary scales to capture the experiences of under-represented patient groups fully. Further studies employing confirmatory factor analysis are desirable to ensure stability, relevance and sensitivity, particularly among very old adults and those living with frailty.

Data availability statement

All data relevant to the study are included in the article or uploaded as supplementary information. Not applicable.

Ethics statementsPatient consent for publication

Not applicable.

Ethics approval

This study received ethical approval from the NHS Health Research Authority (21/PR/0458) and the University of Plymouth Faculty of Health Research Ethics and Integrity Committee (2021-2527-1758). Participants gave informed consent to participate in the study before taking part.

Acknowledgments

The authors would like to thank Mr Chris Rollinson (Research Governance Manager, University Hospitals Plymouth NHS Trust) for providing research governance training; Victoria Yates (Senior Research support facilitator, University Hospitals NHS Trust) for assistance with study coordination and all participating NHS Trusts, research staff, and patients.