Retrospective evaluation of an intervention based on training sessions to increase the use of control charts in hospitals

Hospital characteristics

Information about the 20 hospitals from the NHS Digital Peer Finder Tool15 at baseline is summarised in table 1. On average, there were slightly more patient attendances per year in the intervention hospitals (1.7 mil, SD=0.5 mil) than in the matched control hospitals (1.3 mil, SD=0.75 mil). The degree of specialisation score was lower on average in the intervention group (83 739, SD=80 639) than in the matched control group (138 747, SD=135 068). The average 2010 Index of Multiple Deprivation was similar, at 24 (SD=7) in the intervention and 23 (SD=5) in the matched control sample.

Table 1

Hospital characteristics. means with SD in parentheses

Inter-rater reliability and blinding

Percentage agreement was 99.6% (Cohen’s k=0.97) for SPCs, 98.5% (Cohen’s k=0.94) for time series charts, 89.0% (Cohen’s k=0.61) for time series and between group charts, and 89.9% (Cohen’s k=0.80) for quality and safety charts. In no cases was a rater ‘de-blinded’ such that they could discern whether a board paper arose before or after the salient intervention period. There were 12 images referred to the chief project investor because it was unclear whether they were charts (eg, the resolution may have been too poor to tell) and agreement on the appropriate decision was reached in all cases.

Chart characteristics for all charts in intervention and control hospitals

There were 6318 charts identified. However, 31 were either educational SPCs with example data, illustrative data not about the hospital, or they were icons without any data. These charts were removed from the analyses. After excluding these charts, 6287 charts were retained for analyses (see table 2). Nearly one-half of charts (3003/6287, 48%) were quality and safety charts. Time series charts were more common (4741/6287, 75%) than between group charts (640/6287, 10%) and 906/6287 (14%) charts were comprised of both time series and between group presentations (combined). Of all 6287 charts, 449 (7%) were SPCs. Of the 449 SPCs, 63/449 (14%) had a summary icon displayed on them, and the control limits were labelled for 342/449 (76%) of the SPCs. For most charts with labelled limits (191/342, 56%), the label was UCL (‘upper confidence limit’) or LCL (‘lower confidence limit’) rather than specifying where the limit was set (see online supplemental file 6 for further description of the SPCs).

Table 2

Chart characteristics (all charts)

Effects of training intervention (intervention versus control hospitals)All charts

The raw numbers and proportions of SPCs used by group (control and intervention), hospital and time-period (preintervention and postintervention) for all charts are shown in table 3 and figure 5. On average in the control group, there was very little change in use of SPCs from before (9/1585, 0.6%) to after (23/1900, 1.2%) the intervention period (average difference 0%, 95% CI −2% to 2%). In the training intervention group, use of SPCs increased from 89/1235 (7%) to 328/1567 (21%), and the average difference was 22% (95% CI 2% to 42%). On average, the absolute difference in use of SPCs was 17% (95% CI 6% to 27%) higher in the intervention group compared with the control group. Use of SPCs in the postintervention period was nine times higher (95% CI 3 to 32) in the intervention group compared with the control group, adjusting for the preintervention (baseline) proportion of SPCs.

Table 3

SPC usage by group, hospital and period (all charts)

Figure 5Figure 5Figure 5

Use of SPCs—premeasurements and postmeasurements by group. SPC, statistical process control chart.

Subset of quality and safety charts only

As planned, we carried out an analysis restricted to quality and safety charts. The raw number and proportions of SPCs used by group (control, intervention), hospital, and time-period (preintervention versus postintervention) for quality and safety charts are shown in table 4. In the control group, there was very little change in use of SPCs before (7/657, 1%) to after (12/741, 2%) the training intervention period (average difference 0%, 95% CI −3% to 4%). In the training intervention group, use of SPCs was 71/684 (10%) before and 213/921 (23%) after the training, and the average difference was 21% (95% CI 0% to 42%). On average, the difference in use of SPCs was 18% (95% CI 7% to 29%) higher in the intervention group compared with the control group. In model-based analyses, use of SPCs in the postintervention period was nine times higher (95% CI 2 to 41) in the intervention group compared with the control group.

Table 4

SPC usage by group, hospital and period (planned subgroup analysis—quality and safety charts only)

Subset of time series charts

Further analyses regarding changing the exposures to time series charts and between group charts are reported in online supplemental file 7, Tables S7-2 and S7-3. For the model with the time series chart exposure, the results were broadly similar to the main analysis.

Subset of time series and between group charts

For the model with the times series and between group exposure, the average difference in use of SPCs was 10% (95% CI 0% to 20%) higher in the intervention group compared with the control group. The zero-inflated negative binomial model did not converge for these data, possibly due to the high number of zero cells in the outcome (37/40 observations).

Thematic analysis of qualitative data

Written responses from the feedback forms were available for 7 out of 10 hospitals in the training intervention sample, including two hospitals that increased the SPCs in board papers by less than 10%. Most comments consisted of a few words or one sentence. The main themes relating to responses to the question about what went well were the general format, content and delivery of the training (n=21/66), such as ‘Topic relevant and timely’; practical and personal examples that use own hospitals’ data (n=19/66), such as ‘trust (hospital) data brought it alive’; conversation, discussion and interaction (n=10/66), such as ‘interactive opportunity to discuss examples’; formatting, use and insights (n=10/66), such as ‘good explanation of SPC rules’ and other general comments (n=6/66).

The question about what could have been done differently during the training elicited fewer responses overall (n=32) than did the question about what went well (n=66); this was true across hospitals, including those that changed their use of SPCs both more and less than 10%. The main themes relating to what could have been done differently were the session format (n=15/32), such as ‘more time for discussion’ and ‘break out into groups’; no suggestions for doing anything differently (5/32); the training content (4/32), such as having a ‘technical supplement’ and ‘more on the calculation of control limits’ and requests for more examples using own hospital data (3/32), providing handouts (3/32) and other (2/32).

Most participants mentioned awareness of SPCs themselves as a key takeaway (n=29/70). Others commented on the general use of SPCs (n=23/70), such as trend lines, tools and templates, and understanding ‘how poor presentation can lead to poor decisions’. Several participants commented that the training changed how they interpret data (n=6/70), intend to report data (6/70) or generally think about data and reporting (4/70). The other comments (n=2/70) were about encouraging others and timelines for implementation.

Finally, when asked for any other comments, most participants made generally positive comments on the training (25/26). Only one (1/26) participant suggested that ‘next steps are important’, which may reference the need to consider implementation steps in training.

留言 (0)

沒有登入
gif