The use of checklists in surgery is a best practice.1 There is a plethora of evidence that suggests using the WHO Surgical Safety Checklist (SSC) reduces complications such as pneumonia,2 intraoperative blood loss,2 3 sepsis,2 unplanned intubation,2 urinary tract infections,2 wound infections,2–4 30-day readmissions and 30-day mortality.2–4 The SSC has three components, which need to be carried out for each phase of a surgical procedure, including sign-in, timeout and sign-out.5 The SSC serves as an aide memoir that includes vital information to prompt team discussions and actions that may otherwise be overlooked or forgotten, thereby promoting clear, consistent and timely communications among team members that prevents errors and enhances patient safety.6 Importantly, the SSC is more than merely a routine activity. When used as intended, it can promote effective teamwork and communication and enable surgical team members to voice their concerns regardless of their professional role.7

The challenges in SSC implementation are well documented4 8 9 and attributed to a myriad of factors that involve every level of the healthcare organisation. Contextual challenges that impact checklist adoption include the level of physician involvement,4 8 workflow patterns,3 10 a lack of organisational support8 and a lack of customisation.8 Therefore, it is no surprise that previous research has demonstrated inconsistent results concerning SSC compliance, particularly for frequency and completeness of usage.3 4 In the context of planned surgeries, a systematic review and meta-analysis of 10 studies found that observed checklist compliance varied, with rates ranging from 12% to 100%.5 Although checklist compliance is increasingly being monitored in healthcare settings, it is not enough to simply know if the ‘checks’ are performed in practice.5 Measuring intervention fidelity is necessary as it is an important component of the implementation of any quality improvement (QI) intervention, adding further evidence about the causal relationship between the intervention and outcome.11

Implementation fidelity is a key component of implementation.12 It serves as a key moderator between the intervention and its intended outcomes, which is why fidelity measures are needed to assess the effect of a QI intervention.13 Implementation fidelity encompasses several dimensions including faithfulness to the intervention’s design, the effectiveness of training, the proper delivery of the intervention (ie, if it was executed as intended), the participants’ comprehension and utilisation of the intervention, and the incorporation of the newly acquired practice in daily routine.13 Measuring implementation fidelity enables the interpretation of intervention effectiveness13 and may mitigate potentially false conclusions being drawn about an intervention’s effectiveness. In other words, it helps to explain why an intervention succeeded or failed. Checklist implementation fidelity has traditionally been assessed using electronic health record audits or self-reports, which do not indicate how the SSC is used, that is, the quality of team communications during perioperative safety activities.8 These methods have inherent limitations such as the reliance on the use of data from the electronic health record, which may be incomplete, or missing important contextual data. For instance, team dynamics and level of engagement in the checking processes can only be measured in real time with direct observations.7

There are observational tools specifically developed to assess the fidelity of SSC administration, including the WHO Behaviourally Anchored Rating Scale14 and the Oxford Non-Technical Skills (NOTECHS) system.15 Both tools have demonstrated high inter-rater reliability,15–17 and the NOTECHS has demonstrated acceptable predictive, concurrent and convergent validity.16 However, these are primarily research-based tools. While these instruments share certain elements, they frequently require supplementary training and resources to render them practical for adoption and maintenance in clinical settings. As such, they might not be feasible for implementation in many healthcare systems. In this context, the study by Moyal-Smith and colleagues18 in this issue of BMJ Quality & Safety is a welcome contribution to the literature, reporting on the development of a QI tool, the Checklist Performance Observation for Improvement tool (CheckPOINT) that clinicians can more easily use to routinely assess SSC implementation fidelity in daily practice.

The CheckPOINT tool underwent two rounds of face validity testing with 19 surgical safety experts, clinicians and QI specialists to measure checklist adherence, communication effectiveness, attitude and engagement.18 A 90-minute training programme was developed, and four trained observers independently assessed the CheckPOINT tool’s reliability over 37 video simulations demonstrating excellent inter-rater agreement (intraclass correlation coefficients >0.75 for all checklist phases). Then, two observers were trained to field-test the tool in the operating room. Observers performed a total of 98 observations across 31 sign-ins, 48 timeouts and 19 sign-outs, showing higher ratings and less variation than during reliability testing.18 One week after the completion of the field observations, observers were interviewed regarding the usability of the CheckPOINT tool. Overall, observers reported that the CheckPOINT tool was easy to use and facilitated an understanding of what team behaviours they needed to assess. Moyal-Smith and colleagues18 note that compared with existing tools with a similar intent, CheckPOINT is a more feasible and potentially scalable option for routine monitoring of SSC use. This largely relates to the minimal training requirements needed to embed CheckPOINT as an audit tool.

A key strength of the study was the co-development used in designing the CheckPOINT tool with a multidisciplinary team of clinicians and researchers with expertise in surgery, QI and implementation science.18 Diversity of expertise allows for the implementation of a more comprehensive assessment of implementation fidelity. Also, there is a higher likelihood of engagement and support from key stakeholders. Another advantage of the CheckPOINT tool is its ability to acknowledge the connectedness between team performance and SSC fidelity, offering a means to assess these factors more holistically. Finally, the user-friendly format of the CheckPOINT tool, with each phase of the SSC being represented on one page, reduces assessors’ cognitive load, making the tool easier to use.

As acknowledged by the authors,18 a key limitation is the CheckPOINT tool’s lack of nuance in capturing performance. Specifically, the tool does not discriminate between low, moderate and high performance. Further, in the reliability testing phase of the tool, there was limited score variation with higher observed scores. This lack of variation may be related to the experience and role(s) of the observers, highlighting the importance of differentiating observer ratings from true variation in performance. Nonetheless, observers were trained, which increases our confidence in the findings. Also, there was an over-representation of panel members from the USA and Singapore, which may limit generalisability of the study findings to lower-resource settings. The instrument was field-tested at a single institution, during which time another surgical safety improvement project was already underway, which may have indirectly contributed to the high ratings seen in the audit period. However, the ceiling effects described in this audit period may well change over time. Despite these limitations, Moyal-Smith and colleagues18 have developed a tool to measure SSC implementation fidelity with minimal administrative burden that is easy to use with the possibility of being tailored to meet and reflect contextual nuances inherent in any surgical setting.

As Moyal-Smith and colleagues18 and others16 17 in this field have rightfully identified, tools such as the CheckPOINT are ideally suited for gathering information following the first introduction of an SSC or following changes to an established SSC process. These tools can help staff identify areas where the fidelity of implementation may not be as successful or robust, allowing for the deployment of tailored strategies, such as education, coaching or real-time reinforcement. But is this enough to change team behaviour? Ultimately, meaningful SSC participation relies on a combination of connected factors including team composition, communication, workflow and local safety culture.7

Authentic team participation and engagement in the process is therefore crucial to successful SSC implementation.7 Checklist implementation is almost certainly context specific, varying from one local environment to the next, depending on participant responsiveness, the surgical case and its complexity, resource availability and broader factors such as organisational culture. These factors must be acknowledged when considering the utility of the CheckPOINT or any other tool, and its application in practice. Adaptations may be necessary to improve the fit of the CheckPOINT tool to a particular surgical context.

Using QI tools such as CheckPOINT to audit and monitor implementation fidelity in SSC can guide teams to improve standardisation of processes, increasing the consistency of surgical teams to follow best practice guidelines. CheckPOINT and similar tools may also provide opportunities to develop standardised team training education resources. Conversely, focusing on adherence in practice per se without a focus on the other elements such as attitude and engagement may also have some negative effects. For instance, over-reliance on and rigid adherence to checklists may create a false sense of safety with team members being overly task-focused and potentially overlooking other important components of patient care.7 Moreover, achieving implementation fidelity in any patient safety activity is only possible with support from the organisation and system to enable individuals to authentically engage in the process of checklist execution.8

As a novel QI initiative, the CheckPOINT tool can be introduced into surgical care settings at the beginning of a patient safety programme, taking advantage of the demonstrated substantial harm reduction benefits from the successful adoption and implementation of the SSC.18 Evaluating the execution and adherence to the SSC using a tool like CheckPOINT can be particularly useful to identify checklist ‘fatigue’ or ‘drift’ in the ongoing use of the SSC. In considering checklist implementation fidelity, the work of Moyal-Smith and colleagues18 reminds us about the importance of shifting the focus from merely ticking all of the boxes of the checklist towards enacting the behaviours the SSC is seeking to encourage. The SSC is a highly effective, well-intentioned tool that risks being lost in translation if we focus too much on checklist adherence rather than on the enactment of behaviours that are intended to promote the delivery of safe surgical care.

Not required.

Not applicable.