Design

Semistructured interviews were conducted with key informants of the NCEAs scheme, between July and August 2021. Reporting of this study is guided by Standards for Reporting Qualitative Research.11

Sample and recruitment strategy

We sought to recruit up to 25 key informants, to include a sample of current ACCEA subcommittee assessors (professional, employer and lay assessors), applicants, representatives of professional organisations and other key stakeholders. We purposively sampled informants on the basis of current membership of ACCEA subcommittees in England and Wales or through membership of relevant national organisations such as Royal Colleges or groups representing doctors. Current ACCEA subcommittee assessors were invited via email invitation from the ACCEA main committee on behalf of the research team. Representatives of national-level organisations and bodies were identified in consultation with the ACCEA main committee and invited via email from the research team. Replies were directly to the research team to prevent the ACCEA main committee from knowing who had replied. From those who expressed interest, we purposively selected a range of individuals to maximise variation in terms of profession, gender and ethnicity. Participant’s identity was kept entirely confidential within the research team and not disclosed to any external agencies or bodies, including funding bodies and ACCEA.

Interview materials and procedure

Two semistructured interview topic guides were developed; one for use with assessors, and one for use with representatives of professional organisations and other stakeholders. The topic guides were informed by a review of literature, the research team’s knowledge of the NCEA scheme, regular liaison meetings with the main committee for ACCEA, and input from a project-specific patient and public involvement group. We piloted the topic guides with a previous assessor, known to the research team. Both topic guides were organised into three sections, reflecting the aims of the study: (1) role and experience in relation to assessing and applying for CEAs; (2) ‘excellence’: what should be rewarded and how should excellence be defined and (3) equity within the application and assessment process. Interviews lasted up to 1 hour and were conducted via online platforms (Microsoft Teams or Zoom) or on the telephone according to interviewee preferences.

Patient and public involvement

A patient and public advisory group provided written and verbal feedback on the interview topic guides and on initial analysis of the interviews. Resulting changes made to the interview guide included providing a list of current assessment domains and a description of the current scoring scale for key informants.

Data processing and analysis

Interviews were audio recorded and transcribed. Transcripts were checked by BMT and anonymised. Interview transcripts were analysed inductively with a reflexive thematic analysis consisting of six recursive phases.12 13 Together, BMT and EP developed an initial list of subthemes. Individual codes with similar semantic meanings were grouped together into subthemes, and stand-alone codes were transferred directly into subthemes. Potential overarching themes were developed and iteratively refined with JLC. JLC brought relevant experience to inform the overall study. To facilitate reflexivity, JLC was not involved in data collection and only in late stages of analysis and interpretation.

In a reflexive thematic analysis, the coding process is unstructured, which meant that codes evolved to capture the researchers’ deepening understanding of the data and research area. During the six phases of analysis, BMT reflected on her assumptions surrounding the data and research area and how they might shape analysis. Professional characteristics of the core research team were also reflected on during the final analysis and write up stages, to ensure that such characteristics did not inappropriately shape findings. The results presented in this paper focus on the findings relating to how clinical excellence is defined and considerations of equity in scoring. Detailed comments made around the numerical scoring scales informed subsequent stages of research.

FindingsCharacteristics of key informants

Twenty-five key informants were interviewed. Most were ACCEA subcommittee assessors (n=14), others held dual roles as assessor and representative of professional organisation (n=6) or were solely representative of professional organisation (n=5). Six key informants reported holding an NCEA, and one participant reported being unsuccessful in application. A near equal proportion of interviewees were female (n=12) and male (n=13). Despite concerted efforts to recruit a diverse ethnic sample through purposive sampling, most key informants were of white ethnic background (n=19). A small number of informants were of other ethnic backgrounds (Indian or Indian Mmixed background n=3, African or African mixed background n=1, Chinese or Chinese mixed background n=1). Nearly all informants were aged 45 years or older (n=24). Eleven potential key informants, who had expressed an interest in being involved in the interviews, did not respond to further invitations to participate.

Overarching themes

Our inductive reflexive thematic analysis generated three overarching themes around how interviewees defined clinical excellence, differentiated between levels of excellence and discussed definitions for and scoring of excellence that are unbiased. The overarching themes and their subthemes explore how: (1) ‘Clinical excellence’ is multifaceted, and a range of behaviours and activities should be rewarded; (2) Assessors develop their own personal strategies to guard against bias and perceived challenges with the current scoring scheme and (3) There are perceived inequities for marginalised groups of doctors surrounding producing evidence and due to the self-nomination process. A number of subthemes were also identified (table 2).

Table 2

Illustration of overarching and descriptive themes from qualitative interviews around defining clinical excellence, scoring and equity

‘Clinical excellence’ is multifaceted, and a range of behaviours and activities should be rewarded

Informants’ views on what constitutes clinical excellence appeared to be related to their current role as a medical professional, and/or past experiences as patients of the NHS.

Nearly all informants reported that clinical excellence means demonstrating going over and above job expectations and the expectations of their colleagues, in terms of hours worked, taking on additional and more senior tasks, and relieving workload of colleagues:

…to be able to really demonstrate what do I do that’s over and above. My job plan says this, my contract says that, actually I have excelled in the way that I have delivered on my contract … (Female; White ethnicity; Lay assessor).

Some also argued that doctors should not be further rewarded for activities that they are already being remunerated for:

A lot of these posts such as clinical director or divisional director are renumerated at quite a high level so divisional director is getting paid 30 thousand pounds in our hospital on top of their NHS salary. So I don’t feel that… should carry weight in terms of clinical excellence award because they’re already being paid for that… (Male; White ethnicity, Professional assessor, and award holder).

Many informants explained that making an exceptional difference to patients and the NHS constitutes clinical excellence and should be rewarded, being a perceived fundamental aim of the NHS and those working within it. A marker of excellence included patient preferences to be treated by certain doctors:

‘I want to go and see this doctor for this disease,’ because the care that they give or the team that delivers that care is just brilliant and they’re fabulous. We want to reward that—that’s clinical excellence. (Male; White ethnicity; Professional assessor, Representative of professional organisation, award holder).

Many informants explained the importance of outlining the impact of excellence, such as on patients, their colleagues and the NHS overall. Some informants noted that as this is an NCEA Scheme, national and international impact of activities or otherwise beyond local impact should be demonstrated and rewarded:

I think clinical excellence should be where people are… really showing that excellence at the regional or national level. (Female; non-White ethnicity; Professional assessor).

But others noted that excellence demonstrated at local level should not be overlooked:

And if you read the definitions of national it doesn’t have to be national it can be anything outside of your immediate Trust, so you could be applying for a national award because you’ve influenced the Trust next door. (Female; White ethnicity; Professional assessor).

In assessing clinical excellence, many informants described the importance of recognising the context or setting through which excellence is achieved. Challenges for applicants were reported in respect of generating research excellence in non-university hospitals, or within specialties that offered less time and fewer resources to undertake research, such as emergency medicine and radiology:

… the man or woman sitting in District Hospital X where they’re struggling just to manage the volume of work, and they cannot score particularly highly in domain four (research and innovation) because they just don’t have the time, whereas the individual who’s got half their week as an academic can get big publications. (Male; non-White ethnicity; Professional assessor and past award holder).

Assessors develop their own personal strategies to guard against bias and against perceived challenges with the scoring scheme

Key informants who were assessors explained how they had developed personal strategies to assess clinical excellence and to try to ensure fairness. Strategies appeared to be in response to assessors’ experiences, and the written applications themselves. These informants did not specify that such strategies were recommended or guided by ACCEA, but were rather strategies that they had personally developed.

Assessors described obtaining examples of relevant job plans to tackle the diversity of specialties and job roles in applications that were assigned to them. Lay assessors, and occasionally other assessors, suggested that their lack of knowledge about the typical working day of some specialties, especially with sometimes limited details provided under the job plan section, made it difficult for them to assess what constitutes a contribution that was ‘over and above’. They, therefore, sought information about the job role in focus:

I know a little bit because we’ve got a big neurosciences centre… But I’m merely a lay person and I’m judging someone’s form on neuroscience, because what do I know? So, if the Societies give me a leg-up and support me that helps. (Male; non-White ethnicity; Professional assessor and past award holder).

In assessing awards, informants reflected on the subjectivity of the process. Informants who were assessors described maintaining a ‘hawk’ or ‘dove’ assessment style to address unconscious bias, and that being aware of one’s own potential biases and performance as an assessor was important to diffuse issues with subjectivity:

If you were a hawk you need to be a consistent hawk. You can’t be a hawk for one person and a dove for another… (Female; non-White ethnicity; Professional Assessor).

Many noted several factors that could shape one’s perception and expectation of excellence, such as the assessors’ own job, life experiences, protected characteristics and whether the applicant is a current award holder.:

I do four days of [speciality], 7am to 7pm… So my standard is really very high because of what I do for a living… (Male; White British ethnicity; Professional Assessor, representative of professional organisation, and award holder).

Assessor-informants described modifying scores to reflect application structure and content, when applicants’ presentation of material was particularly difficult to read. Examples included the use of undefined acronyms, variable use of bulleting and the use of excessive and poorly focused sentences and presenting the same evidence in multiple domains or presenting evidence in the incorrect domains:

… abbreviations …Sometimes they bother to explain them earlier and sometimes they don’t…If you’re marking them and it’s all bullet-points, it’s pretty easy to scan through, but often you don’t get a sense of a journey or a development. Whereas if it’s all dense text, it’s much more difficult to read through…. (Male; White ethnicity; Professional assessor, representative of professional organisation, and award holder).

Informants also emphasised the importance of considering several sources of excellence evidence presented, and that assessors need a comprehensive overview of applicants’ excellence, in terms of role and contribution, from the applicants’ emic perspective, as well as the etic perspective of colleagues and patients, in order to fairly score:

All evidence submitted says this person is a wonderful person you know they’re really good at their work… so yes if it was obligatory that we had to have citations from a number of different people then that would make it better. (Female; White ethnicity; Professional Assessor).

Some informants suggested that citations from Royal Colleges were not entirely helpful and sometimes overlooked, as they were generally positive, and not all applicants were a member of a College.

There are perceived inequities for marginalised groups of doctors in producing evidence and due to the self-nomination process

Key informants discussed a variety of inequities within the assessment scheme, which were suggested to be underpinned by long-standing inequities in the NHS. Informants explained that such inequities act as a perceived barrier to being successful in the scheme and thus discourage doctors from applying.

Informants from various clinical specialties reported that there are difficulties for certain specialties to generate suitable evidence, such as in pathology, anaesthesia and emergency medicine, where doctors may not be able to generate appropriate evidence for all domains as easily as doctors from other specialties:

Radiologists sit in a dark room looking at pictures and don’t have that much contact with many patients—so how do you assess how their patient benefit is, and how they get quality feedback from a patient who may never have seen them? Anaesthetists have a similar sort of problem… (Male; White ethnicity; Professional Assessor, representative of professional organisation, and award holder).

Other informants suggested that it is difficult for service-focussed doctors, especially those working in district general hospital settings, to generate as much research evidence as academic-focussed doctors working in university teaching hospitals. Some informants also explained that there is a general assumption among doctors that most of these awards are given to doctors working within some specialties, such as surgery; potentially dissuading doctors from other specialties from applying.

Many informants also reported time challenges for less than full-time applicants to generate evidence, because of the perception that they are unable to go over and above their work hours, or to take on additional tasks, and thus have restricted opportunities to develop evidence across all five domains, unlike full-time applicants, who were suggested to be able to do this:

…because you’ve only got a 0.8 whole time equivalent contract, and so what you’re delivering at your baseline appears to be lower than what someone who’s on a full-time contract delivers, and then you’ve got to then not only bridge that gap, but also then go over and above, and I think that’s really difficult to demonstrate. (Male; White ethnicity; Professional assessor, representative of professional organisation, and award holder).

Eight informants explained that there are gender and ethnic disadvantages incurred through the self-nomination process of the scheme. Male and female informants, and informants from a variety of ethnic backgrounds, explained that female doctors and doctors from ethnic minority groups are less likely to recognise their excellence and to self-nominate themselves for an award; and when they do so, their presentation of material is sometimes seen as too modest:

I think another one from somebody who is not British, for example, and there is a very different way of writing, there’s a very different way of expressing themselves, sometimes a lot more modest. You look at the difference between a female applicant and a male applicant and how they present their evidence. (Female; White ethnicity; Lay assessor).

It was suggested that female doctors and doctors from ethnic minority groups do not always feel comfortable taking credit for team effort in a self-nomination process application:

As an ethnic minority person, you do suffer from imposter syndrome unlike say my white colleague who might have done half the things that I did but are absolutely fine in going there. Secondly, if you’re a female, then it’s even worse because you never really reach that stage where you think that you’re good enough until somebody tells you. (Female; non-White ethnicity; Professional assessor).

Informants from a range of specialties and backgrounds explained that there is widespread penalisation for teamwork in the application. Informants explained that the narrative nature of the application focused on the applicant only, and that mentioning too much teamwork could be costly:

You can spend a lot of time writing things: ‘I did this, I did that.’ Or was it, ‘We did this or we did that, or was I part of the team that did this?’ and what’s more powerful. I think that plenty of folks are less good at taking the credit for stuff than others… (Male; White ethnicity; Professional assessor, representative of professional organisation, and award holder).

Nearly all informants provided practical suggestions which might be considered in relation to improving the scoring process and equitability of the scheme. Many suggested the need for advertising the scheme more broadly, such as through members of ACCEA visiting trusts to talk about the scheme to underrepresented specialties and groups of doctors, and allocating champions within trusts to encourage applications. Many also recommended that ACCEA should create support materials for prospective applicants, such as videos about how to collect evidence, or whom to approach for advice on applications. Finally, some informants advised that for assessors; more frequent training, contact and information about the range of specialties for assessors, would improve their contribution.