Participants

Thirty of 42 who expressed interest in participating were available and consented to take part in a focus group. We had 4–6 people in each group and the majority of participants described their primary role as being academic, researcher, or methodologist, however, many individuals were multidisciplinary themselves (Table 3). For example, we had academics and methodologists who were previously healthcare providers. The majority of participants were familiar with summary formats with most reporting familiarity with abstracts (n = 29), plain language summaries (n = 26), and summary of findings tables (n = 24).

Table 3 Participant demographics

We coded to 79 of the 94 recommendations and generated three new codes relating to describing caveats, the ability to navigate, and color preference. These 79 recommendations were categorized into structure, style, and content for ease of presentation. Of note, the 15 recommendations that we did not code to were largely only from one supporting study, often dealt with minute details (e.g., ‘shade rows’, ‘present positive results first, then negative’) or perhaps may not have been deemed relevant for our participant population (i.e., all participants were native-English speakers and did not discuss ‘use end-user’s native language’). The 15 recommendations are available in Additional file 1 with red strikethrough text. In addition to text being coded to the recommendation itself, text was coded as ‘Not Supporting’ to indicate if a participant objected or did not agree with a recommendation. At most, this was 7% of the coding coverage in any of the 6 transcripts, and special attention was given to these codes to present a balanced presentation of the results below.

Structure and style

Although summaries of varying lengths were discussed, including executive summaries of nearly ten pages, most participants agreed that “whether it is a policy brief or a plain language summary or an executive summary…” 1–2 pages was preferred over “…a five pager or six pager” [039]. One-page summaries were preferred by most. It was recognized that “it takes such effort to get things simplified and readable but it’s so worth it” [033] but it has its benefits as “sometimes the more you write the more you have to explain…” [027].

Participants also generally agreed that a visual format may be more helpful as “visual representation can get the message across very powerfully” [022] although it was recognized that some topics may be easier to present visually: “this thing was an easy thing to put an image of, which you can’t do with everything” [012]. While a format that balances visual and textual information was desired by many, concerns about resource constraints were expressed with one participant reporting “that the biggest barrier is for people to actually be able to produce them, I'd love to be able to do it if I had the software and the skills and the time” [027].

Even if the resources are available, participants did not like overly illustrated approaches as this was seen as largely inaccessible, with one participant noting that “there’s a lot of time given to…making things…visually appealing for some people that will actually limit it for other people…high level summaries…should be accessible to everyone” [028]. Aspects of accessibility and readability that several participants commented on were the “real balance” [037] of text and white space, bullet points, subheadings, easily identifiable hyperlinks, and simple color schemes which emphasized important information such as key messages. Most participants agreed that key messages were one of the most important aspects of a summary, stating: “the focus in a summary is more on the results than the methods” [027]. There was no agreement on whether the key message should be presented first or last in a summary but as one participant noted, “if it's high level enough and if you have done it right, you should want people to work through everything before they get to…the key messages” [037]. Hyperlinks “inserted in all the places where they’re relevant as opposed to having at the end tools and resources section” were praised by many.

Flexibility in the presentation of information was discussed (e.g., collapsible sections with interactive formats) but as one participant expressed: “it would be nice to have that but also you’d have to make sure that there’s some level of control that people are getting the key message still, they’re not just picking and choosing pieces that they want to see” [032]. Flexibility was also discussed in relation to reading preferences for printed versus online summaries. Some preferred printable formats as they “just find it easier to digest something that’s actually on a piece of paper in front of me as opposed to on a screen” [020]. Even for those who “prints the 80 page report. And reads it and highlights it and you know underlines…,” it was acceptable to have “the one open on the laptop and clicking the hyperlink…” [012]. Hyperlinks were also generally preferred to footnotes as they’re “…more immediate. A footnote takes up text and a link, it's optional whether you go to it or not and it's immediate for the reader” [011].

Regarding subheadings and structure, most participants did not like the IMRaD (Introduction, Methods, Results, and Discussion) academic format, which has been the predominant structure used in academic articles since the 1970s [20]. For synthesis summaries, they expressed that “it’s more user friendly for everyone involved” [012] not to use it (IMRaD). They liked smaller structured sections that “have it broken up” [037] with signposting to important information, stating that “it can be nicer to read a shorter piece and then go on to the next…as opposed to being…presented with a huge block of text that’s a bit overwhelming" [012]. These discussions related to the expressed need to clarify the audience for the evidence summary. Accessibility for all end-users was emphasized over individual separate tailored summaries: “you are never going to get something that suits everybody and someone is going to want more information. But if you have appropriate links to that other information I think a one size fits all could work” [015].

Accessibility for all end-users was also related to the need to consistently format summaries over time. It was recognized that “each organisation will have their own format” [024] but that “consistency in the format is really important it's…marketing so you have to…be…consistent because I think people get used to using a certain thing and they get familiar with it and they like it” [009]. In addition to being helpful from an organizational point-of-view, decision-makers also viewed this consistency as “great…it gave me an idea of…the direction of flow or…what needed to be completed” [001] and it helps end-users “know exactly where to find stuff” [016].

Content

This consistency from organizations or summary producers was also related to participant’s trust in the summary findings, with some expressing that “there needs to be clear ownership of where…” a summary “…has come from because people will be copying and pasting it…” One participant noted that “you’d have confidence in certain institutions…that the assessment was rigorous” [010] with another echoing that “I think the reputation of the organisation is very important…” [024] Recognizable logos and links to organization websites or the first author et al. were preferred over a full list of authors. Disclosures of “some sort of funding and conflict of interest” [040] were also deemed important even if it was a simple note of “no conflicts of interest or conflicts are reported, go here for information. You don’t need every single thing on the summary itself but an indication that there are or not” [040]. This was “important, particularly for pharmaceutical policies or medicine policies” [024].

Other important information to signpost included key messages and the need to communicate “why there is a summary of the evidence” [014] and properly framing the context by describing the PICO (patients, interventions, comparators, and outcomes) as it “tells you everything you need to know about…who it is you’re talking about, what it is you’re talking about and in terms of intervention and…what outcomes you’re interested in” [034]. It was noted that defining the scope of the question being addressed could be particularly helpful when a “guideline group will discuss something that’s not there and they’ll say well what about this paper. And you have to go well you didn’t ask that question” [034]. PICO information was deemed essential to include with a strong preference for a narrative format, i.e., “covered in terms of the introduction” [004], or a “smart art graphic as opposed to an absolute table” [024]. There was some disagreement about whether participants (did not) like a P-I-C-O bulleted presentation. The context or scope of the findings was emphasized as the “inclusion criteria and exclusion criteria” were essential to “judge the results in terms of how something has been done…” [015].

Additional important contextual information was the synthesis's search and/or publication date. However, caution was expressed that putting both may confuse readers, as one participant stated, "I would automatically assume that I was wrong, if I saw something that was like March 2020 but they just published it March 2022. I’d think maybe there was something wrong” [014]. Those involved in synthesizing the evidence noted that “they take quite a while between searching and publishing, I think the search date would be more useful than publication date” [008] highlighting the importance of knowing the recency of the evidence base—“especially in something like COVID where there’s very rapidly developing evidence. You’d want to know how far back does this go into” [032]. Ultimately, “whether or not that includes the dates or not, I guess it depends on how relevant that might be for the key message” [030].

Methodology

Aside from the search and publication dates, there was much discussion about the evidence synthesis methodology (Fig. 3). It was broadly agreed that important information such as the type of review, “like this was a rapid review as opposed to…a systematic review…” [032] should be included but “…we’re not talking about the steps of a systematic review” [032]. A majority of participants, even methodologists,

“don’t think detailed methods work belongs in the summary…We know where to go if we want to find anything to do with methods and it doesn’t go into the summary, people know to go to the report for that…I don’t even think it's a nice to have, I think it can confuse a reader as to the point of the report. And it might indicate that what they’re reading is not for them.” [042]

Fig. 3

Participants agreed that the “methods absolutely have to be reported for transparency” but that information generally should be in “an appendix where if people wanted to…review it in that level of detail,” [018] they could. Framing methods “a bit differently, like what did we do?” [027] was viewed as more accessible as “it's already more conversational and easier to understand than talking about methodology and…that is jargon at the end of the day” [027]. Most participants agreed that the details such as which assessment tools were used, would be within the full technical report, not the summary: “if people wanted to know more about what the steps were in that review then again I would…maybe put in a link and they can review that those steps have been taken” [016].

Statistical information

The idea that “bombarding them with methodology probably isn’t the best way to go” [027] extended to providing definitions of statistical terms which was also seen as a “waste of your word count” [027], particularly for clinicians who “want the bigger picture and what impact that’s going to have on their clinical practice” [020]. However, it was noted that it was “important to have those definitions in there…” for things that could be “…easily misinterpreted by an unfamiliar reader” [028]. Interpreting statistical findings was emphasized over actual numerical statistics; it was “better not to use statistical terms or even any statistical association or like odds ratio…” [031] and that it is more helpful to add “a contextualization part to add a bit of meaning behind the statistic…” meaning “…what is the result saying in terms of the finding that you are talking about. So is it going to lead to an increased risk, a decreased risk” [026].

If possible “statistics…in diagram form” [011] such as “stick people…coloured in to show numbers, figures, and things like that” [035] were helpful because “if you have a big paragraph that’s just giving you statistical information with R values, that’s numbing. While if you can pull it out and actually showcase it through like imagery…it can be seen in a context” [014]. Exceptions to minimizing numerical information were discussed such as when “it's more nuanced or more borderline” [040] or when “it's like a major thing that’s going to really impact the guideline or impact or practice…it can be helpful just to have like the actual P value or confidence interval there…sometimes to see like how wide is that interval or what are we actually dealing with here…” [030]. Yet still, minimizing information was preferred: “you don’t want a high-level summary just to be full of like P values or some other tests” [030].

Presenting statistical information for guideline development can often come in the form of a summary of findings tables. These were discussed as “very important for guideline development groups” [027] with some stating that “definitely summary finding tables should be added.” [031] However, it was noted that they can be “overwhelming…” particularly when there are.

“…multiple time points, multiple outcomes…” because “…then the reader is kind of left wondering well which is the most important time point, which is the most important outcome?” [027] It was suggested that one could “break up the summary tables findings…into categories or something like that. So that it’s not one big, long table…” [015].

Certainty of evidence

The importance of including an assessment of the quality of evidence (largely in reference to the Grading of Recommendations Assessment, Development, and Evaluation or GRADE scale) in an evidence summary was discussed at length. Many felt that, in addition to the key messages, “the most important thing is the person would walk away from reading your summary and say it's either high or low quality” [027]. Some participants strongly felt that providing “information on the GRADEing of the evidence” was “how we can generate the trust on the evidence” [031]. But others cautioned that while “GRADE is obviously an ideal…sometimes though it can be misinterpreted so if something is very low certainty evidence it doesn’t necessarily mean it doesn’t work or is bad” [025]. Others echoed the concern with using and the need to explain GRADE, expressing that they “think the GRADE is way too deep. That scale doesn’t need to be explained at all.” [014] Participants agreed that if GRADE is used it should be a “generic overview” [028] or explained “in a very easy way or in an explainable way. Not using the jargon” [031].

Recommendations

There was some disagreement regarding putting recommendations in an evidence summary. Some thought, "if a guideline has been commissioned to inform policy…it's an opportunity not to leave that unsaid” [011]. As one participant framed it, if a synthesis was from a governmental statutory body or clinical program for a “specific context then it’s appropriate. So it totally depends on where you sit within that decision making context” [027]. However, another participant preferred a more cautious presentation, noting that “in Cochrane reviews they provide something called author’s conclusions…it is clearly written that it is the authors opinion based on the summary or based on the evidence” [031]. They expressed the belief that “we are providing the evidence so our task is not to recommend anything. Our task is to present the result and to present the…quality of evidence…whether this will be converted into a guideline or not so that is the decision for the decision makers” [031]. However, “from a clinicians’ point of view…the recommendations are the most important and…how they can be implemented in the clinical setting” [005]. As these evidence summaries are informing clinical guidelines, one patient frankly stated “that what they need is to know what to do…there isn’t time for fluffing and faffing” [038].

Framing findings, whether recommendations or conclusions, “within the country or the system you’re working in” [028] was noted as a “critical piece” [030] to report in a summary. This information is helpful for any guideline development group members who may be involved in an implementation with some saying “to me implementation is quite crucial because you can say here’s the recommendations, here’s the key points. But you need to figure out how to actually implement it” [032].

Guidance for summary producers

As one participant noted, “there isn’t really anything… for a general evidence summary that isn’t targeted at a lay audience” [027], thus we aimed to summarise the results from our direct content analysis in a visual (Fig. 4) and one-page-summary (Additional file 5) format to help provide clear and accessible guidance for summary producers. All participants were presented with a draft version of Fig. 4 via email and invited to give comments and attend a debriefing session where the lead facilitator (MKS) explained each item more thoroughly and summarised the results across the six groups. Participants requested minor clarifications and text editing but there were no objections to the content of the guidance.

Fig. 4

Guidance for summary producers