Preliminary considerations

The process of the critical appraisal is comprised of two phases: the preliminary considerations and the main critical appraisal tool. Before embarking on the critical appraisal assessment, it is useful for the assessors to identify important aspects of the study which will guide the appraisal process (Fig. 2). The study design and the research outcomes are key characteristics and should be captured. Moreover, the assessors should examine the SPRs put forward by the study and define whether they will focus on all of the recommendations or only on specific ones, as guided by the review question/s.

Fig. 2

ECR-P (Evidence Communication Rules for Policy) critical appraisal tool preliminary considerations

A review (systematic or not) of policy recommendations might aim to identify and synthesize various policy recommendations relevant to one area or issue or multiple areas and issues. On the other hand, a review might aim to focus on a specific policy recommendation made for a specific issue, bringing together evidence from various sources. If only part and not all of the policy recommendations are included in the scope of the review question, it should be defined early on in this preliminary stage as it will guide the appraisal process.

ECR-P domains

ECR-P consists of five domains, each corresponding to one of the five rules of evidence communication [12]. The overview of the tool is presented in Table 1. Each domain includes a set of signalling questions that are further grouped into two levels: the study level and the policy recommendations level. The first focuses on the evidence base (findings and conclusions) of the study, and the second focuses on the SPRs of the study.

Table 1 Evidence Communication Rules for Policy (ECR-P) critical appraisal toolInform, not persuade

This domain aims to assess whether researchers have been open about motivations and limitations. It has been established that trustworthiness of communication is judged not only by expertise and honesty but also by evidencing good intentions [24]. Authors should be clear about the aims and objectives of their study and disclose any factors that introduce limitations to their findings and the consequent policy recommendations. Ideally, recognising limitations should also be accompanied by proposing future solutions to mitigate them. A clear connection between the study findings and conclusions and policy recommendations is key in evidencing trustworthiness of communication.

In communicating evidence, authors should present findings and policy recommendations in a neutral manner. Emotive language, which can be persuasive, should be avoided. Emotive language is wording that is used in order to elicit an undue emotional response in the reader. Undue, here, refers to trying to evoke an emotion based on unsubstantiated statements. In addition, when authors are composing the policy recommendation section, they should keep in mind that their readers will include policymakers who might not be familiar with their scientific field. Therefore, they should strive to avoid scientific jargon as much as possible in the policy recommendations section to improve both accessibility and informativeness.

Offer balance, not false balance

In this domain, the balance of the communicated evidence is assessed. In the interest of informing fully, a balanced account of evidence should be provided. The focus is first on the completeness of reporting. All aspects of study results should be reported whether a hypothesis has been verified or not. In many scientific fields, a guideline is followed in order to exhibit the quality and completeness of reporting (e.g. CONSORT [25], STROBE [26], CHEERS [27] etc.). Implementing such a guideline is advisable. Regarding the policy recommendations section, all foreseeable aspects of a recommended policy should be discussed, whether positive or negative. Knowledge of the policy status quo and how this might be affected should also be demonstrated. Developing and implementing a new policy is very likely to have negative implications as well. Authors should investigate and report on their SPRs’ implications always keeping in mind the end user. End users in this sense being both those who will enforce the policy recommendations as well as the groups affected by them.

Disclose uncertainties

This domain focuses on whether what we don’t know is clearly communicated. A strategy that could be adopted is for authors to state: what they know; what they don’t know; what research could be done in the future to find out more; what people could do in the meantime to be on the safe side; and that initial recommendations might be subject to change [12, 28]. Reporting uncertainties might be seen to be easier and more straightforward if studies included quantitative analysis (e.g. statistical confidence intervals). Nevertheless, studies should also explain contextually what this uncertainty means in terms of their findings. In addition, authors must consider how the uncertainty of their findings might create uncertainty in their policy recommendations. If uncertainty exists, it could be the case that the precautionary principle should be adopted in the meantime [29].

State evidence quality

This domain addresses the communication of evidence quality. The credibility of a communication and the reliability of a study finding can be affected by the quality of the underlying evidence. Whether the data used in a study is primary, collected by the authors, or secondary, retrieved from other sources, its quality should be reported and considered. The use of a standardised quality metric, if one exists for the specific area, is advisable. Undoubtedly, the quality of the underlying scientific evidence that the study is based upon will affect the quality of the policy recommendations that have been developed based on them. A consideration of this interaction is key.

Pre-empt misunderstandings

This domain addresses the repercussions that can stem from inadequate understanding, be it due to a lack of adequate information, a lack of clarity in presented information, an overload of information that reduces clarity, for instance, for non-experts, or the existence of misinformation, to name but a few potential scenarios. Effort must be made to pre-empt misunderstandings and inoculate against misinformation via “prebunking” [30]. In order to achieve this goal, one must anticipate potential issues arising from misunderstandings or even disinformation. Especially in the policy recommendations section, knowing your audience and offering clear and practical recommendations can help address this issue.

Signalling questions

Signalling questions were designed to obtain information around one or more dimensions of quality addressing the context of each rule for evidence communication. It should be noted that there is significant interplay and some overlap between certain elements of the dimensions of quality that are examined by ECR-P. Mapping to the three dimensions of quality is presented in Additional file 2.

The available responses for each signalling question are predefined (Table 1). The affirmative responses ‘Yes’ and ‘Probably Yes’ are associated with low concerns for RoB and therefore high-quality outcomes. Consequently, the negative responses ‘No’ and ‘Probably No’ are associated with high concerns for RoB and low-quality outcomes. The options ‘Probably Yes’ and ‘Probably No’ should be used in the case where the determination had to fall back on a judgement made by the assessor. Such judgements are made when clear objective evidence is not available in the study but can be safely inferred from the context. The ‘No information’ response should be used when not enough information is reported for the assessor to make a ‘probably yes’ or ‘probably no’ judgement within the context of the study. The ‘not applicable’ option is mainly used when a signalling question is connected to a previous one that has not been answered positively. Nuances of the different responses for each of the signalling questions are described in detail in the elaboration and explanation document (see Additional file 1). Each response must be justified in a free text box. Assessors may use direct quotations from the papers to justify their responses when possible.

Domain and overall judgement

The critical appraisal assessment results are expressed as RoB judgements. The term RoB is often used to describe concerns regarding the results of a study specifically arising from areas of internal validity [4]. We decided to use the same term here for two reasons. First, ECR-P touches upon key internal validity issues, especially in the study level of each domain (see Additional file 2). Second, assessors are familiar with this term being used in the quality appraisal process within systematic reviews, as established by many existing quality appraisal tools. Third, using this terminology would facilitate a seamless incorporation of ECR-P assessment outcomes into frameworks for developing body of evidence summaries such as GRADE (Grading of Recommendations, Assessment, Development, and Evaluations) [31].

RoB judgement will be reached for each domain separately and there will also be an overall RoB judgement. RoB judgement per level of domain will be based on an algorithm depending on the responses to each of the signalling questions. Consequently, the ratings of the two levels will be combined in a rating per domain. The algorithms are provided in the accompanying explanation and elaboration document (see Additional file 1). It should be noted that the decision trees are our suggestion so the judgement can be overridden by the assessors by providing appropriate justifications. The algorithm incorporates considerations of both the study level as well as the policy recommendations level within each domain.

The RoB judgement options for domain and overall judgement are as follows: Low risk of bias; Some concerns and High risk of bias, corresponding to high, moderate and low quality. The overall RoB judgement is mapped to the domain level judgements. The worst rating across domains will be carried over to the overall RoB judgement. We advise that a paper should be rated as low RoB, indicating the highest quality, only when all domains have low RoB.

Target users

ECR-P was initially designed to fit into the systematic reviewing process but also as a tool to be used in evidence-based policymaking. ECR-P can be used more generally to critically appraise SPRs in any domain. Potential users of the tool include policymakers and policymaking organisations of all levels as well as researchers and funders that are interested in evidence-based policymaking. Furthermore, ECR-P can be a useful tool in the peer-review process providing a clear and transparent critical appraisal roadmap for manuscript reviewers and journal editors. Target users can be expanded in the future with user-defined refinement.