Article Access Statistics    Viewed556        Printed1384        Emailed0        PDF Downloaded18        Comments [Add]    

Click on image for details.

 

         ORIGINAL ARTICLE Year : 2022  |  Volume : 68  |  Issue : 4  |  Page : 207-212

Assessment of the quality of randomized controlled trials in surgery using Jadad score: Where do we stand?

S Mohsina, B Gurushankari, R Niranjan, S Sureshkumar, GS Sreenath, V Kate
Department of Surgery, Jawaharlal Institute of Postgraduate Medical Education and Research, Pondicherry, India

Date of Submission01-Feb-2021Date of Decision27-May-2021Date of Acceptance21-Jun-2021Date of Web Publication13-Apr-2022

Correspondence Address:
V Kate
Department of Surgery, Jawaharlal Institute of Postgraduate Medical Education and Research, Pondicherry
India

Source of Support: None, Conflict of Interest: None

Check

DOI: 10.4103/jpgm.JPGM_104_21

Introduction: Quality assessment of randomized controlled trials (RCTs) is important to prevent clinical application of erroneous results.
Materials and Methods: This was an assessment of published RCTs in surgical subspecialties during 2011–2018 based on MEDLINE and EMBASE search. The primary objective of the present study was to quantitatively and qualitatively analyze the RCTs published from India based on year of publication, geographical distribution, and subspecialty using the modified Jadad score (high quality if score is ≥3; or ≥2 if blinded design was not feasible). Its secondary objective was to identify factors affecting the quality of RCTs.
Results: Among 1304 trials identified, 162 were analyzed. Of these 96 (59%) had a score of ≥3; and 104 (64.2%) were of high quality (score ≥2). Year-wise there was no significant quantitative (P = 0.329) or qualitative (P = 0.255) variation. Geographic regions had similar quantity (P = 0.206) and quality (P = 0.068). The RCTs among subspecialties too were comparable in quantity and quality. Higher impact factor of journal (P = 0.013) and assessment by Institute Review Board (IRB) (P = 0.004) were significantly associated with a better study quality. Type of institution, number of authors, centricity, assistance by a statistician, and source of funding did not affect the quality of RCTs.
Conclusions: The quantity and quality of surgical RCTs were stable and comparable over the years and across geographical regions and subspecialties. Higher impact factor of journal and review by IRB were significantly associated with a better study quality.

Keywords: Blinding, drop-out, quality assessment, randomization, reliability

How to cite this article:
Mohsina S, Gurushankari B, Niranjan R, Sureshkumar S, Sreenath G S, Kate V. Assessment of the quality of randomized controlled trials in surgery using Jadad score: Where do we stand?. J Postgrad Med 2022;68:207-12
How to cite this URL:
Mohsina S, Gurushankari B, Niranjan R, Sureshkumar S, Sreenath G S, Kate V. Assessment of the quality of randomized controlled trials in surgery using Jadad score: Where do we stand?. J Postgrad Med [serial online] 2022 [cited 2022 Nov 7];68:207-12. Available from: https://www.jpgmonline.com/text.asp?2022/68/4/207/343151  :: Introduction 

In the current era of evidence-based health care, randomized controlled trials (RCTs) are considered as the gold standard for establishing the safety and efficacy of a new clinical intervention.[1],[2] The quality assessment of RCTs by assessment of the study design, conduct, and analysis helps in improving the quality of research.[3] Some of the commonly followed methods used to assess an RCT include the Jadad score, van Tulder scale score, the Delphi List, the Consolidated Standards of Reporting Trials (CONSORT) statement, and the Cochrane Back Review Group criteria.[4],[5],[6],[7],[8],[9] The Jadad scale, also known as Jadad scoring or the Oxford Quality Scoring System, is one of the most commonly used methods to independently assess the methodological quality of a clinical trial.[4] It is a three-item five-point scale that assesses randomization, blinding, and attrition. Jadad scoring system remains one of the widely used scores as it is easy to use and brief, assesses the key factors causing bias, and is known to have good validity and reliability.[10],[11],[12]

There are few quality-assessment reports in the literature; however, studies focusing on surgical trials are lacking. Among surgical reports, quality assessment of RCTs is reported for plastic pediatric surgery trials and surgical gastroenterology (SGE).[13],[14],[15],[16] Data regarding the standards of published RCTs in the field of surgery are overall limited, especially from India.[17] Moreover, large-scale quality analysis over a period of time is required to identify the trends in the quality of published trials and there are very few reports addressing these issues. As global literature has similar assessment of trials, this was focused on trials published from India. Hence, this study was carried out with the aim of quantitatively and qualitatively analyzing the RCTs in the field of surgery from India published during 2011–2018 based on year of publication, regional division within the country, and subspecialty. The secondary objective was to identify various factors associated with the quality of RCTs, such as the intervention type, funding source, type of institution, review by Institute Review Board (IRB), number of authors, assistance of statistician, and impact factor of journal.

 :: Materials and Methods 

This study was a retrospective quality assessment of RCTs in various surgical subspecialties from India during the years 2011–2018 published in various international and national journals. The study was approved by the Institute Research Council and Ethics Committee and was reported in line with the STROCSS(Strengthening the reporting of cohort studies in surgery) criteria.[18]

Search strategy and selection of trials

MEDLINE and EMBASE searches were carried out by four independent investigators (SM, RN, BG, and DS) with the keywords such as “randomized controlled trials,” “random,” “randomly,” “blinded,” “surgery,” “surgical,” “neurosurgery,” “surgical gastroenterology,” “plastic surgery,” “pediatric surgery,” “urology,” and “India” with Boolean operators such as AND, OR to retrieve information about the published articles. Studies were included only if they were truly randomized, human study and on surgical/allied topics based and were published from India during the years 2011–2018. Trials reported from other specialties such as obstetrics and gynecology, otolaryngology, ophthalmology, and anesthesia and published protocols were excluded. Articles were screened independently by four investigators (SM, RN, BG, and DS) and the retrieved results were matched for inclusion.

Data retrieval

Full reports were retrieved for the studies which met the criteria for inclusion in the study. The articles were scrutinized and the following information was recorded: the year of publication, number of authors, centricity of the trial, geographical region within the country (classified based on the location of the center of the principal investigator), impact factor of the journal, surgical subspecialty, type of institution (public/private sector), type of intervention studied, involvement of a statistician, mention of review by IRB in the manuscript, and source of funding. These data were collected independently by six investigators (SM, RN, BG, DS, SS, and GSS) and any discrepancies were settled by discussion.

Quality assessment of trials

Qualitative assessment of the RCTs was carried out using the Jadad score.[4] It is a five-point scale that comprises points related to randomization, blinding, and dropout. Mention of randomization, blinding, and adequate data on dropout's merits is one point each. Further, based on the appropriateness of methods of randomization and blinding, described one additional point each is awarded to or subtracted from the score, respectively. The scoring was carried out independently by five investigators (SM, BG, SS, TM, and VK) and median of the final scores were considered so as to increase the reliability. In majority of the surgical trials, a blinded design is not feasible because of the difficulty in designing a placebo for most surgical interventions and hence can be falsely rated as a poor-quality study (score <3). Hence, the modified Jadad score was utilized for categorizing the studies after assessing the study design.[5] In the modified Jadad score, the study was considered as high quality for the total Jadad score of ≥3 if blinding was feasible. Study designs in which blinding was not feasible, a score of ≥2 was considered as high quality.[5] The studies were categorized according to the final quality-assessment score and were expressed as proportions.

Statistical analysis

The statistical analysis was carried out using SPSS version 22. The trials were scored as high or low quality based on the Jadad scale and were expressed as proportions. Chi-square test or Fisher's exact test was used to analyze the difference in high-quality trials based on region, year, and subspecialty. Kruskal–Wallis test or Mann–Whitney U test was used to compare the mean/median Jadad scores between the various subgroups. Univariate followed by multivariate analysis was carried out to identify the factors associated with the quality of the trial. A P value <0.05 was considered statistically significant.

 :: Results 

The search results and fate of the identified articles are shown in [Figure 1]. Among the 1304 trials identified by the search strategy, 162 were included in the analysis based on the inclusion criteria. The inter-reviewer agreement was good with a kappa score of >0.7 for all the variables included. Of the 162 trials, a blinded design was not feasible in 48 trials. Ninety six (59%) of the published trials were found to have a Jadad score of ≥3. When the modified Jadad score was applied for studies in which blinded design was not feasible, 64.2% (104/162) of the trials were found to be of high quality. Among overall trials, randomization was adequate in 82%, blinding appropriate in 28%, and dropouts were appropriately reported in 57% of the studies.

Assessment of RCTs with time, geographical distribution, and subspecialty

From 2011 to 2018, there was no significant quantitative variation in the published articles based on the year of publication, although an increasing trend of the score was noted (P = 0.329). The quality of trials in terms of the median Jadad score has also remained stable over the eight years (P = 0.255). The difference in quality and quantity of RCTs based on the year of publication is shown in [Figure 2]. Based on geographic division within the country, there was no significant quantitative (P = 0.206) and qualitative variation (P = 0.068) as shown in [Table 1]. Although not significant, a higher proportion of high-quality trials were found to be reported from South India. Based on subspecialty, a larger number of trials were reported from SGE (27%), urology (27%), and general surgery (27%). The RCTs among various subspecialties were however comparable in terms of quality and quantity [Table 1].

Figure 2: Assessment of overall surgical trials with time: (a) comparison of median Jadad scores over time and (b) comparison of number of high-quality trials over time

Click here to view

Table 1: Assessment of surgical trials with regions within the country and between surgical subspecialties

Click here to view

Factors associated with the quality of trials

On assessing various factors associated with better quality of the study, higher impact factor of the journal (P = 0.013) and review by IRB (P = 0.004) were found to be significantly associated with a better study quality [Table 2]. Type of institution, intervention, centricity, number of authors, assistance of statistician, and funding source of trial did not have a significant association with the study quality.

 :: Discussion 

The assessment of the quality of RCTs, which is considered to be the standard method for “rational therapeutics” in medicine, is essential because poor-quality trials can affect the interpretations about the available evidence.[2] There is a lack of focused studies evaluating the quality of surgical RCTs from India. In the present study, the published RCTs from India in various surgical specialties over past 8 years were assessed for quality using the Jadad score. The surgical trials from India were found to be of optimum quality. A relatively higher number of good-quality trials (64.5%) were reported in the present study when compared to 34–38% in the previous reports.[5],[17] A better study quality was found to have an association with higher impact factor and scrutiny by IRB. Study quality was not associated with the type of institution, intervention, centricity, number of authors, assistance of statistician, and funding source of trial.

Van Tulder scale score, the Delphi List, the CONSORT statement, and the Cochrane Back Review Group criteria are some of the common scoring systems used to assess the methodological quality of RCTs.[4],[5],[6],[7],[8],[9] Although these are elaborate and well-validated scoring systems, it can be potentially tedious in assessing the study quality. Jadad score is one of the most frequently used scores for assessing the methodological quality of a trial.[10] Blinding, randomization, and description of dropouts are the three criteria recommended as the basic minimum assessment tools before inclusion of trials in meta-analysis and systematic reviews by the Cochrane Review Group. It is known to have good validity and reliability.[10],[11],[12] Its brevity and ease of use makes it one of the most widely used scales. In the present study, blinding was not done in 83.17% of the trials analyzed; however, it was found that blinding was not possible in 48 (47.5%) trials studied. The use of the modified Jadad score thus helped to avoid misinterpreting the quality of studies.

Balasubramanian et al.[17] analyzed the quality of reporting of surgical RCTs published in 2003 in top-10 impact factor journals using the Jadad score, allocation concealment score, and a modified CONSORT score. The study reported 62% of RCTs to be of poor quality with a Jadad score of <3. However, in this study, there was no evaluation of the study design and feasibility of blinding before scoring the studies in their report, which could significantly alter the scores. In the present study over a period of 8 years, on using the Jadad score criteria of ≥3, only 59% were of high quality; however, on application of a modified Jadad score, 64.5% of the studies were found to be of good quality. The use of a modified Jadad score in the present study has contributed to this difference, as use of cutoff score of 3 is likely to falsely identify trials in which blinding is not feasible as poor quality. This underscores the need for using the modified criteria especially in surgical RCTs, where a blinded design is not always possible. The reporting of surgical trials was found to be suboptimal in the previous report published in 2006.[17] A quality assessment report of RCTs was carried out by Chung et al.,[5] which identified 34% of the surgical trials to be of good quality with the Jadad score. The authors found the Jadad scoring to be in agreement with other scales utilized in the same study.

Chan et al.[19] in a previous report assessed the epidemiology and reporting characteristics of all PubMed-indexed RCTs published in 2005. The authors found adequate reporting of randomization, blinding, and attrition in only 21%, 48%, and 34% of the reported trials. Zhang et al., in a similar quality assessment of RCTs conducted in China, found that randomization and blinding were adequate in only one-third of the studies; however, dropouts were appropriately mentioned in 53% of the studies.[20],[21] In a study on quality assessment of RCTs in oral and maxillofacial surgery, the mean scale of the Jadad score was 3.06 points.[22] In the present study, randomization was adequate in 82%, blinding appropriate in 28%, and dropouts were clearly mentioned in 57% of the studies. Many journals around the year 2007 made it mandatory to mention the registration number of the RCTs while submitting the manuscript, which led to a widespread awareness and registration of trials. This could be one of the reasons for the overall higher quality of trials in the present study when compared to Chan et al.[19] and other studies.[5],[20],[21]

In the present study, although not significant, analysis of trials based on regional divisions showed that better quality trials were conducted from the southern states, although the number of trials reported was relatively higher from the northern part of the country. Among subspecialties, the quality of trials was comparable; however, a relatively higher proportion of high-quality trials were from neurosurgery and general surgery when compared to the other surgical subspecialties. Azad et al.[23] also found a higher quality of trials in neurosurgery improvised with time. Assessment of quality and quantity of RCTs in the field of orthopedics also showed a rise in both quality and quantity from the year 2001 to 2013.[24] A relatively higher proportion of higher quality studies from general surgery may be attributed to the fact that majority of the subspecialty centers are relatively recent when compared with the general-surgical centers and hence a better experienced research methodology and reporting in general surgery may have contributed to the difference.

Analyzing the various facilitating and hindering factors is important for improvising the standards of research and reporting. In the present study, although a majority of RCTs were reported from public sector institutions, there was no difference in the quality of the research from private institutions, when compared to public sector institutions. The number of authors was not significantly associated with the Jadad score or quality of the study; however, with >10 authors, trend of reduced quality was seen. This may be an indirect indication of existence of guest or gift authorship, which is known to hinder the quality of the research. In the present study, most of the RCTs (160/162) included were single-center studies. Large number of authors may be a norm for multicentric studies due to the involvement of many centers. Similar results were reported in a previous study, which showed involvement of six to nine authors and multicentric trials to be of higher quality.[17] In the present study, as only two multicentric trials were included, it was not possible to analyze the effect of multicentricity on the quality of the reported trials. The involvement of statistician was associated with relatively increased proportions of higher quality studies (85% vs. 62%); however, the association was not significant probably owing to the large discrepancy in the numbers with and without involvement of a statistician. In the present study, the statistician involvement was analyzed based on the author affiliations only. This could have potentially caused some missing data on the involvement of statistician and could have affected the results. Statistician's involvement helps to improvise the statistical analysis resulting into an overall good-quality publication.

Impact factor of journal and quality of trials has been correlated in many reports.[5],[12] In a previous report, a good correlation was found between impact factor of the journal and the CONSORT score of the studies evaluated.[5] Hassan et al.[12] reported a moderate positive correlation between impact factor and Jadad scores of the included trials. Similar results were found in the present study. A higher impact factor of the journal implies strict quality-control measure, adherence to reporting guidelines, and an effective peer review system, which can contribute to ensuring quality of the trials published in those journals. Review by IRB was found to be associated with a better quality in the present study. Chung et al.[5] in a previous study reported that articles approved by IRB to be of higher quality. Review and scrutiny by an IRB help to ensure the feasibility of the study design and the validity of the protocol being proposed. This helps in improvising the quality of the conduct of studies. In the present study, IRB review was assessed based only on the reporting in the manuscript by the authors. Although it is a fundamental requirement for conduct of RCT, review by IRB was reported only in 86% of the studies. However, it is possible that although IRB approval must have been taken, a statement regarding that may not have been included in the manuscript. As this study was over a period of last 8 years, it was not possible to verify the same from the individual journal or authors. IRB scrutiny was found to be significantly associated with the quality of study as well and hence mandatory verification and mention in the manuscript of IRB review can potentially improve the quality. Funding source was not mentioned in 147/162 trials in the present study. In this study, analysis was carried out based on the information of mention of the funding source by the authors. However, a differentiation between academic or investigator-initiated and pharma-sponsored trials was not carried out as this was not a part of the study. This is in contrast to the previous epidemiological report where the funding source was unavailable only in 10% of included trials.[19] In the present study, the quality of trials was not found to be correlating with funding source as very few trials had mentioned them. Overall funded trials should have a higher quality, as most of these will undergo a further scrutiny at the funding level. Apart from the overall study, it is also important to adhere to the abstract guidelines of the abstract checklist of RCTs. Gallo et al.[25] reported a limited adherence to the CONSORT for abstracts checklist among the leading five plastic surgery journals.

The study is not without limitations. In the present study, the authors have utilized only Jadad score for quality assessment. This score, in spite of its advantages, has a few inherent limitations such as lack of assessment of allocation concealment and inter-rater variability. However, in the present study, the scoring was carried out by five independent reviewers (senior resident and consultant level) who are familiar with the research methodology and a median score was used to improve the reliability and inter-reviewer agreement was also analyzed.[12] Previous reports using multiple scoring systems demonstrated a moderate correlation between Jadad scores, allocation concealment, and CONSORT scores, which further emphasize the validity of the Jadad score indirectly. Owing to its ease of use and the fact that it assesses the three key factors causing bias, the authors preferred the Jadad score as the assessment tool. The authors utilized the modified score, as in surgical trials a blinded design is not always possible, thus leading to a false assessment as low quality when using a usual cutoff score of 3. The assessment of the feasibility of blinding in the study design can be subjective and hence assessment was carried out by five independent assessors to reduce potential bias.

 :: Conclusions 

The quantity and quality of surgical RCTs reported from India have remained stable over 8 years. The RCTs from various geographical regions in India and subspecialties were comparable in quality and quantity. Higher impact factor and review by IRB were found to be significantly associated with a better study quality. Type of institution, intervention, centricity, number of authors, assistance of statistician, and funding source of trial did not have a significant association with the study quality. Although the quality of Indian trials in surgical specialties is optimal, the scientific community should work together to further improve quality of surgical trials. A better understanding of randomization and blinding and thorough review by a research committee can help in improvising the design and conduct of a good-quality study. Strict adherence to the CONSORT guidelines can further improve the standards of reporting of trials. The key message of this study is that the quality and quantity of surgical trials have been stable over the years; geographical regions and subspecialties did not affect the quality and quantity of the trials; and a higher impact journal and review by IRB were associated with a better study quality. Type of institution, number of authors, involvement of statistician, and funding source of trial did not affect the quality of the trial.

Acknowledgement

The authors would like to thank Dr. Tulasingam Mahalakshmy, Additional Professor and Dr. Premkumar Ramasubramani, Junior Resident, Department of Preventive and Social Medicine (PSM), Jawaharlal Institute of Postgraduate Medical Education & Research (JIPMER), Pondicherry, India, for their help in the statistical analysis. The authors would also like to thank Dr. Souraja Dutta, former Intern, Department of Surgery, JIPMER, Pondicherry, India, for her help in data collection.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

 

 :: References 
1.Jadad AR. Randomised controlled trials: A user's guide. London, England: BMJ Books; 1998.  
    2.Maria K, Christian R, Monika SG, Maria B. Randomized controlled trials. Series on evaluation of scientific publications. Dtsch Arztebl Int 2011;108:663–8.  
    3.Chung W, Lee KW, Hwang IH, Lee DH, Kim SY. Quality assessment of randomized controlled trials in the Journal of the Korean Academy of Family Medicine. Korean J Fam Med 2009;30:626-31.  
    4.Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Control Clin Trials 1996;17:1-12.  
    5.Chung JH, Kang DH, Jo JK, Lee SW. Assessing the quality of randomized controlled trials published in the Journal of Korean Medical Science from 1986 to 2011. J Korean Med Sci 2012;27:973-80.  
    6.Berger VW, Alperson SY. A general framework for the evaluation of clinical trial quality. Rev Recent Clin Trials 2009;4:79–88.  
    7.van Tulder M, Furlan A, Bombardier C, Bouter L. Updated method guidelines for systematic reviews in the Cochrane Collaboration Back Review Group. Spine 2003;28:1290-9.  
    8.Verhagen AP, Henrica CW, Robert A, Kessels AG, Boers M, Bouter LM, et al. The Delphi List: A Criterial List for Quality Assessment of Randomized Clinical Trials for Conducting Systematic Reviews Developed by Delphi Consensus. J Clin Epidemiol 1998;51:1235–41.  
    9.Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al. The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Ann Intern Med 2001;134:663–94.  
    10.Olivo SA, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ. Scales to assess the quality of randomized controlled trials: A systematic review. Phys Ther 2008;88:156-75.  
    11.Bhatt A. Quality of clinical trials: A moving target. Perspect Clin Res 2011;2:124–8.  
  [Full text]  12.Hassan WM, Mehanna M, Nour R, Yehia D, Atta N, Alfaar A. Association between quality of clinical trials and human development index in heart failure using JADAD scale. Principles Practice Clin Res 2015;1:40-6.  
    13.Karri V. Randomised clinical trials in plastic surgery: Survey of output and quality of reporting. J Plast Reconstr Aesthet Surg 2006;59:787–96.  
    14.Moss RL, Henry MCW, Dimmit RA, Rangel S, Geraghty N, Skarsgard ED. The role of prospective randomized clinical trials in pediatric surgery: State of the art? J Pediatr Surg 2001;36:1182–6.  
    15.Kaido T, Uemoto S. Quality assessment of randomized controlled trials involving liver surgery. Hepatogastroenterology 2008;55:1678–80.  
    16.Kaido T. Recent randomized controlled trials in pancreaticoduodenectomy. Pancreas 2006;33:228-32.  
    17.Balasubramanian SP, Wiener M, Alshameeri Z, Tiruvoipati R, Elbourne D, Reed MW. Standards of reporting of randomized controlled trials in general surgery: Can we do better? Ann Surg 2006;244:663-7.  
    18.Agha R, Abdall-Razak A, Crossley E, Dowlut N, Losifidis C, Mathew G, et al. STROCSS 2019 guideline: Strengthening the reporting of cohort studies in surgery. Int J Surg 2019;72:156-65.  
    19.Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet 2005;365:1159–62.  
    20.Zhang D, Yin P, Freemantle N, Jordan R, Zhong N, Cheng KK. An assessment of the quality of randomised controlled trials conducted in China. Trials 2008;9:22.  
    21.Zhang J, Chen X, Zhu Q, Cui Q, Cao L, Su J. Methodological reporting quality of randomized controlled trials: A survey of seven core journals of orthopaedics from mainland China over 5 years following the CONSORT statement. Orthop Traumatol Surg Res 2016;102:933-8.  
    22.Trevisan B, Garcia RD, Musskopf ML. Quality assessment of randomised controlled trials in oral and maxillofacial surgery. Br J Oral Maxillofac Surg 2020;58:647-51.  
    23.Azad TD, Feng AY, Mehta S, Bak AB, Johnson E, Mittal V, et al. Randomized controlled trials in functional neurosurgery—Association of device approval status and trial quality. Neuromodulation 2020;23:496-501.  
    24.Smith CS, Mollon B, Vannabouathong C, Fu JM, Sales B, Bhandari M, et al. An assessment of randomized controlled trial quality in the journal of bone and joint surgery: Update from 2001 to 2013. J Bone Joint Surg Am 2020;102:e116.  
    25.Gallo L, Wakeham S, Dunn E, Avram R, Thoma A, Voineskos S. The reporting quality of randomized controlled trial abstracts in plastic surgery. Aesthet Surg J 2020;40:335-41.  
    
  [Figure 1], [Figure 2]
 
 
  [Table 1], [Table 2]