Comparative analysis of Cochrane and non-Cochrane reviews over three decades

In this observational study, we found that the number of published systematic reviews has increased considerably, whereas Cochrane’s relative share of systematic reviews has decreased throughout its existence. While systematic reviews increased, the publication of Cochrane reviews has decreased steadily since 2015. During the last decade, Chinese first authors produced a large portion of systematic reviews. Gold open access journals were the largest publishers of systematic reviews from 2013 to 2022, whereas high-impact journals were the largest publishers of systematic reviews from 1993 to 2002. The number of authors seemed to increase over time.

We found that publication of systematic reviews is growing rapidly. Compared with other general publications in life sciences, where authors found a growth rate of 5.1% [1], systematic reviews are growing faster. Our findings show that while the motivation for publishing systematic reviews was high, Cochrane reviews were being published less frequently. Cochrane reviews are known to take a long time to complete [21,22,23] compared with systematic reviews [24,25,26]. This may be partly due to the process of a Cochrane review having more mandatory steps involved, e.g., minimum requirements for databases searched, searching trial registries, and requirements regarding which analyses to conduct. This may also explain why they are known overall to be of higher quality [27,28,29,30,31,32] and are valuable for developing healthcare policies [33]. The thought of high demands for resources and long publication times may intimidate potential authors. However, despite this, Cochrane’s author satisfaction was generally high [34]. Recently, Cochrane has begun rethinking their publication model [35] with several key projects aimed at improving the author experience and optimizing the publication process [36].

We found that balances had shifted relating to production and publication of systematic reviews over the past 10 years. Regarding production, especially systematic reviews written by Chinese first authors have expanded, increasing their publication of systematic reviews by a factor of 25 from 2003–2012 to 2013–2022. The same productivity increase among Chinese authors was not seen in Cochrane reviews. Despite the rapid increase in productivity, one Chinese study found similar methodological and reporting quality between Chinese systematic reviews and those from the USA [37]. Regarding publication, gold open access journals had grown to be the largest publishers of systematic reviews from 2013 to 2022. Open access has been gaining popularity and may be a step in the right direction for science [38]. However, gold open access journals operate with article processing charges that authors must pay to get their papers published. These charges can be steep and have been increasing rapidly over time [39]. There is a wish within the scientific community to transition to a diamond open-access model, where articles are published open access but without authors or readers having to pay exorbitant fees [40]. While this is undoubtedly desirable, there are still costs associated with academic publishing and determining how the diamond open-access model should be financed is not an easy matter. With the increasing amount of research production, there has been discussion of “research waste” [41], i.e., unnecessary and non-contributing research. It was argued already in 1994, at the beginning of our inclusion period, that the scientific community should aim towards producing fewer reviews and instead focus on increasing the quality of publications [42]. Since then, we have had 30 years of exponential increase in evidence synthesis. Relating to this concept, several suggestions towards reducing redundancy of reviews have been put forward [43]. Addressing research waste seems to be a topic gaining focus [43, 44]. We also found an increasing number of authors through time. This aligns with the findings of other studies [45, 46]. This may relate to an increasing amount of data in modern research thereby increasing complexity [47]. One study, however, found that adjusting for factors relating to increased complexity did not explain the increase in authorship when it came to trials and non-randomized studies [48]. Furthermore, systematic reviews differ from clinical trials in that data are available remotely through searches, and therefore, the resource demands may be lower. Interestingly, Cochrane reviews had fewer authors than systematic reviews. This may seem counterintuitive, as Cochrane reviews generally have higher demands and should therefore demand more resources. It is not clear what is causing the increasing number of authors in medicine.

Our study had several strengths. Firstly, we analyzed a very large dataset with the goal of having generalizable conclusions. We only searched PubMed, which is a biomedical database, where records are manually screened for indexation as systematic reviews. We thereby avoided a great deal of false positives that traditional searching would produce. This was demonstrated by validating the search and finding a practical precision of 98%. Generally, searches in large bibliographic databases have low precision [49]. One of the authors, a professional data analyst, conducted data cleaning to adjust for wrongly registered data. Furthermore, we collected data on publication dates from the CDSR, which is the primary source of Cochrane reviews. Therefore, we are confident that Cochrane reviews were correctly and sufficiently identified. However, our study also had some limitations. With large datasets, it is likely there were some miscategorized records. Based on our precision estimate and after data cleaning, we estimate this to be about 2% overall, but some areas may have had more than others. Despite the National Library of Medicine’s personal screening of records, we still found some protocols indexed as systematic reviews, indicating that the systematic review filter, while good, is not 100% perfect. Still, it is a more reliable method of retrieving systematic reviews than through searches and the most feasible method owing to the size of our sample. While searching PubMed ensured high external validity, there are undoubtedly more systematic reviews produced in the period indexed in other databases. However, correcting for this bias in our results would only exaggerate the conclusion that systematic reviews are increasing rapidly and constitute an increasingly larger portion of all systematic reviews compared with Cochrane reviews. On the other hand, a small portion of the increase in systematic reviews may be attributed to the post-indexation of older records and journals in PubMed. We do not believe this potential bias would impact the overall conclusions. Generally, we are confident that the assumed miscategorized data would not change the overall conclusions. This study cannot address the quality of the included systematic reviews or if there is an overlap of the research question of these systematic reviews, thus, whether unnecessary systematic reviews are being published.

It is unclear why Cochrane reviews were being published less frequently in the past 10 years. As Cochrane reviews are expected to be updated and despite them not always being updated frequently [50], we expected the number to increase given the combination of new Cochrane reviews being published and older ones being updated. It may be favorable to have more Cochrane reviews published, as they are generally of high quality owing to their thorough peer review process and rigorous methodology [27,28,29,30,31,32]. Some of the decrease in publication of Cochrane reviews may be due to them generally taking a long time to complete [21,22,23] and including many procedural and methodological demands [9, 10]. Future studies exploring why Cochrane reviews were being published less frequently are warranted. The continually increasing publication rates of systematic reviews may risk an information overload, where stakeholders cannot reasonably stay orientated and updated with new literature. It is, however, unclear to which degree the increase in publication of systematic reviews results from overlapping or redundant reviews being published, or if the new studies, in general, are of value to the research community. Further research into field overlap and the methodological quality of newer systematic reviews may clarify this.

In conclusion, the publication of systematic reviews has increased rapidly in the past 30 years, while fewer Cochrane reviews have been published since 2015. Especially Chinese first authors conducted many systematic reviews through the last 10 years. For the same period, gold open access journals were the largest publishers of systematic reviews. It may be favorable to have more Cochrane reviews published, as these are generally of higher quality than systematic reviews. Furthermore, further research regarding metrics related to research waste may clarify questions regarding the utility and value of systematic reviews throughout time.

留言 (0)

沒有登入
gif