This project aimed to investigate the capabilities of AI in conducting scientific writing. The process of scientific writing is multifaceted and requires many different elements: time, a thorough review of relevant literature, ethical use of referenced literature, a comprehensive understanding of scientific writing, accuracy, and editing. Our primary objective was to identify how AI compares to humans in these metrics.

Regardless of the modality used, AI reduced the number of hours required to write a review by a wide margin. The human-written paper took over 35 h more to complete when compared to either AI paper. When examining the difference between the AIO and human paper, three key time differences became evident. The human paper required a literature review and took longer to physically write. AIO conducted no real literature review but rather pulled directly from its knowledge base to write immediately, and it was a faster writer. On the other hand, AIO required extensive fact-checking compared to the human paper. The AIA paper took a similar amount of time to write as the human paper, primarily due to the large number of queries required to complete the paper. The same literature review time was given to both the human and the AIA papers, as all references used in the AIA paper were found in the human-conducted literature review. However, the actuality of the AIA paper is that it would require significantly less time to conduct a literature review. The literature review conducted for the human paper encompassed both finding the appropriate sources and reading the articles. If the literature review was conducted solely for the AIA paper, it would only require identification of appropriate sources and brief skimming of the articles versus the full-length reading, thus saving time. Overall, the AIA and AIO papers took less time to write than the human-written review and have the potential to save time in scientific writing.

When examining other aspects of the writing process, it became clear that each review presented its own unique challenges and weaknesses. The primary issue found in the human-written paper, in addition to the time spent, was the extent of editing. This paper spent the most time in edits and had the lowest similarity score, primarily to rewrite sections to improve readability.

The AIO review presented two problems which were, in essence, the exact opposite of those found in the human paper. While the AIO paper had excellent writing technique, the content was both superficial and inaccurate. The superficiality and inaccuracy of the paper required extensive human editing, so it also had a low similarity score and spent a similar amount of time in the editing process. The inaccuracy bled into the references used, and because of this over 90% of the references in the AIO review were added manually during editing.

It may be assumed that human issues with writing flow and AI factual inaccuracies may be remedied by a human-conducted literature review and then using ChatGPT for writing purposes. This is mostly true, as the AIA paper had a similar ease of readability as the AIO paper and was just as accurate as the human paper. However, the AIA paper brought to light the challenge of plagiarism. To generate the AIA review, ChatGPT reads the referenced article and draws conclusions. During that process, it appeared to oftentimes describe the study by directly copying the abstract of that article. This issue undoubtedly had to be addressed, and a thorough examination of the AIA paper was conducted with subsequent editing to certify that the review would not be considered plagiarized. This was evaluated using the Turnitin software and examining the plagiarism similarity indices by comparing the manuscript to published literature and sources on the internet. Typically, a plagiarism similarity index of under 24% is “green” or not considered to be plagiarized. As such, although the first draft of the AIA paper was “yellow” and thus can be flagged for potential plagiarism, the final draft of the review was under the 24% threshold.

All review articles were peer-review as per standard Current Osteoporosis Reports procedure. During the revision stage, the AIO paper came back with only minor comments and changes. However, the human and AIA papers came back with more extensive revisions, both primarily focused on reorganization of the text. The critique similarities, primarily concerns about ineffective flow of ideas, may suggest that AI is able to generate outlines with more clarity and better organization than humans, revealing another important advantage of using AI.

The primary objective of this endeavor was to demonstrate the current capacity of AI to generate an appropriate scientific review. We have discovered that, with no human guidance, AI is not yet capable of writing an appropriate, publishable review article due to its fabrication of references, inaccuracies, and lack of detail. Nonetheless, it seems as though its writing ability from a non-scientific, standard essay standpoint is equivalent to or better than the average individual. Moreover, as observed in the AIA paper, it can potentially augment human writing and simultaneously reduce the time required to write, although manual examination of the written text is required to ensure ethical writing. While AI, as it currently stands, likely requires years of learning/updating before it can be seriously considered to write a scientific review with minimal human intervention, it can currently be used as a tool for improved human writing and efficiency.