Objective: To evaluate whether the performance of the large language model (LLM) Llama2 improves with pre-training and fine-tuning, and to compare its predictive accuracy with that of a linear regression model for rheumatoid arthritis (RA) disease activity. Methods: Clinical data from 11,865 patients in the cohort were used to predict disease activity at two years on four indices (Disease Activity Score (DAS) 28-Erythrocyte sedimentation rate (ESR), DAS28-C-reactive protein (CRP), Clinical Disease Activity Index (CDAI) or Japanese Health Assessment Questionnaire (J-HAQ)). Logistic regression was employed for the linear model for comparison. The predictive performance was assessed using area under the curve (AUC). Additional performance metrics including precision, recall, and F1 score were calculated. Results: Pre-training significantly improved AUC of Meditron (Llama2 pre-trained with medical data) for DAS28-ESR >5.1, DAS28-ESR <2.6, DAS28-CRP <2.3, J-HAQ score >2.5, and J-HAQ score <0.5 (P <0.05). Fine-tuning resulted in significant improvements in AUC for Llama2 across all indices (P <0.05) except CDAI >22, and for Meditron in DAS28-ESR <2.6, DAS28-CRP >4.1, DAS28-CRP <2.3 and CDAI ≤2.8 (P <0.05). Both LLMs significantly outperformed linear regression in predicting DAS28-ESR <2.6, DAS28-CRP >4.1, DAS28-CRP <2.3, J-HAQ score >2.5, and J-HAQ score <0.5 (P <0.05). Furthermore, DAS28-CRP >4.1, DAS28-CRP <2.3, J-HAQ score >2.5 and J-HAQ score <0.5, Llama2 or Meditron consistently outperformed linear regression across all performance metrics. Conclusion: Both pre-training and fine-tuning significantly improved the performance of Llama2. Both LLMs outperformed the linear regression model in predicting 5 out of the 8 categories of indices.

Competing interests S.H. and M.F. declares no conflicts of interest. K.I. has received speakers fee from Asahi Kasei Pharma Co., Astellas Pharma Inc., AbbVie Japan GK, Ayumi Pharmaceutical Corporation, Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Ltd., Eisai Co., Ltd., Eli Lilly Japan K.K., Janssen Pharmaceutical K.K., Kaken Pharmaceutical Co. Ltd., Mitsubishi Tanabe Pharma Co., Pfizer Japan Inc., Takeda Pharmaceutical Co. Ltd., Teijin Pharma Ltd, and UCB Japan Co. Ltd. Division of Multidisciplinary Management of Rheumatic Diseases is an endowment department, supported with an unrestricted grant from Ayumi Pharmaceutical Corp., Chugai Pharmaceutical Co., Ltd., Mitsubishi Tanabe Pharma Co., Mochida Pharmaceutical Co., Ltd., Nippon Kayaku Co., Ltd. Teijin Pharma Ltd. ET has received lecture fees or consulting fees from AbbVie Japan GK, Asahi Kasei Corp., Astellas Pharma Inc., Ayumi Pharmaceutical Co., Boehringer Ingelheim Japan, Inc., Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Ltd., Daiichi-Sankyo, Inc., Eisai Co., Ltd., Eli Lilly Japan K.K., Gilead Sciences, Inc., Pfizer Japan Inc, Nichi-Iko Pharmaceutical Co., Ltd., Taisho Pharmaceutical Co., Ltd, Takeda Pharmaceutical Co., Ltd, Mitsubishi Tanabe Pharma Co., UCB Japan Co. Ltd. and Viatris Inc. ET has received research funding from Pfizer Inc. and UCB Japan Co. Ltd. MH has received research grants from AbbVie Japan GK, Asahi Kasei Corp., Ayumi Pharmaceutical Co., Boehringer Ingelheim Japan, Inc., Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Eisai Co., Ltd., Eli Lilly Japan K.K., Kaken Pharmaceutical Co., Ltd., Mitsubishi Tanabe Pharma Co., Mochida Pharmaceutical Co., Ltd., Nippon Kayaku Co., Ltd., Nippon Shinyaku Co., Ltd., Pfizer Japan Inc., Taisho Pharmaceutical Co., Ltd., Teijin Pharma Ltd., UCB Japan Co., Ltd., and Viatris Japan. MH has received speakers fee from AbbVie Japan GK, Asahi Kasei Corp., Astra Zeneca K. K., Ayumi Pharmaceutical Co., Boehringer Ingelheim Japan, Inc., Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Ltd., Eisai Co., Ltd., Eli Lilly Japan K.K., GlaxoSmithKline K.K., Gilead Sciences Inc., Janssen Pharmaceutical K.K., Kissei Pharmaceutical Co., Ltd., Mitsubishi Tanabe Pharma Co., Mochida Pharmaceutical Co., Ltd., Ono Pharmaceutical Co., Ltd., Pfizer Japan Inc., Taisho Pharmaceutical Co., Ltd., and Teijin Pharma Ltd. MH is a consultant for AbbVie, Boehringer-ingelheim, Bristol Myers Squibb Co., Kissei Pharmaceutical Co., Ltd. and Teijin Pharma.

Funding This work was supported by JSPS KAKENHI Grant Numbers JP23K24465 and Research grant from the Program for the Promotion of Precision Medicine in Rheumatoid Arthritis, Japan College of Rheumatology, 2023

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

Ethics approval The IORRA cohort study (#2952-R and #2922-R16) were approved by the ethics committee of Tokyo Women's Medical University, and informed consent was obtained from all patients before each survey.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes