Background: Disability progression is a key milestone in the disease evolution of people with multiple sclerosis (PwMS). Prediction models of disability progression have not yet reached the level of trust needed to be adopted in the clinic. A common benchmark to assess model development in multiple sclerosis is also currently lacking. Methods: Data of adult PwMS with a follow-up of at least three years from 146 MS centers, spread over 40 countries and collected by the MSBase consortium was used. With basic inclusion criteria for quality requirements, it represents a total of 15,240 PwMS. External validation was performed and repeated five times to assess the significance of the results. TRIPOD guidelines were followed. Confirmed disability progression after two years was predicted, with a confirmation window of six months. Only routinely collected variables were used such as the expended disability status scale, treatment, relapse information, and MS course. To learn the probability of disability progression, state-of-the-art machine learning models were investigated. The discrimination performance of the models is evaluated on their area under the receiver operator curve (ROC-AUC) and under the precision recall curve (AUC-PR), and their calibration via the Brier score and the expected calibration error. Findings: A temporal attention model was the best model. It achieved a ROC-AUC of 0.71 ± 0.01, an AUC-PR of $0.26 ± 0.02$, a Brier score of $0.1 ± 0.01$ and an expected calibration error of $0.07 ± 0.04$. The history of disability progression is more predictive for future disability progression than the treatment or relapses. Interpretation: Good discrimination and calibration performance on an external validation set is achieved, using only routinely collected variables. This makes these models ready for a clinical impact study. % in MS centers participating in MSBase. All our preprocessing and model code is available at https://gitlab.com/edebrouwer/ms_benchmark, making this task an ideal benchmark for predicting disability progression in MS.

Dana Horakova received speaker honoraria and consulting fees from Biogen, Merck, Teva, Roche, Sanofi Genzyme, and Novartis, as well as support for research activities from Biogen and Czech Minsitry of Education. Eva Kubala Havrdova received honoraria/research support from Biogen, Merck Serono, Novars, Roche, and Teva; has been member of advisory boards for Actelion, Biogen, Celgene, Merck Serono, Novars, and Sanofi Genzyme; has been supported by the Czech Ministry of Educaon research project PROGRES Q27/LF1. Francesco Patti received speaker honoraria and advisory board fees from Almirall, Bayer, Biogen, Celgene, Merck, Novartis, Roche, Sanofi-Genzyme and TEVA. He received research funding from Biogen, Merck, FISM (Fondazione Italiana Sclerosi Multipla), Reload Onlus Association and University of Catania. Guillermo Izquierdo received speaking honoraria from Biogen, Novartis, Sanofi, Merck, Roche, Almirall and Teva. Sara Eichau received speaker honoraria and consultant fees from Biogen Idec, Novartis, Merck, Bayer, Sanofi Genzyme, Roche and Teva. Marc Girard received consulting fees from Teva Canada Innovation, Biogen, Novartis and Genzyme Sanofi; lecture payments from Teva Canada Innovation, Novartis and EMD . He has also received a research grant from Canadian Institutes of Health Research. Pierre Duquette served on editorial boards and has been supported to attend meetings by EMD, Biogen, Novartis, Genzyme, and TEVA Neuroscience. He holds grants from the CIHR and the MS Society of Canada and has received funding for investigator initiated trials from Biogen, Novartis, and Genzyme. Alessandra Lugaresi has served as a Biogen, Bristol Myers Squibb, Merck Serono, Novartis, Roche, Sanofi/ Genzyme and Teva Advisory Board Member. She received congress and travel/accommodation expense compensations or speaker honoraria from Biogen, Merck, Mylan, Novartis, Roche, Sanofi/Genzyme, Teva and Fondazione Italiana Sclerosi Multipla (FISM). Her institutions received research grants from Novartis and Sanofi Genzyme. Pierre Grammond has served in advisory boards for Novartis, EMD Serono, Roche, Biogen idec, Sanofi Genzyme, Pendopharm and has received grant support from Genzyme and Roche, has received research grants for his institution from Biogen idec, Sanofi Genzyme, EMD Serono. Tomas Kalincik served on scientific advisory boards for BMS, Roche, Janssen, Sanofi Genzyme, Novartis, Merck and Biogen, steering committee for Brain Atrophy Initiative by Sanofi Genzyme, received conference travel support and/or speaker honoraria from WebMD Global, Eisai, Novartis, Biogen, Sanofi Genzyme, Teva, BioCSL and Merck and received research or educational event support from Biogen, Novartis, Genzyme, Roche, Celgene and Merck. Raed Alroughani received honoraria as a speaker and for serving on scientific advisory boards from Bayer, Biogen, GSK, Merck, Novartis, Roche and Sanofi Genzyme. Maria Pia Amato received honoraria as consultant on scientific advisory boards by Biogen, Bayer Schering, Merck, Teva and Sanofi Aventis; has received research grants by Biogen, Bayer Schering, Merck, Teva and Novartis. Francois GrandMaison received honoraria or research funding from Biogen, Genzyme, Novartis, Teva Neurosciences, Mitsubishi and ONO Pharmaceuticals. Katherine Buzzard received honoraria and consulting fees from Biogen, Teva, Novartis, Genzyme Sanofi, Roche, Merck, CSL and Grifols. Murat Terzi received travel grants from Novartis, Bayer Schering, Merck and Teva; has participated in clinical trials by Sanofi Aventis, Roche and Novartis. Cavit Boz received conference travel support from Biogen, Novartis, Bayer Schering, Merck and Teva; has participated in clinical trials by Sanofi Aventis, Roche and Novartis. Jeannette Lechner Scott travel compensation from Novartis, Biogen, Roche and Merck. Her institution receives the honoraria for talks and advisory board commitment as well as research grants from Biogen, Merck, Roche, TEVA and Novartis. Samia J. Khoury received compensation for participation in the Novartis Maestro program. Vincent van Pesch has received travel grants from Merck, Biogen, Sanofi, Bristol Myers Squibb, Almirall and Roche; his institution receives honoraria for consultancy and lectures and research grants from Roche, Biogen, Sanofi, Merck, Bristol Myers Squibb, Janssen, Almirall and Novartis Pharma. Radek Ampapa received conference travel support from Novartis, Teva, Biogen, Bayer and Merck and has participated in a clinical trials by Biogen, Novartis, Teva and Actelion. Julie Prevost accepted travel compensation from Novartis, Biogen, Genzyme, Teva, and speaking honoraria from Biogen, Novartis, Genzyme and Teva. Daniele Spitaleri received honoraria as a consultant on scientific advisory boards by Bayer-Schering, Novartis and Sanofi Aventis and compensation for travel from Novartis, Biogen, Sanofi Aventis, Teva and Merck. Cristina Ramo Tello received research funding, compensation for travel or speaker honoraria from Biogen, Novartis, Genzyme and Almirall. Claudio Solaro served on scientific advisory boards for Merck, Genzyme, Almirall, and Biogen; received honoraria and travel grants from Sanofi Aventis, Novartis, Biogen, Merck, Genzyme and Teva. Davide Maimone served on scientific advisory boards for Bayer, Biogen, Merck, Sanofi Genzyme, Novartis, Roche, and Almirall; received honoraria and travel grants from Sanofi Genzyme, Novartis, Biogen, Merck, and Roche. Gerardo Iuliano had travel/accommodations/meeting expenses funded by Bayer Schering, Biogen, Merck , Novartis, Sanofi Aventis, and Teva. Celia Oreja Guevara received honoraria as consultant on scientific advisory boards from Biogen, Celgene, Merck, Novartis, Roche, Sanofi Genzyme and TEVA. Bart Van Wijmeersch received research and rravel grants, honoraria for MS Expert advisor and Speaker fees from Bayer Schering, Biogen, Sanofi Genzyme, Merck, Novartis, Roche and Teva. Pamela McCombe received speakers fees and travel grants from Novartis, Biogen, Tevalua, Sanofi Mark Slee has participated in, but not received honoraria for, advisory board activity for Biogen, Merck , Bayer Schering, Sanofi Aventis and Novartis. Tamara Castillo Trivino received speaking/consulting fees and/or travel funding from Bayer, Biogen, Merck, Novartis, Roche, Sanofi Genzyme and Teva. Jose Luis Sanchez Menoyo accepted travel compensation from Novartis, Merck and Biogen, speaking honoraria from Biogen, Novartis, Sanofi, Merck, Almirall, Bayer and Teva and has participated in clinical trials by Biogen, Merck and Roche Ricardo Fernandez Bolanos received speaking honoraria from Biogen, Novartis, Merck and Teva. Marie Dhooghe received consultancy and advisory board fees from Roche, Sanofi Genzyme, Biogen, Merck Serono, Bayer Schering, Novartis and Allergan; received congress support from Biogen, Merck Serono, Teva and Roche. She has also received research support from Novartis, Biogen, Roche, FWO (Research Foundation Flanders) and Fonds D.V. (Ligue Nationale Belge de la Sclerose en Plaques, Fondation Roi Baudouin). Guy Laureys received travel and/or consultancy compensation from Sanofi Genzyme, Roche, Teva, Merck, Novartis, Celgene, Biogen. Anneke van der Walt served on advisory boards and receives unrestricted research grants from Novartis, Biogen, Merck and Roche She has received speakers honoraria and travel support from Novartis, Roche, and Merck. She receives grant support from the National Health and Medical Research Council of Australia and MS Research Australia. Helmut Butzkueven Institution (Monash university) has received compensation for consulting, talks, advisory / steering board activities from Biogen, Merck, Novartis, Genzyme, Alfred Health; research support from Novartis, Biogen, Roche, Merck, NHMRC, Pennycook Foundation, MSRA. HB has received compensation for same activities from Oxford Health Policy Forum, Merck, Biogen, Novartis. Michael Barnett served on scientific advisory boards for Biogen, Novartis and Genzyme and has received conference travel support from Biogen and Novartis. He serves on steering committees for trials conducted by Novartis. His institution has received research support from Biogen, Merck and Novartis. Jiwon Oh has received research funding from the MS Society of Canada, National MS Society, Brain Canada, Biogen, Roche, EMD Serono (an affiliate of Merck KGaA); and personal compensation for consulting or speaking from Alexion, Biogen, Celgene (BMS), EMD Serono (an affiliate of Merck KGaA), Novartis, Roche, and Sanofi Genzyme. Ayse Altintas received speaker honoraria from Merck, Alexion,; received travel and registration grants from Merck, Biogen Gen Pharma, Roche, Sanofi Genzyme. Yara Fragoso received honoraria as a consultant on scientific advisory boards by Novartis, Teva, Roche and Sanofi Aventis and compensation for travel from Novartis, Biogen, Sanofi Aventis, Teva, Roche and Merck. Edgardo Cristiano received honoraria as consultant on scientific advisory boards by Biogen, Bayer Schering, Merck, Genzyme and Novartis; has participated in clinical trials/other research projects by Merck, Roche and Novartis. Tunde Csepany received speaker honoraria/ conference travel support from Bayer Schering, Biogen, Merck , Novartis, Roche, Sanofi Aventis and Teva. Suzanne Hodgkinson received honoraria and consulting fees from Novartis, Bayer Schering and Sanofi, and travel grants from Novartis, Biogen Idec and Bayer Schering. Norma Deri received funding from Bayer, Merck , Biogen, Genzyme and Novartis. Shlomo Flechter received research funding, speaker honoraria and compensation for travel from and served as a consultant on advisory board for Bayer Schering, Teva, Biogen, Merck , Genzyme and Novartis. Bruce Taylor received funding for travel and speaker honoraria from Bayer Schering Pharma, CSL Australia, Biogen and Novartis, and has served on advisory boards for Biogen, Novartis, Roche and CSL Australia. Fraser Moore participated in clinical trials sponsored by EMD Serono and Novartis. Orla Gray received honoraria as consultant on scientific advisory boards for Genzyme, Biogen, Merck, Roche and Novartis; has received travel grants from Biogen, Merck, Roche and Novartis; has participated in clinical trials by Biogen and Merck. Csilla Rozsa received speaker honoraria from Bayer Schering, Novartis and Biogen, congress and travel expense compensations from Biogen, Teva, Merck and Bayer Schering. Allan Kermode received speaker honoraria and scientific advisory board fees from Bayer, BioCSL, Biogen, Genzyme, Innate Immunotherapeutics, Merck , Novartis, Sanofi, Sanofi Aventis, and Teva. Magdolna Simo received speaker honoraria from Novartis, Biogen, Bayer Schering; congress/travel compensation from Teva, Biogen, Merck , Bayer Schering. Cameron Shaw received travel assistance from Biogen and Novartis. Todd Hardy has received speaking fees or received honoraria for serving on advisory boards for Biogen, Merck, Teva, Novartis, Roche, Bristol Myers Squibb and Sanofi Genzyme, is Co Editor of Advances in Clinical Neurosciences and Rehabilitation, and serves on the editorial board of Journal of Neuroimmunology and Frontiers in Neurology. Pamela McCombe received honoraria and consulting fees from Novartis, Bayer Schering and Sanofi and travel grants from Novartis, Biogen and Bayer Schering. Bhim Singhal received consultancy honoraria and compensation for travel from Biogen and Merck. Ilya Kister served on scientific advisory board for Biogen and received research support from Guthy Jackson Charitable Foundation, National Multiple Sclerosis Society, Biogen, , and Novartis. Neil Shuey received travel compensation from Bayer Schering, Novartis, and Biogen Idec. Tatjana Petkovska Boskova received congress and travel expense compensations from Biogen Idec and Teva. Nikolaos Grigoriadis received honoraria, consultancy/lecture fees, travel support and research grants from Biogen Idec, Biologix, Novartis, TEVA, Bayer, Merck Serono, Genesis Pharma, Sanofi Genzyme, ROCHE, Cellgene , ELPEN and research grants from Hellenic Ministry of Development.

Edward De Brouwer is funded by a FWO-SB grant. This research received funding from the Flemish Government under the Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen programme.

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

Approval to access MSBase dataset was approved by the MSBase Scientifc Leading Group Research Committee. Request project code : 2020-015. Due to the collaborative nature of the MSBase registry, individual ethical approval for each participating center is required by the MSBase central ethics committee. Patient consent is required for all patients.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Yes

The data used in study originates from the MSBase data set which can be requested through the MSBase organization that facilitates data sharing agreements with each individual center.

https://www.msbase.org/