Patients with serious mental illnesses (SMI), such as schizophrenia, major depression, and bipolar disorder, face higher mortality incidence rates and premature mortality risks, and this is especially so for ethnic minorities and those with lower socioeconomic status (Bertolini et al., 2023; De Mooij et al., 2019; Rotenberg, 2019). These issues are coupled with the widely-discussed issues of inequality and the lack of mental health resources in such populations (Saxena et al., 2007; Xiang et al., 2020). Patients with SMI also face a higher risk of physical health problems (Fiorillo and Sartorius, 2021), including cardiovascular diseases, metabolic syndrome, and inflammation. Moreover, failure to conduct a timely and comprehensive assessment of the psychological and physiological status of patients with SMI can lead to symptom recurrence (Taylor et al., 2005; Rodriguez-Villa et al., 2021), potentially aggravating their mental health issues. In response to these problems, previous studies have emphasized the importance of regular physical and mental status, exercise ability, and other forms of assessments for such patients (Latoo et al., 2015) to facilitate medical staff's jobs of identifying suicide risks, cognitive impairment, and psychiatric symptom recurrence (Health Assessment Guide for Nurses, 2023). These procedures can also support the provision of timely interventions for patients with SMI. Notably, the treatment and rehabilitation processes for most patients with SMI occur predominantly within community settings (Parker et al., 2019; van Genk et al., 2023). Against this backdrop, continuous monitoring and assessment by medical and community care professionals are crucial, as these efforts may aid in the timely identification and addressing of a myriad of challenges and needs that patients with SMI may encounter. This sustained attention is vital for preventing disease recurrence and plays a significant role in enhancing patients’ self-management capabilities, rehabilitation outcomes, and quality of life, as well as contributes to reducing overall healthcare costs (Dalton-Locke et al., 2021; Frost et al., 2017; Ronaldson et al., 2020).

Rehabilitation strategies for SMI must consider a multitude of factors, including patient lifestyle, economic status, physical health, and education level (Carswell et al., 2022; Koomen et al., 2022). For example, a study conducted with patients with SMI treated with antipsychotic medication found that behaviors such as smoking, drinking alcohol, and using marijuana or cocaine were associated with readmission risk and that the level of medication adherence also had a significant effect on readmission risk (Portela et al., 2022). Another study focused on the self-care ability of patients with SMI in Beijing found that factors such as education level, guardian support for independent living, age, disease process, and physical health status were associated with the self-care ability (Chen et al., 2021; Correll et al., 2017; Jones et al., 2004). Physiological indicators are also important, with one study noting a significant association between psychiatric symptoms and cardiovascular diseases in patients with SMI (Nielsen et al., 2021). Furthermore, cardiovascular diseases are more prevalent in patients with SMI, which may be related to their lifestyle, medication side effects, and the physiological impact of the SMI. The studies cited thus far clearly show that given the multifaceted complexity of factors that must be considered during the rehabilitation of patients with SMI, traditional clinical approaches to rehabilitation assessment may not fully capture all key factors affecting their rehabilitation, especially when these factors are intertwined with patients’ daily lives and community settings. Therefore, advanced methods are needed in addition to traditional clinical interviews, as the latter may not capture the full range of patient data, are limited by their time-consuming nature, and rely on patient collaboration (Chekroud et al., 2021; National Institute of Mental Health, 2017; Stein et al., 2022).

In recent years, data-based methods, particularly machine learning models, have shown great potential for rehabilitation assessment (Dwyer et al., 2018; Sijia et al., 2022; Sijia et al., 2022). However, these models have significant challenges. They often grapple with the intricacies of multidimensional indicators and multimodal data, such as textual medical records, physiological markers, and behavioral observations, which can hinder their ability to fully account for individual variance (Zhang et al., 2022). Additionally, concerns regarding model bias and a lack of interpretability persist, compromising the trustworthiness and practical utility of these approaches in clinical settings (Lambert et al., 2022; Yogarajan et al., 2022).

Multimodal and multitask learning strategies for artificial intelligence (AI) tools may provide new solutions for the aforementioned challenges. Multimodal learning can help the model understand patient information more comprehensively by integrating information with different data structures (e.g., text records, physiological indicators, and behavioral data), enhancing its prediction accuracy and robustness (Boehm et al., 2022; Ochoa et al., 2017; Qiu et al., 2022; Ramachandram and Taylor, 2017). Meanwhile, multitask learning improves not only prediction accuracy by processing multiple related tasks in parallel but also the model's generalization ability by exploring potential correlations between tasks (Liu et al., 2015; Ruder, 2017); it also helps mitigate the model's sensitivity to individual differences and data biases (Harutyunyan et al., 2019; Ruder, 2017). The combination of these technologies is expected to overcome the shortcomings of traditional single-task machine learning models in dealing with individual differences in and the multidimensional complexity of data and provide more accurate and reliable support for personalized rehabilitation assessment and strategy formulation for patients with SMI (Kline et al., 2022; Liu et al., 2015).

Accordingly, we propose a multimodal multitask learning model that combines Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2018) and TabNet (Arik and Pfister, 2021). BERT, with its powerful text processing capabilities, can effectively parse patients’ medical records, whereas TabNet can handle structured diagnostic data with excellence. We hold that the model not only effectively handles the demographic, pathological, and follow-up data of patients with SMI but also improves prediction performance and reduces overfitting by mining potential connections between different rehabilitation assessment tasks.

Considering the rehabilitation assessment needs of patients with SMI, we identified four key clinical indicators as predictive tasks for our model, as follows: “Referral risk,” “Dangerous behavior,” “Medication adherence,” and “Self-awareness”(Fig. 1). These indicators are highly clinically relevant and integral to the recovery process. “Referral risk” assesses the urgency of patients to receive more intensive treatment or professional care, aiding in the early identification of patients at risk of relapse or deterioration who require additional intervention (Bolduc et al., 2015; Kline and Schiffman, 2014; Munk-Olsen et al., 2009; Yang et al., 2022). The “Dangerous behavior” indicator evaluates the actions exhibited by patients that could potentially harm themselves or others (Torrey, 1994; Malakhov et al., 2023; Yesavage, 1984). “Medication adherence” signifies the extent to which patients follow their treatment plans and has a direct effect on treatment outcomes (Higashi et al., 2013; Edgcomb and Zima, 2018; Velligan et al., 2017). “Self-awareness” reflects patients’ cognition and understanding of their own condition status, which is a crucial factor in promoting active patient treatment participation (Bedford and David, 2014; Lysaker et al., 2022).

In the clinical practice of psychiatric rehabilitation, these four indicators do not exist in isolation but are interconnected and collectively influence patient rehabilitation outcomes (Buckley et al., 2004; Budak and Yılmaz, 2019). For instance, a patient's level of self-awareness can directly impact medication adherence, whereas good medication adherence can significantly reduce the risk of referral and dangerous behaviors (Morken et al., 2008; Novick et al., 2010; Swanson et al., 2004; Swartz et al., 1998). The interplay between these indicators suggests that considering their interconnectedness in the model provides a more comprehensive and accurate perspective for assessing patient rehabilitation status. Therefore, the multitask learning model adopted in this study enables the simultaneous evaluation of these interrelated indicators, enhancing model prediction accuracy for each individual indicator and the model's capacity to identify interactions among indicators. Furthermore, by leveraging the advantages of multimodal learning strategies, the model is enabled to delve into and manage the complexity of multisource data, thereby improving model performance and reliability for psychiatric rehabilitation assessment.

To the best of our knowledge, this study marks the inaugural application of multimodal and multitask learning strategies tailored for the rehabilitation assessment of patients with SMI. Unlike previous research that predominantly focused on AI applications for mental disorder diagnosis, we focused on evaluating rehabilitation status, which is crucial for tailoring patient-centric recovery plans. Our objective was to provide clinicians with a tool that encapsulates a broader spectrum of patient data, thereby enhancing the precision and comprehensiveness of the support provided by the AI model for clinical decisions. This is anticipated to refine rehabilitation strategies, improve treatment outcomes, and secure tailored approaches in the continuum of mental healthcare.