Introduction

A Serious Mental Illness (SMI) interferes with a person’s life and ability to function (1). SMIs include but are not limited to Major Depression, Schizophrenia, Schizo-affective Disorder, and Bipolar Disorder. Persons with SMI face severe threats to their health, well-being, and autonomy. Beyond their symptoms, including depressed mood, hallucinations, delusions, and mania, they may face additional complications in varying degrees of severity across various SMIs that add substantially to the disease burden. Specifically, there are five notable complications. Persons with SMI can become isolated (2), both because their symptoms and behavior repel others but also due to their waning capacity for meaningful social interaction (3). Social isolation reduces the likelihood of support in the community and aggravates mental illness (3–5), creating a harmful feedback loop. They become impaired, losing their ability to function in society, including deficits in self-care (6). Persons with SMI are at higher risk for suicide and other forms of self-harm (7, 8). Persons with SMI may engage in aggression/violence (9, 10). As much as 50% of persons with SMI have a co-morbid Substance Use Disorder (SUD) (11), and SUDs are both a mental disorder and a complication of SMI.

Depending on life circumstances, the eventual outcomes of SMI can be devastating for some individuals. Isolation and impairment can lead to loss of employment and income (12), leading to poverty, poor diet (13), and homelessness (14, 15). Their physical health can be affected (16, 17), and poor physical health aggravates mental illness (18), creating a second vicious circle. Aggression/violence often leads to criminal charges and detention in the correctional (19) or forensic mental health systems (20, 21). The result can be physical, psychological, and social deterioration, including shortened life expectancy (22, 23).

The disease process is punctuated by crises when symptoms are most severe, and the risk for harm is greatest. Often, mental health crises are precipitated by intoxication by alcohol or some other psychoactive substance (24). Deterioration, risk of injury, and harm to others impose an obligation on society to prevent these outcomes. Crises often lead to hospitalization to minimize or prevent harm or further deterioration. Historical developments such as deinstitutionalization (25), the widespread acceptance of the recovery model (26), and the development of more comprehensive community services have left inpatient psychiatry with a diminished but critical role in the continuum of psychiatric care: amelioration of symptoms and complications and rapid return to the community.

In this environment, the quality of mental health care provided in hospitals, both in effectiveness and efficiency, becomes of critical importance and widespread interest. Current discussions of healthcare quality have evolved from Donabedian’s conceptual framework (27–29), which considers quality of care proportional to the improvement in health status brought about by medical intervention. Donabedian’s framework defined “health status” as a metric that quantifies the patient’s overall level of health at a particular point in time. When the practitioner assesses the patient’s health status before and after the provision of care, the magnitude of the positive difference is an outcome reflecting the quality of care.

Donabedian’s model conceived of quality of care as involving three elements. Structure included the physical infrastructure, materials, and human resources required to provide care. Process was the work of medical professionals providing care. Hospital quality improvement programs have easily identified structures and processes, incorporating them into their balanced scorecards. However, hospitals can improve the third element, implementing meaningful Outcomes. Given the importance of outcomes to establishing inpatient mental health care quality, it is surprising how few measures have been made available. The Health of the Nation Outcome Scales (HoNOS) (30, 31) is a clinician-rated measure for use with adults in contact with mental health services. It contains 22 items assessing behavior, impairment, symptoms, and social functioning. Each item is scored on a 5-point scale from zero (no problem) to 5 (severe problem). It is used in the UK, Australia, New Zealand, and many parts of Europe. Its use in North America has been limited.

The present work explores the development of a new outcome measure based on the Resident Assessment Instrument-Mental Health (RAI-MH) (32). interRAI (33) is a not-for-profit international network that develops and implements assessment methods to support improved quality of care among vulnerable persons with complex needs in health and social service settings. The Resident Assessment Instrument-Mental Health (RAI-MH) (32), comprised of 460 items, was developed for inpatient mental health services, including acute, forensic, long-stay, and geriatric psychiatry. It was created through a systematic multi-step process including literature reviews, consultations with clinicians and experts, inclusion of items from other interRAI instruments, expert working group sessions, surveys of front-line staff, debriefing sessions after reliability testing, focus groups, and nursing retreats. As a result, the RAI-MH has 19 content areas assessing mental and physical health, status, functioning, cognitive performance, substance use, support systems, and relationships, as well as health service use. In 2005, the Government of Ontario mandated the RAI-MH for use in all government-funded inpatient psychiatric services in Ontario. Since 2005, according to protocol, the RAI-MH has been administered to psychiatric inpatients in Ontario on admission, discharge, and every three months for long-stay patients or whenever there was a significant change in clinical status. As of 2020, over 1.4 million assessments have been completed on over 320,000 unique individuals in Canada (34). Though designed for use in Inpatient Psychiatry, interRAI has adapted the RAI-MH for use in other mental health care settings, including Community Mental Health (interRAI CMH) (35), Emergency Psychiatry (interRAI-ESP) (36), and Child and Youth Mental Health (interRAI ChYMH) (37). interRAI refers to these instruments as “integrated” because they share a common language, conceptual basis, clinical emphasis, collection method, core elements, and care planning protocols. The RAI-MH has been updated and improved, and the new instrument is called the interRAI-MH (38). For the current study, we used the RAI-MH, but the revised and improved instrument contains all the elements that we used for our research. In this paper, we use the terms RAI-MH and interRAI-MH interchangeably.

To examine the feasibility of developing mental health quality indicators (MHQIs) based on the RAI-MH, Perlman and colleagues (39) performed retrospective analyses on two large data sets from over 70 facilities in Ontario. They derived potential MHQIs by examining the empirical improvement patterns in the severity of depressive symptoms (DSI) and cognitive performance (CPS) levels across facilities. While mean scores on these scales showed substantial variation among facilities, these authors concluded that deriving MHQIs using data from the RAI-MH is a feasible approach to assessing the quality of care. Tolonen and colleagues (40) used a multidimensional model of mental illness and various scales from the interRAI-CMH to evaluate outcomes for participants in a residential psychiatric rehabilitation program. They scored the scales before and after a 29-month (median) duration of treatment. Depression, mania, and positive and negative symptoms of psychosis showed significant pre-post improvement. The performance of daily activities and risks for self-harm and harm to others improved from before to after rehabilitation.

The Donabedian model represents a person’s multidimensional health status as a two-dimensional matrix. The vertical dimension quantifies the level of function or performance (health status). The horizontal dimension represents various functional areas (symptoms, impairment, isolation, risk to self, and danger to the public) that contribute to health status. Donabedian suggested that an average status across the different functional areas could represent an individual’s overall health status. We will refer to these functional areas as “components” of mental health status and their combination as the Composite Index of Inpatient Mental Health Status (CIIMHS).

MethodsParticipants

The data analyses reported here used a sample (N=719) that includes all patients in hospital care at a large tertiary care facility during one fiscal year. Their average age was 43 years. Nearly three-quarters were male (73%). Most were never married (69%); 14% were married or had a live-in partner at the time of their admission to the hospital, and 17% were widowed, separated, or divorced. Almost all were English-speaking (99%). Even though more than half (54%) had graduated from high school, and 22% had some post-secondary education, only 8% were employed, and 13% had no income. All others were on a pension, social assistance, or disability.

The research described in this manuscript was subject to Waypoint’s Research Ethics Board Approval of an application entitled “The Development of Clinical Outcome Measures based on the interRAI-MH among clinical programs at Waypoint Centre for Mental Health Care Certificate # CRRA#12.03.01.

Setting

Waypoint Centre for Mental Health Care is a large psychiatric hospital and forensic mental health research facility in Penetanguishene, Ontario, Canada. The hospital is one of four specialty mental health care facilities in Ontario. It provides extensive acute and longer-term psychiatric inpatient and outpatient services to the surrounding region and has Ontario’s only high-secure forensic mental health program.

Table 1 presents salient information regarding Waypoint’s ten inpatient programs, including patients’ legal status (whether patients are voluntary, civilly committed, or forensic), length of stay, clinical complexity, bed count, level of security, whether the program admits female patients and the number of patients who received care in the program during the fiscal year.

Table 1. Hospital programs and mean program CIIMHS scores.

A Forensic patient is an individual charged with a criminal offense with a confirmed or suspected mental disorder who is sent to Waypoint for assessment or treatment by a court or a review board under the terms in the Criminal Code of Canada (CCC). Civil patients either volunteer for inpatient care or are committed to the hospital on an involuntary basis under the terms of the Mental Health Act (Ontario).

The hospital divides its programs into categories depending on the length of stay and the complexity of the mental illnesses typically seen in the program. Acute programs care for patients over a short period (generally less than thirty days} or for slightly more extended periods (31-150 days) because of fixed-program curricula or a fixed period of hospitalization specified in an order from the court. Chronic programs care for patients over an extended period (>150 days). Further, the hospital divides its Chronic programs according to the complexity of the diagnoses of their typical patients. Simple Chronic programs serve patients with severe and persistent mental illness (SPMI). Complex Chronic programs serve patients with SPMI, plus complicating additional diagnoses (e.g., Dementia or Intellectual/Developmental Disabilities).

The Forensic Personality Disorders Program serves men brought into the forensic mental health system before 1992 when Canadian courts often found a person who committed a criminal offense Not Guilty by Reason of Insanity (NGRI) because of a personality disorder alone (PD; usually Antisocial). Due to their high risk to public safety, authorities have refused to transfer these patients to lower levels of security or release them to the community. As a result, these PD-only patients have remained at Waypoint. Since 1992, according to the Criminal Code of Canada (CCC), the court must hear evidence that the accused person has a diagnosis of SMI (including psychosis) to be admitted to the forensic system. Consequently, the number of PD-only patients has remained static. The hospital’s Personality Disorder Program serves men who have been in the hospital for a long time but do not suffer from an SMI, providing our study with a convenient no-SMI comparison group.

Materials

Hirdes and colleagues (41) listed 17 RAI-MH scales that yield a score based on an RAI-MH assessment. From among these, we chose scales that could be categorized into one of seven putative causes of inpatient admission (symptoms of depression, symptoms of psychosis, isolation, impairment, risk of self-harm, aggression/violence, and substance abuse). Seeking scales that would be sensitive to the changes brought about by treatment, we eliminated scales that contained a preponderance of static or historical items that would not change with treatment. Structural validity (42) is related to content validity and relates to the statistical interdependencies among scale scores on a multidimensional instrument and whether or not these subscales reflect coherent and distinct underlying factors. Accordingly, we were interested in deriving CIIMHS components that were relatively independent of one another (orthogonal). To that end, we eliminated scales whose computation depended on scores from other scales. Table 2 presents the results of our scale selection. We chose eight scales, as indicated in the table. The psychometric properties of these scales are good (43, 44).

Table 2. Components of the CIIMHS.

Procedure

Waypoint Decision Support supplied a data file derived from RAI-MH assessments for all 1,491 assessments completed at the hospital during the relevant fiscal year. For each assessment, the file contained a unique but anonymous identifying number for each patient, their program assignment at the time of the assessment, the dates for each assessment, the assessment type (admission, discharge, quarterly, or change in status), and item scores for the eight scales described above.

There were 719 unique patients provided inpatient care at Waypoint during the relevant fiscal year. First, we sought every patient’s most recent RAI-MH assessment before the end of each of the four fiscal quarters. We will refer to these as assessments at T2. Then, for each patient, we sought the immediate previous assessment. We will refer to these as assessments at T1. For patients with numerous T1-T2 pairings, we chose a single T1-T2 pair on the following bases. First, we eliminated a pairing if the patient was transferred to a different program between T1 and T2, unless the patient had only one T1T2 pairing. Second, we chose the pair with the most extended interval between T1 and T2. If there were two or more pairings with roughly equal intervals, we chose the earlier pairing. Finally, to ensure that the choice of T1 T2 pairings was independent of scale scores, we finalized the eventual data set of T1 T2 pairings before the computation of scale scores.

Calculation of the CIIMHS and component scores

Table 2 presents the conceptual framework we used to determine CIIMHS components. Column 1 presents the mental disorders. Column 2 presents the symptoms of depression and psychosis. Column 3 presents the complications of these mental disorders (impairment, isolation, risk of harm to self, and aggression/violence). Column 3 lists the eight dynamic scales derived from the RAI-MH sorted into these categories.

Hirdes and colleagues (41) provide directions for calculating scale scores. All statistical analyses described here were done with SPSS Version 29. To evaluate internal consistency (reliability), Coefficient Alpha was calculated for each of the eight scales. Table 3 presents Coefficient Alpha for each scale, separately for T1 and T2, as well as for the eventual component scores. Additionally, the table includes Alphas for scales previously published by Hirdes et al. (2020) (34).

Table 3. Cronbach's Alpha (N-719) at T1 and T2 for seven RAI-MH Scales and the CIIMHS component scales.

We conducted several Principal Components Analyses (PCA; Factor Analyses) of the eight RAI MH scales. Extraction was based on Eigenvalues greater than 1.00 and Varimax Rotation with Kaiser Normalization. All rotations converged in 5 or fewer iterations. The top panel of Table 4 presents the results of our PCAs. The first three columns show the results from T1, identifying three factors accounting for 68% of the variance. The first factor reflected functional impairment in daily tasks (ADL, IADL) and cognitive (CPS) performance. The second included the scales involving aggression and violence (ABS, VS) together with psychotic symptoms (PSS). The third included scales reflecting depressed mood (DSI, SWS). The second three columns in the top panel present the PCA results from T2. This PCA found three factors, accounting for 71% of the variance. Again, the first factor reflected functional impairment in daily tasks (ADL, IADL) and cognition (CPS). But here, the second factor included scales reflecting depressed mood (DSI, SWS) and psychotic symptoms (PSS). At T2, the third factor included aggression and violence scales (ABS, VS). Therefore, the eight scales’ factor structure was unstable from T1 to T2. Specifically, the scale PSS was associated with violence and aggression at T1 and with depressed mood at T2. The instability of the factor structure from T1 to T2 presents the risk that factors might indicate changes from T1 to T2 when, in actuality, the factors represent a different set of mental health components at T1 versus T2. We recalculated each of these two PCAs, excluding the PSS scale. These results are presented in the lower panel on the right-hand side of Table 4. As can be seen, the factor structure is identical at T1 and T2, with the factors accounting for 74% and 76% of the variance, respectively. At T1 and T2, the first factor reflected functional impairment in performing daily tasks (ADL, IADL) and cognition (CPS). The second factor included the aggression and violence scales (ABS, VS). The third included scales reflecting depressed mood (DSI, SWS).

Table 4. Principal Components Analysis of RAI MH scales separately at T1 and T2.

The distributions of these eight scale scores differed widely in statistical properties (means, standard deviations, and ranges). If we had combined raw scale scores to calculate the composite score, scales with greater ranges would have added greater weight to the composite score. To avoid this, we standardized scale scores to the Z distribution so that each scale had a mean of zero and a standard deviation of 1.00. We made transformations of the standardized scores to make the numbers more “clinician-friendly.” We eliminated the negative numbers by transforming z scores so that each scale ranged from 0.00 to 10.00.

As a result of our data analysis above (PCAs), we considered the following as mental health components. The first would include the scales ADL, IADL, and CPS; the component would be named “impairment.” The second would consist of the ABS and VS scales and be called “aggression.” The third would include DSI and SWS scales, called “depression.” The PSS scale would represent our measure of “Psychosis.”. We calculated component scales as follows: Impairment = (ADL0-10+IADL0-10+ CPS0-10)/3; Aggression = (ABS0-10 +VS00-10)/2; Depression = (DSI0-10 + VS0-10)/2; and Psychosis = PSS0-10. We calculated the CIIMHS as the sum of the 4 component scores and scaled the component measures to have similar ranges to the composite by multiplying each component score by 5. Table 5 presents descriptive statistics for the CIIMHS and its components separately for T1 and T2.

Table 5. Descriptive statistics for the CIIMHS and component scores at T1 and T2 (N=719).

Validity, hypotheses, and statistical analysis

Content Validity evaluates the extent to which a test measures a representative sample of the subject matter under investigation and how well a test covers all relevant aspects of the construct it aims to measure (45). Concurrent validity refers to how closely a new measure relates to a known criterion or gold standard. We hypothesized that our new measure would significantly relate to hospital program assignment (length of stay, complexity of diagnosis). Predictive Validity refers to a measure’s ability to predict the occurrence of a particular outcome. We hypothesized that the CIIMHS would predict hospital discharge. Construct validity concerns how well a measure represents or reflects a concept that is not directly measurable (46). The ultimate purpose of developing outcome measures is to support treatment program evaluation and improvement. Construct validity relates to the outcome measure’s responsiveness to change in health status when it occurs. We hypothesized that our new measure would reflect pre (T1) to post (T2) changes when they occur.

Concurrent validity

Hypothesis 1. Mean program CIIMHS scores will follow the order of PD-only patients (lowest mean scores), acute patients, simple chronic patients, and complex chronic patients (highest mean scores). Between groups ANOVA will be used to test this hypothesis, with planned orthogonal comparisons followed by the Tukey Post Hoc test. Planned orthogonal comparisons included the PD-only group versus all other programs combined, acute programs versus chronic programs, and simple chronic programs versus complex chronic programs.

Hypothesis 2. Discharged patients will have lower mean CIIMHS than current patients, and the CIIMHS will predict discharge from the hospital.

Predictive validity

Hypothesis 3. We evaluate our ability to predict future behavior using a statistical analysis called Receiver Operator Characteristics (ROC) (47, 48). The ROC curve plots the trade-off between true and false positives across a test’s measurement range. An AUC value of.50 indicates a ‘chance’ relationship between predictor and outcome, while an AUC value of 1.00 represents a ‘perfect’ prediction (rare). Good AUC values typically range between.65 and.80, with AUCs above.80 considered large. The AUC represents the probability that the CIIMHS score for a randomly chosen discharged patient is lower than that of a randomly chosen current patient.

Construct validity

Hypothesis 4. The mean CIIMHS will reduce from T1 to T2 throughout the hospital.

Hypothesis 5. Acute programs will show significant reductions in mean CIIMHS over the study interval, but chronic programs will not.

Hypothesis 6. Discharged patients will show a larger reduction in mean CIIMHS than current patients.

Mixed Between-Groups Within-Ss (T1 v T2) ANOVAs will be used to test these hypotheses. The single factor within-subjects ANOVA provides three estimates of variance (Sums of Squares) (1): the difference between the means at T1 and T2 (Effect) (2); differences among patients’ marginal scores (T1 + T2); and (3) error.

SSTotal= SSEffect+ SSPatients+ SSError

The variance estimate for error represents the consistency of change from T1 to T2 across patients. The more similar the change is from T1 to T2 across patients, the smaller the error.

We will make important comparisons between programs’ T1 vs. T2 effect sizes. The Effect Size (ES) (49) is calculated as:

Partial Eta Squared (ηp2) =SSeffect/(SSeffect+ SSerror)

Therefore, two factors determine the effect size. The larger the difference between the means at T1 and T2 and the more consistent the T1 to T2 changes among patients, the larger the ES. This makes the Partial Eta Squared an ideal statistical model for quality of care as defined by the Donabedian model.

ResultsContent validity of the CIIMHS

Table 2 presents the conceptual framework used to evaluate the content validity of the CIIMHS. We were unable to find scales representing the Risk of Self-harm and Substance Abuse that met our selection criteria. The Severity of Self-Harm scale included the PSS, DSI, and CPS in its calculation, and the History of Suicide Attempt (a static item) is a major component in its calculation. The CAGE scale contained a preponderance of static items. Five component scales derived from 8 scales from the RAI-MH quantify the severity or extent of the five remaining putative causes of mental health crises. Accordingly, our analysis of Content Validity finds that the CIIMHS contains 5 of 7 relevant components. As seen in Table 5, concerning the weighting of the five components of the composite measure, with most scales ranging from 0-50, the mean component scores mainly were within 2.00 of the mean CIIMHS, indicating roughly equal weighting.

Concurrent validity

Table 1 orders the hospital programs according to their mean CIIMHS scores.

Hypothesis 1. Hospital Programs differed widely in mean CIIMHS scores, F(9,709) = 52.25, p<.001. The Forensic PD program (no-SMI) had the lowest mean CIIMHS at T1 (2.24), and the Civil Chronic SPMI + Dementia program had the highest mean CIIMHS (12.40). Concerning our planned comparisons, the PD-only program had a significantly lower CIIMHS mean score at T1 than other programs combined, F(1,709) = 67.09, p<.001. Among the remaining programs, Chronic programs had significantly higher CIIMHS mean scores at T1 than Acute programs, F(1,709) = 141.89, p<.001, and Complex Chronic programs had higher CIIMHS mean scores at T1 than Simple Chronic programs, F(1,709) = 93.72, p<.001 The 10th column in Table 1 presents the results of the Tukey Post Hoc Test (Alpha = 0.01). These results were all in line with our predictions.

Hypothesis 2. Using data (N=353) from 4 hospital programs that routinely discharge to the community (for other programs, discharges involved transfers to other hospital programs, either at Waypoint or other hospitals in the province), the mean CIIMHS score (3.53) of discharged patients (N=257) was significantly lower than the mean CIIMHS score (5.82) of current inpatients (N=96), F(1,351) =33.62, p<.001.

Predictive validity

Hypothesis 3. The results of the ROC analysis are presented graphically in Figure 1. CIIMHS at T1 yields an AUC of 0.69 (95% CI = .64-.75). This AUC was statistically significant (p<.001), and its numerical value indicated a good level of prediction accuracy. CIIMHS at T2 yielded an AUC of.87 (95% CI.83-.90), was statistically significant (p<.001), and is regarded as a high level of prediction accuracy. Since each AUC value is outside the other’s CIs, the CIIMHS at T2 is a significantly better predictor than the CIIMHS at T1.

Figure 1. Predictive validity. ROC Curves for the prediction of patient discharge by CIIMHS (N=353). The lower curve is for CIMHS at T1 (AUC=.69; 95% CI = .64-.75) and the upper curve is for CIMHS at T2 (AUC=.87; 95% CI = .83-.90).

Construct validity

On average, the time between T1 and T2 was 55.33 days and ranged between 1 and 180 days. (Mode = 92 days; Median = 68 days). The time between assessments varied among programs, F(13, 1477) = 175.12, p<.001. The Civil Acute SMI program had the shortest mean interval between assessments (13.5 days). The Forensic Acute SMI program and the Voluntary Acute SMI + SUD program had the second shortest mean interval between assessments (58.7 and 69.5 days, respectively). The remaining programs had mean intervals between assessments from 79.8 to 92.9 days. Usually, a researcher would be concerned that shorter intervals between assessments would compromise a program’s apparent effectiveness since shorter intervals provide less time for treatment effects to be realized. However, as will be seen, the programs with shorter intervals between assessments showed the most significant improvement in mental health status.

Table 6 presents mean CIIMHS scores for all hospital programs at T1 and T2. We used hospital programs as the grouping factor in a 2 (T1 vs T2) within by 10 between groups design.

Table 6. Statistical comparisons T1 versus T2 by hospital program.

Hypothesis 4. Over the whole hospital, as predicted, there was a reduction in CIIMHS from T1 (mean = 5.64) to T2 (mean = 4.34), F(1,718) =95.02, p<.001; Partial Eta Squared (ηp2) = 0.117, a medium effect size (50, 51).

Hypothesis 5. The changes from T1 to T2 were not uniform across hospital programs, as the interaction between T1 vs T2 and programs was significant, F(1,709) = 3.15, p<.001. As predicted, Acute programs significantly reduced CIIMHS scores from T1 to T2, but Chronic programs did not.

Table 6 presents separate within-subjects ANOVAs for the individual programs. The Civil Acute SMI+SUD program shows a decline in mean CIIMHS score from T1 (2.78) to T2 (1.14) that was statistically significant, F(1,58) = 20.55, p<.001, with a large effect size (ηp2) = 0.262. The Civil Acute SMI program showed a significant decline in mean CIIMHS scores from T1 (3.44) to T2 (1.54), F (1,240) = 73.54, p<.001, with a large effect size (ηp2) of .235. The Forensic Acute Assessment program showed a decline from T1 (7.29) to T2 (5.52), F(1,139) = 21.81, p<.001, with a medium effect size of (ηp2) = 0.136.

The only significant reduction in the CIIMHS among the Chronic programs occurred in the Forensic PD program, F(1,41) = 5.12, p<.05, with a medium effect size (ηp2) of.112, due to a significant reduction in the Aggression component.

Hypothesis 6. To test our prediction concerning discharged versus current patients, we selected data from patients (N=353) in programs that routinely discharged patients into the community. Within subjects ANOVA, comparing mean CIIMHS scores at T1 and T2 indicated that for patients not discharged from the hospital, the mean CIIMHS score at T2 (5.50) was not different from the mean CIIMHS score at T1 (5.82). For discharged patients, the mean CIIMHS score at T2 (1.66) was significantly reduced from that at T1 (3.53). The reduction in discharged patients was significantly greater than in current patients, F(1,351) = 14.97, p<.001.

Discussion

A Donabedian model conceptualized outcomes for inpatient psychiatry, construing inpatient mental health status as a multidimensional entity with six components, each a precursor and putative contributory cause of emergency admission to a psychiatric hospital. These were depression, psychosis, impairment, aggression/violence, self-harm, and substance use. To compute an outcome measure, we chose eight scales from the interRAI-MH based on their ability to reflect an individual’s level on one of these dimensions and their ability to reflect a change in level when it occurred. Four component measures were computed after averaging correlated measures. Based on the four measures, the CIIMHS was examined for content validity and tested for concurrent, predictive, and construct validity using a large sample of inpatients in a mental health specialty hospital. We found strong evidence for validity in all four spheres.

Regarding content validity, the CIIMHS included four of the six content areas specified above, providing a reasonably representative sample. The content validity of the CIIMHS can be evaluated by comparison with other empirically evaluated outcome measures. The HoNOS (30, 31) has very similar item content to the CIIMS. Of its 12 items, 6 have an obvious equivalent item among the 8 RAI-MH items used in the present study, and 4 pertain to issues not necessarily pertinent to inpatient care for SMI. The remaining two deal with substance abuse and self-harm, which were not included in our measure, as explained above. The basis-32 (52) is a self-report outcome measure to assess mental health treatment outcomes. It contains five of our six primary symptom and functioning domains, including psychosis, depression, addictive behavior, social relations, and daily living and role functioning.

Regarding concurrent validity, we predicted differences among hospital programs in terms of the severity of mental illness and then calculated statistical comparisons. Hospital programs’ mean CIIMHS scores accorded with their predicted mental health status. Acute programs had lower mean scores than chronic programs. Simple Chronic programs had lower mean scores than Complex Chronic programs, and the Personality Disorder program had lower scores than those serving SMI patients. Patients discharged during the quarter had lower scores than patients who remained in the hospital. Regarding predictive validity, CIIMHS scores significantly predicted discharge from the hospital at T1 and T2, with more accurate prediction at T2.

Concerning construct validity, the change in the hospital’s CIIMHS mean score from T1 to T2 was statistically significant overall. Moreover, as predicted, we demonstrated that acute programs were more likely to significantly reduce CIIMHS scores than chronic programs. Patients discharged during the quarter had more significant decreases in mean CIIMHS scores from T1 to T2 than those who remained in the hospital. The only Chronic Program that showed a significant reduction in CIIMHS was the PD Program. This reduction was due to a significant reduction in the Aggression/Violence component. Hospital programs provide continuous interventions to reduce violence, which may explain this outcome.

The three Acute programs showed significant reductions in CIIMHS. These reductions support the validity of the CIIMHS as an outcome measure. The Civil/Voluntary Acute SMI program admits patients from regional ERs and provides evidence-based psychiatric treatment. Staffed by fully qualified psychiatrists and psychiatric nurses, patients receive proven psychotherapeutic medications and supportive care from allied health professionals. The Forensic Acute SMI program admits patients from the courts for assessments of fitness to stand trial or criminal responsibility or for treatment to make fit in preparation for a return to court. The primary treatment is proven antipsychotic medication.

Waypoint designed the Voluntary Acute SMI+SUD program in 2005 as an integrated mental illness and substance abuse treatment based on evidence-based programs for Concurrent Disorders (Dual Diagnosis) described in the empirical literature. The hospital developed new manuals and trained all program clinicians in several empirically supported clinical interventions, including CBT. McKee and colleagues (53) used a quasi-experimental wait-list control design to evaluate the new program. Eighty-six participants who completed the program showed clinically significant improvements in mental health symptoms, acquisition of knowledge and skills, higher self-esteem, and higher satisfaction with the program compared with the wait-list control group. Our findings are consistent with those of McKee et al.

The primary purpose of outcome studies is to evaluate program effectiveness by comparing outcomes over time and between different treatment conditions. Comparing programs in terms of their statistical effects could be done by looking at the reduction in mean CIIMHS from T1 to T2. An alternative is the Effect Size (Partial Eta Squared). In the present study, the three Acute programs’ changes from T1 to T2 are ranked similarly using differences between means or effect sizes: Civil SMI+SUD > Civil SMI > Forensic SMI. However, a similar ranking would not necessarily be obtained in all circumstances since the ES depends additionally on the consistency among patients in their changes from T1 to T2. Notably, the ES allows comparisons among groups assessed with different instruments using different scales. Partial Eta Squared (ηp2) was calculated from 11,044 inferential statistics reflecting treatment effect sizes reported in the Journal of Counseling Psychology from 1970 to 1979 and resulted in a median value of.08 (54).

Chronic SPMI programs did not exhibit statistically significant changes from T1 to T2. The time between T1 and T2 may be an essential variable. In the study reported here, the average time between T1 and T2 for Chronic programs was between 80 and 90 days. Careful examination of Table 4 reveals that the Civil and Forensic SPMI Simple Chronic programs show changes from T1 to T2 that are approaching levels of traditional statistical significance. Evaluating change over a longer interval may improve outcomes for these groups. Tolonon and colleagues (40) described a long-term rehabilitation program for young adults with Severe and Persistent Mental Disorders. Over a treatment period of (median length) 29 months, patients demonstrated reduced depressive symptoms, lowered risk of harm to self and others, and increased ADLs. Similarly, Van Kranenburg and associates (55) described a long-term treatment (over four years) of homeless persons suffering from Severe and Persistent Mental Illness and Substance Use Disorders in which participants demonstrated an increase in Global Psychosocial Functioning, reductions in risk of harm to self and others, reduced symptoms and an increase in skills for daily living. These authors provided effect sizes of 0.19 to 0.33 per annum and 0.42 and 0.73 over two years.

The CIIMHS represents a reasonable assessment of inpatient mental health status. However, the four CIIMHS components will not represent the breadth of relevant mental health status in all clinical venues. The CIIMHS does not include a scale representing anxiety, a frequent component of mental illness. Anxiety is a common experience for inpatients but is a rare cause of hospital admission by itself (56). The absence of a scale representing recovery or other community-relevant issues would be problematic in many settings. Indicators for recovery are dealt with in the interRAI Self-reported Quality of Life Survey for Mental Health and Addictions (57, 58). An outpatient or community setting would require different components in a composite measure.

The CIIMHS provides a standardized assessment to be used consistently in all hospital programs, allowing program managers and the hospital administration to make meaningful program comparisons. However, specialized programs may want to supplement their outcome measures with specialized instruments. For example, inpatient programs treating SUDs will want to include outcome measures relating to substance abuse, and forensic programs will want to have outcome measures relating to criminal behavior (59). Standardized measures do not imply that they reflect a complete picture of outcomes for all programs.

The findings reported here have broad and immediate applicability. As mentioned above, since 2005, according to protocol, the RAI-MH has been administered to psychiatric inpatients in Ontario on admission, discharge, and every three months for long-stay patients or whenever there was a significant change in clinical status. These findings could be replicated easily with data readily available in every psychiatric inpatient unit in Ontario. In addition, as mentioned above, the RAI-MH has been adapted to other mental health care settings, including Community Mental Health (35), Emergency Psychiatry (36), and Child and Youth Mental Health (37). These instruments contain the items our study used to derive similar outcome measures in these settings.

Finally, it’s important to say that while these results support the validity of the measures, they have not demonstrated the effectiveness of any intervention or treatment. Further studies using adequate controls to eliminate potential confounding variables (placebo effect, passage of time, change in environment) are required. The current results validate a measure that could be used in appropriately designed studies examining the effectiveness of inpatient treatment.

One problematic aspect of this approach to outcome assessment is the number of steps and time it takes to compute the CIIMHS and its components. We are developing software to make these calculations less burdensome.

From a research perspective, one drawback of the study is that data were acquired from clinicians’ scoring of the instruments as part of their regular clinical duties. Had the scoring been done as part of a rigorous research protocol, the reliability of these scales might have been better. On the other hand, our data reflect what can reasonably be done to assess the quality of care based on records available in many settings.

In summary, our validation of the CIIMHS was successful. These results have encouraged the implementation of the CIIMHS as a Routine Outcome Monitor (ROM) (60) at Waypoint. This outcome measure has been included with the structure and process QIs on the hospital’s balanced scorecard. These outcomes allow the hospital to evaluate its efforts to improve the quality of patient care over the long term.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by the research described in this manuscript was subject to Waypoint’s Research Ethics Board Approval of an application entitled “The Development of Clinical Outcome Measures based on the interRAI-MH among clinical programs at Waypoint Centre for Mental Health Care Certificate # CRRA#12.03.01. The studies were conducted in accordance with the local legislation and institutional requirements. Written informed consent for participation was not required from the participants or the participants’ legal guardians/next of kin because we were provided with an anonymized data file after most patients had left the hospital. We had no contact with participants as part of this research, and it would have been impossible to locate inpatients months after their discharge.

Author contributions

HB: Conceptualization, Formal Analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Writing – original draft, Writing – review & editing. CP: Conceptualization, Methodology, Writing – review & editing. EH: Data curation, Writing – review & editing, Methodology. GB: Conceptualization, Writing – review & editing. JH: Conceptualization, Investigation, Methodology, Resources, Supervision, Writing – review & editing.

Funding

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Acknowledgments

Author HB used Grammarly for grammar and spell check.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. McGinty EE, Baller J, Azrin ST, Juliano-Bult D, Daumit GL. Quality of medical care for persons with serious mental illness: A comprehensive review. In: Schizophrenia research. New York: Elsevier, vol. 165. (2015). doi: 10.1016/j.schres.2015.04.010

PubMed Abstract | Crossref Full Text | Google Scholar

2. Wang J, Lloyd-Evans B, Giacco D, Forsyth R, Nebo C, Mann F, et al. Social isolation in mental health: a conceptual and methodological review. Soc Psychiatry Psychiatr Epidemiol. (2017) 52:969–77. doi: 10.1007/s00127-017-1446-1

PubMed Abstract | Crossref Full Text | Google Scholar

4. Pancani L, Marinucci M, Aureli N, Riva P. Forced social isolation and mental health: A study on 1,006 italians under COVID-19 lockdown. Front Psychol. (2021) 12:663799. doi: 10.3389/fpsyg.2021.663799

PubMed Abstract | Crossref Full Text | Google Scholar

6. Chen C, Chen Y, Huang Q, Yan S, Zhu J. Self-care ability of patients with severe mental disorders: based on community patients investigation in beijing, China. Front Public Health. (2022) 10:847098. doi: 10.3389/fpubh.2022.847098

PubMed Abstract | Crossref Full Text | Google Scholar

8. Singhal A, Ross J, Seminog O, Hawton K, Goldacre MJ. Risk of self-harm and suicide in people with specific psychiatric and physical disorders: Comparisons between disorders using English national record linkage. J R Soc Med. (2014) 107:194–204. doi: 10.1177/0141076814522033

PubMed Abstract | Crossref Full Text | Google Scholar

10. Swanson JW, McGinty EE, Fazel S, Mays VM. Mental illness and reduction of gun violence and suicide: Bringing epidemiologic research to policy. Ann Epidemiol. (2015) 25:366–76. doi: 10.1016/j.annepidem.2014.03.004

PubMed Abstract | Crossref Full Text | Google Scholar

13. Sparling TM, Cheng B, Deeney M, Santoso MV, Pfeiffer E, Emerson JA, et al. Global mental health and nutr