Purpose

To develop and psychometrically evaluate a brief measure of the telemental health experience among people receiving psychiatric and psychological care—the Telepsych User Experience Scale (TUES).

Methods

The TUES was administered at 6 months to 364 study participants who screened positive for posttraumatic stress disorder and/or bipolar disorder and used telepsych services. The factor structure of the TUES was examined using exploratory and confirmatory factor analyses. Psychometric testing of the final scale examined (1) internal reliability, (2) criterion validity, (3) convergent validity, and (4) test-retest reliability using omega coefficients, negative binomial regression, and correlations, respectively. A week after the 12-month follow-up survey was completed, a retest was administered to 29 patients to assess reliability.

Findings

Factor analytic methods indicated a single latent factor (user experience) and correlated error variance (method effect of item wording) for 2 items. To enhance clinical utility, we removed the 2 negatively worded items, resulting in a 5-item scale. Confirmatory analyses indicated excellent fit of the final model, which retained the best performing items from each hypothesized construct. The TUES demonstrated evidence of internal consistency (omega = 0.88-0.90), convergent validity, (r = 0.58), and criterion validity through telepsych engagement (incidence rate ratio = 1.19, P < .001), though test-retest reliability was unacceptable (r = 0.41).

Conclusion

The TUES is a pragmatic instrument with evidence of validity and internal reliability. Replication is necessary, but this initial psychometric evaluation suggests the TUES is a promising, brief yet comprehensive measure of telemental health user experience with clinical populations in rural settings.