Classification of DREEP in the context of other app-based sleep assessment technologies

The development and validation of novel technologies for easy and reliable diagnostics or sleep monitoring without impeding patient comfort have become a huge research priority in medicine and the use of home-based sleep testing is increasing. A study by Tiron et al. [21] examined the Firefly app-based sensing technology, which also offers the potential to significantly lower the barrier of entry to OSA screening, as no hardware (other than the user’s personal smartphone) is required. The subtle breathing patterns of a person in bed are measured via a smartphone using the Firefly app technology platform and artificial intelligence (AI) algorithms to identify detailed sleep stages, respiration rate, snoring, and OSA patterns. This technology showed a performance comparable to outpatient OSA screening devices and other smartphone screening apps, with a sensitivity of 88.3% and specificity of 80.0%, with a receiver operating characteristic AUC of 0.92, for a clinical threshold for the AHI of ≥ 15 events/h of detected sleep time [21]. This is comparable to the results shown in our study, although it has to be considered that the applications are based on completely different processes.

To our knowledge, this is the first validation attempt of a web-based application designed to screen for various common chronic sleep disorders at the subjective level. Like other subjective sleep assessment methods, the DREEP app is not intended to make a definitive diagnosis, but rather to assist specialists in their clinical evaluation by providing a high pre-test probability for a specific sleep problem. While there are many validated one-dimensional questionnaires available to sleep medicine to check for the presence of one particular condition, DREEP offers a more holistic approach. Its multidimensional structure is based on a collection of several adapted, shortened, or modified test items derived from four well-established rating scales routinely used for the assessment of sleep-related (movement) disorders. Unlike this clear diagnostic purpose of the DREEP app, the focus of the vast majority of sleep apps available in commercial app stores today is on sleep tracking. Primarily, such tracking apps are supposed to fulfil the purpose of prevention and raise awareness of one’s own sleep, which can possibly entail behavioral changes to achieve improved sleep hygiene. At the same time, the use of such apps may also have a detrimental effect on sleep quality as part of an exaggerated self-optimization. Given the high performance and reliability of some of these apps, it is of course to be expected that this kind of long-term sleep monitoring will occasionally also recognize indications for the presence of a sleep disorder requiring treatment. However, since most of these apps have not yet been validated, their use as a clinical screening tool is still viewed critically [20]. In contrast, other sleep apps have already proven their valuable contribution to sleep health care, but as they were predominantly developed for therapeutic purposes, e.g., as a digital unguided self-help alternative to cognitive behavioral therapy for the treatment of insomnia [21], they can be ignored here.

Another computerized tool for the assessment of sleep disorders is the expert system Sleep-EVAL, which was introduced in 1990 and validated against polysomnography in 1999 [27]. It is a non-monotonic, level‑2 knowledge-based system endowed with a causal reasoning mode designed to provide homogeneous and standardized diagnostic evaluations. Similar to DREEP, it is designed like traditional questionnaires but much more complex. The initial responses are used to form diagnostic hypotheses. Decisional trees are then created for each hypothesis by formulating additional questions that need to be asked to complete the information on that specific diagnostic pathway. Since this is repeated for all hypotheses, the tool allows for differential diagnosis [27]. It also differs from DREEP in the way it is presented to the patient. While DREEP is a browser-based application that can be used with or without the supervision of a sleep specialist, Sleep-EVAL is a structured diagnostic interview that requires some training to conduct. On average, such an interview lasts 40.4 ± 20.0 min [28], which is much longer than the time it takes to complete the browser-based questionnaire DREEP.

Comparison of the study results with other multidimensional subjective sleep assessment methods

For the following discussion, the present preliminary DREEP results are therefore compared with diagnostic testing accuracy measures of other multidimensional subjective sleep assessment methods that, like DREEP, can be completed without any assistance, but are, as far as we know, at best available as a computerized answer sheet lacking all advantages of a mobile app. Table 5 shows some psychometric properties of four comprehensive rating scales for global sleep assessment with respect to the sleep disorder entities insomnia, OSA, and PLMD/RLS. Their discriminative potential is quantified by sensitivity and specificity indices and, where indicated, by AUC.

Table 5 Overview of the clinical accuracy of some other validated multidimensional questionnaires

First, we found that the DREEP algorithm correctly classified a large percentage of individuals who did indeed suffer primarily from problems falling asleep and/or staying asleep as insomnia patients. Furthermore, the high specificity value suggests that most otherwise sleep-disordered individuals were correctly identified as non-insomniacs by the DREEP algorithm. The global measure of diagnostic accuracy, or AUC, was 0.83, which can be considered very good. This means that the DREEP model can distinguish between positive and negative cases of chronic insomnia with a probability of 83%. This finding on the discriminative ability of DREEP related to insomnia is even superior to those of other multidimensional sleep scales. Specifically, the AUC values of the Global Sleep Assessment Questionnaire (GSAQ) and the Holland Sleep Disorders Questionnaire (HSDQ) were only 0.72 and 0.69, respectively (Table 5), indicating that the DREEP algorithm is a promising screening alternative for insomnia. The discrepancy in discriminatory performance between DREEP and the HSDQ may be partly explained by the extent to which other sleep disorders co-occurred in the subgroup of insomnia patients. Kerkhof et al. [26], who developed the HSDQ, reported a large percentage of patients from other diagnostic groups who were comorbid with insomnia, causing considerable symptom overlap, with the result that many insomnia patients also showed a positive test result on other subscales. In the present initial validation study of the DREEP app, however, the comorbidities of the insomnia patients were limited only to sleep-related breathing disorders (Table 1), which should have contributed to an overall lower symptom overlap and thus have had less influence on test performance.

Second, the DREEP app distinguishes between individuals with and without OSA with a probability of 85%, which is also considered very good. The corresponding AUCs obtained for the GSAQ and the HSDQ were slightly higher but still in the same diagnostic accuracy category, indicating that the discriminative performances were quite similar between these approaches (Table 5). Sensitivity indices of the DREEP app determined for the total OSA group were also comparable with those of the other four self-assessment instruments (Table 5). A comparison of diagnostic accuracy by OSA severity, on the other hand, was not possible here because this distinction was not made in any analysis of the other four questionnaires. However, this aspect is very important in clinical sleep medicine, because incorrect severity classification, i.e., persons with severe OSA erroneously classified as mild or vice versa, can be associated with unnecessary effort and costs, e.g., due to performing an additional PSG, which should be strictly avoided in view of the current resource scarcity in sleep medicine. Overall, the present preliminary validation data show that DREEP is suitable to detect individuals with OSA in the clinical setting.

Third, the DREEP algorithm discriminates between positive and negative cases of PLMD/RLS with a probability of 80%, which is again slightly lower than the AUC values of the GSAQ and the HSDQ (Table 5), but still very good. In addition, the DREEP app correctly identified around 79% of all individuals with clinically confirmed PLMD/RLS. The specificity value of 0.80 indicates that DREEP correctly classified otherwise sleep-disordered individuals as “negative” for PLMD/RLS to a sufficient extent for clinical purposes. These two indices were highest for patients with moderate and lowest for patients with mild PLMD/RLS, suggesting that DREEP underestimates the presence of mild cases. Compared to the other four questionnaires, the test attributes of the DREEP app are mid-range and most similar to those of the HDSQ and the SLEEP-50. Both indices are much better than those found in the Sleep Disorders Questionnaire, and the specificity is also higher than that in the GSAQ. In terms of sensitivity, however, DREEP performs worse than the GSAQ (Table 5).

Finally, a previously unknown case of narcolepsy was correctly diagnosed by DREEP during the study. However, with a total of one affected patient in the present study, it was impossible to come to any conclusions about the diagnostic abilities of DREEP with regard to the prediction of narcolepsy.

Nevertheless, based on the test performances for the other sleep disorders, we assume that the presented web application is sufficient to make an initial assessment of whether insomnia, sleep-disordered breathing, or a sleep-related movement disorder is present or not.

Strengths and limitations of the DREEP web application

DREEP provides several benefits for future application because it is easy to use, inexpensive, does not require a smartphone or similar, and works paperless and is therefore sustainable. In addition, it is faster than separately completing the various questionnaires used in DREEP, or other computerized tools such as the Sleep-EVAL system [28].

Besides these strengths, we have to face some limitations. Children and adolescents were excluded from the study, and therefore the conclusions of the present initial validation study apply to adults only. Thus, DREEP is not currently suitable for use in children and adolescents with disturbed sleep. In addition, only subjects who experienced non-restorative sleep on at least three days a week for more than one month were recruited. Therefore, the use of DREEP is limited to individuals with common chronic sleep disorders and is not appropriate for use in those with acute or situational sleep disorders. Although DREEP can detect several rare chronic sleep disorders, the current study provides no validation data for this. One individual with previously undetected narcolepsy was correctly identified by the algorithm, but this observation does not allow any statements to be made about the diagnostic ability of DREEP in adults with narcolepsy. Further, although the diagnosis suggested by DREEP was often correct in relation to common chronic sleep disorders, the application provides this diagnosis only with some degree of certainty and not with a final clinical diagnosis. Therefore, it is not suitable for providing an unequivocal diagnosis or for reliably excluding the presence of a specific sleep disorder as the cause of chronically disturbed sleep. Finally, the browser-based application DREEP needs to be further investigated. In a follow-up study, a healthy control group should also be analyzed to be able to improve the clinical accuracy of DREEP. Ideally, this investigation should then not only take place in a single tertiary care center, but in a multicenter study.