Participants

This study is a secondary analysis of pooled data with research samples collected from 2008 to 2017 from several projects aimed at identifying ASD biomarkers, randomized control trials of social skills training, and developing an early ASD screening instrument. All the participants were enrolled via patient referrals from child and adolescent psychiatric, pediatric and child rehabilitation departments, and communities such as local clinics and daycare centers, recruitment posters on online/offline bulletin boards of public institutions, and online parenting communities. Participants from the social skills training programs consisted of participants with ASD; participants recruited for identification of ASD biomarkers and development of the early ASD screening instrument included both participants with ASD and without ASD. The examiners were blinded to the diagnostic characteristics of the participants, and clinical best-estimate diagnoses were determined by experienced clinicians, including two licensed child psychiatrists. One institution was in charge of recruiting participants and conducting all evaluations for all projects.

A total of 2158 participants were included in this study (mean age [standard deviation] = 79.7 [64.0] months; age range = 12–393 months; 1588 males; Toddler Module, n = 289; Module 1,Footnote 1n = 642; Module 2 n = 574; Module 3 n = 411; Module 4, n = 242; 1473 participants with ASD, 685 participants without ASD, and 123 participants with OD). Participants with OD consisted of participants who were diagnosed as not having ASD based on clinical best-estimate diagnosis and obtained scores lower than 80 in either the full-scale intelligence quotients (FSIQ) or Korean Vineland Adaptive Behavior Scales, Second Edition (K-VABS; [33]) and therefore were considered as a subgroup of participants without ASD.

We aimed to categorize the OD group to represent individuals with potential intellectual disabilities or developmental delays. Although we were not able to confirm the clinical diagnostic status of the OD group, we wanted to, at least preliminarily, examine if the ADOS-2 can be used to differentiate individuals with ASD from individuals with at least some developmental problems in terms of adaptive skills and intellectual functioning. Diagnostic criteria of intellectual disability include deficits in intellectual and adaptive functionings observed during the developmental period [1], and, therefore, we used the FSIQs and K-VABS scores to identify individuals who may have an intellectual disability. We included participants with IQ scores lower than 80 to include those who have borderline intellectual functioning (i.e., individuals who function on the border between intellectual disability and normal intellectual functioning; [34]). Because the construct of adaptive behavior captures whether an individual has conceptual, social, practical skills expected of their age, development, and culture [35,36,37], we used the VABS score as a proxy for potential developmental delay.Footnote 2

Diagnostic procedures are presented in the Procedures section. Detailed characteristics of the total participants and participants by module are included in Tables 1 and 2. Information on participant characteristics for each developmental cell of the Toddler Module, Module 1, and Module 2 is available in Additional file 1: Table S1. Detailed characteristics of the OD participants are available in Additional file 1: Table S2.

Table 1 Participant characteristicsTable 2 Participant characteristics by moduleProcedures

Participants and their parents completed a battery of tests during their one-time visit, including the K-ADOS or K-ADOS-2, ADI-R, the Korean version of Childhood Autism Rating Scale (K-CARS), Korean Vineland Social Maturity Scale (K-SMS), and cognitive tests measuring FSIQs. Questionnaires, such as the Social Responsiveness Scale-2 (SRS-2), Social Communication Questionnaire (SCQ), and K-VABS, were mailed and filled out prior to the visit. The K-ADOS or K-ADOS-2 and ADI-R were administered by research-reliable professionals or research assistants who worked alongside them in the same laboratory on a daily basis and were trained prior to the actual administration. The scales were administered only after an adequate level of inter-reliability with the research-reliable professionals (> 80%) was reached. All administrations of the K-ADOS or K-ADOS-2 and ADI-R were videotaped and double-checked by these professionals to confirm the quality and reliability.

Subsequently, two board-certified psychiatrists made the best-estimate clinical diagnostic criteria for ASD and non-ASD based on DSM-5 [1]. The clinical best-estimate diagnosis was made according to the information gathered collectively from all tests administered, including the K-ADOS/K-ADOS-2, ADI-R, SCQ, SRS-2, K-CARS, SMS, VABS, IQ assessments, and observed clinical impressions. The study was approved by the Institutional Review Board (IRB) of Seoul National University Bundang Hospital (IRB no. B-2110–716-102).

MeasuresAutism Diagnostic Observation Schedule and Autism Diagnostic Observation Schedule-2 (ADOS and ADOS-2 [4, 5])

This study used the Korean translated versions of the ADOS/ADOS-2, approved by its publisher Western Psychological Services. Data collected prior to July 2017, when the ADOS-2 was published in Korea, were administered using the original K-ADOS. The results from the K-ADOS were rescored based on the K-ADOS-2 algorithm for this study. The modules range from the Toddler Module, for children aged 30 months and younger, to Module 4, for verbally fluent older adolescents and adults. The diagnostic algorithms for the Toddler Module and Modules 1 and 2 are further subdivided into developmental cells based on age/language. The algorithm for the Toddler Module is divided into two developmental cells: 12–20 months/nonverbal 21–30 months toddlers (12–20/NV21–30) and 21–30 months toddlers with some words (21–30SW). The algorithm of Module 1 is divided into two developmental cells based on expressive language level: no words (NW) and some words (SW). The algorithm of Module 2 is divided into two developmental cells based on age groups: < 5 years and ≥ 5 years.

All modules provide two cutoff points in the classification algorithms. For Modules 1 through 4, there is a higher cutoff in the classification algorithms for stringent classification (i.e., autism) and a lower cutoff in the classification algorithms for more inclusive classification (that is, autism spectrum disorder; ASD). For Module 4, we applied the revised algorithm from Hus and Lord [38]. The Toddler Module also has a higher cutoff in the classification algorithms for stringent classification (moderate–severe concern) and a lower cutoff in the classification algorithms for more inclusive classification (mild–moderate concern), which were specified in Esler et al. [39]. Alternatively, Luyster et al. [40] provided the single research cutoff point for the Toddler Module and explained that the single cutoff needs to be applied in the Toddler Module due to the relative lack of diagnostic stability in younger children. In this study, we primarily relied on the results calculated based on the ASD cutoff for Modules 1–4 and the Luyster et al. [40]’s cutoff point for Toddler Module to make the decisions regarding validity. The diagnostic validity of the Toddler Module calculated based on Esler et al. [39]’s cutoff point system is presented in Additional file 1: Table S3.

Autism Diagnostic Interview-Revised (ADI-R [6])

The ADI-R is a semi-structured caregivers’ interview used to diagnose or evaluate the core symptoms of ASD. Each item is scored and converted on a scale of 0, 1, and 2, with higher scores indicating a greater number of and/or clear symptoms of ASD. The ADI-R includes 93 items describing four diagnostic domains: social interaction, communication, RRBs, and abnormality of development evident at or before 36 months. Each domain has a diagnostic criterion, but individuals must exceed all four cutoff scores to be classified as ASD. While the majority of the algorithm score consists of parents’ descriptions of a child’s behaviors between the ages of 4–5 years, some items ask whether the behavior has ever been present during the child's lifetime. For children under 4 years of age, ratings on current behaviors are used. The Korean translation of the ADI-R [25], approved by its publisher Western Psychological Services, was used in this study.

Social Communication Questionnaire [41]

The SCQ is a caregiver-report screening instrument for ASD designed to evaluate an individual’s behavior in three domains: social interaction, language and communication, and RRB. The SCQ includes 40 items to be rated as either “yes” or “no.” It consists of two forms: the Lifetime Form, which focuses on an individual’s developmental history, and the Current Form, which inspects an individual’s behaviors over the past three months. The total score in the Lifetime Form is used to determine if an individual is likely to have ASD, and whether a more extended diagnostic evaluation needs to be undertaken. In this study, we used a cutoff score of 10, for children under 47 months of age, and 12, for children over 48 months, based on a standardization study conducted in Korea [42].

Social Responsiveness Scale-2 (SRS-2 [43])

The SRS-2 is a 65-item parent-report questionnaire that assesses the severity of ASD-related symptoms on a 4-point scale, with higher total scores reflecting more severe ASD symptomatology. It consists of five subscales: social awareness, social cognition, social communication, social motivation, and autistic mannerisms. The SRS-2 has been used extensively in the ASD literature as a diagnostic measure [44] and is reported to have good internal consistency and concurrent, discriminant validity [45]. Chun et al. [46] demonstrated adequate levels of sensitivity and specificity of the Korean translated version of the SRS-2. A cutoff T-score of 65 was applied regardless of gender in the preschool form of the SRS-2, and cutoff T-scores of 70 and 63 were used for female and male participants, respectively, for the school-age and adult forms of the SRS-2 because these values are widely used across clinical settings in South Korea.

Korean version of the Childhood Autism Rating Scale (K-CARS [47])

The CARS [48] is a clinician-rated scale developed to screen for ASD. Consisting of 15 items rating the presence and severity of symptoms associated with ASD, the CARS is scored from 1 (no impairment observed or reported) to 4 (severe impairment). There is no consensus on the cutoff score of the K-CARS; Shin and Kim [49] suggested a cutoff score of 28, while others recommend 24 [50]. Therefore, we utilized both cutoff scores in this study.

Full-Scale Intelligence Quotients (FSIQ)

The following instruments were used to calculate FSIQ in this study: the Wechsler Preschool and Primary Scale of Intelligence (WPPSI) [51] for children aged 2 years and 6 months to 6 years, Wechsler Intelligence Scale for Children (WISC) [52] for children aged 6–16 years, and Wechsler Adult Intelligence Scale (WAIS) [53] for individuals over 16 years of age. These instruments utilize chronological age standardization with a mean of 100 and a standard deviation of 15.

Korean version of the Vineland Adaptive Behavior Scale, second edition (K-VABS [33, 54])

The VABS is a parent or other caregiver’s rating of a person’s adaptive functioning and social self-sufficiency from birth to adulthood. The VABS consists of five domains: communication, daily living skills, socialization, motor skills, and maladaptive behavior. It is scored on a 0–2 rating scale, with a higher score representing skills used more frequently. The five domains together yield a total adaptive behavior composite score. The normative mean of the composite score is 100, with a standard deviation of 15. We used the Korean version of the parent/caregiver rating form of VABS, which was highly correlated with the survey interview form of VABS and showed sufficient validity among Koreans [55].

Korean Vineland Social Maturity Scale (K-SMS [56])

The K-SMS is a clinician-rated instrument that assesses social and adaptive maturity. Originally developed using the Doll’s Vineland Social Maturity Scale [57], the K-SMS includes 89 items grouped by behavioral milestones that are expected at each age. It consists of eight subdomains (communication, general self-help, locomotion, occupation, self-direction, self-help eating, self-help dressing, and socialization skills) and provides a global social age and social quotient.

Nonverbal mental age

Data were collected from multiple studies aiming to fulfill different objectives; the age range of participants recruited for each study and, consequently, the scales used to assess the nonverbal mental age of participants varied across studies. Depending on the type and age range of the studies, we used the Beery-Buktenica Developmental Test of Visual-Motor Integration (VMI) or Leiter International Performance Scale in addition to the nonverbal subscale of WPPSI or WISC. The Beery-Buktenica Developmental Test measures the ability of an individual to integrate their visual perception and motor coordination [58]. The Leiter International Performance Scale assesses nonverbal performance intelligence and cognitive abilities [59]. Many participants were not able to participate in these assessments of nonverbal mental age due to lack of cooperation, and, additionally, some could not participate because they did not meet the minimum age range for participation. For instance, we could not collect the information about the nonverbal mental age of participants in the Toddler Module. However, we present the information on the nonverbal mental age of participants in Module 1, analyzed using the collected data, since Gotham et al. [22] reported that the specificity was low when Module 1 was applied to children with nonverbal mental age lower than 15 months.

We identified the nonverbal mental age of 30 participants in Module 1, calculated based on the WPPSI or WISC scores, and, of these 30 participants, none of the participants in Module 1 had a nonverbal mental age lower than 15 months. We also identified the VMI scores from 74 participants of the participants in Module 1, and the developmental age calculated based on the VMI scores of all 74 participants exceeded 35 months (mean developmental age = 43.4 months, SD = 10.4). Additionally, we identified the Leiter International Performance Scale of 169 participants in Module 1, and five participants with ASD from Module 1 had a nonverbal mental age lower than 15 months. We conducted sensitivity tests of the entire analysis on Module 1, Module 1 SW, and Module 1 NW after eliminating these five participants, and eliminating these participants resulted in very minimal changes in analyses.

Statistical analyses

Initially, we computed a set of independent t tests comparing the age, FSIQ, and scores from K-ADOS-2, ADI-R, K-CARS, SCQ, and SRS-2 of participants with ASD and those without ASD. Calibrated severity scores (CSS; i.e., a severity metric that takes age and language level into account [60] were used to compare the K-ADOS-2 scores.

To address the first aim, the sensitivity, specificity, PPV, NPV, and Cohen’s kappa (k) between ASD and non-ASD were calculated to check for consistency between the best-estimate clinical diagnosis and diagnosis based on ASD cutoff for K-ADOS-2 Modules 1–4 and Luyster et al.’s [40] cutoff point for Toddler Module. This analysis was conducted on all modules combined, each module (including Toddler Module and Modules 1, 2, 3, and 4) individually, and each developmental cell (12–20/NV21–30 and 21–30 SW in Toddler Module, NW and SW in Module 1, and under and over 5 years of age in Module 2). We also computed the area under the receiver operating characteristic (ROC) curve of all items by developmental cell to explore if all items included in the algorithm have sufficient diagnostic accuracy according to the area under the curve (AUC).

To investigate the second aim, we computed Pearson’s r correlation coefficients between the total scores of K-ADOS-2 and those of existing ASD diagnostic instruments (i.e., ADI-R, K-CARS, SCQ, and SRS-2) for all modules combined, each module individually, and each developmental cell. Additionally, k values were calculated between the diagnosis based on the K-ADOS-2 ASD cutoff (and Luyster et al.’s [40] cutoff point for Toddler Module) and the diagnosis based on the existing ASD diagnostic instruments. The k values were interpreted based on McHugh’s [61] criteria (0–0.2, none; 0.21–0.39, minimal; 0.40–0.59, weak; 0.60–0.79, moderate; 0.8–0.9, strong; above 0.9, almost perfect). For the third aim, Cronbach’s α values for the algorithm items and values after an item was removed were computed to examine the internal consistency of each developmental cell.

Finally, we calculated the sensitivity, specificity, PPV, NPV, and k values to examine how accurately the K-ADOS-2 ASD cutoff can distinguish ASD from OD for all modules combined, each module individually, and each developmental cell. We did not compare the diagnostic validity between OD and the remaining participants without ASD (i.e., participants who were not diagnosed with ASD and did not have FSIQ or VABS scores lower than 80) because this sample included a few participants for whom we did not have all FSIQ and VABS scores and therefore would have been categorized as OD if all relevant information was available.

All analyses except for the calculation of Cronbach’s α values were repeated using the Autism cutoff for Modules 1–4 and moderate–severe concern for the Toddler Module. All statistical analyses were performed using Excel and SPSS Statistics (version 23.0; IBM Corp., Armonk, NY, USA).