Background

Most children with mental health problem who receive an empirically supported treatment get significantly better and do so more rapidly than with other treatments or no treatment (1). A crucial step in arriving at an empirically supported treatment is clinical assessment, which leads to a correct initial diagnosis. The assessment needs to focus on symptoms, the functioning of the child and the family, and common risk factors for mental health problems. A standardized intake instrument facilitates the triaging between different services of mental health and community services (2); in addition, it makes the clinical assessment more efficient (3). A standardized intake instrument may also serve as a baseline for evaluation of treatment outcomes. The utility of an intake instrument depends not only on its ability to identify psychiatric disorders and relevant risk factors but also on how the instrument is perceived by clinicians and patients, which is crucial for implementation (4). Studies have shown that a too burdensome procedure risks increasing attrition causing decreased response rate (5). Despite the advantages of adopting a standardized intake instrument, many outpatient child and adolescent mental health services (CAMHS) still base their intake decisions on unstructured interviews (6). The Brief Child and Family Phone Interview (BCFPI) (6, 7) is increasingly used for screening owing to its brevity and format friendliness to both family and clinician.

The BCFPI is a semi-structured, computer-assisted clinical intake and follow-up phone interview. It was developed with the goal of improving intake screening, treatment planning as well as being an instrument for outcome measurement. The broad approach targeting symptoms, child and family functioning and risk factors makes BCFPI suitable for triaging between different levels of mental healthcare and community services (6). The six subscales measuring symptoms correspond to the diagnostic categories of attention-deficit hyperactivity disorder (ADHD), oppositional defiant disorder (ODD), conduct disorder (CD), separation anxiety disorder (SAD), generalized anxiety disorder (GAD), and major depressive disorder (MDD). The BCFPI has been found useful and is appreciated by both parents and clinicians (5). Its approach makes BCFPI valuable for evaluation of treatment, and it has been used in numerous studies (8-11).

Overall, the BCFPI has shown good reliability and validity (6, 7, 12, 13). With the exception of the CD subscale (α = 0.68), reliability coefficients for subscales in field trials ranged from 0.75 to 0.85. The BCFPI has shown good test retest reliability and sensitivity to change (7). Cunningham and colleagues conducted a confirmatory factor analysis and measuring invariance across age and sex in a clinical sample, a community sample, and an implementation sample, including altogether 56,825 children. The item structure of the six symptom subscales was supported and showed good internal consistency in all three samples (6).

In a study by Boyle et al. (12), the concurrent validity of BCFPI was examined by comparing it with the Diagnostic Interview Schedule for Children, version IV (DISC-IV), administered to parents. The sample consisted of children and adolescents referred to outpatients CAMHS (n = 399). BCFPI was administered at baseline, after two months and after 13 months and showed a test-retest reliability that exceeded 0.50 for all disorders except MDD. The concurrent validity of the BCFPI scales compared with counterparts in the DISC-IV was somewhat better for externalizing than for internalizing disorders. Kappa estimates ranged from 0.40 to 0.49 for ADHD, ODD, and CD and from 0.28 to 0.37 for SAD, GAD, and MDD. Area under the curve (AUC) was overall good (ADHD = 0.81, ODD = 0.81, CD = 0.86, SAD = 0.83, GAD=0.76, and MDD=0.75). The BCFPI classified mental health problems reasonably well compared with the more extensive DISC-IV interview. However, the DISC-IV diagnoses were elicited by an interview with the same parent on the same day as the BCFPI, thus just adding more questions to the same diagnostic categories.

In a more recent study (13), the BCFPI was compared with Conner’s Rating Scales. High-risk and highly symptomatic inpatients were included (n = 227). The symptom subscales of BCFPI were correlated with selected measures of Conner’s’ Rating Scales. The correlations were moderate to strong supporting the convergent validity of the BCFPI.

A problem that occurs when validating an instrument in children and adolescent mental health is that there is no obvious gold standard psychiatric diagnosis. However, Spitzer (14) proposed the Longitudinal Expert All Data (LEAD) procedure as the best proxy to a gold standard psychiatric diagnosis. According to Spitzer LEAD diagnoses should be based on all available information about the patient from more than one point in time and arrived at in consensus with the experts in the field. The LEAD procedure should include a structured interview and observational time for further evaluation and outcome (15). The validity of the BCFPI has been supported in studies comparing it with other measures based on parent reports (12, 13), but it has yet to be tested against and compared with diagnoses based on a more comprehensive LEAD procedure including a full diagnostic workup.

Method Aims

The aim was to investigate the criterion validity of the BCFPI against LEAD diagnoses in child psychiatric outpatients. In addition, the Child Behavior Checklist (CBCL) (16, 17), the well-established and validated parent report form, will be used as a comparator.

Participants

The participants were recruited from four Swedish CAMHS and selected from all new referrals from the time period between December 2009 and January 2013. Exclusion criterion was inability to participate in the BCFPI. Altogether 5908 parents were routinely interviewed with BCFPI and, at the end, asked to participate in a semi-structured research interview. The 307 patients selected for further study were interviewed with the 2009 version of the K-SADS-PL (Kiddie Schedule for Affective Disorders and Schizophrenia for School-age Children, Present and Lifetime Version) and completed the CBCL within six weeks of the BCFPI. Forty cases were discarded owing to protocol violations or failure of the diagnostician to report data, leaving in total 267 patients. The forty discarded cases consisted of slightly more boys than girls and somewhat more children from the younger age group.

The children were six to 17 years of age, and a slightly larger number of boys were included. None of the children had Children’s Global Assessment Scale scores in the non-clinical range of 70 or above (18). There were slightly more children than adolescents in the sample, but the boys to girls ratio was in line with the large consecutive sample of 5908 new referrals. The patients included in the study had more externalizing and internalizing symptoms as assessed by the BCFPI compared with the large sample (Table 1).

TABLE 1.

Demographic characteristics and symptom load in the study sample compared with all admitted patients

Instruments Brief Child and Family Phone Interview

The BCFPI originated with the Ontario Child Health study completed in 1989 (19). It is a semi-structured, computer-assisted clinical intake and follow-up phone interview (6). The parent interview begins with an open question on the major concern about the child. This is followed by 36 structured questions regarding common symptoms of child mental health problems. These questions are divided into six subscales that correspond to the diagnostic categories (Cronbach’s α in parentheses), ADHD (α = 0.76), ODD (α = 0.85), CD (α = 0.48), SAD (α = 0.79), GAD (α = 0.72), and MDD (α = 0.86). The three subscales for ADHD, ODD, and CD are merged to the scale ‘externalizing’ (α = 0.84) and the three subscales for SAD, GAD, and MDD are merged to the scale ‘internalizing’ (α = 0.84). Each question has three different response options: never 0); sometimes 1); and often 2). The subscale for ADHD is divided into two parts. The first part contains three questions on attentional problems and is referred to as an ADD subscale. The other part contains three questions on hyperactivity. The results are presented as a T-score for each subscale, making it easy to compare the child with the population norms. The BCFPI has standard norms for the four groups, boys six to 12 years, boys 13 to 17 years, girls six to 12 years, and girls 13 to 17 years. In addition to questions about symptoms, BCFPI also focuses on the child’s overall functioning, family functioning, family distress, parental depression, neglect, and abuse. The BCFPI takes approximately 30 to 45 minutes to perform and should be administered by a clinician trained in the procedure (6, 7).

Child Behavior Checklist

The CBCL is a widespread scale in both child psychiatric research and clinical settings. A number of studies have shown its good psychometric properties 16). The CBCL includes 118 questions about behavioral and emotional symptoms. Each question has three response options: not true (0), somewhat or sometimes true 1, and very true or often true 2. The questions are coded into empirically based symptom scales or Diagnostic and Statistical Manual of Mental Disorders (DSM)-oriented subscales. The DSM-oriented subscales have been developed by experts’ identifying questions in the CBCL that are consistent with diagnoses according to the DSM-IV (17). The five DSM-oriented subscales that are used in this study are depression, anxiety, ADHD, ODD, and CD.

Kiddie Schedule for Affective Disorders and Schizophrenia for School-age Children, Present and Lifetime Version

The K-SADS-PL was a component of every LEAD procedure. It is a comprehensive semi-structured diagnostic interview, widely used in both clinical and research settings. K-SADS-PL has shown good psychometric properties (20, 21). It consists of three sections: 1) introductory interview; 2) screen interview; and 3) eight optional diagnostic supplements.

The introductory interview begins with an open question about the major complaint, and then the interviewer seeks information on the child’s functional level, demographics, and the family history of mental health problems.

The screening interview consists of 105 symptom probes representing 23 different diagnostic domains. If any probe reaches threshold, the clinician is urged to use the indicated supplement.

The supplements cover all aspects for arriving at the proper diagnoses arrived from the DSM-IV. There are eight different supplements: 1) affective disorders; 2) psychotic disorders; 3) anxiety disorders; 4) behavioral disorders; 5) substance use disorders; 6) eating disorders; 7) tic disorders and 8) autism spectrum disorders (20, 21).

Procedures

Parent(s) of every new referral was interviewed with BCFPI in the standard intake procedure. At the end of the interview, the parent was informed about the study and asked to give oral consent. The K-SADS-PL was performed within six weeks of the BCFPI. The selection was made consecutively, but a few adjustments were made by the local monitor at the late stage of the study to arrive at roughly equal numbers of children and adolescents and of boys and girls in each age group. Children were at that stage actively selected to include more girls from the younger age group.

Residents or specialists in child and adolescent psychiatry, who had gone through a comprehensive training and had passed a course examination, conducted the K-SADS-PL. Parents and patients were interviewed separately for up to three hours each. If uncertainty about diagnoses arose, the resident or specialist was to make contact with a senior clinician for guidance. They were also allowed, if necessary, to contact the patient’s school for more diagnostic information. Parents filled out the CBCL on the same day as the K-SADS-PL. Clinicians were blind to the BCFPI and to the CBCL.

Longitudinal Expert All Data

A senior clinician (HJ) performed all LEAD diagnoses. They were based on full information from the K-SADS-PL interview and all available medical records, which could include reports from teachers, psychological assessments and the outcome of treatment and reassessment of senior clinicians. The information was gathered at least 6 months after the K-SADS-PL (mean = 1.2 range = 0.1 to 3.1 years). No information from the BCFPI or the CBCL was available for the LEAD diagnoses. All cases that changed diagnosis from the K-SADS-PL or contained inconclusive information were reviewed by the third author (TI) to arrive at a consensus LEAD diagnosis. A blinded reliability test of LEAD diagnoses in 30 cases (HJ and TI) showed excellent interrater reliability with kappa values in the 0.9 to 1.0 range. More details can be found in a sister paper validating the KSADS-PL (22).

Statistical Analyses

Spectrum diagnoses like ADHD were collapsed into one variable named ADHD that included LEAD diagnosis for ADHD inattentive, hyperactive, combined and not otherwise specified type. Likewise major depression, dysthymia and depressive disorders not otherwise specified were collapsed into one variable named MDD. Cases of behavioral disorder not otherwise specified were coded into the ODD variable. The BCFPI subscale INTERNALIZING was compared with LEAD diagnoses of any depressive disorder plus any anxiety disorder except specific phobia. The BCFPI subscale EXTERNALIZING was compared with LEAD diagnoses of any ADHD disorder plus any behavioral disorder. The BCFPI ADD subscale, consisting of only three questions of inattentive problems, was compared with any ADHD disorder. The primary outcome measure was Receiver Operating Characteristics (ROC). Cutoffs for interpreting the ROC “Area Under Curve” (AUC) values were as follows: 0.60 to 0.70 (poor), 0.70 to 0.80 (fair), 0.80 to 0.90 (good), 0.90 to 1.00 (excellent). Agreements between LEAD diagnoses and the subscales of the BCFPI and the CBCL were analyzed with Kappa statistics. Landis and Koch (23) have suggested the following cutoffs for interpreting the Kappa values as follows: 0 to 0.20 (poor agreement), 0.21 to 0.40 (fair agreement), 0.41 to 0.60 (moderate agreement), 0.61 to 0.80 (good agreement), and 0.81 to 1.00 (very good agreement). Sensitivity measures the proportion of cases with a LEAD diagnosis that are correctly identified with the BCFPI or the CBCL. Specificity measures the proportion of cases without the LEAD diagnosis that are correctly identified with BCFPI or CBCL. Positive predictive value is the proportion of patients the BCFPI or the CBCL identifies as having a diagnosis that are correctly classified (i.e., are true positives), whereas negative predictive value is the proportion of patients that the BCFPI or the CBCL identifies as not having a diagnosis that are correctly classified (i.e., are true negatives).

Results

Table 2 presents the statistics for BCFPI and CBCL using the LEAD diagnoses as criterion. The estimated kappa values used a cutoff point with the same prevalence for the BCFPI and the CBCL as in the LEAD diagnoses. The first column used the raw values as cutoff for a binary diagnosis of each disorder. AUC estimates were high (0.80 to 0.90) for the subscales ADD, ADHD, SAD, and CD. Fair AUC (0.70 to 0.80) were found for all the other measures. Kappa estimates for the BCFPI were highest for subscales ADD, ADHD, and SAD, with moderate agreement (0.41 to 0.60) to LEAD. The subscales MDD, EXT, INT, EXT, and CD were somewhat lower, but still with fair agreement (0.41 to 0.60) to LEAD. Kappa estimates for subscales GAD and CD were in the poor range (0 to 0.20).

TABLE 2.

BCFPI mean values (SD in parentheses) divided by sex and

The diagnostic sensitivity (Table 3) of the BCFPI varied between subscales; ADD and ADHD had the highest level of sensitivity, whereas the sensitivity for the subscales GAD and CD was very low. Sensitivity was higher for externalizing than for internalizing problems. The figures for specificity were inversed of sensitivity. The subscales ADD, ADHD, and ODD had the lowest specificity, whereas the subscales SAD, GAD, and CD had very high specificity. The subscales GAD and CD had a very low prevalence in this sample, whereas the prevalence for ADD, ADHD, and ODD was very high.

TABLE 3.

Screening properties of the BCFPI and the CBCL versus a LEAD diagnosis

The subscales ADD and ADHD had high positive predictive value, whereas the subscales SAD, GAD, and CD showed very high negative predictive value. This suggests that a BCFPI screen positive ADHD (at the chosen cutoff score) is relatively reliable, that is, there is a high probability that a child will be correctly identified with these disorders, whereas for a child with a BCFPI screen negative for SAD, GAD, or CD, the probability is low that the child will be diagnosed with these disorders.

The last two columns of Table 3 show AUC and kappa estimates for the CBCL. The AUC values of the CBCL were on a similar level as the BCFPI, but the CBCL performed better regarding the diagnoses of CD and ODD. The BCFPI was better at identifying SAD. Kappa values for ADHD were similar and moderate. CBCL was better at identifying ODD and CD, whereas BCFPI had a better kappa value for SAD. Both BCFPI and CBCL had poor kappa values for GAD.

Table 4 presents AUC for BCFPI versus LEAD diagnoses stratified by sex and age (6 to 12 or 13 to 17 years). There were no significant differences across sex or age groups.

TABLE 4.

Screening properties for BCFPI versus LEAD divided into sex and age

Discussion

The BCFPI performed well as a screening tool for the major child psychiatric diagnoses with overall good criterion validity according to a LEAD diagnosis for the four symptom subscales measuring ADHD, ODD, SAD, and MDD. The BCFPI did equally well across sex and age groups and performed about as well as the comprehensive CBCL. The results support earlier findings (12, 13) that BCFPI is a valid instrument that can be used in CAMHS. However, the results were less convincing for the symptom subscales CD and GAD. The low prevalence of CD and GAD could partly explain this as it is challenging to reliably screen for disorders of low prevalence (24).

In line with earlier studies (12, 13), ADHD could be reliably identified with both the BCFPI and the CBCL. The three questions in the BCFPI on attentional problems were equally good at identifying ADHD as the full ADHD subscale that also included three questions on hyperactivity. Our data suggest that clinical assessment regarding ADHD should be considered for children with elevated scores on the three-item ADD subscale even if the scores for the full ADHD subscale are low.

The BCFPI performed fairly in identifying ODD and behavioral disorders not otherwise specified, but often identified ODD in adolescents better than in younger children. Most children 12 years and younger diagnosed with behavioral disorders also had ADHD. This suggests that the BCFPI might be less accurate in identifying behavioral disorders comorbid with ADHD among children. The CBCL was better at identifying ODD but the results from BCFPI still were fair. In this clinical sample with only a 5% prevalence of CD, the BCFPI had a good AUC but a poor kappa value. The lower discriminatory abilities of BCFPI in CD compared with the study by Boyle et al. (12) may partly be owing to the limited number of children, only 13, with CD in the present study. However, the more comprehensive CBCL with 13 questions concerning CD performed better than the BCFPI with six questions on CD.

Screening for anxiety disorders with parent reports is known to be difficult as the children are the best informants on subjective aspects of the disorders (25). However, the BCFPI performed very well for SAD. Using parent reports when screening for SAD though has the advantage that SAD more often, compared with other anxiety disorders, contains behavioral symptoms like clinging to a parent and fear of sleeping alone. In contrast to SAD, the BCFPI was only fair at identifying children with GAD, which was expected, as symptoms of GAD are more covert than symptoms of SAD and therefore more difficult for parents to identify. Furthermore, symptoms of generalized anxiety in the screening tools will not be diagnosed as GAD in the LEAD if they are better explained by other disorders like depression or autism (26). The lower agreement between the CBCL and the LEAD diagnoses could be understood as a result of the CBCL anxiety subscale measuring anxiety disorders in general and not SAD and GAD specifically.

Depressive disorders were to a fair degree identified with both the BCFPI and the CBCL. Both the BCFPI and the CBCL are parent reports. Research suggests that parent reports and child reports often differ and that parents report fewer symptoms (27). In clinical practice, when the open question or the overall pattern of the BCFPI raises suspicion of significant depressive symptoms that are not reported by the parent, the BCFPI youth scale could be added.

Strengths and limitations

The main strength of the study is that the BCFPI is compared with proxy gold standard diagnoses with a LEAD procedure based on both K-SADS interviews, a wealth of clinical information from a year of treatment and an expert consensus with high interrater agreement. Furthermore, all patients were new referrals; thus, they had not previously been psychiatrically assessed or been involved in psychiatric care, either of which could have biased the BCFPI responses.

The study has some limitations concerning the sample representability in relation to the total population of children contacting CAP at the time. Although the study sample was representative in sex and age, the study sample comprised patients with a larger number of symptoms than the average patient. This could improve the ability of the screening methods, as well-defined cases might more easily be identified with a screening method. On the contrary, patients with more comorbidity might be more difficult to correctly screen. Diagnoses within the same spectrum were collapsed into one category before data analysis, which may inflate the agreement of measures, as diagnoses not otherwise specified need a smaller number of symptoms than the full DSM syndrome. Emergency referrals were not included in the regular intake service at most clinics, which probably lower the number of children with bipolar and psychotic disorders. The low prevalence of GAD and CD makes results regarding those subscales tentative.

Clinical significance

The BCFPI is a concise and effective parent interview, identifying most major child and adolescent psychiatric disorders. The criterion validity is good and is similar to the more comprehensive CBCL. The BCFPI is well suited as an intake instrument in clinical services to support triaging and diagnostic procedures.

Acknowledgments

The authors would like to thank all parents and children and clinicians who agreed to participate in this study.

Funding

The study was funded by The Söderström-Königska Foundation (2008-22893), Development and Education (FOUU), Region Halland, Halmstad, Sweden (110361), and FOU grants from Region Skåne, Sweden (133821).

Availability of data and materials

The data set used and analyzed during the current study is available from the corresponding author on reasonable request.

Authors’ contribution

All authors contributed to the design of the study and in writing the manuscript. HJ, TI, MR, and MA coordinated and supervised the study. MA and MB drafted the first version of the manuscript and conducted the statistical analysis. HJ and MA coordinated and supervised data collection. All authors substantially contributed toward and have approved the final manuscript.

Conflict of interest

Drs. Andersson and Jarbin report grants from Söderström-Königska Foundation and from Development and Education (FoUU) Region Halland and Skåne, Sweden. Dr. Markus Andersson is supported from the BCFPI Inc. for being the national coordinator of the BCFPI in Sweden. The other authors declare no conflict of interest.

Ethics approval and consent to participate

The Ethical Review Board at Lund University approved the study. Patients aged 15 years and above and all parents with custody were informed and consented in participating in the study.