The overall sample of this study comprised 1,879 children (aged 6 to 12 years), mostly with symptoms or diagnoses of externalizing behavior disorders. Data were collected within the scope of five studies on the treatment of externalizing behaviors, which were coordinated by the Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy at the University Hospital of Cologne, Germany: (1) ADOPT study—Affective Dysregulation—Optimizing Prevention and Treatment [26], (2) Self-Help Comparison Study—Behavioral Versus Nonbehavioral Guided Self-Help for Parents of Children with Externalizing Disorders [27], (3) Enhancement Study—Telephone-Assisted Self-Help for Parents of Children with Attention-Deficit/Hyperactivity Disorder who have Residual Functional Impairment despite Methylphenidate Treatment [28], (4) ESCAschool Study—Evidence-Based Stepped Care of ADHD [29], and (5) WASH Study—Efficacy of Web-Assisted Self-Help for Parents of Children with ADHD [30]. From the research consortium ADOPT, three subprojects provided data for this study: ADOPT Online, ADOPT Treatment, and ADOPT Institution. ESCAschool is a sub-study within the research consortium ESCA-life. Across all subprojects, caregiver ratings of school-age children (6.00 to 12.11) were included.

The majority of the participants had a suspected or confirmed diagnosis of attention-deficit/hyperactivity disorder (ADHD), oppositional defiant disorder (ODD), hyperkinetic disorder (HKD), and/or current symptoms of ADHD/ODD/HKD or affective dysregulation (AD). Across the studies, inclusion criteria were no severe cognitive impairment in the children, and willingness for study participation of a parent. In the ADOPT and ESCA study, children's informed consent was also obtained. Exclusion criteria included severe intellectual disability (e.g., IQ < 80), pervasive developmental disorder, another primary disorder (e.g., autism spectrum disorder), or severe psychiatric conditions such as schizophrenia, bipolar disorder, or major depressive episode. Children undergoing current or planned behavioral therapy or indication for inpatient treatment. The samples were recruited Germany-wide through facilities of the local health care systems, residents’ registration offices, and out-of-home care institutions.

Data of 1,879 parents of children aged between 6.00 and 12.11 years were included (nADOPT = 695; nSelf-help comparison = 104; nEnhancement = 114; nESCA = 559; nWASH = 407). Children who were still attending kindergarten were excluded from the data set of the self-help comparison study (n = 45).

The Assessment Scale of Positive and Negative Parenting Behavior (FPNE—Fragebogen zum positiven und negativen Erziehungsverhalten) was constructed based on items of established parenting scales and self-constructed items, which were selected by a focus group of experts in behavioral child and adolescent psychotherapy. We used all 13 items of the German Questions on Parenting (FZEV; 17), which assesses parenting behavior on a four-point rating scale (0 = never to 3 = very often), with higher scores indicating more positive reported parenting behavior. The FZEV is a German adaptation of the Parent Practices Scale PPS; [17, 18], and was developed and used to evaluate a Triple-P parenting training intervention and a prevention program for externalizing problem behavior. Internal consistencies of the total scale ranged from α = 0.84 to α = 0.87 [30, 31]; α = 0.86 in the present study. Additionally, 13 items were selected from the Management of Children’s Behavior Scale– Revised MCBS, [23], which were partly modified during the translation process into German. The original version of the MCBS consists of 37 items rated on a three-point Likert scale (1 = not like me, 2 = somewhat like me, and 3 = like me), with higher scores indicating stronger identification with the stated parenting behavior. The scale covers coercive communication, acknowledgment of good behavior, physical punishment, harsh punishment, inconsistent parental control, and negative reinforcement of deviant behavior. From each category, we selected one to five items which demonstrated a part-whole-corrected item-total correlation of rit ≥ 0.30 in the analysis by Perepletchikova and Kazdin [23] and which lent themselves well to translation into German. The original 37-item version of the MCBS has shown an internal consistency of α = 0.84 [23]. Furthermore, a focus group of therapy experts created sixteen additional self-constructed items relating to dysfunctional parenting (example: “Whether or not I impose a punishment often depends on my mood.”). As the response format for the FPNE, we chose the four-point rating scale (1 = never to 4 = very often) used in the FZEV. For the items adopted from the MCBS scale and the self-constructed items, the scale was adjusted accordingly. A detailed overview of the items and their origin can be found in Table S1 of the supplement.

The following instruments were used to assess the validity of the newly constructed scale: The Symptom Checklists for Attention-Deficit/Hyperactivity Disorder (FBB-ADHS; German: “Fremdbeurteilungsbogen für Aufmerksamkeitsdefizit-/Hyperaktivitätsstörungen”, 20 items, [32]) and for Oppositional Defiant Disorder and Conduct Disorder (FBB-SSV; German: “Fremdbeurteilungsbogen für Störungen des Sozialverhaltens”, 24 items, [32]) assess ADHD and ODD/conduct disorder symptoms according to the DSM-5 and ICD-10. The ADHD total scale and the ODD scale were used in the present study. Both scales have shown satisfactory internal consistency and factorial validity (ADHD Total, 20 items, α = 0.94; ODD Total, 8 items, α = 0.87; [33]). Internal consistencies in the present study were α = 0.95 (ADHD Total) and α = 0.81 (ODD Total). All items were rated on a four-point Likert-type scale ranging from 0 to 3 (0 = not at all to 3 = very much), with higher scores indicating higher symptom severity. Both instruments are part of the Diagnostic System for Mental Disorders in Childhood and Adolescence DISYPS-III; [32]. For studies conducted with the previous version of the DISYPS– the DISYPS-II [34]– the data were combined.

From the German translation of the Child Behavior Checklist for Ages 6–18 CBCL 6-18R, 113 items, [35], we used the symptom scales to assess externalizing symptoms (CBCL External, 35 items) and internalizing symptoms (CBCL Internal, 32 items). Each item was rated on three-point Likert scale (0 = not true to 2 = true). Scale scores were calculated by summing the individual item scores, with higher scores indicating more severe behavior problems. Both second-order scales have shown satisfactory validity and internal consistencies above 0.80 (CBCL External α = 0.93; CBCL Internal α = 0.86; 34). In the present study, internal consistencies were α = 0.90 (CBCL External) and α = 0.84 (CBCL Internal).

The Depression Anxiety Stress Scales (DASS-21; 36; German: Depressions-Angst-Stress-Skalen, 21 items) measure symptoms of depression, anxiety, and stress referring to the previous week. We employed a German adaptation of the 42-item version by Lovibond and Lovibond [36]. The items were rated on a four-point Likert scale (0 = never to 3 = very often), with higher scores indicating higher symptom severity. The questionnaire consists of the three scales Depression (7 items; α = 0.88), Anxiety (7 items; α = 0.76), and Stress (7 items; α = 0.86; 36), which have shown satisfactory internal consistencies and factorial validity [37]. In the present study, internal consistencies were α = 0.86 (Depression), α = 0.73 (Anxiety), and α = 0.82 (Stress).

The KIDSCREEN-10 10 items [38] assesses various aspects of health-related quality of life and well-being in children and adolescents. The total quality of life score was used in the present study (KIDSCREEN Total, 10 items). The items were rated on a five-point Likert scale (0 = never/not at all to 4 = very often), with higher scores indicating higher quality of life and/or well-being. Internal consistency for the total scale was satisfactory, at α = 0.82, and factorial validity was demonstrated [38]. In the present study, internal consistency was α = 0.77.

All statistical analyses were performed using SPSS 28, R 4.4.0 or Mplus. Descriptive, reliability, and validity analyses were conducted separately in the total sample and in the subsamples originating from the different included studies.

An analysis of missing data in the total sample revealed 2.6% missing values for items of the FPNE across all caregivers. In cases of less than 10% missing values in the questionnaire per caregiver, they were conservatively replaced by 1 ("behavior is not shown"; n = 5). Questionnaires with more than 10% missing values were excluded from the analysis (n = 4).

For initial item selection and scale development of the FPNE, we performed exploratory factor analyses (EFA). To determine the best-fitting model, we estimated and sequentially compared two two-factor CFA models (first-order correlated-factors model, modified first-order correlated-factors model) and used exploratory structural equation modeling (ESEM). Validation of the factor structure was undertaken by randomly dividing the total sample into two halves, performing the EFA in the first subsample and the CFA and ESEM in the second subsample.

In psychological research, ordinal data often studied using factor analysis with Pearson correlation, constructed for metric and normally distributed data e.g., [3]. Our analyses of skewness and kurtosis, as well as our tests for normal distribution, showed significant deviations from the normal distribution for several items. The use of a polychoric correlation matrix is actually the more reliable method to create a more accurate measurement model that better reflects the latent relationships between items [39]. After careful consideration, we decided to combine both methods. To ensure methodological comparability with other studies, an exploratory factor analysis (principal axis analysis using oblimin rotation) based on Pearson correlations was first performed in SPSS initially without specifying the number of factors and second with specification of 2, 3, and 4 factors. The two-factor model was based on the assumption that there could be a global positive and a negative factor. To determine the appropriate number of factors to retain in the exploratory factor analysis (EFA), we used several criteria: the Kaiser criterion (eigenvalues > 1), the MAP test, and parallel analysis. We also chose a threshold for factor loadings of ≥|.32|, which is a commonly used threshold in factor analysis, to ensure that the item contributions to the factors are significant [40]. The two-factor model was then tested and compared in R using the psych and lavaan packages based on polychoric correlations.

For confirmatory factor analysis, we used the robust weighted least squares with mean and variance adjustment (WLSMV) estimator, as it does not assume normally distributed data [41]. To evaluate the model fit, the comparative fit index (CFI), the Tucker–Lewis index (TLI), the root mean square error of approximation (RMSEA) with corresponding 90% confidence intervals, and the standardized root mean square residual (SRMR) were considered as goodness-of-fit indices [42]. Given its dependence on the sample size, less attention was paid to the χ2 test. An RMSEA and SRMR ≤ 0.08 and CFI/TLI values > 0.90 were considered to be indicative of acceptable model fit. CFI/TLI values > 0.95 and RMSEA and SRMR values ≤ 0.05 were considered to indicate good model fit [42, 43]. Regarding the CFA, in the first-order correlated-factors model, correlations between the positive and the negative parenting factor were allowed. In contrast to the CFA models, the ESEM permitted cross-loadings between all items to prevent overestimation of factor loadings.

Additionally, we tested the measurement invariance of the final model for children’s age (6.00 to 9.56 vs. 9.57 to 12.11) and gender (male vs. female). Based on the median, the children were divided into two groups (Mdn = 9.56). For this purpose, stepwise restrictions were performed by constraining parameters to be equal across the groups. Strong factorial invariance is given when 1) configural, 2) metric, and 3) scalar invariance is achieved [44]. Restrictiveness increases with the different types of measurement invariance [41]. For configural measurement invariance, the same model was estimated in both groups, and the factor loadings, intercepts, and residual variances were allowed to vary freely. Configural measurement invariance was given in the case of the same loading pattern and number of factors. Metric invariance means that all groups have the same conceptual understanding of the latent constructs. For the examination of metric invariance, the loadings are additionally set to be equal in all groups. Scalar invariance means that the item difficulties do not differ between the groups. Additionally, the intercepts of the manifest variables in all groups have to be the same [45]. As the difference test is sample-sensitive, Chen's [46] rule of thumb was used to assess the model comparisons. Accordingly, the CFI should not decrease by more than 0.02 and the RMSEA should not increase by more than 0.015.

Following the testing for measurement invariance, latent mean differences between age and gender groups were examined, starting from the full scalar invariance model. To compare differences in latent means between groups, the mean for younger children and girls was set to zero as the reference group, while the means for older children and boys were estimated freely. This approach does not estimate absolute means for each group but rather assesses mean differences in the latent variables between the groups. Latent mean differences were evaluated using the critical ratio (CR) index, where CR values of 1.96 or above indicate significant differences. A positive CR value suggests that the comparison group has higher latent means than the reference group. Effect sizes were evaluated using Cohen’s d, which expresses the group mean difference as a proportion of the pooled within-group standard deviation. Following Cohen’s guidelines [47], we used the following interpretations for d: 0.20 ≤ d ≤ 0.39 as small, 0.40 ≤ d ≤ 0.79 as moderate, d ≤ 0.80 as large.

Subsequently, internal consistency (Omega, Cronbach’s alpha) and concurrent validity of the final model were examined. To assess validity, correlations between scale scores and parent-reported child ADHD/HKD/ODD symptoms (FBB-ADHS, FBB-SSV), child externalizing or internalizing symptoms (CBCL/6-18R), children’s quality of life (KIDSCREEN-10), and parental psychological symptoms (DASS-21) were examined using Pearson correlation coefficients (r). Based on the results of Pinquart [9, 10], we hypothesized that negative parenting would show a small to moderate association with psychopathological symptoms of children and parents; in relation to quality of life, we expected negative low to medium correlations. With regard to positive parenting behavior, we expected negative correlations in the low range with psychological problems of children and parents, and a small to moderate positive correlation with quality of life, as found in previous studies (e.g., 9, 10).