Ethical approval for this observational comparative study, which utilized retrospectively analyzed data, was provided by the Ethics Committee of the University of Witten/Herdecke (approval no. 291/2021). The original pool of CBCT images was obtained from 547 patients who were treated at the Dental Clinic of the University of Witten/Herdecke, Germany in the years 2015–2016. Sample size determination was not undertaken for this preliminary study.

All CBCT images were generated in Digital Imaging and Communications in Medicine (DICOM) format with GALILEOS Comfort (Sirona Dental Systems GmbH, Germany) at an X-ray exposure of 85 kV and 5–7 mA (14 s; field of view: 150 × 150 mm, 200 singular images), yielding a voxel size of 0.027 mm3 and slice thickness of 300 µm.

The inclusion criteria for this study were as follows: patients aged between 8 and 40 years who had not received prior orthodontic or surgical treatment and had a CBCT image of adequate quality.

Patients with a history of craniofacial anomalies such as cleft lip and palate, cysts or tumors in the maxillary region, and CBCT scans with subjectively insufficient image quality were excluded.

The patient data were pseudonymized at the source. All CBCT scans were analyzed using OsiriX Lite version 11.0 (Pixmeo SARL, Bernex, Switzerland) in a dark room with an X-ray reporting monitor under the same screen settings. The 179 CBCT images were initially assessed and classified by a single examiner, a dentist who had received training in CBCT diagnosis. This examiner utilized dynamic free scanning to classify the images based on the classification system developed by Angelieri et al. [5]. We refer to the existing literature for the exact definitions of each class.

A second opinion from an experienced and trained orthodontist in CBCT diagnosis was sought when there were uncertainties regarding the classification. Through collaboration, a mutually agreed upon classification was established.

Subsequently, for a sample of 60 CBCT images selected randomly, an additional measurement was conducted two weeks later in order to assess the intra-rater reliability. The examiner’s classification was set as the benchmark for the following assessments by the three examiner groups.

A total of ten CBCT scans were selected at random, while ensuring that all defined stages were included at least once. These scans were assessed by three distinct groups of investigators, comprising dental students, orthodontic residents, and practicing orthodontists (each group comprised of five examiners). The assessment was conducted using a dynamic free-screening procedure. The examiners performed an independent evaluation and were required to orient the CBCT slices themselves, as instructed in training, to analyze midpalatal suture maturation. The number of examiners in each group was based on the study by Obuchowski, in which medical imaging studies were performed by 5–10 examiners [13]. The examiners were provided with training materials and an evaluation scheme in the form of a handout describing the radiological features at each maturity stage and accurate figures and flowcharts as described by Angelieri et al. to assess the midpalatal suture maturation stage. Prior to the assessment of the ten selected CBCT scans, all examiners were provided five different CBCT scans for training purposes. Examiners who had not acquired CBCT expertise were instructed using the software. Students were selected from higher semesters to ensure that they had adequate anatomical knowledge. A repeat assessment was performed after two weeks to measure the intra-rater reliability for each examiner. The software programs Medas (EDV Grundysteme, Margetshöchheim, Germany) and IBM SPSS Statistics 29 (IBM, Armonk, USA) were used for statistical analyses. Data distribution was analyzed using descriptive statistics and the Kolmogorov–Smirnov test. Weighted Cohen´s kappa was employed to determine the intrarater reliability for the benchmark classification and for inter- and intraexaminer agreement of the pooled performance for each group of dental students, orthodontic residents and orthodontists. Additionally, the agreement to the benchmark classification for each group was assessed using Cohen's kappa (k). The interpretation of kappa values was conducted in accordance with McHugh's classification system, which categorizes the level of agreement as follows: k ≤ 0.20 as none, k = 0.21 ≤ 0.39 as minimal, k = 0.40 ≤ 0.59 as weak, k = 0.60 ≤ 0.79 as moderate, k = 0.80 ≤ 0.90 as strong, and k > 0.90 as almost perfect [14].

For the level of intra-rater reliability exhibited by the individuals, a Kendall-tau correlation analysis was conducted and assessed in accordance with the classification system established by Chan, in which ratings of r < 0.3 were deemed poor, r = 0.3 ≤ 0.5 were considered fair, r = 0.6 ≤ 0.8 were categorized as moderately strong, and r > 0.8 were characterized as very strong correlation [15].

In addition to evaluate the deviation from the benchmark classification and compare the respective groups for diagnostic performance, stages A-E were coded in numbers 0–4. The classified values were summed for the respective examiners and the values of the benchmark classification were subtracted. Finally, the mean values for the respective groups were calculated, in analogy to the study of Barbosa et al. utilizing median values [16], and Wilcoxon pairwise comparison analysis was performed to compare for significant differences between the groups. The significance level for all statistical tests was set at p < 0.05.