The first objective of the EQA pilot scheme was the identification and validation of adequate control material for TMB estimation. In this respect, several different commercial reference material manufacturers were invited through a tender. Special requirements for the supply of samples were specified, including the potential yield of DNA, variability of TMB values, and availability of cell lines as FFPE blocks. Seracare (LGC Clinical Diagnostics, Inc.) was awarded the supply contract, and the IQNPath panel of experts selected from the list provided the following nine cell lines: NCI-H2009, NCI-H2126, NCI-H2171, NCI-H1437, NCIH23, NCIH322, SKMEL2, C33A, and IGROV1.

We performed a validation phase on the selected cell lines using three different commercial NGS panels (see “Materials and Methods”; report files are uploaded at link: 10.5281/zenodo.6563860). Two independent gDNA extractions with the GeneRead™ DNA FFPE Kit (Qiagen) obtained comparable results in terms of gDNA yield (data not shown). Similarly, two independent NGS runs using the OTML, TSO500, and QIAseq Targeted DNA Panel were conducted to ensure the consistency of the results. The FFPE cell lines were also tested with the F1CDx assay (reported in figures and tables as FMI). The results are summarized in Fig. 1.

Results of the internal validation phase on 9 cell lines. The TMB values were calculated using three different commercial NGS panels (OTML, TSO500, and QIAseq Targeted DNA Panel) and compared with the reference F1CDx assay

All the samples returned valid TMB results with the four NGS panels, and between the two parallel runs, the standard deviation between the same panel was very low (0.01–5.07; 0–6.98; 0–0.54; 0–7.13 for TSO500, OTML, QIAseq, and F1CDx, respectively). However, there were significant differences between the TMB scores obtained from the four NGS assays (Fig. 1 and Supplementary Table 1). In particular, the concordance of the TMB values was low among the TSO500, OTML, and QIAseq assays and between these panels and the reference F1CDx test. Although the cutoff of the different tests is likely to be different, the important differences found could result in a misclassification of the samples in high vs low TMB.

Following this validation phase, five cell lines, NCI-H2171 (TMB1), NCI-H1437 (TMB2), NCI-H2126 (TMB3), NCI-H2009 (TMB4), and C33A (TMB5), covering a TMB range “low to high,” were chosen as the EQA materials. Before shipment of the samples to the participating centers, two additional runs were performed on these five samples from two new independent gDNA extractions (Supplementary Fig. 1). These results confirmed the data of the initial validation. Finally, thirty laboratories were selected to participate in the pilot EQA through a survey, the results of which have been recently published. These laboratories were selected on the basis of several criteria including their expertise and experience in NGS technologies and the method they used to assess TMB [29].

Twenty-nine laboratories confirmed their participation in the EQA scheme and received the samples, specifically from twelve different countries registered with AIOM (N = 3), ESP (N = 4), EMQN (N = 14), GenQA (N = 6), and Gen&Tiss (N = 2). Twenty-three (79.3%) laboratories submitted results. One laboratory submitted results using two different panels (Lab6 and Lab6bis). Out of 29 participating laboratories, two (6.9%) withdrew from the scheme because of the absence of “normal” (non-tumor tissue) matched samples, and four (13.8%) did not submit results without providing any explanation. All participating laboratories provided details of the DNA extraction and NGS methodology that they used to assess TMB. Eleven different commercially available kits for DNA extraction from FFPE tissue were used for DNA extraction (Table 1). The GeneRead FFPE DNA kit (Qiagen) was the most popular kit followed by the QIAmp DNA FFPE kit (Qiagen). Notably, no participating laboratory reported a critical issue in this preanalytical phase, confirming the results obtained in the validation phase of the EQA and suggesting that sufficient good-quality DNA was present in the samples.

A wide array of NGS panels and methods for library preparation were employed by the participating laboratories (Table 2). The majority of participating laboratories used a targeted sequencing approach with multigene testing panels. Only two laboratories performed WES and two assessed TMB by clinical exome. The OTML was the most widely used commercially available panel, although five laboratories used custom panels. The characteristics of the NGS methods are summarized in Supplementary Table 2.

A general overview of the results submitted and compared with the TMB values assessed by the F1CDx test is shown in Table 3 and Fig. 2. As expected, the use of different technologies for TMB testing led to significant variability in the reported TMB values. The variability was higher for sample TMB5, which had the highest TMB value according to the F1CDx test, when compared to the other testing methods. All the laboratories reported higher TMB values for sample TMB2 as compared with the F1CDx result, which was confirmed by a repeated analysis with the F1CDx panel. Importantly, excluding outliers, good reproducibility of the TMB score was shown by laboratories using the same panel. In this respect, the aim of the pilot was achieved, and the commercial material used appears suitable as a reference for TMB evaluation using a variety of different testing methods. The majority of laboratories did not indicate a TMB cut-off value, and when a value was reported, it was not clear if the cut-off value provided was effectively validated. Therefore, it was not possible to provide any feedback about the clinical interpretation of the TMB value. Due to the challenging nature of this biomarker, no scoring of the results from the participating laboratories was applied. The assessment process has been conducted by benchmarking the performance of each laboratory in two different ways: comparing the reported results against the reference F1CDx test and comparing the reported results against other centers using the same method.

Summary of results for TMB test submitted by different laboratories. The five selected samples were tested by participating laboratories using different methods. The circles represent the TMB value reported by each participating center, while the horizontal bar indicates the TMB value obtained with the reference F1CDx assay

The results submitted by laboratories using the OTML showed good inter-laboratory reproducibility, although some variability was observed; such variability was higher in the case of TMB5 compared with other samples (Supplementary Fig. 2). To better understand the reason for this variability, we analyzed the results taking into account the bioinformatics pipeline used by the laboratories to call TMB. In particular, four laboratories (Lab 2, Lab 6bis, Lab 19, and Lab 24) included only non-synonymous mutations in the assessment of TMB, whereas three centers (Laboratories 16, 17, and 18) used both non-synonymous and synonymous variants. Dividing the laboratories into two groups taking into consideration the different pipelines used to call TMB, we observed a lower inter-laboratory variability in the group of laboratories that included only non-synonymous variants. In addition, the values reported by these centers were closer to values of F1CDx as compared to the values of the laboratories that used both synonymous and non-synonymous alterations for TMB estimation (Table 4). The recommended TMB evaluation algorithm for the OTML panel includes only non-synonymous variants.

Regarding the three laboratories that used the Oncomine Comprehensive Assay Plus (OCA Plus), an almost perfect reproducibility of the results for all the samples was observed. In fact, the submitted results showed a very low standard deviation ranging between 0.38 and 0.87 (data not shown). Nevertheless, the TMB value obtained with OCA Plus in 3/5 samples was overestimated as compared to FMI values (Supplementary Fig. 3). On the other hand, for laboratories that used the TSO500 panel, excluding a clear outlier (Lab 1), they presented a good inter-laboratory reproducibility. However, all the laboratories (apart from Lab 1) using the TSO500 underestimated the TMB value of TMB1, and they also showed a greater variability for TMB5 (Supplementary Fig. 4). Of the two labs that used WES, Lab 23 overestimated TMB for 3 out of four samples as compared to F1CDx, whereas the results submitted by Lab 11 were in line with the reference score (Supplementary Fig. 5). The laboratories that performed “Clinical exome” testing showed results very close to the F1CDx values even if they overestimated TMB2 and TMB4 samples (Supplementary Fig. 6). Lastly, for laboratories which used unique custom panels, they submitted results similar to the F1CDx score. For this group of laboratories, it is important to note a higher variability for TMB5 and the overestimation for TMB2 (Supplementary Fig. 7).