This study showed that scoring PACSS on pre-operative CTA imaging in popliteal and infrapopliteal lesions can be performed reliably if prior scoring arrangements are made. In addition, scoring the calcification of the entire TV appeared to be reliable.

One previous study tested the inter- and intra-observer reliability of semi-quantitative peripheral calcification scores [16]. The referred study scored the PACSS and PARC on DSA and the Fanelli scoring system on both DSA and CTA imaging. All scoring systems were dichotomized and tested on femoropopliteal lesions. The inter- and intra-observer agreement of the binary PACSS were fair (κ = 0.32) and moderate to substantial (κ = 0.52–0.62) according to the scale recommended by Landis and Koch, respectively. The inter- and intra-observer agreement in the other calcification scores were fair (κ = 0.38–0.40) and highly variable (κ = 0.36–0.92), respectively.

The PACSS score has been studied primarily in femoropopliteal lesions, in which higher PACSS grades were significantly associated with loss of primary patency, increased target lesion revascularization (TLR), increased MALE and increased mortality [8,9,10,11,12]. Regarding infrapopliteal disease, one retrospective study found that PACSS grade 4 was associated with failed guidewire crossing of below-the-knee chronic total occlusions in univariate analysis [20]. All these studies scored the grade of calcification on two-dimensional DSA instead of CTA imaging. DSA is superior in assessing vessel patency, the absence of which lowers the likelihood of achieving lesion revascularization. However, CTA imaging has the advantages of three-dimensionality and non-invasiveness, and can be performed preoperatively for risk stratification. In addition, previous studies concluded that DSA is limited in its ability to identify calcium as compared with intravascular ultrasound (IVUS). Most importantly, the inter- and intra-observer agreement of PACSS on DSA were unacceptably low [16, 17]. Therefore, these studies raised serious concerns regarding the accuracy and reliability of the currently most commonly used angiographic calcification scores for PAD.

This study demonstrates that the original 5-step PACSS can be scored reliably on CTA imaging. Besides the original PACSS score, we also tested the reliability of a binary PACSS score. Semi-quantitative scores with more than two categories are suitable for determining the prognosis, however, in clinical practice the interventionalist is confronted with a binary question, namely is the calcification present severe enough to choose for an additional calcium modifying treatment such as atherectomy or specialty balloons[16, 21, 22]. The inter-observer agreement of the binary score was slightly higher than the original 5-step PACSS score, while the intra-observer agreement was similarly strong.

In addition, we decided to add the calcification grade of the entire TV, as it might be a more accurate representation of total limb calcification than TL calcification alone, as measured in the original PACSS score. Two studies already demonstrated that TV calcification of infrapopliteal arteries is a significant predictor of technical success, limb salvage amputation-free survival (AFS) and major adverse cardiovascular events (MACE) [6, 15].

The inter-observer agreement of the trichotomic TV calcification and 7-step mPACSS score was substantial (κ = 0.64) and moderate (κ = 0.48), respectively, while the intra-observer agreement was almost perfect (κ = 0.87) and substantial (κ = 0.77), respectively. The inter-observer reliability in all scores was calculated with Fleiss’ kappa, which is suitable for calculating the absolute agreement of nominal scores without any intrinsic ordering or ranking [19]. On the other hand, in the case of true ordinal scores, weighted kappa values should be calculated. Regarding PACSS and mPACSS, it is debatable whether these scores should be considered nominal or truly ordinal. In case both scores are considered truly ordinal, the weighted kappa values of inter-observer agreement would be substantially higher (PACSS: κ = 0.71–0.76, mPACSS: κ = 0.72–0.77). Supplementary Table 1 summarizes all weighted kappa values of the PACSS score and its modifications.

Previous studies using CTA imaging to score the grade of infrapopliteal calcification used the quantitative tibial artery calcification (TAC) score and found associations between severe TAC and lower technical success, limb salvage, AFS and survival free of MACE [8, 17]. The inter-observer agreement was high for the quantitative TAC score, which makes quantitative calcium scoring a promising method that deserves further evaluation in future research. On the other hand, quantitative scores are not widely available, time consuming and dependent on a certain level of expertise. This study has shown that semi-quantitative scoring of PACSS on CTA is reliable and fast in popliteal and infrapopliteal disease. Next steps are to test the reliability in femoropopliteal disease and to validate this calcium score on a patient cohort with treatment outcomes, preferably a large prospective multi-center cohort. The use of virtual non-contrast techniques may help improve calcium scoring. Ideally, the scoring system should aid the clinician’s decision in the approach for revascularization, thereby improving revascularization success.

Limitations of this retrospective study are the biased study population, small differences in imaging protocols and limited data regarding intra-observer reliability. In this study, 94% of the limbs were included for CLTI and only popliteal and infrapopliteal lesions were included because the study population was extracted from the THRILLER registry. This may affect the generalizability of the findings of this study to the overall PAD population. Secondly, the protocols for CTA imaging were generally consistent, but minor differences existed. For example, slice thickness varied from 1 to 1.5 mm, which made scoring of calcium bilaterality slightly more difficult in the scans with thicker slices, particularly in the infrapopliteal arteries. Thirdly, only one rater scored PACSS twice, which resulted in limited data regarding intra-observer reliability. However, for this rater, substantial to almost perfect agreement was achieved in all scoring components.

Finally, prior scoring arrangements were essential to improve inter-observer agreement, but may have conflicted with the original assignment of PACSS grades. For example, lesions smaller than 5 cm could not be scored in grades 2 and 4. Nevertheless, the exact interpretation of the different PACSS grades remains unclear and is subject for future scientific research [3].