Inter- and Intrarater Agreement of CT Brain Calcification Scoring in Primary Familial Brain Calcification

Patient Selection

This study was conducted in accordance to the Strengthening the Reporting of Observational Studies in Epidemiology guidelines. Ethical approval was obtained from the local Dutch Medical Ethical Review Committee NedMec (registration numbers 21–170 and 22–1005). All patients gave written informed consent.

In this retrospective study, all patients aged ≥18 years who were diagnosed with PFBC or Fahr syndrome and visited the outpatient clinic of the University Medical Center Utrecht (UMCU), the Netherlands, between September 1, 2019 and July 1, 2023, were eligible for inclusion. The UMCU is an academic hospital with expertise in PFBC and Fahr syndrome. PFBC was diagnosed based on: 1) presence of bilateral basal ganglia calcifications as seen on CT, 2) TCS above the age-specific threshold (original TCS >0 in age <40 years, >4 in age 40 to 60 years, and >5 in >60 years),⁵ and 3) clinical symptoms consistent with the diagnosis. A genetic mutation associated with PFBC confirmed the diagnosis, but was not a mandatory criterion for inclusion. Patients were diagnosed with Fahr syndrome if they met the aforementioned criteria for PFBC, but a secondary cause associated with bilateral basal ganglia calcification was identified, for example, endocrine disorders such as hypoparathyroidism.¹ All consecutive patients who were scanned in an outpatient setting according to the PFBC CT scanning protocol in the UMCU were included in this study. Patients were excluded if they were scanned according to a different CT scanning protocol or in another hospital.

CT Imaging

The CT scanning protocol was an unenhanced scan of the full brain at 120 voltage (peak) (kVp) and 250 milliampere. All scans were acquired at dual-layer detector scanners (Iqon, CT7500, Philips Healthcare). The scan data were reconstructed at a slice thickness of 0.9 mm by using iterative model-based reconstruction. When a patient was scanned multiple times during the study period, the most recent scan was included.

Modified TCS

The original TCS is a visual rating scale that quantifies the amount of calcification in 18 different locations in the brain: left and right lenticular nucleus, left and right caudate nucleus, left and right thalamus, left and right cerebral subcortical white matter, cerebral cortex, left and right cerebellar hemisphere, vermis, left and right midbrain, pons, medulla, and left and right internal capsule only if independent of other calcifications. Each location gets attributed a score ranging from 0 to 5: 0 = no calcification; 1 = punctate; 2 = faint; 3 = moderate; 4 = severe; and 5 = severe and confluent. Examples of CT scans with different degrees of calcification in several locations are included in the article that developed the TCS.⁵ The TCS is the sum of score points of all locations and this overall score ranges from 0 to 90 points.⁵

After working with the original TCS in previous research, we updated the original TCS by modifying 3 aspects.² First, we removed the items “independent internal capsule calcification left and right” from the score. Independent internal capsule calcifications have not been observed in patients with PFBC or Fahr syndrome before on CT, neither in our study population (assessed by raters B.M.G.S. and P.A.d.J.), nor in the literature.^2,5 When the internal capsule is calcified, these calcifications are always confluent with calcifications in nearby locations and never independent.^2,5 Second, we divided the item “cerebral cortex calcification” into a left and right cerebral cortex, which is in accordance with the other locations that are already scored separately. Third, we added the items “hippocampal calcification left and right” to the score. We commonly observed hippocampal calcification in PFBC in our clinical practice, but the hippocampus is not included in the original TCS. Hippocampal calcifications were scored bilaterally as: 0 = no calcification; 1 = punctate; 3 = moderate; or 5 = severe. We did not include score 2 and 4 because the hippocampus is too small to quantify the calcifications on a 5-point scale. Examples of the hippocampal calcification score are shown in the Online Supplemental Data. An example of the modified TCS scoring form is shown in the Online Supplemental Data. The modified TCS ranges from 0 to 95 points.

CT Scan Assessment

Fifteen raters with different backgrounds and levels of expertise from the UMCU visually assessed 40 consecutive CT scans of the brain of 40 individual patients on 120 kVp reconstructions at least once, by using the modified TCS. Level of expertise for evaluating neurologic CT scans was predetermined based on the entrustable professional activity (EPA) level: I = has knowledge; II = may act under full supervision; III = may act under moderate supervision; IV = may act independently; and V = may act as supervisor and instructor.⁶ EPA level V applied to residents or radiologists with extensive experience in neuroradiology. Among the raters were: 3 radiologists (D.R.R., P.A.d.J., W.P.T.M.M., EPA levels V, IV, and IV, respectively), 3 radiology residents (M.M.v.B., S.V.d.L., S.M.U.V., EPA level IV, V, and III), 1 geriatrician (N.M.S.G., EPA level II), 1 internist (M.J.L.P., EPA level II), 5 geriatric residents (B.M.G.S., M.M.v.B., M.J.R., R.M.P., T.K., all EPA level II), 1 internal medicine resident (W.R.V., EPA level II), and 1 final-year medical student (B.D.W.T.L., EPA level II). Each rater received an instruction manual including an overview of the anatomy of the brain and examples of calcification scores in different brain regions, the latter was adapted from the developmental study of the TCS.⁵ Examples of hippocampal calcification were collected from a previous study that evaluated hippocampal calcification in a cohort of memory clinic patients.⁷ Three raters (B.M.G.S., P.A.d.J., W.P.T.M.M.) scored the 40 scans twice with a minimum time interval of 3 months between the first and second assessment to determine the intrarater reproducibility. During the second assessment, raters were blinded for scores of the first assessment. The scores of the first assessment were used to determine the interrater agreement. Each scan was examined in axial, coronal, and sagittal view. All raters were blinded for clinical variables.

Statistical Analysis

Data were collected based on the date of scanning and included: age, sex, diagnosis, and results of genetic testing (if available). Data were presented by using number with percentage for categoric variables, mean with standard deviation for continuous nonskewed variables, and median with interquartile range for skewed variables. Prevalence of calcified brain locations was calculated based on the scores of the rater with the highest level of expertise and most years of experience in neuroradiology (neuroradiologist D.R.R.). The interrater agreement for the overall score of the modified TCS and for the subscore of each individual item was assessed by using the intraclass correlation coefficient (ICC) with its 95% CI and Kendall concordance coefficient W, respectively. The interrater agreement was determined for all raters and per level of expertise. The intrarater agreement was assessed by using the ICC and weighted Cohen κ, respectively, with their 95% CIs.⁸ An ICC of <0.5 can be interpreted as poor, of 0.5–0.75 as moderate, of 0.75–0.9 as good, and of >0.90 as excellent agreement.⁹ Similarly, a weighted Cohen κ of <0.0 can be interpreted as poor, of 0.0–0.2 as slight, of 0.21–0.4 as fair, of 0.41–0.6 as moderate, of 0.61–0.8 as substantial, and of 0.81–1.0 as almost perfect.¹⁰ There is no general agreement on cutoff values to interpret Kendall W, which ranges from 0 (no agreement) to 1 (complete agreement). The higher the W score, the better the overall agreement.⁸ For the purpose of this study, the W scores were interpreted in the same way as the ICC (which also ranges from 0 to 1 and is interpreted to conform to the same rule “the higher, the better”). Statistical analyses were performed by using SPSS Statistics for Windows, version 29.0 (IBM).