Interrater Agreement of BT-RADS for Evaluation of Follow-Up MRI in Treated Primary Brain Tumor Patients

BACKGROUND AND PURPOSE: The Brain Tumor Reporting and Data System (BT-RADS) is a structured radiology reporting algorithm that was introduced to provide uniformity in post-treatment primary brain tumor follow-up and reporting, but its interrater reliability (IRR) assessment has not been widely studied. Our goal is to evaluate the IRR among neuroradiologists and radiology residents in the use of BT-RADS.

MATERIALS AND METHODS: This retrospective study reviewed 103 consecutive MR studies in 98 adult patients previously diagnosed with and treated for primary brain tumor (January 2019 to February 2019). Six readers with varied experience (4 neuroradiologists and 2 radiology residents) independently evaluated each case and assigned a BT-RADS score. Readers were blinded to the original score reports and the reports from other readers. Cases in which at least one neuroradiologist scored differently were subjected to consensus scoring. After the study, a post-hoc reference score was also assigned by 2 readers using future imaging and clinical information previously unavailable to readers. The interrater reliabilities were assessed using Gwet’s AC2 index with ordinal weights and percent agreement.

RESULTS: Of the 98 patients evaluated (median age, 53 years; interquartile range, 41-66 years), 53% were males. The most common tumor type was astrocytoma (77%) of which 56% were grade 4 glioblastoma. Gwet’s index for interrater reliability among all six readers was 0.83 (95% CI: 0.78, 0.87). The Gwet’s index for the neuroradiologists’ group (0.84 [95% CI: 0.79, 0.89]) was not statistically different from that for the residents’ group (0.79 [95% CI: 0.72, 0.86]) (χ² = 0.85; p = 0.36). All four neuroradiologists agreed on the same BT-RADS score in 57 of the 103 studies, three neuroradiologists agreed in 21 of the 103 studies, and two neuroradiologists agreed in 21 of the 103 studies. Percent agreement between neuroradiologist blinded scores and post-hoc reference scores ranged from 41%-52%.

CONCLUSIONS: A very good interrater agreement was found when tumor reports were interpreted by independent blinded readers using BT-RADS criteria. Further study is needed to determine if this high overall agreement can translate into greater consistency in clinical care.

ABBREVIATIONS: BI-RADS = Breast Imaging Reporting and Data System; BT-RADS = Brain Tumor Reporting and Data System; IQR = interquartile range; IRR = interrater reliability; NI-RADS = Neck Imaging Reporting and Data System.