Evaluation of an Artificial Intelligence Model for Identification of Mass Effect and Vasogenic Edema on CT of the Head

Study Design

This retrospective stand-alone model performance study was conducted using radiology cases from 5 hospitals within the Mass General Brigham network between 2016 and 2022 using methods similar to those found in a previously published study about intracranial hemorrhage identification.¹² The study examined the performance for the binary detection of mass effect and vasogenic edema by the AI model. It was approved by the Mass General Brigham institutional review board with a waiver of informed consent. It was conducted in accordance with relevant guidelines and regulations including the Health Insurance Portability and Accountability Act. This article follows the Standards for Reporting Diagnostic Accuracy (STARD 2015; https://www.equator-network.org/reporting-guidelines/stard/) reporting guidelines.

Case Selection

The cohorts for mass effect and vasogenic edema were selected in a consecutive manner on the basis of the original radiology reports. The cohort size for each of the positive and negative cases was based on powering calculations as described in the statistical analysis section below. The positive cases were identified through a natural language-processing search engine (Nuance mPower Clinical Analytics; https://www.nuance.com/healthcare/diagnostics-solutions/radiology-performance-analytics/mpower-clinical-analytics.html) followed by a manual report review. The negative cases were identified by taking the next negative case acquired on the same CT scanner after the positive cases to avoid temporal and technical bias. The next negative cases were taken after every Nth positive case based on the ratio of positive-to-negative cases to ensure that the principles of consecutive selection applied.

The cohort considered all CT head cases performed at a hospital, including inpatients and outpatients; there were no limitations on the original CT head clinical indication. The CT head cases were obtained from patients at least 18 years of age. The CT head cases were taken from unique patients; only the first CT head from a given patient was included. It was possible for a case to be included in both cohorts (ie, both mass effect and vasogenic edema); there were 8 cases that were included in both cohorts.

All cases were de-identified and underwent an image-quality review by an American Board of Radiology–certified neuroradiologist. The relevant series for the model interpretations were selected at the same time as described under the Series Selection section below. The review was performed using the FDA-cleared eUnity image visualization software (Version 6 or higher; Mach7) and an internal Web-based annotation system that used the REDCap electronic data-capture tools hosted at Mass General Brigam.^13,14

Series Selection

The model was provided with a single selected series at the time of model inference. These series were noncontrast thin (≤1.5 mm) and/or thick (>1.5 and ≤5 mm) axial series for each CT head case. The series were selected so that the thin series was the thinnest available series ≤1.5 mm; the thick series was randomized between the thinnest and thickest available series in the range >1.5 and ≤5 mm to ensure representation of series thicknesses across the entire range. The series were selected at the same time as the image-quality review. After series selection, a DICOM metadata review was additionally performed to ensure that the section thickness was within the appropriate range and that there was a consistent section interval (with tolerance of ±0.2 mm).

Ground Truth Interpretations

Ground truth interpretations were performed by up to 3 American Board of Radiology–certified neuroradiologists. They answered whether the relevant finding was “Present” or “Absent.” The definition provided to the neuroradiologists for mass effect was “mass effect as evidenced by effacement of ventricles, basal cisterns or cerebral sulci, midline shift, or brain herniation (eg, tonsillar herniation or uncal herniation).” The definition of vasogenic edema was “deep white matter hypodensity extending into subcortical white matter.” The neuroradiologists also answered whether a “parenchymal abnormality including ischemia/mass/cyst/encephalomalacia” was present. They provided their interpretations independently, without access to the original radiology reports and in different worklist orders. They used the same image-visualization software and annotation system as used in the image-quality review. They had access to the entire CT head case (ie, they were not restricted to the series selected for model inference). For determining consensus interpretations for the presence of mass effect or vasogenic edema, we used a “2 + 1” strategy: The first 2 neuroradiologists interpreted every case and a third neuroradiologist then interpreted cases with discrepant interpretations. A parenchymal abnormality was considered present if either of the first 2 neuroradiologists annotated it as present; the third neuroradiologist was not asked about its presence.

Model Inference

The evaluated AI model was Version 3.1.0 of the Annalise Enterprise CTB Triage Trauma device (https://annalise.ai/annalise-triage/). It is the same AI model used by the Annalise Enterprise (CTB module) device, which is commercially available in some non-US markets and whose development has been previously described.⁹ In brief, it consists of an ensemble of 5 neural networks with 3 heads: one for classification, one for left-right localization, and one for segmentation. It can identify 130 different radiologic findings and was trained on >200,000 CT head cases, which were each labeled by at least 3 radiologists. These training cases came from 8 different scanner manufacturers and >90 different scanner models. The training cases were completely independent of the cases used for this stand-alone model performance study.

The Annalise Enterprise CTB Triage Trauma device provides only binary classification output about the identification of findings, which is consistent with FDA regulations for CADt devices. The model was installed at Mass General Brigham for use in this study and received only the DICOM-formatted CT head cases. It output a classification score between 0 and 1 for mass effect and vasogenic edema. A binary output for these findings could be derived using prespecified operating points. As part of model inference, the device contains multiple filters to look at attributes of the series to be interpreted to ensure that the model does not perform inference on unsuitable images; in these situations, the device does not produce an output, which is referred to as “unsuccessful model inference” within this study. While not part of the current study, internal bench testing indicated a model turn-around time of 81.6 seconds (95% CI, 80.3–82.9 seconds).^10,11

Statistical Analysis

The statistical analysis was performed in R (Version 4.0.2; http://www.r-project.org) on the full analysis set. The predefined end points included the areas under the receiver operating characteristic curves (AUCs) for the identification of mass effect and vasogenic edema for each of the thin and thick series. The AUCs were calculated using the consensus annotations and the classification scores from the AI model. The prespecified end points also included the sensitivity and specificity at predetermined operating points; this article reports the performance at operating points that have received FDA clearance. They were calculated by comparing the binary model output at each operating point with the consensus annotations (ie, by calculating the number of true-positive, false-negative, true-negative, and false-positive cases).

The positive predictive values (PPVs) and negative predictive values (NPVs) were calculated as exploratory analyses at assumed prevalences of 0.05, 0.10, 0.15, and 0.20. The sensitivities and specificities were calculated as exploratory analyses for the subgroups of presence or absence of a parenchymal abnormality. The AUCs, sensitivities, and specificities were calculated as exploratory analyses for the subgroups of sex, age, ethnicity, race, and CT scanner manufacturer. These parameters were derived from clinical databases or DICOM fields for each radiology case. Any missing data were treated as “unknown” or “unavailable,” and no data were imputed.

All CIs were calculated using bootstrapped intervals with 2000 resamples. The sample sizes for each of the findings were powered on the basis of preliminary model results at a balanced operating point to ensure that the lower bound of the 95% CI for sensitivity was >80%, and for specificity, >80%.