Automated Detection of Steno-Occlusive Lesion on Time-of-Flight MR Angiography: An Observer Performance Study

Study Cohort

Sixty-two individuals were collected from the SNUBH database between October 2014 and August 2019 as an internal test set, including 30 with intracranial stenosis or occlusion (stenosis group) and 32 without steno-occlusive lesions (healthy group). The inclusion criteria for the stenosis group were as follows: 1) age >18 years, 2) TOF-MRA and DSA performance within a 1-month interval, and 3) moderate or severe degree stenosis according to the DSA report (>50% stenosis by using the Warfarin Versus Aspirin for Symptomatic Intracranial Disease method as follows: % stenosis = (1−[D_stenosis/D_normal]) × 100). The inclusion criteria for the healthy group were as follows: 1) age >18 years, 2) TOF-MRA performance, and 3) normal MRA findings according to the radiologic report.

In addition, 76 individuals were collected as an external test set from the SSMH database from January 2016 to December 2019, comprising 30 with intracranial stenosis or occlusion (stenosis group) and 46 without steno-occlusive lesions (healthy group). The inclusion criteria for the SSMH cohort were as follows: 1) age >18 years and 2) TOF-MRA results.

Patients in the stenosis group were randomly selected. For the healthy group, the patients were randomly chosen among individuals who underwent brain MR imaging as part of health screenings and received normal MRA reports. In total, we collected 138 individuals from 2 institutions, with 60 in the stenosis group and 78 in the healthy group.

TOF-MRA Acquisition

For SNUBH, 3D TOF-MRA examinations were performed using a 1.5T (Amira, Siemens Healthineers, or Intera, Philips Healthcare) or 3T scanner (Achieva or Ingenia, Philips Healthcare). The scan parameters were as follows: TR, 20–27 ms; TE, 3.45–7.15 ms; flip angle, 18°–25°; field of view, 132–230 mm; section thickness, 0.5–1.6 mm; matrix, 256 to 704 × 163 to 360.

For SSMH 3D TOF-MRA was performed using a 1.5T (Avanto, Siemens Healthineers; Achieva) or 3T (Verio or Vida, Siemens Healthineers; Ingenia) scanner. The scan parameters were as follows: TR, 17.9–25 ms; TE, 3.5–7 ms; flip angle, 18°–23°; field of view, 170–240 mm; section thickness, 0.4–1.2 mm; matrix, 384 to 512 × 214 to 331.

AI Model

We used a deep-learning algorithm to detect steno-occlusive lesions by using traces of intracranial arteries (Fig 1). Our model utilized an image segmentation model such as U-Net as a backbone,¹¹ augmented by additional modules to detect steno-occlusive lesions. Specifically, we designed a backbone model termed Spider U-Net, which is a modified version of U-Net that adds a long short-term memory network. The vessel segmentation performance of Spider U-Net outperformed that of U-Net.¹² A multitask learning method based on Spider U-Net demonstrated that the detection performance of steno-occlusive lesions while extracting blood vessels was higher than that while detecting steno-occlusive lesions without extracting blood vessels.¹⁰ The model used a training set from SNUBH similar to that used previously and demonstrated superior overall detection performance compared with other models.¹⁰ Details of the model have been published elsewhere.^10,12 Code details can be accessed through the following link: https://github.com/djchoi1742/MRA_ICAD.

• Download figure
• Open in new tab
• Download powerpoint

FIG 1.

Representative image of our proposed model. A, Initial TOF-MRA shows multiple steno-occlusive lesions, including left middle cerebral artery occlusion. B, Our proposed AI model identified steno-occlusive lesions, shown as red and yellow markings on the heat map.

Image Interpretation and Observer Performance Study

A neuroradiologist (L.S., with 13 years of clinical experience), a board-certified radiologist (H.L., with 5 years of clinical experience), and a neurosurgeon (T.K., with 13 years of clinical experience) thoroughly reviewed all TOF-MRA examinations and accessible DSA studies, establishing a reference standard by consensus for the number and location of steno-occlusive lesions. The reference standards for steno-occlusive lesion detection were confined to the distal ICA, the A1-2 segments of the anterior cerebral artery, the M1-2 segments of the MCA, the P1-2 segments of the posterior cerebral artery, the V4 segments of the vertebral artery, and the basilar artery.

Five radiologists, including 2 neuroradiologists (S.H.B. and S.J.C., with 10 and 9 years of experience in neuroradiology, respectively) and 3 radiology residents (J.H.J., H.S.C., and H.U.C., with 4, 3, and 3 years of clinical experience, respectively), participated as observers. The radiologists who determined the reference standard did not participate. Each reviewer conducted 2 separate assessments of all TOF-MRA studies (n = 138) across 2 sessions. The studies were randomly divided into 2 blocks (Block A and Block B, each containing 69 studies). During the first session, studies in Block A were reviewed with AI assistance, and those in Block B were reviewed without AI. To mitigate bias and memory recall effects, a compulsory 4-week washout period was placed between the sessions. After this interval, the review conditions were swapped: During the second session, studies initially reviewed with AI (Block A) were assessed without AI, and vice versa for Block B (Fig 2). The reviewers were blinded to the patient information and reference standards. They were instructed to assess whether there is stenosis or occlusion in the intracranial artery by using a commercial PACS. Upon conducting an AI-assisted review, the heat map generated by the AI model was superimposed onto the MIP TOF-MRA images. The heat map images were reviewed using an additional viewer (Windows Photo Viewer, Microsoft). The participants were instructed to mark a stenotic or occlusive lesion by recording the coordinates of the central portion of the lesion. The confidence rating for each lesion ranged from 1 (very uncertain) to 5 (absolutely certain). The overall reading time was recorded for each case. During the review, they were encouraged to evaluate both the source and MIP TOF-MRA images.

• Download figure
• Open in new tab
• Download powerpoint

FIG 2.

A diagram illustrating the observer performance study. Each observer conducted 2 separate TOF-MRA reviews across 2 reading sessions: 1 without AI and another with AI. There was a washout period of 4 weeks or longer between these sessions.

The observers’ performances were stratified into neuroradiologist and radiology resident groups and were combined eventually. An overlap between the lesions marked by the observer (indicated by the coordinates) and reference standard (annotated across the entire length) was classified as a true-positive finding; otherwise, it was classified as a false-positive finding.

Statistical Analysis

We performed a jackknife free-response receiver operating characteristic (JAFROC) analysis to evaluate the reviewers’ localization performance on a per-lesion basis.¹³ The area under the JAFROC curve (AUFROC) indicates the probability that the lesion rating marked in the diseased case is greater than the highest rating in the healthy case. We calculated the AUFROC according to AI use and performed a comparison test by using the Dorfman-Berbaum-Metz method.¹⁴ Comparisons based on AI use were calculated as fixed-reader random cases; for pooled reviewers, they were calculated as random-reader random cases. We computed the area under the receiver operating characteristic curve (AUC) to measure the performance on a per-patient basis. We calculated the average of the reviewers’ ratings per patient. Similar to JAFROC, we performed a comparison test of the AUC by using an identical method for each reviewer and pooled reviewers.

We calculated the sensitivity and specificity to measure the diagnostic accuracy of the reviewers. The sensitivity was calculated for each lesion and per patient. Sensitivity per lesion was calculated under a cutoff of the sum of the 1 – false-positive fraction, and the lesion localization fraction was maximized. The sensitivity per patient was calculated by using an optimal cutoff to maximize the Youden J statistic.¹⁵ Specificity was calculated for each patient.

We used the intraclass correlation coefficient (ICC)¹⁶ to measure interobserver agreement among the reviewers. The ICC values <0.5, 0.5–0.75, 0.75–0.9, or ≥0.9 indicated poor, moderate, good, or excellent agreement, respectively.¹⁷ We calculated the ICC based on AI use. All statistical analyses were performed by using the R statistical software version 3.6.3 (R Foundation for Statistical Computing). Statistical significance was set at P < .05.