Identification of Small, Regularly Shaped Cerebral Aneurysms Prone to Rupture

American Journal of Neuroradiology

Download figure
Open in new tab
Download powerpoint
FIG 1.
Performance of different machine learning models on an independent, external validation data set: A, ROC curves. B, Precision/recall curves. The best performance is achieved by the SVM model (red curves). BG indicates bagging or bootstrap aggregating; RF, random forest; SVM, support vector machine; KNN, K-nearest neighbor; LR, logistic regression.
Download figure
Open in new tab
Download powerpoint
FIG 2.
Variable importance for the SVM model applied to the validation data set. Variables are ordered from top to bottom according to their importance determined by the AUC of the ROC as the measure of variable importance. See the Online Supplemental Data for an explanation of the terms.

Table 1:

Summary of evaluation metrics for machine learning predictive models during repeated internal 10-fold cross-validation

Model	AUC (mean) (maximum)	TPR (mean) (maximum)	FPR (mean) (minimum)	Misclassification Error (mean) (minimum)
BG	0.84 (0.91)	0.75 (0.84)	0.23 (0.18)	0.23 (0.14)
RF	0.84 (0.92)	0.78 (0.89)	0.25 (0.17)	0.25 (0.16)
SVM	0.85 (0.90)	0.82 (0.95)	0.26 (0.16)	0.23 (0.17)
KNN	0.82 (0.90)	0.74 (0.90)	0.23 (0.14)	0.23 (0.15)
LR	0.83 (0.92)	0.73 (0.95)	0.23 (0.15)	0.24 (0.16)

Note:—AUC indicates area under the curve; BG, bagging or bootstrap aggregating; FPR, false-positive rate; KNN, K-nearest neighbor; LR, logistic regression; RF, random forest; SVM, support vector machine; TPR, true-positive rate.

Table 2:

Performance measures for each machine learning model applied to the external testing data set^{^a}

Model	AUC	TPR	FPR	PPV	NPV	F1 Score	Balanced Accuracy	Misclassification Error
BG	0.74	0.71	0.35	0.24	0.93	0.36	0.68	0.34
RF	0.76	0.71	0.32	0.26	0.94	0.38	0.70	0.32
SVM	0.84	0.93	0.35	0.30	0.98	0.45	0.79	0.31
KNN	0.76	0.79	0.36	0.26	0.95	0.39	0.71	0.34
LR	0.77	0.86	0.37	0.27	0.96	0.41	0.74	0.34

Note:—NPV indicates negative predictive value, the number of true-negatives divided by the number of true- and false-negatives; AUC, area under curve; FPR, false-positive rate (1-specificity = number of false-positives divided by all negatives); PPV, positive predictive value (precision = number of true-positives divided by number of true- and false-positives); TPR, true-positive rate (sensitivity or recall = number of true-positives divided by all positives).
↵a F1 = 2 × PPV × TPR / (PPV + TPR) is the harmonic mean of precision and recall. Balanced accuracy is accuracy accounting for class imbalance [(sensitivity + specificity)/ 2]. Misclassification error is the number of incorrect classifications divided by sample size.