Accurate detection and quantification of unruptured intracranial aneurysms (UIAs) is important for rupture risk assessment and to allow an informed treatment decision to be made. Currently, 2D manual measures used to assess UIAs on Time-of-Flight magnetic resonance angiographies (TOF-MRAs) lack 3D information and there is substantial inter-observer variability for both aneurysm detection and assessment of aneurysm size and growth (Forbes et al., 1996; Kim et al., 2017; White et al., 2000). 3D measures could be helpful to improve aneurysm detection and quantification but are time-consuming and would therefore benefit from a reliable automatic UIA detection and segmentation method. The Aneurysm Detection and segMentation (ADAM) challenge was organised in which methods for automatic UIA detection and segmentation were developed and submitted to be evaluated on a diverse clinical TOF-MRA dataset. A training set (113 cases with a total of 129 UIAs) was released, each case including a TOF-MRA, a structural MR image (T1, T2 or FLAIR), annotation of any present UIA(s) and the centre voxel of the UIA(s). A test set of 141 cases (with 153 UIAs) was used for evaluation. Two tasks were proposed: (1) detection and (2) segmentation of UIAs on TOF-MRAs. Teams developed and submitted containerised methods to be evaluated on the test set. Task 1 was evaluated using metrics of sensitivity and false positive count. Task 2 was evaluated using dice similarity coefficient, modified hausdorff distance (95th percentile) and volumetric similarity. For each task, a ranking was made based on the average of the metrics. In total, eleven teams participated in task 1 and nine of those teams participated in task 2. Task 1 was won by a method specifically designed for the detection task (i.e. not participating in task 2). Based on segmentation metrics, the top two methods for task 2 performed statistically significantly better than all other methods. The detection performance of the top-ranking methods was comparable to visual inspection for larger aneurysms. Segmentation performance of the top ranking method, after selection of true UIAs, was similar to interobserver performance. The ADAM challenge remains open for future submissions and improved submissions, with a live leaderboard to provide benchmarking for method developments at https://adam.isi.uu.nl/.

Comparing methods of detecting and segmenting unruptured intracranial aneurysms on TOF-MRAS: the ADAM challenge / Timmins, Kimberley M; van der Schaaf, Irene C; Bennink, Edwin; Ruigrok, Ynte M; An, Xingle; Baumgartner, Michael; Bourdon, Pascal; De Feo, Riccardo; Noto, Tommaso Di; Dubost, Florian; Fava-Sanches, Augusto; Feng, Xue; Giroud, Corentin; Group, Inteneural; Hu, Minghui; Jaeger, Paul F; Kaiponen, Juhana; Klimont, Michał; Li, Yuexiang; Li, Hongwei; Lin, Yi; Loehr, Timo; Ma, Jun; Maier-Hein, Klaus H; Marie, Guillaume; Menze, Bjoern; Richiardi, Jonas; Rjiba, Saifeddine; Shah, Dhaval; Shit, Suprosanna; Tohka, Jussi; Urruty, Thierry; Walińska, Urszula; Yang, Xiaoping; Yang, Yunqiao; Yin, Yin; Velthuis, Birgitta K; Kuijf, Hugo J. - In: NEUROIMAGE. - ISSN 1053-8119. - 238:(2021), pp. 1-21. [10.1016/j.neuroimage.2021.118216]

Comparing methods of detecting and segmenting unruptured intracranial aneurysms on TOF-MRAS: the ADAM challenge

De Feo, Riccardo;
2021

Abstract

Accurate detection and quantification of unruptured intracranial aneurysms (UIAs) is important for rupture risk assessment and to allow an informed treatment decision to be made. Currently, 2D manual measures used to assess UIAs on Time-of-Flight magnetic resonance angiographies (TOF-MRAs) lack 3D information and there is substantial inter-observer variability for both aneurysm detection and assessment of aneurysm size and growth (Forbes et al., 1996; Kim et al., 2017; White et al., 2000). 3D measures could be helpful to improve aneurysm detection and quantification but are time-consuming and would therefore benefit from a reliable automatic UIA detection and segmentation method. The Aneurysm Detection and segMentation (ADAM) challenge was organised in which methods for automatic UIA detection and segmentation were developed and submitted to be evaluated on a diverse clinical TOF-MRA dataset. A training set (113 cases with a total of 129 UIAs) was released, each case including a TOF-MRA, a structural MR image (T1, T2 or FLAIR), annotation of any present UIA(s) and the centre voxel of the UIA(s). A test set of 141 cases (with 153 UIAs) was used for evaluation. Two tasks were proposed: (1) detection and (2) segmentation of UIAs on TOF-MRAs. Teams developed and submitted containerised methods to be evaluated on the test set. Task 1 was evaluated using metrics of sensitivity and false positive count. Task 2 was evaluated using dice similarity coefficient, modified hausdorff distance (95th percentile) and volumetric similarity. For each task, a ranking was made based on the average of the metrics. In total, eleven teams participated in task 1 and nine of those teams participated in task 2. Task 1 was won by a method specifically designed for the detection task (i.e. not participating in task 2). Based on segmentation metrics, the top two methods for task 2 performed statistically significantly better than all other methods. The detection performance of the top-ranking methods was comparable to visual inspection for larger aneurysms. Segmentation performance of the top ranking method, after selection of true UIAs, was similar to interobserver performance. The ADAM challenge remains open for future submissions and improved submissions, with a live leaderboard to provide benchmarking for method developments at https://adam.isi.uu.nl/.
2021
challenge; aneurysms; angiography; detection; segmentation
01 Pubblicazione su rivista::01a Articolo in rivista
Comparing methods of detecting and segmenting unruptured intracranial aneurysms on TOF-MRAS: the ADAM challenge / Timmins, Kimberley M; van der Schaaf, Irene C; Bennink, Edwin; Ruigrok, Ynte M; An, Xingle; Baumgartner, Michael; Bourdon, Pascal; De Feo, Riccardo; Noto, Tommaso Di; Dubost, Florian; Fava-Sanches, Augusto; Feng, Xue; Giroud, Corentin; Group, Inteneural; Hu, Minghui; Jaeger, Paul F; Kaiponen, Juhana; Klimont, Michał; Li, Yuexiang; Li, Hongwei; Lin, Yi; Loehr, Timo; Ma, Jun; Maier-Hein, Klaus H; Marie, Guillaume; Menze, Bjoern; Richiardi, Jonas; Rjiba, Saifeddine; Shah, Dhaval; Shit, Suprosanna; Tohka, Jussi; Urruty, Thierry; Walińska, Urszula; Yang, Xiaoping; Yang, Yunqiao; Yin, Yin; Velthuis, Birgitta K; Kuijf, Hugo J. - In: NEUROIMAGE. - ISSN 1053-8119. - 238:(2021), pp. 1-21. [10.1016/j.neuroimage.2021.118216]
File allegati a questo prodotto
File Dimensione Formato  
Timmins_comparing_2021.pdf

accesso aperto

Tipologia: Versione editoriale (versione pubblicata con il layout dell'editore)
Licenza: Creative commons
Dimensione 3.96 MB
Formato Adobe PDF
3.96 MB Adobe PDF

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11573/1550042
Citazioni
  • ???jsp.display-item.citation.pmc??? 0
  • Scopus 29
  • ???jsp.display-item.citation.isi??? 25
social impact