Beyond rankings: learning (more) from algorithm validation
Challenges have become the state-of-the-art approach to benchmark image analysis algorithms in a comparative manner. While the validation on identical data sets was a great step forward, results analysis is often restricted to pure ranking tables, leaving relevant questions unanswered. Specifically,...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article (Journal) |
| Language: | English |
| Published: |
23 March 2023
|
| In: |
Medical image analysis
Year: 2023, Volume: 86, Pages: 1-12 |
| ISSN: | 1361-8423 |
| DOI: | 10.1016/j.media.2023.102765 |
| Online Access: | Verlag, lizenzpflichtig, Volltext: https://doi.org/10.1016/j.media.2023.102765 Verlag, lizenzpflichtig, Volltext: https://www.sciencedirect.com/science/article/pii/S1361841523000269 
|
| Author Notes: | Tobias Roß, Pierangela Bruno, Annika Reinke, Manuel Wiesenfarth, Lisa Koeppel, Peter M. Full, Bünyamin Pekdemir, Patrick Godau, Darya Trofimova, Fabian Isensee, Tim J. Adler, Thuy N. Tran, Sara Moccia, Francesco Calimeri, Beat P. Müller-Stich, Annette Kopp-Schneider, Lena Maier-Hein |
MARC
| LEADER | 00000caa a2200000 c 4500 | ||
|---|---|---|---|
| 001 | 1850719098 | ||
| 003 | DE-627 | ||
| 005 | 20250901171110.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 230621s2023 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1016/j.media.2023.102765 |2 doi | |
| 035 | |a (DE-627)1850719098 | ||
| 035 | |a (DE-599)KXP1850719098 | ||
| 035 | |a (OCoLC)1389528154 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rda | ||
| 041 | |a eng | ||
| 084 | |a 33 |2 sdnb | ||
| 100 | 1 | |a Roß, Tobias |d 1990- |e VerfasserIn |0 (DE-588)118364955X |0 (DE-627)1663271143 |4 aut | |
| 245 | 1 | 0 | |a Beyond rankings |b learning (more) from algorithm validation |c Tobias Roß, Pierangela Bruno, Annika Reinke, Manuel Wiesenfarth, Lisa Koeppel, Peter M. Full, Bünyamin Pekdemir, Patrick Godau, Darya Trofimova, Fabian Isensee, Tim J. Adler, Thuy N. Tran, Sara Moccia, Francesco Calimeri, Beat P. Müller-Stich, Annette Kopp-Schneider, Lena Maier-Hein |
| 264 | 1 | |c 23 March 2023 | |
| 300 | |a 12 | ||
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a Computermedien |b c |2 rdamedia | ||
| 338 | |a Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Online verfügbar 1. März 2023, Artikelversion 23. März 2023 | ||
| 500 | |a Gesehen am 21.06.2023 | ||
| 520 | |a Challenges have become the state-of-the-art approach to benchmark image analysis algorithms in a comparative manner. While the validation on identical data sets was a great step forward, results analysis is often restricted to pure ranking tables, leaving relevant questions unanswered. Specifically, little effort has been put into the systematic investigation on what characterizes images in which state-of-the-art algorithms fail. To address this gap in the literature, we (1) present a statistical framework for learning from challenges and (2) instantiate it for the specific task of instrument instance segmentation in laparoscopic videos. Our framework relies on the semantic meta data annotation of images, which serves as foundation for a General Linear Mixed Models (GLMM) analysis. Based on 51,542 meta data annotations performed on 2,728 images, we applied our approach to the results of the Robust Medical Instrument Segmentation Challenge (ROBUST-MIS) challenge 2019 and revealed underexposure, motion and occlusion of instruments as well as the presence of smoke or other objects in the background as major sources of algorithm failure. Our subsequent method development, tailored to the specific remaining issues, yielded a deep learning model with state-of-the-art overall performance and specific strengths in the processing of images in which previous methods tended to fail. Due to the objectivity and generic applicability of our approach, it could become a valuable tool for validation in the field of medical image analysis and beyond. | ||
| 650 | 4 | |a Artificial intelligence | |
| 650 | 4 | |a Biomedical image analysis challenges | |
| 650 | 4 | |a Deep learning | |
| 650 | 4 | |a Endoscopic vision | |
| 650 | 4 | |a Generalized linear mixed models | |
| 650 | 4 | |a Grand challenges | |
| 650 | 4 | |a Image characteristics driven algorithm development | |
| 650 | 4 | |a Instrument segmentation | |
| 650 | 4 | |a Minimally invasive surgery | |
| 650 | 4 | |a Surgical data science | |
| 700 | 1 | |a Bruno, Pierangela |e VerfasserIn |4 aut | |
| 700 | 1 | |a Reinke, Annika |d 1993- |e VerfasserIn |0 (DE-588)1219684252 |0 (DE-627)1735662143 |4 aut | |
| 700 | 1 | |a Wiesenfarth, Manuel |e VerfasserIn |4 aut | |
| 700 | 1 | |a Köppel, Lisa |e VerfasserIn |0 (DE-588)1222994089 |0 (DE-627)1742265537 |4 aut | |
| 700 | 1 | |a Full, Peter M. |e VerfasserIn |0 (DE-588)1219695777 |0 (DE-627)1735698784 |4 aut | |
| 700 | 1 | |a Pekdemir, Bünyamin |e VerfasserIn |4 aut | |
| 700 | 1 | |a Godau, Patrick |d 1993- |e VerfasserIn |0 (DE-588)1179282515 |0 (DE-627)1066788200 |0 (DE-576)518174212 |4 aut | |
| 700 | 1 | |a Trofimova, Darya |e VerfasserIn |0 (DE-588)1062761766 |0 (DE-627)806884908 |0 (DE-576)420194851 |4 aut | |
| 700 | 1 | |a Isensee, Fabian |d 1990- |e VerfasserIn |0 (DE-588)1207568430 |0 (DE-627)1694044998 |4 aut | |
| 700 | 1 | |a Adler, Tim |d 1991- |e VerfasserIn |0 (DE-588)1194987672 |0 (DE-627)1677181680 |4 aut | |
| 700 | 1 | |a Tran, Thuy |e VerfasserIn |0 (DE-588)1293398039 |0 (DE-627)1850720029 |4 aut | |
| 700 | 1 | |a Moccia, Sara |e VerfasserIn |4 aut | |
| 700 | 1 | |a Calimeri, Francesco |e VerfasserIn |4 aut | |
| 700 | 1 | |a Müller, Beat P. |d 1971- |e VerfasserIn |0 (DE-588)14066209X |0 (DE-627)70374819X |0 (DE-576)317992287 |4 aut | |
| 700 | 1 | |a Kopp-Schneider, Annette |e VerfasserIn |0 (DE-588)1119160545 |0 (DE-627)872460444 |0 (DE-576)178153206 |4 aut | |
| 700 | 1 | |a Maier-Hein, Lena |d 1980- |e VerfasserIn |0 (DE-588)1075029252 |0 (DE-627)832869899 |0 (DE-576)190090804 |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Medical image analysis |d Amsterdam [u.a.] : Elsevier Science, 1996 |g 86(2023) vom: März, Artikel-ID 102765, Seite 1-12 |h Online-Ressource |w (DE-627)306365081 |w (DE-600)1497450-2 |w (DE-576)091204941 |x 1361-8423 |7 nnas |a Beyond rankings learning (more) from algorithm validation |
| 773 | 1 | 8 | |g volume:86 |g year:2023 |g month:03 |g elocationid:102765 |g pages:1-12 |g extent:12 |a Beyond rankings learning (more) from algorithm validation |
| 856 | 4 | 0 | |u https://doi.org/10.1016/j.media.2023.102765 |x Verlag |x Resolving-System |z lizenzpflichtig |3 Volltext |
| 856 | 4 | 0 | |u https://www.sciencedirect.com/science/article/pii/S1361841523000269 |x Verlag |z lizenzpflichtig |3 Volltext |
| 951 | |a AR | ||
| 992 | |a 20230621 | ||
| 993 | |a Article | ||
| 994 | |a 2023 | ||
| 998 | |g 1075029252 |a Maier-Hein, Lena |m 1075029252:Maier-Hein, Lena |d 110000 |e 110000PM1075029252 |k 0/110000/ |p 17 |y j | ||
| 998 | |g 1119160545 |a Kopp-Schneider, Annette |m 1119160545:Kopp-Schneider, Annette |d 50000 |e 50000PK1119160545 |k 0/50000/ |p 16 | ||
| 998 | |g 14066209X |a Müller, Beat P. |m 14066209X:Müller, Beat P. |d 50000 |e 50000PM14066209X |k 0/50000/ |p 15 | ||
| 998 | |g 1293398039 |a Tran, Thuy |m 1293398039:Tran, Thuy |d 110000 |e 110000PT1293398039 |k 0/110000/ |p 12 | ||
| 998 | |g 1194987672 |a Adler, Tim |m 1194987672:Adler, Tim |p 11 | ||
| 998 | |g 1207568430 |a Isensee, Fabian |m 1207568430:Isensee, Fabian |p 10 | ||
| 998 | |g 1062761766 |a Trofimova, Darya |m 1062761766:Trofimova, Darya |p 9 | ||
| 998 | |g 1179282515 |a Godau, Patrick |m 1179282515:Godau, Patrick |d 110000 |e 110000PG1179282515 |k 0/110000/ |p 8 | ||
| 998 | |g 1219695777 |a Full, Peter M. |m 1219695777:Full, Peter M. |d 50000 |e 50000PF1219695777 |k 0/50000/ |p 6 | ||
| 998 | |g 1222994089 |a Köppel, Lisa |m 1222994089:Köppel, Lisa |d 910000 |d 911700 |e 910000PK1222994089 |e 911700PK1222994089 |k 0/910000/ |k 1/910000/911700/ |p 5 | ||
| 998 | |g 1219684252 |a Reinke, Annika |m 1219684252:Reinke, Annika |d 50000 |e 50000PR1219684252 |k 0/50000/ |p 3 | ||
| 999 | |a KXP-PPN1850719098 |e 4341189123 | ||
| BIB | |a Y | ||
| SER | |a journal | ||
| JSO | |a {"recId":"1850719098","id":{"doi":["10.1016/j.media.2023.102765"],"eki":["1850719098"]},"title":[{"subtitle":"learning (more) from algorithm validation","title":"Beyond rankings","title_sort":"Beyond rankings"}],"note":["Online verfügbar 1. März 2023, Artikelversion 23. März 2023","Gesehen am 21.06.2023"],"person":[{"family":"Roß","roleDisplay":"VerfasserIn","display":"Roß, Tobias","role":"aut","given":"Tobias"},{"family":"Bruno","roleDisplay":"VerfasserIn","given":"Pierangela","role":"aut","display":"Bruno, Pierangela"},{"display":"Reinke, Annika","role":"aut","given":"Annika","family":"Reinke","roleDisplay":"VerfasserIn"},{"roleDisplay":"VerfasserIn","family":"Wiesenfarth","role":"aut","display":"Wiesenfarth, Manuel","given":"Manuel"},{"family":"Köppel","roleDisplay":"VerfasserIn","role":"aut","display":"Köppel, Lisa","given":"Lisa"},{"role":"aut","display":"Full, Peter M.","given":"Peter M.","roleDisplay":"VerfasserIn","family":"Full"},{"roleDisplay":"VerfasserIn","family":"Pekdemir","display":"Pekdemir, Bünyamin","role":"aut","given":"Bünyamin"},{"display":"Godau, Patrick","role":"aut","given":"Patrick","roleDisplay":"VerfasserIn","family":"Godau"},{"given":"Darya","role":"aut","display":"Trofimova, Darya","roleDisplay":"VerfasserIn","family":"Trofimova"},{"roleDisplay":"VerfasserIn","family":"Isensee","given":"Fabian","role":"aut","display":"Isensee, Fabian"},{"role":"aut","display":"Adler, Tim","given":"Tim","family":"Adler","roleDisplay":"VerfasserIn"},{"roleDisplay":"VerfasserIn","family":"Tran","given":"Thuy","display":"Tran, Thuy","role":"aut"},{"family":"Moccia","roleDisplay":"VerfasserIn","given":"Sara","role":"aut","display":"Moccia, Sara"},{"given":"Francesco","display":"Calimeri, Francesco","role":"aut","family":"Calimeri","roleDisplay":"VerfasserIn"},{"display":"Müller, Beat P.","role":"aut","given":"Beat P.","family":"Müller","roleDisplay":"VerfasserIn"},{"family":"Kopp-Schneider","roleDisplay":"VerfasserIn","given":"Annette","role":"aut","display":"Kopp-Schneider, Annette"},{"given":"Lena","display":"Maier-Hein, Lena","role":"aut","family":"Maier-Hein","roleDisplay":"VerfasserIn"}],"name":{"displayForm":["Tobias Roß, Pierangela Bruno, Annika Reinke, Manuel Wiesenfarth, Lisa Koeppel, Peter M. Full, Bünyamin Pekdemir, Patrick Godau, Darya Trofimova, Fabian Isensee, Tim J. Adler, Thuy N. Tran, Sara Moccia, Francesco Calimeri, Beat P. Müller-Stich, Annette Kopp-Schneider, Lena Maier-Hein"]},"physDesc":[{"extent":"12 S."}],"relHost":[{"recId":"306365081","titleAlt":[{"title":"Medical image analysis online"}],"pubHistory":["1.1996/97 -"],"language":["eng"],"origin":[{"publisher":"Elsevier Science","dateIssuedKey":"1996","publisherPlace":"Amsterdam [u.a.]","dateIssuedDisp":"1996-"}],"type":{"bibl":"periodical","media":"Online-Ressource"},"part":{"year":"2023","extent":"12","pages":"1-12","text":"86(2023) vom: März, Artikel-ID 102765, Seite 1-12","volume":"86"},"physDesc":[{"extent":"Online-Ressource"}],"title":[{"title":"Medical image analysis","title_sort":"Medical image analysis"}],"id":{"issn":["1361-8423"],"zdb":["1497450-2"],"eki":["306365081"]},"disp":"Beyond rankings learning (more) from algorithm validationMedical image analysis","note":["Gesehen am 16.05.23"]}],"type":{"bibl":"article-journal","media":"Online-Ressource"},"origin":[{"dateIssuedKey":"2023","dateIssuedDisp":"23 March 2023"}],"language":["eng"]} | ||
| SRT | |a ROSSTOBIASBEYONDRANK2320 | ||