Mutually improved endoscopic image synthesis and landmark detection in unpaired image-to-image translation
The CycleGAN framework allows for unsupervised image-to-image translation of unpaired data. In a scenario of surgical training on a physical surgical simulator, this method can be used to transform endoscopic images of phantoms into images which more closely resemble the intra-operative appearance o...
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Dokumenttyp: | Article (Journal) |
| Sprache: | Englisch |
| Veröffentlicht: |
January 2022
|
| In: |
IEEE journal of biomedical and health informatics
Year: 2022, Jahrgang: 26, Heft: 1, Pages: 127-138 |
| ISSN: | 2168-2208 |
| Online-Zugang: |
|
| Verfasserangaben: | Lalith Sharan, Gabriele Romano, Sven Koehler, Halvar Kelm, Matthias Karck, Raffaele De Simone and Sandy Engelhardt |
MARC
| LEADER | 00000caa a2200000 c 4500 | ||
|---|---|---|---|
| 001 | 1798141248 | ||
| 003 | DE-627 | ||
| 005 | 20220820163309.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 220407s2022 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1109/JBHI.2021.3099858 |2 doi | |
| 035 | |a (DE-627)1798141248 | ||
| 035 | |a (DE-599)KXP1798141248 | ||
| 035 | |a (OCoLC)1341458836 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rda | ||
| 041 | |a eng | ||
| 084 | |a 33 |2 sdnb | ||
| 100 | 1 | |a Sharan, Lalith |d 1993- |e VerfasserIn |0 (DE-588)1255069554 |0 (DE-627)1798142104 |4 aut | |
| 245 | 1 | 0 | |a Mutually improved endoscopic image synthesis and landmark detection in unpaired image-to-image translation |c Lalith Sharan, Gabriele Romano, Sven Koehler, Halvar Kelm, Matthias Karck, Raffaele De Simone and Sandy Engelhardt |
| 264 | 1 | |c January 2022 | |
| 300 | |a 11 | ||
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a Computermedien |b c |2 rdamedia | ||
| 338 | |a Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Gesehen am 07.04.2022 | ||
| 520 | |a The CycleGAN framework allows for unsupervised image-to-image translation of unpaired data. In a scenario of surgical training on a physical surgical simulator, this method can be used to transform endoscopic images of phantoms into images which more closely resemble the intra-operative appearance of the same surgical target structure. This can be viewed as a novel augmented reality approach, which we coined Hyperrealism in previous work. In this use case, it is of paramount importance to display objects like needles, sutures or instruments consistent in both domains while altering the style to a more tissue-like appearance. Segmentation of these objects would allow for a direct transfer, however, contouring of these, partly tiny and thin foreground objects is cumbersome and perhaps inaccurate. Instead, we propose to use landmark detection on the points when sutures pass into the tissue. This objective is directly incorporated into a CycleGAN framework by treating the performance of pre-trained detector models as an additional optimization goal. We show that a task defined on these sparse landmark labels improves consistency of synthesis by the generator network in both domains. Comparing a baseline CycleGAN architecture to our proposed extension (DetCycleGAN), mean precision (PPV) improved by $+61.32$, mean sensitivity (TPR) by $+37.91$, and mean $F_1$ score by $+0.4743$. Furthermore, it could be shown that by dataset fusion, generated intra-operative images can be leveraged as additional training data for the detection network itself. | ||
| 650 | 4 | |a CycleGAN | |
| 650 | 4 | |a Generative adversarial networks | |
| 650 | 4 | |a landmark detection | |
| 650 | 4 | |a landmark localization | |
| 650 | 4 | |a Maintenance engineering | |
| 650 | 4 | |a mitral valve repair | |
| 650 | 4 | |a Semantics | |
| 650 | 4 | |a Surgery | |
| 650 | 4 | |a surgical simulation | |
| 650 | 4 | |a surgical training | |
| 650 | 4 | |a Task analysis | |
| 650 | 4 | |a Training | |
| 650 | 4 | |a Valves | |
| 700 | 1 | |a Romano, Gabriele |d 1991- |e VerfasserIn |0 (DE-588)1204102082 |0 (DE-627)168924464X |4 aut | |
| 700 | 1 | |a Köhler, Sven |e VerfasserIn |0 (DE-588)124712830X |0 (DE-627)1780697864 |4 aut | |
| 700 | 1 | |a Kelm, Halvar |e VerfasserIn |0 (DE-588)1255070277 |0 (DE-627)179814350X |4 aut | |
| 700 | 1 | |a Karck, Matthias |d 1961- |e VerfasserIn |0 (DE-588)130833525 |0 (DE-627)505909340 |0 (DE-576)298366169 |4 aut | |
| 700 | 1 | |a De Simone, Raffaele |d 1958- |e VerfasserIn |0 (DE-588)1075518466 |0 (DE-627)833385674 |0 (DE-576)443621667 |4 aut | |
| 700 | 1 | |a Engelhardt, Sandy |d 1987- |e VerfasserIn |0 (DE-588)1122674465 |0 (DE-627)876003080 |0 (DE-576)481436049 |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |a Institute of Electrical and Electronics Engineers |t IEEE journal of biomedical and health informatics |d New York, NY : IEEE, 2013 |g 26(2022), 1, Seite 127-138 |h Online-Ressource |w (DE-627)728472279 |w (DE-600)2687987-6 |w (DE-576)375637281 |x 2168-2208 |7 nnas |
| 773 | 1 | 8 | |g volume:26 |g year:2022 |g number:1 |g pages:127-138 |g extent:11 |a Mutually improved endoscopic image synthesis and landmark detection in unpaired image-to-image translation |
| 951 | |a AR | ||
| 992 | |a 20220407 | ||
| 993 | |a Article | ||
| 994 | |a 2022 | ||
| 998 | |g 1122674465 |a Engelhardt, Sandy |m 1122674465:Engelhardt, Sandy |d 910000 |d 910100 |e 910000PE1122674465 |e 910100PE1122674465 |k 0/910000/ |k 1/910000/910100/ |p 7 |y j | ||
| 998 | |g 1075518466 |a De Simone, Raffaele |m 1075518466:De Simone, Raffaele |d 910000 |d 910200 |d 50000 |e 910000PD1075518466 |e 910200PD1075518466 |e 50000PD1075518466 |k 0/910000/ |k 1/910000/910200/ |k 0/50000/ |p 6 | ||
| 998 | |g 130833525 |a Karck, Matthias |m 130833525:Karck, Matthias |d 910000 |d 910200 |e 910000PK130833525 |e 910200PK130833525 |k 0/910000/ |k 1/910000/910200/ |p 5 | ||
| 998 | |g 1255070277 |a Kelm, Halvar |m 1255070277:Kelm, Halvar |p 4 | ||
| 998 | |g 124712830X |a Köhler, Sven |m 124712830X:Köhler, Sven |d 910000 |d 910100 |e 910000PK124712830X |e 910100PK124712830X |k 0/910000/ |k 1/910000/910100/ |p 3 | ||
| 998 | |g 1204102082 |a Romano, Gabriele |m 1204102082:Romano, Gabriele |d 910000 |d 910200 |e 910000PR1204102082 |e 910200PR1204102082 |k 0/910000/ |k 1/910000/910200/ |p 2 | ||
| 998 | |g 1255069554 |a Sharan, Lalith |m 1255069554:Sharan, Lalith |d 910000 |d 910100 |e 910000PS1255069554 |e 910100PS1255069554 |k 0/910000/ |k 1/910000/910100/ |p 1 |x j | ||
| 999 | |a KXP-PPN1798141248 |e 4113496064 | ||
| BIB | |a Y | ||
| SER | |a journal | ||
| JSO | |a {"physDesc":[{"extent":"11 S."}],"recId":"1798141248","note":["Gesehen am 07.04.2022"],"language":["eng"],"type":{"bibl":"article-journal","media":"Online-Ressource"},"title":[{"title":"Mutually improved endoscopic image synthesis and landmark detection in unpaired image-to-image translation","title_sort":"Mutually improved endoscopic image synthesis and landmark detection in unpaired image-to-image translation"}],"relHost":[{"name":{"displayForm":["Institute of Electrical and Electronics Engineers"]},"pubHistory":["17.2013 -"],"titleAlt":[{"title":"Biomedical and health informatics"}],"corporate":[{"display":"Institute of Electrical and Electronics Engineers","role":"aut"}],"id":{"zdb":["2687987-6"],"doi":["10.1109/JBHI.6221020"],"issn":["2168-2208"],"eki":["728472279"]},"part":{"year":"2022","volume":"26","pages":"127-138","issue":"1","extent":"11","text":"26(2022), 1, Seite 127-138"},"origin":[{"publisherPlace":"New York, NY","dateIssuedKey":"2013","dateIssuedDisp":"2013-","publisher":"IEEE"}],"title":[{"title":"IEEE journal of biomedical and health informatics","title_sort":"IEEE journal of biomedical and health informatics"}],"language":["eng"],"type":{"media":"Online-Ressource","bibl":"periodical"},"note":["Gesehen am 14.03.2023"],"recId":"728472279","physDesc":[{"extent":"Online-Ressource"}],"disp":"Institute of Electrical and Electronics EngineersIEEE journal of biomedical and health informatics"}],"person":[{"given":"Lalith","role":"aut","family":"Sharan","display":"Sharan, Lalith"},{"display":"Romano, Gabriele","family":"Romano","role":"aut","given":"Gabriele"},{"family":"Köhler","role":"aut","given":"Sven","display":"Köhler, Sven"},{"family":"Kelm","given":"Halvar","role":"aut","display":"Kelm, Halvar"},{"display":"Karck, Matthias","family":"Karck","role":"aut","given":"Matthias"},{"given":"Raffaele","role":"aut","family":"De Simone","display":"De Simone, Raffaele"},{"display":"Engelhardt, Sandy","family":"Engelhardt","role":"aut","given":"Sandy"}],"origin":[{"dateIssuedDisp":"January 2022","dateIssuedKey":"2022"}],"id":{"eki":["1798141248"],"doi":["10.1109/JBHI.2021.3099858"]},"name":{"displayForm":["Lalith Sharan, Gabriele Romano, Sven Koehler, Halvar Kelm, Matthias Karck, Raffaele De Simone and Sandy Engelhardt"]}} | ||
| SRT | |a SHARANLALIMUTUALLYIM2022 | ||