Stable reliability diagrams for probabilistic classifiers
A probability forecast or probabilistic classifier is reliable or calibrated if the predicted probabilities are matched by ex post observed frequencies, as examined visually in reliability diagrams. The classical binning and counting approach to plotting reliability diagrams has been hampered by a l...
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Dokumenttyp: | Article (Journal) |
| Sprache: | Englisch |
| Veröffentlicht: |
February 17, 2021
|
| In: |
Proceedings of the National Academy of Sciences of the United States of America
Year: 2021, Jahrgang: 118, Heft: 8, Pages: 1-10 |
| ISSN: | 1091-6490 |
| DOI: | 10.1073/pnas.2016191118 |
| Online-Zugang: | Verlag, lizenzpflichtig, Volltext: https://doi.org/10.1073/pnas.2016191118 Verlag, lizenzpflichtig, Volltext: https://www.pnas.org/content/118/8/e2016191118 
|
| Verfasserangaben: | Timo Dimitriadis, Tilmann Gneiting, and Alexander I. Jordan |
MARC
| LEADER | 00000caa a2200000 c 4500 | ||
|---|---|---|---|
| 001 | 1753154731 | ||
| 003 | DE-627 | ||
| 005 | 20220819154241.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 210406s2021 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1073/pnas.2016191118 |2 doi | |
| 035 | |a (DE-627)1753154731 | ||
| 035 | |a (DE-599)KXP1753154731 | ||
| 035 | |a (OCoLC)1341403386 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rda | ||
| 041 | |a eng | ||
| 084 | |a 17 |2 sdnb | ||
| 100 | 1 | |a Dimitriadis, Timo |e VerfasserIn |0 (DE-588)1230883045 |0 (DE-627)1753224217 |4 aut | |
| 245 | 1 | 0 | |a Stable reliability diagrams for probabilistic classifiers |c Timo Dimitriadis, Tilmann Gneiting, and Alexander I. Jordan |
| 264 | 1 | |c February 17, 2021 | |
| 300 | |a 10 | ||
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a Computermedien |b c |2 rdamedia | ||
| 338 | |a Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Gesehen am 07.04.2021 | ||
| 520 | |a A probability forecast or probabilistic classifier is reliable or calibrated if the predicted probabilities are matched by ex post observed frequencies, as examined visually in reliability diagrams. The classical binning and counting approach to plotting reliability diagrams has been hampered by a lack of stability under unavoidable, ad hoc implementation decisions. Here, we introduce the CORP approach, which generates provably statistically consistent, optimally binned, and reproducible reliability diagrams in an automated way. CORP is based on nonparametric isotonic regression and implemented via the pool-adjacent-violators (PAV) algorithm—essentially, the CORP reliability diagram shows the graph of the PAV-(re)calibrated forecast probabilities. The CORP approach allows for uncertainty quantification via either resampling techniques or asymptotic theory, furnishes a numerical measure of miscalibration, and provides a CORP-based Brier-score decomposition that generalizes to any proper scoring rule. We anticipate that judicious uses of the PAV algorithm yield improved tools for diagnostics and inference for a very wide range of statistical and machine learning methods. | ||
| 650 | 4 | |a calibration | |
| 650 | 4 | |a discrimination ability | |
| 650 | 4 | |a probability forecast | |
| 650 | 4 | |a score decomposition | |
| 650 | 4 | |a weather prediction | |
| 700 | 1 | |a Gneiting, Tilmann |e VerfasserIn |0 (DE-588)1019627484 |0 (DE-627)690974809 |0 (DE-576)358470323 |4 aut | |
| 700 | 1 | |a Jordan, Alexander I. |e VerfasserIn |0 (DE-588)1027203264 |0 (DE-627)72857747X |0 (DE-576)372589928 |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |a National Academy of Sciences (Washington, DC) |t Proceedings of the National Academy of Sciences of the United States of America |d Washington, DC : National Acad. of Sciences, 1915 |g 118(2021), 8, Artikel-ID e2016191118, Seite 1-10 |h Online-Ressource |w (DE-627)254235379 |w (DE-600)1461794-8 |w (DE-576)073260509 |x 1091-6490 |7 nnas |
| 773 | 1 | 8 | |g volume:118 |g year:2021 |g number:8 |g elocationid:e2016191118 |g pages:1-10 |g extent:10 |a Stable reliability diagrams for probabilistic classifiers |
| 856 | 4 | 0 | |u https://doi.org/10.1073/pnas.2016191118 |x Verlag |x Resolving-System |z lizenzpflichtig |3 Volltext |
| 856 | 4 | 0 | |u https://www.pnas.org/content/118/8/e2016191118 |x Verlag |z lizenzpflichtig |3 Volltext |
| 951 | |a AR | ||
| 992 | |a 20210406 | ||
| 993 | |a Article | ||
| 994 | |a 2021 | ||
| 998 | |g 1027203264 |a Jordan, Alexander |m 1027203264:Jordan, Alexander |p 3 |y j | ||
| 998 | |g 1019627484 |a Gneiting, Tilmann |m 1019627484:Gneiting, Tilmann |p 2 | ||
| 998 | |g 1230883045 |a Dimitriadis, Timo |m 1230883045:Dimitriadis, Timo |d 180000 |d 181000 |e 180000PD1230883045 |e 181000PD1230883045 |k 0/180000/ |k 1/180000/181000/ |p 1 |x j | ||
| 999 | |a KXP-PPN1753154731 |e 3902111550 | ||
| BIB | |a Y | ||
| SER | |a journal | ||
| JSO | |a {"physDesc":[{"extent":"10 S."}],"relHost":[{"part":{"extent":"10","text":"118(2021), 8, Artikel-ID e2016191118, Seite 1-10","volume":"118","issue":"8","pages":"1-10","year":"2021"},"titleAlt":[{"title":"PNAS online"},{"title":"PNAS"}],"pubHistory":["1.1915 -"],"recId":"254235379","corporate":[{"roleDisplay":"VerfasserIn","display":"National Academy of Sciences","role":"aut"}],"language":["eng"],"disp":"National Academy of Sciences (Washington, DC)Proceedings of the National Academy of Sciences of the United States of America","note":["Gesehen am 09.03.2022"],"type":{"media":"Online-Ressource","bibl":"periodical"},"title":[{"title":"Proceedings of the National Academy of Sciences of the United States of America","title_sort":"Proceedings of the National Academy of Sciences of the United States of America"}],"physDesc":[{"extent":"Online-Ressource"}],"id":{"issn":["1091-6490"],"eki":["254235379"],"zdb":["1461794-8"]},"origin":[{"publisherPlace":"Washington, DC","publisher":"National Acad. of Sciences","dateIssuedKey":"1915","dateIssuedDisp":"1915-"}]}],"origin":[{"dateIssuedDisp":"February 17, 2021","dateIssuedKey":"2021"}],"id":{"doi":["10.1073/pnas.2016191118"],"eki":["1753154731"]},"name":{"displayForm":["Timo Dimitriadis, Tilmann Gneiting, and Alexander I. Jordan"]},"note":["Gesehen am 07.04.2021"],"type":{"media":"Online-Ressource","bibl":"article-journal"},"language":["eng"],"recId":"1753154731","title":[{"title_sort":"Stable reliability diagrams for probabilistic classifiers","title":"Stable reliability diagrams for probabilistic classifiers"}],"person":[{"family":"Dimitriadis","given":"Timo","roleDisplay":"VerfasserIn","display":"Dimitriadis, Timo","role":"aut"},{"role":"aut","roleDisplay":"VerfasserIn","display":"Gneiting, Tilmann","given":"Tilmann","family":"Gneiting"},{"role":"aut","display":"Jordan, Alexander I.","roleDisplay":"VerfasserIn","given":"Alexander I.","family":"Jordan"}]} | ||
| SRT | |a DIMITRIADISTABLERELI1720 | ||