Evaluating probabilistic classifiers: the triptych
Probability forecasts for binary outcomes, often referred to as probabilistic classifiers or confidence scores, are ubiquitous in science and society, and methods for evaluating and comparing them are in great demand. We propose and study a triptych of diagnostic graphics focusing on distinct and co...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article (Journal) |
| Language: | English |
| Published: |
July-September 2024
|
| In: |
International journal of forecasting
Year: 2024, Volume: 40, Issue: 3, Pages: 1101-1122 |
| ISSN: | 0169-2070 |
| DOI: | 10.1016/j.ijforecast.2023.09.007 |
| Subjects: | |
| Online Access: | Verlag, kostenfrei: https://www.sciencedirect.com/science/article/pii/S0169207023000997/pdfft?md5=bd26faa9dd0165399770a39be8802f6a&pid=1-s2.0-S0169207023000997-main.pdf Resolving-System, kostenfrei: https://doi.org/10.1016/j.ijforecast.2023.09.007 
|
| Author Notes: | Timo Dimitriadis, Tilmann Gneiting, Alexander I. Jordan, Peter Vogel |
MARC
| LEADER | 00000caa a2200000 c 4500 | ||
|---|---|---|---|
| 001 | 1891212710 | ||
| 003 | DE-627 | ||
| 005 | 20241205142147.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 240613s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1016/j.ijforecast.2023.09.007 |2 doi | |
| 035 | |a (DE-627)1891212710 | ||
| 035 | |a (DE-599)KXP1891212710 | ||
| 035 | |a (OCoLC)1475299550 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rda | ||
| 041 | |a eng | ||
| 084 | |a 17 |2 sdnb | ||
| 100 | 1 | |a Dimitriadis, Timo |e VerfasserIn |0 (DE-588)1230883045 |0 (DE-627)1753224217 |4 aut | |
| 245 | 1 | 0 | |a Evaluating probabilistic classifiers |b the triptych |c Timo Dimitriadis, Tilmann Gneiting, Alexander I. Jordan, Peter Vogel |
| 264 | 1 | |c July-September 2024 | |
| 300 | |b Illustrationen | ||
| 300 | |a 22 | ||
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a Computermedien |b c |2 rdamedia | ||
| 338 | |a Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Online verfügbar: 4. November 2023, Artikelversion: 31. Mai 2024 | ||
| 520 | |a Probability forecasts for binary outcomes, often referred to as probabilistic classifiers or confidence scores, are ubiquitous in science and society, and methods for evaluating and comparing them are in great demand. We propose and study a triptych of diagnostic graphics focusing on distinct and complementary aspects of forecast performance: Reliability curves address calibration, receiver operating characteristic (ROC) curves diagnose discrimination ability, and Murphy curves visualize overall predictive performance and value. A Murphy curve shows a forecast’s mean elementary scores, including the widely used misclassification rate, and the area under a Murphy curve equals the mean Brier score. For a calibrated forecast, the reliability curve lies on the diagonal, and for competing calibrated forecasts, the ROC and Murphy curves share the same number of crossing points. We invoke the recently developed CORP (Consistent, Optimally binned, Reproducible, and Pool-Adjacent-Violators (PAV) algorithm-based) approach to craft reliability curves and decompose a mean score into miscalibration (MCB), discrimination (DSC), and uncertainty (UNC) components. Plots of the DSC measure of discrimination ability versus the calibration metric MCB visualize classifier performance across multiple competitors. The proposed tools are illustrated in empirical examples from astrophysics, economics, and social science. | ||
| 650 | 4 | |a Calibration error |7 (dpeaa)DE-206 | |
| 650 | 4 | |a Economic utility |7 (dpeaa)DE-206 | |
| 650 | 4 | |a Logarithmic score |7 (dpeaa)DE-206 | |
| 650 | 4 | |a MCB-DSC plot |7 (dpeaa)DE-206 | |
| 650 | 4 | |a Misclassification loss |7 (dpeaa)DE-206 | |
| 650 | 4 | |a Proper scoring rule |7 (dpeaa)DE-206 | |
| 650 | 4 | |a Score decomposition |7 (dpeaa)DE-206 | |
| 650 | 4 | |a Sharpness principle |7 (dpeaa)DE-206 | |
| 655 | 4 | |0 (DE-206)49 |a Aufsatz in Zeitschrift |5 DE-206 | |
| 700 | 1 | |a Gneiting, Tilmann |e VerfasserIn |0 (DE-588)1019627484 |0 (DE-627)690974809 |0 (DE-576)358470323 |4 aut | |
| 700 | 1 | |a Jordan, Alexander I. |e VerfasserIn |0 (DE-588)1027203264 |0 (DE-627)72857747X |0 (DE-576)372589928 |4 aut | |
| 700 | 1 | |a Vogel, Peter |d 198X- |e VerfasserIn |0 (DE-588)1179421345 |0 (DE-627)1067262806 |0 (DE-576)518237478 |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t International journal of forecasting |d Amsterdam [u.a.] : Elsevier Science, 1985 |g 40(2024), 3 vom: Juli/Sept., Seite 1101-1122 |h Online-Ressource |w (DE-627)306313154 |w (DE-600)1495951-3 |w (DE-576)080987435 |x 0169-2070 |7 nnas |a Evaluating probabilistic classifiers the triptych |
| 773 | 1 | 8 | |g volume:40 |g year:2024 |g number:3 |g month:07/09 |g pages:1101-1122 |g extent:22 |a Evaluating probabilistic classifiers the triptych |
| 856 | 4 | 0 | |u https://www.sciencedirect.com/science/article/pii/S0169207023000997/pdfft?md5=bd26faa9dd0165399770a39be8802f6a&pid=1-s2.0-S0169207023000997-main.pdf |x Verlag |z kostenfrei |
| 856 | 4 | 0 | |u https://doi.org/10.1016/j.ijforecast.2023.09.007 |x Resolving-System |z kostenfrei |
| 951 | |a AR | ||
| 992 | |a 20241122 | ||
| 993 | |a Article | ||
| 994 | |a 2024 | ||
| 998 | |g 1230883045 |a Dimitriadis, Timo |m 1230883045:Dimitriadis, Timo |d 180000 |d 181000 |e 180000PD1230883045 |e 181000PD1230883045 |k 0/180000/ |k 1/180000/181000/ |p 1 |x j | ||
| 999 | |a KXP-PPN1891212710 |e 4621131478 | ||
| BIB | |a Y | ||
| SER | |a journal | ||
| JSO | |a {"relHost":[{"title":[{"title_sort":"International journal of forecasting","title":"International journal of forecasting"}],"note":["Gesehen am 29.05.2020"],"disp":"Evaluating probabilistic classifiers the triptychInternational journal of forecasting","type":{"media":"Online-Ressource","bibl":"periodical"},"language":["eng"],"recId":"306313154","pubHistory":["1.1985 -"],"part":{"year":"2024","issue":"3","pages":"1101-1122","text":"40(2024), 3 vom: Juli/Sept., Seite 1101-1122","volume":"40","extent":"22"},"origin":[{"publisher":"Elsevier Science","dateIssuedKey":"1985","dateIssuedDisp":"1985-","publisherPlace":"Amsterdam [u.a.]"}],"id":{"issn":["0169-2070"],"zdb":["1495951-3"],"eki":["306313154"]},"physDesc":[{"extent":"Online-Ressource"}]}],"physDesc":[{"extent":"22 S.","noteIll":"Illustrationen"}],"name":{"displayForm":["Timo Dimitriadis, Tilmann Gneiting, Alexander I. Jordan, Peter Vogel"]},"id":{"doi":["10.1016/j.ijforecast.2023.09.007"],"eki":["1891212710"]},"origin":[{"dateIssuedKey":"2024","dateIssuedDisp":"July-September 2024"}],"language":["eng"],"recId":"1891212710","type":{"bibl":"article-journal","media":"Online-Ressource"},"note":["Online verfügbar: 4. November 2023, Artikelversion: 31. Mai 2024"],"person":[{"family":"Dimitriadis","given":"Timo","roleDisplay":"VerfasserIn","display":"Dimitriadis, Timo","role":"aut"},{"role":"aut","roleDisplay":"VerfasserIn","display":"Gneiting, Tilmann","given":"Tilmann","family":"Gneiting"},{"display":"Jordan, Alexander I.","roleDisplay":"VerfasserIn","role":"aut","family":"Jordan","given":"Alexander I."},{"display":"Vogel, Peter","roleDisplay":"VerfasserIn","role":"aut","family":"Vogel","given":"Peter"}],"title":[{"title":"Evaluating probabilistic classifiers","subtitle":"the triptych","title_sort":"Evaluating probabilistic classifiers"}]} | ||
| SRT | |a DIMITRIADIEVALUATING2024 | ||