What's wrong with my benchmark results?: Studying bad practices in JMH benchmarks
Microbenchmarking frameworks, such as Java's Microbenchmark Harness (JMH), allow developers to write fine-grained performance test suites at the method or statement level. However, due to the complexities of the Java Virtual Machine, developers often struggle with writing expressive JMH benchma...
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Dokumenttyp: | Article (Journal) |
| Sprache: | Englisch |
| Veröffentlicht: |
16 July 2021
|
| In: |
IEEE transactions on software engineering
Year: 2021, Jahrgang: 47, Heft: 7, Pages: 1452-1467 |
| ISSN: | 1939-3520 |
| DOI: | 10.1109/TSE.2019.2925345 |
| Online-Zugang: | Verlag, lizenzpflichtig, Volltext: https://doi.org/10.1109/TSE.2019.2925345 Verlag, lizenzpflichtig, Volltext: https://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=DOISource&SrcApp=WOS&KeyAID=10.1109%2FTSE.2019.2925345&DestApp=DOI&SrcAppSID=F4YMMc9jPB5f9dX7ixC&SrcJTitle=IEEE+TRANSACTIONS+ON+SOFTWARE+ENGINEERING&DestDOIRegistrantName=Institute+of+Electrical+and+Electronics+Engineers 
|
| Verfasserangaben: | Diego Costa, Cor-Paul Bezemer, Philipp Leitner, and Artur Andrzejak |
MARC
| LEADER | 00000caa a2200000 c 4500 | ||
|---|---|---|---|
| 001 | 1769678883 | ||
| 003 | DE-627 | ||
| 005 | 20220820041948.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 210908s2021 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1109/TSE.2019.2925345 |2 doi | |
| 035 | |a (DE-627)1769678883 | ||
| 035 | |a (DE-599)KXP1769678883 | ||
| 035 | |a (OCoLC)1341421133 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rda | ||
| 041 | |a eng | ||
| 084 | |a 28 |2 sdnb | ||
| 100 | 1 | |a Damesceno Costa, Diego Elias |d 1991- |e VerfasserIn |0 (DE-588)1194500528 |0 (DE-627)1676516069 |4 aut | |
| 245 | 1 | 0 | |a What's wrong with my benchmark results? |b Studying bad practices in JMH benchmarks |c Diego Costa, Cor-Paul Bezemer, Philipp Leitner, and Artur Andrzejak |
| 264 | 1 | |c 16 July 2021 | |
| 300 | |a 16 | ||
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a Computermedien |b c |2 rdamedia | ||
| 338 | |a Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Gesehen am 08.09.2021 | ||
| 520 | |a Microbenchmarking frameworks, such as Java's Microbenchmark Harness (JMH), allow developers to write fine-grained performance test suites at the method or statement level. However, due to the complexities of the Java Virtual Machine, developers often struggle with writing expressive JMH benchmarks which accurately represent the performance of such methods or statements. In this paper, we empirically study bad practices of JMH benchmarks. We present a tool that leverages static analysis to identify 5 bad JMH practices. Our empirical study of 123 open source Java-based systems shows that each of these 5 bad practices are prevalent in open source software. Further, we conduct several experiments to quantify the impact of each bad practice in multiple case studies, and find that bad practices often significantly impact the benchmark results. To validate our experimental results, we constructed seven patches that fix the identified bad practices for six of the studied open source projects, of which six were merged into the main branch of the project. In this paper, we show that developers struggle with accurate Java microbenchmarking, and provide several recommendations to developers of microbenchmarking frameworks on how to improve future versions of their framework. | ||
| 650 | 4 | |a bad practices | |
| 650 | 4 | |a Benchmark testing | |
| 650 | 4 | |a Java | |
| 650 | 4 | |a jmh | |
| 650 | 4 | |a microbenchmarking | |
| 650 | 4 | |a Optimization | |
| 650 | 4 | |a Performance testing | |
| 650 | 4 | |a static analysis | |
| 650 | 4 | |a Static analysis | |
| 650 | 4 | |a Tools | |
| 650 | 4 | |a Writing | |
| 700 | 1 | |a Bezemer, Cor-Paul |e VerfasserIn |4 aut | |
| 700 | 1 | |a Leitner, Philipp |e VerfasserIn |4 aut | |
| 700 | 1 | |a Andrzejak, Artur |d 1969- |e VerfasserIn |0 (DE-588)122012216 |0 (DE-627)705736253 |0 (DE-576)29304712X |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |a Institute of Electrical and Electronics Engineers |t IEEE transactions on software engineering |d Piscataway, NJ : IEEE Service Center, 1975 |g 47(2021), 7, Seite 1452-1467 |h Online-Ressource |w (DE-627)324452942 |w (DE-600)2026617-0 |w (DE-576)094402078 |x 1939-3520 |7 nnas |
| 773 | 1 | 8 | |g volume:47 |g year:2021 |g number:7 |g pages:1452-1467 |g extent:16 |a What's wrong with my benchmark results? Studying bad practices in JMH benchmarks |
| 856 | 4 | 0 | |u https://doi.org/10.1109/TSE.2019.2925345 |x Verlag |x Resolving-System |z lizenzpflichtig |3 Volltext |
| 856 | 4 | 0 | |u https://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=DOISource&SrcApp=WOS&KeyAID=10.1109%2FTSE.2019.2925345&DestApp=DOI&SrcAppSID=F4YMMc9jPB5f9dX7ixC&SrcJTitle=IEEE+TRANSACTIONS+ON+SOFTWARE+ENGINEERING&DestDOIRegistrantName=Institute+of+Electrical+and+Electronics+Engineers |x Verlag |z lizenzpflichtig |3 Volltext |
| 951 | |a AR | ||
| 992 | |a 20210908 | ||
| 993 | |a Article | ||
| 994 | |a 2021 | ||
| 998 | |g 122012216 |a Andrzejak, Artur |m 122012216:Andrzejak, Artur |d 110000 |d 110300 |e 110000PA122012216 |e 110300PA122012216 |k 0/110000/ |k 1/110000/110300/ |p 4 |y j | ||
| 998 | |g 1194500528 |a Damesceno Costa, Diego Elias |m 1194500528:Damesceno Costa, Diego Elias |p 1 |x j | ||
| 999 | |a KXP-PPN1769678883 |e 3974848827 | ||
| BIB | |a Y | ||
| SER | |a journal | ||
| JSO | |a {"title":[{"title_sort":"What's wrong with my benchmark results?","subtitle":"Studying bad practices in JMH benchmarks","title":"What's wrong with my benchmark results?"}],"person":[{"role":"aut","roleDisplay":"VerfasserIn","display":"Damesceno Costa, Diego Elias","given":"Diego Elias","family":"Damesceno Costa"},{"display":"Bezemer, Cor-Paul","roleDisplay":"VerfasserIn","role":"aut","family":"Bezemer","given":"Cor-Paul"},{"role":"aut","display":"Leitner, Philipp","roleDisplay":"VerfasserIn","given":"Philipp","family":"Leitner"},{"family":"Andrzejak","given":"Artur","display":"Andrzejak, Artur","roleDisplay":"VerfasserIn","role":"aut"}],"note":["Gesehen am 08.09.2021"],"type":{"media":"Online-Ressource","bibl":"article-journal"},"recId":"1769678883","language":["eng"],"origin":[{"dateIssuedDisp":"16 July 2021","dateIssuedKey":"2021"}],"id":{"doi":["10.1109/TSE.2019.2925345"],"eki":["1769678883"]},"name":{"displayForm":["Diego Costa, Cor-Paul Bezemer, Philipp Leitner, and Artur Andrzejak"]},"physDesc":[{"extent":"16 S."}],"relHost":[{"id":{"issn":["1939-3520"],"eki":["324452942"],"zdb":["2026617-0"]},"origin":[{"dateIssuedDisp":"1975-","dateIssuedKey":"1975","publisher":"IEEE Service Center","publisherPlace":"Piscataway, NJ"}],"name":{"displayForm":["Institute of Electrical and Electronics Engineers"]},"physDesc":[{"extent":"Online-Ressource"}],"title":[{"title_sort":"IEEE transactions on software engineering","subtitle":"TSE","title":"IEEE transactions on software engineering"}],"part":{"year":"2021","issue":"7","pages":"1452-1467","volume":"47","text":"47(2021), 7, Seite 1452-1467","extent":"16"},"titleAlt":[{"title":"Transactions on software engineering"},{"title":"TSE"}],"pubHistory":["1.1975 -"],"recId":"324452942","corporate":[{"display":"Institute of Electrical and Electronics Engineers","roleDisplay":"VerfasserIn","role":"aut"}],"language":["eng"],"disp":"Institute of Electrical and Electronics EngineersIEEE transactions on software engineering","note":["Gesehen am 20.07.2023"],"type":{"bibl":"periodical","media":"Online-Ressource"}}]} | ||
| SRT | |a DAMESCENOCWHATSWRONG1620 | ||