Evidence-based policy or beauty contest?: an LLM-based meta-analysis of EU cohesion policy evaluations
Independent and high-quality evaluations of government policies are an important input for designing evidence-based policy. Lack of incentives and institutions to write such evaluations, on the other hand, carry the risk of turning the system into a costly beauty contest. We study one of the most ad...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Book/Monograph Working Paper |
| Language: | English |
| Published: |
Mannheim, Germany
ZEW - Leibniz Centre for European Economic Research
[2024]
|
| Series: | Discussion paper
no. 24, 037 (06/2024) |
| In: |
Discussion paper (no. 24, 037 (06/2024))
|
| Subjects: | |
| Online Access: | Verlag, kostenfrei: https://madoc.bib.uni-mannheim.de/67663 Verlag, kostenfrei: https://ftp.zew.de/pub/zew-docs/dp/dp24037.pdf?_gl=1*1xnt69m*_ga*NjI4NzAxODI3LjE3MjA0NTg2MTk.*_ga_KFD4G5CY27*MTcyMDQ1ODYxOC4xLjEuMTcyMDQ1ODcxOC4wLjAuMA.. Verlag, kostenfrei: https://www.zew.de/publikationen/evidence-based-policy-or-beauty-contest-an-llm-based-meta-analysis-of-eu-cohesion-policy-evaluations Resolving-System, kostenfrei: https://hdl.handle.net/10419/300241 
|
| Author Notes: | Zareh Asatryan, Carlo Birkholz, Friedrich Heinemann |
MARC
| LEADER | 00000cam a2200000 c 4500 | ||
|---|---|---|---|
| 001 | 1894679865 | ||
| 003 | DE-627 | ||
| 005 | 20240822121733.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 240708s2024 gw |||||o 00| ||eng c | ||
| 024 | 7 | |a urn:nbn:de:bsz:180-madoc-676639 |2 urn | |
| 024 | 7 | |a 10419/300241 |2 hdl | |
| 035 | |a (DE-627)1894679865 | ||
| 035 | |a (DE-599)KXP1894679865 | ||
| 035 | |a (OCoLC)1444105595 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rda | ||
| 041 | |a eng | ||
| 044 | |c XA-DE | ||
| 084 | |a 17 |2 sdnb | ||
| 084 | |a A11 |a C45 |a D83 |a H43 |a H54 |2 jelc | ||
| 100 | 1 | |a Asatryan, Zareh |e VerfasserIn |0 (DE-588)1152036319 |0 (DE-627)1013254406 |0 (DE-576)392362821 |4 aut | |
| 245 | 1 | 0 | |a Evidence-based policy or beauty contest? |b an LLM-based meta-analysis of EU cohesion policy evaluations |c Zareh Asatryan, Carlo Birkholz, Friedrich Heinemann |
| 264 | 1 | |a Mannheim, Germany |b ZEW - Leibniz Centre for European Economic Research |c [2024] | |
| 300 | |a 1 Online-Ressource (53 Seiten) |b Illustrationen | ||
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a Computermedien |b c |2 rdamedia | ||
| 338 | |a Online-Ressource |b cr |2 rdacarrier | ||
| 490 | 1 | |a Discussion paper / ZEW |v no. 24, 037 (06/2024) | |
| 520 | |a Independent and high-quality evaluations of government policies are an important input for designing evidence-based policy. Lack of incentives and institutions to write such evaluations, on the other hand, carry the risk of turning the system into a costly beauty contest. We study one of the most advanced markets of policy evaluations in the world, the evaluations of EU Cohesion Policies by its Member States (MS). We use large language models quantify the findings of about 2,300 evaluations, and complement this data with our own survey of the authors. We show that the findings of evaluations are inconsistent with those of the academic literature on the output impacts of Cohesion Policy. Using further variation across MS, our analysis suggests that the market of evaluations is rather oligopolistic within MS, that it is very fragmented across the EU, and that there is often a strong involvement of managing authorities in the work of formally independent evaluators. These factors contribute to making the findings of the evaluations overly optimistic (beautiful) risking their overall usefulness (evidence-based policy). We conclude by discussing reform options to make the evaluations of EU Cohesion Policies more unbiased and effective. | ||
| 650 | 4 | |a Policy Evaluation |7 (dpeaa)DE-206 | |
| 650 | 4 | |a EU Cohesion Policy |7 (dpeaa)DE-206 | |
| 650 | 4 | |a Large Language Model |7 (dpeaa)DE-206 | |
| 655 | 4 | |0 (DE-206)34 |a Graue Literatur |5 DE-206 | |
| 700 | 1 | |a Birkholz, Carlo |d 1992- |e VerfasserIn |0 (DE-588)1232420638 |0 (DE-627)1756560803 |4 aut | |
| 700 | 1 | |a Heinemann, Friedrich |d 1964- |e VerfasserIn |0 (DE-588)131390244 |0 (DE-627)508766591 |0 (DE-576)169084329 |4 aut | |
| 830 | 0 | |a Discussion paper |v no. 24, 037 (06/2024) |9 2024,37 |w (DE-627)1677802278 |w (DE-600)2985404-0 |7 am | |
| 856 | 4 | 0 | |u https://madoc.bib.uni-mannheim.de/67663 |x Verlag |z kostenfrei |
| 856 | 4 | 0 | |u https://ftp.zew.de/pub/zew-docs/dp/dp24037.pdf?_gl=1*1xnt69m*_ga*NjI4NzAxODI3LjE3MjA0NTg2MTk.*_ga_KFD4G5CY27*MTcyMDQ1ODYxOC4xLjEuMTcyMDQ1ODcxOC4wLjAuMA.. |x Verlag |z kostenfrei |
| 856 | 4 | 0 | |u https://www.zew.de/publikationen/evidence-based-policy-or-beauty-contest-an-llm-based-meta-analysis-of-eu-cohesion-policy-evaluations |x Verlag |z kostenfrei |
| 856 | 4 | 0 | |u https://hdl.handle.net/10419/300241 |x Resolving-System |z kostenfrei |
| 951 | |a BO | ||
| 992 | |a 20240822 | ||
| 993 | |a WorkingPaper | ||
| 994 | |a 2024 | ||
| 998 | |g 131390244 |a Heinemann, Friedrich |m 131390244:Heinemann, Friedrich |d 180000 |e 180000PH131390244 |k 0/180000/ |p 3 |y j | ||
| 999 | |a KXP-PPN1894679865 |e 4569839878 | ||
| BIB | |a Y | ||
| JSO | |a {"relMultPart":[{"dispAlt":"Discussion paper","physDesc":[{"extent":"Online-Ressource"}],"id":{"eki":["1677802278"],"zdb":["2985404-0"],"hdl":["10419/17"]},"origin":[{"publisher":"[ZEW]","dateIssuedKey":"2019","dateIssuedDisp":"2019-","publisherPlace":"[Mannheim]"}],"name":{"displayForm":["ZEW"]},"titleAlt":[{"title":"ZEW discussion paper"}],"part":{"number_sort":["2024,37"],"number":["no. 24, 037 (06/2024)"]},"pubHistory":["No. 19,0001 (2019, 01)-"],"recId":"1677802278","language":["eng"],"corporate":[{"roleDisplay":"Herausgebendes Organ","display":"ZEW - Leibniz-Zentrum für Europäische Wirtschaftsforschung","role":"isb"}],"type":{"bibl":"serial","media":"Online-Ressource"},"disp":"Discussion paper / ZEW","title":[{"title_sort":"Discussion paper","title":"Discussion paper"}]}],"physDesc":[{"noteIll":"Illustrationen","extent":"1 Online-Ressource (53 Seiten)"}],"type":{"media":"Online-Ressource","bibl":"book"},"language":["eng"],"recId":"1894679865","origin":[{"publisherPlace":"Mannheim, Germany","dateIssuedDisp":"[2024]","dateIssuedKey":"2024","publisher":"ZEW - Leibniz Centre for European Economic Research"}],"title":[{"title_sort":"Evidence-based policy or beauty contest?","title":"Evidence-based policy or beauty contest?","subtitle":"an LLM-based meta-analysis of EU cohesion policy evaluations"}],"id":{"hdl":["10419/300241"],"uri":["urn:nbn:de:bsz:180-madoc-676639"],"eki":["1894679865"]},"name":{"displayForm":["Zareh Asatryan, Carlo Birkholz, Friedrich Heinemann"]},"person":[{"roleDisplay":"VerfasserIn","display":"Asatryan, Zareh","role":"aut","family":"Asatryan","given":"Zareh"},{"family":"Birkholz","given":"Carlo","roleDisplay":"VerfasserIn","display":"Birkholz, Carlo","role":"aut"},{"given":"Friedrich","family":"Heinemann","role":"aut","display":"Heinemann, Friedrich","roleDisplay":"VerfasserIn"}]} | ||
| SRT | |a ASATRYANZAEVIDENCEBA2024 | ||