Same data, different conclusions: radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left uncons...

Full description

Saved in:

Bibliographic Details
Main Authors:	Schweinsberg, Martin (Author) , Feldman, Michael (Author) , Staub, Nicola (Author) , Akker, Olmo R. van den (Author) , Aert, Robbie C. M. van (Author) , Assen, Marcel A. L. M. van (Author) , Liu, Yang (Author) , Althoff, Tim (Author) , Heer, Jeffrey (Author) , Kale, Alex (Author) , Mohamed, Zainab (Author) , Amireh, Hashem (Author) , Prasad, Vaishali Venkatesh (Author) , Bernstein, Abraham (Author) , Robinson, Emily (Author) , Snellman, Kaisa (Author) , Sommer, S. Amy (Author) , Otner, Sarah M. G. (Author) , Robinson, David (Author) , Madan, Nikhil (Author) , Silberzahn, Raphael (Author) , Goldstein, Pavel (Author) , Tierney, Warren (Author) , Murase, Toshio (Author) , Mandl, Benjamin (Author) , Viganola, Domenico (Author) , Strobl, Carolin (Author) , Schaumans, Catherine B. C. (Author) , Kelchtermans, Stijn (Author) , Naseeb, Chan (Author) , Garrison, S. Mason (Author) , Yarkoni, Tal (Author) , Chan, C. S. Richard (Author) , Adie, Prestone (Author) , Alaburda, Paulius (Author) , Albers, Casper (Author) , Alspaugh, Sara (Author) , Alstott, Jeff (Author) , Nelson, Andrew A. (Author) , Ariño, Eduardo de la Rubia (Author) , Arzi, Adbi (Author) , Bahník, Štěpán (Author) , Baik, Jason (Author) , Balling, Laura Winther (Author) , Banker, Sachin (Author) , Baranger, David A. A. (Author) , Barr, Dale J. (Author) , Barros-Rivera, Brenda (Author) , Bauer, Matt (Author) , Blaise, Enuh (Author) , Boelen, Lisa (Author) , Carbonell, Katerina Bohle (Author) , Briers, Robert A. (Author) , Burkhard, Oliver (Author) , Canela, Miguel-Angel (Author) , Castrillo, Laura (Author) , Catlett, Timothy (Author) , Chen, Olivia (Author) , Clark, Michael (Author) , Cohn, Brent (Author) , Coppock, Alex (Author) , Cugueró-Escofet, Natàlia (Author) , Curran, Paul G. (Author) , Cyrus-Lai, Wilson (Author) , Dai, David (Author) , Dalla Riva, Giulio Valentino (Author) , Danielsson, Henrik (Author) , Russo, Rosária de F. S. Macri (Author) , De Silva, Niko (Author) , Derungs, Curdin (Author) , Dondelinger, Frank (Author) , Duarte de Souza, Carolina (Author) , Dube, B. Tyson (Author) , Dubova, Marina (Author) , Dunn, Ben Mark (Author) , Edelsbrunner, Peter Adriaan (Author) , Finley, Sara (Author) , Fox, Nick (Author) , Gnambs, Timo (Author) , Gong, Yuanyuan (Author) , Grand, Erin (Author) , Greenawalt, Brandon (Author) , Han, Dan (Author) , Hanel, Paul H. P. (Author) , Hong, Antony B. (Author) , Hood, David (Author) , Hsueh, Justin (Author) , Huang, Lilian (Author) , Hui, Kent N. (Author) , Hultman, Keith A. (Author) , Javaid, Azka (Author) , Jiang, Lily Ji (Author) , Jong, Jonathan (Author) , Kamdar, Jash (Author) , Kane, David (Author) , Kappler, Gregor (Author) , Kaszubowski, Erikson (Author) , Kavanagh, Christopher M. (Author) , Khabsa, Madian (Author) , Kleinberg, Bennett (Author) , Kouros, Jens (Author) , Krause, Heather (Author) , Krypotos, Angelos-Miltiadis (Author) , Lavbič, Dejan (Author) , Lee, Rui Ling (Author) , Leffel, Timothy (Author) , Lim, Wei Yang (Author) , Liverani, Silvia (Author) , Loh, Bianca (Author) , Lønsmann, Dorte (Author) , Low, Jia Wei (Author) , Lu, Alton (Author) , MacDonald, Kyle (Author) , Madan, Christopher R. (Author) , Madsen, Lasse Hjorth (Author) , Maimone, Christina (Author) , Mangold, Alexandra (Author) , Marshall, Adrienne (Author) , Matskewich, Helena Ester (Author) , Mavon, Kimia (Author) , McLain, Katherine L. (Author) , McNamara, Amelia A. (Author) , McNeill, Mhairi (Author) , Mertens, Ulf K. (Author) , Miller, David (Author) , Moore, Ben (Author) , Moore, Andrew (Author) , Nantz, Eric (Author) , Nasrullah, Ziauddin (Author) , Nejkovic, Valentina (Author) , Nell, Colleen S. (Author) , Nelson, Andrew Arthur (Author) , Nilsonne, Gustav (Author) , Nolan, Rory (Author) , O'Brien, Christopher E. (Author) , ONeill, Patrick (Author) , O'Shea, Kieran (Author) , Olita, Toto (Author) , Otterbacher, Jahna (Author) , Palsetia, Diana (Author) , Pereira, Bianca (Author) , Pozdniakov, Ivan (Author) , Protzko, John (Author) , Reyt, Jean-Nicolas (Author) , Riddle, Travis (Author) , Ridhwan Omar Ali, Amal (Author) , Ropovik, Ivan (Author) , Rosenberg, Joshua M. (Author) , Rothen, Stephane (Author) , Schulte-Mecklenbeck, Michael (Author) , Sharma, Nirek (Author) , Shotwell, Gordon (Author) , Skarzynski, Martin (Author) , Stedden, William (Author) , Stodden, Victoria (Author) , Stoffel, Martin A. (Author) , Stoltzman, Scott (Author) , Subbaiah, Subashini (Author) , Tatman, Rachael (Author) , Thibodeau, Paul (Author) , Tomkins, Sabina (Author) , Valdivia, Ana (Author) , Woestijne, Gerrieke B Druijj-van de (Author) , Viana, Laura (Author) , Villesèche, Florence (Author) , Wadsworth, W. Duncan (Author) , Wanders, Florian (Author) , Watts, Krista (Author) , Wells, Jason D. (Author) , Whelpley, Christopher E. (Author) , Won, Andy (Author) , Wu, Lawrence (Author) , Yip, Arthur (Author) , Youngflesh, Casey (Author) , Yu, Ju-Chi (Author) , Zandian, Arash (Author) , Zhang, Leilei (Author) , Zibman, Chava (Author) , Uhlmann, Eric Luis (Author)
Format:	Article (Journal)
Language:	English
Published:	17 June 2021
In:	Organizational behavior and human decision processes Year: 2021, Volume: 165, Pages: 228-249
ISSN:	0749-5978
DOI:	10.1016/j.obhdp.2021.02.003
Online Access:	Verlag, lizenzpflichtig, Volltext: https://doi.org/10.1016/j.obhdp.2021.02.003 Verlag, lizenzpflichtig, Volltext: https://www.sciencedirect.com/science/article/pii/S0749597821000200
Author Notes:	Martin Schweinsberg, Michael Feldman, Nicola Staub, Olmo R. van den Akker, Robbie C.M. van Aert, Marcel A.L.M. van Assen, Yang Liu, Tim Althoff, Jeffrey Heer, Alex Kale, Zainab Mohamed, Hashem Amireh, Vaishali Venkatesh Prasad, Abraham Bernstein, Emily Robinson, Kaisa Snellman, S. Amy Sommer, Sarah M.G. Otner, David Robinson, Nikhil Madan, Raphael Silberzahn, Pavel Goldstein, Warren Tierney, Toshio Murase, Benjamin Mandl, Domenico Viganola, Carolin Strobl, Catherine B.C. Schaumans, Stijn Kelchtermans, Chan Naseeb, S. Mason Garrison, Tal Yarkoni, C.S. Richard Chan, Prestone Adie, Paulius Alaburda, Casper Albers, Sara Alspaugh, Jeff Alstott, Andrew A. Nelson, Eduardo Ariño de la Rubia, Adbi Arzi, Štěpán Bahník, Jason Baik, Laura Winther Balling, Sachin Banker, David AA Baranger, Dale J. Barr, Brenda Barros-Rivera, Matt Bauer, Enuh Blaise, Lisa Boelen, Katerina Bohle Carbonell, Robert A. Briers, Oliver Burkhard, Miguel-Angel Canela, Laura Castrillo, Timothy Catlett, Olivia Chen, Michael Clark, Brent Cohn, Alex Coppock, Natàlia Cugueró-Escofet, Paul G. Curran, Wilson Cyrus-Lai, David Dai, Giulio Valentino Dalla Riva, Henrik Danielsson, Rosaria de F.S.M. Russo, Niko de Silva, Curdin Derungs, Frank Dondelinger, Carolina Duarte de Souza, B. Tyson Dube, Marina Dubova, Ben Mark Dunn, Peter Adriaan Edelsbrunner, Sara Finley, Nick Fox, Timo Gnambs, Yuanyuan Gong, Erin Grand, Brandon Greenawalt, Dan Han, Paul H.P. Hanel, Antony B. Hong, David Hood, Justin Hsueh, Lilian Huang, Kent N. Hui, Keith A. Hultman, Azka Javaid, Lily Ji Jiang, Jonathan Jong, Jash Kamdar, David Kane, Gregor Kappler, Erikson Kaszubowski, Christopher M. Kavanagh, Madian Khabsa, Bennett Kleinberg, Jens Kouros, Heather Krause, Angelos-Miltiadis Krypotos, Dejan Lavbič, Rui Ling Lee, Timothy Leffel, Wei Yang Lim, Silvia Liverani, Bianca Loh, Dorte Lønsmann, Jia Wei Low, Alton Lu, Kyle MacDonald, Christopher R. Madan, Lasse Hjorth Madsen, Christina Maimone, Alexandra Mangold, Adrienne Marshall, Helena Ester Matskewich, Kimia Mavon, Katherine L. McLain, Amelia A. McNamara, Mhairi McNeill, Ulf Mertens, David Miller, Ben Moore, Andrew Moore, Eric Nantz, Ziauddin Nasrullah, Valentina Nejkovic, Colleen S Nell, Andrew Arthur Nelson, Gustav Nilsonne, Rory Nolan, Christopher E. O'Brien, Patrick O'Neill, Kieran O'Shea, Toto Olita, Jahna Otterbacher, Diana Palsetia, Bianca Pereira, Ivan Pozdniakov, John Protzko, Jean-Nicolas Reyt, Travis Riddle, Amal (Akmal) Ridhwan Omar Ali, Ivan Ropovik, Joshua M. Rosenberg, Stephane Rothen, Michael Schulte-Mecklenbeck, Nirek Sharma, Gordon Shotwell, Martin Skarzynski, William Stedden, Victoria Stodden, Martin A. Stoffel, Scott Stoltzman, Subashini Subbaiah, Rachael Tatman, Paul H. Thibodeau, Sabina Tomkins, Ana Valdivia, Gerrieke B. Druijff-van de Woestijne, Laura Viana, Florence Villesèche, W. Duncan Wadsworth, Florian Wanders, Krista Watts, Jason D Wells, Christopher E. Whelpley, Andy Won, Lawrence Wu, Arthur Yip, Casey Youngflesh, Ju-Chi Yu, Arash Zandian, Leilei Zhang, Chava Zibman, Eric Luis Uhlmann

Description
Summary:	In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed.
Item Description:	Gesehen am 03.11.2021
Physical Description:	Online Resource
ISSN:	0749-5978
DOI:	10.1016/j.obhdp.2021.02.003

Same data, different conclusions: radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

Similar Items