Reconciling Transparency and Theoretical Coherence: An Argumentation Framework to Design of Follow-Up Studies

Dr. Kenzo Nera (FRS-FNRS, ULB, Belgium)

Abstract: We contend that an overlooked cause of the tendency not to report null results – namely, the “file drawer” problem – lies in the difficulty of making coherent arguments out of mixed and non-significant results. We first describe a “gold prospecting mindset” in the design of follow-up studies. While not fundamentally incompatible with transparency, this mindset encourages the reopening of the file-drawer because it prioritizes the search for significant results over the theoretical coherence of a research program. We then argue that confirmation bias in researchers’ evaluation of their own methods fosters the perception that significant results are more informative than nonsignificant ones – even when the data can be interpreted as supporting the null hypothesis. Against this view, we propose that both significant and non-significant findings can be informative within the domain of validity set by the conducted studies. We outline an argumentation framework for designing follow-up studies so that the results will be theoretically informative, regardless of statistical significance. We identify two follow-up goals requiring distinct methodological adjustments: strengthening a conclusion within its domain of validity, and delineating said domain. In this perspective, researchers should aim not only to discover effects, but specify contexts in which their hypotheses may, or may not, apply. This framework may help reconcile the transparent reporting of null and mixed results with theoretical coherence.

The seminar will take place in the CeSCuP seminar room, as well as online via this link.