The replication crisis
Ioannidis (2005, p. 1) reported that with current research practices “it is more likely for a research claim to be false than true, suggesting that every study that shows significant effects should be taken with a grain of salt, if not downright scrutinized. It follows, then, that replication studies, i. e., studies aimed at verifying research results by applying the same approach to a new set of data, should be commonplace. Yet, in the so-called “hard sciences” as well as psychology, a replication crisis has been attested. The crisis is that most studies do not successfully replicate previous findings and replication attempts are too few altogether. And though Ioannidis’ statement was printed in a medical journal, we find that this holds for linguistics as well. With the quantitative turn in the discipline – an increase in the use of quantitative research designs since the early 2000s (Kortmann, 2021) – a need for replication studies arose, but it is yet to be addressed: Recently, for experimental linguistics, Kobrock and Roettger (2022) have found that only 1 in 1600 studies is directly replicated by independent researchers.
The lack of replication in linguistics may have different reasons, some of which we discuss in the following. First, there is still little awareness on the matter, a problem that is perhaps gradually being amended, as indicated by the publication of a special issue of Linguistics titled “The replication crisis: Implications for linguistics” (see Sönning & Werner, 2021). Second, some studies may simply elude exact replication: linguistic phenomena can be highly context-sensitive and thus impossible to reproduce in contexts that differ from the original. In addition, current institutional practices and publication strategies inhibit replication: journals will usually prefer original research that reports significant effects, prompting researchers to deliver exactly that, while, at the same time, discouraging agencies from funding different kinds of projects. Finally, there is a deficit in the teaching of quantitative practices and the rationale behind them, which can lead to inadequate study designs and a lack of methodological rigor.
Why student theses should be collaborative research studies
How may these challenges be met? In addressing the potential of student theses, Quintana (2021) has proposed a solution similar to the method that Roettger and Bear-Henney (2018) report to have employed in their teaching: letting university students contribute by way of replication.
It is this exact idea that we strive to develop: Student theses could present a particularly useful resource for replication. First, students as new members of the academic community are not as heavily constrained by institutionalized practices, publication and funding demands as senior members who may feel coerced to work in accordance with current practice. Second, students are usually required to produce a thesis. In doing a replication study for their thesis, they are equipped with a research design template in form of a published study, allowing them to gather crucial research experience by following this example. By the end of their work, a student will have had the chance to submit a thesis that adheres to principles of scientific rigor and that contributes to the respective field in a meaningful way.
Similar approaches have already been implemented: FORRT (Framework for Open and Reproducible Research Training), CREP (Collaborative Replications and Education Project) and the German Reproducibility Network (GRN) are community-led organizations dedicated to teaching open science and reproducible research practices to educators and students alike.
Introducing STReNeL
Our goal at STReNeL (Student Theses Replication Network Linguistics) is to achieve better and more reliable scientific advances and generalizations by creating solutions to the urgent replication crisis. Educating fellow young researchers, we aim to form a collaborative network that tackles issues related to the replication crisis.
In detail, we plan to 1) form a network of students in the language sciences, 2) define criteria and guidelines for student-lead replications, 3) determine studies that warrant replication and collect them in a database, 4) pilot replications to show the field the benefits of collaborative student replication, 5) raise awareness.
For that we need your participation and support. Find us on OSF and join our mailing list.
References
Ioannidis, John P. A. (2005): Why Most Published Research Findings Are False. PLoS Medicine 2(8). e124. doi:10.1371/journal.pmed.0020124.
Kobrock, Kristina & Timo Benjamin Roettger (2022): Assessing the replication landscape in experimental linguistics. Preprint PsyArXiv. doi:10.31234/osf.io/fzngs.
Kortmann, Bernd (2021): Reflecting on the quantitative turn in linguistics. Linguistics 59(5). 1207–1226. doi:10.1515/ling-2019-0046.
Quintana, Daniel S. (2021): Replication studies for undergraduate theses to improve science and education. Nature Human Behaviour 5(9). 1117–1118. doi:10.1038/s41562-021-01192-8.
Roettger, Timo B. & Dinah Baer-Henney (2019): Towards a replication culture in phonetic research: Speech production research in the classroom. Phonological Data and Analysis 1(4). 1–23. doi:10.3765/pda.v1art4.13.
Sönning, Lukas & Valentin Werner (2021): The replication crisis, scientific revolutions, and linguistics. Linguistics 59(5). 1179–1206. doi:10.1515/ling-2019-0045.