Open Science Fundamentals
PNB 3EE3

Maya Flannery

2024-01-22

Overview

Objectives

  • What is open science?
  • Why care about open science?
  • How do you do open science?
  • Connection to this course!

Background: The Replication Crisis

What is replication?

  • “[A] study for which any outcome would be considered diagnostic evidence about a claim from prior research.”[1] (p. 1)

Why is replication important?

  • This is how we establish credibility of scientific claims

The Replication Crisis

[2]
  • Evidence for extraordinary claims!

Issue

  • ESP is real or…
  • Something is wrong with the way we do science!

[5]
  • Psychologists’ methods become under extreme critique

Massive replication studies

[6]
  • Many studies do not replicate!
  • In some fields, less than 50%

How can this happen?

Problems: Questionable Practices (QRPs and QMPs)

Questionable research practices [7]

Questionable measurement practices [8]

Failing to report

  • all dependent measures
  • all conditions

Probability of rejecting null hypothesis increases with the number of tests.

[9]

Selective reporting

Collecting data

  • After seeing whether results are significant
  • Stopping after achieving desired result
  • Excluding data after seeing impact of doing do

See p-hacking demo

Analysis

  • Rounding down p-values
  • Falsifying data
  • Claim results are unaffected by demographics
    • Western Educated Industrialized Rich Democratic (WEIRD) populations

Claim to have predicted an unexpected finding

HARKing [10]

Why do researchers do this?

  • Careers depend on ability to “publish or perish”
    • It is very difficult to publish null results

Solution: Open Science

[11]

[11]

FOSTER Open Science

Open Data

  • Promotes reproducibility
  • Saves resources
  • Must be prepared carefully

Output from JSPsych

Open Reproducible Research

  • Data stored, organized, formatted appropriately
  • Code is clean, readable, well-documented
  • Instructions for reproduction

Open Science Tools

Open Access

Incentives

[11]

More Things to Think About

Metametascience

[12]

Knowledge?

[13]

University of Oxford

Open Science in PNB 3EE3

Reproducible Research and Open Science Tools

Open repositories

Open source/notebooks

What about p-hacking?

Preregistration

OSF Prereg Template

Assignment

References

[1]
Nosek BA, Errington TM. What is replication? PLoS Biol 2020;18:e3000691. https://doi.org/10.1371/journal.pbio.3000691.
[2]
Bem DJ. Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology 2011;100:407–25. https://doi.org/10.1037/a0021524.
[3]
Lakens D. The 20% Statistician: A pre-publication peer-review of the “Feeling The Future” meta-analysis. The 20% Statistician 2014. http://daniellakens.blogspot.com/2014/05/a-pre-publication-peer-review-of-meta.html (accessed September 16, 2023).
[4]
Wagenmakers E-J. Open Science Collaboration Blog · Bem is Back: A Skeptic’s Review of a Meta-Analysis on Psi. Open Science Collaboration 2014. http://osc.centerforopenscience.org/2014/06/25/a-skeptics-review/ (accessed September 16, 2023).
[5]
Wagenmakers E-J, Wetzels R, Borsboom D, Van Der Maas HLJ. Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology 2011;100:426–32. https://doi.org/10.1037/a0022790.
[6]
Klein RA, Vianello M, Hasselman F, Adams BG, Adams RB, Alper S, et al. Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in Methods and Practices in Psychological Science 2018;1:443–90. https://doi.org/10.1177/2515245918810225.
[7]
John LK, Loewenstein G, Prelec D. Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychol Sci 2012;23:524–32. https://doi.org/10.1177/0956797611430953.
[8]
Flake JK, Fried EI. Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them. Advances in Methods and Practices in Psychological Science 2020;3:456–65. https://doi.org/10.1177/2515245920952393.
[9]
xkcd. Significant. Xkcd n.d. https://xkcd.com/882/.
[10]
Kerr NL. HARKing: Hypothesizing After the Results are Known. Pers Soc Psychol Rev 1998;2:196–217. https://doi.org/10.1207/s15327957pspr0203_4.
[11]
Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie Du Sert N, et al. A manifesto for reproducible science. Nat Hum Behav 2017;1:0021. https://doi.org/10.1038/s41562-016-0021.
[12]
Whitaker K, Guest O. #bropenscience is broken science. The British Psychological Society 2020. https://www.bps.org.uk/psychologist/bropenscience-broken-science (accessed September 16, 2023).
[13]
Hall BL, Tandon R. Decolonization of knowledge, epistemicide, participatory research and higher education. Research for All 2017;1. https://doi.org/10.18546/RFA.01.1.02.

PNB 3EE3: Open Science Fundamentals — Maya Flannery