Good Science, Bad
Science
Latest News
[This section contains
the latest news with respect to the course. Check this section for
last-minute changes in logistics, useful hints on homework exercises,
and
general advice.]
1. Schedule and homework exercises can be found below.
2. The schedule is tentative and depends on the progress we make as a group,
so be sure to check it regularly.
3. BlackBoard should
be used for discussions and for handing in homework. The deadline
for handing in homework expires automatically as soon as the next class
starts.
4. The grade for assignments that are not handed in is an automatic 4.
5. Your grade in this course is based on a weighted average of your
individual assigments, where the two
presentation assignments count double.
6.
You are graded on individual quality and effort. Unless stated
otherwise, you are not allowed to collaborate on the individual
assignments. Please contact me in case of doubt.
7. Attendance to all lectures is mandatory.
8.
The go-to Email address for correspondence about the course
is GoodScience2015@gmail.com. Urgent Email can be sent to
EJ.Wagenmakers@gmail.com.
9. Sjoerd Glaser is TA for this course.
Goals
1. To understand the current "crisis of confidence" in psychological
science, and learn what can be done to fix it.
2.
To obtain guidance and practical experience in designing informative
and honest replication experiments through preregistration.
3. To
submit four high-quality preregistration proposals to peer-reviewed
international journals for "in principle acceptance" (IPA, https://osf.io/8mpji/wiki/home/).
Should
this goal be realized, those students who wish to contribute to
data collection and analysis (outside of the context of this course)
will co-author the manuscripts.
Class
1 (Mo, 31-08): Introduction
In the first class we will cover the philosophy and
goals of this course.
Homework Assignments for Class 2
[1] Read the following article
(published 27-08-2015): Open Science Collaboration (2015). Estimating
the reproducibility of psychological science. Science,
349, 943.
[2] Read the background
available at the Open Science Framework (the details you do not
need to read are here).
[3] The article was covered in many online newspapers and magazines. An
overview is here.
Read the coverage from the outlets listed below, and rate the quality
of each piece on a scale from 1 (=awful) to 10 (=brilliant). Integer
scores only please. These are the outlets: the Christian
Science Monitor, CBS
News, The
Guardian, The
New York Times, The
Daily Mail, The
Independent, and Buzzfeed.
Write down your scores and bring them to class.
[4]
A journalist for the NY Times has asked you about the pros and
cons of the Reproducibility Project. (a) Assume the role of a proponent
and indicate the pros (in no more than half a page); (b) Assume the
role of a detractor and indicate the cons (in no more than half a
page). Note: Throughout this entire
course, it is OK to find inspiration online (e.g., here)
or in academic writings, but you need to acknowledge this explicitly
[e.g., As Charlie Chaplin already observed "Failure is unimportant. It
takes courage to make a fool of yourself"
(http://www.brainyquote.com/quotes/authors/c/charlie_chaplin.html)].
This assignment needs to be uploaded to Blackboard before the start of
the next class. You will be graded on this assignment.
Class
2 (Thu, 03-09): Reproducibility Project
This class starts with a discussion exercise about the Reproducibility
Project. Guidelines for the discussion exercises are here.
Following this exercise Dora Matzke will tell about her experience
with adversarial collaborations (for instance here),
preregistration, and replications.
Homework Assignments for Class 3
[1] When you have been assigned to group "De Groot", read the following
paper: De Groot, A. D. (1956/2014). The
meaning of "significance" for different types of research.
Translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom,
Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora
Matzke, Don Mellenbergh, and Han L. J. van der Maas. Acta
Psychologica, 148,
188-194. When you have been assigned to group "Chambers", read the
following paper: Chambers, C. D., Feredoes, E., Muthukumaraswamy, S.
D., & Etchells, P. (2014). Instead
of "playing the game" it is time to change the rules: Registered Reports
at AIMS Neuroscience and beyond. AIMS
Neuroscience, 1, 4-17.
[2]
A journalist for the NY Times has asked you about the pros
and cons of preregistration, specifically with respect to the paper by
De Groot (for those in the "De Groot" group) and the paper by Chambers
et al. (for those in the "Chambers" group). (a) Assume the role of a
proponent and
indicate the pros (in no more than half a page); (b) Assume the role of
a detractor and indicate the cons (in no more than half a page). This
assignment needs to be uploaded to Blackboard before the start of
the next class. You will be graded on this assignment.
Class
3 (Mo, 07-09): Preregistration
This
class starts with an exercise on the benefits of preregistration. I
will bring a series of volumes from high-impact journals such as Psychological Science and you get to
play statistical detective: how many analyses can you spot that were
definitely not pre-planned?
Homework Assignments for Class 4
[1]
Find a good candidate for a preregistered replication study. There are
no restrictions, but keep in mind that the study needs to be simple,
feasible, and relatively straightforward. So no longitudinal fMRI
experiments on heroin addicts. It also helps if the effect is
theoretically important, taught in textbooks, published in a
prestigious journal, or the topic of a recent dispute. One list of
experiments that researchers like to see replicated can be found at the
PsychFileDrawer website
(but many of these experiments are relatively involved). Browsing issues of
Science, PNAS,
Psychological Science, and Journal
of Personality and Social Psychology
should provide ample source of inspiration. Another idea is to seek
replication of a really famous experiment (e.g., the one about
cognitive dissonance).
[2]
Prepare a 3-minute (!) Powerpoint presentation in which you explain
what the candidate study is about and why you would want to see it
replicated. Upload your presentation to Blackboard and
send them to GoodScience2015@gmail.com. You will receive a combined
grade for your Powerpoint presentation and your delivery during class
(advice: practice, practice, practice!)
Class
4 (Thu, 10-09): RR Candidate Selection
In this class all students give 3-minute presentations
promoting their Registered Replication candidate. A jury of experts
will select the most promising four proposals. Each student is then
assigned to one of these four proposals (i.e., about 5 students per
proposal). In the remainder of this course we will try to refine these
proposals until they are ready to be submitted to a peer-reviewed
journal.
Homework Assignments for Class 5
[1] Read the following paper: Wagenmakers, E.-J., Morey, R. D., & Lee,
M. D. (2015). Bayesian
benefits for the pragmatic researcher. Manuscript submitted for
publication.
[2] A hypothetical reviewer, Dr. Pearman, wonders why you used Bayesian
methods for the analysis of your replication study. Explain to Dr.
Pearman why Bayesian methods are particularly apt for replication
studies (no more than half an A4). Also, outline three ways in which
the "Bayesian benefits" paper can be improved (no more than half an
A4). This assignment needs to be uploaded to Blackboard before the
start of
the next class. You will be graded on this assignment.
Class
5 (Mo, 14-09): Bayesian Statistics with JASP
This class demonstrates the use of JASP,
a free and open-source software program with an inviting GUI which is
undergoing rapid development in my lab. With JASP you can easily conduct
both
classical and Bayesian analyses.
Homework Assignments for Class 6
[1] Write an analysis plan
for your assigned RR proposal. You can be inspired by earlier proposals from
my lab (e.g., here,
here,
here,
here, or
here;
NOTE: go into these articles and find the
links to the preregistration proposals that are available online;
for Bayesian alternatives see also here).
It is also acceptable to carry out a classical power analysis (e.g., using
the G*Power
software). This assignment needs to be uploaded to Blackboard before
the start of
the next class. You will be graded on this assignment. Another
note:
you are free to provide more context by informing me about the
Materials and Procedure, but what I'll grade is the analysis plan.
[2] Together with the other members of your group, create a Google doc file
and an OSF directory for your assigned RR
proposal.
Class
6 (Thu, 17-09): RR Analysis Plan
This class aims to finalize the analysis plans for each of the four RR
proposals.
Homework Assignments for Class 7
[1]
Start writing a short introduction to your RR proposal. Explain the
motivation, the importance, and the added value of preregistration.
Include references to the relevant literature. Print your introduction
and bring it with you to the next class. This will be vitally important
for the in-class work. This assignment will not be graded.
[2] Read the following paper: Neuroskeptic (2012). The
nine circles of scientific hell. Perspectives
on Psychological Science, 7, 643-644. Think about what circles you
aim to avoid with your RR proposal. This assignment will not be graded.
Class
7 (Mo, 21-09): RR Introduction
This class aims to finalize the introductions for each of the four RR
proposals.
Homework Assignments for Class 8
[1]
As a group: fine-tune your introductions. You should now have a good
first draft of your introduction and analysis plan, that is, a good
first draft of your intended RR proposal.
[2] Before
Tuesday at noon: Send your first draft to RR proposal
GoodScience2015@gmail.com, and we will distribute it to the other students
in this class.
[3]
Read the other three RR proposals, and provide suggestions for
improvement. Keep in mind the three Cs: be concise, constructive, and
concrete! Send your feedback to the students who drafted the RR
proposal. Also, upload your feedback to Blackboard before the
start of
the next class. You will be graded on this assignment.
Class
8 (Thu, 24-09): RR Revisions
This class aims to develop a plan to revise the RR proposals in light of the
suggestions for improvement.
Homework Assignments for Class 9
[1] As a group: take the suggestions for improvement into account and revise
your RR proposal.
[2]
Create a fake data set according to the constraints improsed by your
preregistration plan. Analyze the data set using JASP. In other words,
pretend that your are conducting the actual experiment and observe data
according to the plan that you've specified. The data can be generated
automatically (for instance using R) or Stapel-style (by hand). Store
the results in a ,jasp file and upload this on Blackboard; also upload
a text file that explains what you would conclude from the output. The
best
examples will be included in the OSF directory as a concrete and
unambiguous blueprint of how the real data will be analyzed. You will
be graded on this assignment. NB. In case your analysis cannot yet be
carried out in JASP you are free to use a different program.
Class
9 (Mo, 28-09): Contacting the Author
This
class discusses the analysis files. Also, we discuss Kahneman's
recommendation that, for replication research, it is a good idea to
contact the original authors and seek their advice.
Homework Assignments for Class 10
[1]
In no more than one A4, write the original authors to seek their
advice regarding your RR proposal. This assignment needs to be
uploaded to Blackboard before the start of
the next class. You will be graded on this assignment.
[2] Read the following papers: Button et al. (2013). Power
failure: Why small sample size undermines the reliability of
neuroscience. Nature Reviews
Neuroscience, 14,
1-12 and Wagenmakers, E.-J., Verhagen, A. J., Ly, A., Bakker, M., Lee,
M. D., Matzke, D., Rouder, J. N., & Morey, R. D. (in press). A
power fallacy. Behavior Research
Methods.
Class
10 (Thu, 01-10): Power
This class starts with a discussion exercise about power.
Homework Assignments for Class 11
[1] Those in the Francis group read the following paper: Francis, G. (2012).
The psychology of
replication and replication in psychology. Perspectives
on Psychological Science, 7, 585-594.
Those in the Schimmack group read: Schimmack, U. (2014). Quantifying
statistical research integrity: The replicability-index. Available on this
website.
[2]
For those in the Francis group: write a one-page critique of the
Francis method (if you find inspiration in the literature then cite it
and explain why you find the argument compelling). For those in the
Schimmack group: write a one-page critique of the Schimmack method (if
you find inspiration in the literature then cite it and explain why you
find the argument compelling). Next: channel Francis or Schimmack and
write a one-page rebuttal to the critique. This assignment needs to be
uploaded to Blackboard before the start of
the next class. You will be graded on this assignment.
Class
11 (Mo, 05-10): Francis
& Schimmack
This
class starts with a discussion exercise about the Francis and Schimmack
methods, discussing their advantages and limitations, and we'll take it
from there.
Homework Assignments for Class 12
[1] Read the following two papers, available here:
(1) Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A
key to the file-drawer. Journal of
Experimental Psychology: General, 143,
534-547; (2) Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014).
p-curve and effect size: Correcting for publication bias using only
significant results. Perspectives on
Psychological Science, 9, 666-681.
[2] Apply a P-curve analysis to the studies that are related to your RR
proposal. This assignment needs to be
uploaded to Blackboard before the start of
the next class. You will be graded on this assignment.
Class
12 (Thu, 08-10): P-Curves
This class starts with a discussion about P-curves, and we'll take it from
there.
Homework Assignments for Class 13
[1] Those in the Simmons group, read the following paper: Simmons, J. P.,
Nelson, L. D., Simonsohn, U. (2011). False-positive
psychology: Undisclosed flexibility in data collection and analysis allows
presenting anything as significant. Psychological
Science, 22, 1359-1366 (see also
here). Their data are published in the Journal
of Open Psychology Data here.
Those in the Peirce group, read the following section in a paper by
Peirce (originally published in 1883, "A theory of probable
inference"): Hartshorne, C., & Weiss, P. (1932). Collected papers
of Charles Sanders Peirce: Volume II: Elements of logic. Cambridge:
Harvard University Press: pp.
461-467 (section 8).
[2] On page 466, footnote 1, Peirce
says: "The physicians seem to use the maxim that you cannot reason from post hoc to propter
hoc
to mean (rather obscurely) that cases must not be used to prove a
proposition that has only been suggested by these cases themselves."
This suggests that the general principle (i.e., don't use the data
twice, but focus on pure prediction) was already known before Peirce.
Find writings that predate Peirce and that state the same
principle. This assignment needs to be
uploaded to Blackboard before the start of
the next class. You will be graded on this assignment.
Class
13 (Mo, 12-10): QPRs
This
class starts with a discussion exercise about QPRs using this
website.
Homework Assignments for Class 14
[1] Read the Transparency
and Openness Promotion (TOP) Guidelines. Also,
read the following paper: Nosek, B. A., Spies, J. R., & Motyl, M.
(2012). Scientific
utopia: II. Restructuring incentives and practices to promote truth over
publishability. Perspectives on
Psychological Science, 7, 615-631.
[2] Propose another TOP guideline that you believe may be useful (one page
max). This assignment needs to be
uploaded to Blackboard before the start of
the next class. You will be graded on this assignment.
Class
14 (Thu, 15-10): TOP Guidelines
This class starts with a discussion exercise about the TOP guidelines, and
we'll take it from there...
Homework Assignments for Class 15
[1] Each group starts to work on a final presentation of their RR proposal
(for details see homework for class 16).
Class
15 (Mo, 19-10): Finalizing the RRs
In this class we aim to finalize the RRs and make specific plans for the
future.
Homework Assignments for Class 16
[1]
Each group prepares a final presentation of their RR proposal. Each
individual group member prepares a 3-5 minute (!) Powerpoint
presentation in which they explain one part of the project. Upload your
presentation to Blackboard and
send them to GoodScience2015@gmail.com. You will receive a combined
grade for your Powerpoint presentation and your delivery during class
(advice: practice, practice, practice!) .
Class
16 (Thu, 22-10): Presentation of Finalized RRs
In this class each of the four groups present their RR proposal.