Good Science, Bad Science

Latest News

[This section contains the latest news with respect to the course. Check this section for last-minute changes in logistics, useful hints on homework exercises, and general advice.]

1. Schedule and homework exercises can be found below.
2. The schedule is tentative and depends on the progress we make as a group, so be sure to check it regularly.
3. BlackBoard should be used for discussions and for handing in homework. The deadline for handing in homework expires automatically as soon as the next class starts.
4. The grade for assignments that are not handed in is an automatic 4.
5. Your grade in this course is based on a weighted average of your individual assigments, where the two presentation assignments count double. 
6. You are graded on individual quality and effort. Unless stated otherwise, you are not allowed to collaborate on the individual assignments. Please contact me in case of doubt.
7. Attendance to all lectures is mandatory.
8. The go-to Email address for correspondence about the course is GoodScience2015@gmail.com. Urgent Email can be sent to EJ.Wagenmakers@gmail.com.
9. Sjoerd Glaser is TA for this course.

Goals

1. To understand the current "crisis of confidence" in psychological science, and learn what can be done to fix it.
2. To obtain guidance and practical experience in designing informative and honest replication experiments through preregistration.
3. To submit four high-quality preregistration proposals to peer-reviewed international journals for "in principle acceptance" (IPA, https://osf.io/8mpji/wiki/home/). Should this goal be realized, those students who wish to contribute to data collection and analysis (outside of the context of this course) will co-author the manuscripts.

Class 1 (Mo, 31-08): Introduction
In the first class we will cover the philosophy and goals of this course.

Homework Assignments for Class 2

[1] Read the following article (published 27-08-2015):  Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349, 943.

[2] Read the background available at the Open Science Framework (the details you do not need to read are here).

[3] The article was covered in many online newspapers and magazines. An overview is here. Read the coverage from the outlets listed below, and rate the quality of each piece on a scale from 1 (=awful) to 10 (=brilliant). Integer scores only please. These are the outlets: the Christian Science Monitor, CBS NewsThe Guardian, The New York Times, The Daily Mail, The Independent, and Buzzfeed. Write down your scores and bring them to class.

[4] A journalist for the NY Times has asked you about the pros and cons of the Reproducibility Project. (a) Assume the role of a proponent and indicate the pros (in no more than half a page); (b) Assume the role of a detractor and indicate the cons (in no more than half a page). Note: Throughout this entire course, it is OK to find inspiration online (e.g., here) or in academic writings, but you need to acknowledge this explicitly [e.g., As Charlie Chaplin already observed "Failure is unimportant. It takes courage to make a fool of yourself" (http://www.brainyquote.com/quotes/authors/c/charlie_chaplin.html)]. This assignment needs to be uploaded to Blackboard before the start of the next class. You will be graded on this assignment.

Class 2 (Thu, 03-09): Reproducibility Project
This class starts with a discussion exercise about the Reproducibility Project. Guidelines for the discussion exercises are here. Following this exercise Dora Matzke will tell about her experience with adversarial collaborations (for instance here), preregistration, and replications.

Homework Assignments for Class 3

[1] When you have been assigned to group "De Groot", read the following paper: De Groot, A. D. (1956/2014). The meaning of "significance" for different types of research. Translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas. Acta Psychologica, 148, 188-194. When you have been assigned to group "Chambers", read the following paper: Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D., & Etchells, P. (2014). Instead of "playing the game" it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS Neuroscience, 1, 4-17.

[2]  A journalist for the NY Times has asked you about the pros and cons of preregistration, specifically with respect to the paper by De Groot (for those in the "De Groot" group) and the paper by Chambers et al. (for those in the "Chambers" group). (a) Assume the role of a proponent and indicate the pros (in no more than half a page); (b) Assume the role of a detractor and indicate the cons (in no more than half a page). This assignment needs to be uploaded to Blackboard before the start of the next class. You will be graded on this assignment.

Class 3 (Mo, 07-09): Preregistration
This class starts with an exercise on the benefits of preregistration. I will bring a series of volumes from high-impact journals such as Psychological Science and you get to play statistical detective: how many analyses can you spot that were definitely not pre-planned?

Homework Assignments for Class 4

[1] Find a good candidate for a preregistered replication study. There are no restrictions, but keep in mind that the study needs to be simple, feasible, and relatively straightforward. So no longitudinal fMRI experiments on heroin addicts. It also helps if the effect is theoretically important, taught in textbooks, published in a prestigious journal, or the topic of a recent dispute. One list of experiments that researchers like to see replicated can be found at the PsychFileDrawer website (but many of these experiments are relatively involved). Browsing issues of Science, PNAS, Psychological Science, and Journal of Personality and Social Psychology should provide ample source of inspiration. Another idea is to seek replication of a really famous experiment (e.g., the one about cognitive dissonance).

[2] Prepare a 3-minute (!) Powerpoint presentation in which you explain what the candidate study is about and why you would want to see it replicated. Upload your presentation to Blackboard and send them to GoodScience2015@gmail.com. You will receive a combined grade for your Powerpoint presentation and your delivery during class (advice: practice, practice, practice!) 

Class 4 (Thu, 10-09): RR Candidate Selection
In this class all students give 3-minute presentations promoting their Registered Replication candidate. A jury of experts will select the most promising four proposals. Each student is then assigned to one of these four proposals (i.e., about 5 students per proposal). In the remainder of this course we will try to refine these proposals until they are ready to be submitted to a peer-reviewed journal.

Homework Assignments for Class 5

[1] Read the following paper: Wagenmakers, E.-J., Morey, R. D., & Lee, M. D. (2015). Bayesian benefits for the pragmatic researcher. Manuscript submitted for publication.

[2] A hypothetical reviewer, Dr. Pearman, wonders why you used Bayesian methods for the analysis of your replication study. Explain to Dr. Pearman why Bayesian methods are particularly apt for replication studies (no more than half an A4). Also, outline three ways in which the "Bayesian benefits" paper can be improved (no more than half an A4). This assignment needs to be uploaded to Blackboard before the start of the next class. You will be graded on this assignment.

Class 5 (Mo, 14-09): Bayesian Statistics with JASP
This class demonstrates the use of JASP, a free and open-source software program with an inviting GUI which is undergoing rapid development in my lab. With JASP you can easily conduct both classical and Bayesian analyses.

Homework Assignments for Class 6

[1] Write an analysis plan for your assigned RR proposal. You can be inspired by earlier proposals from my lab (e.g., here, here, here, here, or here; NOTE: go into these articles and find the links to the preregistration proposals that are available online; for Bayesian alternatives see also here). It is also acceptable to carry out a classical power analysis (e.g., using the G*Power software). This assignment needs to be uploaded to Blackboard before the start of the next class. You will be graded on this assignment. Another note: you are free to provide more context by informing me about the Materials and Procedure, but what I'll grade is the analysis plan.

[2] Together with the other members of your group, create a Google doc file and an OSF directory for your assigned RR proposal. 

Class 6 (Thu, 17-09): RR Analysis Plan
This class aims to finalize the analysis plans for each of the four RR proposals.

Homework Assignments for Class 7

[1] Start writing a short introduction to your RR proposal. Explain the motivation, the importance, and the added value of preregistration. Include references to the relevant literature. Print your introduction and bring it with you to the next class. This will be vitally important for the in-class work. This assignment will not be graded.   

[2] Read the following paper: Neuroskeptic (2012). The nine circles of scientific hell. Perspectives on Psychological Science, 7, 643-644. Think about what circles you aim to avoid with your RR proposal. This assignment will not be graded.

Class 7 (Mo, 21-09): RR Introduction
This class aims to finalize the introductions for each of the four RR proposals.

Homework Assignments for Class 8

[1] As a group: fine-tune your introductions. You should now have a good first draft of your introduction and analysis plan, that is, a good first draft of your intended RR proposal.

[2] Before Tuesday at noon: Send your first draft to RR proposal GoodScience2015@gmail.com, and we will distribute it to the other students in this class.

[3] Read the other three RR proposals, and provide suggestions for improvement. Keep in mind the three Cs: be concise, constructive, and concrete! Send your feedback to the students who drafted the RR proposal. Also, upload your feedback to Blackboard before the start of the next class. You will be graded on this assignment.
Class 8 (Thu, 24-09): RR Revisions
This class aims to develop a plan to revise the RR proposals in light of the suggestions for improvement.

Homework Assignments for Class 9

[1] As a group: take the suggestions for improvement into account and revise your RR proposal.

[2] Create a fake data set according to the constraints improsed by your preregistration plan. Analyze the data set using JASP. In other words, pretend that your are conducting the actual experiment and observe data according to the plan that you've specified. The data can be generated automatically (for instance using R) or Stapel-style (by hand). Store the results in a ,jasp file and upload this on Blackboard; also upload a text file that explains what you would conclude from the output. The best examples will be included in the OSF directory as a concrete and unambiguous blueprint of how the real data will be analyzed. You will be graded on this assignment. NB. In case your analysis cannot yet be carried out in JASP you are free to use a different program.
Class 9 (Mo, 28-09): Contacting the Author
This class discusses the analysis files. Also, we discuss Kahneman's recommendation that, for replication research, it is a good idea to contact the original authors and seek their advice.

Homework Assignments for Class 10

[1]  In no more than one A4, write the original authors to seek their advice regarding your RR proposal. This assignment needs to be uploaded to Blackboard before the start of the next class. You will be graded on this assignment.

[2] Read the following papers:  Button et al. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 1-12 and Wagenmakers, E.-J., Verhagen, A. J., Ly, A., Bakker, M., Lee, M. D., Matzke, D., Rouder, J. N., & Morey, R. D. (in press). A power fallacy. Behavior Research Methods.

Class 10 (Thu, 01-10): Power
This class starts with a discussion exercise about power.

Homework Assignments for Class 11

[1] Those in the Francis group read the following paper: Francis, G. (2012). The psychology of replication and replication in psychology. Perspectives on Psychological Science, 7, 585-594. Those in the Schimmack group read:  Schimmack, U. (2014). Quantifying statistical research integrity: The replicability-index. Available on this website.

[2] For those in the Francis group: write a one-page critique of the Francis method (if you find inspiration in the literature then cite it and explain why you find the argument compelling). For those in the Schimmack group: write a one-page critique of the Schimmack method (if you find inspiration in the literature then cite it and explain why you find the argument compelling). Next: channel Francis or Schimmack and write a one-page rebuttal to the critique. This assignment needs to be uploaded to Blackboard before the start of the next class. You will be graded on this assignment.
Class 11 (Mo, 05-10): Francis & Schimmack
This class starts with a discussion exercise about the Francis and Schimmack methods, discussing their advantages and limitations, and we'll take it from there.

Homework Assignments for Class 12

[1] Read the following two papers, available here: (1) Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534-547; (2) Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). p-curve and effect size: Correcting for publication bias using only significant results. Perspectives on Psychological Science, 9, 666-681.

[2] Apply a P-curve analysis to the studies that are related to your RR proposal. This assignment needs to be uploaded to Blackboard before the start of the next class. You will be graded on this assignment.

Class 12 (Thu, 08-10): P-Curves
This class starts with a discussion about P-curves, and we'll take it from there.

Homework Assignments for Class 13

[1] Those in the Simmons group, read the following paper: Simmons, J. P., Nelson, L. D., Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366 (see also here). Their data are published in the Journal of Open Psychology Data here. Those in the Peirce group, read the following section in a paper by Peirce (originally published in 1883, "A theory of probable inference"): Hartshorne, C., & Weiss, P. (1932). Collected papers of Charles Sanders Peirce: Volume II: Elements of logic. Cambridge: Harvard University Press: pp. 461-467 (section 8).

[2] On page 466, footnote 1, Peirce says: "The physicians seem to use the maxim that you cannot reason from post hoc to propter hoc to mean (rather obscurely) that cases must not be used to prove a proposition that has only been suggested by these cases themselves." This suggests that the general principle (i.e., don't use the data twice, but focus on pure prediction) was already known before Peirce. Find writings that predate Peirce and that state the same principle. This assignment needs to be uploaded to Blackboard before the start of the next class. You will be graded on this assignment.
Class 13 (Mo, 12-10): QPRs
This class starts with a discussion exercise about QPRs  using this website.

Homework Assignments for Class 14

[1] Read the Transparency and Openness Promotion (TOP) Guidelines. Also, read the following paper: Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631.

[2] Propose another TOP guideline that you believe may be useful (one page max). This assignment needs to be uploaded to Blackboard before the start of the next class. You will be graded on this assignment.
Class 14 (Thu, 15-10): TOP Guidelines 
This class starts with a discussion exercise about the TOP guidelines, and we'll take it from there...

Homework Assignments for Class 15

[1] Each group starts to work on a final presentation of their RR proposal (for details see homework for class 16).

Class 15 (Mo, 19-10): Finalizing the RRs
In this class we aim to finalize the RRs and make specific plans for the future.

Homework Assignments for Class 16

[1] Each group prepares a final presentation of their RR proposal. Each individual group member prepares a 3-5 minute (!) Powerpoint presentation in which they explain one part of the project. Upload your presentation to Blackboard and send them to GoodScience2015@gmail.com. You will receive a combined grade for your Powerpoint presentation and your delivery during class (advice: practice, practice, practice!) .

Class 16 (Thu, 22-10): Presentation of Finalized RRs
In this class each of the four groups present their RR proposal.