Reproducibility is more than the RPP

A bit less than a year ago, in one of the biggest events in psychology in recent memory, a group of researchers published the Reproducibility Project: Psychology, a landmark effort to reproduce 100 findings in psychology journals.  The major result was that, depending on how you measure "reproducibility", between 39% and 47% of the original results were successfully reproduced.  Today, a new comment on the RPP has been published that makes the bold claim that the reproducibility estimates reported by the original team were drastically wrong, and are "statistically indistinguishable from 100%".

Although the commentary does raise some good points -- they note that some of the studies in the reproducibility project depart from the original studies in ways that are likely problematic -- I also think it's easy to lose sight of the broader context when critiquing a single project.  (For those interested, there also may be some problems with the basic claims of the critique)

Reviewing peer review (and its flaws)

Peer review is viewed as the arbiter of good science.  In fact, passing peer review is typically a prerequisite for professional advancement -- a scientific paper will not be published unless it is judged worthy of publication by one or more peers, and likewise a grant will not be awarded unless a group of scientific peers judge a proposal to be of sufficiently high quality.  I would argue that, because so many means of professional advancement are conditional on satisfying reviewers, satisfying reviewers is one of the most important tasks that a career scientist faces.

Because of its importance to career scientists, scientists have plenty of opinions about peer review.  However, peer review is not often taken as an object of scientific study.  My goal in this post, then, is to conduct a short review peer review.  I will structure my discussion around the following three questions, after which I will give some concluding thoughts:

1. What are the goals of peer review?
2. What are the costs of peer review?  Who bears these costs?
3. What are benefits of peer review?  Who reaps these benefits?

Mapping "prejudice" research reveals its preoccupation with implicit bias

One of the many difficulties of doing social science is that the concepts that we study are often fuzzy.  Precisely defining concepts like "attitudes", "cognition", and the "self" can be challenging, which sometimes leads to dramatic differences in how scientists use the terms.

The challenges are only enhanced when the object of study is a politically charged concept like my chosen field of study, prejudice.  I believe this fuzziness in the definition of "prejudice" has exerted a distorting influence on research on the topic, affecting the questions researchers ask, the measures researchers use, and the interventions researchers develop.

Today, I'm going to focus on a small piece of this issue by answering the following questions:

1. When contemporary researchers choose to study "prejudice", how do they use the term?
2. What does contemporary researchers' use of the term "prejudice" reveal about their (often unstated) definitions of of the term?

What we can learn from the LaCour data fabrication incident

Mike LaCour, author of a paper on
canvassing that was later retracted

About two weeks ago, news broke that Michael LaCour, the first author of a study about how, purportedly, gay canvassers can successfully improve people's attitudes toward gay men and same-sex marriage initiatives, likely fabricated his data. Although news about fraud is always troubling, this news was particularly troubling -- after all, the study was published in Science, one of the most high-profile journals for scientific research (as the joke goes, the shorter the title, the more prestigious the journal). In addition, the methods of the study appeared to be rigorous, and the findings just "felt good" -- according to the study, brief, 20-minute conversations with canvassers who admitted they were gay created dramatic changes in attitude that persisted up to nine months.

Diederik Stapel and the frequency of scientific shenanigans

On August 27, two junior researchers working with the Dutch social psychologist Diederik Stapel at Tilburg University contacted a university administrator with suspicions that their senior colleague was using faked data.  As one of the worst forms of academic shenanigans that fall under the broad umbrella of "academic misconduct", an allegation of data fabrication was quite serious.  This is especially true because Diederik Stapel was in the early stages of a prolific scientific career; he served on the editorial board of six different academic journals and had received the 2007 "Early Career Award" from the International Society for Self and Identity (ISSI).  He had also published many articles that received generous press attention, including one in Science that claimed that messy environments promote discrimination.

Nonetheless, a little over a week and one university investigation later, Stapel admitted to making up data and was sacked from Tilburg University.

The quest for (social) scientific truth

Every now and then when I tell a stranger that I study scientific psychology, I get a reaction that is perhaps best summarized by this comic from xkcd:

Click for a larger image

Essentially, the reaction is that, as a social science, psychology isn't a "real" science like biology, chemistry, or physics.  I encounter this reaction often enough, even among otherwise scientifically savvy people, that I am often tempted to do silly things like shout, punch things, or at the very least launch into an ill-advised rant.

Unraveling the "obviousness" bias in psychology

In my last post, I argued that the pseudoscience of parapsychology (and in particular the publication of a paper in the Journal of Personality and Social Psychology claiming to provide evidence of precognition) hurts the perception of psychology as a science. This may seem like an obvious argument to make; Bem's paper was published in the flagship journal of social psychology, so it is easy to make the logical jump that this article is representative of the kind of research most social psychologists do. Therefore, the reasoning goes, social psychology is not a real science at all; to quote one comment on the media coverage of the Bem article,

Psychology is such a joke. A demonstration of future events influencing present events would be one of the most important (if not *the* most important) findings in the history of mankind. Yet this demonstration doesn't end up in Science or Nature, but is published in the Journal of Personality and Social Psychology? And some wonder why psychology is still considered pseudoscience....

"Evidence" for precognition hurts the perception of psychology as a science

There is a a paper in press at the most prestigious journal in my field, Journal of Personality and Social Psychology, that claims to have found evidence of ESP.

In this paper, Daryl Bem, a well-known and well-respected psychologist at Cornell University, conducted 9 time-reversed versions of classic social psychology experiments. For example, one common way of testing for associations between sets of concepts is through priming, in which an picture is flashed very quickly on a computer screen, after which the participant must categorize a second object. People are usually faster at categorizing the second object when the two objects share a relationship to each other; for example, when a picture of a rainbow (which most people agree is "good") is displayed before the participant must categorize the word "puppy" (which most people also agree is "good") as either good or bad.

Science and storytelling

While catching up on the latest issue of Atlantic, I stumbled upon this article about a massive longitudinal study about what makes us happy. Expecting an article full of wonderful statistical analysis, I eagerly began reading.

The article was not what I expected.