Psychology Replication Test
Scientific studies about how people act or think can rarely be replicated by outside experts, according to a new study that raises questions about the seriousness of psychology research.
A team of 270 scientists tried reproducing 100 psychology and social science studies that had been published in three top peer-reviewed US journals in 2008.
Just 39 per cent came out with same results as the initial reports, according an international team of researchers known as The Open Science Collaboration.
Their findings are reported in the journal Science .
The topics of studies reviewed ranged from people's social lives and interactions with others to research involving perception, attention and memory.
No medical therapies were called into question as a result of the study, although a separate effort is underway to evaluate cancer biology studies.
"It's important to note that this somewhat disappointing outcome does not speak directly to the validity or the falsity of the theories," says Gilbert Chin, a psychologist and senior editor at the journal Science.
"What it does say is that we should be less confident about many of the original experimental results," says Chin, who was not involved in the study.
Study co-author Brian Nosek from the University of Virginia says the research shows the need for scientists to continually question themselves.
"A scientific claim doesn't become believable because of the status or authority of the person that generated it," says Nosek.
"Credibility of the claim depends in part on the repeatability of its supporting evidence."
Skewed picture
Problems can arise when scientists cherry-pick their data to include only what is deemed "significant," or when study sizes are so small that false negatives or false positives arise.
Nosek says scientists are also under pressure to publish their research regularly and in top journals, and the process can lead to a skewed picture.
"Not everything we do gets published. Novel, positive and tidy results are more likely to survive peer review and this can lead to publication biases that leave out negative results and studies that do not fit the story that we have," he says.
"If this occurs on a broad scale, then the published literature may become more beautiful than the reality."
Some experts said the problem may be even worse that the current study suggests.
John Ioannidis, a biologist at Stanford University in Palo Alto, told Science magazine he suspects about 25 per cent of psychology papers would hold up under scrutiny, about the same "as what we see in many biomedical disciplines."
Key caution
One study author who participated in the project as both a reviewer and reviewee was E J Masicampo, assistant professor at Wake Forest College in North Carolina.
He was part of a team that was able to replicate a study that found people who are faced with a confrontational task, like having to play a violent video game, prefer to listen to angry music and think about negative experiences beforehand.
But when outside researchers tried to replicate Masicampo's study -- which hypothesised that a sugary drink can help college students do better at making a complicated decision -- they were not successful.
Masicampo chalks up the differences to geographical factors, stressing that the experiment showed how complicated it can be to do a high-quality replication of a study.
"As an original author whose work was being replicated, I felt like my research was being treated in the best way possible," he says.
There are ways to fix the process so that findings are more likely to hold up under scrutiny, says Dorothy Bishop, professor of developmental neuropsychology at the University of Oxford.
"I see this study as illustrating that we have a problem, one that could be tackled," says Bishop, who was not involved in research.
She urged mandatory registration of research methods beforehand to prevent scientists from picking only the most favourable data for analysis, as well as requiring adequate sample sizes and wider reporting of studies that show null result, or in other words, those that do not support the hypothesis initially put forward.
Scientists could also publish their methods and data in detail so that others could try to replicate their experiments more easily.
These are "simply ways of ensuring that we are doing science as well as we can," says Bishop.
0 comments :