Peer review is like a lottery
OPINION: Peer review is not the automatic seal of approval that it may seem. It depends more on what the journals want to publish.
One thing is guaranteed to make an academic bang their head against the desk: the peer review. The process of converting your work into an article for publication in a scientific journal, followed by a review by two or three of your peers.
The peer reviewers’ task is to decide whether your results are new and exciting enough for publication in their journal.
The ‘publish or perish’ mantra strikes fear in the heart of academics as soon as they receive the long awaited email from the journal editor with a decision as to whether their article has been accepted.
Every PhD student will experience the process of peer review at some time or other, and here I will share my own personal experience.
Peer review: a crash course in humility
I was halfway into my PhD when I submitted my first draft of a scientific article to my supervisor. My assertion that he could just add his changes and that I would submit it to a journal by the end of the month were met with a puzzled look.
"Uh, no. We’re not there yet!" he said.
One and half years later, the article was published, and my arrogance replaced with humility.
The first journal politely said, “No thanks”. The refusal went like this: "I cannot escape the conclusion that this paper in its current form does not yield generalizable results. In its current form, it lacks a strong enough connection two theory and a sufficient empirical foundation."
This is academic for “this study is crap.”
Three reviewers looked at my study, and this was my first lesson about peer review: There is rarely a consensus. Here is what a couple of them had to say:
“This research does an excellent job of considering Danish science journalism practices in the context of international science journalism.”
“After reading the manuscript, I do not see what is novel or important about this research.”
Peer review is a lottery
I shared my frustrations with my colleagues, and out came their many personal anecdotes.
One had the same article refused by five different journals before finally getting it published. Another had an article rejected by the editor—who did not even send it out for review. A third told me that his now most cited article had so many rejections that they nearly abandoned it all together.
And I have not become better friends with the system after stumbling upon scientific articles that conclude the likelihood of having a paper published is, statistically speaking, random.
In other words, it is a lottery as to whether or not your paper will be accepted for publication. The idea of a quality check is also an illusion.
The ratings of peer reviewers are as unpredictable as film reviews. The latest Star Wars film received anything from two to six stars. It is a matter of taste.
Close to acceptance in a large journal but long way from publishing in a small journal
As a rule you will typically receive one or three responses to your journal submission. Either the editor accepts it, rejects it, or asks you to rewrite and resubmit.
My next article fell into the second category. It was rejected by the editor.
“Unfortunately, the shortcomings of the paper are not of the type that can be overcome by a simple revision,” he wrote.
But the reviewers own recommendation contradicted their assessment. One accepted it, one recommended it to be rejected, and the third suggested a rewrite. Two out of three gave it a chance, but this ultimately meant nothing if the editor then rejects it.
In a moment of madness I began to think about the system logically. Big. Mistake. I sent the article to a prestigious journal first, where it was almost accepted. I rewrote it as the reviewers’ suggested and sent it to a less recognised journal with only two reviewers.
My rational argument was that if the article is close to qualifying for the Olympics, then it must be good enough for the regional championships of a less renowned journal.
Alas, while the article was almost accepted by the prestigious journal, it felt like the less prestigious journal ran it through a paper shredder.
The peer review system does not benefit research
A PhD is a research qualification and part of the training includes learning how to navigate the system of peer review. But I still find it hard to see how the system actually benefits research.
You learn how to satisfy reviewers and editors by mastering the academic style of writing and the format of scientific papers. You learn that the transparency of your results are not so important.
So it's not about getting research findings published for the benefit of bring new knowledge to society.
Instead, it is about demonstrating your own ability and to pimp your CV. I have not made this up. This is exactly what one professor told me after my first rejection.
Newton, Einstein, and Bohr had no peer review
Sometimes it seems as if scientists have forgotten that the peer review system is not even 100 years old. Neither Newton, Darwin, Einstein, nor Bohr went through the system. As such, it is not a fundamental part of scientific endeavour, so why maintain it when it is proven to be random and slow?
Perhaps the most important lesson for PhD students is that the peer review system is a necessary evil, but you should not lose any sleep over the outcome. The system that we think is guarding the credibility of science is more like a bottleneck--keeping the total number of annual publications down to about a million.
There are also lessons for science journalists: A peer review does not guarantee high quality scientific research.
----------------
Read the Danish version of this story on Videnskab.dk
Translated by: Catherine Jex
Scientific links
- Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone?, Brain (2000), doi: 10.1093/brain/123.9.1964
- The Assessment of Science: The Relative Merits of Post-Publication Review, the Impact Factor, and the Number of Citations, PLOS (2013), DOI: 10.1371/journal.pbio.1001675
