BMJ 2003;326:241 (1 February)
Little evidence for effectiveness of scientific peer review
Caroline White, London
Despite its widespread use and costs, little hard evidence
exists that peer review improves the quality of published
biomedical research, concludes a systematic review from the
international Cochrane Collaboration.
Yet the system, which has been used for at least 200 years,
has only recently come under scrutiny, with its assumptions
about fairness and objectivity rarely tested, say the review
authors. With few exceptions, journal editors and
clinicians around the world continue to see it as the
hallmark of serious scientific endeavour.
Published last week, the review is the third in a series
from the Cochrane Collaboration Methods Group. The other
reviews look at the grant application process and technical
Only the latter escapes a drubbing, with the reviewers
concluding that technical editing does improve the
readability, accuracy, and overall quality of published
The Cochrane reviewers based their findings on 21 studies of
the peer review process from an original trawl of only 135.
These were drawn from a comprehensive search of biomedical
print and online databases, and information received from
bodies such as the World Association of Medical Editors.
Almost half of the available research focused on the effects
of concealing the identity of reviewers and/or authors,
which, the Cochrane authors conclude, has little impact on
quality. Few studies assessed the impact of peer review on
the importance, usefulness, relevance, or quality of
research. Only one small study tested the validity of the
peer review procedure itself.
On the basis of the current evidence, "the practice of peer
review is based on faith in its effects, rather than on
facts," state the authors, who call for large, government
funded research programmes to test the effectiveness of the
system and investigate possible alternatives.
"As the information revolution gathers pace, an empirically
proven method of quality assurance is of paramount
importance," they contend.
Professor Tom Jefferson, who led the Cochrane review,
suggested that further research might prove that peer
review, or an evolved form of it, worked. At the very least,
it needed to be more open and accountable.
But he said that there had never even been any consensus on
its aims and that it would be more appropriate to refer to
it as "competitive review."
Not only did peer review pander to egos and give researchers
licence to knife each other in the back with impunity, he
said, but it was also "completely useless at detecting
research fraud" and let editors off the hook for publishing
poor quality studies.
In the latest report from the Committee on Publication
Ethics, Professor Peter Lachmann, until recently president
of the UK Academy of Medical Sciences, comments: "Peer
review is to science what democracy is to politics. It's not
the most efficient mechanism, but it's the least
The report can be accessed from the National electronic
Library for Health (www.nelh.nhs.uk)
© 2003 BMJ Publishing Group Ltd