Friday 29 November 2013

Peer Review - Ignored?

Earlier this year I was discussing John Bohannon's "experiment" with the peer review process of open access journals (CBC News, 2013) with a friend of mine in the sciences. The "sting operation" he performed has been criticized as biased since then (Taylor, Wedel & Naish, 2013), and was published largely as a news item even then, so it's clearly not an ideal examination of the peer review process. However, what we ended up discussing was her experience as a reviewer, as she had recently been asked to perform a review in place of her busy supervisor.

When my friend was reviewing her first ever pier-reviewed paper she felt terribly guilty for being highly critical of it, but she found it essentially flawed and didn't recommend it for publishing. As a participant in the experimental process she knew how much work had gone into the paper, and felt bad for the scientists involved, but she knew that her duty as a reviewer was to be critical and honest about adherence to the scientific process and the quality of research. After agonizing over the review, she was very surprised later on to find that the paper she thought she had condemned to rewriting had been published despite her input. She also told me that many of her colleagues had reviewed papers in the past that they deemed not fit for publishing, and had seen them go on to be published in peer-reviewed journals. Further, all of the members of her lab review for non-open-access journals.

While it's true that the peer review process involves multiple reviewers and the input of a single scientist can't be taken as the sole possible view of a paper's worth, this raises concerns about the validity of the peer review process even when it is working as originally designed in a closed-access journal of repute. While there is much debate about the potential harm that can be done when changing the peer review process, there are still many questions that can be asked about the quality of the traditional process as it is implemented today. How many of the reviewers are qualified to perform the review - and how many of them pass it on to a less experienced assistant or student working under them? How many of the reviewers approach the review process seriously and pay close attention to the details of the paper? How well do editors and selection committees adhere to the advice the reviewers give? And how well does the peer review process truly represent the opinions of the scientific community they are meant to embody?

(2013, Oct 14). Bogus science paper reveals peer review's flaws. CBC News. Retrieved from http://www.cbc.ca/news/technology/bogus-science-paper-reveals-peer-review-s-flaws-1.2054004
Taylor, M., Wedel, M., & Naish, D. (2013, Oct 7). Anti-tutorial: how to design and execute a really bad study. Retrieved from http://svpow.com/2013/10/07/anti-tutorial-how-to-design-and-execute-a-really-bad-study/

2 comments:

  1. Thanks, Vanessa, for pointing out these concerns. I am shocked. Before reading your post, I would have trusted a scientific journal that was peer reviewed. I would have believed that the articles had been sanctioned by peers. I am wondering if a system could be put in place that allows for monitoring or transparency. If an editor has already decided to publish something and ignores reviews, then it is corrupt practice on his/her behalf to say the article was peer reviewed. Is it possible to monitor processes? Is transparency possible? The other route to go would be to not have peer reviews at all and let the readership know that everything published in any journal was potential sloppy, inaccurate and not worth reading. Read at your own risk.

    ReplyDelete
  2. Thanks for your post, Vanessa. It reminded me of Barbara Sherwin, a professor at McGill who was supposed to teach my Hons. psychology seminar. A few weeks before the term started, however, she was withdrawn from the course and placed under academic suspension for ghostwriting for the pharmaceutical industry. Her defense was that the paper was peer-reviewed, so although she probably should have disclosed she was not the principal author, the methodology must have been good, so no harm done, right?

    "I believe the article, which was peer-reviewed, represented sound and thorough scholarship, and in no way could be construed as promotion for any particular product or company." http://www.papsyblog.org/2011/08/mcgill-reprimands-prof-over.html

    In the worst cases, I think the rituals of publications have been confused with the scientific pursuit itself. Publication in peer-reviewed journals = credibility. Nothing else matters.

    ReplyDelete