Peer Reviewing my Peers…

Well, I am quite late for this post.  To make up for my tardiness, I decided that I would take this as an opportunity to respond to all of my group members previous posts on this topic.  Think of it as a sort of peer review (and like peer review, it comes long after the original submissions!)

Akash has found some other interesting incidents involving controversy in peer review.  It is interesting to note that some incidents of peer review failure were cases where the research was perfectly valid, but it was completely plagiarized.  I think this serves as a good example of where peer review might not be able to hold up a standard.  I think that most peer reviewers would only being doing a review of an article to see if the ideas that it is espousing contain any merit.  I would not be surprised if many peer reviewers would be caught off guard by an idea that is perfectly sound, but completely unoriginal.  I think that this is especially true if you consider the fact that most research does not occur in isolation; it would be very difficult to be certain if a good idea in one paper hasn't already been thought of in another.

I like Lauren's discussion of Wikipedia as a source of peer review.  And I agree with her assessment that, despite Fitzpatrick's assertion that Wikipedia is a platform of ongoing peer review, the reality does not hold up that standard.  It is good to point out that most Wikipedians are not experts, as I think that this very much does make a difference in the quality of the review.  I have seen and heard of some of the infamous "edit wars" that can occur in Wikipedia over the simple placement of a single word.  A controversy caused by the placement of a word is less the exception, and more a rule.  I believe that this is for the precise reason that Lauren indicates: Wikipedians are mostly non-experts; but you do not need to be an expert to have an opinion on word placement.

I also think that this is why "Wikigroaning" exists.  "Wikigroaning" is a term coined for the practice of comparing the word count of a regular, serious encyclopedia article with a pop-culture one.  The resulting groan you will experience is how this term achieved its name.  Or, do a Wikigroan with my favourite method: compare an article on serious subject on Wikipedia: http://en.wikipedia.org/wiki/Spanish_Civil_War (word count: ~14000)
With an article on the Star Wars Wookiepedia:  http://starwars.wikia.com/wiki/Wicket_Wystri_Warrick (word count: ~25000)

I don't know if I completely agree with Jess' point on the Sokal affair (sorry!): That if satire slips past peer review it is a sign of sickness in the discipline itself.  I think that I will go back to my first point, about a peer reviewer being caught off guard by a plagiarized concept, and say that the same is true here.  If you are not expecting to look for satire, it can be extremely hard to spot.  Especially if you are the one who the satire is being direct towards.  We can have a very large blind spot when it comes to our own ideas and beliefs, and satire can very easily slip right past us.  And oftentimes, the point of the satire is completely lost on those it is directed against.  As in the Sokal affair, where the publishers of the journal said that they did in fact read Sokal's article, but saw nothing wrong with it, it just needed to be better written.  When something like this happens, you can start to lose track of who is "trolling" whom…

Vanessa's post about the peer review process was quite interesting.  Hearing about her friend's experience with peer review makes me wonder more about the process as well.  How is it that her friend could find a paper fundamentally flawed, yet others see it as being fit for publishing?  The questions that Vanessa raises at the end of her post are all good points, and I think that there are no easy answers to any of them.  Hearing her friend's experience with peer review does make me start to see the more troubling implications of peer review (I am just speculating here): What if her friend really wasn't qualified to read the paper, and completely misinterpreted it? Or, even worse, what if she was the only one who actually read the paper, and found it flawed?  Not something that you would want to think about too much…

I like Cythia's point about the personal impact of peer review.  When you are choosing to publish a paper, it is really up to you on where you want to publish it and what kind of feedback you are looking for.  While I am sure that many academics would like to see their work in the most prestigious journal, it is really more about seeking the feedback and recognition that will help your ideas to grow.  It is up to the individuals who publish papers to decide on how they want to have their work recognized, and which peer review process works or doesn't work is really a choice that is made by them.

I found myself laughing when reading Eva's post.  I liked her assessment of the Sokal affair, and I found that it was in line with my own thinking: it only created a controversy, and did not really advance a new topic or enlightened change in the system.  What made me laugh was that Eva's post made me think of Sokal's hoax being assessed as if it were his paper being submitted (as a sort of meta-analyis).  Eva's summation of it not being a very original idea is what gave me a chuckle.  Eva also has a good point about peer review being flexible in regards to open vs closed review.  I agree that a combination of both would allow for more constructive reviews.  And I think this relates to Eva's point about author's being flexible in their choice of peer review.

Courtney's post about peer review and the full stomach reminded me of my legal work.  And that I think I had heard about her professor's research in my studies as well.  It is a well accepted fact of legal work that judges are humans too, and are subject to the same weaknesses as everyone else (so it is never wise to do anything to annoy them; including presenting your case at lunch hour)  Though Courtney relates this to reviewers having an empty stomach, and thus giving a negative review of a paper, I think that the observation that others have made about the possible bias of reviewers is a more probable concern.  An empty stomach is the least of your worries when you are dealing with humans with all the same biases and weaknesses that you have.  (I still think that attaching a cookie to your paper is an excellent idea).

Ghaddar makes a good point about nonsensical writings not being related to single papers, but entire disciplines.  Whenever I encounter the subject of pseudoscience, I always think about phrenology, and because I am a Simpsons fan, I think about Mr. Burns' response when Smithers says that "phrenology has been dismissed as quackery 160 years ago" Mr. Burns says, "Of course you'd say that...you have the brainpan of a stagecoach tilter!" (Simpsons, 3F06).  I think that this quote actually teaches us something about peer review: that it is useless when an entire discipline is caught in its own nonsense.  Ghaddar makes a good point in the usefulness of allowing public review of academic papers.  This might help to shed some light in the darker corners academia, where peer review would make less difference.

Well, this has been the longest post ever.  I hope that I was able to hit on some good points in response to each person, and that my peer review of their work helps them out.  Also, I have no idea what the correct format for citing Simpsons episodes is, so I hope that my attempt works.  And now, I hereby deem all of the previous posts to have passed my peer review and be suitable for publishing!