I started to write a short post about blind peer review, but it fragmented into a bunch of barely-related musings.
TL;DR: Something something peer review.
[A] With blind peer review, there is the danger that a referee will have heard the paper at a conference or read a draft on-line.1 On balance, I think the potential benefits of scholarly communication outweigh the potential harm of maybe breaking the seal of anonymity.
Here’s the normative situation as I see it: Authors shouldn’t hide their work from other philosophers, and they should share freely. The only expectation of secrecy is in actually submitting the paper.
So it is great to share drafts with people even though they might eventually be considered as a potential referee for the paper. If they give helpful feedback, then the paper you end up submitting will be better. Yet it would be vicious just to e-mail potential referees and tell them about the paper so that they know it’s yours.
In preparing a paper for submission, one tries to remove identifying references. It’s hard to do when one has written on the topic before, but one must try.
[B] As a referee, you’re not supposed to know who the author is. I recently agreed to review a paper, started reading, and halfway through thought that I had probably heard a distant predecessor of the paper at a conference. I couldn’t clearly remember who had presented the dimly-recalled conference paper, and I wasn’t sure this was the same author in any case. I mentioned my vague recollection to the editors when I turned in my report. The editors could decide how to weigh my fuzzy non-blindness in their decision.
I’m not convinced we should struggle to make the system more anonymous than this.
[C] In discussions of a recent hoax-paper broohaha, one point that several commentators made is that peer review is meant to catch incompetently bad papers but not fraudulent ones. Referees do not make a forensic review of the research to ensure integrity. For disciplines that rely on data, a paper that reports interesting results can often be worth publishing even if its analysis is a bit nutty.
For disciplines that do not rely on data, it is less clear what it means to be a fraudulent paper. Imagine that an author makes convincing arguments but makes them insincerely. This can make a contribution to the literature in a way that presenting intriguing but falsified data cannot.
[D] I once submitted a paper to a journal that practices triple-blind review.2 Shortly after submitting, I received an invitation to referee my paper. The instructions from the journal were adamant that if I knew who the author was— even if I was the author— I should decline without comment. Saying anything to the editor might contaminate the triple-blind evaluation. So I said nothing.
Given that the instructions to potential referees covered the case in which it was the referee’s own paper, this must be a regular occurrence. I wonder what would have happened if I had agreed to be a referee and had reviewed the paper.3
- I’ve blogged about this before, in the distant past.
- This means that authors do not know who is refereeing their paper, referees are not told whose paper they are reviewing, and the editors do not know the author when they invite referees.
- It would have been too brazen to laud my own paper and recommend acceptance, so I probably would have pointed out a few shortcomings and recommended that it be revised.
In the applied physics/mat sci world, journals are all single-blind, and this is the only way that makes sense. We all cite our own papers in the introduction, so even with the author’s names stripped out you can pretty much tell what group a paper came from.
Interestingly, though, it does create a bit of an issue with regard to objectivity. Let it be said that most of the papers I get to review are garbage, and I recommend rejection for probably 80-85%. But for the ones that are valid (i.e. the data supports the conclusions), what we struggle with is /importance/. Because science journals are hierarchical due to impact factor, one always says “yes, this is ok, but is it really worthy of APL?” And it becomes a problem particularly in that the very top tier journals like Science and Nature will readily publish the work of the most famous people, because of name recognition moreso than based on the quality of the particular paper. This sets up a feedback loop where such papers beget more funding, thus more work, and thus more papers in high impact journals. Not that these people don’t do good work, but that their prestige is certainly part of the evaluation.
In philosophy: Citation to an author’s own work can be in the third person, and some journals even encourage to just remove as many of those self-references as possible. Alternately, references to an author’s own work can be replaced with something like [redacted for the purpose of blind review]. The latter strategy has the problem that it may be clear who’s been redacted, if the author is someone who ought to be cited.
There is still an issue with deciding whether something is significant enough, but double-blind review is usually more-or-less possible and reduces (even if it doesn’t eliminate) the influence of an author’s prestige.