Can Double-Blind Peer Review Be Reformed?, Ctd

by Freddie deBoer

Lots and lots of great reader responses to my post on peer review. Here’s a sample. Many readers ding me, correctly, for over-generalizing: much of academic peer review is not double-blind.

Interesting post, and thanks for the link to the Rossman piece. I don’t have a great story for you on peer review, but wanted to add that most natural science journals actually use single-blind. This may be changing, for example this article which also provides some support for my assertion.

Another:

I am a senior academic, so I have been on both sides of the peer review process, and I have counseled younger colleagues who have voiced complaints similar to those outlined by deBoer. Although some of the problems mentioned are insoluble, others could be remedied or at least reduced by clearer policies on the part of professional associations within various disciplines and academic journals.  These include:

1. The professional associations should establish and publicize standards governing professional conduct in undertaking and completing reviews. There may be no way to enforce these standards directly, but publicizing them might be helpful, in part in helping editors call wayward and/or tardy reviewers to account. The associations could also collect and publicize data on the period from submission to publication for various journals. These data might inform authors of what journals to seek out and which to avoid.

2. Academic journals should set a deadline for the completion of reviews, communicated to the authors and to the reviewers ahead of time. Six weeks seems a reasonable period to me, but I am not wedded to that. If the reviewer cannot guarantee a review within the prescribed period, another reviewer should be selected. Reviewers who fail to meet their obligations should not be asked by the journal to review future articles, and journal editors should be proactive in pressuring tardy reviewers. Those submitting articles should be free, after three months without a response from a journal, to submit his/her work to another journal as well.

3. Using the same reviewers for the “revise and resubmit” review as for the original review, so that new objections/concerns are less likely to be raised.  The reviewers should be asked simply as to whether the problems they identified have been adequately addressed or not, and a final determination should be made.  Multiple “revise and resubmits” should not occur. If the author’s work doesn’t merit publication after he/she has had a chance to revise it, the author should be told so (and be free to submit to another journal).

Similar thoughts:

Necessary background: I’m a research meteorologist, almost 21 years past my Ph.D.  I don’t have as many peer-reviewed papers published as anyone would like me to have but the ones I do have appear in 5 different journals published by the American Meteorological Society (AMS), 2 published by the American Geophysical Union (AGU), and another published by the European Geosciences Union (EGU).  Over the years I have served as a reviewer for most of the journals in which I’ve published, and for at least 1 AGU journal in which I have not yet published.  The AMS or the AGU are usually the primary professional society with which U.S. based meteorologists who are not forecasters are affiliated.  (The National Weather Association (NWA) is often the primary for forecasters.). …

I would take serious issue with your statement that “Peer review, at the vast majority of credible journals, is built on a double blind system.”  I have never run into a double-blind review system, either as a writer or a reviewer.  I do not think I have ever heard a colleague talk about being part of one, either.  So this may be very seriously field-specific.  The AMS & AGU journals have, to the best of my knowledge, very high credibility in the meteorology, atmospheric sciences, oceanographic, and related fields. I’ll leave exploration of metrics to you, if you’re interested; all that matters to me is the opinion of my colleagues & “bosses”.  I believe that I have heard of trouble with some editors once one starts getting into topics related to climate change, but as it’s well outside my direct experience I’d prefer to leave it at that.  I’d refer you to the blogs of Prof. Roger Pielke, Jr. or Prof. Judith Curry if you want to explore that further.

You have my sympathies! I just got a paper published after 4 revisions.  Round 1 came back, and we (thought we) revised reasonably based on the comments & concerns; we certainly took them seriously.  Round 2 came back; Reviewer 1 felt we’d blown off his comments, so insisted more forcefully and with more detail, while Reviewer 2 was satisfied.  We did a lot more work and rewrote.  Round 3 came back; Reviewer 1 was satisfied, but Reviewer 2 no longer believed us!  We did a lot more work and rewrote.  Round 4 came back; Reviewers 1 & 2 were now satisfied, and Reviewer 3 (who had been a supportive but rigorous presence throughout) had a few more suggestions and questions.  So we rewrote a little bit more, and finally were done.

Throughout this process we had the same non-anonymous editor, who was patient and understanding and did an awful lot of work IMO.  I’ve definitely had other editors go the extra mile, too.  (And some who weren’t so diligent, but <shrug> not everybody can be excellent at everything.)  With the AMS journals, you always know who your editor is, and I think editors may even stay with “open” papers after their term as editor ends.  That openness makes a big difference, I’m sure.  My most recent experiences with an AGU journal also involved anonymous reviewers but a known editor.  Furthermore, you *always* know what Reviewer 1 told you to do, what Reviewer 2 told you to do, and what the Editor told you to do.

Another:

I guess the norms of peer review differ somewhat by discipline. I’ve been writing papers for biomedical journals for a little over a decade now, and reviewing regularly for nearly as long, and the norm for me has been single-blind review: the reviewers know the authors, but not the other way around. I can only think of one journal for whom I’ve reviewed that was double blind.

Anyway, this may seem hopelessly idealistic, but the standard to which I hold myself when reviewing is really quite simple: I pretend it isn’t blinded at all. I don’t write anything in a review that I wouldn’t sign my name to and be willing to have published, nor anything I wouldn’t say to the authors in person at a conference, in a lab meeting, or over a beer. More importantly than keeping things civil, professional, and non-petty (though that is all very important!) it forces me to make sure that I have my own facts straight and that I can back whatever I say up.

As to my experience of being reviewed while it has been generally fair and constructive for me, it is very clear that my own standards are not universally applied.

Personally, I think doing away with anonymity in peer review altogether would be preferable.

Still another:

I’m a Ph.D candidate and I read your post, nodding my head along the whole time. In my limited experience, the process of peer-review (the depth of review, critique, and the ultimate decision to approve or project) is highly variable. You’ll get two reviews and one will ask for minor changes and the other reviewer will want the entire paper re-written so as to incorporate a topic you don’t even mention (“how could you not mention NGOs?”) “Well because this paper 1) isn’t about NGOs and 2) NGOs don’t apply here and 3) we don’t have the expertise to study or comment on NGOs”.

But for me, the time factor and the accessibility factor are the much bigger issues. Six months to two years to get something published. An adaptive institution this is not. As I’m writing, Ghostbusters (1984) is on TV and Egon just said “print is dead.” I laugh every time. I’m aware there are alternative outlets to traditional publication, but until we have a movement of young academics willing to challenge the norm, or academic institutions with the foresight to change the rules of the game, what are we supposed to do. I know I’m trying to get my stuff into regular ole high-ranked traditional journals. And what about when you have to settle for a lower-tiered journal. It still counts, but is anybody reading it?

More to come.