Can Double-Blind Peer Review Be Reformed?

by Freddie deBoer

Peer review, the vetting of academic writing by subject-matter experts, is an essential element of academic progress. But the peer review process is also dysfunctional, sometimes out-and-out broken– and that brokenness stems from the well-meaning ideals peer review is meant to protect.

Gabriel Rossman wrote a fantastic piece illustrating the difficulties with peer review, and I highly urge you to read it if you are at all interested. I think Rossman is perfectly right in arguing that it’s the self-same people who complain about peer review as authors who often turn around and exemplify its worst tendencies when reviewing. As a peer reviewer myself, I try to always place myself in the position of the author, and in particular, I try never to review an article by thinking about what I would have done differently, but rather to ask if there are glaring theoretical holes, methodological errors, or problems with presentation. Far, far too much peer review becomes a matter of reviewers telling you what you should have done rather than making the work you did write better. As Rossman writes,

Rather, fixing peer review has to begin with you, the reviewer, telling yourself “maybe I would have done it another way myself, but it’s not my paper.” You need to adopt a mentality of “is it good how the author did it” rather than “how could this paper be made better” (read: how would I have done it). That is the whole of being a good reviewer, the rest is commentary.

But the particular problems with how peer review happens are less important than the basic structural problem. The fundamental issue is this. Peer review, at the vast majority of credible journals, is built on a double blind system. In order to ensure that a big name academic’s big name doesn’t get inferior work published, and so that reviewers can respond honestly without fear of retribution from people with disciplinary and institutional power, neither author nor reviewer knows the other’s name. That’s a sound idea, but it has a perverse effect, particularly given how important publishing is to an academic career. Reviewers and editors have enormous power to make or break careers; one major journal article could mean the difference between launching a professional career and having that career die on the vine. And with no knowledge of who exactly is responsible, we’re left with unaccountable power, which is never a good idea even when people are trying their best and mean well.

Though I’m talking about peer review, it’s also worth saying that this can apply to the whole academic publishing process. You might know the names of the editors you’re working with, but going public with complaints, in the event those complaints are fair and warranted, could be disastrous if you aren’t established or tenured.

For the better part of a year, I’ve been wrestling with an onerous, deeply unpleasant review process at an established journal. All editing, of course, is to some degree an unhappy business. But I’ve gotten peer reviews, even for rejections, that have been smart, fair, sympathetic, and constructive. This is not one of those times. The initial reviews were actually quite positive, though they came months after submission. I made those changes without complaint. Since then, there have been additional requests for changes again and again, each time separated by a period of months. I’ve made those changes to the best of my ability, but the requests have sometimes been unclear, rude, or worse.  These changes have been, at times, plainly contradictory of previous requests, in the most direct and unambiguous sense. It’s frequently unclear what requests for changes are coming from the peer reviewers and which are coming from the editors, or if the peer reviewers are even still involved in the process at all. At some point it became clear that I was being edited by several different editors and that these editors were not communicating with each other. I would receive questions, answer them, and get the same questions again, months later. And so on.

Is it possible that I’m just wrong, about everything, and they’re just right? Sure. But the fact is that if I was right even hypothetically, there would be no way for me to fix the problem. I don’t know who the reviewers are, so there’s no way to expect individual accountability. And as someone who lacks the benefit of employment, tenure, or prestige, speaking out publicly about the journal by name would be professional suicide. Even this missive, in and of itself, is likely to be seen as violating proper academic decorum, even though there’s no way to tell what journal I’m talking about. Under those conditions, how could we expect fairness or accountability? I think a lot of peer reviewers do a great job, and for no money. So do most journal editors, who if they are paid, are paid a pittance in most fields. It’s a lot of work. But I don’t know how to deal with problems with peer review and editing when the professional stakes are so high, the personal accountability so low, and when notions of collegiality and respect prevent people from making complaints like this one.

What makes all of this worse is that the double blind system was designed as a bulwark against the corrosive effects of power imbalances. The whole idea is that an unknown graduate student should have the same chance to publish in the biggest journals as the most respected academic celebrity. But that tenured prof can write books, publish research on his or her own web site, and be sure to receive respect and fair process from editors. Younger academics need to have their work vetted if they want to build a career, and they have to do so without complaining. The right to register grievance when grievance is warranted should be available to everyone, but the current structure of academic publishing makes that right unavailable to the most vulnerable.

I’m still plugging away on the article, but the communication has become so acrimonious that I’ve never represented the article on any of my professional documents and have essentially written off ever seeing the piece get published. Which is a shame, because I think it’s a good piece, as the reviewers did, to say nothing of the dozens of hours I’ve spent over the past year writing, researching, and revising the piece. That time represents a major opportunity cost at a critical juncture in my work and my life. And with the review process at many journals being so slow, it’s not reasonable to expect that I could withdraw the piece, get it published elsewhere, and get appropriate credit for it in time for it to help me on the job market.

Are you an academic who’s been caught in review hell? Are you a peer reviewer or editor who thinks your role is misunderstood? Or am I just full of it? Write in to and let the Dish know.

(Thumbnail Photo by Nic McPhee)