I’ve been sending articles out for review for a few years now, and I’ve also had the opportunity to review several papers for journals. Having been through each side of the ordeal, I’ve begun to form some opinions on the peer review process as a whole. This is something other political scientists have written extensively about. For example, Michael Allen recently addressed the issue on this blog, and Nathan Jensen recently wrote an
interesting piece about one of his paper’s journeys through the political
science peer review process. However, some of my recent experiences have prompted me to think about the process from the vantage point of the reviewer. Most of what I can recall reading on the subject tends to focus on the process as a whole, or discusses the process from the point of view of the submitter, rather than the reviewer. So I thought I’d reflect on my experiences and what practices I’ve tried to incorporate into my reviews.
1. Take reviews seriously: This might seem like an obvious point, and it’s one Jensen makes in his post, but for anyone who has been through the process you know you can’t always take this for granted. We’ve all gotten terrible reviews in some way shape or form—some of them are condescending in their tone, some of them seem to address topics or ideas that are completely irrelevant to the paper in question (even after much reflection), some of them are incredibly short, and some of them are just wrong. And these points just concern content. Some reviewers just don’t respond
to review requests from journals, and others don’t seem to get around to the
review until several months after the paper was submitted.
For the most part, all of these things are well within the power of the reviewer to control. I say “for the most part” because sometimes comments or suggestions made in a review may be wrong, but I’d like to believe reviewers are not dishing out incorrect information on purpose. And maybe sometimes people just aren’t communicating their thoughts well, thereby leaving the reader with no idea as to how those thoughts apply to their paper. Also, sometimes life does get in the way, causing unexpected delays. But, quite often, you as a reviewer can control these sorts of
factors.
The review process can and does put a burden on journal editors and reviewers—there’s no question about that. As Jensen notes in his post, senior scholars in particular can be swamped with review requests. But reviewing is probably
one area in which scholars can have the greatest impact on the professional
fortunes of other scholars. And I don’t mean to imply that reviewers should
fail to reject a paper just to provide a boost to someone else’s career. Not every paper deserves to be accepted or given a revise and resubmit, but editors rely to varying degrees on the advice given to them by reviewers. We also know that there are plenty of hard working people out there who face rejection after rejection, many of which are for really silly reasons. If you’re going to reject a paper, the author deserves to know why. A flippant one-line response is not helpful to anyone.
I typically try to get my reviews back within a month or two, if possible. I’m still pretty early on in my career, at a point where not many people know much of my work, so it’s probably pretty easy for me to meet this goal as compared to more senior scholars. On the one occasion that I’ve known in advance that I wasn’t able to get reviews back in a timely manner, I’ve communicated this to the editor. It’s easy to blow this responsibility off—I mean, this isn’t your work, right? You are taking time away from your own busy schedule to review someone else’s work. Still, more likely than not you are asking others to do the same for you.
2. Be clear and detailed: Along the lines of the previous point, a lack of detail in a review doesn’t help anyone. I once received reviews on a paper that ran the gamut from a single sentence saying to publish the article as is, to a rather lengthy, and in no uncertain terms, reject. As nice as it was to see that someone enjoyed your paper so much that they suggested accepting it, I’m not sure that such a brief reply helps. Truthfully, some of the more lengthy and critical reviews provided me with ideas for changes that I really do believe made the paper a better one.
More often than not, papers are going to be rejected, and reviewers should know this. Even if you like a paper so much that you want to see it accepted as is, or with minor revisions, there will usually be at least one other reviewer on a given paper, and it is quite possible that this other reviewer will have a very different take. No paper is perfect, so it can be helpful to know why a reviewer believes that the strengths of a paper outweigh any weaknesses. Especially when other reviewers are less supportive. It’s also helpful to be able to compare reviews to see if there are any common themes/complaints that are popping up across readers. Given the high rate of rejection, I think it’s important to remember that it’s not really just about the one or two reviews that the author receives on a given round of submission. Setting aside for a moment the fact that one person can get the same paper to review over the course of that paper’s journey through the review process, if reviewers are clearly raising the same concerns through multiple rounds of reviews that might be a pretty good signal that something needs to be changed. Sometimes these sorts of things are easy to spot from one set of reviews, and at other times they’re not, but the authors can’t tell if the
reviewer doesn’t provide the detail. Given the high probability of rejection, if you, as a reviewer, think that a paper’s strengths outweigh its weaknesses, a failure to highlight those very features could result in them being erased in future rounds of reviews.
I’ve gotten to the point where I offer a brief statement at the beginning of the review in which I explicitly state whether or not I’m recommending acceptance, rejection, etc. I also try to offer a brief summation of what factors I’m considering in making this recommendation. I also try as hard as possible to not waver in my judgment—if
you have “mixed feelings” or you “could be convinced” that a paper could be worthy of publication, don’t try to pass the buck to the editor and the other reviewers. Just recommend an R&R and move on. Nothing says you have to accept the second or third round of revisions, but given how competitive the review process is, this kind of waffling is effectively the same thing as rejecting the paper outright, and it’s probably not all that helpful to the author.
Beyond the introduction, I also try to clearly identify each individual point that I want to address. And yes, this often involves an enumerated list and/or bullet points. To some extent I think this is just a matter of style. However, it can be difficult in a meandering review to discern the exact points that the reviewer wants you to address, and as much as it is the author’s responsibility to take their revisions seriously,
it is also the reviewer’s responsibility to make clear those points that need addressing. Furthermore, we hold ourselves to a pretty high standard when it comes to organizing our thoughts and arguments in the context of an academic paper, so why shouldn’t it bethe same in reviews? It is, after all, all part of the same, broader, knowledge-generating process.
3. Offer constructive criticism: Again, this seems like an obvious one, but you can’t take it for granted. We spend an awful lot of time in college and graduate school learning/practicing how to pick apart other people’s arguments. So while pointing out the flaws in someone’s article is certainly an important part of the review process, I also think it’s equally important to be able to point out its strengths.
For the most part, my experience on this point leads me to believe that most reviewers are pretty good about this. However, I have received reviews that are highly critical, with little in the way of constructive commentary. “Complainy” might be the most appropriate made-up adjective to describe these reviews. If you as a reviewer think that there are some deficiencies with an argument or method, you need to spell that out for the author. If such deficiencies were clear to all, then they probably wouldn’t be in the paper to begin with. Further, I think the degree to which any flaw is damning to the overall article is contingent upon the availability of feasible alternatives. If you don’t like a measure that the author uses, don’t just complain and then move on. At the very least, ask the author to clarify whether or not superior measures are actually available. If you are aware of one that you believe is superior, then please share it and
explain why you believe it to be better suited to the analysis at hand. Again, this is a collaborative venture, and part of the review process is being able to access the knowledge of other scholars.
I would also say that personally, as a junior scholar, I really do appreciate a good review when I get one. Some people have clearly taken the time to think about my work, and I appreciate it. As I said before, I’ve been fortunate enough to get some great feedback that I have subsequently incorporated into my work, and I do think the work is better for it. Criticism accompanied by constructive feedback and advice generally seems to be more helpful than merely pointing out the faults of a particular project.
4. Be transparent when it comes to your own limitations: As I stated above, sometimes reviewer comments are just wrong. Sometimes this is an honest mistake,
and the reviewer is trying to offer sincere advice or criticism, but just gets it wrong. But, as an author this can be incredibly annoying—particularly if some factually incorrect statement could have been a major contributing factor to your paper’s rejection. If I’m reviewing a paper and I’m unclear about a measure, statistic, or method, I typically try to be entirely transparent when it comes to my own ignorance on the matter. Now I’m pretty confident in the substantive and methodological training that I received in graduate school, but this doesn’t mean I’ve mastered every analytical approach or estimation technique. Nor does it mean that I can’t learn even more about those topics and methods that I have been exposed to. Indeed, I’ve learned a lot over the past seven years, but as this SMBC comic so nicely conveys, sometimes expanding your knowledge base can raise new questions. So, when you think you don’t know, or you know you don’t know, just say, “I don’t know.” Feel free to even pose a question on the issue to the author in your review. After all, there is a certain irony in the idea that certain people, having devoted themselves to an industry that (ostensibly) is based on the discovery and accumulation of knowledge, would be so reluctant to utter those three words.
I’m sure senior scholars may have a different take on some of these issues. As I said, I’m still very early on in my career, so I’m not facing the same kinds of pressures from as many sources as they are. I don’t have to deal with dissertation committees or a never ending stream of review requests. Still, I don’t mind reviews all that much. They can provide me with some insights into what people in my general area of expertise are working on, and they often lead me to read works that I’ve not previously noticed or thought much about. But perhaps my views will change given time.
