Contributing to Public Goods: My 20 Rules for Refereeing

Last updated on January 27, 2012

The development economics blogosphere has been abuzz with talk of refereeing lately. Here are some words of advice from Quarterly Journal of Economics editor Larry Katz in an interview with Berk Özler on the Development Impacts blog, here is David McKenzie on the same blog, and here is Chris Blattman.

I cannot possibly claim to be among the best referees, but by my count, I have refereed 56 57 papers and two book manuscripts since 2005, and I do take pride in my refereeing, which might explain why I was asked to become associate editor at the American Journal of Agricultural Economics for 2012-2015.

As such, I figured I should chime in with my own advice about how to referee papers. I cannot say I always follow every single one of the 20 rules that follow but those are, by and large, the rules I try to live by as a referee. Some of those rules are derived from a similar list by Chris Barrett, who gave the students in his graduate empirical development micro class a list of such rules.

Because the following list is highly idiosyncratic, I would be very happy to hear about your own rules in the comments. And because this is a specialized post, I’m placing the list under the fold.

When You Receive a Refereeing Request

1. Read the title and the abstract of the paper, determine whether you are a good fit for the paper, and respond immediately. You’d be surprised the number of times I’ve been told by journal editors that some people simply do not respond to refereeing requests.
2. Sometimes you might think you are not a good fit for the paper. But at the end of the day, the editor asked you to referee the paper for a reason. Often, the editor wants your advice as a nonexpert, to see if the paper makes sense to (and will be cited) by people who are not experts on the topic.
3. Unless their fit with the paper is especially bad (e.g., a theorist is asked to referee a paper without any theory, or an empiricist is asked to referee a theoretical paper), graduate students and assistant professors should rarely, if ever, decline a refereeing request. “I’m too busy” is true for everyone in this profession, so unless you and your significant other are expecting soon and you are teaching three or more classes, you really have no excuse to decline a refereeing request. If you are asked to serve as associate editor, co-editor, or editor of some journal before you get tenure, you can become a bit more choosy in the refereeing requests you choose to respond favorably to.
4. That said, if you have refereed the paper before, or if there is a conflict of interest (i.e., the paper was written by one of your colleagues, one of your coauthors, one of your students, or one of your advisors), you should let the editor know about it and give her a chance to reconsider.

Once You Have Agreed to Referee a Paper

1. Don’t wait until it’s too late to do the review. You can certainly wait until you get a first reminder from the journal that your review is due, but do not wait until the second such reminder. Really, the earlier you submit your review, the better for everyone involved — if there is one thing I’ve learned from taking Ted O’Donoghue‘s behavioral economics class, it’s that you don’t know your (future) self as well as you think you to. While I was on research leave in 2009-2010, I once received a refereeing request around lunch time; by the time I went home for dinner, I had submitted my review to the editor. Generally, I read the papers I have to referee on flights.
2. Once you decide to get started on your review, read the title, the abstract, the introduction, and the conclusion of the paper. Do you have a good idea of what the authors are doing? Perhaps more importantly, are you convinced that it’s a worthwhile topic? If it’s an empirical paper, can you understand from the tables what relationships the authors are after? If you answer “No” to any of those questions, you should encourage the editor to reject the paper. This may sound harsh, but before submitting, authors should work hard on the “sell” of their paper, i.e., on convincing the reader that the paper is worth their time. And generally speaking, the profession could certainly benefit from better writing. It is true that a groundbreaking good idea that suffers from a bad sell deserves a second chance. The two, however, are rarely orthogonal to one another.
3. A corollary to the previous rule: As a referee, you don’t “accept,” “give a revise and resubmit,” or “reject” a paper, you merely give your advice as to what’s to be done in a cover letter which the editor will not share with the authors. So to claim that you’ve “rejected” this paper and “given a revise and resubmit” to that paper seriously overstates your role in the editorial process. More on the cover letter in a minute.
4. If you believe the paper should be rejected after reading the abstract, the introduction, the conclusion and, if the paper contains any, the tables, write up your review. Make sure you explain clearly why the paper is a nonstarter. More importantly, offer some constructive ways in which the authors can make their paper ready for prime time. Do so in less than two single-spaced pages, and skip the tedium (e.g., typos, missing references, etc.)
5. If you believe the paper should be given a chance after reading the abstract, the introduction, the conclusion and, if the paper contains any, the tables, read the paper thoroughly, then write up your review. Do so in as much space as you need, and do not skip anything, even the tedium.
6. A good review should start by summarizing the paper in one or two short paragraphs. Do not paraphrase the abstract! This is your chance to give your point of view to the authors as to what the paper really does and sometimes, your interpretation of what matters in the paper differs from theirs. Then, offer a numbered list of general constructive comments — those are the potential deal breakers, i.e., the things the authors have to do to get the paper published — and a numbered list of specific comments — those are the small things you’d like the authors to do in order to improve upon the paper. If you are going to recommend rejection, you can skip the specific comments. These two lists are a clear indication of what’s negotiable or not, and numbered comments increase efficiency — the authors can refer to specific comments more easily when responding to your review.
7. Make sure you give the authors suggestions as to how to address your general (i.e., non-negotiable) comments. If you don’t think the authors can reasonably address your comments, you should recommend rejection so as to not waste anyone’s time.
8. What’s a constructive comment? One that can actually be addressed, and one that is not demeaning to the authors. When in doubt as to whether a comment will be seen as demeaning, err on the side of being nicer.
9. That being said, you are not a coauthor of the authors, so you should not push the authors to write the paper you would have written on the topic. Moreover, it is bad form to push the authors to cite all of your papers on the topic. Encourage them, however, to cite the ones that are truly germane to their work. And encourage the authors to polish their writing. More specifically, if you can think of more — or better — motivations for their work, do not hold back — this will enhance the paper’s citation potential, and the editor will be grateful for that since for her, citations are the coin of the realm.
10. Once your review is written up, it’s time to write your cover letter to the editor. That letter is confidential and will not be shared with the authors, so you can be as candid as you want in it. Specifically, you make your recommendation (“reject,” “weak revise and resubmit,” “strong revise and resubmit,” or “accept”) to the editor, and you give your arguments as to why you make this specific recommendation. Do not cut and paste from your own review — rather, this is an occasion for you to be as frank as you want about the reasons behind your recommendation.
11. The cover letter is also your chance to flag potential ethical problems. For example, if you believe the authors are “double dipping” (i.e., publishing slightly different versions of the same paper), if you believe the paper is simultaneously being reviewed somewhere else, or if you believe the authors have plagiarized part of their paper, the cover letter is the place to mention it.
12. Once your review and your cover letter are written up and converted into .pdf documents, submit them to the editor. There is just no good reason to sit on them. As long as you write clearly and cogently, no one will care about your prose in a referee report or cover letter.

Once You Have Submitted Your Referee Report

1. The more considerate editors will let you know the decision they have taken on the basis of your review. Often, they will agree with you. This is especially likely when you recommend rejection, as several editors need the majority or all of the referees to recommend a “revise and resubmit” before they ask the authors for a revised version. Sometimes, however, the editors will go against your review. Don’t take those cases personally. But if this happens often, you should take some time to think about why it does.
2. You will eventually meet many of the authors of papers you have refereed. Resist the temptation to out yourself as a referee, even if you think you were especially kind as a referee — you never know whether your well-intended constructive remark has been interpreted as an insult by the authors.
3. Think carefully before breaking any of these rules. Many seem to believe that they can get away with not responding to editors, with taking an unduly long amount of time before submitting their reviews, with being insulting to the authors of the papers they referee, etc. You get the idea. Those people forget that editors will remember such bad behavior and keep track of who is a good “citizen” of the profession and who is not.
4. The converse of the previous rule is that good refereeing is a very good way to build a solid reputation for good “citizenship” in the profession — something that is overlooked all too often in a profession supposedly full of smart people who understand incentives. Not only are good referees more likely to get asked to become associate editors, but the editors of the journals in your field are often the ones being asked to write your external (anonymous) letters of reference come tenure and promotion time. Don’t forget what Tyler Durden told the police chief in Fight Club: “We cook your meals, we haul your trash, we connect your calls, we drive your ambulances. We guard you while you sleep. Do not. F***. With us.”