« The Queen | Main | Assorted links »
How to be a good referee
Here are my tips:
1. Assume that no referee reports are truly anonymous. It is fine to be critical but always be polite.
2. Unless it is immediate junk, read the paper once and return to it a week later with deeper thoughts and a fresh mind.
3. Your report should not assume that the editor has a working knowledge of the paper in his mind.
4. Respond within a month. First it is considerate to the author. Second, the less "fresh" the task is, the more painful it will be.
5. A properly critical and useful "accept" report is harder to write. Don't look for excuses to quickly reject a potentially good paper.
6. The editor might have chosen you as referee for a reason. You need not go along with the editor's grand plan or desired outcome, but be aware it may exist.
7. Don't check the references to see if you are cited.
Here is a short article full of good advice. Here is the longer piece (which I have not read) on how to publish in top journals. Thanks to Elaine Hawley for the pointer.
Barkley Rosser, a frequent MR commentator who has sent me JEBO papers to referee, may comment on this post but not on my adherence to these standards, especially #4.
Posted by Tyler Cowen on October 24, 2006 at 09:54 AM in Education | Permalink
Comments
I like 1-6 but disagree with 7. Part of the referee's job is to make sure that the paper properly references the literature. Given that referees are not assigned at random, the relevant literature likely includes one or more papers by the referee. Doing this part of the job is especially important in literatures that span disciplines or where strong "schools" exist within economics.
I would also add that you should not include your recommendation in the report itself. It should go only in the cover letter to the editor. This gives the editor the most flexibility. Being an editor is largely, though not entirely public goods provision; as such, editors deserve some flexibility.
Jeff
Posted by: Jeff Smith at Oct 24, 2006 3:03:51 PM
In the longer piece (on how to publish in top journals) the following advice is given:
"Avoid writing comments on others papers".
My co-authored research in EJW confirms on the decline in commentary confirms that this is good advice CURRENTLY (and shows that in the 1960s and 1970s it would have been bad advice) for getting into top journals in economics:
www.econjournalwatch.org/pdf/CoelhoetalEconomicsInPracticeAugust2005.pdf
My subsequent co-authored research in EJW sheds light on why top journals have curtailed commentary:
www.econjournalwatch.org/pdf/CoelhoMcClureEconomicsInPracticeMay2006.pdf
Just as "nature abhors a vacuum", the economics profession abhors the absence of intellectual exchanges in the form or note, comments, replies and rejoinders; ergo EJW.
Posted by: jim at Oct 24, 2006 4:00:48 PM
8) Any time Dwyane Wade misses a shot from within 10 feet, call a foul.
Posted by: BillWallace at Oct 24, 2006 4:33:55 PM
Well, Tyler has put me on the spot on this one,
so I guess I shall comment. I start by noting
that indeed Tyler is a high quality referee, when
you can get him to do it. One of the problems
with high quality referees is that they are busy
people and lots of people want them to do referee
reports. A careful editor tries not to hit on
such people too frequently; refereeing resources
are scarce, although I know a couple of economists
who referee as many as 100 papers a year, and
these are people who even without doing that are
busier than probably 99% of the readers of this blog.
(Once when I had an editor dumping too many papers
on me I said, "if you are going to keep doing this,
then put me on your editorial board." He did so,
and then hardly sent me any papers thereafter.)
I largely agree with Jeff Smith, and to a lesser
extent with jim. Don't know nothing about Dwayne
Wade, so no recs from me about him.
Point 1. This is a complicated matter. It depends
on how in with the in crowd of the literature in
question you are. If you are in, then you are
probably not anonymous. However, you may be anon.
Anyway, one should not be impolite, ever.
I would, however, say that a referee should be
decisive, even forceful. It is editors who really
must be nice, and actually, even if many authors do
not think so, generally we are sympathetic to authors.
That most of us are rejecting most of the papers that
are submitted to us does not mean that we are sitting
around sneering and gloating at those authors. Far
from it, especially in close call cases, and there
are many of those, many hard decisions. (Of course
there are some people who act like assholes and piss
us off; more on that below.)
Regarding this decisiveness business. You will be
doing the editor a favor by making a definite recommendation,
even if that is revise and resubmit. I really get ticked
off at reports that say "well, this paper could be accepted
or maybe it should not be accepted." Duh. Also, you will
be doing the editor (and actually the authors, although
they may not thinks so) a service if when you recommend
rejection, you give strong and forceful reasons for doing
so. That gives the editor a clear signal and reason for
following the recommendation. However, in such cases, there
is all the more reason to be polite. No need to engage
in gratuitous insults, e.g. "this author is really an
idiot" (even if they appear to be).
BTW, the flip side of this anonymity issue is that even
with double blind refereeing often the referee can find
out who the author is, and this is getting easier and
easier with googling, websites, posted working papers,
and so forth. When I took over editing JEBO five years
ago, I switched it from single to double blind, but many
have argued I should not have, or should switch back,
because of this ease of discovery. I keep double blind
out of at least an illusion of fairness and even-handedness.
I would also say that one should be less quick to seek out
the identity of the author than one should be to look at
the References, which I agree with Jeff Smith one should
do (although one looks petty to an editor to demand that
one's own papers get cited, except when there is a really
damned good reason that they should be). I have even
had people tell me that papers that are not already
publicly available must be "no good" and should not be
published. I do not agree with this argument. I have
also been on the wrong side of misidentification, with
having papers rejected at leading journals with referees
saying, "oh, this is obviously another crappy paper by
so and so," who just happened not to be me.
Point 2. Reasonable advice.
Point 3. Wise advice. It is generally a good idea to
do a summary of the main arguments in the paper and
how it fits into existing literature (more reason
to look at the damned References), whatever one is
recommending, just to make sure the editor has some
idea what the heck is going on here. Needless to say,
the issue of how it fits into the existing literature
is very important, as most journals are looking for
some kind of "value-added" or innovativeness in the
paper. That means showing, if one is making a favorable
recommendation, how the paper is succeeding at this
by noting what it is doing that has not already appeared
in the literature. And, of course, this can be grounds
for recommending rejection: there is really nothing new
in this paper, just a rehash of the same-old same-old.
Point 4. This is in general a sticky wicket. I would
note that the linked pieces on refereeing and writing
papers make some curious remarks in this regard. Thus,
the refereeing one says one should not send a report in
too soon or one may get a rep as someone who can get sent
lots of papers (if an editor starts doing this to you,
it is a reasonable reason to beg off refereeing some of
them, or alternatively, to demand to be on the board of
editors if you so wish). The writing one says, "don't
make an inquiry to an editor until at least six months,"
although an earlier inquiry to make sure the editor has
received the paper if you have received no acknowledgement
of said receipt is OK (not noted in that link).
This latter implies what is in fact probably the actual
current norm in the profession. You will look good, but
not "too good," if you get it in a month to four months
after getting the paper. After six months, you are starting
to get old and the paper is beginning to stink. Editors
know that authors will start bugging them and will have
good reason for doing so on the basis of the rarely
stated norms out there. So, do try to get it in by six
months, but it really is better to be faster than that.
There is a more serious problem here, one for the
economics discipline as a whole. Based on this logic of
"I don't want to be too quick or I'll be a sucker," we
have had a kind of prisoner's dilemma problem that has
led to this creep towards longer "first response times."
Although I know people who claim this is optimal based
on signaling and efficiency arguments, I think it is a
bad social norm that has evolved over time. Other
disciplines do not have such long periods. I have
both published and refereed in physics, math, comp sci,
and biology. In all of those the norms are much shorter,
like two weeks. It is a major embarrassment, and I
regularly get berated by people from some of those
disciplines over our scandalous practices in this regard.
BTW, most journals now not only accept electronic
submissions but much prefer them, with some not even
allowing hard copy submissions at all anymore. My
observation is that the general shift to electronic
processing seems to be shortening these times some.
Anyway, I would recommend not waiting a full six
months.
Another btw, it is much better to tell an editor up
front that you cannot referee a paper than to take
it and sit on it forever, especially if you refuse to
reply to inquiries! This latter is exceptionally
bad form.
Point 5. This is a subtle point and one that may
depend on the journal. Very conventional journals,
especially general ones with a committee running it,
are more likely to be pleased to reject a paper
because it has some minor technical mistakes in it.
The fine art is making a judgment about how really
innovative and interesting a paper is. If it really
is innovative, the minor technical mistakes can usually
be fixed in a revision process.
Point 6. Definitely valid. Often an editor is
looking for a balance of views or perspectives,
a theoretician and an econometrician, or someone
who might be favorably disposed to the paper and
someone who might not be (this makes things easy
for the editor if they agree).
Of course a lot of
this will depend on how much an editor knows about
both you and the topic. The more an editors know
about the topic, the more likely it is that they
have a clear idea what they are about when they
approach you. However, if they are largely ignorant
about the topic, it is very likely that you have
been selected because you are in the References
but not the Acknowledgements.
There is also the matter here of whether or not to
accept the request. Yes, it is a public service to
referee, and I remind that I know people who are
very busy and also referee as many as 100 papers
per year. Nevertheless, there are acceptable reasons
for refusing. One is that you really do not know
much about the topic, although again the editor may
be aware of this and wants you for some particular
knowledge you have of a particular aspect of the paper.
Another is that you are too busy. Now, here you
must be careful. You can get away with this if you
are a journal editor or coeditor yourself. You
might also be able to if you are an associate editor
for one of the top 4 journals (AER, JPE, QJE, Econometrica).
Also, if you just got the Nobel Prize, well, if that
is the case the editor was probably an idiot to send
you the paper in the first place unless he is your
closest bosom buddy whom you owe big time or whatever.
And, of course, if the editor has recently sent you
a whole bunch of papers. I would say that three in
the last six months is prima facie "too much," unless
you are a member of the editorial board.
However, let me issue a very serious warning, and I
am not as worked up about this as some other editors.
The biggest no-no you can possibly pull off is to
refuse to referee on grounds of "being too busy"
(unless you are in one of those special cases big time)
if you have recently published in the journal. This
is viewed as really despicable and beneath contempt.
Presume that if you pull that you will never publish
again in that journal as long as that editor is in
place, and quite possibly never again even after that
(as former editors tend to lurk about and get called
on for advice and inform new editors of blacklists of
known plagiarists and all around jerks... ).
Point 7. Again, I agree with Jeff Smith and not
Tyler on this one. However, I definitely agree that
one should be very cautious about suggesting that
authors should cite one's own work. There are
definitely circumstances where this is legitimate
and appropriate. However, be careful and moderate
in doing so, and I would generally suggest that some
other additional citations be suggested as well. Both
editors and would-be authors tend to quickly develop
a low opinion of someone whose main recommendation
is to cite "me, me, me, and me again!" endlessly, and
there are unfortunately plently of referees who engage
in this sort of unprofessional crap.
There is generally a lot of good advice in the linked
sites that Tyler has provided. Check them out.
Some final comments. One of the responses that editors
are doing in the face of both rising submission rates
and the scarcity of refereeing resoucrces issue is to
reject papers upfront without sending them out for
refereeing. Papers on how editorial processes work
seem to be very mixed on this, some approving and
some disapproving.
I have personally had to do this,
and am now, frankly, rejecting about half of submitted
papers upfront without sending them out. Sometimes I
get authors who complain about this and think they
were treated unfairly. However, I would note that, even
though you may feel annoyed, embarrassed, humiliated, or
whatever, the editor is probably doing you a favor. The
editor has made a judgment that your paper is probably
not going to be accepted, no matter what. So, you do
not waste a long period of time (possibly well over six
months if you get into revising and all that, sometimes
much more than that, although we as editors really get
embarrassed about such situations) going through what
is ultimately going to be a doomed process. Especially
for junior faculty who are on a track to go up for tenure
at a particular time, time is valuable and the ability to
turn that sucker around and get it back out to another
journal quickly is something not to be resented or sneered at.
A more general point, although some of you may view this as
"editor special pleading," I would strongly recommend against
complaining about or attempting to undo a clear rejection
decision. First of all, you are very unlikely to succeed,
unless there has been some really serious mistake or evidence
of foul or unethical conduct or whatever. Editors do not like
to undo their decisions, even if the referee reports have all
kinds of problems in them. Most often, the editor will simply
reject your paper again, but on different grounds, assuming
you had any decent grounds to complain about in the first place.
Also, it pisses editors off, especially if you come across as
all whiney and complaining, unless, again, you have some really,
really strong case of total stupidity or foul play or unethical
conduct. Even in many of those cases, you are better off
making any revisions quickly that might have been suggested
by referees that are reasonable and then getting that paper off
again to another journal. I know especially for newer faculty
(and some egomaniacal older ones too), being rejected is hard
and there are scads of incompetent and poorly done referee
reports out there. Be like a duck and view these rejections
as like water that should just roll off your back, no matter
how "unfair" or "stupid" or whatever. Get past it and move
on to the next journal, the sooner the better.
Posted by: Barkley Rosser at Oct 24, 2006 6:34:53 PM
Barley wrote:
"I think it is a bad social norm that has evolved over time. Other
disciplines do not have such long periods [for reviews]. I have
both published and refereed in physics, math, comp sci,
and biology. In all of those the norms are much shorter,
like two weeks. It is a major embarrassment, and I
regularly get berated by people from some of those
disciplines over our scandalous practices in this regard."
I agree, but I'd take it a step further and call it, for lack of a better term, BULL****, given that there are sometimes impacts upon tenure decisions and promotions.
Posted by: jim at Oct 24, 2006 9:18:58 PM
I'd add one recommendation: Be clear about the limits of your knowledge in your report. It's okay to say "I looked at this part of the paper in depth and it's good, but I am not sure about this other part because it's so far outside my expertise." At least in my field (computer security/cryptography), it's pretty common to be able to review one part of a paper at great depth, but not some other parts.
Posted by: albatross at Oct 25, 2006 9:12:19 AM
jim,
I will stick to calling it scandalous.
albatross,
Excellent point.
Posted by: Barkley Rosser at Oct 25, 2006 3:08:44 PM
If the question is "How to be a Good Referee?" this begs the question:
Good for what?
If one means good for the advancement of our understanding of operational matters, then being a good referee would certainly require a prejudice against highly complex mathematical economic theories [because there is now (since 2005) clear evidence that supports Donald Gordon's hypothesis of a negative relationship between operationalism and mathematical complexity in economic theory]:
"Theory versus Application: Does Complexity Crowd Out Evidence", Southern Ec. Jrnl., 71 (3), 556-565.
But as this paper also shows, there is a significant trend among top journals (AER, EJ, JPE, and QJE) toward publishing a larger percentage of articles that are highly mathematically complex.
Posted by: jim at Oct 25, 2006 3:22:05 PM
Barkley,
I agree with you, it IS scandalous (and it is worse considering the impacts upon people up for tenure and promotion)
Posted by: jim at Oct 25, 2006 5:41:46 PM
jim,
Not sure what year that Gordon paper is (you did not provide it, but I
think it was a while ago), but the more recent trend has actually been
away from highly mathematical theory, especially axiomatic math, towards
empiricism of various sorts and other sorts of math, like computer simulation.
Think J.B. Clark awards to Matt Rabin, Steve Levitt, and Daron Acemoglu,
not to mention Nobels to Vernon Smith, Danny Kahneman, Tom Schelling, and
even Ned Phelps, although Robert Aumann still fits the very mathematical
mode.
Regarding this timing issue, it is a very serious problem, one that may
be getting a bit better since the widespread introduction of electronic
processing, but which is still bad. However, I am concerned that junior
faculty know it is there and plan for it. That is one of the reasons why
I think it is a very big mistake for junior faculty to contest a negative
decision at a journal, no matter how apparently unfair. Move on, time
is wasting.
Posted by: Barkley Rosser at Oct 25, 2006 9:43:50 PM
Barkely,
You are correct that there is a pick-up in the publication of empirical work and simulation; still there are trends in the top journals toward the usage of complex mathematical techniques. The SEJ paper I cited above provides the evidence. Additionally it shows that papers in top journals that use the more complex mathematics are: 1) less likely to themselves contain empiricism; and 2) less likely to be cited in a 5 year window following publication by any paper that presents empirical analysis.
Gordon's paper was in 1955.
Posted by: jim at Oct 25, 2006 10:02:13 PM
Barkely,
I agree that junior faculty are best served letting rejections roll off their backs unless maybe, as you mention, there is outrageous incompetence in reviews. Even then, as you say, junior faculty should probably just move on. I agree with you.
Posted by: jim at Oct 25, 2006 10:05:04 PM
Barkely,
Another thing occurs to me: Deirdre McCloskey has been trying for many years to persuade economics to give up their fascination with existence proofs in particular and what she terms the "mathematics of the math department" in general; instead, she has been trying to persuade economists to apply the "mathematics of the engineering or physics departments" (that is driven by the operational concerns for whatever the purposes are at hand). I find her argument persuasive.
You are correct that economists are using simulations with greater frequency. But the simulations that one often finds in economics are not, in McCloskey's words, "calibrated" so as to have any relvance to anything that we observe in this world. Because such simulations are unlike those typical of the engineering and physics departments (again that are driven by practical concerns regarding an issue at hand), McCloskey views them as largely inferior "works". Again, I find her argument persuasive.
Posted by: jim at Oct 26, 2006 10:13:51 AM
jim,
The article you were citing is January 2005 by Coelho and
McClure, testing the hypoethesis of Gordon from 1955. Their
tests involved ranking articles by their use of the words
"lemma" and "multiple equilibria." The first of these is
certainly linked to the sort of axiomatic mathematics that
I am speaking of, although the second is not necessarily so.
McCloskey has a lot of interesting and provocative thoughts
on all this but sometimes overdoes it, indulging in a lot of
fun rhetoric, "blackboard math" and "playing in the sandbox"
and such like at times. BTW, a lot of the simulations that
are done are calibrated. That is not such a big deal these
days, although I know some people who think calibrating is
not a good idea.
A good discussion of how changes in how math is done in
economics have reflected changes in how math itself is done
can be found in the recent excellent book by E. Roy Weintraub
from Duke University Press, _How Economics Became a Mathematical
Science_. His bottom line is that there is no going back to
the days of the non-mathematical Old Institutionalists (or
hermeneutic Austrians), but that the math is indeed a lot
less formal and axiomatic, a lot more oriented to seriously
dealing with empirical realities.
Posted by: Barkley Rosser at Oct 26, 2006 2:12:04 PM
Barkely,
I have another paper in the works with Coelho that presents empirical evidence over a longer period than in our SEJ article on the use of complex mathematics. So I'm very curious to learn what measures were used in Weintraub's book (thanks for alerting me to it).
I agree with the statement "there is no going back to
the days of the non-mathematical Old Institutionalists (or hermeneutic Austrians)". Mathematics is a valuable tool for the development of operational propositions in economics. I am decidedly not against the usage of mathematics in economics.
What Gordon did NOT say was that complexity rules out operationalism; his hypothesis is that mathematical complexity in economic theory introduces a bias against the generation of operational propositions. The evidence that Coelho and I present in the SEJ does not persuade us that there is any reason to discard Gordon's hypothesis.
Posted by: jim at Oct 26, 2006 6:17:14 PM
jim,
Ah, so you are the McClure of Coehlo and McClure. Should
discuss these matters offlist as they are far afield from
the thread.
Posted by: Barkley Rosser at Oct 30, 2006 3:09:17 AM
Barkley,
Really nice chatting with you. The library I have immediate access to here at BSU does not have the book you recommended that I read ("How Economics Became a Mathematical Science"). Nevertheless, I have sent for it via interlibrary loan though; thanks again for alerting me to it.
Posted by: jim at Oct 30, 2006 4:10:15 PM
I've found this thread extremely useful. Currently, I am working to organize a web-based peer review system (with interactive manuscript reader). Our organization was considering a number of variations on an Open Review system. What suggestions can you make to increase the fruitfulness of this endeavor?
Posted by: Paul Allen at Feb 6, 2007 4:20:48 PM
