How to Referee a (Math) Paper

Refereeing papers is one of those essential skills of a professional research mathematician that you are never taught how to do but, one day, you will find a referee request in your inbox. Suddenly, you are thrown into the world of refereeing without any proper training. In this blog post, I will give my own take on how to referee (math) papers.

You can run but you can’t hide. Eventually, an editor will find you and send you a referee request.

Now what?

If you do not have any previous experience on what to do next, this is a blog post for you. Also, read this piece by Arend Bayer on “Writing, and reading, referee reports,” and this piece on “What makes a good PRIMUS review.” Finally, this paper by Bjorn Poonen on “Practical suggestions for mathematical writing” can be a great source of feedback for authors.

Why do we referee papers?

Refereeing papers is a service that mathematicians provide to the community. Math papers can be long and complicated, and the refereeing process gives you the opportunity to have other research mathematicians read your paper carefully for correctness and for suggestions, before it is published. It is a hard job, it can take many, many hours, and it is unpaid. But we publish papers, and others referee our papers so, in turn, we return the favor by refereeing papers by others.

This blog post is not about the math publication system, which deserves an entire different entry. Here I will limit myself to the task of refereeing a paper, and we will leave the editorial commentary on journals, predatory journals, “publish or perish,” the tenure system and the need to publish, etc., for another time.

When do mathematicians start refereeing?

Before you accept a referee job, there are several important factors to consider. But before we go into these factors, let us first address the question “when should you start taking on referee jobs?” Or, more generally, “who should be a referee?”

The most important qualification in order to be a referee is that you need to be an “expert” in the topic of the paper, which in this case means that (i) you have enough background to follow and digest the arguments and techniques used in the paper under review, and (ii) you are familiar with the literature on the subject, enough to know how this result fits in the published record. If you are invited to referee, then the editor believes you are sufficiently qualified to write a review of the paper, so now it is up to you to decide if you are a good fit for the job.

In light of all this, typically, mathematicians start refereeing after (a) they graduate with a PhD, and (b) they have published at least one paper. And the first paper you are asked to referee is probably related to your thesis, or to the topics of your first papers.

Note that some grad students are asked to referee sometimes… I do not think this is, in general, a good idea or fair to the student or the author. Perhaps a better idea would be for the PhD advisor and grad student to collaborate on a review, which would provide a training opportunity in refereeing papers, but I am also not sure this is a great idea either, since someone else’s paper and career is on the line.

Should I accept the referee job?

When you receive a referee request, you will be able to see a copy of the paper so you can decide if you can accept the refereeing job at this time. Go ahead, have a look, and then consider the following factors:

• You are under no obligation whatsoever to referee papers. As I mentioned above, this is an unpaid job, so you can always politely decline an invitation to referee. However, if you are a research mathematician that publishes papers, then you should consider reviewing papers as part of your service that keeps the community going.
• Is this a journal you know about and we should be refereeing for? Please be aware that there are many publications out there that are “fake” or dishonest, so only accept referee jobs from reputable journals.
• Is there a conflict of interests that disqualifies you for this job? If you cannot be an impartial referee then you should not accept the job. Simply let the editor know, and bow out. Here is a list of common conflicts: the paper is by your advisor, one of your students, a close collaborator, a family member, a close friend or a person you have a personal conflict with, the paper’s results are very much like a paper you are writing yourself, etc. If you think there might be a conflict of interests, there is probably a conflict. You can always consult with the editor of the journal, and let them decide. Note that some fields are really small, so there are a lot of connections that may be unavoidable. In summary: if for whatever reason, you think you will not be an impartial referee, then please reject the job.
• Do you have time for this job? As I mentioned above, refereeing is a hard job and to do it well, it takes time (probably many hours). Ask the editor when the referee job is needed by, and if you cannot possibly have it ready by their deadline or soon after, then let them know you are too busy to take on this job at this time. Please keep in mind that some mathematicians’ careers, particularly graduate students and postdocs, are in the balance here, and timely refereeing can make a huge difference in their next job search.
• Are you refereeing too much? If you accept too many jobs, then it might jeopardize your time for your own research. As a general rule, I referee about twice or three times as many papers as I submit to journals. Why? Many journals require two different referees, so I figure that two people kindly took the time to referee my paper, so I need to give back to the community two refereeing jobs for every paper I publish myself. If I have already accepted a refereeing job(s), and I am too busy with it, I will simply let the editors know that I am not available at this time to write a good and timely report.
• Are you a good fit for the job? Have a look at the paper and try to get a sense of the topic, and the techniques used in the proofs. If you are unfamiliar with them, then you may not be the “expert” they are looking for, and it may take you an enormous time to familiarize yourself with the techniques and the literature on the subject, so you should be honest with the editor and simply say that this is far from your area of expertise, and you are not a good fit. Refereeing is not the time to learn a new area, when someone else’s career is on the line. Of course, the paper is most likely brand new research, so you will learn a lot reading and reviewing the paper! But it shouldn’t be too far afield.

What kind of a referee job am I being asked to do?

Typically, editors will ask for one of two types of referee jobs: a quick opinion, or a full referee report. In a quick opinion, you are only asked to evaluate if the paper is a good fit for the journal, and the results are interesting enough for the refereeing process to continue ahead. Typically, this opinion is not even shared with the authors, so it is an internal editorial process, and the editors just want a quick note back from you (one paragraph or two) about the paper with your first impressions (see below for more comments about how to evaluate the fit of a paper).

If the editor is asking you for a long-form referee report, then read on.

Should I reject a paper right away?

Assuming you have answered yes to all the questions above, then it is time to get started: accept the job, download the paper, and start lightly browsing its contents.

The first decision you need to make is if the paper should be rejected right away because, in your opinion, the paper is not a good fit for the journal. This is a hard call to make, so you can ask the editor for more information on what kind of papers they are looking to publish. Another good idea is to go through the journal’s archives, and look for other papers in the same area that they have recently published. Is the paper under review, in principle, at about the same level or above than recent papers that have appeared in the same journal? If so, then go ahead with the job. If the paper is clearly not a fit, if the result is known, if the combination of results and techniques are not strong enough for the journal, if the paper needs a huge amount of work,… then reply to the editors with a rejection. The sooner the better, and if you can, please offer a quick opinion, and suggestions of better journal fits.

Please do not (ever!) be mean when you reject a paper, or if you write a quick opinion. Harsh words are completely unnecessary. Just be professional, and imagine you are the one at the receiving end of the rejection letter. Be honest and direct, but always try to offer some constructive criticism.

If you are in the middle of refereeing and you find a big problem with a proof, then stop right away, and consider for a while if that’s a mistake that cannot be salvaged. If so, you may need to reject the paper on those grounds. Or at least ask the authors for clarification.

By the way, never contact the authors directly. All communication should go through the editor and the online editorial system. The anonymous nature of refereeing ensures that referees can be impartial and honest.

How much time should I spend refereeing this thing??

Refereeing can take many hours, and if the paper is long, it can be months of work. Make sure the editors have given you a deadline that is reasonable so that you don’t have to put everything else aside to review the paper. Let the editors know what is a manageable deadline to have a report ready.

That said, once you start refereeing, if the job is taking longer that you imagined, then there might be other factors to consider. If the ref job is taking too long because something came up, you might want to let the editors know so they can reassign the job if needed. If the ref job is taking way too long because the paper is just not well written, or the arguments are confusing, or you are spending too much time fixing small steps of their proofs… then consider rejecting the paper on those grounds.

Note that a rejection is not necessarily a death sentence for the paper. Most journals offer sending back the paper to the authors for “light revisions” or “major revisions.” If you don’t want to quite reject the paper, but you think that it needs a great amount of work before it is ready for you to review it again, you can send it back with an initial set of general comments indicating what the authors would need to do for you to consider it. For example, you can ask the authors to restructure the paper, to add more detail in proofs, to add more results in a certain direction that seems to be conspicuously missing from the paper, etc.

What am I actually looking for while refereeing?

You are now in the thick of it, reading the paper, it looks like a good fit, and the paper seems worth looking at in detail. What now? What is an editor and an author actually looking for?

The amount of detail and time you put into a report is a personal choice. The bare minimum amount of work a referee needs to do is to check that all the arguments are mathematically correct. In other words, make sure the proofs are correct, and the theorems are stated correctly. However, most of us go an extra mile, and give feedback to improve the paper in several ways.

• Should I worry about grammar and sentence structure? This is optional, because it can be a very time consuming job to go into this level of detail. I do care about this, and I can’t let it go, so I will go into all sorts of grammar comments, but that’s just me. The key is that I wan the paper to be readable, and easily understandable by others, so if bad sentence structure is getting in the way of the math, then I will definitely comment on it and suggest alternative sentences that would be male a clearer, easier to digest argument.
• Should I check every piece of math line by line? This is tricky. You need to check that the arguments are mathematically sound, so you need to go into enough detail to ascertain as much. If you are not checking certain arguments in the paper (e.g., because they are standard, or not main point of the paper) then let the editor know, or simply write it in the referee report.
• Should I provide suggestions? Yes!! Absolutely. The reason you are doing this job is because you are an expert in the field. You are the target audience! So any suggestions you may have, are very much welcome, and that’s the kind of referee report that enriches the refereeing experience and improves papers. You can offer references, alternative proofs, short cuts, examples, or any other kind of suggestion that you think would improve the quality of the paper (particularly if it improves its readability). However, you cannot expect that the authors will overhaul the paper with your suggestions… after all, it is their paper and you are not a coauthor.
• Should I be tough? No. Do not, in any way, write comments that can be construed as offensive. You should be an impartial, professional, honest and direct referee. So if things are missing, or if there are glaring mistakes, simply point them out in a plain way, and let them deal with the mistakes. If your comment is going to read like “the authors should know that…” then remove that comment and think of a way to point the problem out in a neutral way.
• I am in a pissy mood. Should I referee at this time? No. It will not go well. You will be annoyed by every single little thing, and you might end up rejecting the paper for some minor thing. Step away, relax, watch a movie, go for a walk, sleep on it, and when you are back in a constructive mood, go back to the paper and keep going.
• Should I be really nice? You do not have to go out of your way to be complimentary to the authors, but (negative) referee reports can be hard pills to swallow, particularly for early stage mathematicians. So I try to sound encouraging about the good parts, and offer constructive criticism and ideas whenever possible. The key is to strike a balance so that your report is useful.
• I found a mistake. Should I reject the paper? Not yet. How big of a mistake is it? Is it a simple error that can be fixed? Offer a solution (though you are not obligated to do so). Is it a complicated issue that you cannot fix yourself in a reasonable amount of time? Write it down in the report, and let them deal with it (this may be a minor or major revision, depending on the size of the gap in the proof). Is it a catastrophic error? Then, yes, contact the editor, let them know there is a serious issue with the paper, and reject it.
• Should I evaluate the overall quality of the paper? Yes. This is a very hard thing to do, but yes, absolutely, the editor will want to know your overall impression after you have looked at the entire paper. First, I gain a first impression of the paper, enough to decide whether the paper is a good fit for the paper and I am going ahead with the process. And then I wait until I have read the paper in detail to decide on an overall impression of the paper.
• How do I actually referee? That’s your personal choice, but I print a hardcopy of the paper, and write all my comments on the paper itself and margins, so that when I am ready to write, I go comment by comment and expand on it in the report.

The referee report

It is time to write all your comments and feedback on the actual report. Consider adding the following components to your reports:

• NOT anywhere in the report: your name, affiliation, email address. Make sure the report is anonymous and that you are not writing things in a way that will easily identify yourself.
• Title and authors of the paper under review.
• Journal where the paper is submitted (this is mostly for my records, because sometimes you get to referee the same paper twice for different journals!).
• Overview: a summary of the results of the paper, so the editor and authors know that you have actually read the paper. It is also a place to state the main results in your opinion, which may differ from the results that the authors think are the main results! This section is a neutral zone, however, so you are just stating results without colorful commentary.
• Recommendation: a narrative of the strengths and weaknesses of the paper, in your expert opinion, which concludes with a recommendation for the editors: reject, accept, minor revision, major revision, etc. You can include here big items that the authors need to address before the paper is accepted, and general comments about the paper.
• Detailed comments: this is an itemized list of comments. Please include pages and lines and theorem numbers that you are referring to, so that the authors know what you are exactly talking about.
• Conclusion: any other general comments that may improve the paper, or thoughts about the paper itself.

Once you are done, send the anonymous referee report to the editors, in their preferred contact method, probably through their online editorial system.

What happens then?

After the report is sent back to the editors, the editorial team may be waiting for other referees to also send their reports. Once they have all their reports, they will make a decision. If they ask the authors for a revision, they might ask you to look at the paper one more time. I usually accept because it is efficient, since I am already familiar with the paper, but again, it is your call if you are available or busy at this time.

Finally, thanks for taking time to do a great job refereeing papers! Authors definitely appreciate the hard work of a referee.