This post contains useful guidelines (links to the originals) on writing referee reports. I review them every time before I write one. However, after reading some of the reports I received and hearing about complaints from others about their experiences, it seems that not every referee is aware of these guidelines. I list three of them below and attach a few thoughts, comments, and a rough proposal in the end.
How to write an effective referee report and improve the scientific review process, written by Johnathan B. Berk, Campbell R. Harvey, and David Hirshleifer, published in Journal of Economic Perspectives in 2017. https://www.aeaweb.org/articles?id=10.1257/jep.31.1.231
Preparing a Referee Report: Guidelines and Perspectives, written by Johnathan B. Berk, Campbell R. Harvey, and David Hirshleifer. https://www.aeaweb.org/content/file?id=222
Econometrica referee guidelines: https://www.econometricsociety.org/publications/econometrica/guidelines-referees
Comments:
Personally, I think if I didn't review these guidelines before writing a report, I would be acting irresponsibly.
There appears to be a widespread belief that "any paper can be rejected because one can always find flaws in any paper." These guidelines prevent good papers from being rejected for small flaws. So no, not all papers can be rejected. Papers do not get rejected for just having typos. Published papers have typos too.
When refereeing, we need to distinguish between "major flaws" from "minor blemishes." It would make us look silly if we recommend rejecting because of some easily fixable errors, or some unexplored alternative explanations that can be tested in one afternoon. Keep in mind this advice from Berk et al. (JEP 2017): "Do not dismiss papers that attack large issues merely because flaws can be found. The important question that you need to assess is whether the flaws actually invalidate the contribution."
...
There are too many great points in the guidelines. They simply must be read by anyone who is serious about the refereeing responsibility. I add a few of my own thoughts below:
Before attempting to reject a paper, ask ourselves "If this paper is not publishable, what fraction of recently published papers become unpublishable by this standard?" If the answer is zero, then the paper is clearly below bar. If the answer is a positive number, re-think whether we are being too harsh, too unfair, or too biased. Our standard cannot be higher than the journals' revealed standard. Yes, journals can have bad papers, but those bad papers are still considered good enough by many. We can hold too high a standard for ourselves, but, hopefully, not for others.
Avoid rejecting without understanding the paper. We are not the first reader of the paper. If we reject because we think "the paper is confusing" or we "cannot understand the paper" or we think the paper "is wrong," we risk appearing silly because the editor has decided that the paper is good enough to send to referees. I have seen a case where one referee claimed to be unable to understand the paper "despite his/her best effort" while the other referee says the paper reads well. I imagine this would not reflect well on the one who failed to understand.
Our minds like to trick us into thinking that "we have known this all along." A paper's contribution is relative to what the literature knows, not relative to what we (the know-it-all referees) know. A paper with new, important, but "obvious results" are better than you think because the "obvious results" have never been shown, which implies that they are not as obvious as you thought, or, they are obvious only to you. Furthermore, many good ideas are obvious only ex post, but hard to think of ex ante. Finally, good writers make even the deepest secrets plain to see. So do not complain about the results for just being "not surprising." Results that are new, important, and obvious should be obviously publishable.
Subjectivity is unavoidable, but we need to demonstrate fairness in our judgement to the editors and the authors.
In sum, there exist good guidelines for referees. Hence, I am hopeful that responsible referees make up the majority. However, we perhaps cannot rely entirely and only on people's good faith. As in all systems, some monitoring mechanisms need to be in place to keep referees in check. A rough idea I propose here:
Open-referee-report initiative. Authors can commit to disclose the referee reports they will receive. They can note this commitment during the paper submission and/or in the paper itself. Then the editor would/could inform the referees that the reports will be disclosed. The advantages of this ex ante commitment option include:
Referees who are short in time or have other constraints will not accept the request.
High-quality papers are more likely to select this option, which help attract serious referees.
Good reports can be seen and even rewarded.
Everyone can learn from the reports.
Referees will behave more responsibly because of the public monitoring.
If this option creates extra costs for the journal, the journal can charge a fee for it.
Most importantly, the publication quality will likely improve as a result. This quality improvement will help pay for the costs.
I have spent three hours writing this post. Hope it helps.