Action Editors play a central role in making the ARR process smooth and in ensuring that the reviews are of high quality and provide constructive feedback to the authors. In other words, it is the job of the AEs to make sure that each ARR submission receives (at least) three high-quality reviews by the time the reviews are scheduled to be released to the authors. As the job of AEs is to ensure the high quality of the reviews, please also thoroughly read the guidelines for reviewers in addition to reading each paper.
The key tasks for an AE are:
- Checking for violations of submission requirements.
- Checking and adjusting the reviewer assignments that were generated by the matching algorithm (3 reviewers are automatically selected but may not be a great fit).
- Nudging reviewers who write low quality reviews to reconsider the paper and update their review.
- Following up on late reviews and assigning emergency reviewers if necessary.
- Encouraging discussion between reviewers, both before and after the author respone.
- Writing a meta-review with clear recommendations for authors (regarding any important revisions) and for conference program committees (regarding whether it could be accepted in its current form).
The reviewers may have opposing views on the merits and the shortcomings of a submission. A good AE will thoroughly read all reviews, will understand the main points of praise and criticism by each of the reviewers, and will instigate the discussion centered around the disagreeing views about the submission’s merits. Particularly if there are disagreeing opinions among the reviewers, the AE should also carefully read the paper to be able to lead a meaningful discussion.
Improving Review Quality
Crucially, AEs are encouraged to request from the reviewers to further substantiate and to improve their reviews, if judged to be of insufficient quality. Each review claim, i.e., each point of praise and/or criticism needs to be adequately substantiated. Specifically, AEs need to pay attention to:
- unsubstantiated claims and to blanket statements (e.g., “not novel enough”, “too few experiments”)
- claims of missing key papers without the relevant citations made by the reviewers
- unjustified/unacceptable reasons for negative evaluation of the submission, listed in the instructions for reviewers (e.g., “this is a resource paper”, “the results are not better than SOTA”). These are, on their own and without adequate substantiation, not acceptable reasons for negative evaluation of a submission and reviewers should be warned if they resort to those. The AEs are also encouraged to ask reviewers to change their reviews when the review language is hostile, or when the scores do not match the content of the reviews (e.g., low scores but no or little criticism in the review).
Reviewer Discussion
Typically, and especially when reviews disagree with each other, it is a good idea to start a discussion by asking reviewers to read each others’ reviews and see whether there is anything else to add. A reviewer may see a point that causes them to change their score, or one reviewer may help clarify something about the paper that another reviewer was confused about.
Discussion can occur both before and after the author response period in the cycle.
It is OK if differences between reviewers remain, as long as you understand the rationale behind the disagreement.
The discussion can also serve to get the reviewers to agree on top priorities for revision when the reviews contain many suggestions. It may help to list what you see as the top opportunities for improvement and ask reviewers whether they would raise their score for a revision that satisfies those points.
Also, keep in mind that resubmissions do not have a higher page limit for addressing reviewer feedback, so if major additions are suggested, consider asking reviewers what can be removed to make space (or whether the format should be switched between short and long).
Writing the Meta-Review
The meta-review should aggregate the final position of the reviewers after a thorough discussion (especially in the case of initially opposing views and evaluations): it should summarize the most prominent strengths and weaknesses of the submission and should explicitly judge whether the former outweigh the latter (or vice-versa).
Your overall assessment as an action editor is different from the overall assessments made by the reviewers. Your overall assessment should a) help authors figure out what type of revision (if any) they should aim for, and b) help conference organizers make decisions. You should be signaling, both to the authors and to any program chairs, the amount of work it would take for this paper to be publication ready. So if, for example, many more experiments need to be done, then that probably means a major revision. But if the reviewers’ questions - however many - could be answered simply by adding details to the paper, that probably means a minor revision (which the authors could do between accept and camera ready). This means that a paper may receive a 1, 2, 3 or (rarely) 4 from reviewers, but be a “major revision” (3) from you, or a 4, 3 or (rarely) 2 from reviewers, but be a “minor revision” (4) from you.
Also, of course, you may look across all the reviews and the paper and identify themes or gaps, which may influence your overall assessment. You are an action editor because you are an experienced expert; you do not have to simply average across the reviewers’ scores and summarize their comments.
For transparency to authors, you can be explicit in the meta-review that, for example, reviewers reached consensus on some point in the discussion.
Reviewing a Resubmission
A few additional considerations when meta-reviewing a resubmission:
- The meta-review should comment on the relationship to the previous submission (e.g., whether it satisfactorily addressed major objections/suggestions from that round, and whether it introduced new positive contributions or new elements that were subject to criticism).
- Check whether the set of reviewers has changed.
- If any of the reviewers on the previous submission are not reviewing the resubmission, take care to check whether the revision responds to their important concerns.
- A new reviewer brings a fresh set of eyes to the paper, which can be useful for getting feedback focused on the revised version. At the same time, a new reviewer may raise objections that could have been raised about the original submission, but weren’t. Unless any of these objections are truly crucial points that were missed in the previous round, try to avoid creating a moving target for authors in your assessment of whether the paper has improved or not.
- Consider stating in the metareview whether the set of reviewers is the same, or whether there are any new reviewers. This helps make the process more transparent to the authors.
Acknowledgements This tutorial was authored with input from Goran Glavaš, Nathan Schneider, Amanda Stent and the ACL 2022 program chairs.