For Action Editors

TL;DR

An action editor (AE) is a person who oversees a team of reviewers in reviewing the paper, and in the end provides a meta-review summarizing the review process. The main duties are

checking for violations of submission requirements
selecting appropriate reviewers with the aid of heuristics (3 reviewers are algorithmically selected but may not be a great fit)
supervising reviewers
writing a meta-review with clear recommendations for authors (regarding any important revisions) and conference program committees (regarding whether it could be accepted in its current form)

AEs are grouped into topical areas. The AEs in an area are supervised by Senior Action Editors (SAEs).

Quick Links:

What is an Action Editor

An action editor (AE) is a person who oversees a team of reviewers in reviewing the paper, and in the end provides a meta-review summarizing the review process.

Each month, after the submission deadline, action editors who are available that month will be automatically assigned submissions. The action editor will first confirm they have no conflict of interest with each submission. Then, they will quickly check the submission for any major violations in formatting or other factors and desk reject the submission if it is in violation. The action editor will then be presented with a list of reviewers automatically suggested. The action editor will choose reviewers, using this automatic suggestion as a base. This will take place within 7 days of the paper submission deadline.

Each month, starting around 24 days after the submission deadline, the action editor will remind any reviewer who has not yet completed their review.

Each month, between 30 and 35 days after the submission deadline, the action editor will read the reviews, discuss with the reviewers if necessary, and:

Summarize the review results in a meta-review (see the meta-review form and review form).
Perform a compliance check, checking the paper meets formatting and other requirements.

You can see more information about the reviewing schedule on the Dates and Venues page.

AE Time Commitment

AEs serve for an initial term of one year, renewable. We will have a centralized system that keeps track of “review balance”, how much reviewing people have done, and tries to distribute it equitably. In addition, some people prefer to handle many papers at a time as opposed to a few every month.

Action editors may set a max load per month (any value, including 0). To do so, please follow these instructions.

FAQ

This FAQ is intended to help reviewers and action editors navigate OpenReview and the review process. Please direct inquiries to: support@aclrollingreview.org

You may also find this guide helpful.

Timeline

Q: What is the normal review timeline?
A: See the dates and venues page. The most important deadline is the review deadline, which is the 15th of the month following paper submission, and the meta-review deadline, which is the 20th of the month following paper submission. There is a Google calendar here that you can import into your own calendar to help you keep track.

Q: What is the timeline for emergency reviews?
A: Sometimes it is necessary to have an emergency review because an originally-assigned reviewer was not able to perform the review. In this case, we still hope that we can maintain the original timeline (finishing reviews one month after the paper was submitted), but depending on the timing it may not be possible for the reviewer to finish a high-quality review by this deadline. In this case, please consult with the reviewer and decide a feasible timeline, keeping in mind that the authors of the paper will be eagerly awaiting their review feedback.

Q: What if I’ll be unavailable or have time for only one or two papers? A: Action editors may choose to mark themselves as unavailable for any cycle. Instructions to do so can be found here. The form is released shortly before each cycle and a link is shared via email.

Communication

Q: How can I communicate with reviewers?
A: You may communicate with reviewers using the discussion forum or by sending them email directly (reviewers and AEs are not blind to each-other). To send a note via the discussion forum in OpenReview, you can use a button labeled “Official Comment” at the top of the paper forum. Click on it to open a message form. You can select to exclusively message certain sub-groups, e.g. only reviewer 2.

Q: Who contacts reviewers to kick off reviewing?
A: The editors-in-chief email all reviewers to kick off reviewing.

Q: Who sends reminders to reviewers?
A: You may remind reviewers by posting in a paper’s Discussion Forum.

Q: How do I communicate to the editors in chief?
A: Email editors@aclrollingreview.org.

Q: What do I do if I’m unavailable in a particular month?
A: Action editors may choose to mark themselves as unavailable for any cycle. Instructions to do so can be found here. The form is released shortly before each cycle and a link is shared via email.

Q: I think I should have heard from ARR, but I haven’t seen any emails. What’s going on?
A: It’s possible that the emails have been marked as spam – check your spam folder!

Reviewer Assignment

Q: How do I check and modify reviewer assignments?
A: At the beginning of the review period, from about the 17th of the month to the 22nd of the month, you will be asked to check and potentially modify the assignments of the reviewers to papers that you are in charge of.

Checking Initial Assignments (up to the 22nd): When you get an email notifying you of being in charge of a paper, please log in to check the assignments. At the top of the Area Chairs Console there is an item “Reviewer Browser: Modify Reviewer Assignments”. Click the link to access the Reviewer Browser, which shows reviewer suggestions and tracks the status of invitations. Here is how to modify assignments using the “Edge Browser”: https://openreview.net/faq#question-edge-browser-AC. It is important to select reviewers with expertise appropriate to the paper. Some suggestions are listed based on an automatic score, but this score may not always be reliable, so feel free to make modifications if they are not reliable. Click on a reviewer’s name to see their profile and publications; if they are a good match and do not have too many assignments already, you can unassign one of the currrent reviewers and add a new reviewer. Once you are happy with the assignment (even if you make no modifications), there is nothing more to do. A notification will be sent out on the 23rd of the month notifying your selected reviewers that they have been assigned papers and that they should commence reviewing.
Modifying Assignments (23rd and beyond): At some point after reviewers have been assigned you may have to add new reviewers, either because a reviewer contacts you regarding not being a good match for the paper, or because they are unresponsive. If you need to do this, we would suggest you either think of reviewers who you think are appropriate, or use the reviewer browser and sorting by “Affinity Score” to find reviewers who are appropriate. Because the newly assigned reviewer will have a somewhat compressed reviewing timeline (since they were not assigned right at the beginning of the reviewing process), it would be good to reach out to them via email and confirm their availability before assigning them. Here is how to modify assignments using the “Edge Browser”: https://openreview.net/faq#question-edge-browser-AC.

Q: How many reviewers are there per paper?
A: Four reviewers should be assigned per paper at the beginning of the review process. Each paper must get at least three reviews (four is also acceptable) before the AE writes the meta-review and the reviews are released to the authors.

Q: Does automatic review assignment check if the reviewer is an action editor?
A: We hope that action editors will also be able to serve as reviewers in some cases, as action editors are often the most experienced members of our community who can provide high-quality reviews on some topics. However, when we assign papers we will attempt to balance your load so the overall load of action-edited and reviewed papers is manageable. That being said, you should never be assigned as reviewer of a paper where you are action editor, so if this happens please contact us to notify us of the issue, then unassign yourself and assign a different reviewer.

Q: In what order are alternative reviewers for a submission displayed (eg alphabetical, by match score, random)?
A: You can choose several orders. If you choose “affinity score” they will be sorted by the matching score given by an ML model (specifically, as of September this is the SPECTRE-MFR model provided by OpenReview).

Q: Can we see the number of submissions a reviewer is already assigned?
A: Yes, look for the number next to “assignments” by their name. Please do consider this when making new assignments.

Q: Can we see author information?
A: No, we are managing a two-way anonymized process up through action editors. It is not possible for you to assign a reviewer who has a COI that is automatically identifiable.

Q: What can I do to make sure I get good papers as AE?
A: Please fill out your OpenReview profile with (1) a link to your DBLP profile and click “import papers from DBLP”, and (2) a link to your Semantic Scholar profile. We import papers from these two sources, so having both filled in will give us the fullest view of your publication record for automatic paper assignment.

Q: Does a reviewer showing up at the top of the list in green mean that they’ve accepted the assignment?
A: Reviewers are assigned, not invited. If a reviewer cannot review your paper, they will contact you via email or via the discussion forum in OpenReview.

Compliance and Desk Rejects

Q: What is the compliance checklist?
A:

Is the content of this paper appropriate to a *ACL venue?
Is the paper in English?
Does this paper use the ARR template?
Does this paper adhere to length constraints for the paper type (long: 8, short: 4, in both cases exclusive of references and appendices)?
Is the submission (including paper and any supplemental materials) anonymized?
Is this paper not already published elsewhere?

Q: One of my papers doesn’t follow the compliance checklist, what should I do?
A: The paper should be desk-rejected, please get in touch with the editors in chief.

Q: One of my assigned submissions is more than four pages but less than eight pages in length. What should I do?
A: Treat it as a long paper submission.

Q: Can the ethics statement extend into a 5th/9th page?
A: Yes.

Q: How do I actually desk reject a submission?
A: In OpenReview, you should see a button labeled “Desk Reject”.

Meta-Reviewing

Q: How do I enter a meta-review?
A: Click on the “official meta-review” button in the system.

Q: Why can I not press the “meta-review” button?
A: The meta-review button may not be clickable if not all reviews are submitted yet. In this case, please try to encourage the reviewers to submit their reviews, or recruit a new emergency reviewer.

Q: Why is my overall assessment question different from the one reviewers see?
A: Your overall assessment as an action editor is different from the overall assessments made by the reviewers. Your overall assessment should a) help authors figure out what type of revision (if any) they should aim for, and b) help conference organizers make decisions. You should be signaling, both to the authors and to any program chairs, the amount of work it would take for this paper to be publication ready. So if, for example, many more experiments need to be done, then that probably means a major revision. But if the reviewers’ questions - however many - could be answered simply by adding details to the paper, that probably means a minor revision (which the authors could do between accept and camera ready). This means that a paper may receive a 1, 2, 3 or (rarely) 4 from reviewers, but be a “major revision” (3) from you, or a 4, 3 or (rarely) 2 from reviewers, but be a “minor revision” (4) from you.

Also, of course, you may look across all the reviews and the paper and identify themes or gaps, which may influence your overall assessment. You are an action editor because you are an experienced expert; you do not have to simply average across the reviewers’ scores and summarize their comments.

Q: How do I flag a submission for ethics review?
A: Please email the editors-in-chief of ARR.

ACL Rolling Review