For Action Editors

What is an Action Editor

An action editor (AE) is a person who oversees a team of reviewers in reviewing the paper, and in the end provides a meta-review summarizing the review process.

Each month, after the submission deadline, action editors who are available that month will be automatically assigned submissions. The action editor will first confirm they have no conflict of interest with each submission. Then, they will quickly check the submission for any major violations in formatting or other factors and desk reject the submission if it is in violation. The action editor will then be presented with a list of reviewers automatically suggested. The action editor will choose reviewers, using this automatic suggestion as a base. This will take place within 7 days of the paper submission deadline.

Each month, starting around 24 days after the submission deadline, the action editor will remind any reviewer who has not yet completed their review.

Each month, between 30 and 35 days after the submission deadline, the action editor will read the reviews, discuss with the reviewers if necessary, and:

  • Summarize the review results in a meta-review (see the meta-review form and review form).
  • Perform a compliance check, checking the paper meets formatting and other requirements.

You can see more information about the reviewing schedule on the Dates and Venues page.

AE Time Commitment

AEs serve for an initial term of one year, renewable. We will have a centralized system that keeps track of “review balance”, how much reviewing people have done, and tries to distribute it equitably. In addition, some people prefer to handle many papers at a time as opposed to a few every month. Thus, at the beginning of the monthly reviewing cycle, action editors will have a chance to specify how many papers they would like to handle. In addition, we will add the ability of action editors to indicate that they are on vacation or otherwise unavailable.

Interest Survey

Interested in reviewing for ACL Rolling Review?

FAQ

This FAQ is intended to help reviewers and action editors navigate OpenReview and the review process. Please direct inquiries to: support@aclrollingreview.org

Timeline

What is the timeline?

See the dates and venues page.

Communication

How can I communicate with reviewers?

You may communicate with reviewers using the discussion forum or by sending them email directly (reviewers and AEs are not blind to each-other).

To send a note via the discussion forum in OpenReview, you can use a button labeled “Official Comment” at the top of the paper forum. Click on it to open a message form. You can select to exclusively message certain sub-groups, e.g. only reviewer 2.

Who contacts reviewers to kick off reviewing?

The editors-in-chief email all reviewers to kick off reviewing.

How do I communicate to the editors in chief?

Email editors@aclrollingreview.org.

What do I do if I’m unavailable in a particular month?

For now, editors@aclrollingreview.org.

Reviewer Assignment

How many reviewers per paper?

There are 4 reviewers assigned per paper in the initial round, and we hope to get at least three reviews per paper before the AE writes the meta-review and the reviews are released to the authors.

How do I modify reviewer assignments?

At the top of the Area Chairs Console there is an item “Reviewer Browser: Modify Reviewer Assignments”. Click the link to access the Reviewer Browser, which shows reviewer suggestions and tracks the status of invitations. See this guide from NeurIPS 2021.

It is important to select reviewers with expertise appropriate to the paper. Some suggestions are listed based on an automatic score, but this score may not always be reliable.

Click on a reviewer’s name to see their profile and publications; if they are a good match and do not have too many assignments already, click “Invite Assignment”. You will receive an email notification when the reviewer accepts or declines the invitation. Once a reviewer has accepted the invitation they will be listed under the paper in the Area Chairs Console. If someone declines, or does not respond to the invitation after a few days, invite someone else until 3 reviewers are assigned.

Does automatic review assignment check if the reviewer is an action editor?

We hope that action editors will also be able to serve as reviewers in some cases, as action editors are often the most experienced members of our community who can provide high-quality reviews on some topics. However, when we assign papers we will attempt to balance your load so the overall load of action-edited and reviewed papers is manageable. That being said, you should never be assigned as reviewer of a paper where you are action editor, so if this happens please contact us to notify us of the issue, then unassign yourself and assign a different reviewer.

Should I assign and invite someone else to review a paper where the reviewer has indicated that they are unavailable or unqualified?

Yes, and you can do this by clicking through to “Modify Reviewer Assignments” from the Area Chair Console.

Compliance and Desk Rejects

What is the compliance checklist?

  • Is the content of this paper appropriate to a *ACL venue?
  • Is the paper in English?
  • Does this paper use the ARR template?
  • Does this paper adhere to length constraints for the paper type (long: 8, short: 4, in both cases exclusive of references and appendices)?
  • Is the submission (including paper and any supplemental materials) anonymized?
  • Is this paper not already published elsewhere?

One of my papers doesn’t follow the compliance checklist, what should I do?

The paper should be desk-rejected, please use the “desk reject” button in OpenReview. If you’re not sure about the decision, please get in touch with the editors in chief.

One of my assigned submissions is more than four pages but less than eight pages in length. What should I do?

Treat it as a long paper submission.

Can the ethics statement extend into a 5th/9th page?

Yes.

How do I actually desk reject a submission?

In OpenReview, you should see a button labeled “Desk Reject”.

Reviewer Assignment

In what order are alternative reviewers for a submission displayed (eg alphabetical, by match score, random)?

You can choose several orders. If you choose “affinity score” they will be sorted by the matching score given by an ML model (specifically, as of September this is the SPECTRE-MFR model provided by OpenReview).

Can we see the number of submissions a reviewer is already assigned?

Yes, look for the number next to “assignments” by their name. Please do consider this when making new assignments.

Can we see author information?

No, we are managing a two-way anonymized process up through action editors. It is not possible for you to assign a reviewer who has a COI that is automatically identifiable.

Meta-Reviewing

How do I enter a meta-review?

Click on the “official meta-review” button in the system.

Why can I not press the “meta-review” button?

The meta-review button may not be clickable if not all reviews are submitted yet. In this case, please try to encourage the reviewers to submit their reviews, or recruit a new emergency reviewer.

How do I flag a submission for ethics review?

Please email the editors-in-chief of ARR.

Why was I assigned to be a reviewer for a paper where I’m an action editor?

This was a bug in our assignment in the September iteration. Apologies, and we’ll fix this in October. For the time being, please unassign yourself and assign someone else.

Other Questions

Is the list of ARR action editors published somewhere?

Yes, see here.

I got 4 papers. Should I expect this load in the future?

We aim for the average monthly load in any given month to be 1-2 papers, but in certain periods (close to large conferences), it’s possible that the load may be higher, and in slow periods the load may be lower.

Am I expected to accept all invitations? or should I select a subset that I am most interested in?

You are free to decline to review papers, but keep in mind that ARR will be the sole reviewing process for ACL/NAACL in 2022, so a) you won’t get separate review requests from conference organizers, and b) the volume should come down in the following months.

Why did I get an invitation to be AE of something later than the normal time?

It is possible that an EICs may have sent an emergency action editor request for a paper where the original AE was not able to handle it or we could not get in contact. For these papers the deadlines for reviews and meta-reviews can be extended flexibly as necessary (although we obviously hope the paper can be done sooner rather than later).

I got a review invitation that I’d like to decline, but when I click the decline button, I get the following error message on Openreview: “Wrong key, please refer back to the recruitment email.” What do I have to do to be able to accept/decline invitations? Is this a known problem?”

This is a known issue as the link (as opposed to the link text) uses %-encoding for umlauts (or even any non-ASCII characters). Openreview currently send emails in plain text and they are planning to send them in HTML format with all the links already translated. We hope that will solve the problem.