For Reviewers

Reviewer Responsibilities

ARR reviewers give their expert assessment on the merit of the submitted papers.

Each month, around 7 days after the submission deadline, reviewers who are available that month will receive submissions to review. Reviewers should immediately check that they have no conflict of interest with their assigned submissions. Reviews should be completed by 30 days after the submission deadline (see the review form).

Each month, around 30 days after the submission deadline, the action editor will read the reviews and discuss with the reviewers if necessary. Reviewers should be responsive during this time.

So, reviewers will want to be especially available from the 15th to the 22nd of each month.

Reviewer Time Commitment

An initial term of one year, renewable. We will eventually have a centralized system that keeps track of “review balance”, how much reviewing people have done, and tries to distribute it equitably. In addition, some people prefer to review many papers at a time as opposed to a few every month. Thus, at the beginning of the monthly reviewing cycle, reviewers will have a chance to specify how many papers they would like to handle. In addition, we will add the ability of reviewers to indicate that they are on vacation or otherwise unavailable.

Reviewers may choose to mark themselves as unavailable for some period. For now, please email the editors.

Reviewer Interest Survey

Interested in reviewing for ACL Rolling Review?

FAQ

This FAQ is intended to help reviewers navigate OpenReview and the review process. Please direct inquiries to: support@aclrollingreview.org

Timeline

What is the timeline?

See the dates and venues page.

Communication

Who contacts reviewers to kick off reviewing?

The editor-in-chief emails all reviewers to kick off reviewing.

How can I communicate with the action editor?

You can either email the action editor directly or contact through OpenReview.

To go through OpenReview, you should see a button labeled “Official Comment” at the top of the paper forum. Click on it to open a message form. For messaging exclusively the action editors, please select only “area chairs” in the “Readers” field of the form.

Desk Rejects

What should I do if I think the paper is in violation of the ARR guidelines?

Please inform the action editor in charge of your paper.

What is the compliance checklist?

  • Is the content of this paper appropriate to a *ACL venue?
  • Is the paper in English?
  • Does this paper use the ARR template?
  • Does this paper adhere to length constraints for the paper type (long: 8, short: 4, in both cases exclusive of references and appendices)?
  • Is the submission (including paper and any supplemental materials) anonymized?
  • Is this paper already published elsewhere?

Other Questions

Does automatic review assignment check if the reviewer is an action editor?

We hope that action editors will also be able to serve as reviewers in some cases, as action editors are often the most experienced members of our community who can provide high-quality reviews on some topics. However, when we assign papers we will attempt to balance your load so the overall load of action-edited and reviewed papers is manageable.

Is the list of ARR reviewers and action editors published somewhere?

Yes, see here.

I got 4 papers. Should I expect this load in the future?

The average monthly load for reviewers should be much closer to 2, but in certain periods (close to large conferences), it’s possible that the load is higher.

Am I expected to accept all invitations? or should I select a subset that I am most interested in?

You are free to decline to review papers, but keep in mind that ARR will be the sole reviewing process for ACL/NAACL in 2022, so a) you won’t get separate review requests from conference organizers, and b) the volume should come down in the following months. We will be changing the mechanism in the following months such that reviewers will be expected to review the papers they are assigned (unless they feel very unqualified to do so).

Why did I get an invitation to review something when the deadline is already past?

It is possible that an AE may send as an emergency review request for one of the prior month submissions that we’re running late with. There is no strict deadline, but the sooner you could provide the review, the better.

I got a review invitation that I’d like to decline, but when I click the decline button, I get the following error message on Openreview: “Wrong key, please refer back to the recruitment email.” What do I have to do to be able to accept/decline invitations? Is this a known problem?”

This is a known issue as the link (as opposed to the link text) uses %-encoding for umlauts (or even any non-ASCII characters). Openreview currently send emails in plain text and they are planning to send them in HTML format with all the links already translated. We hope that will solve the problem.