For Reviewers

New to reviewing, or need a refresher? Check out our reviewer tutorial! Also, see this guide for help with common OpenReview tasks associated with reviewing.

Reviewer Responsibilities

ARR reviewers give their expert assessment on the merit of the submitted papers.

Each month, around 7 days after the submission deadline, reviewers who are available that month will receive submissions to review. Reviewers should immediately check that they have no conflict of interest with their assigned submissions. Reviews should be completed by 30 days after the submission deadline (see the review form).

Each month, around 30 days after the submission deadline, the action editor will read the reviews and discuss with the reviewers if necessary. Reviewers should be responsive during this time.

So, reviewers will want to be especially available from the 15th to the 22nd of each month.

Reviewer Time Commitment

An initial term of one year, renewable. We will eventually have a centralized system that keeps track of “review balance”, how much reviewing people have done, and tries to distribute it equitably. In addition, some people prefer to review many papers at a time as opposed to a few every month. Thus, at the beginning of the monthly reviewing cycle, reviewers will have a chance to specify how many papers they would like to handle. In addition, we will add the ability of reviewers to indicate that they are on vacation or otherwise unavailable.

Reviewers may choose to mark themselves as unavailable for some period. To do so, please complete this form.

Reviewer Interest Survey

Interested in reviewing for ACL Rolling Review?

We are also looking for volunteer emergency reviewers:

FAQ

This FAQ is intended to help reviewers navigate OpenReview and the review process. Please direct inquiries to: support@aclrollingreview.org.

You may find this guide helpful with common OpenReview tasks.

Timeline

Q: What is the normal review timeline?
A: See the dates and venues page. The most important deadline is the review deadline, which is the 15th of the month following paper submission.

Q: What is the timeline for emergency reviews?
A: Sometimes it is necessary to have an emergency review because an originally-assigned reviewer was not able to perform the review. In this case, we still hope that we can maintain the original timeline (finishing reviews one month after the paper was submitted), but depending on the timing it may not be possible to finish a high-quality review by this deadline. In this case, please consult with the action editor in charge of your paper (see how to contact them below) and decide a feasible timeline, keeping in mind that the authors of the paper will be eagerly awaiting their review feedback.

Q: What if I’ll be unavailable? A: Reviewers may choose to mark themselves as unavailable for some period. To do so, please complete this form. Please do so before the 15th of the first month in which you will be unavailable.

Paper Assignment

Q: What can I do to make sure I get good paper assignments?
A: Please fill out your OpenReview profile with (1) a link to your DBLP profile and click “import papers from DBLP”, and (2) a link to your Semantic Scholar profile. We import papers from these two sources, so having both filled in will give us the fullest view of your publication record for automatic paper assignment.

Q: Does automatic review assignment check if the reviewer is an action editor?
A: We hope that action editors will also be able to serve as reviewers in some cases, as action editors are often the most experienced members of our community who can provide high-quality reviews on some topics. However, when we assign papers we will attempt to balance your load so the overall load of action-edited and reviewed papers is manageable.

Q: I got 4 papers. Should I expect this load in the future?
A: The average monthly load for reviewers should be much closer to 2, but in certain periods (close to large conferences), it’s possible that the load is higher.

Q: How can I set a maximum review load for myself? A: Use this form. Please complete it before the 15th of the month in which you first want this maximum load.

Q: Is it allowed to have a secondary reviewer for ARR?
A: This can be done in two ways:

  • You can ask someone to send the review to you and upload it as your own. This way, you can mentor a junior reviewer. However, they will not be acknowledged in OpenReview as a reviewer.
  • You can ask your action editor to assign the secondary reviewer as a reviewer (in place of you). This way, they will be acknowledged in OpenReview as a reviewer. However, you won’t be able to see their review unless they share it with you.

Communication

Q: Who contacts reviewers to kick off reviewing?
A: The editor-in-chief emails all reviewers to kick off reviewing, which will usually happen on or around the 23rd of the month.

Q: How can I communicate with the action editor in charge of one of my papers?
A: You can either email the action editor directly or contact through OpenReview.

  • To communicate through OpenReview, you should see a button labeled “Official Comment” at the top of the paper forum. Click on it to open a message form. For messaging exclusively the action editors, please select only “area chairs” in the “Readers” field of the form.
  • To communicate through email, you can find the name of the action editor on the “Paper Assignments” page in OpenReview, and click on their name. Their email will be anonymized on the OpenReview page itself (for privacy reasons), but you can often find it by clicking on the “Homepage” link or otherwise searching for their contact information by following the links in the profile.

Desk Rejects

Q: What should I do if I think the paper is in violation of the ARR guidelines?
A: Please inform the action editor in charge of your paper.

Q: What is the compliance checklist?
A:

  • Is the content of this paper appropriate to a *ACL venue?
  • Is the paper in English?
  • Does this paper use the ARR template?
  • Does this paper adhere to length constraints for the paper type (long: 8, short: 4, in both cases exclusive of references and appendices)?
  • Is the submission (including paper and any supplemental materials) anonymized?
  • Is this paper already published elsewhere?

Resubmissions

Q: If I’m reviewing a resubmission (that I reviewed before) do I have to write a very different review?
A: No! You may choose to, and perhaps should, focus on whether and how the authors responded to the previous reviews. It’s very helpful if you can indicate that you read their response and say whether you feel weaknesses/revisions were addressed or not.

Other Questions

Q: Is the list of ARR reviewers and action editors published somewhere?
A: Yes, see here.

Q: Are there multiple action editors per paper?
A: No

Q: Am I expected to accept all invitations? or should I select a subset that I am most interested in?
A: You are free to decline to review papers, but keep in mind that ARR will be the sole reviewing process for ACL/NAACL in 2022, so a) you won’t get separate review requests from conference organizers, and b) the volume should come down in the following months. We will be changing the mechanism in the following months such that reviewers will be expected to review the papers they are assigned (unless they feel very unqualified to do so).

Q: I got a review invitation that I’d like to decline, but when I click the decline button, I get the following error message on Openreview: “Wrong key, please refer back to the recruitment email.” What do I have to do to be able to accept/decline invitations? Is this a known problem?”
A: This is a known issue as the link (as opposed to the link text) uses %-encoding for umlauts (or even any non-ASCII characters). Openreview currently send emails in plain text and they are planning to send them in HTML format with all the links already translated. We hope that will solve the problem.

Q: Is there any plan of integrating the review system of ARR into Publons so that there is a method for showing reviewing contributions?
A: We haven’t considered it yet (primarily because at this moment we have several issues of higher priority to sort out), but it’s definitely something we will consider in the following months.

Q: Is ARR open to first-time reviewers? If yes, is there a training process?
A: Yes, ARR is definitely open to first-time reviewers, but we do expect that reviewers have some ACL publications. Please fill out this response form, and we’ll get in contact when the time comes!

Q: I got an invitation to review but the title of the message and the indicated deadline seem outdated. Is that correct? or did this message reach me too late?
A: It means you are invited as an emergency reviewer and the deadline does not apply to you. In general, our guidance is to take the time you need to give a high-quality review, while keeping in mind that authors are waiting for the reviews to come back. Unfortunately, OpenReview sends the same original invite and does not allow AEs to modify the message.