ARR Reviewing Process

Timeline

Below is an approximate timeline for a February ARR review cycle. The other cycles follow the same pattern. The submission deadline is always the 15th of the month; other dates vary due to the length of months, timing of weekends, etc. There are also some simplifications, e.g., the ethics process runs mostly in parallel to these steps, and there are additional tasks for first-time reviewers.

Month (example)	Date / Date range	What	Who
January	25	Confirm availability to review	reviewers + ACs
February	1	Submission site opens	authors
February	15	Submission deadline	authors
February	17	Deadline to withdraw without penalty	authors
February	18-21	Finalize AC assignments and check papers	SACs
February	22-25	Finalize reviewer assignments and check papers	ACs
February	29	Receive and check assignments	reviewers
March	20	Reviews due	reviewers
March	18-20	Check review quality and prompt reviewers to reconsider the paper where appropriate	ACs
March	26-30	Author response	authors
March - April	March 31 - April 2	Reviewer discussion, updating reviews	reviewers + ACs
April	7	Meta-reviews due	ACs
April	1-11	Complete ethics review process	Ethics reviewers
April	14	Final reviews released to authors

Peer review process participants

Peer review for ACL venues is made possible by the efforts of a huge number of people, almost all of them volunteers. This page breaks down the different roles within the scientific review process in ARR.

Editors in Chief coordinate each cycle of ARR review – both the scientific considerations and the supporting infrastructure. They are available to answer questions from members of the review team.
Senior Area Chairs check and refine the automatic assignment of papers to ACs and supervise the review effort. Unlike a conference, SACs do not make accept / reject recommendations.
Area Chairs (a.k.a. meta-reviewers) coordinate review of submissions, including (a) checking and refining the automatic assignment of reviewers, (b) coordinating reviewer discussions, and (c) writing meta-reviews. ACs and authors are anonymous to each other. ACs know the identity of their reviewers, but not vice versa (by default).
Reviewers read papers and write thorough reviews. A review should briefly state what the paper is about, highlight its strengths/weaknesses, and offer constructive advice to authors (regarding potential improvements) while signaling whether it merits acceptance in its current form. Reviewers and authors are anonymous to each other.
The Ethics Chairs coordinate the process of reviewing potential ethical issues in submissions flagged by reviewers, ACs, or SACs as needing such review.
Ethics Reviewers carefully read and review papers that have ethical issues deemed significant enough by the Ethics Chairs.
Venue Program Committees (typically a conference or workshop team formed and headed by PC chairs) receive a pool of papers with completed reviews and meta-reviews and decide which ones to accept. They are outside the ARR process, receiving reviews from ARR.

Click on one of the roles for a detailed explanation.

Reviewers

Each submission receives at least 3 reviews. The reviewer’s main tasks are (1) checking the appropriateness of the submission and its assignment to this reviewer, (2) writing a strong review, (3) updating the review as appropriate based on author-reviewer discussion.

ARR maintains a pool of reviewers. If you have not reviewed for ARR before and would like to volunteer, contact the editors. All authors of submissions are also expected to be willing to review and those with enough papers in DBLP are added as reviewers. Before every cycle, there is a request for availability with two questions: (1) what is your maximum load for new assignments, where 0 indicates you are not able to review new submissions, and (2) whether you are willing to review resubmissions, even if you answered ‘0’ to (1). It is helpful to agree to review any resubmissions so the authors can get the same reviewers again (assuming they did not request new reviewers).

Quick links:

Area Chairs

Every submission is assigned to an area chair (AC), whose job is to oversee a team of reviewers evaluating the paper, and to write a meta-review summarizing the conclusions. You should treat the papers as submissions to ACL when evaluating them.

All AC tasks occur through the AC console for the cycle. Access it by going to the OpenReview page for the cycle and opening the “Area Chair Console” link.

Pre-cycle tasks

The first time you are an AC, you must complete a form indicating which areas you have the expertise to be an AC for and confirming that your OpenReview profile is complete.

Before every cycle, there is a request for availability with two questions: (1) what is your maximum load for new assignments, where 0 indicates you are not able to be an AC for any new submissions, and (2) whether you are willing to be an AC for resubmissions, even if you answered ‘0’ to (1). It is helpful to agree to return as an AC for any resubmissions so the authors have continuity (assuming they did not request a change of AC).

In a cycle right before a major conference commitment date, the average AC load is 7 papers (the range has been 1-12). In other cycles, where there are fewer submissions, the average AC load is 2 papers (range 1-5). Note, these values are calculated based only on the ACs who were assigned papers. In the quieter cycles, most ACs are not assigned any papers.

ACs should also take care that their OpenReview profile and list of imported papers is up to date to facilitate COI detection and paper matching. Being up to date means having: links to DBLP and Semantic Scholar profiles, current affiliation, and current and past email addresses.

In-cycle tasks

Area Chairs have duties at every stage of the two-month review cycle.

Checking for violations of submission requirements.
Checking and adjusting the reviewer assignments that were generated by the matching algorithm (3 reviewers are automatically selected but may not be a great fit).
Nudging reviewers who write low quality reviews to reconsider the paper and update their review.
Following up on late reviews and assigning emergency reviewers if necessary.
Encouraging discussion between reviewers, both before and after the author respone.
Writing a meta-review with clear recommendations for authors (regarding any important revisions) and for conference program committees (regarding whether it could be accepted in its current form).

The Area Chair tutorial has advice for interacting productively with reviewers and writing the meta-review.

ACs can communicate with authors, reviewers, and senior members of the review team by posting messages (or “comments”) in the OpenReview interface. In particular, the SAC for the paper can help you figure out how to deal with concerns about the content of a review or how to handle missing reviews. The process for communication is the same as the process for reviewers to make comments (see step-by-step instructions here). Slides 20-22 of the Area Chairs’ Guide to OpenReview also show other ways to contact reviewers, including how to get their email address if they are unresponsive on OpenReview. Note that reviewers of a paper are anonymous to each other, identified by pseudonyms in the discussion period, and do not see the identity of their AC in OpenReview. For issues with OpenReview itself, contact tech@aclrollingreview.org.

Quick Links:

Senior Area Chairs

ARR submissions include a metadata field for topic area (see the Call for Papers) to help direct it to appropriate area chairs and reviewers. Each area is supervised by Senior Area Chairs (SACs), who are appointed by the Editors-in-Chief. The main duties of SACs are:

Checking the assignments of papers to ACs in the area
Monitoring and following up with ACs to ensure review steps are on track; assigning emergency ACs if necessary
Responding to AC questions about the process (in OpenReview or by email)
Responding to confidential author questions raised in OpenReview

Unlike the traditional conference model, SACs do not make acceptance/rejection recommendations. Those are made by venues in the commitment phase.

ACs, reviewers, and authors are not told who their SAC is. SACs are able to see the identities of ACs and reviewers, but not authors.

SACs can communicate with authors, reviewers, and area chairs in two ways:

Via email. Using the SAC Console, select papers and press the ‘Message Reviewers’ button, then select a group to contact, write a message and send it. Note that this approach maintains anonymity. If ACs or reviewers are unresponsive, their email address is visible in the SAC console. Note that reviewers of a paper are anonymous to each other, identified by pseudonyms in the discussion period, and do not see the identity of their AC in OpenReview.
Via messages (or “comments”) in the OpenReview interface for each paper. The process for communication is the same as the process for reviewers to make comments (see step-by-step instructions here).

SAC FAQ

Q: How can I modify reviewer assignments?
A: You can access and modify these as follows:

Go to Senior Area Chair console
Click Paper Status tab
In the table that shows up, next to each paper there should be a “Modify Reviewer Assignments” link. Click on that.
It shows the Area Chair assignments (not reviewer). So next you need to look for the paper you care about and click on “Assignments (x) »”.
Edit the options that appear.

SACs should direct questions about the process to the Editors-in-Chief.

Ethics Chairs/Reviewers

During ARR review, some submissions are flagged for ethics review. These submissions are considered by the ethics chairs and if necessary, sent to ethics reviewers, who are supervised by the ethics chairs. Ethics reviewers read the paper and offer supplementary feedback targeted at potential ethical issues. See the ethics guidelines for specific criteria and examples.

Ethics reviewing happens in parallel to regular reviewing, and the ethics reviews are not shown to regular reviewers. After flagging a paper, reviewers and ACs should complete their work without further consideration for the ethical issues raised.

Papers can be flagged for ethics review at any time in the review process. Most are flagged early, through the checklists ACs and reviewers complete. Some are only flagged during the main review process. In either case, the process is the same. The guidelines for flagging papers for ethics review are here.

Venue Program Committees

Any conference or workshop that would like to consider ARR-reviewed submissions (either exclusively or in hybrid fashion, alongside direct submissions) is welcome to subscribe to ARR. The venue sets a commitment deadline, which will be advertised on the ARR website. Authors of submissions with complete ARR reviews/meta-reviews may send them for the venue’s consideration: this is called commitment. Venues select their own program committees, enact their own decision-making procedures (maintaining confidentiality of the submissions/reviews), and communicate acceptance decisions directly to authors. See this page for details.

Note that the ARR review process is not customized to any particular venue. All ARR submissions are anonymous and in the format of a short paper (up to 4 pages) or long paper (up to 8 pages). ARR reviewers are instructed to write a review as if the paper were being considered for the ACL conference. Thus low scores may indicate the submission is not a good fit for ACL, even if it may be a good fit for other/more specialized venues.

Questions from venues about subscribing to ARR should be directed to the Editors-in-Chief.