Reviewing

Peer review for ACL venues is made possible by the efforts of a huge number of people, almost all of them volunteers. This page breaks down the different roles within the scientific review process in ARR.

Editors in Chief coordinate each cycle of ARR review – both the scientific considerations and the supporting infrastructure. They are available to answer questions from members of the review team.
Senior Action Editors (a.k.a. Senior Area Chairs) check and refine the automatic assignment of papers to AEs and supervise the review effort. Unlike a conference, SAEs do not make accept / reject recommendations.
Action Editors (a.k.a. Area Chairs, or meta-reviewers) coordinate review of submissions, including (a) checking and refining the automatic assignment of reviewers, (b) coordinating reviewer discussions, and (c) writing meta-reviews. AEs and authors are anonymous to each other. AEs know the identity of their reviewers, but not vice versa (by default).
Reviewers read papers and write thorough reviews. A review should briefly state what the paper is about, highlight its strengths/weaknesses, and offer constructive advice to authors (regarding potential improvements) while signaling whether it merits acceptance in its current form. Reviewers and authors are anonymous to each other.
The Ethics Chairs coordinate the process of reviewing potential ethical issues in submissions flagged by reviewers, AEs, or SAEs as needing such review.
Ethics Reviewers carefully read and review papers that have ethical issues deemed significant enough by the Ethics Chairs.
Venue Program Committees (typically a conference or workshop team formed and headed by PC chairs) receive a pool of papers with completed reviews and meta-reviews and decide which ones to accept. They are outside the ARR process, receiving reviews from ARR.

Click on one of the roles for a detailed explanation.

Timeline

Below is an approximate timeline for the February ARR review cycle. The other cycles follow the same pattern. The submission deadline is always the 15th of the month; other dates vary due to the length of months, timing of weekends, etc. There are also some simplifications, e.g., the ethics process runs mostly in parallel to these steps, and there are additional tasks for first-time reviewers.

Month (example)	Date / Date range	What	Who
January	25	Confirm availability to review	reviewers + AEs
February	1	Submission site opens	authors
February	15	Submission deadline	authors
February	17	Deadline to withdraw without penalty	authors
February	18-21	Finalize AE assignments and check papers	SAEs
February	22-25	Finalize reviewer assignments and check papers	AEs
February	29	Receive and check assignments	reviewers
March	20	Reviews due	reviewers
March	18-20	Check review quality and prompt reviewers to reconsider the paper where appropriate	AEs
March	26-30	Author response	authors
March - April	March 31 - April 2	Reviewer discussion, updating reviews	reviewers + AEs
April	7	Meta-reviews due	AEs
April	1-11	Complete ethics review process	Ethics reviewers
April	14	Final reviews released to authors

Reviewers

Each submission receives at least 3 reviews. A reviewer’s job is to offer an evaluation of the submission that highlights its contributions, strengths, and weaknesses. The review should be written for two audiences: the other members of the reviewing team, including a hypothetical venue that may consider the paper for acceptance in the future; and the authors, so that they can react in a constructive manner, e.g. by revising the paper or committing it to a venue.

ARR maintains a pool of reviewers. If you have not reviewed for ARR before and would like to volunteer, contact the editors. All authors of submissions are also expected to be willing to review and those with enough papers in DBLP are added as reviewers.

Pre-cycle tasks

The first time you are a reviewer, you must complete two forms. The registration form asks for the tracks you are capable of reviewing for, the languages you have expertise for, and checks that your profile is complete. The expertise selection form asks you to import papers you have authored, which will be used to inform our matching algorithm.

Before every cycle, there is a request for availability with two questions: (1) what is your maximum load for new assignments, where 0 indicates you are not able to review new submissions, and (2) whether you are willing to review resubmissions, even if you answered ‘0’ to (1). It is helpful to agree to review any resubmissions so the authors can get the same reviewers again (assuming they did not request new reviewers).

In a cycle right before a major conference commitment date, the average reviewer load is 3 papers (the range has been 1-9). In other cycles, where there are fewer submissions, the median reviewer load is 1 paper (range 1-6). Note, these values are calculated based only on the reviewers who were assigned papers. In the quieter cycles, most reviewers are not assigned any papers.

Reviewers should also take care that their OpenReview profile and list of imported papers is up to date to facilitate COI detection and paper matching. Being up to date means having: links to DBLP and Semantic Scholar profiles, current affiliation, and current and past email addresses.

In-cycle tasks

As a reviewer, your main duties for a submission are:

Checking the assignment when you receive it to ensure that (i) you are familiar with the topic and (ii) you can evaluate the paper without a conflict of interest or knowledge of author identities. Submit the paper checklist to confirm there are no major problems with the submission.
Reading the paper thoroughly, critically assessing its strengths and weaknesses, and writing the review. It is important to
- Defer searching for related work until after you have read the paper in case it deanonymizes the authors
- In your comments, maintain a professional tone and strive to be constructive and specific, even if you find aspects of the paper frustrating
- Include any questions you have that can be answered in the in-cycle author response
- If you have suggestions, indicate whether they would substantially improve your view of the paper if incorporated in a resubmission
- Evaluate the paper as if it was a submission to the main ACL conference
- Resubmissions: If the paper is a resubmission and you reviewed the previous version, your review can focus on the changes. If you did not review the previous version, write your review based on the revised version alone. Once this is submitted, you will gain access to the previous reviews so you can check for additional points you had not considered.
Checking the responsible research checklist and deciding whether ethics review would be appropriate for the submission, following these guidelines
Being responsive to email inquiries; keeping the AE informed if there is a delay in submitting the review
Reading the author response (if any) and engaging in discussion (new in October 2023).
Reading the other reviews, participating in reviewer discussions, and revising your review if you have anything to add

For a step-by-step guide to reviewing, including advice on what aspects of the paper to think about and what criticisms are considered fair, see the reviewer tutorial.

In some cases, an assigned reviewer becomes unavailable and an emergency review is needed. If you are invited to provide an emergency review, consider whether you are prepared to do a good job in a short timeframe; if not, it is better to decline.

Reviewers can communicate with AEs and senior members of the review team by posting messages (or “comments”) in the OpenReview interface. For issues with OpenReview itself, contact tech@aclrollingreview.org.

Quick links:

Reviewer FAQ

Q: How can I communicate with the action editor in charge of one of my papers?
A: You can contact them through OpenReview. See step-by-step instructions here. You should see a button labeled “Official Comment” at the top of the paper forum. Click on it to open a message form. For messaging exclusively the action editors, please select only “area chairs” in the “Readers” field of the form.

Q: Can I designate a secondary or substitute reviewer for ARR?
A: An assigned reviewer may wish to recruit another individual—often a mentee—to help with the review. This is called a secondary reviewer. You can involve a secondary reviewer by coordinating the review offline and submitting it in OpenReview as your own. In this case, we encourage you to leave a confidential note to the action editor indicating who the secondary reviewer was (see the question above for how to do this). However, they will not be acknowledged in OpenReview as a reviewer. Alternatively, if you wish to hand over full reviewing duties to a colleague or mentee, you can ask your action editor to reassign the paper to a substitute reviewer in your stead. They will be acknowledged in OpenReview as a reviewer. You will no longer be listed as a reviewer of the paper and will be unable to access the paper and its reviews in the system.

Q: I think I should have heard from ARR, but I haven’t seen any emails. What’s going on?
A: It’s possible that the emails have been marked as spam – check your spam folder! ARR does not use email tracking: we know if email bounces, but we don’t know if you have seen a message.

Q: What should I do if I think the paper is in violation of the ARR guidelines?
A: Please inform the action editor in charge of your paper.

Q: What is the paper checklist?
A: This is a form for each paper that helps identify issues early in the review process. Your response is not visible to the authors or other reviewers. It is visible to your AE, SAE, and the EICs.

Is the content of this paper appropriate to a *ACL venue?
Is the paper in English?
Does this paper use the ARR template?
Does this paper adhere to length constraints for the paper type (long: 8, short: 4, in both cases exclusive of references and appendices)?
Is the submission (including paper and any supplemental materials) anonymized?
Is this paper already published elsewhere?

Q: What is the review report license agreement about?
A: At ARR you can donate your peer review reports to an open public dataset of peer reviews and paper drafts (see our blogpost on the data collection for details). You can contribute your data to the corpus via signing the review report license transfer agreement, which you can find in your task log on OpenReview. You have to make this decision for each cycle and you donate all review reports of a cycle in bulk.

Q: How are paper assignments determined? In particular, how are areas used?
A: We generate an assignment automatically based on areas, prior work, and languages involved. We also try to ensure at least one reviewer is a non-student, and that no two reviewers are from the same institution. This assignment is checked by the AE, who changes reviewers as needed. Regarding areas, reviewers choose multiple areas that they have expertise in while authors choose a single area at submission time. We strongly encourage the matching algorithm to assign reviewers that match the area.

Q: I no longer wish to be a reviewer for ARR. How can I withdraw myself from the pool?
A: Please contact the EICs to have yourself removed.

Action Editors

Every submission is assigned to an action editor (AE), whose job is to oversee a team of reviewers evaluating the paper, and to write a meta-review summarizing the conclusions. You should treat the papers as submissions to ACL when evaluating them.

All AE tasks occur through the AE console for the cycle. Access it by going to the OpenReview page for the cycle and opening the “Area Chair Console” link.

Pre-cycle tasks

The first time you are an AE, you must complete a form indicating which areas you have the expertise to be an AE for and confirming that your OpenReview profile is complete.

Before every cycle, there is a request for availability with two questions: (1) what is your maximum load for new assignments, where 0 indicates you are not able to be an AE for any new submissions, and (2) whether you are willing to be an AE for resubmissions, even if you answered ‘0’ to (1). It is helpful to agree to return as an AE for any resubmissions so the authors have continuity (assuming they did not request a change of AE).

In a cycle right before a major conference commitment date, the average AE load is 7 papers (the range has been 1-12). In other cycles, where there are fewer submissions, the average AE load is 2 papers (range 1-5). Note, these values are calculated based only on the AEs who were assigned papers. In the quieter cycles, most AEs are not assigned any papers.

AEs should also take care that their OpenReview profile and list of imported papers is up to date to facilitate COI detection and paper matching. Being up to date means having: links to DBLP and Semantic Scholar profiles, current affiliation, and current and past email addresses.

In-cycle tasks

Action editors have duties at every stage of the two-month review cycle.

Checking for violations of submission requirements.
Checking and adjusting the reviewer assignments that were generated by the matching algorithm (3 reviewers are automatically selected but may not be a great fit).
Nudging reviewers who write low quality reviews to reconsider the paper and update their review.
Following up on late reviews and assigning emergency reviewers if necessary.
Encouraging discussion between reviewers, both before and after the author respone.
Writing a meta-review with clear recommendations for authors (regarding any important revisions) and for conference program committees (regarding whether it could be accepted in its current form).

The Action Editor tutorial has advice for interacting productively with reviewers and writing the meta-review.

AEs can communicate with authors, reviewers, and senior members of the review team by posting messages (or “comments”) in the OpenReview interface. In particular, the SAE for the paper can help you figure out how to deal with concerns about the content of a review or how to handle missing reviews. The process for communication is the same as the process for reviewers to make comments (see step-by-step instructions here). Slides 20-22 of the Action Editors’ Guide to OpenReview also show other ways to contact reviewers, including how to get their email address if they are unresponsive on OpenReview. Note that reviewers of a paper are anonymous to each other, identified by pseudonyms in the discussion period, and do not see the identity of their AE in OpenReview. For issues with OpenReview itself, contact tech@aclrollingreview.org.

Quick Links:

Senior Action Editors

ARR submissions include a metadata field for topic area (see the Call for Papers) to help direct it to appropriate action editors and reviewers. Each area is supervised by Senior Action Editors (SAEs), who are appointed by the Editors-in-Chief. The main duties of SAEs are:

Checking the assignments of papers to AEs in the area
Monitoring and following up with AEs to ensure review steps are on track; assigning emergency AEs if necessary
Responding to AE questions about the process (in OpenReview or by email)
Responding to confidential author questions raised in OpenReview

Unlike the traditional conference model, SAEs do not make acceptance/rejection recommendations. Those are made by venues in the commitment phase.

AEs, reviewers, and authors are not told who their SAE is. SAEs are able to see the identities of AEs and reviewers, but not authors.

SAEs can communicate with authors, reviewers, and action editors in two ways:

Via email. Using the SAE Console, select papers and press the ‘Message Reviewers’ button, then select a group to contact, write a message and send it. Note that this approach maintains anonymity. If AEs or reviewers are unresponsive, their email address is visible in the SAE console. Note that reviewers of a paper are anonymous to each other, identified by pseudonyms in the discussion period, and do not see the identity of their AE in OpenReview.
Via messages (or “comments”) in the OpenReview interface for each paper. The process for communication is the same as the process for reviewers to make comments (see step-by-step instructions here).

SAE FAQ

Q: How can I modify reviewer assignments?
A: You can access and modify these as follows:

Go to Senior Area Chair console
Click Paper Status tab
In the table that shows up, next to each paper there should be a “Modify Reviewer Assignments” link. Click on that.
It shows the Action Editor assignments (not reviewer). So next you need to look for the paper you care about and click on “Assignments (x) »”.
Edit the options that appear.

SAEs should direct questions about the process to the Editors-in-Chief.

Ethics Chairs/Reviewers

During ARR review, some submissions are flagged for ethics review. These submissions are considered by the ethics chairs and if necessary, sent to ethics reviewers, who are supervised by the ethics chairs. Ethics reviewers read the paper and offer supplementary feedback targeted at potential ethical issues. See the ethics guidelines for specific criteria and examples.

Ethics reviewing happens in parallel to regular reviewing, and the ethics reviews are not shown to regular reviewers. After flagging a paper, reviewers and AEs should complete their work without further consideration for the ethical issues raised.

Papers can be flagged for ethics review at any time in the review process. Most are flagged early, through the checklists AEs and reviewers complete. Some are only flagged during the main review process. In either case, the process is the same. The guidelines for flagging papers for ethics review are here.

Venue Program Committees

Any conference or workshop that would like to consider ARR-reviewed submissions (either exclusively or in hybrid fashion, alongside direct submissions) is welcome to subscribe to ARR. The venue sets a commitment deadline, which will be advertised on the ARR website. Authors of submissions with complete ARR reviews/meta-reviews may send them for the venue’s consideration: this is called commitment. Venues select their own program committees, enact their own decision-making procedures (maintaining confidentiality of the submissions/reviews), and communicate acceptance decisions directly to authors. See this page for details.

Note that the ARR review process is not customized to any particular venue. All ARR submissions are anonymous and in the format of a short paper (up to 4 pages) or long paper (up to 8 pages). ARR reviewers are instructed to write a review as if the paper were being considered for the ACL conference. Thus low scores may indicate the submission is not a good fit for ACL, even if it may be a good fit for other/more specialized venues.

Questions from venues about subscribing to ARR should be directed to the Editors-in-Chief.