Peer review for ACL venues
Peer review for ACL venues is made possible by the efforts of a huge number of people, almost all of them volunteers. This page breaks down the different roles within the scientific review process in ARR.
- Editors in Chief coordinate each cycle of ARR review – both the scientific considerations and the supporting infrastructure. They are available to answer questions from members of the review team.
- Senior Area Chairs check and refine the automatic assignment of papers to ACs and supervise the review effort. Unlike a conference, SACs do not make accept / reject recommendations.
- Area Chairs (a.k.a. meta-reviewers) coordinate review of submissions, including (a) checking and refining the automatic assignment of reviewers, (b) coordinating reviewer discussions, and (c) writing meta-reviews. ACs and authors are anonymous to each other. ACs know the identity of their reviewers, but not vice versa (by default).
- Reviewers read papers and write thorough reviews. A review should briefly state what the paper is about, highlight its strengths/weaknesses, and offer constructive advice to authors (regarding potential improvements) while signaling whether it merits acceptance in its current form. Reviewers and authors are anonymous to each other.
- The Ethics Chairs coordinate the process of reviewing potential ethical issues in submissions flagged by reviewers, ACs, or SACs as needing such review.
- Ethics Reviewers carefully read and review papers that have ethical issues deemed significant enough by the Ethics Chairs.
- Venue Program Committees (typically a conference or workshop team formed and headed by PC chairs) receive a pool of papers with completed reviews and meta-reviews and decide which ones to accept. They are outside the ARR process, receiving reviews from ARR.
Click on one of the roles for a detailed explanation.
Timeline
Below is an approximate timeline for the February ARR review cycle. The other cycles follow the same pattern. The submission deadline is always the 15th of the month; other dates vary due to the length of months, timing of weekends, etc. There are also some simplifications, e.g., the ethics process runs mostly in parallel to these steps, and there are additional tasks for first-time reviewers.
Month (example) | Date / Date range | What | Who |
---|---|---|---|
January | 25 | Confirm availability to review | reviewers + ACs |
February | 1 | Submission site opens | authors |
February | 15 | Submission deadline | authors |
February | 17 | Deadline to withdraw without penalty | authors |
February | 18-21 | Finalize AC assignments and check papers | SACs |
February | 22-25 | Finalize reviewer assignments and check papers | ACs |
February | 29 | Receive and check assignments | reviewers |
March | 20 | Reviews due | reviewers |
March | 18-20 | Check review quality and prompt reviewers to reconsider the paper where appropriate | ACs |
March | 26-30 | Author response | authors |
March - April | March 31 - April 2 | Reviewer discussion, updating reviews | reviewers + ACs |
April | 7 | Meta-reviews due | ACs |
April | 1-11 | Complete ethics review process | Ethics reviewers |
April | 14 | Final reviews released to authors |
Reviewers
Each submission receives at least 3 reviews. A reviewer’s job is to offer an evaluation of the submission that highlights its contributions, strengths, and weaknesses. The review should be written for two audiences: the other members of the reviewing team, including a hypothetical venue that may consider the paper for acceptance in the future; and the authors, so that they can react in a constructive manner, e.g. by revising the paper or committing it to a venue.
ARR maintains a pool of reviewers. If you have not reviewed for ARR before and would like to volunteer, contact the editors. All authors of submissions are also expected to be willing to review and those with enough papers in DBLP are added as reviewers.
Pre-cycle tasks
The first time you are a reviewer, you must complete two forms. The registration form asks for the tracks you are capable of reviewing for, the languages you have expertise for, and checks that your profile is complete. The expertise selection form asks you to import papers you have authored, which will be used to inform our matching algorithm.
Before every cycle, there is a request for availability with two questions: (1) what is your maximum load for new assignments, where 0 indicates you are not able to review new submissions, and (2) whether you are willing to review resubmissions, even if you answered ‘0’ to (1). It is helpful to agree to review any resubmissions so the authors can get the same reviewers again (assuming they did not request new reviewers).
In a cycle right before a major conference commitment date, the average reviewer load is 3 papers (the range has been 1-9). In other cycles, where there are fewer submissions, the median reviewer load is 1 paper (range 1-6). Note, these values are calculated based only on the reviewers who were assigned papers. In the quieter cycles, most reviewers are not assigned any papers.
Reviewers should also take care that their OpenReview profile and list of imported papers is up to date to facilitate COI detection and paper matching. Being up to date means having: links to DBLP and Semantic Scholar profiles, current affiliation, and current and past email addresses.
In-cycle tasks
As a reviewer, your main duties for a submission are:
- Checking the assignment when you receive it to ensure that (i) you are familiar with the topic and (ii) you can evaluate the paper without a conflict of interest or knowledge of author identities. Submit the paper checklist to confirm there are no major problems with the submission.
- Reading the paper thoroughly, critically assessing its strengths and weaknesses, and writing the review. It is important to
- Defer searching for related work until after you have read the paper in case it deanonymizes the authors
- In your comments, maintain a professional tone and strive to be constructive and specific, even if you find aspects of the paper frustrating
- Include any questions you have that can be answered in the in-cycle author response
- If you have suggestions, indicate whether they would substantially improve your view of the paper if incorporated in a resubmission
- Evaluate the paper as if it was a submission to the main ACL conference
- Resubmissions: If the paper is a resubmission and you reviewed the previous version, your review can focus on the changes. If you did not review the previous version, write your review based on the revised version alone. Once this is submitted, you will gain access to the previous reviews so you can check for additional points you had not considered.
- Checking the responsible research checklist and deciding whether ethics review would be appropriate for the submission, following these guidelines
- Being responsive to email inquiries; keeping the AC informed if there is a delay in submitting the review
- Reading the author response (if any) and engaging in discussion (new in October 2023).
- Reading the other reviews, participating in reviewer discussions, and revising your review if you have anything to add
For a step-by-step guide to reviewing, including advice on what aspects of the paper to think about and what criticisms are considered fair, see the reviewer tutorial.
In some cases, an assigned reviewer becomes unavailable and an emergency review is needed. If you are invited to provide an emergency review, consider whether you are prepared to do a good job in a short timeframe; if not, it is better to decline.
Reviewers can communicate with ACs and senior members of the review team by posting messages (or “comments”) in the OpenReview interface. For issues with OpenReview itself, contact tech@aclrollingreview.org.
Quick links:
- Reviewers’ Guide to OpenReview
- Review Form
- Reviewer Tutorial
- How to Set Your Max Load / Availability for a Cycle
Reviewer FAQ
Q: How can I communicate with the area chair in charge of one of my papers?
A: You can contact them through OpenReview. See step-by-step instructions here. You should see a button labeled “Official Comment” at the top of the paper forum. Click on it to open a message form. For messaging exclusively the area chairs, please select only “area chairs” in the “Readers” field of the form.
Q: Can I designate a secondary or substitute reviewer for ARR?
A: An assigned reviewer may wish to recruit another individual—often a mentee—to help with the review. This is called a secondary reviewer. You can involve a secondary reviewer by coordinating the review offline and submitting it in OpenReview as your own. In this case, we encourage you to leave a confidential note to the area chair indicating who the secondary reviewer was (see the question above for how to do this). However, they will not be acknowledged in OpenReview as a reviewer.
Alternatively, if you wish to hand over full reviewing duties to a colleague or mentee, you can ask your area chair to reassign the paper to a substitute reviewer in your stead. They will be acknowledged in OpenReview as a reviewer. You will no longer be listed as a reviewer of the paper and will be unable to access the paper and its reviews in the system.
Q: I think I should have heard from ARR, but I haven’t seen any emails. What’s going on?
A: It’s possible that the emails have been marked as spam – check your spam folder! ARR does not use email tracking: we know if email bounces, but we don’t know if you have seen a message.
Q: What should I do if I think the paper is in violation of the ARR guidelines?
A: Please inform the area chair in charge of your paper.
Q: What is the paper checklist?
A: This is a form for each paper that helps identify issues early in the review process. Your response is not visible to the authors or other reviewers. It is visible to your AC, SAC, and the EICs.
- Is the content of this paper appropriate to a *ACL venue?
- Is the paper in English?
- Does this paper use the ARR template?
- Does this paper adhere to length constraints for the paper type (long: 8, short: 4, in both cases exclusive of references and appendices)?
- Is the submission (including paper and any supplemental materials) anonymized?
- Is this paper already published elsewhere?
Q: What is the review report license agreement about?
A: At ARR you can donate your peer review reports to an open public dataset of peer reviews and paper drafts (see our blogpost on the data collection for details). You can contribute your data to the corpus via signing the review report license transfer agreement, which you can find in your task log on OpenReview. You have to make this decision for each cycle and you donate all review reports of a cycle in bulk.
Q: How are paper assignments determined? In particular, how are areas used?
A: We generate an assignment automatically based on areas, prior work, and languages involved. We also try to ensure at least one reviewer is a non-student, and that no two reviewers are from the same institution. This assignment is checked by the AC, who changes reviewers as needed. Regarding areas, reviewers choose multiple areas that they have expertise in while authors choose a single area at submission time. We strongly encourage the matching algorithm to assign reviewers that match the area.
Q: I no longer wish to be a reviewer for ARR. How can I withdraw myself from the pool?
A: Please contact the EICs to have yourself removed.
Area Chairs
Every submission is assigned to an area chair (AC), whose job is to oversee a team of reviewers evaluating the paper, and to write a meta-review summarizing the conclusions. You should treat the papers as submissions to ACL when evaluating them.
All AC tasks occur through the AC console for the cycle. Access it by going to the OpenReview page for the cycle and opening the “Area Chair Console” link.
Pre-cycle tasks
The first time you are an AC, you must complete a form indicating which areas you have the expertise to be an AC for and confirming that your OpenReview profile is complete.
Before every cycle, there is a request for availability with two questions: (1) what is your maximum load for new assignments, where 0 indicates you are not able to be an AC for any new submissions, and (2) whether you are willing to be an AC for resubmissions, even if you answered ‘0’ to (1). It is helpful to agree to return as an AC for any resubmissions so the authors have continuity (assuming they did not request a change of AC).
In a cycle right before a major conference commitment date, the average AC load is 7 papers (the range has been 1-12). In other cycles, where there are fewer submissions, the average AC load is 2 papers (range 1-5). Note, these values are calculated based only on the ACs who were assigned papers. In the quieter cycles, most ACs are not assigned any papers.
ACs should also take care that their OpenReview profile and list of imported papers is up to date to facilitate COI detection and paper matching. Being up to date means having: links to DBLP and Semantic Scholar profiles, current affiliation, and current and past email addresses.
In-cycle tasks
Area Chairs have duties at every stage of the two-month review cycle.
- Checking for violations of submission requirements.
- Checking and adjusting the reviewer assignments that were generated by the matching algorithm (3 reviewers are automatically selected but may not be a great fit).
- Nudging reviewers who write low quality reviews to reconsider the paper and update their review.
- Following up on late reviews and assigning emergency reviewers if necessary.
- Encouraging discussion between reviewers, both before and after the author respone.
- Writing a meta-review with clear recommendations for authors (regarding any important revisions) and for conference program committees (regarding whether it could be accepted in its current form).
The Area Chair tutorial has advice for interacting productively with reviewers and writing the meta-review.
ACs can communicate with authors, reviewers, and senior members of the review team by posting messages (or “comments”) in the OpenReview interface. In particular, the SAC for the paper can help you figure out how to deal with concerns about the content of a review or how to handle missing reviews. The process for communication is the same as the process for reviewers to make comments (see step-by-step instructions here). Slides 20-22 of the Area Chairs’ Guide to OpenReview also show other ways to contact reviewers, including how to get their email address if they are unresponsive on OpenReview. Note that reviewers of a paper are anonymous to each other, identified by pseudonyms in the discussion period, and do not see the identity of their AC in OpenReview. For issues with OpenReview itself, contact tech@aclrollingreview.org.
Quick Links:
- Area Chairs’ Guide to OpenReview
- Review Form and Meta-review Form
- Area Chair Tutorial
- How to Set Your Max Load / Availability for a Cycle
Senior Area Chairs
ARR submissions include a metadata field for topic area (see the Call for Papers) to help direct it to appropriate area chairs and reviewers. Each area is supervised by Senior Area Chairs (SACs), who are appointed by the Editors-in-Chief. The main duties of SACs are:
- Checking the assignments of papers to ACs in the area
- Monitoring and following up with ACs to ensure review steps are on track; assigning emergency ACs if necessary
- Responding to AC questions about the process (in OpenReview or by email)
- Responding to confidential author questions raised in OpenReview
Unlike the traditional conference model, SACs do not make acceptance/rejection recommendations. Those are made by venues in the commitment phase.
ACs, reviewers, and authors are not told who their SAC is. SACs are able to see the identities of ACs and reviewers, but not authors.
SACs can communicate with authors, reviewers, and area chairs in two ways:
- Via email. Using the SAC Console, select papers and press the ‘Message Reviewers’ button, then select a group to contact, write a message and send it. Note that this approach maintains anonymity. If ACs or reviewers are unresponsive, their email address is visible in the SAC console. Note that reviewers of a paper are anonymous to each other, identified by pseudonyms in the discussion period, and do not see the identity of their AC in OpenReview.
- Via messages (or “comments”) in the OpenReview interface for each paper. The process for communication is the same as the process for reviewers to make comments (see step-by-step instructions here).
SAC FAQ
Q: How can I modify reviewer assignments?
A: You can access and modify these as follows:
- Go to Senior Area Chair console
- Click Paper Status tab
- In the table that shows up, next to each paper there should be a “Modify Reviewer Assignments” link. Click on that.
- It shows the Area Chair assignments (not reviewer). So next you need to look for the paper you care about and click on “Assignments (x) »”.
- Edit the options that appear.
SACs should direct questions about the process to the Editors-in-Chief.
Ethics Chairs/Reviewers
During ARR review, some submissions are flagged for ethics review. These submissions are considered by the ethics chairs and if necessary, sent to ethics reviewers, who are supervised by the ethics chairs. Ethics reviewers read the paper and offer supplementary feedback targeted at potential ethical issues. See the ethics guidelines for specific criteria and examples.
Ethics reviewing happens in parallel to regular reviewing, and the ethics reviews are not shown to regular reviewers. After flagging a paper, reviewers and ACs should complete their work without further consideration for the ethical issues raised.
Papers can be flagged for ethics review at any time in the review process. Most are flagged early, through the checklists ACs and reviewers complete. Some are only flagged during the main review process. In either case, the process is the same. The guidelines for flagging papers for ethics review are here.
Venue Program Committees
Any conference or workshop that would like to consider ARR-reviewed submissions (either exclusively or in hybrid fashion, alongside direct submissions) is welcome to subscribe to ARR. The venue sets a commitment deadline, which will be advertised on the ARR website. Authors of submissions with complete ARR reviews/meta-reviews may send them for the venue’s consideration: this is called commitment. Venues select their own program committees, enact their own decision-making procedures (maintaining confidentiality of the submissions/reviews), and communicate acceptance decisions directly to authors. See this page for details.
Note that the ARR review process is not customized to any particular venue. All ARR submissions are anonymous and in the format of a short paper (up to 4 pages) or long paper (up to 8 pages). ARR reviewers are instructed to write a review as if the paper were being considered for the ACL conference. Thus low scores may indicate the submission is not a good fit for ACL, even if it may be a good fit for other/more specialized venues.
Questions from venues about subscribing to ARR should be directed to the Editors-in-Chief.