Our options

We had to make a quick decision to keep the cycle on track once we had all necessary information about reviewer registration. We discussed several options, including: (1) requiring all qualified authors to take the same assignment burden; (2) lowering the qualification threshold for reviewers; and (3) requiring authors with multiple submissions to take on higher service burdens. In the end, we adopted the second and third options. We are fully cognizant of the additional effort this entails, especially as this extra burden was unplanned. We are also aware that this may negatively affect the quality of the review process. We nevertheless hope that the community supports the cycle with high-quality reviews, given the situation. We genuinely appreciate the effort you put into ensuring scientific quality to the highest extent possible.

Deficiencies in our communication

While we were deciding how to handle the substantial increase in submissions, there were lapses in communication. For this, we apologize. In particular:

• We should have alerted the community much sooner that the reviewing assignments would need to increase in response to the increase in submissions. The exact availability of reviewers can only be calculated when we have identified the qualifications of the authors, which takes several days after submission. Nevertheless, we should have given everyone more preparatory time, and will do so when this situation arises in future cycles.
• Many of you are upset about receiving a high reviewing load, especially when you had specified a low reviewing load in the maximum availability form. We did not make it sufficiently clear that the Reviewer Unavailability and Maximum Load Request form applies to voluntary reviewing service and not to the mandatory reciprocal service required of authors. We did not realize this confusion existed due to the wording of the forms. We are taking steps to repair this problem for future cycles.
• Some authors reached out to us in confusion or with understandable stress at the increased load. We did not always respond to these authors in a satisfactory manner. The editors-in-chief, program chairs, and technical support collectively handle an enormous number of inquiries on a daily basis, which leads to impersonal and sometimes abrupt-seeming responses in the interest of dealing with issues in a timely way. We are reviewing our standards for responding to inquiries about mandatory service requirements for authors.
• We are running behind schedule on responding to your emails asking for support. Be assured that we are working as fast as we can to handle the backlog. Many of your emails require additional discussion behind the scenes prior to response.

Requests to reduce reviewing assignments

With regret, we will not be able to reduce the reciprocal reviewing assignment load in this cycle. If we reduce the reciprocal load for any individual author, we will be unable to find volunteers to take over their efforts. However, for authors who wish to withdraw their submissions in this cycle to avoid the high reciprocal reviewing load, we will waive the withdrawal penalty in this cycle so these papers will be allowed to be re-submitted in the next cycle.

Looking towards the future

Based on our experience as editors-in-chief and program chairs, the reality is that there appears to be too many good-faith, properly formatted research papers being submitted for the system to handle in certain cycles. It is a “luxury problem”: the community is simply too productive by present research standards. The ACL executive and peer review committees are currently reviewing options for addressing the scaling problem with reviewing capacity, including options for limiting submissions for the first time in ACL’s history.

Here are two things you can do as an author to help the ARR process in the future:

• If you meet the qualifications, please volunteer for service roles, particularly reviewer and area chair, even for cycles where you are not an author.
• Submit your papers in lower-demand cycles. ARR-reviewed papers can be committed to any conference that subscribes to ARR, even if it is not reviewed in a cycle that coincides with the conference.

The submission crisis affects not only our community, but also adjacent communities in machine learning and other currently-popular technical and scientific fields. Solutions will require input and discussions from ARR, ACL, and the academic world as a whole, especially to address the underlying causes that brought about this situation.

What is REVAS?

REVAS is a system developed at TU Darmstadt and MBZUAI that — with support from the ACL exec — is intended to be continuously maintained and developed for the next two years to help improve the peer review quality at *CL venues. It looks like this:

A key principle distinguishing REVAS from other review assistants is that it adheres to the publication ethics guidelines: it does not suggest content for the review. It only provides feedback on whether the review satisfies some of the criteria that are in general desirable for a good review, such as specificity. From the authors’ perspective, the reviewer adherence to such general criteria is both desirable and expected.

As with any automated tool, REVAS is intended only as a source of suggestions, not a replacement for human judgement: i.e., a review not flagged by REVAS is not necessarily amazing, and a review flagged by REVAS is not necessarily truly at fault. ARR is experimenting with REVAS to establish whether such a tool can help to raise the overall review quality level and improve the community awareness of the relevant peer review norms and guidelines.

The REVAS pilot is completely independent of (and complementary to) the EMNLP 2026 initiative on providing authors with AI-generated feedback. The REVAS experiment is an ongoing initiative of ARR, rather than any specific *CL conference.

What review quality criteria does REVAS support?

In March 2026, the following criteria were supported:

• Actionability
• Grounding & Specificity
• Verifiability
• Helpfulness

See more details on how these criteria were operationalized in this EMNLP-25 paper.

For May 2026, the tool was extended to more directly support the ARR reviewer guidelines, in particular 10 of the 17 currently listed reviewer heuristics are now checked by REVAS:

• H1. The results are not surprising
• H3. The results are not novel
• H5. The results do not surpass the latest SOTA
• H7. This method is too simple
• H9. The topic is too niche
• H11. The paper has language errors
• H13. The authors could also do [extra experiment X]
• H14. The authors should compare to a ‘closed’ model X
• H15. The authors should have done [X] instead
• H16. Limitations != weaknesses

Preliminary evaluation of the tool by the REVAS team, based on 700 manually annotated review comments, was promising, and the ARR editor team decided to move forward with a larger-scale experiment in May.

How will this initiative proceed and be evaluated?

A link to REVAS was made available to reviewers in the March 2026 cycle. In 120 reviews, REVAS flagged 209 comments in the initial review on the 4 metrics supported at that time. Subsequently, reviewers have taken the feedback into account and edited their review. According to the REVAS analysis, this resulted in an overall improvement as measured by the tool:

These results are promising, but, needless to say, a wider-scale evaluation is needed. Experimental use in the May cycle will allow more reviewers to test the tool, and ARR will then be able to evaluate the quality based not only on user perceptions or automated metrics, but also on the effects (if any) on the number and/or composition of author-raised issues with peer review quality, which are a part of the ARR system in OpenReview.

What role does REVAS play in the ARR review process?

REVAS is an external tool accessible via a link in the review form. The reviewers may opt in to paste the weaknesses section of their draft review to REVAS and receive feedback. It is entirely up to them to evaluate the feedback and act on it if the suggestions are appropriate. REVAS does not suggest the review scores and plays no role in any further actions in the peer review pipeline or decision-making.

How can the authors tell if REVAS was used in a given review?

The review form includes a question about disclosure of use of AI tools by the reviewer. This question is visible to the authors, and disclosing the use of REVAS is one of the provided options.

How is confidentiality preserved?

• REVAS does not have access to any submission materials except the text of the review.
• REVAS is fully self-hosted by MBZUAI. No data is shared with third parties, or commericial model providers, and all models are run in the MBZUAI compute cluster.
• All data is carefully encrypted and only authorized users can see their own data.

Is it possible to get involved or provide feedback?

The REVAS team actively collects feedback on the tool here. If you’d like to stay involved with what’s coming next, you can sign up as a beta user. If you’re an expert in peer review or AI4Science and would like to get involved with REVAS, feel free to reach out to Prof. Iryna Gurevych.

At ARR, we would appreciate help from experienced volunteers (area chair and above) to assist with validating/annotating/analysing the results. If you would like to volunteer for that, please sign up here. For general feedback about this experiment to the editors, please reach out to editors@aclrollingreview.org. Discussion can also occur in the business meetings at *CL conferences, which typically have representatives from both the ACL exec and ARR.

The main updates clarify the types of peer review data that may be donated and collected. Under the updated protocol, collected data may include paper drafts, review text, review scores, metadata, before-and-after-rebuttal versions of reviews and scores, and author-flagged review issues.

The updated agreements also clarify the grace period for public release of donated data. Donated data for accepted papers may be made publicly available after acceptance. For papers that are not accepted, donated data will remain confidential for two years following the submission date and may be released publicly thereafter.

Authors, reviewers, and meta-reviewers can find the full updated protocol here: https://arr-data.aclweb.org/protocol/

ARR Data Collection is a community-driven effort to collect and curate a large-scale dataset of peer reviews and associated metadata from the ACL community, with the goal of supporting research on peer review and improving reviewing practices in our conferences.

If they get assignments, AC checklists must be completed by Jan 18th.

Reviewer checklists must be completed by Jan 21st. The deadline for reviews is February 7th, and for meta-reviews March 3rd. In the event of any emergencies, the chairs should be notified via the emergency declaration form.

From this cycle, we intend to strictly monitor adherence to reviewing responsibilities.

• Papers whose authors have not registered as reviewers or ACs or have not completed their checklists as Reviewers or ACs by the designated deadline will be automatically desk-rejected.

• Reviewers/ACs who remain unresponsive and do not submit their reviews or meta-reviews on time will be prevented from accessing reviews and/or meta-reviews for their own papers.

• Highly irresponsible reviewers may become ineligible from committing their paper(s) to ACL 2026 and (re-)submit or commit to the next cycle. The submitting authors should (a) make sure that all other authors are aware of this policy, and (b) check that everybody on their team(s) submits their checklists and their (meta-)reviews on time and in accordance with the guidelines.

All authors will have to register as reviewers or meta-reviewers (Area Chairs-ACs) on ARR. If you are already serving as Senior Area Chair (SAC) this is not necessary.

Papers whose authors have not registered as reviewers or ACs or have not completed their checklists as Reviewers or ACs by the designated deadline will be automatically desk rejected.

Reviewers/ACs who remain unresponsive and do not submit their reviews or meta-reviews on time will be prevented from accessing reviews and/or meta-reviews for their own papers. Further penalties may be incurred, for example by barring submission to a subsequent cycle.

After analyzing the server logs, OpenReview leadership met with ARR and informed us that the impact on ARR was very minor in comparison to other conferences hosted by OpenReview (especially ICLR). Since nearly all reviews had been finalized when the incident happened, only a small number of very late third reviews could have possibly been unduly influenced. However, there were nine papers where the area chair or senior area chair’s identity had been compromised. Consequently, we decided to replace the ACs and/or SACs of these papers to ensure that they received objective meta-reviews and to reduce the risk of retaliation against the AC or SAC for a negative meta-review. Please note though that we have no evidence that the unauthorized queries were issued by the authors, so the action we took was out of precaution.

For the next few cycles, ARR will additionally ensure that resubmissions receive new reviewers if the identities of the earlier submission’s reviewers were leaked.

The ARR October 2025 Editors-in-Chief and EACL 2026 Program Chairs

Our past experience suggests that only a small fraction of the papers submitted through ARR are based on the MS Word Template. However, they receive a comparably high number of desk reject flags, because most reviewers are not used to the considerably different layout resulting from this template. The processing of this desk rejects increases the workload on (S)ACs, PCs, and EiCs, and is stressful for the authors.

The rationale for maintaining the Word template has been to remain open to research communities from other disciplines who may be less used to LaTeX. However, at this point Overleaf also offers a WYSIWYG interface, the ACL template is available on this platform, and to the best of our knowledge it is not banned in any countries that have restrictions on e.g., Google products. This makes the burden for the few authors still relying on the MS Word Template more bearable and reduces the learning curve to adapt to LaTeX. At the same time, the overall benefit for the community is high enough to justify this move.

This move would also facilitate the effort of automating some of the formatting checks post-submission, in order to reduce the workload on Reviewers, (S)ACs, PCs, and EiCs. In the long run, we hope it will help to shorten the paper checklist.

ARR is increasingly used as the main submission channel for *CL conferences. In the recent two cycles for ACL and EMNLP, respectively, ARR has received over 7000 submissions in each cycle. To continue supporting these large cycles, there is a pressing need to scale the EiCs team managing the 10-week cycles. EiCs rotate to lead reviewing cycles. However, the entire team coordinates and oversees the following activities:

• Recruit and manage (senior) area chairs ((S)ACs) and reviewers
• Coordinate with conference program chairs, becoming guest PCs during cycles tied to *CL conferences
• Oversee review forms and CFPs
• Review and update submission tracks
• Make final desk reject decisions (in collaboration with PCs in conference-associated cycles)
• Provide reports to the ACL Exec
• General work on the ARR system hosted on OpenReview
• Coordinate with the rest of the ARR team, including support, communications, etc.
• Participate in review policy development in coordination with ACL peer review committee, Publication Ethics Committee and other ACL bodies

The ideal candidate would have a strong publication record across more than one NLP research area and a solid record as an outstanding member of the *CL community. Experience serving in ARR as SAC and/or AC is highly valued, as is having served as a past PC in one of our top *CL conferences and/or as an EiC for one of our *CL journals. At a minimum, candidates should have at least some experience in managing submission and reviewing processes, for example, as workshop co-organizers.

Being an ARR Editor-in-Chief is a major, ongoing time commitment requiring, on average, 2-3 hours a week but absorbing a significant amount of time during peak cycle phases. When an EiC is co-leading a cycle, it is expected that they can dedicate a large portion of their time during these peak critical times. However, contributing to our community this way is also a very rewarding experience. We are aware that the ARR process needs to continue to improve; we are looking for candidates with ideas on improving what we do who are also eager to dedicate their energy and time to help us implement better reviewing practices and workflows.

ARR is a diverse team, and we are determined to keep it that way. This includes researchers at different career stages, although we recognize that due to the requirements we specified, most junior researchers might not yet be eligible.

Nomination procedure

Nominations should be submitted via email to EiC member Jing Jiang (jing.jiang@anu.edu.au), with a Cc to EiC member Xiaodan Zhu (xiaodan.zhu@queensu.ca), by August 24th, 2025, subject line “Nomination for ARR co-EiC”, and should contain the following information:

• Name, email address, and website of the nominee
• If not a self-nomination, confirmation that the nominee is willing to serve (a dated excerpt of an email from the nominee suffices); if a self-nomination, a statement affirming willingness to serve.
• Statement of nominee’s goals/vision for ARR and/or relevant prior experience and/or reasons for interest in the position. The length of the statement is up to the nominator, but 1 paragraph may suffice, and more than 4 paragraphs are not anticipated to be necessary.

Nominations will be acknowledged by reply email; should acknowledgment not be received within 3 days, please resend your email.

The existing ARR EiC team will evaluate the nominations and make decisions, after which the search committee chair will contact the selected nominee via email.

While we recognize that case-by-case consideration might still be required, we provide the following as a rough guideline to inform both authors and future ARR Editors-in-Chief (and conference Program Chairs). The categories below stem from the assumption that reviewing as a service is a mandatory part of our work as researchers and inextricably linked to authoring research papers, consistent with the view that submissions foster a partnership between authors-reviewers and the conference. Most of us are busy people, but when we commit to being a co-author of a given submission, we also accept the responsibilities that come with that, and that includes budgeting the time for reviewing (or making plans with secondary reviewers). If an author is serving the same community under a different role, we consider their obligation fulfilled. The categorization has been developed by the EMNLP 2025 PCs together with ARR, and may evolve in response to community feedback in future cycles.

We consider the following peer review service–related reasons acceptable as exceptions to reviewing requirement (* marks NLP venues or strongly related venues with significant overlap in dates):

• AC/SAC for the same venue or a related* venue
• Program co-chair/local chair/general chair in the same or related* venue
• Editor-in-chief of a major journal*

We also consider the following personal reasons as acceptable under certain circumstances (e.g., the condition being unexpected or beyond the control of the individual):

• Medical emergency
• Parental leave
• Family medical leave (e.g., taking care of a sick parent)
• Other emergency (e.g., natural disaster)

Finally, we will consider exemptions for authors that although technically qualifying, feel that they cannot fulfil their duties due to lack of expertise. Again, these will be approved on a case-by-case basis.

We consider the following as unacceptable reasons:

• Department Chair, Dean, Associate Dean, Center Director
• Emeritus professors
• Being on sabbatical
• Being on vacation
• Reviewing for other conferences (whose service dates aren’t overlapping)
• Performing other service (not listed above)
• Changing jobs
• Studying for an exam
• Busy with work / over-commitment
• Known or pre-existing medical conditions (present at the time of writing the paper)

While some authors in these circumstances may feel that they are too busy to adequately review, ARR does allow the use of secondary reviewers that may help ameliorate concerns around time commitments.

What this means

If your paper gets accepted to EMNLP 2025, your completed responsible NLP checklist will be published alongside your paper as an appendix. This is a new requirement that will also apply to subsequent *ACL conferences.

Background

Authors are already familiar with the Responsible NLP Research Checklist - it has been part of ARR submissions for some time now. It covers areas such as:

• How you collected and annotated data
• What datasets you used and their limitations
• Your training and evaluation setup
• Potential impacts of your work
• Computational costs

We have always used these checklists during review. Filling them out incorrectly or misleadingly can result in desk rejection.

What changes for authors

If you are planning to commit to EMNLP 2025, here is what you need to know:

Your checklist responses will be public if your paper is accepted. Authors will therefore want to be extra careful and thorough when completing the checklist. Checklists do not count against page limits. They are included as an appendix.

Why we are doing this

Making these checklists public should help with transparency. Other researchers will be able to see how you handled ethical considerations, which datasets you used, what limitations you acknowledged, and so on. This may also encourage people to think more carefully about these issues when they know their answers will be visible to the broader community.