In the March 2026 ARR cycle, we for the first time encouraged reviewers to try the experimental review support tool called REVAS. The instructions were communicated to the cycle reviewers, but we realized that we never wrote a blog post describing the initiative in more detail. We do this now, and apologize for the delay.
What is REVAS?
REVAS is a system developed at TU Darmstadt and MBZUAI that — with support from the ACL exec — is intended to be continuously maintained and developed for the next two years to help improve the peer review quality at *CL venues. It looks like this:
A key principle distinguishing REVAS from other review assistants is that it adheres to the publication ethics guidelines: it does not suggest content for the review. It only provides feedback on whether the review satisfies some of the criteria that are in general desirable for a good review, such as specificity. From the authors’ perspective, the reviewer adherence to such general criteria is both desirable and expected.
As with any automated tool, REVAS is intended only as a source of suggestions, not a replacement for human judgement: i.e., a review not flagged by REVAS is not necessarily amazing, and a review flagged by REVAS is not necessarily truly at fault. ARR is experimenting with REVAS to establish whether such a tool can help to raise the overall review quality level and improve the community awareness of the relevant peer review norms and guidelines.
The REVAS pilot is completely independent of (and complementary to) the EMNLP 2026 initiative on providing authors with AI-generated feedback. The REVAS experiment is an ongoing initiative of ARR, rather than any specific *CL conference.
What review quality criteria does REVAS support?
In March 2026, the following criteria were supported: * Actionability * Grounding & Specificity * Verifiability * Helpfulness
See more details on how these criteria were operationalized in this EMNLP-25 paper.
For May 2026, the tool was extended to more directly support the ARR reviewer guidelines, in particular 10 of the 17 currently listed reviewer heuristics are now checked by REVAS:
- H1. The results are not surprising
- H3. The results are not novel
- H5. The results do not surpass the latest SOTA
- H7. This method is too simple
- H9. The topic is too niche
- H11. The paper has language errors
- H13. The authors could also do [extra experiment X]
- H14. The authors should compare to a ‘closed’ model X
- H15. The authors should have done [X] instead
- H16. Limitations != weaknesses
Preliminary evaluation of the tool by the REVAS team, based on 700 manually annotated review comments, was promising, and the ARR editor team decided to move forward with a larger-scale experiment in May.
How will this initiative proceed and be evaluated?
A link to REVAS was made available to reviewers in the March 2026 cycle. In 120 reviews, REVAS flagged 209 comments in the initial review on the 4 metrics supported at that time. Subsequently, reviewers have taken the feedback into account and edited their review. According to the REVAS analysis, this resulted in an overall improvement as measured by the tool:
These results are promising, but, needless to say, a wider-scale evaluation is needed. Experimental use in the May cycle will allow more reviewers to test the tool, and ARR will then be able to evaluate the quality based not only on user perceptions or automated metrics, but also on the effects (if any) on the number and/or composition of author-raised issues with peer review quality, which are a part of the ARR system in OpenReview.
What role does REVAS play in the ARR review process?
REVAS is an external tool accessible via a link in the review form. The reviewers may opt in to paste the weaknesses section of their draft review to REVAS and receive feedback. It is entirely up to them to evaluate the feedback and act on it if the suggestions are appropriate. REVAS does not suggest the review scores and plays no role in any further actions in the peer review pipeline or decision-making.
How can the authors tell if REVAS was used in a given review?
The review form includes a question about disclosure of use of AI tools by the reviewer. This question is visible to the authors, and disclosing the use of REVAS is one of the provided options.
How is confidentiality preserved?
- REVAS does not have access to any submission materials except the text of the review.
- REVAS is fully self-hosted by MBZUAI. No data is shared with third parties, or commericial model providers, and all models are run in the MBZUAI compute cluster.
- All data is carefully encrypted and only authorized users can see their own data.
Is it possible to get involved or provide feedback?
The REVAS team actively collects feedback on the tool here. If you’d like to stay involved with what’s coming next, you can sign up as a beta user. If you’re an expert in peer review or AI4Science and would like to get involved with REVAS, feel free to reach out to Prof. Iryna Gurevych.
At ARR, we would appreciate help from experienced volunteers (area chair and above) to assist with validating/annotating/analysing the results. If you would like to volunteer for that, please sign up here. For general feedback about this experiment to the editors, please reach out to editors@aclrollingreview.org. Discussion can also occur in the business meetings at *CL conferences, which typically have representatives from both the ACL exec and ARR.