ARR Changelog

One major benefit of ARR is that we can steadily improve the peer review process. Some ideas for these changes come from the ARR team and others came from the community. This page describes what we have done and what we are working on. This page lists some ideas for future work that have been suggested, and we are considering.

Currently in the pipeline

  • Hiring a part-time editorial assistant
  • Hiring a part-time workflow manager (to support OpenReview functionality development and ongoing bug fixes)
  • Improving support of the work by best paper committees
  • Improving the ethics review process integration
  • Improving reviewed paper information display for SACs
  • ARR EiCs participate in the work of ACL Standing Peer Review Committee, which is currently considering the core design of the peer review process at ACL
  • Reviewer history (in development at OpenReview with feedback from venues using OpenReview, including ARR)

Changelog

October 2024

  • Updating CFP and author checklist (based on common problems in recent submissions)
  • Flagging review issues by authors, and reporting action on these flags by the ACs (lead: Anna, Jonathan)
  • Improving instructions for author response period (lead: Mausam, Anna)
  • Major updates for the AC and reviewer guidelines (lead: Anna, Michael)
  • Improving ARR email templates (lead: Michael)
  • Creating a tutorial for PCs who work with ARR (lead: Anna, Viviane, Vincent)
  • Updating the Responsible NLP checklist (lead: Anna)
  • Improved public statistics for review cycles (lead: Sudipta)

August 2024

  • Fixing some functionality we lost in the OpenReview API update (lead: Jonathan, Sudipta)
  • Onboarding three new EiCs
  • Starting the hiring process for the OpenReview support position (lead: Thamar)

June 2024

  • Switching to the OpenReview API v2. This broke a lot of our scripts and processes, which we will work to restore in future cycles.
  • ACL board votes to approve funding for supporting ARR infrastructure, including a dedicated contractor for supporting our OpenReview customizations (lead: Thamar)
  • Refining our process for checking the reviewing volunteers (lead: Jonathan)

Past improvements (prior to April 2024)

These are some of the ideas that have been implemented since ARR started.

Process and Policy

  • Do not show previous reviews to new reviewers until after they have submitted their review. This avoids biasing their opinion.
  • Return to 3 reviews per paper, to reduce reviewing load.
  • Create a board to oversee ARR.
  • Make requesting new reviewers and/or a new AE for a resubmission easier and have the request be accepted by default.
  • Add tracks.
  • Ensure there is at least one senior reviewer per paper.
  • Ensure that no two reviewers for a paper are from the same research group.
  • Focus the review process on *CL conferences only, to make expectations clearer.
  • Switch to a longer cycle to ensure all reviews are received on time.
  • Introduce author response.
  • Introduce senior action editors to help manage the review process.
  • Introduce ethics review process.
  • Introduce ‘soundness’ to the review form.

OpenReview UI changes

  • Form for AEs to nominate good/bad reviewers.
  • Improve and clarify the user interface to encourage more discussion.
  • New emergency reviewer tracking and recruitment interface.
  • Integrate the responsible research checklist into the submission form.

Website

  • Create clear and complete guides for authors, reviewers, and venue organizers.

Other

  • Dual-submission checking support.
  • Extracting service records, to speed up the creation of service certificates.