One major benefit of ARR is that we can steadily improve the peer review process. Some ideas for these changes come from the ARR team and others came from the community. This page describes what we have done and what are are considering for the future.
Plans
April 2024
- Switching to the OpenReview API v2, this is a necessary change that will have some benefits in terms of what we can do.
Suggestions
The following ideas have been suggested and may be considered in future ARR development:
Process and Policy
- Adding a category of “training” reviewers. Assign them to papers in addition to the normal reviewers. This will help grow the pool of reviewers over time.
OpenReview UI
- Show AEs and SAEs the meta-review scores in the list of papers.
- Show SAEs, AEs, and reviewers the planned schedule in the OpenReview interface.
- A formal way to indicate secondary reviewer contributions.
- Give authors the ability to download all materials for a paper as a PDF, maintaining anonymisation.
- Mechanism for SAEs to nominate good/bad AEs.
Website
- Create a public record of who actually did reviewing, etc for each cycle.
- On the stats page, show resubmission stats as distributions rather than with box plots (at the moment it is hard to see what the median is).
Other
- A mentoring system for new reviewers.
- Create a system that detects low-quality review comments at the sentence level. This could either (a) be shared with reviewers, encouraging them to make changes, or (b) shared with AEs, who can then nudge reviewers.
- Encourage authors of resubmissions to mark up the changes. Maybe do it automatically?
- Automatic checks for length violations.
- Integrating the review system of ARR into OrcID so that there is a method for showing reviewing contributions.
Implemented Improvements
These are some of the ideas that have been implemented since ARR started.
Process and Policy
- Do not show previous reviews to new reviewers until after they have submitted their review. This avoids biasing their opinion.
- Return to 3 reviews per paper, to reduce reviewing load.
- Create a board to oversee ARR.
- Make requesting new reviewers and/or a new AE for a resubmission easier and have the request be accepted by default.
- Add tracks.
- Ensure there is at least one senior reviewer per paper.
- Ensure that no two reviewers for a paper are from the same research group.
- Focus the review process on *CL conferences only, to make expectations clearer.
- Switch to a longer cycle to ensure all reviews are received on time.
- Introduce author response.
- Introduce senior action editors to help manage the review process.
- Introduce ethics review process.
- Introduce ‘soundness’ to the review form.
OpenReview UI changes
- Form for AEs to nominate good/bad reviewers.
- Improve and clarify the user interface to encourage more discussion.
- New emergency reviewer tracking and recruitment interface.
- Integrate the responsible research checklist into the submission form.
Website
- Create clear and complete guides for authors, reviewers, and venue organizers.
Other
- Dual-submission checking support.
- Extracting service records, to speed up the creation of service certificates.