In May and June, the ACL reviewing committee conducted a survey on the future of reviewing in the ACL community. Follow this link for results and analysis of the survey, including both the multiple choice questions and the free-text responses.
Based on those responses, the panel discussion at ACL, feedback from the ACL and NAACL Program Chairs (PCs), and discussion with the ACL Executive, we have a range of updates about ARR:
Changes that are already in effect: * An ARR board is being created to oversee ARR, including approving new ARR initiatives, approving new ARR Editor in Chiefs (EICs), and managing collaboration between ARR and conferences. The board will have five members: an ARR EIC, an ACL Executive member, two past PCs, and one other member of the community. * ACL is providing funding to OpenReview (OR) to get the ongoing support of an OR employee for ARR activities as well as all other ACL activities that are carried out through OR. * Additional EICs are being recruited to spread the load of running ARR. As part of recruitment, we are working to ensure continued diversity in the EIC team. The plan is to grow the team to seven EIC in the coming weeks.
Plan for future changes and ARR activities (note, the goal is to complete items by the end of the month they are listed in)
August
- A monthly blog post to share updates on ARR.
September
- Make requesting new reviewers and/or a new AE for a resubmission easier and have the request be accepted by default.
- Adding the ability for authors to identify relevant track(s) for their paper, for AEs and reviews to indicate relevant track(s) for themselves, and for that information to be considered in AE and reviewer assignments. This corresponds to the ‘soft tracks’ option from the survey.
- Adjust reviewer assignment matching to consider demographics to ensure there is at least one senior reviewer and that no two reviewers for a paper are from the same research group.
- Implement author evaluation of reviews.
- Adjusting the review form to support the new ACL Awards Policy.
October
- Release a paper on arXiv about findings from the first year of ARR.
- Reconsidering the wording and structure of the review form to make it be explicitly focused on the main ACL conferences (ACL, AACL, EACL, EMNLP, NAACL).
November
- Refine guidelines for meta reviewers and consider running a tutorial session or mentoring process.
December
- Switch to an 8 week cycle and guarantee all reviews will be ready by the end of the cycle.
- Explore aligning cycles with conferences.
February 2023
- Develop a mentoring system for new reviewers (either new to reviewing or new to the field).
May 2023
- Conduct another survey to get feedback from the community.
July 2023
- Present results and analysis of the survey.
Under discussion / Exploration
- Adding support for subreviewers / secondary reviewers.
- Allow papers that are missing reviews or a meta-review to be committed to conference deadlines.
- Improvements to the user interface and reviewing process to encourage more reviewer discussion.
Thank you to all of the community members who provided input!