As we continue to add incremental changes to our review process, and OpenReview (OR) infrastructure, we want to highlight an important change that will be implemented for the upcoming April cycle: a reviewing workload requirement.
We are very happy to see the larger number of submissions coming due to our major conferences adopting ARR as their default reviewing mode. We hope this continues and would like to clarify that we’re always open to receiving submissions from anyone, especially newcomers to the field!
However, with increasingly larger numbers of submissions, we face the need to implement a more robust approach to covering reviewing needs. Therefore, with the support of the ARR Board, we have decided to adopt a reviewing workload requirement for papers. This requirement has to be satisfied at the time of submission.
What does this mean for authors?
It means that during submission time, authors have to specify which person, among the coauthors in the current submission, will be covering reviewing for at least three papers and to certify that whoever has been specified as reviewer meets the following conditions:
a) Has an updated OR profile1 (the person has provided their semantic scholar and an updated email address where they can receive OR messages).
b) Has published at least three papers in our *CL conferences2 in the last 5 years (note exceptions below).
c) Once added as a reviewer, the person promptly completes their reviewer registration form for the cycle.
Understandably, there will be several exceptions to this requirement, and we will be more than happy to accommodate papers falling under reasonable exceptions, including:
- A paper whose authors are new to our community.
- A paper whose authors have not met requirement b).
- A paper whose authors are already involved in the reviewing process in some capacity (as Area Chairs, as Senior Area Chairs, etc).
Authors can indicate another reason to request an exception that will then be reviewed by the EiCs.
Unless the paper falls under categories 1-3 above, or there is another valid reason to request an exception, the paper will be subject to desk rejection if authors fail to comply with the reviewing workload. This means we will verify that we have a complete OR profile and that those authors identified as reviewers have fulfilled their max load and expertise task. In the past, as per our CFP, all authors qualifying to review were automatically added to the reviewing pool. But we faced challenges with many of these authors, they either: (1) did not have a complete OR profile, and thus we were unable to assign them papers, or (2) never answered our email requests to specify max load/expertise.
This new policy may seem too harsh to some or even a tit-for-tat transaction. We would like to point out that helping out to cover reviewing for three papers is a low bar that doesn’t really cover the workload of having a paper complete a full review cycle. A full review cycle includes effort from the Senior Area Chair to check that the reviewing assignments are sensible, an Area Chair (AC) that will help monitor reviewers’ progress, as well as help to engage them during author rebuttal. Then, the AC needs to review the paper, digest the reviews, the authors’ response, and the reviewers’ reply, if any, in order to then write a meta-review. Finally, there is another set of people who may engage with the paper during commitment. This can’t be considered a tit-for-tat. Our goal is that the review process is a bit less stressful by securing some fraction of the reviewing capacity needed during the submission process. Furthermore, the more people are in the reviewing pool, the better paper-reviewer matches can be secured, which means a better experience for reviewers and higher-quality feedback for the authors.
As always, we welcome your feedback, send an email to editors@aclrollingreview.org. We will continue to review our processes with the goal of improving the overall reviewing experience for everyone, and increasing trust in the community.
Footnotes:
-
Regarding the first requirement. To understand what an up to date OR profile looks like, see this example:
In contrast, this example profile is missing all the key fields:
↩
-
The term *CL conferences is used loosely here. We include AACL/IJCNLP, ACL, CL, COLING, CONLL, EACL, EMNLP, NAACL, and TACL. We don’t distinguish between short and long papers, and we do count Findings papers. ↩