The most recent service qualifications at ARR are based on the sustainable reviewing policy, and supercede the earlier qualifications criteria.
Reviewer qualifications
The updated reviewer qualifications are as follows:
- Those who have obtained their PhDs and/or have senior academic positions (postdoc/faculty), automatically qualify if they have at least 2 publications in major ACL events, Findings of the ACL, or major ML venues.
- Doctoral students (but not bachelor and master students), as well as industry researchers with at least a Masters degree, automatically qualify if they have at least 2 publications in major ACL events or Findings or major ML venues and a total of >= 3 publications in total.
- Those who do not automatically qualify by these criteria can serve as secondary reviewers, mentored by a qualified primary reviewer (see (4) below), but cannot be designated service contributors by themselves.
Venues considered ‘major ACL’ are: ACL, CL, COLING, CoNLL, EACL, EMNLP, HLT, IJCNLP / AACL, LREC, NAACL, TACL, *SEM. Venues considered ‘major ML’ are: AAAI, COLM, CVPR, ECCV, FAccT, ICCV, ICLR, ICML, IJCAI, JAIR, JMLR, NeurIPS, TMLR, TPAMI. Venues considered major may be expanded over time and additional qualification criteria may be added over time as appropriate. For some of ARR tracks, major relevant venues include SIGIR, ECIR, WWW, ICWSM, CHI, Interspeech and ICASSP.
AC qualifications
The ARR area chairs must qualify as reviewers, and additionally:
- Either a completed PhD, or a main conference *CL publication from more than 5 years ago
- Four papers in main ACL events or Findings
- Two additional papers in either main ACL events or major ML/AI venues
- Extensive reviewing experience
How to check if I am qualified?
The ARR submission form opens two weeks in advance (available from September 28 for October’26 review cycle). ARR team is currently working on an automated tool to check reviewer qualifications and streamline applications for manual verification. This tool will be announced separately as soon as possible, and linked from the submission form.
How to apply for manual verification?
Those who do not formally qualify under the above criteria, but have substantially equivalent experience, may apply with documentation of their expertise and review experience for case-by-case consideration (e.g. with records from other venues where they served). In particular, those who have been sufficiently mentored as secondary reviewers at major ACL venues (see (4) below) may be accepted as reviewers, even if they do not meet the strict publication thresholds, on a case-by-case basis.
The sustainable reviewing policy also proposes for ARR to develop a reviewer certification system, based on a combination of knowledge of ARR review criteria and secondary reviewer record. Doctoral students who do not yet have a sufficient publication record themselves, may be considered qualified if they pass a test of knowledge of ARR review criteria (to be developed), have successfully served as secondary reviewers for at least 10 submissions across at least 2 review cycles (with no ‘irresponsible reviewer’ flags from area chairs, active engagement during the author rebuttal, no evidence of using AI to generate reviews, etc.), and are highly recommended by the primary reviewers who mentored them.
ARR is currently in the process of streamlining the manual application process. More detailed instructions will be posted here soon.