Welcome to the team Anna!
]]>Ethics reviews are a cornerstone of maintaining integrity and responsibility in academic research. As we navigate through a multitude of submissions for our upcoming conference proceedings, it’s vital to ensure that our papers adhere to ethical standards, rules, and regulations. This guide aims to assist reviewers and action editors (AEs) in detecting papers that may need an in-depth ethics review, as well as preparing the reviewers to conduct effective and efficient ethics reviews, ensuring a high standard of compliance throughout our conference proceedings.
Ethics reviews are not just a formality; they are critical to the integrity of academic discourse. In particular, the ethics reviews are meant to assess whether the submitted research exhibits any substantial ethical issues, that is, issues that pose an increased risk of harm outside the current norms of NLP or CL research. It is especially critical to uphold these standards as we engage in research that deals with sensitive data and/or controversial topics. As ethics reviewers, you play a pivotal role in safeguarding these standards.
Before diving into the review process, familiarize yourself with the specific guidelines for ethics reviewing that ARR has drafted. Remember, an ethics review is not just about ticking boxes, nor is it a technical review; it’s about engaging critically with the material and its implications. You can find more information about this and about the process in our previous blog posts:
Guidelines for Ethics Reviewing: https://aclrollingreview.org/ethicsreviewertutorial
ACL code of conduct: https://aclrollingreview.org/responsibleNLPresearch/
One of the critical steps in the ARR pipeline is the flagging of papers for ethics reviewing. This is done by the action editors during the early stages of the reviewing cycle, or by the reviewers during the deeper technical review they perform. They are required to provide a justification along with flagging any papers they deem to require an in-depth ethics review.
This is an important step that routes papers to the ethics chairs which are then assigned for full in-depth ethics reviews after an initial assessment. While ethics reviews are essential in maintaining the integrity of research, it’s vital to discern when they are truly needed. Over-flagging not only burdens the review process but might also hinder valuable research from moving forward.
Based on the past few cycles of ARR ethics review operationalization, we have identified a few patterns that result in false flags. We surface these recurring patterns here as we believe it will be helpful for reviewers and action editors to correctly identify the papers that should be flagged for deeper ethics reviewing, while also eliminating false flags stemming from misconceptions about the ethics reviewing.
During the cycle, we asked Action Editors to indicate reviewers who did a particularly great job. We’d like to publicly acknowledge you all - thank you!
Milind Agarwal, Sumeet Agarwal, Nader Akoury, Miguel A. Alonso, Ion Androutsopoulos, Rico Angell, Diego Antognini, Ekaterina Artemova, Ioana Baldini, Yu Bao, Lisa Beinborn, Gábor Berend, Aditya Bhargava, Laura Biester, Terra Blevins, Emanuela Boros, Ana Brassard, Sky CH-Wang, Pere-Lluís Huguet Cabot, Pengshan Cai, Rui Cao, Yixin Cao, Yekun Chai, Chung-Chi Chen, Guanyi Chen, Yi Chen, Cheng-Han Chiang, Patricia Chiril, Hyundong Justin Cho, Fenia Christopoulou, Iulia Maria Comsa, Sam Davidson, Björn Deiseroth, Daryna Dementieva, Yang Deng, Ritam Dutt, Mohamed Elgaar, Guy Emerson, Nils Feldhus, Besnik Fetahu, Anette Frank, Dayne Freitag, William Gantt, Vagrant Gautam, Ariel Gera, Thamme Gowda, Jialiang Guo, Zhen Guo, Richard Hahnloser, Chikara Hashimoto, Shreya Havaldar, Timothy J. Hazen, Zhengqi He, Maartje Ter Hoeve, Ari Holtzman, Seokhee Hong, Mark Hopkins, Tom Hosking, David M Howcroft, Kung-Hsiang Huang, Taichi Iki, Md Saiful Islam, Eugene Jang, Harsh Jhamtani, Pei Ke, Katherine A. Keith, Douwe Kiela, Tracy Holloway King, Svetlana Kiritchenko, Venelin Kovatchev, Saurabh Kulshreshtha, Jenny Kunz, Tatsuki Kuribayashi, Shuhei Kurita, Shuaiqi LIU, Tsz Kin Lam, Dongkyu Lee, Ji-Ung Lee, Colin Leong, Bryan Li, Dongyuan Li, Lucy H. Lin, Xudong Lin, Tal Linzen, Enrico Liscio, Aiwei Liu, Fenglin Liu, Lu Liu, Qian Liu, Kyle Lo, Yi Lu, Ziqiao Ma, Craig MacDonald, Sumit Madan, Mounica Maddela, Rahmad Mahendra, Mieradilijiang Maimaiti, Sandeep Mathias, R. Thomas McCoy, Niloofar Mireshghallah, Syrielle Montariol, Hyeonseok Moon, Luca Moroni, Maximilian Mozes, Diane Napolitano, Aurélie Névéol, Ansong Ni, Garrett Nicolai, Dmitry Nikolaev, Shu Okabe, Eda Okur, Peyman Passban, Sachin Pawar, Pavel Pecina, Massimo Piccardi, Francesco Piccinno, Tiago Pimentel, Joan Plepi, Linlu Qiu, Leonardo Ranaldi, Yanghui Rao, Vipul Kumar Rathore, Agha Ali Raza, Siyu Ren, Darcey Riley, Wiem Ben Rim, Anthony Rios, Rudolf Rosa, Domenic Rosati, Mukund Rungta, Yves Scherrer, David Schlangen, Sebastian Schuster, Amit Seker, Rico Sennrich, Ori Shapira, Qiang Sheng, Anthony Sicilia, Damien Sileo, Mohammad Golam Sohrab, Yixiao Song, Felix Stahlberg, Julius Steen, Ian Stewart, Shane Storks, Phillip Benjamin Ströbel, Hanbo Sun, Rui Sun, Makoto Takamoto, Ece Takmaz, Chris Thomas, Julien Tourille, Martin Tutek, Asahi Ushio, Sowmya Vajjala, Vincent Vandeghinste, Vijay Viswanathan, MinhDuc Vo, Henning Wachsmuth, Liang Wang, Ruoyao Wang, Zhaowei Wang, Leon Weber-Genzel, Orion Weller, Rodrigo Wilkens, Steven R. Wilson, Hongqiu Wu, Amelie Wuehrl, Kaige Xie, Pengyu Xu, Shuntaro Yada, Jing Nathan Yan, Chenghao Yang, Torsten Zesch, Chen Zhang, Haoran Ranran Zhang, Zixuan Zhang, Jie Zhao, Xiaoyan Zhao, Mingyang Zhou, Shengqi Zhu, Wanrong Zhu, Caleb Ziems
]]>The new policy takes effect in ARR beginning with the Feb. 15 submission and commitment cycles. Under the new policy, submissions will remain anonymous during peer review, but authors are free to post and discuss non-anonymous preprints at any time. To protect anonymity during peer review, ARR will take measures to prioritize reviews by reviewers who are not aware of the author identities. Authors are reminded that widely sharing the work will make it harder to recruit reviewers. (Venues will also institute awards for unpublicized work. These awards are decided by program committees, not ARR.)
The new policy is now reflected in the ARR call for papers, author page, reviewing page, review form, reviewer tutorial, and guidelines for commitment venues.
]]>Reviews are due Nov 20th. A detailed guide has been shared with reviewers and the support team is regularly checking for and responding to questions.
Many thanks for all the support from reviewers in this process!
]]>AEs and SAEs are now reviewing their assignments and making minor tweaks to reviewing assignments where required. We will start the review process on Monday October 30th. We will be sending out instructions to reviewers soon.
October cycle by the numbers:
More soon!
]]>The dates and cycles relevant for this coordinated submission process are as follows (Table 1 below shows key dates by conference and month):
Each cycle will include its corresponding author response period before the meta-review stage. Exact time will be announced in the conference CFPs. Relevant aspects of the plan above are discussed below.
Anonymity period Unless the ACL enacts a change in policy, each conference will follow the current ACL wide anonymity embargo policy where authors cannot release a deanonymized preprint during the review cycle or between commitment and decisions, plus one month before submission and commitment. More information here: ARR Call For Papers and ARR Author instructions.
Acceptance rates To calculate each conference’s acceptance rate, we will record, at the time of submission to ARR, authors’ intention to commit to one of the three conferences. This doesn’t imply a hard commitment. Authors may choose to commit to a different venue after receiving their reviews.
Month | EACL 2024 | NAACL 2024 | ACL 2024 |
---|---|---|---|
Oct 2023 | Last ARR submission deadline: 10/15 | ||
Dec 2023 | ARR reviews available: 12/15 Commitment deadline: 12/20 |
Last ARR submission deadline: 12/15 | |
Jan 2024 | Decision notification: 01/15 | ||
Feb 2024 | ARR reviews available: 02/15 Commitment deadline: 02/20 |
Last ARR submission deadline: 02/15 | |
Mar 2024 | Decision notification: 03/15 | ||
Apr 2024 | ARR reviews available: 04/15 Commitment deadline: 04/20 |
||
May 2024 | Decision notification: 05/15 |
Table 1. Timeline of submission and reviewing activities for the upcoming conferences
Program Committee Each set of PCs will recruit their own team of Senior Area Chairs, these SACs will be the same group of people in ARR and the conference for the corresponding submission cycle. So for example, for the upcoming October cycle, EACL PCs will select their team of SACs. They will serve as SACs in ARR for that cycle, and then also support PCs during the final assessment of papers committed to EACL. This is the team that will help make final accept/reject decisions after commitment of papers. PCs will encourage, and facilitate, discussions among reviewers/ACs and SACs after the commitment of papers, to further improve the final decision process. However, reviewers, and area chairs will be recruited by all teams, although this may happen in stages. So please be prepared to receive an invite to join the reviewing process soon!
Resubmissions to ARR and Commitments to Conferences Once a paper has been fully reviewed, authors have the following (mutually exclusive) options as next steps for their manuscript:
Figure 1 below shows a diagram of the process for an ARR reviewed paper:
Figure 1. Wrofklow diagram of the author choices once a submission has been fully reviewed by ARR.
More detailed information and FAQ about the process can be found on the Authors page
There will be more information coming out soon, but reach out to any of the PCs teams or to ARR in case you have questions.
Email contacts:
Some questions have simple clear answers. Other questions involve policy decisions where we provide the current answer. We acknowledge that other policies could exist. A survey will be conducted in 2024 to get input from the community on the path forward for these kinds of policies.
The system is very convoluted and complicated to understand even for usual ACL members. How do we ensure this doesn’t act as a deterrent for newcomers in the field and hence doesn’t affect diversity?
The process used to be so complex that it excluded some of us who were not as informed/used to it. Can you please make more efforts in simplifying it and explaining it in a SHORT post?
I have just commented on ARR: decision-making is very complicated at ARR. I think it should be made similar to SoftConf
We want to make the process as clear as possible. We have developed new materials to guide authors (video and author page) and we are happy to answer questions. At a high level, the process is the same as submission to a conference, except that after receiving reviews, authors decide whether to have their paper considered for the conference or go through another round of reviews.
Will there be clear cut-offs for when to submit to ARR in order to guarantee reviews back before a conference commitment deadline?
How will timely reviews be guaranteed?
Will the system ensure a max delay to get the reviews to a paper?
Yes, ARR now guarantees reviews are ready within two months after the submission date. Since switching to two month cycles we have consistently achieved this goal.
After I receive reviews, can I revise and commit at the same time? Or I can only commit after receiving reviews from a previous cycle?
Wouldn’t it be great that one can commit in parallel to conferences with the same commitment deadline? Like AACL 23 and EMNLP 23
No, you can choose either to revise or to commit. This is equivalent to standard policies for double-submissions to conferences: while your paper is being considered for acceptance at one conference you cannot simultaneously submit for review to another conference.
Do we submit again or resubmit i.e. do we know the link with the initial paper. Is there any trace kept of the reviewing steps of a given paper (number of review) and who has access to this history ?
Each cycle has a separate submission page. When you make your resubmission you include the link to the previous version. This is provided to reviewers of the current paper. Note, if new reviewers are assigned to the paper they do not have access to the old reviews until after they submit their review (new in August 2023).
The ARR editors and tech/support teams have access to the history of papers and use this to calculate aggregate statistics as shown on the ARR Stats page.
What will the tracks be? Can reviewers chose more than one? If so will review load be considered overall?
The tracks are listed in the submission form and in our CFP. Reviewers can choose multiple tracks. Their total load is computed across all tracks. Authors choose exactly one track for each submission.
Is there a standard for how much a paper must be modified in order to request that old reviews get hidden?
Old reviews are never completely hidden in the current ARR process. As of August 2023, we give access to new reviewers after they have independently reviewed the current paper (before that they cannot see the old reviews).
The biggest problem with ARR now is that ‘no decision’ is made, which leads to frustration and further delays in publication. How can the system deal with that?
The day one design flaw was the fact that you can’t get a decision. In a journal or conference you get a decision. Whereas, no matter how good your ARR scores are your paper may not be accepted. Is there consideration by the ACL Exec or ARR Board of changing that, so you actually get a decision?
With the current structure of ARR, this depends on when conference PCs decide to set their commitment deadlines. This year, the October, December, and February cycles have commitment deadlines immediately following the cycle, which will provide decisions (e.g., a submission on October 15th that is committed to EACL will get a decision by January 15th). In other words, in a conference review system without ARR, a typical review+decision cycle is about 3.5 months. In the same way, in the system with ARR, we can easily simulate older systems by committing to the adjoining conference irrespective of the strength of reviews. ARR actually makes the system strictly more flexible, by creating a post-review decision point for authors, giving them the ability to take more informed decisions in the best interest of their paper. Therefore, while ARR may have this as the biggest problem, it is still strictly better than existing conference review systems.
In the future, assuming ARR continues and the community prefers all cycles to have decisions, there are a range of ways to achieve that:
the motivation for arr was supposed to be to reduce a surge of reviews and to spread across the year but given there is a “submission by” deadline for each conference how is this accomplished?
Spreading reviewer load is one of several motivations for ARR. The hope is that papers that are ready earlier get reviewed in earlier cycles, with a possibility of being posted on arxiv, and then committed to the conference when a suitable deadline comes along. This does happen, though cycles without an immediate commitment deadline have been smaller. This is understandable since the change to several cycles in a year is so recent and our community is still in the process of getting used to our conference deadlines. We expect that as ARR continues to offer multiple cycles, then the community will slowly adapt to this new way of getting our papers reviewed. This, in the long run, should lead to spreading reviewer loads.
Will LREC, COLING, CONLL, WMT, and other NLP conferences be part of this initiative?
Will all workshops be part of the same process?
ARR is happy to provide reviews to a range of venues. It is up to the PCs of those events to choose whether to participate. Note, however, that we instruct reviewers to assume they are reviewing a submission for ACL. That may not be a good fit for venues with narrower and specialised foci, such as workshops.
Are reviews still sticky, or will authors have the option to suppress previous reviews/restart from zero state?
Reviews are sticky, i.e., when a paper is resubmitted the old reviews are available to reviewers. Note, reviewers and AEs are not sticky. Authors may choose to get new reviewers and/or a new AE when they resubmit. If authors do request new reviewers, the current implementation hides old reviews from the new reviewers until they submit their own review, independently.
Will we ever consider public reviews like for ICLR?
Why not make the reviews open in public domain? Making them public would increase the responsibility of the reviewers as well as the authors.
Why not considering making un-anonymous reviews (sign by the reviewers). Quality of reviews will improve and could be publicly accessible. Some journal process this way and the result is much better.
There is no plan to add public reviews to ARR at this time or to reveal reviewer identities to the public. These are both possible within OpenReview and could be implemented if the community strongly supports them.
Did you consider making ARR a full TACL-style journal and thus completely decouple acceptance from conference presentation?
ARR was created based on community input to the ACL reviewing committee. For their original proposal and context for the decision process, see this page.
I think there should be at least two non-student reviewers on each paper. What is the motivation for the relaxed restriction in the current policy?
For context, we currently require (technically speaking, strongly encourage the matching algorithm to produce) at least one non-student reviewer per paper. Requiring more non-student reviewers would be difficult to do while also (a) avoiding heavy reviewing loads and (b) getting the best expertise matching we can. It’s also worth noting that many student reviewers are senior students with significant experience.
Will there be new conferences or will the ACL conference deadlines be more spread out throughout the year?
The timing of each conference’s commitment deadlines is up to the PCs of that conference. For 2024, EACL, NAACL, and ACL coordinated their deadlines with each other and ARR, leading to deadlines in October, December, and February respectively. We are not aware of plans for other conferences next year.
Why does the conference still make an accept/reject decision after ARR decides that a paper is good? Doesn’t this create extra work? What will the role of conference PCs in the future be?
Isn’t it unfair that a 4 or 5 paper is then rejected? What are authors supposed to do with this decision?
Separating accept/reject decisions from the review process is how we preserve PCs’ agency in the creation of the technical program. ARR is meant to support the review process only.
In a typical conference, these steps happen:
ARR only does (1) and part of (2) since Action Editors in ARR provide a score, but not an explicit recommendation.
Note that in a regular conference it is also the case that seemingly “strong papers” get rejected because a score is not the only way to assess the relevance/quality of a paper and PCs have full agency to decide how to create the technical program.
Will the reject after committing to a conference be justified? This time decisions were made without further notice.
Can you highlights reasons why some ARR papers that scored 5 may have been rejected? What could authors do to those papers?
The commitment process is run by PCs. They could provide a justification to authors, but it was noted in the session that it would involve a significant additional effort on their part. Note that in standard conference reviewing, papers with positive scores are sometimes rejected and no additional comments are provided beyond the meta-review. We hope to improve this part of the process with cooperation from conference PCs.
Why was ARR chosen as the sole submission system for 2024 conferences when the ACL 2023 statistics show clear preference for softconf?
This question makes the assumption that submission patterns in 2023 show a clear preference. During 2023, the ARR EICs intentionally worked with PCs to schedule the deadlines so that the direct submission deadline was slightly later, which encouraged people to go direct (since it gave more time before submission). This succeeded, leading to smaller cycles in which the ARR team could focus on improving the system.
The decision to use ARR only for EACL, NAACL, and ACL in 2024 was made by the PCs of the respective conferences.
Finally, we need to separate the infrastructure from the review process. Softconf and OpenReview are software tools we use to support the review process. ARR and direct conference submission are two styles of reviewing processes, run using those tools. PCs could choose various combinations of these. For example, EMNLP 2023 direct reviewing is happening on OpenReview.
if you think the survey results are not valid why not continue in hybrid submission mode and then do a new survey?
We are not sure which survey this is referring to. The 2022 survey indicated strong support for ARR, but also raised a range of issues, which we have worked to address over the last year (see the presentation at ACL 2023). Another survey will be conducted in 2024 to get community input based on the experience of ARR-only submissions at EACL, NAACL, and ACL.
What plans do you have to ensure the community remains informed about all of the steps and all of the changes and questions that arise so that the community feels like it understands the system?
We will be working with the PCs of conferences to share updates on their websites, our blog, and social media. We are also in the process of updating our website (thank you to Nathan Schneider for lots of helpful input and edits!), and have prepare a detailed video with guidance to authors.
Can we remove anonymity period restrictions for ARR submissions? Neurips does not have a ban against pre prints being uploaded at any point so long as they are not advertised as under peer review.
What is the point of the anonimity period? You can still find the article if it was submitted more than a month before. What are you trying to achieve?
How does anonymity/ axXiv work with ARR? Can one post after receiving reviews? If so, does one have to wait 30 days afterwards before resubmitting? Before committing?
How does the anonymity period work in the new system ? I.e. for submitting versions on ARXIV?
Is there a plan to change the policy on non-anonymized pre-prints during ARR? Other similar (non-ACL) conferences allow these to be uploaded to arxiv during the review period.
Are there two anonymous periods or just one, if ARR is going to be used for all ACL confs?
The anonymity policy is set by the ACL Executive. ARR will follow any changes to the policy.
The simplest way to understand the process is that there is an anonymity period for the ARR cycle and an anonymity period for commitment:
Our new video explains this in detail. These two examples may be helpful:
ARR also provides the option of anonymous preprints. When you submit to a cycle you can choose to opt-in and have your paper available with no indication of the author identities.
As a follow-up, how is arxiv and Twitter anonymous, regardless of when it’s done?
Neither arXiv nor Twitter are anonymous. ARR’s provided preprint process is anonymous.
Perhaps use conference acceptance statistics to help measure solidity of individual reviewers?
Not sure we agree. Just because a reviewer agreed or disagreed with an eventual decision should not reflect the quality of their assessment of the paper. A paper may be accepted even when valid points get raised by a reviewer; and a paper may be rejected in spite of some strengths highlighted by a reviewer.
There is no quality information about ARR reviews. The ARR SACs do not take acceptance decisions. So we will have many unexpected rejects a long time down the line. Novelty suffers. Not good.
ARR reviewers are from the broad ACL community only. Therefore, ARR reviews are not expected to be of lower quality. If anything, since reviewing loads may get spread out, at least in leaner cycles, we expect ARR reviews to be of marginally higher quality. As described above, SACs come into the picture at decision making time, not at reviewing time (in both ARR and conferences); so the process is not much different. Indeed, even in typical conference reviewing some papers get repeatedly rejected and their novelty suffers. We do not anticipate that this phenomena will increase because of the decoupling of reviewing and decision making.
ARR is contributing, with reviewer and author consent, to the creation of a dataset of reviews. We hope that resources like that and efforts by our team will lead to improvement in review quality over time.
How do we make sure reviewers don’t ignore rebuttals and actively engage with the authors during multiple rounds
Due to shortened review cycles in ARR, author response is meant for factual inaccuracies only. If your paper indeed receives conflicting opinions, it may be hard to reconcile during the author response phase. It may be better to resubmit to the next cycle and ask for a different set of reviewers.
How do we solve the issue of reviewers not doing the work? That is 2 lines reviews. How do we force non students to review way more? Can we force 1 publication = 6 reviews at least?
This is indeed an issue in our community for all reviewing (not just ARR). ARR does not have a magical solution here. We do agree that creating a centralized reviewing system enables some possibilities that were not there in standalone conference reviewing (e.g., tracking service over time). We have no plans for rules like those proposed in the question. Our current focus is successful implementation for the EACL, NAACL and ACL 2024 cycles.
Over 800 papers reported each R1/R2/R3 during the ACL’23 review cycle (ACL2023 reviewing report). If these many authors request new reviewers in the ARR cycle, is there a large enough reviewing pool?
Our pool is large enough to accommodate many requests. Note that a reviewer may be reported for issues on one paper but do a great job on another that is a better fit for their expertise. Very few of the ACL 2023 reviewers were reported on multiple papers. This means they are not completely removed from the pool for such a report and can continue to contribute, reviewing a paper other than the one they were reported on.
Should reviewer “quality” over time be monitored?
Yes, this is one of the motivations for ARR, but not our current focus.
Will student and non-student reviews be weighted differently? Should the degree obtained (BS/MS/PHD) be the main determiner not the stage (student, non student)? What about speciality areas?
We do not instruct action editors to weigh reviews differently based on reviewer expertise.
Recommendation: let meta reviewers make two venue suggestions if they are happy with the paper.
Thanks for this suggestion. The current structure of ARR is that all reviews and decisions are done with the assumption that papers are intended for ACL or similar conferences.
For ACL23 there was approximately 1 month more time for direct submission than ARR. Will time from submission to publication increase?
The timing of the 2023 conference deadlines and the ARR deadlines was intentionally set to reduce the load on ARR. In the coming year, when conferences are ARR-only, the time will be shorter (e.g., if you submit on October 15th, and commit to EACL on December 15th, you will receive a decision by January 15th or thereabouts).
What will be the relation with TACL and ARR in the future? Will TACL also move to the ARR platform?
What role does TACL serve in an ARR-only world?
We are not aware of plans to move TACL to the ARR platform. For the foreseeable future, TACL will continue as it is now, both in terms of the review process and appearance at conferences.
I submitted to ARR last year. Revised and resubmitted. Did not request new reviewers but still got THREE new reviewers…what is the point if there is no continuity?
There are two possibilities here. First (more likely), early in 2022 there were issues with the reviewer reassignment process, which have now been resolved. Second (less likely), if all three reviewers indicated they were unavailable then we would not have been able to assign them. Note, now when we ask reviewers for their availability, we give them the option to say that they will have bandwidth to review resubmissions, but not new papers.
How do I know when my reviews are “good enough”?
For context, this question was asked online during the session after a comment that you can go through multiple cycles of ARR until you are happy with your reviews. The decision is subjective and depends on the reviews. In terms of scores, as shown at ACL, papers with a meta-review score of 5 almost always get in, papers with a score of 4 usually get in, and papers with a score of 3 sometimes get in. The reason there is not a clear mapping from numbers to decisions is that the content of the reviews is important - the issues raised by reviewers may vary in significance.
Are there clear guidelines for what counts as a “good review”? Is there numerical guidance per conference for what is likely accept or reject?
In order to evaluate the quality of a paper we need to know to which conference this paper will be intended/committed. Reviews are conditioned on the venue and ARR is decoupling this.
As a SAC at NAACL 2022 it was extremely difficult to make decisions because the (meta) reviewers didn’t make recommendations for a specific venue - how should ACs make recs w/o knowing the venue?
We ask all reviewers to treat papers as submissions to ACL or a comparable conference. We have guidelines for reviewers and the numerical scores have descriptions.
What is the evolution on reviewer load? Early sign-ups had the problem of committing to (up to) 6 papers each cycle, AFAIK.
In the October, December, and February cycles, we expect loads to be comparable to regular conference loads. For cycles that do not have a conference commitment deadline immediately following, we expect loads to be low (at most 1-2 papers per cycle).
Many of your focus is on helping authors. How are you working on reviewer burnout?
Every cycle, we ask reviewers what load they are able to take on. This allows them to avoid burnout by adapting based on their own personal situation. Moreover, some of the reviews are for resubmissions, which require significantly less time to review.
How are we going to deal with proceedings compilation such as SoftConf?
An effort outside of ARR is underway to improve the proceedings compilation process from OpenReview that is supported by the ACL exec (and is needed for EMNLP 2023).
why don’t we do a simple referendum to the NLP community– do you want ARR or not. We can decide on that rather than some small group deciding on something that affects the future of many.
Why is the ACL community (as we pay the memberships) not a part of deciding this important part of the research workflow?
ARR has been the focus of two large community surveys, first in 2020 for general feedback on the proposal, then in 2022 for feedback on the first experience with ARR. The second survey included Likert-scale questions to gauge support for ARR, and the community broadly supported it (see the full results here). The survey also raised a range of issues, which we have worked to address in the past year.
In addition, we want to highlight that the decision for going fully ARR for the upcoming EACL, NAACL, and ACL conferences was made by each PC team after discussions with ARR and with input from the ACL exec and the ARR board. As has been the case in past *CL conferences, PCs have full agency on deciding how to manage their review process, and this time it is not an exception to this.
Process question: If I commit a paper to a conference but it gets rejected, am I allowed to try again and commit it to another conference?
Yes.
For ACL, there were six sources of presentations that appeared on the main conference schedule. (1) Direct submissions (2) ARR (3) SRW (4) TACL (5) CL (6) Industry track Which will use ARR now?
Of those, “(1) Direct submission” will not exist in 2024 for EACL, NAACL, and ACL. The others will continue as they are.
Will ARR adopt the soundness/excitement breakdown? How frequently will there be changes/experimentations with new scoring systems?
will the multidimensional review score approach be taken ? if so which one?
Options for this are being discussed with the EACL, NAACL, and ACL PCs.
Regarding the future, we are currently focused on running the ARR process smoothly, before considering further changes.
Why not doing the following: reviewers submit reviews, authors respond to reviewers and – at the same time – revise the paper accordingly. If the paper is accepted, that will be the camera ready!
We are not currently considering this proposal, Any changes to our review process need to be made in consultation with the ARR board. For major changes, consultation with the community would also be needed.
Does ARR necessarily imply OpenReview? Can we consider doing one change but not the other?
Of these options:
1, 2, and 4 can be done, but 3 would be difficult. In the 2022 survey (see here), we described (2) as “Do not keep rolling review, but do have an integrated reviewing system, with improvements / modifications”.
how will we communicate acceptance rates in a way that will be meaningful to those who make hiring and tenure decisions based on elite publication metrics?
Several options were proposed in the original ARR proposal. ACL 2022 reported two values in the proceedings:
“(a) (Number of accepted papers at ACL 2022) / (Number of papers that selected ACL 2022 as the preferred venue in ARR or were committed to ACL 2022). For ACL 2022, for the denominator we consider the 3,378 papers as explained above. Thus, the acceptance rate is 701 / 3,378 = 20.75% for the Main conference.”
“(b) (Number of accepted papers at ACL 2022) / (Number of papers committed to ACL 2022). For the denominator, we had 1,918 papers committed to ACL 2022, and thus, the acceptance rate is 701 / 1,918 = 36.54% for the Main conference.”
If people resubmit once before the deadline, how is this decreasing the reviewing load?
This question assumes that everyone submits once before the deadline. In reality some authors choose to submit to non-conference cycles. Moreover, for resubmissions, if the authors do not request new reviewers then the review process should be less of a load as the reviewers are already familiar with the paper.
Shorter people have trouble with QR codes etc when they are at the very bottom of the screen
This comment is about the layout of our slides in the plenary session. Thanks for mentioning this, we will keep it in mind for future presentations.
How do you obtain transparency without being open which has the possibility that leads to dissemination of harmful code?
We have an ethics reviewing process, which deals with potential harms of research. If by the question, you mean an author purposely providing harmful code, while saying something else in the paper - this is hard to handle right now, along with other similar issues like an author purposefully fabricating the data.
Do you think a four-months round for improvement (2 for review, 2 for incorporating changes) is satisfiably fast in a fast-pacing field like ours? Won’t the work on conferences become more outdated?
Any peer review process needs to give a meaningful amount of time to the peers for providing a rigorous review. Shortening this time will result in a further reduction in review quality and timeliness - issues that our community is already dealing with.
Most conferences have an exclusivity policy. How does that work at ARR?
See our CFP for details on our dual submission policy. The general principle is the same as for other venues: while a paper is under review at ARR it cannot be submitted elsewhere and vice versa.
Will the conferences in the coming year all be using the same review form, the ARR form? What advice would you give to PCs about interpreting the main score in the form?
Yes, they will all use the same form. This is simplest for PCs as it means they can get papers from any cycle and have consistent information in the reviews. We are guiding reviewers to treat the overall score as they would for submissions to ACL and PCs are aware of that approach.
]]>We’re super excited to have you on board, Malihe; welcome!
]]>