Quick links: 2024 Submission Sites * How to Submit on OpenReview
This page contains a step-by-step guide and FAQ for authors who wish to have their work reviewed by ARR: - Step 0: Is ARR right for your paper? - Step 1: Submit your paper for ARR review - Step 2: Wait for reviews - Step 3: Receive reviews and choose next step - Step 4: Resubmit for another round of review (optional) - Step 5: Commitment - Step 6: Camera-ready - Frequently Asked Questions
We have also created a detailed video presentation covering this information: https://www.youtube.com/watch?v=DaoCLEghXyU.
Step 0: Is ARR right for your paper?
ARR provides reviews—and only reviews—for submissions. The reviews will not be specific to a conference/venue, but the standards expected by reviewers are those of a traditional direct submission as a long or short main conference paper to ACL (or the other major conferences operated by the ACL). - Short papers (up to 4 pages) as well as long papers (up to 8 pages) - All NLP/computational linguistics topics are within scope. See the CFP for example areas. - Empirical papers, theoretical papers, resource papers, application papers, opinion pieces, and surveys - Submissions must be anonymous - Submissions must be original work - Submissions must not be under review, published, or on track to be published in another (archival) venue
Some caveats: - System demonstrations should NOT be submitted to ARR. Conferences typically have a separate review process for them with different policies (e.g., different paper lengths and single-blind submissions). If you do submit a paper to ARR that describes a system, it will be reviewed as a normal main conference paper and your reviews will only be eligible for commitment to venues listed on our venues page (which does not include system demonstration tracks at conferences). - If you intend to send your paper to a workshop or to a more specialized conference like CoNLL, *SEM, INLG, etc., check whether it is listed on the venues page. If not, there may be venue-specific review processes. - Some venues are hybrid, meaning they will consider ARR-reviewed papers as well as direct submissions.
Step 1: Submit your paper for ARR review
There is a submission deadline every two months: see dates and venues. Deadlines are firm.
The submission form asks for: - The PDF of your paper, adhering to the submission guidelines, including the formatting in the submission templates, which is available as an Overleaf template - Optional: supplementary materials, including software and/or code - Answers to questions from a responsible research checklist - A short sentence describing your paper - Select tracks from the list in the CFP - URLs for any non-anonymous preprints - [for resubmissions] The link to your previous submission (e.g., https://openreview.net/forum?id=abcd1234) - [for resubmissions] A PDF describing how you changed your paper in response to the reviews
Submissions are made in OpenReview: submission points for 2023. In case of system instability it is recommended to submit at least one hour before the deadline. All authors will receive a confirmation email upon submission.
You may edit all aspects of your submission up until the deadline.
All authors must be registered in OpenReview. Your OpenReview profile must be up to date (including affiliations and a recent import of any DBLP-listed publications) to aid in the detection of conflicts of interest.
All submitted materials must be anonymized. Submissions and any supplementary materials must not include authors’ names and affiliations, or the acknowledgments section. Self-references that reveal the authors’ identities must be avoided. The submissions should not have links to non-anonymized repositories: the code should be either submitted as supplementary material (zip or tgz files), or as a link to an anonymized repository (e.g., Anonymous GitHub), though take note of their limitations). Please avoid any links to storage services like Dropbox (which may track the reviewers downloading the resources).
Before submitting, ensure that: - Your paper is not already under review, published, or on track to be published at any archival venue. (“Under review” includes papers for which prior ARR reviews have been committed to a venue, and a decision is pending—see Step 5.) - Your paper satisfies everything in the author checklist - Check the reviewing workload requirement. See this blog post for further information about this new (as of April 2024) requirement.
At the time of submission you may opt to have ARR publish your paper as an anonymous preprint. These will remain anonymous during and after the review process. If you wish to remove an anonymous preprint, please contact the ARR editors (see People).
Step 2: Wait for reviews
As soon as you have submitted your paper, it will be considered under review by ARR. You may not submit the paper elsewhere during this period.
You may withdraw your submission during this period (before it receives a meta-review), but if you do so more than 48 hours after the submission deadline, your paper will be ineligible for resubmission in the next cycle. Contact the editors if you encounter special circumstances and are not sure whether withdrawal is appropriate. To withdraw, log into OpenReview, select your submission and then click on the “trash can” icon in the top right.
A submission may be desk rejected if the topic or style of contribution is not suited to ARR or if it is a dual submission or for a technical violation of the submission requirements—e.g., accidentally revealing the authors’ identities, exceeding the page limit, not completing the reproducibility checklist, not adding Limitations section, or otherwise failing to adhere to the style guidelines. It may also be desk rejected if it is a resubmission and ignores relevant feedback provided during the previous review round. Sometimes a violation is not immediately obvious, but does get noticed later in the cycle. Due to that, desk rejection can occur at any point in the review cycle, even after receiving initial reviews or even a meta-review. If a technical violation is discovered, the paper can be submitted afresh in the next review cycle (the desk-rejected version does not count as a prior submission).
Once the main reviews are available you will receive the opportunity to submit an author response. The exact timing varies by cycle and is listed on the dates and venues page. The intention of author response at ARR is different from that at conferences. At a conference, an author’s response is directly tied to an accept/reject decision, and therefore the authors often try to argue with the reviewers to accept their paper. On the other hand, ARR is only a reviewing platform and its reviewer do not have a direct say in the decision-making. Moreover, while conferences are a stateless system, in ARR, an author can always resubmit their paper in the next cycle after improving the paper and can request for the same or a different set of reviewers.
Due to these differences, the ARR author response also has different goals than a typical conference author response, and in some cases different workflows are more appropriate. Some examples follow:
- If a review has low soundness scores due to any minor misunderstandings or factual inaccuracies, author rebuttal can and should focus on this.
- If a reviewer makes significant clarity-related objections, it might be best to revise and resubmit.
- If a reviewer has strong (and, in your opinion, unfair) objections about your paper, it might be prudent to simply request a different set of reviewers in the next iteration.
One common issue is whether to report new experimental results in a response. For ARR, it is OK to present new experimental results that are in direct response to a reviewer’s question – these experiments should be minor add-ons to existing experiments, such as a new ablation, or a different hyperparameter setting, or a comparison with a different baseline. However, you are not allowed to submit unsolicited new results or fresh results that would indicate substantial additional work after the paper submission (e.g., results of new improved models).
We note that author response is a choice, and there is no compulsion to respond to the reviews. Your response must be text-only, and no external links are allowed. The paper will be desk-rejected if this is violated.
Your response will immediately be visible to the reviewers. If a reviewer responds to your response, it will be possible to have a back-and-forth conversation using the ‘Official Comment’ button. Similar to the spirit of the first author response, the goal of the back-and-forth conversation is also to clarify any further misunderstandings; repeatedly arguing your point of view to a strongly opinionated reviewer will not be helpful and is not recommended. If you believe that the reviewer is violating the reviewer guidelines, it’s best to flag this using the confidential comment, specifying exactly what guideline is violated and why you believe so.
All back-and-forth comments can be made until a final deadline, after which the area chairs will write their meta-review. The meta-review will be written taking your response and subsequent discussion (if any) into account. Your paper’s review is considered complete as soon as you receive the meta-review, and you are free to commit it to a venue that accepts ARR reviews, submit it to some other venue, or resubmit it to ARR in the next cycle.
Step 3: Receive reviews and choose next step
Once the final reviews (including the meta-review) are delivered, you will know how the paper has been evaluated. There will be at least 3 reviews and a meta-review. Each review will provide text and scores conveying the reviewer’s impressions of the paper and suggestions for improvement. The meta-review will give an overall impression from the Action Editor’s perspective and indicate whether the paper would benefit from revision or whether it might merit publication in its current form.
You have several choices for the next step of the paper:
- Quickly address reviewer feedback, and submit a revision to the next ARR review deadline (~1 week). → Step 4
- Spend more time on revisions and submit to a subsequent ARR deadline (2+ months). → Step 4
- Commit the reviews to an ARR venue for a decision on acceptance. → Step 5
- Submit for direct (i.e. non-ARR) review by a venue. If you are submitting to a venue that has both direct submissions and ARR submissions, see their policies for how to go from ARR reviewing to being directly reviewed. Note that venues may have other restrictions: for example, TACL requires a 9 month gap between submission to ARR and submission to TACL.
These options are mutually exclusive: you cannot submit a revision and in parallel commit to a venue, for example. See the CFP for all of the rules on dual submissions.
Considerations: - Reviewer feedback and scores do not directly indicate whether the paper will be accepted if committed to a venue. In general, it is expected that reviews that are very positive on the whole will most likely lead to acceptance, and reviews that are negative on the whole will most likely lead to rejection. There are no guarantees, however: the program committee members making the acceptance/rejection decision will consider the totality of the reviews while drawing on their own expertise and taking into account the entire pool of papers committed to the venue. - Different venues (e.g., main conferences vs. workshops) that consider ARR-reviewed papers may have different qualitative thresholds for acceptance.
Step 4: Resubmit for another round of review (optional)
If you can see ways to markedly improve your paper or are unhappy with your reviews, consider revising it and resubmitting to a subsequent ARR review cycle.
Here you submit the same information as Step 1, as well as: - The link to your previous submission (e.g., https://openreview.net/forum?id=abcd1234) - An explanation of revisions that responds to reviews from the previous round - A preference regarding reviewers: same or all new - A preference regarding the meta-reviewer: same or new
You can resubmit a paper as many times as you like (though expect diminishing returns after the first couple of rounds). The submission requirements are the same as first-round review (4-page limit for short papers, 8 pages for long papers).
Some things to consider: - If you felt the previous-round reviews were constructive, consider asking for the same reviewers. If you satisfactorily addressed their concerns in the revision, they will likely improve their evaluation of the paper. - ARR will attempt to reassign the same reviewers if requested. Because of availability constraints, it is still possible that you will receive some new reviewers. - Repeat reviewers (as well as those substituting for the previous reviewers if unavailable) are instructed to focus on whether the revisions adequately address the weaknesses previously identified, and are discouraged from raising new issues unless the revisions resulted in contradictory findings or questionable results not present in the previous round of reviews. - If you felt the previous-round reviews reflected a lack of expertise or engagement with the topic of your paper, consider asking for new reviewers. - Generally we accept such requests, but the expectation is that papers have been revised based on the earlier reviews and it is up to the discretion of the action editors or editors in chief whether the request is granted. This request is not considered a mark against the paper. (We understand that a certain amount of subjectivity is present in reviewing and want to give papers a fair shot at brand new reviews.) - New reviewers will not see the prior reviews initially. Only after they submit their own review will they have access to the prior reviews (new in August 2023). Note that new reviewers are instructed to form their own view of the paper rather than to focus primarily on the revisions. - The explanation of revisions should summarize the changes made and include a point-by-point response indicating how you have addressed each weakness and suggestion listed by each reviewer. - A revision does not have to accept every suggestion of every reviewer. Reviewers understand that there are limitations on what can be accomplished in a single paper. We recommend engaging with reviewers’ constructive feedback in the explanation of revisions. Even if you request new reviewers, they will have access to reviews from previous rounds after submitting their own reviews, so ignoring good advice is not recommended. - A paper that has been previously submitted but fails to acknowledge the previous version will be desk rejected.
Once you have submitted the revision, you return to Step 2 above.
Step 5: Commitment
Once the paper has received complete reviews (including a meta-review), and is no longer under review by ARR, it is eligible for submission to a venue for a decision on acceptance. This submission is called commitment.
Commitment occurs through a submission form set up by the PCs of the venue (e.g., in Softconf, OpenReview, or a Microsoft / Google Form). The form asks for information about the paper, including the ARR URL for the paper (e.g., https://openreview.net/forum?id=abcd1234 where “abcd1234” is a random string unique to your paper). After the commitment deadline, ARR provides the reviews to the venue and they make a decision. The venue will contact authors directly to notify you of accept/reject decisions.
Different venues have different commitment deadlines, and therefore, different commitment periods when the paper is under consideration by a venue. ARR does not place any restrictions on committing to multiple venues simultaneously, or committing to one venue and submitting for direct review at another venue. However, venues typically do not permit these types of dual commitments / submissions.
The commitment phase ends when the paper is no longer under consideration by any venue—either due to withdrawal or due to receiving a decision of acceptance or rejection.
If the paper is accepted, it is no longer eligible for further rounds of review at ARR. (Whether you can commit to additional venues is up to the policies of those venues.) → Step 6
If the commitment phase ends without an acceptance, the state of the paper returns to Step 3. That is, it can be committed to another venue, resubmitted for review at ARR, or submitted elsewhere for review.
Step 6: Camera-ready
Once the paper is accepted by a venue, the final step is to prepare and submit the camera-ready version. ARR is not involved in that process. You should consult the instructions from the venue for what is involved.
Questions about the preparation and submission of camera-ready papers should be directed to the Program Chairs of the venue you have committed to.
Questions about when the proceedings will appear on the ACL Anthology should be directed to the venue’s Publication Chairs.
Frequently Asked Questions
Q: Why does ARR not provide a decision with reviews?
A: ARR implements the review stage but not the final acceptance recommendation/decision stage of traditional conference reviewing. Separating the two allows us to return the full set of reviews to authors faster and give authors more choice of what to do next with the paper (Step 3), including the option to revise and resubmit.
For EACL 2024, ACL 2024, and NAACL 2024, note that deadlines are structured such that reviews after the submission deadline cycle can be be immediately committed for consideration by the conference—in which case the total time from submission to decision will be ~3 months, which is similar to the traditional conference review process (if not shorter).
Q: Since reviews follow the paper when it is resubmitted, isn’t there a risk that getting unlucky with reviewers in the first round will doom the paper?
A: For a resubmission (Step 4), authors have the option to request new reviewers, ensuring the paper gets a fresh look. While the old reviews will still be associated with the paper, to avoid anchoring biases, a new reviewer will only have access to them after submitting their own review. Moreover, authors have the opportunity to respond to any problems that they perceive with the previous-round reviews.
Q: Does ARR import reviews from other publication venues?
A: No, ARR is a self-contained review system and all reviews are done within ARR. We do not import reviews that were done by other venues.
Q: I have further questions about ARR submission. Who should I contact?
A: Please contact: support at aclrollingreview.org
Anonymity
Q: What are the requirements of the anonymity policy for authors?
A: ARR follows the ACL Policies for Review and Citation. Submissions must be anonymized as detailed in Step 1.
Beginning with the February 15, 2024 ARR deadlines, there is no anonymity period or limitation on posting or discussing non-anonymous preprints while the work is under peer review.
Contributing to the Paper and Review Dataset
Q: The submission form has a question asking if I agree for the anonymized metadata associated with my submission to be included in a publicly available dataset. What is this?
A: The ACL Exec has approved the creation of an opt-in corpus of submissions and reviews. This corpus is being assembled by a group separate from the ARR editorial team. The review meta-data associated with your submission, i.e. scores, anonymous reviewer identifiers etc., will only be included if both you and the reviewers agree. Submission and review texts are not included in this consent and are handled separately (currently, this data is not processed or collected).
Q: How can I donate my paper draft with associated review reports?
A: At ARR you can donate your peer review reports to an open public dataset of peer reviews and paper drafts (see our blogpost on the data collection for all details). As an author, you will be contacted by email including detailed instructions after your paper is accepted at a *ACL conference. In the forum of your accepted paper, you can find a new button “License Agreement”; after clicking this button, you decide which data to contribute and sign the license agreement digitally. Only one of the authors has to do this on behalf of all co-authors.
Q: Is this the same as the EMNLP 2023 review release?
A: No, the EMNLP 2023 data release includes all accepted papers and opt-in rejected papers. The ARR data collection is entirely voluntary for all papers and requires consent from both authors and reviewers. Note, ARR papers committed to EMNLP 2023 will not be released as part of the EMNLP data release.