This page contains a step-by-step guide and FAQ for authors who wish to have their work reviewed by ARR:
- Step 0: Is ARR right for your paper?
- Step 1: Submit your paper for ARR review
- Step 2: Wait for reviews
- Step 3: Receive reviews and choose next step
- Step 4: Resubmit for another round of review (optional)
- Step 5: Commitment
- Step 6: Camera-ready
- Frequently Asked Questions
We have also created a detailed video presentation covering this information: https://www.youtube.com/watch?v=DaoCLEghXyU.
Step 0: Is ARR right for your paper?
ARR provides reviews—and only reviews—for submissions. The reviews will not be specific to a conference/venue, but the standards expected by reviewers are those of a traditional direct submission as a long or short main conference paper to ACL (or the other major conferences operated by the ACL).
- Short papers (up to 4 pages) as well as long papers (up to 8 pages)
- All NLP/computational linguistics topics are within scope. See the CFP for example areas.
- Empirical papers, theoretical papers, resource papers, application papers, opinion pieces, and surveys
- Submissions must be anonymous
- Submissions must be original work
- Submissions must not be under review, published, or on track to be published in another (archival) venue
- System demonstrations should NOT be submitted to ARR. Conferences typically have a separate review process for them with different policies (e.g., different paper lengths and single-blind submissions). If you do submit a paper to ARR that describes a system, it will be reviewed as a normal main conference paper and your reviews will only be eligible for commitment to venues listed on our venues page (which does not include system demonstration tracks at conferences).
- If you intend to send your paper to a workshop or to a more specialized conference like CoNLL, *SEM, INLG, etc., check whether it is listed on the venues page. If not, there may be venue-specific review processes.
- Some venues are hybrid, meaning they will consider ARR-reviewed papers as well as direct submissions.
Step 1: Submit your paper for ARR review
There is a submission deadline every two months: see dates and venues. Deadlines are firm.
The submission form asks for:
- The PDF of your paper, adhering to the submission guidelines, including the formatting in the submission templates
- Optional: supplementary materials, including software and/or code
- A responsible research checklist
- A short sentence describing your paper
- Select tracks from the list in the CFP
- URLs for any non-anonymous preprints
- [for resubmissions] The link to your previous submission (e.g., https://openreview.net/forum?id=abcd1234)
- [for resubmissions] A PDF describing how you changed your paper in response to the reviews
Submissions are made in OpenReview: submission points for 2023. In case of system instability it is recommended to submit at least one hour before the deadline. All authors will receive a confirmation email upon submission.
You may edit all aspects of your submission up until the deadline.
All authors must be registered in OpenReview. Your OpenReview profile must be up to date (including affiliations and a recent import of any DBLP-listed publications) to aid in the detection of conflicts of interest.
All submitted materials must be anonymized. Submissions and any supplementary materials must not include authors’ names and affiliations, or the acknowledgments section. Self-references that reveal the authors’ identities must be avoided. The submissions should not have links to non-anonymized repositories: the code should be either submitted as supplementary material (zip or tgz files), or as a link to an anonymized repository (e.g., Anonymous GitHub), though take note of their limitations). Please avoid any links to storage services like Dropbox (which may track the reviewers downloading the resources).
Before submitting, ensure that:
- Your paper is not already under review, published, or on track to be published at any archival venue. (“Under review” includes papers for which prior ARR reviews have been committed to a venue, and a decision is pending—see Step 5.)
- You have not posted a non-anonymous preprint of your paper or widely advertised it in the 1 month prior to the submission deadline. (See the ACL Policies on Submission, Review and Citation and the FAQ below)
- Your paper satisfies everything in the author checklist
At the time of submission you may opt to have ARR publish your paper as an anonymous preprint. These will remain anonymous during and after the review process. If you wish to remove an anonymous preprint, please contact the ARR editors (see People).
Step 2: Wait for reviews
As soon as you have submitted your paper it is considered under review by ARR. You may not submit the paper elsewhere during this period, nor submit/update a non-anonymous preprint or widely publicize the work while it is under review.
You may withdraw your submission during this period (before it receives a meta-review), but if you do so more than 48 hours after the submission deadline, your paper will be ineligible for resubmission in the next cycle. Contact the editors if you encounter special circumstances and are not sure whether withdrawal is appropriate. To withdraw, log into OpenReview, select your submission and then click on the “trash can” icon in the top right.
A submission may be desk rejected if the topic or style of contribution is not suited to ARR, or for a technical violation of the submission requirements—e.g., accidentally revealing the authors’ identities, exceeding the page limit, or otherwise failing to adhere to the style guidelines. Desk rejection can occur at any point in the review cycle. If a technical violation is discovered, the paper can be submitted afresh in the next review cycle (the desk-rejected version does not count as a prior submission).
Once the main reviews are available you will receive the opportunity to submit an author response. The exact timing varies by cycle and is listed on the dates and venues page. The intention of this is to clarify any misunderstandings or confusions on the part of the reviewers (not to reargue the case for your paper). There is no requirement to respond. Your response will immediately be visible to the reviewers. In the October 2023 cycle, it will be possible to have a conversation with reviewers during this period (using the ‘Official Comment’ buttons). The structure for future cycles will be updated here once known.
The meta-review will be written taking your response (if any) into account.
Step 3: Receive reviews and choose next step
Once the final reviews (including the meta-review) are delivered, you will know how the paper has been evaluated. There will be at least 3 reviews and a meta-review. Each review will provide text and scores conveying the reviewer’s impressions of the paper and suggestions for improvement. The meta-review will give an overall impression from the Action Editor’s perspective and indicate whether the paper would benefit from revision or whether it might merit publication in its current form.
You have several choices for the next step of the paper:
- Quickly address reviewer feedback, and submit a revision to the next ARR review deadline (~1 week). → Step 4
- Spend more time on revisions and submit to a subsequent ARR deadline (2+ months). Note that with this option, there is a period when your paper is not under review. You are allowed to release a preprint or discuss it publicly in that period, up until a month before you submit again. → Step 4
- Commit the reviews to an ARR venue for a decision on acceptance. → Step 5
- Submit for direct (i.e. non-ARR) review by a venue. If you are submitting to a venue that has both direct submissions and ARR submissions, see their policies for how to go from ARR reviewing to being directly reviewed. Note that venues may have other restrictions: for example, TACL requires a 9 month gap between submission to ARR and submission to TACL.
These options are mutually exclusive: you cannot submit a revision and in parallel commit to a venue, for example. See the CFP for all of the rules on dual submissions.
- Reviewer feedback and scores do not directly indicate whether the paper will be accepted if committed to a venue. In general, it is expected that reviews that are very positive on the whole will most likely lead to acceptance, and reviews that are negative on the whole will most likely lead to rejection. There are no guarantees, however: the program committee members making the acceptance/rejection decision will consider the totality of the reviews while drawing on their own expertise and taking into account the entire pool of papers committed to the venue.
- Different venues (e.g., main conferences vs. workshops) that consider ARR-reviewed papers may have different qualitative thresholds for acceptance.
Step 4: Resubmit for another round of review (optional)
If you can see ways to markedly improve your paper or are unhappy with your reviews, consider revising it and resubmitting to a subsequent ARR review cycle.
Here you submit the same information as Step 1, as well as:
- The link to your previous submission (e.g., https://openreview.net/forum?id=abcd1234)
- An explanation of revisions that responds to reviews from the previous round
- A preference regarding reviewers: same or all new
- A preference regarding the meta-reviewer: same or new
You can resubmit a paper as many times as you like (though expect diminishing returns after the first couple of rounds). The submission requirements are the same as first-round review (4-page limit for short papers, 8 pages for long papers).
Some things to consider:
- If you felt the previous-round reviews were constructive, consider asking for the same reviewers. If you satisfactorily addressed their concerns in the revision, they will likely improve their evaluation of the paper.
- ARR will attempt to reassign the same reviewers if requested. Because of availability constraints, it is still possible that you will receive some new reviewers.
- If you felt the previous-round reviews reflected a lack of expertise or engagement with the topic of your paper, consider asking for new reviewers.
- Generally we accept such requests, but the expectation is that papers have been revised based on the earlier reviews and it is up to the discretion of the action editors or editors in chief whether the request is granted. This request is not considered a mark against the paper. (We understand that a certain amount of subjectivity is present in reviewing and want to give papers a fair shot at brand new reviews.)
- New reviewers will not see the prior reviews initially. Only after they submit their own review will they have access to the prior reviews (new in August 2023).
- A revision does not have to accept every suggestion of every reviewer. Reviewers understand that there are limitations on what can be accomplished in a single paper. We recommend engaging with reviewers’ constructive feedback in the explanation of revisions. Even if you request new reviewers, they will have access to reviews from previous rounds after submitting their own reviews, so ignoring good advice is not recommended.
Once you have submitted the revision, you return to Step 2 above.
Step 5: Commitment
Once the paper has received complete reviews (including a meta-review), and is no longer under review by ARR, it is eligible for submission to a venue for a decision on acceptance. This submission is called commitment.
Commitment occurs through a submission form set up by the PCs of the venue (e.g., in Softconf, OpenReview, or a Microsoft / Google Form). The form asks for information about the paper, including the ARR URL for the paper (e.g., https://openreview.net/forum?id=abcd1234 where “abcd1234” is a random string unique to your paper). After the commitment deadline, ARR provides the reviews to the venue and they make a decision. The venue will contact authors directly to notify you of accept/reject decisions.
Different venues have different commitment deadlines, and therefore, different commitment periods when the paper is under consideration by a venue. ARR does not place any restrictions on committing to multiple venues simultaneously, or committing to one venue and submitting for direct review at another venue. However, venues typically do not permit these types of dual commitments / submissions.
The commitment period is also subject to anonymity requirements: during the commitment period and for the 1 month prior to the commitment deadline, you may not submit/update a nonanonymous preprint of the paper or publicize it widely.
The commitment phase ends when the paper is no longer under consideration by any venue—either due to withdrawal or due to receiving a decision of acceptance or rejection.
If the paper is accepted, it is no longer eligible for further rounds of review at ARR. (Whether you can commit to additional venues is up to the policies of those venues.) → Step 6
If the commitment phase ends without an acceptance, the state of the paper returns to Step 3. That is, it can be committed to another venue, resubmitted for review at ARR, or submitted elsewhere for review.
Step 6: Camera-ready
Once the paper is accepted by a venue, the final step is to prepare and submit the camera-ready version. ARR is not involved in that process. You should consult the instructions from the venue for what is involved.
Questions about the preparation and submission of camera-ready papers should be directed to the Program Chairs of the venue you have committed to.
Questions about when the proceedings will appear on the ACL Anthology should be directed to the venue’s Publication Chairs.
Frequently Asked Questions
Q: Why does ARR not provide a decision with reviews?
A: ARR implements the review stage but not the final acceptance recommendation/decision stage of traditional conference reviewing. Separating the two allows us to return the full set of reviews to authors faster and give authors more choice of what to do next with the paper (Step 3), including the option to revise and resubmit.
For EACL 2024, ACL 2024, and NAACL 2024, note that deadlines are structured such that reviews after the submission deadline cycle can be be immediately committed for consideration by the conference—in which case the total time from submission to decision will be ~3 months, which is similar to the traditional conference review process (if not shorter).
Q: Since reviews follow the paper when it is resubmitted, isn’t there a risk that getting unlucky with reviewers in the first round will doom the paper?
A: For a resubmission (Step 4), authors have the option to request new reviewers, ensuring the paper gets a fresh look. While the old reviews will still be associated with the paper, to avoid anchoring biases, a new reviewer will only have access to them after submitting their own review. Moreover, authors have the opportunity to respond to any problems that they perceive with the previous-round reviews.
Q: Does ARR import reviews from other publication venues?
A: No, ARR is a self-contained review system and all reviews are done within ARR. We do not import reviews that were done by other venues.
Q: I have further questions about ARR submission. Who should I contact?
A: Please contact:
support at aclrollingreview.org
Q: What are the requirements of the anonymity policy for authors?
A: ARR follows the ACL Policies for Submission, Review and Citation. For each review cycle and commitment process there is an anonymity period (see question below for timing). During the anonymity periods, ARR submissions may not be released online in a non-anonymized form (e.g. via arXiv or on a personal website). Non-anonymous preprints that were published before the start of the anonymity period may not be updated until the meta-reviews are released for the cycle. The only exception is for the purpose of correcting names, in which case the EICs should be notified per ACL policy. The existence of non-anonymous preprints must be disclosed in the submission form.
ARR permits talks about work under review in small groups, such as colloquia and workshops, but we ask you not to advertise the preprint (or the content of such talks) on social media, blog about the work, or have it covered in the media during the anonymity period. Anonymous preprints can be posted after the start of the anonymity period, but likewise should not be advertised by their authors or their close colleagues, as that can compromise the review process. If you wish to make an anonymous preprint, ARR provides an option for you to do so at submission. Other anonymous preprint servers are also permitted.
See the ACL policy for further details on what is and is not permitted.
See Step 1 for details on anonymity requirements for the paper itself at submission time.
Q: When do these anonymity requirements apply?
A: The ACL anonymity rules apply to both the review process and the commitment process.
The anonymity period for the review period starts one month before the deadline for the cycle and ends once either meta-reviews have been released, the paper is withdrawn, or the paper is desk-rejected. For example, for the June 2023 cycle, the submission deadline was June 15th and meta-reviews were returned by August 15th, so the anonymity period started on May 15th and ended on August 15th.
For commitment, the anonymity policy is up to the venue. Typically, the anonymity period for commitment starts one month before the commitment deadline and ends once decisions are released or the paper is withdrawn. For example, for ACL 2023, the commitment deadline was March 17th and notifications were May 1st, so the anonymity period started on February 17th and ended on May 1st.
Note, for arXiv, the timings above refer to when your paper is submitted to arXiv (not the date it appears on arXiv). We understand that in some cases this means the timestamp shown on your paper may be inside the anonymity period.
Q: I submitted to ICLR, but want to withdraw my paper to submit to ARR. This withdrawal will cause my paper to become deanonymized. Is that OK?
A: If a non-anonymous version of the paper was made publicly available before the ARR anonymity period, it is OK to withdraw from ICLR and submit to ARR. However, if de-anonymization on withdrawal from ICLR results in the paper being de-anonymized for the first time during the anonymity period, the paper would violate the ARR anonymity policies.
Contributing to the Paper and Review Dataset
Q: The submission form has a question asking if I agree for the anonymized metadata associated with my submission to be included in a publicly available dataset. What is this?
A: The ACL Exec has approved the creation of an opt-in corpus of submissions and reviews. This corpus is being assembled by a group separate from the ARR editorial team. The review meta-data associated with your submission, i.e. scores, anonymous reviewer identifiers etc., will only be included if both you and the reviewers agree. Submission and review texts are not included in this consent and are handled separately (currently, this data is not processed or collected).
Q: How can I donate my paper draft with associated review reports?
A: At ARR you can donate your peer review reports to an open public dataset of peer reviews and paper drafts (see our blogpost on the data collection for all details). As an author, you will be contacted by email including detailed instructions after your paper is accepted at a *ACL conference. In the forum of your accepted paper, you can find a new button “License Agreement”; after clicking this button, you decide which data to contribute and sign the license agreement digitally. Only one of the authors has to do this on behalf of all co-authors.
Q: Is this the same as the EMNLP 2023 review release?
A: No, the EMNLP 2023 data release includes all accepted papers and opt-in rejected papers. The ARR data collection is entirely voluntary for all papers and requires consent from both authors and reviewers. Note, ARR papers committed to EMNLP 2023 will not be released as part of the EMNLP data release.