[tldr:] ARR is expanding its data collection to further insights. Authors can now consent to donate their paper reviews and meta data during paper submission, reviewers and ACs can consent to donate their data for the cycle. Consider donating your peer-review data! You can always revisit this decision as needed.
As the number of submissions increases, our publication model is changing. However, we still do not know much about the path papers take, the decision processes of reviewers, etc. To do just that, we will start collecting data to better understand the submission cycle, gain new insights into review quality and reviewer behavior, and more. This new data will provide ample research opportunities.
With the approval of the ACL Ethics Committee, ACL RR has launched a new peer-review data collection protocol during the April submission cycle. This new protocol builds on the initiative by Gurevych, Kuznetsov and Dycke (2021) and is a community-driven effort. Read more on this: https://arr-data.aclweb.org/
Background
Frequent users of the ARR might remember our previous protocol for requesting data collection from authors and reviewers. That protocol approached authors to donate their data only after accept/reject decisions have been made. The details can be found here: https://aclrollingreview.org/datacollection. As has been documented in the publications using this data [1,2], the resulting data is skewed towards papers that received a positive outcome during the peer review process. Recognizing the need to motivate more research into the design of NLP to support peer-review, we propose to streamline the process of data collection by asking authors to consent to sharing their data during paper submission. In addition, we expand the content being collected to now request metadata donation.
What is new in this Protocol?
Authors now decide on data donation during submission and can revise this decision throughout the reviewing process. Reviewers will now also be requested to donate their metadata (i.e. scores), in addition to the content of the reviews, and have the ability to specify which content they want to donate. Meta-reviewers will be asked to sign a license agreement to donate their meta-reviews of a cycle. The collection of meta-reviews is a new addition to the protocol. For additional information on the data collection protocol, as well as the changes made in comparison to the earlier protocol, see the Protocol page.
[1] Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond Nils Dycke, Ilia Kuznetsov, Iryna Gurevych (2022). Findings of EMNLP.
[2] NLPeer: A Unified Resource for the Computational Study of Peer Review Nils Dycke, Ilia Kuznetsov, Iryna Gurevych (2023). ACL-2023.