Review Form

Paper Summary

Describe what this paper is about. This should help action editors and area chairs to understand the topic of the work and highlight any possible misunderstandings.

Summary of Strengths

What are the major reasons to publish this paper at a selective *ACL venue? These could include novel and useful methodology, insightful empirical results or theoretical analysis, clear organization of related literature, or any other reason why interested readers of *ACL papers may find the paper useful.

Summary of Weaknesses

What are the concerns that you have about the paper that would cause you to recommend against acceptance at *ACL venues? These could include concerns about correctness of the results or argumentation, limited perceived impact of the methods or findings (note that impact can be significant both in broad or in narrow sub-fields), lack of clarity in exposition, or any other reason why interested readers of *ACL papers may gain less from this paper than they would from other papers under consideration. Where possible, please number your concerns so authors may respond to them individually.

Comments/Suggestions/Typos

If you have any comments to the authors about how they may improve their paper, other than addressing the concerns above, please list them here.

Overall Assessment

What is your overall assessment of this paper? Please consider the quality of the work itself and not breadth of audience.

  • 5 = Top-Notch: This paper has great merit, and easily warrants acceptance in a *ACL top-tier venue
  • 4.5
  • 4 = Strong: This paper is of significant interest (for broad or narrow sub-communities), and warrants acceptance in a top-tier *ACL venue if space allows.
  • 3.5
  • 3 = Good: This paper is of interest to the *ACL audience and could be published, but might not be appropriate for a top-tier publication venue. It would likely be a strong paper in a suitable workshop.
  • 2.5
  • 2 = Borderline: This paper has some merit, but also significant flaws. It does not warrant publication at top-tier venues, but might still be a good pick for workshops.
  • 1.5
  • 1 = Poor: This paper has significant flaws, and I would argue against publishing it at any *ACL venue.

Reviewer Confidence

  • 5 = Positive that my evaluation is correct. I read the paper very carefully and am familiar with related work.
  • 4 = Quite sure. I tried to check the important points carefully. It’s unlikely, though conceivable, that I missed something that should affect my ratings.
  • 3 = Pretty sure, but there’s a chance I missed something. Although I have a good feel for this area in general, I did not carefully check the paper’s details, e.g., the math or experimental design.
  • 2 = Willing to defend my evaluation, but it is fairly likely that I missed some details, didn’t understand some central points, or can’t be sure about the novelty of the work.
  • 1 = Not my area, or paper is very hard to understand. My evaluation is just an educated guess.

Best Paper

Could this be a best paper in a top-tier *ACL venue?

  • Yes
  • Maybe
  • No

If the answer is Yes or Maybe, please justify your decision:

Further Questions

Replicability

Will members of the ACL community be able to reproduce or verify the results in this paper?

  • 5 = They could easily reproduce the results.
  • 4 = They could mostly reproduce the results, but there may be some variation because of sample variance or minor variations in their interpretation of the protocol or method.
  • 3 = They could reproduce the results with some difficulty. The settings of parameters are underspecified or subjectively determined, and/or the training/evaluation data are not widely available.
  • 2 = They would be hard pressed to reproduce the results: The contribution depends on data that are simply not available outside the author’s institution or consortium and/or not enough details are provided.
  • 1 = They would not be able to reproduce the results here no matter how hard they tried.

Datasets

If the authors state (in anonymous fashion) that datasets will be released, how valuable will they be to others?

  • 5 = Enabling: The newly released datasets should affect other people’s choice of research or development projects to undertake.
  • 4 = Useful: I would recommend the new datasets to other researchers or developers for their ongoing work.
  • 3 = Potentially useful: Someone might find the new datasets useful for their work.
  • 2 = Documentary: The new datasets will be useful to study or replicate the reported research, although for other purposes they may have limited interest or limited usability. (Still a positive rating)
  • 1 = No usable datasets submitted.

Software

If the authors state (in anonymous fashion) that their software will be available, how valuable will it be to others?

  • 5 = Enabling: The newly released software should affect other people’s choice of research or development projects to undertake.
  • 4 = Useful: I would recommend the new software to other researchers or developers for their ongoing work.
  • 3 = Potentially useful: Someone might find the new software useful for their work.
  • 2 = Documentary: The new software will be useful to study or replicate the reported research, although for other purposes it may have limited interest or limited usability. (Still a positive rating)
  • 1 = No usable software released.

Knowledge of/educated guess at author identity

  • 5 = From a violation of the anonymity-window or other double-blind-submission rules, I know/can guess at least one author’s name.
  • 4 = From an allowed pre-existing preprint or workshop paper, I know/can guess at least one author’s name.
  • 3 = From the contents of the submission itself, I know/can guess at least one author’s name.
  • 2 = From social media/a talk/other informal communication, I know/can guess at least one author’s name.
  • 1 = I do not have even an educated guess about author identity.

(Optional) Conjecture as to author identity. Authors will not see your response.

Ethical Concerns

Independent of your judgement of the quality of the work, please review the ACL code of ethics and list any ethical concerns related to this paper.

Any entry means that the paper will be referred to the Ethics Committee for further ethics review.