Home » Blog » CHI2020 Paper Reviews and Rebuttals

Recent Updates

Past Deadlines

> August 1st, 2019

SIGCHI Student Travel Grant

> September 13th, 2019

Papers: Title, abstract, authors, subcommittee choice, and all other metadata

> September 20th, 2019

Papers: Submission files

> October 16th, 2019

Case Studies, Courses, Doctoral Consortium, Workshops/Symposia

> November 15th, 2019

Gary Marsden Student Development Fund

> December 18th, 2019

Special Interest Groups, Panels

> January 6th, 2020

Alt.CHI, Interactivity/Demos, Late-Breaking WorksStudent Game Competition, Student Research Competition

> February 15th, 2020

Gary Marsden Student Development Fund

CHI2020 Paper Reviews and Rebuttals

There were 3126 complete submissions in the Papers track (an increase of 166 or 6% on CHI 2019). Of these, three were withdrawn and 90 were desk rejected for failing to meet submission criteria for the conference: 

  • 36 for not keeping to the page limit
  • 31 because they were not correctly anonymised
  • 12 for plagiarism
  • 4 for being under submission elsewhere
  • 4 for miscellaneous reasons
  • 3 for being out of scope for the conference

A further 148 submissions were quick rejected. This means they were rejected through a quick review process. Associate chairs quick rejected submissions that they felt were sufficiently weak that they did not go through the full peer review process. These papers received a review from an AC to explain the rationale for the quick rejection.

In total 241 submissions (8%) did not go for full review (fewer than 2019’s 283). This left 2,885 submissions requiring full peer review. These submissions received 11,729 reviews or four reviews each for the vast majority of submissions. One paper has three reviews, 234 papers received five reviews, and four submissions have six reviews (those will require tricky-to-write rebuttals!).

Each paper has an ‘overall score’, which is the mean of the scores assigned by reviewers. The lowest score a reviewer can assign a submission is one, and five is the highest. Reviewers also self-report their expertise as it relates to a given submission. This is rated on a scale from one (least knowledgeable) to four (most knowledgeable). Together, these scores give you an idea of how your paper has fared.

Analytics on Scores

Of the 2,885 papers that went for full review the mean of these mean scores was 2.56 (SD=0.67), up from 2.48 at the same stage last year. At the 23.8% acceptance rate of CHI 2019 (and imagining mean scores were the only criterion), all papers with a score greater than 3.0 would be accepted (647 papers). There are 124 papers with a mean of exactly 3.0. 

There are currently 334 submissions with a mean score of 3.5 or greater (11.5%). Only 105 submissions (3.6%) have an average score of 4.0 or greater. Most submissions (2114, 73%) have a mean score of less than 3.0. Figure 1 shows the distribution of scores.

Figure 1. Distribution of mean scores for submissions that were not desk or quick rejected. This is not just a relabelled version of last year’s chart –the distribution this year is only imperceptibly different to last year’s. 

Writing a Rebuttal

You have a short window (see the timing on the submission page) to submit your rebuttal. There is no requirement that you submit a rebuttal, although reviewers and ACs are often pleased to receive, at a minimum, a brief acknowledgement of the time and effort they have spent on your work.

People often wonder whether it is “worth” submitting a rebuttal at or below a certain score. Scores can and do move. The Papers Chairs this year are encouraging authors of papers with an overall score equal to or above 2.25 to submit a rebuttal. Additionally, if the ACs for one of your submissions have encouraged you to write a rebuttal then it probably makes sense to do so. It is a judgement call. 

A number of members of our community have provided their personal views on when you should write a rebuttal and how to write an effective rebuttal. Here are a handful (most of which have appeared in this blog in previous years):

Joanna McGrenere, Andy Cockburn
Technical Programme Chairs, CHI 2020

Sandy Gould
Analytics Chair, CHI 2020