| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| reviewer_instructions [2011/01/22 22:47] – nickroy | reviewer_instructions [2011/03/15 22:38] (current) – [Detailed Review Instructions] nickroy |
|---|
| [[:home|Home]] | [[:home|Home]] |
| |
| <fs 2em>RSS 2011 Reviewer Instructions</fs> | #####RSS 2011 Reviewer Instructions##### |
| |
| ----- | ----- |
| * Jan 30 Reviews are assigned\\ | * Jan 30 Reviews are assigned\\ |
| * Feb 28 Reviews due \\ | * Feb 28 Reviews due \\ |
| * Mar 7 Author rebuttal begins\\ | * Mar 7 Author response begins\\ |
| * Mar 14 Author rebuttal ends, reviewers answer to rebuttals, discussion begins \\ | * Mar 14 Author response ends, discussion begins, reviewers revise initial reviews \\ |
| * Mar 28 Discussion period ends\\ | * Mar 28 Discussion period ends\\ |
| * Apr 7 Area chair meeting \\ | * Apr 7 Area chair meeting \\ |
| * Apr 15 Acceptance announced\\ | * Apr 15 Acceptance announced\\ |
| * Jun 27 RSS in Los Angeles | * Jun 27 RSS in Los Angeles |
| | |
| | Please be guided by __**this**__ schedule, not the deadlines listed by CMT. All CMT deadlines are listed as 12/25/25. The CMT deadline is the time at which CMT turns off access to parts of the system. In the course of the discussion and the acceptance decisions, the area chairs may ask your (the reviewer) to clarify a point in your review. This cannot happen after the CMT deadline. To allow revisions to rebuttals or the discussion, we put the CMT deadlines to a point substantially in the future after the acceptances are announced. |
| |
| ----- | ----- |
| ===== CMT Instructions ===== | ===== CMT Instructions ===== |
| |
| * First go to https://cmt.research.microsoft.com/RSS2011/ and login. You should be presented with a webpage that looks very much like this image (click image to enlarge) : | * First go to https://cmt.research.microsoft.com/RSS2011/ and login. |
| | * If you have never logged in before, you will need to generate a password. Enter your email, click the "Reset your password" button and follow the instructions. (You should have received an email from CMT sometime in the last three months thanking you for reviewing for us, and inviting you to log in. The email address that received this email is the email address you need to use.) |
| | * Once logged in, you should be presented with a webpage that looks very much like this image (click image to enlarge) : |
| |
| {{:reviewer_desktop.png?200x49&direct}}. | {{gallery>:public_images:reviewer_desktop.png?200x49&lightbox}} |
| |
| * Click on "Download Submission Files for All Assigned Papers" to get a zip file of all your assigned papers. | * Click on "Download Submission Files for All Assigned Papers" to get a zip file of all your assigned papers. |
| * Click on the appropriate paper, and you should be presented with a webpage that looks very much like this image (click image to enlarge): | * Click on the appropriate paper, and you should be presented with a webpage that looks very much like this image (click image to enlarge): |
| |
| {{:reviewer-panel.png?200x50&direct}}. | {{gallery>:public_images:reviewer-panel.png?200x50&lightbox}} |
| |
| * The ranking scores will not be available during Feedback, only the written comments. The red text <html><span style="color:red;">Visible To Authors Only After Decision Notification, Visible To Authors During Feedback</span></html> is contradictory. It should say "Visible To Authors Both During Feedback and After Decision Notification". (We were unable to get this corrected by CMT.) | * The ranking scores will not be available during Feedback, only the written comments. The red text <html><span style="color:red;">Visible To Authors Only After Decision Notification, Visible To Authors During Feedback</span></html> is contradictory. It should say "Visible To Authors Both During Feedback and After Decision Notification". (We were unable to get this corrected by CMT.) |
| - Novelty. | - Novelty. |
| |
| For each criterion you have to assign an a rating. Below we provide a detailed explanation of the different criteria and ratings. While all criteria are important, we want to encourage technically strong and rigorously evaluated papers, and therefore would like you to pay special attention to the first two criterion. However, unlike a journal, we also wish to encourage papers that are speculative, i.e., preliminary experimental work or theoretical results. | For each criterion you have to assign an a rating. Below we provide a detailed explanation of the different criteria and ratings. While all criteria are important, we want to encourage technically strong and rigorously evaluated papers, and therefore would like you to pay special attention to the first two criterion. However, unlike a journal, a conference needs to be a venue for papers that are contain preliminary experimental work or preliminary theoretical results. In any case, your review should be constructive (that is: aimed to improve the paper) and thorough. For each of the criteria, you should provide detailed comments justifying the evaluation along with suggestions for improving the paper. Furthermore, please provide specific information on which issues you would like the authors to address in their rebuttal. |
| Your review should be constructive and thorough. For each of the criteria, you should provide detailed comments justifying the evaluation along with suggestions for improving the paper. Furthermore, please provide specific information on which issues you would like the authors to address in their rebuttal. | |
| |
| Comments to avoid: | Comments to avoid: |
| * Less helpful: "This work is a reinvention of XX" without a citation. More helpful: "The authors unfortunately appear to be unaware of the work of XX presented in YY which largely overlaps with the results in this paper." | * **Less helpful: "This work is a reinvention of XX" without a citation.** More helpful: "The authors unfortunately appear to be unaware of the work of XX presented in YY which largely overlaps with the results in this paper." |
| * Less helpful: "This work contains no new theoretical results." While robotics is largely the marriage of good theory and good experimentation, we welcome papers that provide insight into robotics through rigorous experimentation. If you are unconvinced that the experiments support the claim, then a more helpful comment would be: "This is an experimental paper, but unfortunately the experiments do not give sufficient confidence in the insight the authors describe. We would need to see the following experiments to be convinced of the authors' contribution." Or perhaps if the experimental claim is uninteresting, a more helpful comment might be: "The authors provide experiments to support the claim of XX, but this claim is somewhat unsurprising given the work in YY and ZZ. A more interesting claim might be QQ, but the experiments do not support that result." | * Less helpful: "This work contains no new theoretical results." While robotics is to a great extent the marriage of good theory and good experimentation, we welcome papers that provide insight into robotics through rigorous experimentation. If you are unconvinced that the experiments support the claim, then a more helpful comment would be: "This is an experimental paper, but unfortunately the experiments do not give sufficient confidence in the authors' central claim. We would need to see the following experiments to be convinced of the authors' contribution." Or perhaps if the experimental claim is uninteresting, a more helpful comment might be: "The authors provide experiments to support the claim of XX, but this claim is somewhat unsurprising given the work in YY and ZZ. A more interesting claim might be QQ, but the experiments do not support that result." |
| * Less helpful: "This work contains little or no experimental results." While robotics is largely the marriage of good theory and good experimentation, we welcome papers that provide insight into robotics through novel theoretical or algorithmic contributions. If you are unconvinced that the theory is correct, then a more helpful comment would be: "This is a theoretical paper, but unfortunately the authors make the following assumptions XX, YY and ZZ that undermines confidence that this theory can be applied usefully. We would need to see either experimental evidence, or a relaxation of those assumptions to be convinced of the authors' contribution." Or perhaps if the theoretical claim is uninteresting, a more helpful comment might be: "The authors provide a proof to support the claim of XX, but this claim is somewhat unsurprising given the work in YY and ZZ. A more interesting claim might be QQ, but the proofs do not support that result." | * **Less helpful: "This work contains little or no experimental results."** While robotics is to a great extent the marriage of good experimentation and good theory, we welcome papers that provide insight into robotics through novel theoretical or algorithmic contributions. If you are unconvinced that the theory is correct, then a more helpful comment would be: "This is a theoretical paper, but unfortunately the authors make the following assumptions XX, YY and ZZ that undermine confidence that this theory can be applied usefully. We would need to see either experimental evidence, or a relaxation of those assumptions to be convinced of the authors' contribution." Or perhaps if the theoretical claim is uninteresting, a more helpful comment might be: "The authors provide a proof to support the claim of XX, but this claim is somewhat unsurprising given the work in YY and ZZ. A more interesting claim might be QQ, but the proofs do not support that result." |
| * Less helpful: "This is just a systems paper." While robotics is largely the marriage of good theory and good experimentation, we welcome papers that provide insight into robotics through system development. This probably means that it is an experimental paper, but you are unconvinced by the experimentation, or even the claim of the paper. A more helpful comment would be: "This is an experimental paper, but unfortunately the experiments do not give sufficient confidence in the insight the authors describe. We would need to see the following experiments to be convinced of the authors' contribution." Or perhaps if the experimental claim is itself unclear or uninteresting, a more helpful comment might be: "The authors provide experiments but do not give a clear hypothesis being tested by XX. The results are somewhat unsurprising given the work in YY and ZZ. A more interesting claim might be QQ, but the experiments do not support that result." | * **Less helpful: "This is just a systems paper."** Again, while robotics is to a great extent the marriage of good theory and good experimentation, we welcome papers that provide insight into robotics through system development. This probably means that it is an experimental paper, but you are unconvinced by the experimentation, or even the claim of the paper. A more helpful comment would be: "This is an experimental paper, but unfortunately the experiments do not give sufficient confidence in the insight the authors describe. We would need to see the following experiments to be convinced of the authors' contribution." Or perhaps if the experimental claim is itself unclear or uninteresting, a more helpful comment might be: "The authors provide experiments but do not give a clear hypothesis being tested by XX. The results are somewhat unsurprising given the work in YY and ZZ. A more interesting claim might be QQ, but the experiments do not support that result." |
| | * **Less helpful: "This paper is basically uninteresting."** This comment is not helpful to the author. Even if you, as a reviewer, are tired of reading papers in a particular area, this comment gives no guidance to the authors. Please avoid comments of this form. It's possible to say that a paper is narrow in the problem it addresses, without actually calling it boring. Describing a problem as narrow encourages the authors to broaden the scope of the paper. Describing a paper as boring encourages the authors to set the building on fire out of frustration. |
| |
| Guidance on the meaning of the criteria and their numerical scores (Thanks to Dieter Fox). | Guidance on the meaning of the criteria and their numerical scores (Thanks to Dieter Fox). |