wiki:EarlyReview

Summary email on the early reviews, from Jari Arkko

Perform some reviews that are now happening at IETF Last Call a bit earlier:

Building quality and cross-area review to the process earlier is of course a big effort. For the first item above, we are launching an experiment to make a small change to current directorate review procedures to learn if we can move reviews a little bit earlier. If successful, this experiment will enable working groups to deal with issues before IETF Last Call and IESG review and empower the working groups to be in charge of the documents throughout their life cycle. We are also hoping that document quality will improve and number of issues discussed in the IESG will be lower.

While the number of reviews as such is not changed, some additional effort and care will however be required from the reviewers, directorate coordinators/secretaries, the working group chairs, and other participants. The experiment will show us whether this effort is reasonable and if there are any unexpected effects. The experiment is performed on a voluntary basis by each directorate. As early review requests come in, the directorates can throttle workload by either processing the requests or reverting to the existing procedures. Initially, the Gen-ART, Security Directorate, and Applications Area Directorate are included in this effort. Other directorates may be added later. The experiment will be reviewed after six months.

Care must be taken to avoid a number of possible drawbacks. There may be problematic documents that would require much re-review and effort. Similarly, working groups often perform a number of working group last calls on a document, and it would be undesirable to engage the reviewer before the document was really ready to be sent forward. And when reviewers send comments, it is important that the group listens to the comments in the right way, like you would for a review from a security expert or a general networking expert. E-mail practices around sending and responding to comments have to be carefully managed, as the outside reviewers are typically not list members.

For the working group chairs, you can request a review by sending mail to dir-coord@…, indicating the name of the document. You can ask for a review as soon as a working group last call has ended successfully and the document edits are done. Initially, it would make sense to ask for reviews on documents that you feel are likely the most stable ones. To avoid congesting the reviewer resources, ask for this service only for some documents, not as a wholesale service on every document you submit for publication.

You can ask for the review in parallel with ongoing AD reviews, filling out the shepherd questionnaire, etc. The hope is that reviews can come in in the same time frame as other tasks in this stage, and that you can take the comments into account and revise the document before finally sending it out for IETF Last Call. Reviews will be sent to you and potentially the working group mailing list. You may need to forward comments and/or approve new posters to the list. Discussion on the mailing list is encouraged, but please try to ensure that the reviewer is kept on the Cc line, as he or she may not be on the list itself. Treat the reviews with respect and keep it in mind that outside experts may have opinions that need to be taken into account, even if the working group had not considered those aspects before. Mediate and monitor the discussion actively. Try to keep the reviewer out of e-mail storms and work out solutions separately and then engage the reviewer again.

For the directorate coordinators/secretaries, please monitor the dir-coord@…. If there is a request for a review, dispatch the task to a reviewer. Managing overload is your task. Please acknowledge requests and whether you can accommodate them. Directorates that today review documents twice, at IETF Last Call and then before entering the IESG telechat should continue this practice by doing the early review and then the IESG telechat review. Existing practices such as retaining the same reviewer for checking the same version is useful. Your review tools and practices may need adjustment to accommodate reviews happening at different times.

For the reviewers, take into account the context. Post your reviews to the working group chairs, authors, and Cc the working group mailing list if necessary. If your review team uses a template for these e-mails, some changes may be necessary in the template for the early review. Some of those changes have been discussed, e.g., in the Gen-ART list.

For everyone, please collect your experiences so that in six months we can evaluate how you liked the experience. And thank you for your participation in this effort!

Last modified 8 years ago Last modified on 14/01/14 09:31:57