Transcript Best Practices for the Care and Feeding of a Conference Organization

IBM Research
Best Practices for the Care and Feeding of a
Program Committee, and Other Thoughts on
Conference Organization
Fred Douglis
IBM T.J. Watson Research Center
Best Practices
7/1/2016
© 2008 IBM Corporation
IBM Research
WOW … It’s WOWCS
 First reaction: what a great idea!
 Second reaction: will anyone actually submit?
– Immediate fear: spend time writing something up, only to find
out it’s been canceled
 Final reaction: duty
– I’ve chaired USENIX’98, USITS’99, and a few other
conferences
– Most went well but there were some issues
 End goal: guidebook (wiki) for program/general chairs
– https://wiki.usenix.org/bin/view/Main/Conference/CollectedWisdom
2
Best Practices
7/1/2016
© 2008 IBM Corporation
IBM Research
Wiki
 Goals
– Living document
– Allow discussion
 Two types of info eventually
– Guide for new program chairs or conference organizers
• Review management software
• Best practices
– Ideas for new ways to do things
• How well do rebuttals work? Ratings? Bidding?
3
Best Practices
7/1/2016
© 2008 IBM Corporation
IBM Research
Problem 1: Bad Reviewers
 Not everyone is a good PC member
– Some never do anything
– Some do really crummy reviews
– Some are too argumentative, dominate the rest of the PC, or
have other personality issues
 How to find who is a known bad reviewer?
– Past experience
– Word of mouth… but people tend not to know to ask
 How to expand word of mouth?
– Reviewer database?
4
Best Practices
7/1/2016
© 2008 IBM Corporation
IBM Research
Problem 2: Inexperienced Reviewers
 Always want to bring some “new blood” into a PC
– Search for repeat authors who haven’t served (need a script to
scour Google Scholar for this)
– Caveat: not everyone knows what to expect and may not be
cut out for it. Don’t take too many people you don’t know if
you can help it
 Tips for ensuring the best PC
– Set expectations early (# reviews, timing, … no surprises!)
– Have multiple deadlines: force people to miss early rather
than all at once just before the decisions
– Help with calibration (average scores of reviewers compared
to the scores of their peers on same papers)
5
Best Practices
7/1/2016
© 2008 IBM Corporation
IBM Research
Problem 3: PC Composition
 Avoid inbreeding, or the appearance of it
– Overlap from year to year
– Institutional overlap
 There are enough people who will have published at an
established conference that a chair should draw on them
rather than outside the conference community
– Best fit
– Reward participation
– Avoid problems with calibration
6
Best Practices
7/1/2016
© 2008 IBM Corporation
IBM Research
PC Bells and Whistles
 Options for running a PC
– Rebuttals (Seem to be a nice idea, but add lead time)
– Reviewer ratings (hard to calibrate; don’t want to insult)
7
Best Practices
7/1/2016
© 2008 IBM Corporation
IBM Research
Running a Conference
 Sponsorship
– Some institutions make it a pleasure to run a conference, and
some make it much harder
– Doing it alone has risks (e.g., liability): who “owns” the
conference risks?
 Setting expectations
– It’s hard to know just how many submissions a new conference
will get: better to be swamped (and add PC members) than miss
all your targets
– Publicity is amazingly critical (who knew?)
 Scheduling
– Watch for conflicts and hope the others watch too
– Consider the impact of rejected papers from one conference
being routed to another. (Feed reviews?)
8
Best Practices
7/1/2016
© 2008 IBM Corporation
IBM Research
Backup
9
Best Practices
7/1/2016
© 2008 IBM Corporation
IBM Research
Problem 4: (Self)plagiarism
 Simultaneous submission seems to be a big
problem
– Least publishable units
– True plagiarism
 We need a better mechanism for detecting
– Serendipity doesn’t cut it
10
Best Practices
7/1/2016
© 2008 IBM Corporation
IBM Research
Concrete Proposals/Challenges
 Plagiarism
– Neutral agent to collect papers and analyze for overlap
– Get multiple organizations to buy in
 Reviewer database
– Neutral agent to collect feedback, and if someone gets
multiple negative reports, somehow blacklist
 See Internet Computing columns Sept-Oct/NovDec 2007
11
Best Practices
7/1/2016
© 2008 IBM Corporation