In this post I’ll try to convince you that all the conferences are doing it wrong. JavaOne, Devoxx, and all other conferences should stop using conference program committees to select their speakers, there is a much better way.

Call for Papers

Some of you know I enjoy speaking in public and I’ve done so at many conferences. To get selected to speak at a conference you’ll need to supply a ‘paper’ to the Call for Papers (CfP). In this paper you outline the content of your talk and provide some information about yourself as speaker.

Once all these papers are submitted some people from the community will look at all the proposals and they have the daunting task of selecting the right ones. This is very hard to do, I know this for a fact since I’ve been on several program committees, for example for J-Fall.

The honor from hell

When I got asked to be on a program committee it felt like a real honor and a lot of fun. But in reality it is a painstaking task! You’ll get a lot of papers to review, sometimes up to or over a hundred. And you have to rate them, all of them! Once you are halfway through you can’t really give an honest rating compared to the first papers you’ve reviewed, plainly because you’ve seen too much and can’t remember most of them. There is something very wrong about this system.

Bias

Another popular topic these days is gender bias. Until recently in IT a lot of these program committees were dominated by males. Research has shown that an all-male committee is much more likely to pick male speakers than female speakers for example. This is why currently most conferences try to pick a committee that is as mixed as possible. But is that really solving the problem?

Flooding

A third problem is the flooding. Some people have found out that submitting papers has a bit of chance/luck to it. If you write down the right words people will fall in love with it and you get picked. So instead of submitting one paper, they’ll just submit five or even ten papers, sometimes about the same topic but with other buzzwords. This actually improves their chances of getting in instead of getting a penalty!

Telescope Time assignment

This brings me to a Numberphile YouTube video called: Telescope Time without Tears. It turns out they had exactly the same problems we’re facing with program committees. There are only a couple of powerful telescopes in the world and a lot more researchers that want to get time on these telescopes. So previously they had selection committees and call for papers. They ran into exactly the same problems as outlined above, so they came up with a much better way to do this.

The solution: distributed peer review

I’ll try to explain what the Telescope committee did to solve their problems, but I really suggest you take a look at the video first. It explains everything much better than I possibly can.

Their idea is simple, if you submit a paper, you’ll receive a number of papers in return to review, for example six other papers. This is really what peer-reviewing is about, getting reviewed by actual peers. This instantly solves the problem of flooding, if you submit five papers, you’ll receive thirty (!) papers to review, a real penalty.

This way of reviewing is also much more fair, more people will vote and they only have to rate six papers each. So, now I know what you are thinking: this can’t be right, we can game this method! For example, if I see a paper that is a lot like my own paper, but better… I give it a very poor rating. There is a solution for that as well. You can do this of course, but only you will do this, others will still give it a fair higher rating. This information can then be used to give a penalty to your own submitted paper. On the other hand, if your scoring is much like other voters, you’ve done a fair and good job, your paper gets a small bonus. So people are motivated to really take a good look at the few papers they need to rank.

I really like the idea of getting rid of program committees, for some reason it is always feels like there is this group of in-crowd people and they have the tendency to pick friends and colleagues, the same speakers as always. I’m not blaming them, I’m sometimes part of that too! We don’t do it on purpose, it happens automatically! It is the system that is flawed.

So, for the next JavaOne, Devoxx or J-Fall, instead of selecting a committee and giving them the painstaking task of rating all the papers… maybe we should try distributed peer reviews? It just sounds like a much better approach.

Sources

The main YouTube video: Telescope Time without Tears - Numberphile
The ‘extra footage’ YouTube video with more details: Telescope Time (extra footage) - Deep Sky Videos
The actual research paper (containing the electoral theory): Telescope time without tears – a distributed approach to peer review