A standard clause
When reviewing papers for journals and conferences, I include the following mention:
|
This review is submitted on the condition that the authors will receive exactly the same text as the selection committee, my name and this notice included. Signed reviews are the only way I know to fix the current flawed review system in computer science. |
with a link to the present page.
When I receive a request to do a review I check with the editor that this approach is compatible with the publication's policy; if it's not I decline.
The trouble with anonymous refereeing
It is widely believed that anonymous refereeing helps fairness, by liberating reviewers from the fear that openly stated criticism might hurt their careers. In my experience, the effects of anonymity are worse than this hypothetical damage. Referees too often hide behind anonymity to turn in sloppy reviews; worse, some dismiss contributions unfairly out of personal interest or personal prejudice.
Even honest people will produce bad-quality reviews out of negligence, laziness or lack of time because they know they will not be challenged. Putting your name on an assessment forces you to do a decent job.
The situation may be different in other fields, or even in different parts of computer science. In software engineering, programming languages, programming methodology, user interfaces, development tools, project management, even algorithms, there is room for opinion. In the natural sciences referees can say "your results violate the laws of physics" or "I tried to repeat your experiment and got different results". In mathematics they can say "your proof is wrong". Often in computer science things are not as clearcut: your new development technique or process model may have achieved a productivity gain in three projects, but a referee may doubt that it will work in other cases. Even for topics such as programming language design some subjectivity is inevitable. It is part of makes these topics fun, but also imposes on us a special responsibility to be fair. Putting one's name on the assessment is the best way I know.
A common excuse is that some researchers, especially those who are at the start of their careers, may be scared away from writing bad reviews of work whose authors could have an influence on their professional advancement. This argument is naïve at best and dishonest at worst. First, if you really have a personal relationship with an author, you should either disclose it clearly -- and then write the most competent review you can -- or decline to write a review. But more importantly, the effects of anonymity are worse. If a signed review is too nice because the reviewer is scared, that will be immediately visible to the selection committee. The real unethical opportunity is for someone to kill a potential competitor's contributions under cover of anonymity. If a referee with all too human frailties happens to be applying for a position at a university, which is the most pernicious and most likely case: that he will soften his comments on a paper by one of its current professors, because the review shows his name; or that he will make nasty anonymous comments on a submission whose author is also applying?
Anonymity: an illusion?
The promise of anonymity is precarious anyway. It's so easy to give yourself out, especially if you do your job right: your comments will betray your style; in pointing out missing bibliographic references, you might -- surprise! -- include your own work, and that won't be very difficult to guess for a rejected author. Then, of course, editors make mistakes, more and more common now that all happens through email. (Who has never hit "Reply All" instead of "Reply"?) Some countries even have freedom of information laws that may force the editor to reveal the referees on request.
So if the promise of anonymity lures you into making brash comments on the work of someone you know, you might be in for a bad surprise.
The consequences of openness
In practice, making my name public in reviews has brought occasional trouble. Some authors have been mad at me. For those who complained, I think it was better that they could write to a real person and get an opportunity to hear back, rather than build up frustration at the world's anonymous unfairness. In one occasion an author complained that one of my books used a term that he said he had coined for a paper of his, rejected a year earlier by a conference, for which I had signed my review. I had actually been using that term myself frequently in earlier publications and talks; being able to answer his complaint removed any bad feelings. Such a dialog is healthier than the common case of an author who suspects plagiarism and then develops a conspiracy theory because he can't obtain official confirmation that the supposed plagiarist was responsible for rejecting his paper and stealing his ideas. Openness is the best way to dispose of such misunderstandings quickly, and let everyone move on to more productive pursuits. So yes, it takes some guts to sign your name, especially if what you are signing is negative, but it's part of being a professional.
This is the discipline I have adopted for myself; it keeps me honest and enables me to do the best job I can as a referee. I can't say I would never make an exception, but I haven't yet found an opportunity, and I have not been able to make up a hypothetical case that would require it.
Towards open refereeing
Do I think everyone else should apply the same policy? Basically yes, although I am not on a crusade; I hope the observations on this page will prompt readers to reflect on the issue and make up their own minds. In order of increasing departure from current practices, I do suggest the following changes:
At the very least, every self-respecting publication should
accept referees who insist on signing their reviews
according to the policy described here. That
some well-known organizations demand anonymity is unacceptably arrogant:
how can a publication prevent a referee from insisting
on honesty at his own risk? To me such publications have
something to hide. I consider them unethical and will not referee
for them, write for them, or advise anyone
to submit publications to them.
I think that publications should explicitly tell referees that they
have the option of signing their reviews, even if (in conformance
to current practices, and to avoid bad surprises for reviewers
who expect anonymity to be the rule) the default policy remains
anonymous refereeing.
As these issues get discussed further, I would argue that the
default should change to signed reviews. This, of course, could
only happen if the view developed here gains many supporters.
If you agree with the general recommendation of this page, or at least think it raises questions worth discussing, you may consider linking to it; this may help increase awareness of the issue of fair reviewing in our field.
Fixing the refereeing process
I hope in any case that a serious debate will take place on the ethics of refereeing in computer science. The current practice is highly unsatisfactory; the refereeing rules published by ACM, IEEE or OOPSLA are out of step. We need to take an unbiased look at the current situation and establish fair and workable refereeing standards.
In the meantime, individual decisions to sign one's reviews are a way to take a stand against bad refereeing practices in computer science.
|
|