The naked reviewer: signing reviews

Every event can only be as good as the reviewers that review the submissions (or as the venue, but this is another topic). Reviewers decide on the submissions’ relevance to the field, the accuracy and rigorousness of the methods applied, whether and to which extent a submission may add value to an event, …, and, finally, whether a submission should be accepted for publication or not.

Looking back.

Everybody, I believe, was confronted with a review that raised up the thought of whether a peer-review process or the reviewer him-/herself was fair, reliable, biased or even politically motivated; picturing the reviewer as an ignorant and horrible person hedging behind the anonymity of the review process and holding the sword of Damocles that directly points at our research subject.

I strongly believe that the idea of giving authors the possibility to rate their reviews and irregularities in the process — as done at last year’s MoDELS conference —  would already increase the quality of reviews. I also believe, however, that apart from punishing bad reviewers, we need to cherish the good ones.

At this year’s EASE conference, I was consequently very much surprised and delighted to see that the organisers had prepared a best reviewer award. It was reasonably pointed out that the reviewers too often remain the forgotten ones, and that reviewers often deserve better. Behind this hedge,  we now could picture a stressed researcher who, rather than holding the sword of Damocles, is trying to make an honest review to help the authors in improving their work and — at the same time —  trying to do this apart from his/her daily work.

After the conference dinner, I had the pleasure of discussing with Steve Easterbrook whether or not a reviewer could and should leave his hedge to take credit (or blame). We joked that this might strongly increase the acceptance rates, but — to get to the point — also thought that this could decrease the chance of writing bad reviews, including: (1) ones that are not in tune with the effort the authors might have spent into their research (the usual two-sentence review), and (2) especially ones that are irrationally subjective and to some extent offensive.

Far too many reviewers seem to forget their manners when they write their reviews. Is it that difficult to reject a paper while still being nice? And far too many reviews remain unjustified leaving the authors curious about how they could possibly improve their work. Just to give you an impression about what kind of reviews I am referring to: “Maybe interesting to others, but not to me.” (this was the complete review). Looking at the other side of the review process: far too many co-reviewers do not seriously review a publication. This holds for rejections as well as for acceptances of publications with serious flaws (noticeably, while rating their own expert judgment as high).

Call it an Experiment.

Making a long story short: I really believe that leaving the anonymity behind would bring certain positive effects as every single word I, as a reviewer, would write could be directly associated with my name. Leaving the hedge, this is my personal believe, should bring following advantages. It should:

1. Increase the transparency of the process (not really new).
2. Force the reviewer to commit to a certain accuracy and to reasonably argue why a submission should or should not be accepted.
3. Ensure a certain honesty and the objectivity in the review (keeping in mind that it could also result in the opposite effect).

Besides those more general thoughts, I have (to be honest) no idea which effect non-anonymous review would have in general, because – as it is often the case in software engineering – it simply depends on the human factor and maybe on the self-perception of a reviewer. However, driven by that curiosity, I decided to start signing my reviews. It took not long to experience first effects.

The first effect was that I double and triple checked my reviews and, maybe this is only perceived as such, reviews took me longer as I especially checked whether I’ve hit the right tone. After 8 weeks, I was confronted with another effect. I got a personal reply from an author of a (rejected) paper to thank me for the suggestions I made in the review. The third effect is more a feeling, or rather an assumption than something I have evidence for. Signing my reviews might lead my co-reviewers to feel somehow uncomfortable or even influence their decision – at least, it could be irritating. Therefore, whenever I sign my reviews, I kindly ask my co-reviewers in the confidential remarks field to inform me whether they would like me not to sign my reviews and so far, I was only asked once to remove my signature: by a chair member. The argument was that the community has committed itself to a blind review process. I was not sure how to process that, but the email continued in a very sincere tone and the member stated the same assumption I had as a particular reason — due to respect towards the co-reviewers, which, of course, I fully understood.

Essentially, I have no idea whether signing reviews will increase in the end the quality of a review process and my current feeling is that it should not be enforced by the chair/editorial board for all reviewers, but rather be left to the reviewer. However, at least I found my own way to feel more comfortable with reviews and currently I am still doing this more as a small curiority-driven experiment to see what happens. I’ll keep you informed and if you read in the news about a researcher that disappeared in the week of a notification phase, please let my mother know that it was Steve Easterbrook’s fault.