{"id":3118,"date":"2019-09-16T07:13:25","date_gmt":"2019-09-16T07:13:25","guid":{"rendered":"https:\/\/www.editage.com\/insights\/my-peer-review-wishlist\/"},"modified":"2025-01-15T06:32:51","modified_gmt":"2025-01-15T06:32:51","slug":"my-peer-review-wishlist","status":"publish","type":"post","link":"https:\/\/www.editage.com\/insights\/my-peer-review-wishlist","title":{"rendered":"My peer review wishlist"},"content":{"rendered":"

It\u2019s been about a month since I\u2019ve posted here. Why? I\u2019ve been reading<\/i> instead of writing. I\u2019ve been reviewing SIGCSE<\/a> papers, ICSE<\/a> papers, IEEE Transactions on Software Engineering<\/a> papers, ACM Transactions on Computing Education<\/a> papers. I\u2019ve been editing my 7 wonderful doctoral students\u2019 submissions. I\u2019ve been guiding two students\u2019 NSF graduate research fellowships and one other undergraduate\u2019s graduate school applications. And I\u2019ve been reading a few dozen CRA Outstanding Undergraduate nominations.<\/span><\/span><\/span><\/p>\n

Now, some people don\u2019t like reviewing. I happen to love it, especially peer review! I (occasionally) learn about new ideas, I help my academic community improve its scholarship, and I get to solve a very specific kind of puzzle: what can I say in 500\u20131000 words that will transform the author\u2019s perspective on their own work? The teacher in me savors the chance to try to infer from someone\u2019s writing how they\u2019re thinking, and then find just the right words to shift their thinking toward my own research aesthetic. I don\u2019t always succeed (especially in busy months like this), but I usually have fun doing it.<\/span><\/span><\/span><\/p>\n

Unfortunately, many of the researchers in my communities don\u2019t have as much fun reviewing. And I don\u2019t blame them: we get short timelines, the volunteer work is mostly thankless (due to anonymity), and there\u2019s hardly any interaction with authors or other reviewers, aside from highly asynchronous and highly impersonal discussions on poorly maintained, poorly designed groupware like PCS or EasyChair<\/a> (no slight against the developers who maintain these: you have hard, highly under-resourced jobs, just keeping these sites running).<\/span><\/span><\/span><\/p>\n

After 15 years of reviewing, however, there are a few things that I think might make people actually enjoy it, and would make me want to do even more of it. Here\u2019s my wishlist for academic peer review.<\/span><\/span><\/span><\/p>\n

Evaluate against explicit criteria<\/b><\/span><\/span><\/span><\/h4>\n

Nothing drives me more crazy than a box that says, \u201cWrite your review\u201d here. These completely unguided peer review processes fail in two ways: they result in widely varying opinions about research, but they also fail to signal to new reviewers what aspects of research a community values or how to evaluate them. There isn\u2019t just one kind of reviewing, there are infinite, and it\u2019s an editor\u2019s job to narrow the scope.<\/span><\/span><\/span><\/p>\n

The International Conference on Software Engineering (ICSE) has made some progress on this recently, adding specific criteria such as:<\/span><\/span><\/span><\/p>\n