Vol. 24 - Here is a revised version of your review with improved clarity
ChatGPT is not my peer. It should not review my papers.
Last week, we received a review on a manuscript that was clearly, blatantly written by an LLM. This was easy to figure out because the usual ChatGPT output was quite literally pasted as is in the review.
Me? I think it’s interesting. It highlights how close we are to giving up on peer review.
Peer review is not a formal system. It has codes, but these codes are collectively agreed norms of behavior. And when we let them change, they change.
Who is we?
Reviewers, first.
Using an LLM to write a review is a sign that you want the recognition of the review without investing into the labor of the review. Sure enough, being asked to review by reputable journals is a sign of your reputation. There is a reason why I describe my reviewer activity as “over 50 journals, including Science and Nature”.
But this is the wrong way to look at reputation. Where they review does not change my appraisal of someone’s reputation (unless it’s eLife after late Oct. 2023). But as an editor, the fact that someone consistently writes good reviews does. The reputational boost you can get from thorough reviews is not tied to the journal for which you write them: it is tied to the editor seeing that your review is good.
If we start automating reviews, as reviewers, this sends the message that providing reviews is either a box to check or a line to add on the resume.
Writing automated reviews means we have given up.
Editors, second.
Before we start — the editor in this case did their job really well, by providing guidance about which comments were important, and which could be ignored.
The purpose of an editor is not to count the votes of reviewers, but to read through the reviews and come to a decision. This can (this should!) mean deciding something that is not what the reviewers wanted. I have recommended minor revisions with two reviewers suggesting a reject, or rejected a paper with several positive reviews. The text of the review matters, but it matters in context: the knowledge of the journal, of the body of work, of the dynamics of the field, and of the expertise of the reviewers.
None of this can be adequately captured by an LLM. The purpose of peer review is not to have something to write, it is to give criticism. LLMs do not have the capacity for criticism at a level that is required for a review.
And as an editor, I think an automatically generated review should lead to either a stern comment in the decision letter, or a rescinding of the review. We should not be looking for text, we should be seeking informed advice.
Accepting automated reviews means we have given up.
Authors, finally.
The responses to reviewers is a beautiful invention. It is the only part of peer review not to be fully broken. This is, in large part because it gives authors a little platform to comment on the review process itself.
I submit a manuscript for review in the hope of getting comments from my peers. If this assumption is not met, the entire social contract of peer review is gone. In practical terms, I am fully capable of uploading my writing to ChatGPT (I do not — because I love doing my job). So why would I go through the pretense of peer review if the process is ultimately outsourced to an algorithm? I do not care for the aesthetics of peer review. We are not children playing academics. We are supposed to be doing substantive, not performative work.
And when it is time to respond to the reviewers, I think it is important to respond forcefully. I will not debase myself by answering a machine, nor will I ask my students to do it.
Not combating automated reviews means we have given up.