Reviewing Reviews: An Evaluation of Peer Reviews of Journal Article Submissions

Summary

Publication is the key means by which science is disseminated, with evaluation by journal editors and peer reviewers an important component of the scientific process. Peer reviews are, however, a typically occluded genre of documents not publicly available. Consequently, relatively little is known about peer reviews, including what makes them relevant to editors who are assessing submitted manuscripts. Here we aim to address this knowledge gap by answering the questions: (1) Does reviewer and editor (dis)agreement on the manuscript decision (i.e., accept, minor revision, major revision, reject) relate to how editors assess overall review quality? and, (2) What are the characteristics of review text that distinguish high quality reviews from lower quality reviews for editors? We analyzed 49 reviews of 26 manuscript submissions to Limnology and Oceanography: Letters. We found editor perception of review quality was based on review content rather than if there was agreement on the manuscript decision. Specifically, reviews judged by editors to be ‘highly relevant’ rather than ‘sufficient’ were typically: longer; included more comments related to study goals, analyses conducted, and resulting claims; and contained more descriptive terms related to the manuscript's importance. Reviewers who consider these factors may produce reviews most relevant to editor decisions.