by Kayhan Parsi, JD, PhD and Nanette Elster, JD, MPH
If you try googling the term “peer review is,” one of the top search results is “broken.” This reflects some of the frustration and even cynicism about the peer review process. Regarding the shortcomings of peer review, one is reminded of the famous quote attributed to Churchill (“democracy is the worst form of government, except for all the others.”) Yet peer review is still regarded as the holy grail of publishing—something that purportedly ensures high quality of new scholarship. In this editorial, we would like to define peer review and analyze the current state of peer review, along with some possible changes and innovations.
Peer review has been around for nearly three centuries. According to Shema, modern peer review did not take hold until the middle of the 20th century when journals like JAMA and The Lancet started adopting this practice. Before the advent of email or even the fax machine, papers would have to be sent via traditional mail. And while this process may sound painfully slow, our current process is not necessarily any faster. As one commentator has stated: “It is ironic that, in an era known for the great speed and availability of information – where we could choose to blog our results rather than submit them to journals – publishing papers seems slower and more painful than ever before.” (Although AJOB does boast a relatively speedy turnaround time).
Peer review is simply the process whereby editors send submitted manuscripts to be evaluated by experts in the field. Even in a relatively small field like bioethics, it is nearly impossible for even the most knowledgeable editors to maintain expertise of every subfield within bioethics. Enlisting the help of experts ensures that manuscripts receive a thorough review and critique. As mentioned previously, this practice did not take hold until the middle of the 20th century. Before then, editors reviewed manuscripts without the help of external experts and made decisions on their own about whether to publish new work. Now, editors rely heavily upon the assistance of external reviewers. This external review is intended to not only benefit the editor, but also the reader, the author, and the discipline as a whole. Everyone purportedly benefits because everyone understands that a published manuscript has undergone rigorous review. Given the significance of peer review, the lack of standardized guidance on how to review, who should review, and the lack of evaluation of the process is remarkable.
Different journals have different approaches to peer review. Some journals anonymize only the reviewer to the author. Others anonymize both reviewers and authors from each other (the practice at AJOB), conducting double-blinded reviews. Most journals typically conduct peer reviews pre-publication, although some have called for post-publication review as a way to ensure greater transparency. Our open peer commentaries at AJOB are, in some sense, post-publication reviews of accepted articles. The point of these kinds of reviews isn’t to help editors made decisions about whether or not to accept a piece for publication, but instead, to help clarify, interpret, and elaborate on an article already accepted for publication.
What’s the purpose of peer review? Former BMJ editor Fiona Godlee (citing Lock) has provided a couple of useful metaphors. On the one hand, we can think of peer review as a filter – it keeps the “bad stuff” out and only lets in the “good stuff.” On the other hand, we can think of peer review as a traffic cop – an effective peer review will direct new work to other journals that are more suitable. These metaphors presume that peer review is an objective process—something that Ingelfinger has critiqued (“Even the most conscientious reviewer will find it difficult to wax enthusiastic about an account that undermines his tenets or to disparage a report supporting his work”). Peer review is done by real human beings, with all of their foibles, insecurities, and conflicts. The best that peer review can do is mitigate some of these problems and ensure that the process is as fair as possible.
With increasing access to enormous amounts of information online, much of which is unvetted and of questionable quality, peer review can also serve as a way to help readers/scholars prioritize and assess the credibility of what they are reading. Some might even consider it to be a form of quality assurance. Utilizing plagiarism software coupled with content review by editors and experts alone, however, may not be sufficient to meet the important goals and purposes of peer review. The importance of peer review should be clearly articulated to reviewers, authors, and even readers. Additionally, developing an evaluative tool to assess the quality of the process should also be considered.
How can we improve the peer review process? One study concluded that authors were generally more satisfied with the process when their manuscripts were accepted (not surprisingly). Yet satisfaction with the process should not rise and fall based on whether a manuscript accepted. Rather, an effective process should provide author(s) timely feedback that is relevant and helpful, and yield quality publications that inform, educate, and/or motivate the reader. Moreover, serving as a peer reviewer ought to be seen as part of being a good academic citizen, a good peer, and at times, a good mentor. As Wendler and Miller have argued, “failure to do one’s fair share of peer reviews constitutes a form of free-riding.” Certainly, if someone has successfully published in a journal, that person has a responsibility to also serve as a reviewer. Such reciprocity prevents the kind of free-riding that Wendler and Miller critique.
Another recommendation for improving the peer review process is to provide some type of training for reviewers. In a writing and scholarship course taught by one of us (NE), students learn not only how to receive feedback but how to give feedback. By both giving and getting feedback, students are able to gain a deeper understanding of the process and how the questions, comments, and insights of a peer can strengthen and clarify their own work. Reviewers may also gain skills in providing feedback by pairing more seasoned reviewers with newcomers, wherein both can review a paper and then discuss similarities and differences in their assessment of a manuscript. This process not only serves to educate new reviewers but can also serve as a check and balance on what some might think of as an arbitrary process.
Although editors such as Godlee have recommended training for reviewers new to the peer review process, remarkably little training is given to those who are new to reviewing manuscripts. In an effort to help teach less experienced reviewers how to review, AJOB recently launched a peer-review initiative, in which senior reviewers model their approach to reviewing manuscripts for a less experienced junior faculty member or trainee. Specifically, ten pilot participants (including one of the authors—NE) have been asked to co-review one or more manuscripts with a more junior bioethics colleague and to use that occasion as an opportunity to share what they consider to be excellence in bioethics scholarship. The hope is that the program will help teach a new generation of reviewers how to review, and that AJOB will learn how to provide better resources for reviewers.
Reviewers are typically not paid for their time and effort, so ensuring that the process does not become too arduous and daunting is essential. However, integrating peer review into our standards of professionalism can help ensure integrity and encourage pride in the publications of any given profession. The American Board of Medical Specialists defines professionalism as a: “belief system in which group members (“professionals”) declare (“profess”) to each other and the public the shared competency standards and ethical values they promise to uphold in their work and what the public and individual patients can and should expect from medical professionals.” An accountable and transparent peer review system is one way to ensure that these standards of professionalism are met.