traditional peer review and its defenses

¶ 1 Leave a comment on paragraph 1 0 David Shatz notes in the introduction to his 2004 volume on peer review that his text is not only “the first book-length study of peer review that utilizes methods and resources of contemporary philosophy,” but also “the first wide-ranging treatment of the subject by a scholar in the humanities,” a fact that becomes all the more surprising when he points out:

¶ 2 Leave a comment on paragraph 2 0 Besides its ethical aspect, the topic also has dimensions of epistemological significance, since it implicates such concepts as truth, bias, relativism, conservatism, consensus, and standards of good argument. Philosophers and other humanities scholars have produced a voluminous literature on these subjects. Yet they have not applied their approaches to these topics to peer review itself, that is, to the very procedures and practices that produced much of the voluminous literature in ethics, epistemology, and so many other fields. (Shatz 4)

¶ 3 Leave a comment on paragraph 3 0 Shatz indicates a number of reasons why this may be so, including that the more nebulous (or, rather, problematized) understanding of “truth” in the humanities precludes such scholars from being able to “show that a peer review was wrong” (Shatz 6), and that a critical study of peer review might require empirical work of a sort for which humanists are neither trained nor rewarded. Beyond these factors, however, I’d argue that a critical study of the epistemological practices of peer review requires a form of self-analysis that, as Donald Hall has argued in The Academic Self, many of us resist. Such resistance might suggest an underlying anxiety about the outcome of the analysis, a concern that the time-honored procedures and standards of the humanities might be shown to be flawed – and thus that the work that has developed through those procedures and according to those standards might be even further marginalized within the academy’s mission of knowledge-production. However, as Hall argues, genuinely “owning” our careers and the ways in which we conduct them requires taking the risk of applying our critical skills to an examination of “the textuality of our own profession, its scripts, values, biases, and behavioral norms” (Hall xiv). Too often, such examinations and proposals for change are met with stern reminders that We Have Never Done It That Way Before. The apparently intractable nature of the way things have always been done is precisely the kind of signal that, in other institutions, impels scholars to critical analysis; a resistance to turning the same critical eye on our own seemingly naturalized assumptions may create (or deepen) an atmosphere of intellectual oppression and stultification, as we allow systems in which we do not genuinely have faith to dictate our engagements with the world, and with one another. Opening up the basis of those engagements through a thorough reconsideration of peer review may be precisely what we need in order to allow our work to help shape the ways of knowing of the contemporary world.

¶ 4 Leave a comment on paragraph 4 0 Resistance to considering the merits of a more open mode of publishing often runs something like that expressed – in, I assume, an intentionally hyperbolic fashion – by Shatz:

¶ 5 Leave a comment on paragraph 5 2 It is hard to say who would have the biggest nightmare were open review implemented: readers who have to trek through enormous amounts of junk before finding articles they find rewarding; serious scholars who have to live with the depressing knowledge that flat earth theories now can be said to enjoy ‘scholarly support’; or a public that finds the medical literature flooded with voodoo and quackery. Let us not forget, either, that editors and sponsoring universities would lose power and prestige even while their workload as judges would be eliminated. (16)

¶ 6 Leave a comment on paragraph 6 3 The vehemence of such resistance often reveals something about the nervousness of those who express it, and as in much psychotherapeutic discourse, it’s only after some initial projection and displacement that the real source of that anxiety comes out: the loss of “power and prestige.”[1.9] However, in responding to those earlier displacements of anxiety, one can provide certain kinds of reassurance. The computer technologies that make open review possible also make possible the implementation of analytical tools that can help filter “rewarding” articles from any “junk” they may be mired in, whether those tools employ the results of the open review system themselves or use other modes of sophisticated textual analysis and recommendation. Further, serious scholars depressed by the apparent anything-goes nature of open publishing can see to it, by participating in the review system, that “flat earth” theories obtain the reception that they deserve. In fact, the public is already flooded with voodoo and quackery, a state easily revealed through the most cursory look at the relationship between the pharmaceutical industry and the bulk of publicly available medical information; post-publication review might actually help readers know how to interpret the material that’s out there.[1.10] But finally, if the loss of power and prestige are our primary concerns in clinging to closed review, we would be best served by admitting this to ourselves up front. If we enjoy the privileges that obtain from upholding a closed system of discourse sufficiently that we’re unwilling to subject it to critical scrutiny, we may also need to accept the fact that the mainstream of public intellectual life will continue, in the main, to ignore our work. This can, of course, be rationalized as the inevitable, unenviable fate of genius in a world of mediocrity.

See also Harnad, who presents many of the same concerns: “Every editor of a learned journal, commentary journal or not, is in a position to sample what the literature would have looked like if everything had appeared without review. Not only would a vanity press of raw manuscripts be unnavigable, but the brave souls who had nothing better to do than to sift through all that chaff and post their commentaries to guide us would be the last ones to trust for calibrating one’s finite reading time” (291). The implication, of course, is that without the power to determine whether a manuscript can be published or not, the prestige will drain out of the reviewing process, leaving scholars with only the opinions expressed by the hoi polloi.

One might in particular note Roy and Ashburn, who indicate that it was not in spite of but rather due to the peer review process that published studies of the anti-inflammatory drugs Celebra and Vioxx excluded data about those drugs’ potential for causing heart damage. One might also see the revelation on The Scientist that Elsevier published six fake journals, and that Merck paid the publisher “to produce several volumes of a publication that had the look of a peer-reviewed medical journal, but contained only reprinted or summarized articles – most of which presented data favorable to Merck products” (Grant, “Merck”).

Page 10

Comments

1 Comment on the whole Page

Maria Bustillos 7 October 2009 at 8.01 pm

“One might […] invoke the concept of prematurity here–and see the nineteenth-century observations of Goethe, Herschel, Weir Mitchell, Tourette, and Verrey as having come before their times, so that they could not be integrated into contemporary conceptions. Gunther Stent, writing about ‘prematurity’ in scientific discovery, says, ‘A discovery is premature if its implications cannot be connected by a series of simple logical steps to canonical, or generally accepted, knowledge.’ He discusses this in relation to the classical case of Mendel, and the lesser known but fascinating case of Oswald Avery (who discovered DNA in 1944–a discovery totally overlooked at the time, because of a lack of knowledge that would have enabled scientists to appreciate its importance.” (Oliver Sacks, ‘Scotoma, Neglect and Forgetting in Science,’ collected in Hidden Histories of Science, NYRB 1995.)

As I read your book, what is constantly occurring to me is that opening up scholarly work to a wider field of “peers,” if this were gone about in the right way, could do much to eliminate canonical bias and bring new ideas to light way faster and more efficiently. Really exciting to think about.

Reply to Maria Bustillos

David Parry 22 September 2009 at 2.13 pm

Ironically this is perhaps descriptive of the current state of academic writing and publishing.

Reply to David Parry
1. Nick Mirzoeff 8 February 2010 at 4.48 pm
  
  Not to mention blog and other web publishing!
  
  Reply to Nick Mirzoeff

3 Comments on paragraph 6

michaelroy 29 October 2009 at 12.09 am

Again, i would say that turning up the volume is in order. When you conclude “This can, of course, be rationalized as the inevitable, unenviable fate of genius in a world of mediocrity” , it might also be worth noting that obsolescence has its costs, namely that the maintenance funds often get redirected to the non-obsolete. I worry that you aren’t hitting the high notes about just exactly what is at stake, which is more than the future of the book, but really the future of educational institutions.

Reply to michaelroy
1. Paul F. Gehl 17 January 2010 at 8.38 am
  
  Bingo. This last sentence about rationalization is an elegant throwaway. At the very least michalroy is right that you need to emphasize how wrongheaded and irresponsible such responses are. Power and prestige can only be well defended –assuming they deserve it– if they can take the heat of criticism and come up with alternative forms of validation.
  
  Reply to Paul F. Gehl
abhardy 29 October 2012 at 9.37 am

I’d like to pick up on Gehl’s phrase “alternative forms of validation.” In reading this book and related essays on the future of publishing, I’ve been concerned about the idea of pandering to the public, and the market (even though, I realise you talk about alternate market structures later in the book). I’d like to see future iterations of this text that address more directly what alternative validation and valuation really look like beyond the closed system.

Reply to abhardy

Planned Obsolescence