|
Publishing, Technology, and the Future of the Academy

traditional peer review and its defenses

1 Leave a comment on paragraph 1 0 David Shatz notes in the introduction to his 2004 volume on peer review that his text is not only “the first book-length study of peer review that utilizes methods and resources of contemporary philosophy,” but also “the first wide-ranging treatment of the subject by a scholar in the humanities,” a fact that becomes all the more surprising when he points out:

2 Leave a comment on paragraph 2 0 Besides its ethical aspect, the topic also has dimensions of epistemological significance, since it implicates such concepts as truth, bias, relativism, conservatism, consensus, and standards of good argument. Philosophers and other humanities scholars have produced a voluminous literature on these subjects. Yet they have not applied their approaches to these topics to peer review itself, that is, to the very procedures and practices that produced much of the voluminous literature in ethics, epistemology, and so many other fields. (Shatz 4)

3 Leave a comment on paragraph 3 0 Shatz indicates a number of reasons why this may be so, including that the more nebulous (or, rather, problematized) understanding of “truth” in the humanities precludes such scholars from being able to “show that a peer review was wrong” (Shatz 6), and that a critical study of peer review might require empirical work of a sort for which humanists are neither trained nor rewarded. Beyond these factors, however, I’d argue that a critical study of the epistemological practices of peer review requires a form of self-analysis that, as Donald Hall has argued in The Academic Self, many of us resist. Such resistance might suggest an underlying anxiety about the outcome of the analysis, a concern that the time-honored procedures and standards of the humanities might be shown to be flawed – and thus that the work that has developed through those procedures and according to those standards might be even further marginalized within the academy’s mission of knowledge-production. However, as Hall argues, genuinely “owning” our careers and the ways in which we conduct them requires taking the risk of applying our critical skills to an examination of “the textuality of our own profession, its scripts, values, biases, and behavioral norms” (Hall xiv). Too often, such examinations and proposals for change are met with stern reminders that We Have Never Done It That Way Before. The apparently intractable nature of the way things have always been done is precisely the kind of signal that, in other institutions, impels scholars to critical analysis; a resistance to turning the same critical eye on our own seemingly naturalized assumptions may create (or deepen) an atmosphere of intellectual oppression and stultification, as we allow systems in which we do not genuinely have faith to dictate our engagements with the world, and with one another. Opening up the basis of those engagements through a thorough reconsideration of peer review may be precisely what we need in order to allow our work to help shape the ways of knowing of the contemporary world.

4 Leave a comment on paragraph 4 0 Resistance to considering the merits of a more open mode of publishing often runs something like that expressed – in, I assume, an intentionally hyperbolic fashion – by Shatz:

5 Leave a comment on paragraph 5 2 It is hard to say who would have the biggest nightmare were open review implemented: readers who have to trek through enormous amounts of junk before finding articles they find rewarding; serious scholars who have to live with the depressing knowledge that flat earth theories now can be said to enjoy ‘scholarly support’; or a public that finds the medical literature flooded with voodoo and quackery. Let us not forget, either, that editors and sponsoring universities would lose power and prestige even while their workload as judges would be eliminated. (16)

6 Leave a comment on paragraph 6 3 The vehemence of such resistance often reveals something about the nervousness of those who express it, and as in much psychotherapeutic discourse, it’s only after some initial projection and displacement that the real source of that anxiety comes out: the loss of “power and prestige.”[1.9] However, in responding to those earlier displacements of anxiety, one can provide certain kinds of reassurance. The computer technologies that make open review possible also make possible the implementation of analytical tools that can help filter “rewarding” articles from any “junk” they may be mired in, whether those tools employ the results of the open review system themselves or use other modes of sophisticated textual analysis and recommendation. Further, serious scholars depressed by the apparent anything-goes nature of open publishing can see to it, by participating in the review system, that “flat earth” theories obtain the reception that they deserve. In fact, the public is already flooded with voodoo and quackery, a state easily revealed through the most cursory look at the relationship between the pharmaceutical industry and the bulk of publicly available medical information; post-publication review might actually help readers know how to interpret the material that’s out there.[1.10] But finally, if the loss of power and prestige are our primary concerns in clinging to closed review, we would be best served by admitting this to ourselves up front. If we enjoy the privileges that obtain from upholding a closed system of discourse sufficiently that we’re unwilling to subject it to critical scrutiny, we may also need to accept the fact that the mainstream of public intellectual life will continue, in the main, to ignore our work. This can, of course, be rationalized as the inevitable, unenviable fate of genius in a world of mediocrity.

  • See also Harnad, who presents many of the same concerns: “Every editor of a learned journal, commentary journal or not, is in a position to sample what the literature would have looked like if everything had appeared without review. Not only would a vanity press of raw manuscripts be unnavigable, but the brave souls who had nothing better to do than to sift through all that chaff and post their commentaries to guide us would be the last ones to trust for calibrating one’s finite reading time” (291). The implication, of course, is that without the power to determine whether a manuscript can be published or not, the prestige will drain out of the reviewing process, leaving scholars with only the opinions expressed by the hoi polloi.
  • One might in particular note Roy and Ashburn, who indicate that it was not in spite of but rather due to the peer review process that published studies of the anti-inflammatory drugs Celebra and Vioxx excluded data about those drugs’ potential for causing heart damage. One might also see the revelation on The Scientist that Elsevier published six fake journals, and that Merck paid the publisher “to produce several volumes of a publication that had the look of a peer-reviewed medical journal, but contained only reprinted or summarized articles – most of which presented data favorable to Merck products” (Grant, “Merck”).
  • Page 10

    Source: https://mcpress.media-commons.org/plannedobsolescence/one/traditional-peer-review-and-its-defenses/?replytopara=1