|
A Study of Contexts and Practices

General Comments

6 general comments

  1. Thank you to the authors for providing a preview copy before the AAUP meetings. We would like to correct citation for our 2011 piece, which spends a fair amount of time defining the many functions of peer review: Harley D and Acord SK (2011) Peer Review in Academic Promotion and Publishing: Its Meaning, Locus, and Future. University of California, Berkeley: Center for Studies in Higher Education. Available at: http://escholarship.org/uc/item/1xv148c8  

  2. Overall comment: great work! This is an excellent document that will provide great resources for those of us doing open review, and hopefully inspire those who are not to rethink their assumptions and practices. Thanks for all of the time that went into this, and for inviting our feedback to make it stronger!

  3. First, let me congratulate the authors for creating such a resourceful document on open peer review and practicing what they preach by placing it online for public commentary. If you’ve never done this (e.g., the vast majority of academics), it’s harder than it looks. After re-reading key portions of the text and reviewing other readers’ remarks, my general comments here respond to five broad questions posed by the authors in their <a href=”http://mcpress.media-commons.org/open-review/request-for-feedback/>Request for Feedback</a>. 1) Clarity of purpose (Are our intentions for the document clear? Does it fulfill those promises?)Perhaps this is a characteristic of committee-driven documents, particularly those that seek to satisfy a wide range of members’ opinions on a given topic, but I had difficulty determining the primary purpose of this report. The top half of the <a href=”http://mcpress.media-commons.org/open-review/executive-summary/>executive summary</a> tells us that “The overall objective of these meetings was to help develop a set of community protocols and technical specifications that would help systematize open peer review and ensure that it met both academic expectations for rigor and the digital humanities’ embrace of the openness made possible by social networks and other digital platforms.” At first, it appears as if the goal of the report is to improve our model for open peer review, but that did not fit well in my mind with the bottom half of that page, which emphasized that “no single set of tools or rules can be imposed on open peer review,” which demands a decentralized “structured flexibility.”  My confusion on this particular point led me to wonder about other purposes: Is the report intended to <em>inform</em> audiences about the “merits and pitfalls” of open peer review? Or to <em>advocate</em> for its broader adoption by scholars and publishers? Or to <em>evaluate</em> claims and evidence on whether open peer review produces better-quality scholarship than does traditional practice? In my reading, the report was very informational, but not strong in advocacy or evaluation. If I didn’t already believe in open peer review, this document may have intrigued me about the concept, but probably would not have persuaded me that the merits outweigh the pitfalls. Perhaps the lack of advocacy was the intent of the authors or the result of a committee-driven document. In any case, what is clearer to me now is the need for a careful evaluation of our growing examples of open peer review, and whether or not the evidence shows that “the crowd” produces better developmental editing than traditional practices alone. 2) Organizational concerns (Have we structured the document in a coherent and logical manner? Do sections flow and does the information within them seem to be in the right place?)Yes, the organization of the report makes sense to me, but its integration with this CommentPress website could be improved. For example, the “Request for Feedback” lists 5 general questions for readers, but this would be more effective if they were prominently featured and embedded directly into the “General Comments” section. See one way we tried to do this in <a href=”http://writinghistory.trincoll.edu”>Writing History in the Digital Age</a>. 3) Nuance of argument/perspective (Are we missing key connections between open review and the humanities tradition, key human dynamics, or existing tools that might strengthen our recommendations?)Make a stronger connection between open peer review and speed toward publication. 4) Examples (Are there additional experiments in or explorations of open review that we should include in our consideration?)The text briefly mentions several examples of open peer review, but richer descriptions (or side-bars or vignettes) would clearly strengthen this report for readers who want to know more about what actually happens in practice. If advocacy is a goal of this report, then give it more consideration.  5) Applicability (Are there ways in which the ideas we’ve discussed here might affect your own work that we should consider?)If an author (or group of authors) read this document and wished to experiment with open peer review, is there any “how-to” advice here about practical next steps? Normally I would not raise this as a criteria for a report, but since “applicability” is on your list, I wonder whether this current draft fulfills it.

  4. Here’s what I hope will be a more readable version of the general comment that I hastily posted above: First, let me congratulate the authors for creating such a resourceful document on open peer review and practicing what they preach by placing it online for public commentary. If you’ve never done this (e.g., the vast majority of academics), it’s harder than it looks. After re-reading key portions of the text and reviewing other readers’ remarks, my general comments here respond to five broad questions posed by the authors in their Request for Feedback. 1) Clarity of purpose (Are our intentions for the document clear? Does it fulfill those promises?)Perhaps this is a characteristic of committee-driven documents, particularly those that seek to satisfy a wide range of members’ opinions on a given topic, but I had difficulty determining the primary purpose of this report. The top half of the executive summary tells us that “The overall objective of these meetings was to help develop a set of community protocols and technical specifications that would help systematize open peer review and ensure that it met both academic expectations for rigor and the digital humanities’ embrace of the openness made possible by social networks and other digital platforms.” At first, it appears as if the goal of the report is to improve our model for open peer review, but that did not fit well in my mind with the bottom half of that page, which emphasized that “no single set of tools or rules can be imposed on open peer review,” which demands a decentralized “structured flexibility.” My confusion on this particular point led me to wonder about other purposes: Is the report intended to inform audiences about the “merits and pitfalls” of open peer review? Or to advocate for its broader adoption by scholars and publishers? Or to evaluate claims and evidence on whether open peer review produces better-quality scholarship than does traditional practice? In my reading, the report was very informational, but not strong in advocacy or evaluation. If I didn’t already believe in open peer review, this document may have intrigued me about the concept, but probably would not have persuaded me that the merits outweigh the pitfalls. Perhaps the lack of advocacy was the intent of the authors or the result of a committee-driven document. In any case, what is clearer to me now is the need for a careful evaluation of our growing examples of open peer review, and whether or not the evidence shows that “the crowd” produces better developmental editing than traditional practices alone. 2) Organizational concerns (Have we structured the document in a coherent and logical manner? Do sections flow and does the information within them seem to be in the right place?)Yes, the organization of the report makes sense to me, but its integration with this CommentPress website could be improved. For example, the “Request for Feedback” lists 5 general questions for readers, but this would be more effective if they were prominently featured and embedded directly into the “General Comments” section. See one way we tried to do this in Writing History in the Digital Age. 3) Nuance of argument/perspective (Are we missing key connections between open review and the humanities tradition, key human dynamics, or existing tools that might strengthen our recommendations?)Make a stronger connection between open peer review and speed toward publication. 4) Examples (Are there additional experiments in or explorations of open review that we should include in our consideration?)The text briefly mentions several examples of open peer review, but richer descriptions (or side-bars or vignettes) would clearly strengthen this report for readers who want to know more about what actually happens in practice. If advocacy is a goal of this report, then give it more consideration.  5) Applicability (Are there ways in which the ideas we’ve discussed here might affect your own work that we should consider?)If an author (or group of authors) read this document and wished to experiment with open peer review, do you want any “how-to” advice or resources to appear about practical next steps? Normally I would not raise this as a criteria for a report, but since “applicability” is on your list, I wonder whether this current draft fulfills it.

  5. Great material from what I’ve seen so far. Would you consider making the draft report available for reading in more various & open forms such as a single straight HTML page, PDF for printing, ePIN, etc.? The web-based, section-by-section is quite limiting of how I can access and read the doc, particularly on mobile (currently I’m viewing on iPhone 3GS, iOS 5, Safari) on which it is very difficult to use, particularly because of the floating right column covering most of the main text area. On mobile, I usually read articles only via Instapaper, Readbility, Pocket, etc.
    thanks, Tim.

Leave a Comment

Your details




Your comment

Source: http://mcpress.media-commons.org/open-review/general-comments/