Recommendations for Communities of Practice

Kathleen Fitzpatrick

16 June 2012

¶ 1 Leave a comment on paragraph 1 2 Though discussions about open review typically revolve around technological innovations, one of the key observations repeatedly made by the advisory board was that human systems were primary in establishing criteria for successfully carrying out any review process, closed or open. By human systems we mean the recognition that real people need to work together toward a common purpose within established hierarchies, and with defined roles and objectives, in order to undertake an evaluative and critical engagement with scholarly work, regardless of platforms or tools. Moreover, humans — more than technologies — are essential for attracting participation in open review processes, for developing and modeling norms for participation, for teaching and practicing principles of mutual responsibility and good citizenship, and for enacting a participatory ethos, all of which are essential to a successful open review.

¶ 2 Leave a comment on paragraph 2 4 Creating a successful open review process begins with clearly establishing roles and expectations for participants, not for technologies, which must always take a backseat to human systems. Moreover, a working open review process requires labor from organizers, editors, authors and reviewers. Even enthusiasts must be mobilized to participate. Despite utopian rhetoric about collective intelligence and cultural convergence, open review processes cannot rely on virality or buzz to sustain community engagement any more than they can be successfully carried out through crowd sourcing. Instead, open review processes require dedicated leaders — likely assuming the roles of editors — who act as stewards for a project and guide potential participants to and through the open review process. Developing a clear set of expectations of what “participation” entails is essential for procuring and retaining participants, though even this is not a guarantor of participation.

¶ 3 Leave a comment on paragraph 3 3 Though the likelihood of participation is greater when participants share common interests and affinities, such commonalities do not assure a successful open review process, as pre-existing communities can sometimes reproduce a form of “groupthink” that limits how a work can be evaluated. Of course, a community made up of individuals with divergent and incongruous disciplinary and scholarly backgrounds can frustrate a review process, producing competing evaluatory practices that speak past one another. Stewards for open review processes often act as translators across communities, helping participants recognize affinities in spite of institutional or rhetorical differences.

¶ 4 Leave a comment on paragraph 4 1 From the outset, stewards must work with community members to establish mutually-agreed upon criteria for conducting open peer review. These criteria need to be both flexible and rigorous. Due to the many possible desired outcomes and modes of review available, a rigid set of rules would not only be impossible to develop without severely limiting the possibilities for open peer-to-peer review, but would actually be detrimental to the process. At the same time, rigor is essential precisely because the flexibility open review requires can quickly devolve into ad hoc, unruly, or uncritical forms of assessment and feedback. Open review is too often assumed by skeptics to produce inferior quality evaluations derived from lax and/or non-existent standards; this assumption may be alleviated by clear statements of a community’s standards and expectations for its open review practices. (It should be noted that closed peer review processes can also suffer from all of the above and are equally dependent on human agents investing in a rigorous evaluation system.) A healthy open review process is one in which peers model and thereby reinforce salient, productive norms for participation.

¶ 5 Leave a comment on paragraph 5 0 While flexibility necessitates that each open peer-to-peer review undertaking establish and follow its own rigorous evaluation criteria, there are some identifiable parameters applicable to all open review communities and contexts that should guide criteria determinations:

Desired outcomes:

¶ 6 Leave a comment on paragraph 6 3 As outlined above, peer review can serve many purposes, from gatekeeping and credentialing to promoting dialog and offering advice; from critical engagement to evaluation. It can be both macro and micro in scope, addressing conceptual, organizational, evidentiary, attributional, methodological and stylistic matters at levels that range from completed work (and, in the case of promotion and tenure, across completed works) all the way down to individual words and sentences. It is both a mechanism for authenticating products and an (often invisible) process of intellectual collaboration. Peer review is often all of these things at once, though certain goals might take precedence over others.

¶ 7 Leave a comment on paragraph 7 2 With this in mind, open review communities must be very clear about the goals and desired outcomes of any undertaking in advance of the process: How are works selected for evaluation (through open submission or vetting by editors or community, etc.)? What is being evaluated (an essay, a monograph, a blog post, a multimedia project, a tenure file, the review process itself, etc.) and for what purpose (for eventual publication in either traditional or emerging publication forms and venues; for a work’s scholarly, pedagogical, analytical, prescriptive, polemic and/or creative/innovative/experimental merit; for the purposes of brainstorming, fostering dialog within a community or between a community and a work’s author(s); for credentialing, etc.)? What aspects of a work are to be evaluated (this can range from quality of argument to ability to engage imagined audiences), at what level(s) and through what means (this can range from holistic evaluation of a completed work down to the chapter, section, paragraph, sentence and even word level and can take forms ranging from copy editing a document to embedding comments to “liking” or “disliking” various components)? Determining the means of evaluation requires open peer-to-peer review communities to reach consensus about the types and degrees of openness they wish to embrace.

Openness:

¶ 8 Leave a comment on paragraph 8 4 As previously stated, openness can take several forms. Options include the public accessibility of the review process; the non-anonymity amongst authors and reviewers; opening the process to greater back-and-forth between authors and reviewers and dialog amongst reviewers. Each determination comes with an upside and a downside. For example, making the review process public can help render scholarly processes transparent, but may also blur distinctions between peer groups, wherein individuals with varying degrees and forms of expertise become participants. Of course, the latter is only a “downside” if this is not a desired outcome of openness. Ultimately, open review communities must determine the types and degrees of openness they will pursue in relation to the desired outcomes of the peer review process. Broadly, decisions about openness encompass: the choice between anonymity, pseudonymity or transparency when it comes to reviewer and author identities; the choice to open up the review process to public viewing and/or participation; and the choice to allow and encourage reciprocity between authors and reviewers, as well as amongst reviewers.

Etiquette:

¶ 9 Leave a comment on paragraph 9 3 Extending the choices made with regard to openness, open review communities must also lay out ground rules when it comes to expectations of civility, reciprocity, and revision when it comes to providing and responding to constructive criticism. Such norms for participation are commonplace in every type of review process, blind or open, though there are sometimes concerns raised about open review’s ability to either maintain civility or offer critical feedback. In actuality, blind review also requires participants to learn and practice proper norms of engagement and, unfortunately, unhelpful, dismissive, and mean-spirited feedback can proliferate just as frequently as constructive criticism through the closed review process.

¶ 10 Leave a comment on paragraph 10 1 One of the concerns repeatedly raised about open peer review is that reviewers are less likely to offer candid assessment or harsh critique when their names are attached to their evaluations. This problem is seen as particularly acute for junior scholars wary of upsetting senior colleagues. Though some of these concerns are not easily overcome and might contribute to a community adopting an ethos of openness that embraces either anonymity or pseudonymity, a clearly established set of norms for acceptable and unacceptable modes of discourse offers a community grounds for addressing, managing, and policing uncivil discourse. It also establishes mechanisms for mediating harsh criticism, including, for example, the ability to switch between public, semi-public, and private channels depending on the severity of the critique or the ability to have editors act as intermediaries between authors and reviewers when conflicts arise.

¶ 11 Leave a comment on paragraph 11 2 Beyond concerns about civility, open review raises unique challenges for community etiquette when it comes to interactions between authors and reviewers, as well as amongst reviewers. In trying to achieve a mutuality of participation, an open review community may wish to determine: (a) whether authors and reviewers should be allowed to interact either directly or indirectly, (b) whether interaction is required or merely encouraged, (c) what response time for interactions is appropriate, (d) whether all reviewer feedback must be acknowledged, and if so, in what manner; or if not, how choices should be made about which comments must be addressed, (e) how disagreements amongst reviewers should be resolved in the revision process, and (f) when scholarly conversation has strayed too far off topic to be considered part of the review process.

Labor/Reward/Authority:

¶ 12 Leave a comment on paragraph 12 2 The work involved in peer review is often invisible, but it is not insubstantial. Editors must mobilize reviewers. Reviewers evaluate and make revision recommendations. Authors either revise according to reviewer feedback or they address their decision not to follow certain suggestions. Editors decide whether or not authors have sufficiently addressed reviewer concerns and requests before accepting or rejecting a work. In particular, reviewer labor can be most intensive but it is also the least visible. While editors often make final decisions about publication, they often rely on the feedback provided by reviewers.

¶ 13 Leave a comment on paragraph 13 3 Reviewing the work of colleagues, whether formally (for a journal or press submission) or informally (as part of a writing group) is considered part of the academic gift economy, a service performed with minimal reward, save possibly an anonymous thank you when a work is eventually published (assuming the reviewer recommends “accept”) or a request from an editor to review another work as acknowledgement that previous efforts are indeed appreciated. Scholars review one another’s work for a variety of reasons, ranging from the desire to be good citizens to the wish to stay on top of (and possibly delimit) new scholarship emerging within their fields. While anonymity may permit reviewers the ability to occasionally offer necessary harsh critical feedback without fear of repercussion, it also allows some reviewers to offer unconstructive and uncivil responses that neither help authors to improve their work nor forward scholarly discourse.

¶ 14 Leave a comment on paragraph 14 2 In an open review, the work done–and not done–by reviewers is visible (even if the reviewers remain anonymous). More than this, the work of review may also become the subject of review, as peers can be called upon by editors and authors to affirm or dismiss particular recommendations, deliberate on contradictory revision requests, or socialize participants into community etiquette. At the same time, the visibility of review can become the means by which reviewers attain peer status, garnering recognition for both the substance of their contributions and their investment of time, energy and thought into the process. Finally, the visible contributions that reviewers can make to both a work-in-progress and to scholarly discourse can potentially muddy the waters when it comes to attribution and ownership of ideas. Thus, open review communities might decide in advance how reviewer contributions are to be “counted” and evaluated. The former can range from having each comment posted constitute its own unique citation (which the technical work of the Open Annotation Collaboration is designed to enable) to a granting of co-authorship in the overall work. If these options seem extreme, open review communities should still have policies in place that account for the potential value of ideas shared within a field that rewards intellectual capital. Other options include the use of creative commons licenses that allow reviewers and authors to negotiate terms for incorporating reviewer suggestions into a revised work or a collectively accepted statement about participation in an open gift economy. Alternately, recognition of reviewers can simply take a similar form to traditional print publications, through footnotes or acknowledgement pages. Again, the forms recognition ultimately take are less essential than the community’s clear understanding and agreement on a standard for recognition prior to commencement of the review.

¶ 15 Leave a comment on paragraph 15 3 Evaluation of reviewers can take many forms, ranging from simple “like/dislike” buttons that upgrade or downgrade the status of particular posts (which, in turn, might also delineate peer membership) to mid-tier systems that allow members to mark a post’s recommendations as “required/recommended/provocative/to be disregarded” etc. to the use of more substantive meta-commenting tools that allow communities to assess reviewer posts in much the same manner that they assess the original work under review. Whatever option is decided upon, open review communities must have criteria in place for evaluating both a work under review and reviewers participating in the process. While criteria for evaluating reviews/reviewers will vary according to communities and projects, broad areas for consideration include: tone, contribution to improving/evaluating work under review (at either the macro conceptual level or the micro organizational/grammatical level), contribution to furthering scholarly discourse, and good citizenship, which might include questions like: how active is a reviewer? How engaged? How willing to share resources? etc. Finally, open review communities may also wish to establish where authority ultimately resides in making final decisions about reviewer recommendations and status. Is the process truly democratic, wherein consensus determines a review’s value or a reviewer’s status as “peer,” or do editors (or a pre-selected subgroup of peer evaluators) merely take community suggestions into account when making decisions about the review process?

¶ 16 Leave a comment on paragraph 16 5 When the review process is opened, labor is not only made more visible, it can also become more laborious, particularly for authors trying to respond to multiple asynchronous revision requests and seeking consensus approval on a work’s progression through the publication process. While openness can produce a broader and richer review, it can also produce a longer one, as the ever-expanding middle forestalls finality. The value of endless revision can be progressive, allowing arguments to be rethought in relation to emerging discoveries while privileging the process of intellectual engagement over its end product, but within an academic system that still uses publication as a measuring stick for tenure and promotion and amongst scholars pursuing multiple projects, finality also has its merits. Hence, open review communities need to have protocols in place for moving a work forward from pre-to-post-publication review, which include time limits on the review process, filtering mechanisms that allow authors to prioritize revision recommendations, distinctions between pre-and-post-revision assessment criteria, and easily recognizable differentiations between stages of review. Finally, open review communities must make determinations about archiving/documenting the review process in a manner that effectively demonstrates the labor involved in moving a work through various stages. The latter is particularly important when presenting openly reviewed works to skeptical colleagues, wherein assumptions about lack of rigor are often used to dismiss such works’ merit.