Appendix 2: Functional requirements for an open peer review system | Open Review

Functional requirements for an open peer review system

by Peter Brantley

Basics

¶ 2 Leave a comment on paragraph 2 0 Any open peer review (OPR) system will present a set of compromises in design and functionality. Difficulties arise not only in technical capability but also in the heterogeneous nature of workflow preferences across a wide range of scholarly communities, with varying choices for participation, editorial function, and publication goals.

¶ 3 Leave a comment on paragraph 3 0 In any consideration of appropriate choices for software implementation, it is critical to distinguish between and prioritize different functions for the software platform; persona administration; and workflow management.

¶ 4 Leave a comment on paragraph 4 0 Examples of persona administrative options include support for reviewer profile attributes (anonymous or authenticated); management of reputation metrics; identity management for authors, editors, and reviewers; and the ability to manage user accounts for privileges, inappropriate behavior, and other purposes. Examples of workflow characteristics include the relative degree of openness for solicitation of reviewers; origins of review solicitation (author, editor, or open); comment and annotation moderation; revision control support; progression tracking; scheduling requirements; group-based permissions, and so forth.

¶ 5 Leave a comment on paragraph 5 0 Considerations for software platform choice include the preference for distributed installation, such that each journal or work team has its own local instance of the OPR software; hosted, such that individual projects can be spawned from a common software base at an organizational level, which might be optimal for a publisher or publishing services provider; cloud-based, in which the software is resident on the network and is capable of supporting varying size projects and work groups; or some hybrid of the above. It is worth noting that on this factor, WordPress software has an admirable structure, replicating its core functionality across a variety of modalities, including locally hosted and cloud-based.

Mechanics

¶ 6 Leave a comment on paragraph 6 0 Authoring, editing, commenting, reviewing, revising, and publishing all seem to be vital elements of what we must incorporate. Yet some of these describe what we seek to accomplish – e.g., peer-reviewed, published material – and some describe how it is we might ultimately accomplish it. Authoring, reviewing, and publishing are discrete, essential processes that help us achieve our goal, whereas editing, commenting, and revising describe the functions that enable that goal to be obtained.

¶ 7 Leave a comment on paragraph 7 0 When we consider functional requirements for a software system, we must be clear on the distinction between the processes we are trying to automate and the functions through which we achieve them. For example, in most variations of OPR, it is vital that we manage reviewers adroitly and facilitate their commenting and annotating, as well as summarizing and coalescing their output; reviewing comprises all of these. Each of the activities that comprise the how of any OPR process is characteristically small and discrete. One of our key design goals is to assemble these functions without inducing extraneous complexity through their compilation.

¶ 8 Leave a comment on paragraph 8 0 A member of the committee noted in the January 2012 meeting that the primary work focus would have to be book chapters (vs. an entire book), or discrete articles. Absorbing the management of something greater into a reasonable review process would be too easily daunting for both reviewers and editors. However, this does not yield a summary perspective of the larger work: who reviews the manuscript, when online commentary is couched only at the chapter level?

¶ 9 Leave a comment on paragraph 9 0 This nexus between our ability to attend and focus, combined with how we desire to do the work at hand, must be reflected through the lens of software engineering. Software should not ask more of us than we are capable of. A simple and imperfect solution is preferable to an attempt to articulate a more complex and full-featured software system that will be infinitely more fragile. In sum, OPR systems must be able to segregate their inevitable design shortcomings into arenas where they can be capably handled through separate human or machine interventions. Let the machine handle what it can best handle; the rest will require human intervention.

¶ 10 Leave a comment on paragraph 10 0 Being wise about where failure is permitted is a critical component of design. If it is too much to assume that OPR software systems could reasonably effect the review of both chapter and book as a whole, then the review of the greater manuscript can either take place in a wholly separate workflow, or be conducted out of band of the software itself. Inelegant, yet functional, such an approach permits us to incorporate as much robustness as possible into the OPR system itself.

¶ 11 Leave a comment on paragraph 11 0 Designing for flexibility is fundamental. It is conceivable that one could architect a superbly working system for one functional goal, e.g., a system that would provide for the submission and review of manuscripts geared toward promotion and tenure. But there will be myriad goals for OPR, and many different kinds of workgroups. The use cases enumerated range from evanescent work teams assembling only to deliver a one-off product, to mid-duration projects such as processing book manuscripts, to longer-term endeavors such as open journal publication or deposit. We are better off restricting the goals of our software rather than the purposes to which it may be put.

Requirements

¶ 12 Leave a comment on paragraph 12 0 A set of core requirements is identifiable; taken from Dan Visel’s document on software review.

1.1. Persona Management

¶ 13 Leave a comment on paragraph 13 0
The system must allow anonymity (or pseudonymity) as well as self-identification;
The system should permit the toggling of various levels of identification depending on participant role and review context;
There should be a way to examine a user’s contributions globally across the site for site owners; depending on permissions, editors should have commensurate access to user contributions;
The quality of reviews must be capable of assessment to aid the evaluation of any given reviewer;
The system should permit assessment of reviewer qualifications to aid the evaluation of any given review;
The ability to adequately moderate new content, preventing trolls and spam, is essential;
The system should make it possible to down-vote or deprecate unhelpful comments, and to up-vote and “praise” helpful ones.

1.2. Content Management

¶ 14 Leave a comment on paragraph 14 0
The system should be able to manage in-text conversations among authors, editors, and reviewers in a manner allowing for varying degrees of public/ private/ or mediated access among any combination of the parties.
The system should permit commenting at the page/ paragraph/ sentence/ or word level;
Versioning needs to be accommodated and easily accessible–multiple drafts and editions of texts and comments need to be maintained;
The system should allow readers and reviewers to compare drafts pre- and post- review, or based on versioning;
The system should support the citation or linking of individual comments;
The editing or deletion of reviewer comments should be gracefully handled;
The system should be able to categorize and filter reviews by primary type (i.e., content vs. editorial), and by subcategories, such as argument, evidence, citation, or grammar;
The system should be able to distinguish between reviewer suggestions or recommendations, and required revisions.

1.3. Editorial Basics

¶ 15 Leave a comment on paragraph 15 0
The system should have easy prompts to guide reviewers in their tasks;
The system should enable both public and private discussions;
Editors should be able to “enlist” reviewers out of an available pool;
If the platform permits, authors should be able to recommend reviewers;
Reviewers should be able to receive notifications of new tasks;
The system should be able to accommodate both closed and open-ended (evergreen) publication.

1.4. Intellectual Property

¶ 16 Leave a comment on paragraph 16 0
The system should permit the application of CC licenses to individual content items.

Observations

¶ 17 Leave a comment on paragraph 17 0 Core components of workflow management are subject to nearly infinite customization. In many scenarios, editors will have access to reviews and reviewer commentary. Depending on whether these contributions are considered to be part of the final published work, and whether the work is actually left in any kind of final state itself, editors may have the ability to edit, annotate, or censor. It is also possible to establish byzantine levels of required authorization for certain activities, such as post publication removal of comments, or the blacklisting of a specific reviewer.

¶ 18 Leave a comment on paragraph 18 0 Software is infinitely malleable. Much more thought must be placed on policy and procedure than software requirements. Rather than over-specify preferred functionality, it is far better to be quite clear about what the publishing goals are, and the policies and procedures around submissions, reviews, and publication.

¶ 19 Leave a comment on paragraph 19 0 Most aspects of publishing management have little to do with the nature of peer review. A robust publishing platform is likely to support sophisticated content management, revision control, and some aspects of task assignment. Married to a realistic understanding of the publishing venture’s goals, an appropriate platform will make peer review more straightforward and easy to manage. In contrast, building an entire publishing platform from the ground up in order to support a new open peer review system puts an enormous obligation on software engineering; it’s a bit like building a skyscraper in order to obtain a new marble bathtub.

Sample work streams

¶ 20 Leave a comment on paragraph 20 0 A few sample work streams help elucidate the range of functions for reviewers, editors, and other roles.

¶ 21 Leave a comment on paragraph 21 0 Two phase open review. In a two-phase review, an initial group of “close readers” annotates particular passages or paragraphs (or groupings of these). A second group, such as journal editors, would then write a set of reviews that synthesize and extend the observations of the close-reader group.

¶ 22 Leave a comment on paragraph 22 0 As an example of contributed workflow, the secondary review group could also evaluate the first group’s work by recording the contributed value of each close reading to the higher-level review. Tags could include terms like ‘formative of,’ ‘exemplifies,’ ‘elicits refutation,’ or other terms that characterize the influence of first-round comment on higher-level review. This process would enable commentary on first-stage reviewers: e.g., ‘User X’s comments have elicited 100 direct refutations’; ‘User Y’s observations have made significant contributions toward development of scholarship on topic Z,’ etc.

¶ 23 Leave a comment on paragraph 23 0 Three phase open review. In a three-phase review, a preliminary gating appraisal of submissions by the house editors takes place. Upon provisional acceptance, the submission is released for an open peer review process to the community of reviewers who have registered with, and been accepted by, the publisher. After the review period closes, the authors re-submit their paper to the publisher who then undertakes a final acceptance review of the work. In the large majority of cases, most papers successfully transiting the open peer review process would be accepted.

¶ 24 Leave a comment on paragraph 24 0 Scoring review. In one possible single step review scenario, journal submissions flow automatically into a pool, perhaps categorized by metadata such as subject classifications; these could be derived through machine analysis without the need for manual intervention. Reviewers can “check out” or register for any submission of their choosing, annotating and rating the document. Site thresholds determine the visibility of the paper, or alternatively, if it receives enough voting points, it is automatically approved and “published”.

¶ 25 Leave a comment on paragraph 25 0 Gating review. Some online publishing forums may have very low criteria for acceptance, e.g. incorporating rules against inappropriate narrative profanity, defamation, or other personal attacks; proper citation of literature; originality; and general awareness of the domain. In such a case, the editorial function may be limited to designating or soliciting a threshold number of reviews that evaluate these criteria without paying significant attention to the content. Disagreements would have to be mediated by editors but otherwise reviewer approval would elevate the submission on the publishing site into a publicly accessible status.

¶ 26 Leave a comment on paragraph 26 0 Commentary review. In some publishing streams, the desired outcome is to create a publication combining expert commentary with a submitted, often requisitioned, narrative. In such a case, the submitted work will need an initial review pass to evaluate rhetorical strength and structure, and to elucidate useful edits. This initial analysis could be performed by appointed “pool” reviews. In the second stage of the process, intellectual peers of the submitter would then overlay or integrate their informed commentary and critique into the structure of the work, constructing a new integrated whole. In a potential third stage, this work package would itself potentially be available to a second open review, if desired.

¶ 27 Leave a comment on paragraph 27 0 Monograph review. A monograph will need parallel and possibly multi-staged review. In one track, readers must be assigned to perform a comprehensive and holistic critique of the entire work. These reviewers must include subject matter experts capable of placing the work in its larger context; their reviews would potentially be available for secondary open comment. In parallel, manuscript chapters would have to be pulled off for individual review and comment. Authors will need to accommodate and merge both discrete and holistic comments. It is likely that this will require support for a two-pass review process, with a final curatorial decision by house staff.

Next Steps

¶ 28 Leave a comment on paragraph 28 0 Peer review is a fast moving area in scholarly publishing, and there is substantial institutional interest in software platforms that support its functions. One of the crucial challenges is that every editorial group or publisher is certain to have its own preferred workflow steps, both in the specifics and in higher-level functions as well. For a software tool to be widely utilized, it will have to support a disparate range of flexible configurations. For it to fit a single purpose, it can be narrowly defined.

¶ 29 Leave a comment on paragraph 29 0 Any new open peer review project will have to make a decision whether to support an already active effort or to synthesize its own. At the point where this decision needs to be made, engagement with groups building out software platforms such as Annotum, Open Journal Systems, and PLoS’ Ambra must be made in order to consider the potential for meeting project aims. Ideally, a new effort could join with the aims of an existing one and integrate its requirements into the parent’s engineering workplan.

¶ 30 Leave a comment on paragraph 30 0 The alternative – to build out a new software platform – should be approached with caution and trepidation. It is not trivial to design from scratch a complete, robust, scalable publishing system that supports open peer review, content management, versioning, and other features. It certainly can be built – but it would need to be constructed by a competent engineering team. Partnering with an existing development-ready shop, such as George Mason’s Center for History and New Media, would be an option.

¶ 31 Leave a comment on paragraph 31 0 In either path, an open peer review project would need to acquire a product manager with sufficient technical skill to be able to represent project goals while interacting competently and critically with external software engineering teams.

¶ 32 Leave a comment on paragraph 32 0 An open peer review project would also have to be very clear in its own aims: whether to launch a proof of concept journal of its own device, or to RFP or support an existing online journal seeking a more robust platform or greater functionality. Too much dispersal of goals at too early a stage would be disastrous. For this reason, it would be wise to avoid having full-time academic professionals with the capacity to revise project aims in a supervisory capacity unless project outcomes carried direct career consequences.

¶ 33 Leave a comment on paragraph 33 0 In sum, an open peer review project will need to scope its desires tightly; perform an environmental scan; and proceed on the least expensive path possible in terms of engineering outlay and maintenance.

¶ 34 Leave a comment on paragraph 34 0 Open peer review is a not a revolutionary idea; it is a reaction to how things have been done in the past, recast in the die of distributed, web-based technology. It is a reasonable supposition that the ongoing redefinition of academic publishing’s purpose and operational mechanics will manifest itself quietly but inexorably. This dictates: don’t create something too ambitious, because the future will make it obsolete.

Page 17