Recommendations for Technological Systems

Kathleen Fitzpatrick

16 June 2012

¶ 1 Leave a comment on paragraph 1 0 Given the desire for flexibility that our discussions unearthed, as well as the recognition that different communities of practice will have not only different goals for their review processes but also fundamentally different ways of working, we are not led to propose the development of a new, monolithic platform or review tool. Developing from the ground up a platform with the robustness and ease of use of WordPress, the flexibility of Drupal, the possibilities for multimodal publishing of Omeka or Scalar, and the workflow management of Open Journal Systems would not only be an inordinately expensive proposition, and would not only have to fight the already extensive buy-in that those platforms have developed, but it would also present requirements for ongoing support that are unlikely to be met in the current environment.

¶ 2 Leave a comment on paragraph 2 0 We thus intend in what follows to identify a suite of possibilities, an “ecology” of tools that can work together in different combinations. Much of what follows is drawn from work done for the advisory group by Dan Visel and Peter Brantley; their full reports are included as appendices to this paper.

¶ 3 Leave a comment on paragraph 3 0 Open review tools need to serve a number of basic functions, including but not limited to the following:

0. ¶ 4 Leave a comment on paragraph 4 5
1. User management
   0. ¶ 4 Leave a comment on paragraph 4 5
   1. Identity: A flexible open review system must allow for both anonymity or pseudonymity and for self-identification, and must allow a particular publication or community of practice to determine whether to enforce a real-name policy, or whether anonymous commenting is desirable.
   2. Roles: An open system should allow for the creation of different user types with different privileges and permissions, such as Editor, Moderator, Author, Reviewer, and so forth.
   3. Aggregation: An open review system must be able to gather and display the complete contributions of an individual user.
   4. Reputation: This system should permit the assessment not just of primary texts, but of the quality of contributions of individual reviewers.
   5. Moderation: The system should allow both spam prevention and for comments to be voted down (when unhelpful) or deleted (when trolling).
   6. Nuance: While a reviewer should be able to register lightweight approval or disapproval of content (“likes” and “dislikes”), an emphasis should be placed on more qualitative assessment.
   7. Community norms: The system should provide means to address score-settling behavior and to keep any given user from dominating a conversation, among other such issues.
2. Content management
   0. ¶ 5 Leave a comment on paragraph 5 3
   1. Media: The system should provide the option to embed or upload image, audio, or video comments; similarly, the system should permit commenting on media objects.
   2. Granularity: The system should permit comments to be attached to the page/paragraph/sentence/word level of a text, as well as to the entirety of a text.
   3. Versioning: The system should ideally allow a text to be revised, and should account for multiple editions of texts and comments. Readers and reviewers should be able to compare drafts accordingly.
   4. Citation: The system should allow the lineage of ideas to be traced both forward and backward in time through both texts and comments, using links that function in both directions (that is, a citation of text A in text B should not only result in a link from B to A, but also a corresponding link on A to B).
   5. Filters: The system should allow an author or editor to customize the kinds or quality of comments they see at any given time. Reviews might be filtered by their focus (argumentation, evidence, style), for instance, or by the community’s rating of their helpfulness, by the identity of the reviewer, or by other factors.
3. Workflow management
   0. ¶ 6 Leave a comment on paragraph 6 2
   1. Reviewer selection: The system should allow editors to solicit specific reviewers from an available pool; it could also allow authors to recommend particular reviewers.
   2. Notification: The system should alert members to the presence of new texts available for review, especially where new texts speak to declared member interests, and should alert authors to new comments on their texts.
   3. Prompts: The system should allow editors to create specific and easy prompts to guide reviewers in their tasks.
   4. Privacy: The system should allow both public and private conversations to take place around texts, in keeping with community preferences and policies.
   5. Closure: The system should permit review periods to have a fixed term where desired, as well as permitting open-ended conversation.
   6. Export: The system should allow reviews (including aggregations and filters for nuance, granularity, reputation, and so forth) to be collated and remixed in ways useful to editors, authors, and readers.

¶ 7 Leave a comment on paragraph 7 0 Building a platform from the ground up to support such a wide range of author, editor, reviewer, and reader needs would be a complex and costly process, and would run the risk of producing a fragile, difficult to support piece of software. Far more productive would be to work with existing platforms, to see what can be done with combinations of existing tools, and to explore how ongoing project developers might be led to incorporate the needs of open review into their project roadmaps.

¶ 8 Leave a comment on paragraph 8 0 While there are many content management systems on top of which a robust open review process might be developed, for purposes of example, to convey how one system might function, we will focus in this section on the possibilities presented by WordPress, which has a robust plugin architecture, a relative ease of use, and a vibrant, organized developer community. WordPress’s core architecture includes a number of features that are useful to formal publications, including the ability to support multiple sites within one installation, built-in versioning that includes the editor who made the revisions, customizable user roles, threaded comments, and the like. An installation of WordPress might be combined with a number of plugins to serve many of the open review functions named above; such plugins include:

• ¶ 9 Leave a comment on paragraph 9 0
• BuddyPress (http://www.buddypress.org): a social-network plugin that facilitates the formation of and communication within groups formed in a larger network.
• CommentPress (http://www.futureofthebook.org/commentpress) and Digress.it (http://digress.it): facilitate paragraph-, page-, and document-level commenting.
• Annotum (http://annotum.org/): a plugin designed to extend WordPress as a platform for scientific publishing, by adding support for multiple authors, citations, versioning, and revision comparison, among other features.
• Edit Flow (http://editflow.org/): an editorial workflow plugin facilitating editorial team collaboration, review tracking, reviewer notifications, and more.

¶ 10 Leave a comment on paragraph 10 0 Of course a number of the functions desired in an open review platform remain unserved by these plugins; while CommentPress and Digress.it both provide commenting of a greater granularity than does WordPress out of the box, each is restricted to the paragraph or page level, and revisions to an underlying text can break the relationships between that text and its comments. Similarly, while BuddyPress facilitates the creation of community member profiles and captures a member’s participation within a network, it doesn’t currently provide a means to review or rate that activity, or to transform those ratings into some measure of reputation. Needs such as these will have to be met by extending the features of existing plugins or creating additional plugins.

¶ 11 Leave a comment on paragraph 11 1 The complexity of such a plugin architecture offers some reason for caution; given the many dispersed developers working independently on these components, there is cause to expect that certain plugins might not interoperate well with others, that updates to the WordPress core may break some plugin functions, and that some projects may cease to be updated as their developers move on to other projects. However, working together, a consortium of interested parties (which might include publishers, libraries, scholarly societies, individual scholars, and funders) could assemble an easily installable package of plugins and commit to the development and maintenance of that package on behalf of its user base; the CUNY Academic Commons’s Commons-in-a-Box project might provide a model, and in fact such an open review platform project might be imagined as a fork of this project.[29]

¶ 12 Leave a comment on paragraph 12 1 There are, of course, other notable platforms that might be built upon or learned from, including other content management systems such as Drupal and Joomla, as well as platforms designed specifically for scholarly communication, such as Open Journal Systems from the Public Knowledge Project, or Ambra, the platform on which the PLoS journals are published. An ideal scenario would of course be the development of an open review architecture that could be implemented in a platform-agnostic fashion, allowing reviewer comments to be aggregated across the web, regardless of the system hosting the primary text. (Services such as Disqus provide such a comment-aggregation function, but some concerns about their business models and proprietary data structures make them less-than-ideal as solutions.) The crucial element in any case is flexibility; in order to meet the very different requirements that different communities will bring to open review, and in order to remain sustainable as technologies change, the systems that support such practices will require significant malleability.