Appendix 1: Open Review Software

Kathleen Fitzpatrick

16 June 2012

Open Peer Review software: a report

by Dan Visel, dbvisel@gmail.com. 22 January 2012.
(modified 6 June 2012)

0. Overview: what this report covers

1. The problem to be solved

¶ 2 Leave a comment on paragraph 2 0 This report is looking at current (and near future) tools and platforms that could be used for online peer review with the long-range goal of making a sustainable open source system.

¶ 3 Leave a comment on paragraph 3 0 There are a number of criteria and questions that need to be addressed or answered by any solution.

1.1. Dealing with users

¶ 4 Leave a comment on paragraph 4 0 These questions are common to any social network. The examples of the link aggregators described in section 2.1 might provide useful examples for the design of this system.

• ¶ 5 Leave a comment on paragraph 5 0
• A successful system must allow anonymity (or pseudonymity) as well as self-identification. While a real name policy might be used, there are cases where anonymity is required.
• There should be a way to see in a single page what a user has contributed across the site.
• The quality of reviewers must be assessed. This could be tied to the quality of their contributions; it might also be a separate field.
• A successful system must adequately moderate new content, preventing trolls and spam. It should be possible to down vote unhelpful comments so that they become invisible.
• Reviewing needs to be linear, not binary: while a reviewer should be able to say that something is good or bad, more nuance is also needed.
• How does the system manage score-settling, or the problem of the person who shouts loudest getting the most attention?

1.2. Dealing with content: basics

¶ 6 Leave a comment on paragraph 6 0 These questions are focused on the content that’s in the system: the “texts” to be reviewed and the reviews themselves. Section 3 of this report deals with software that could be useful in managing content.

• ¶ 7 Leave a comment on paragraph 7 0
• Does the system enable embedding/uploading video or audio reviews?
• Does the system allow for commenting at the page/paragraph/sentence/word level? on audio, image or video files?
• Versioning needs to be accommodated and easily accessible: multiple editions of texts (and comments) need to be accounted for. The system should allow readers and reviewers to compare drafts before and after review.
• What kinds of citation practices or other means of tracing the lineage of ideas does the system allow for? Are there means available of citing or linking to individual comments?
• Can filters be customized by author/editor? Can the system filter reviews by categories (i.e., distinguish between reviews dealing with different facets of the project (argument, evidence, writing style, etc.) or different levels of revision suggestions (recommended versus required revisions, etc.)?

1.3. Creating an editorial system

¶ 8 Leave a comment on paragraph 8 0 The questions in this section are less technical and more dependent upon how the system is implemented for ease of use. A basic editorial workflow will need to be imagined to construct a system

• ¶ 9 Leave a comment on paragraph 9 0
• What kinds of prompts are available to guide reviewers?
• What options does the system provide for both public and private discussion?
• Does the system offer mechanisms for alerting members to materials in need of review? Can such mechanisms be customized based on tagging, key words, or author/editor “invitation”?
• How are questions of “finality” addressed within the system: does a review ever end?

1.4. Dealing with intellectual property

¶ 10 Leave a comment on paragraph 10 0 Finally, some questions about intellectual property need to be spelled out as part of the editorial process.

• ¶ 11 Leave a comment on paragraph 11 0
• What are the understandings under which users contribute to the site? How are those understandings spelled out in the terms of use?
• Who “owns” the comments made within the system? What terms of use or other provisions are made for intellectual property and the reuse of material produced within the system?

2. Models for what could be done

¶ 12 Leave a comment on paragraph 12 0 A number of models for what could be done with online peer review already exist. While the majority of what’s described in this section are proprietary (and thus not suitable) solutions, these examples are worth learning from.

2.1. Moderation: useful models

¶ 13 Leave a comment on paragraph 13 0 A number of proposed systems aren’t directly useful as a solution to the problem, but may provide useful models for how commenting and/or moderation could work. These include, in order of discussion:

• ¶ 14 Leave a comment on paragraph 14 0
• Slashdot (http://slashdot.org; the CMS has been released as an open source project called Slash (http://www.slashcode.com).
• Digg (http://digg.com)
• Reddit (http://reddit.com)
• Hacker News (http://news.ycombinator.com)
• Disqus (http://disqus.com)
• Metafilter (http://www.metafilter.com)
• Ask Yahoo! (http://ask.yahoo.com)
• StackOverflow (http://stackoverflow.com)
• Quora (http://quora.com)
• Facebook (http://facebook.com)

¶ 15 Leave a comment on paragraph 15 0 Slashdot is the oldest of these sites, having been around since 1997; it pioneered many of the basic ideas used in commenting and moderation on the web. Slashdot is essentially a blog that presents tech news; most posts are approximately a paragraph long and based around a link. To fully use the site, users must have an account. Posts are user-submitted, though editors choose which posts appear on the site. Slashdot’s moderation has changed an enormous amount over time (largely on account of increased volume of users), but it became notable for a system which allowed users to up- and down-vote articles and comments. This was originally designed to highlight the most interesting comments; as volume grew, the task of moderation increasingly became one of hiding spam and trolls. Each comment has a numerical score from –1 to 5; comments start out at 1, though if other users mark them as insightful, informative, or funny, they receive more points and if users mark them as unhelpful or junk they lose points. Higher-scored comments are shown by default; lower-scored comments are hidden. In addition, the scores of comments are tied to the trust metric of users, called karma (the term originated here and has been picked up by other platforms): if you submit comments or posts that others judge to be good, your karma score goes up. Posts or comments by users with higher karma start with higher scores than posts or comments by users with low karma.

¶ 16 Leave a comment on paragraph 16 0 While Slashdot has declined in popularity over time, in part because of a perceived drop in quality (the “EternalSeptember” problem), the central idea is in use by most blog-like sites with large readerships. Digg might be seen as a stripped-down iteration of the Slashdot concept that arrived in 2004: users could submit links, comment on them, and vote the links or comments up or down. There’s no editorial layer. Until a redesign in 2010 (based on a perceived drop-off in quality), karma was extremely important, which led to the emergence of a class of power users who could essentially control what appeared on the site with their up and down votes. Reddit, founded in 2005, presented a more refined version of Digg’s approach: there’s more of an emphasis on sub-communities and comments. User karma seems to matter less than it does on Digg; there are also more clearly defined cultural norms for what makes a good (as opposed to a simply funny) comment. At the moment, the site is more successful; a good deal of anti-SOPA activism, for example, came from the Reddit communities. Hacker News, started in 2007, works the same territory that Slashdot did; however, the interface is much more stripped down. Posts can only be voted up; if users have enough karma, they can vote comments down, but the site generally works on promotion. Hacker News is generally successful at fostering smart conversations, although much of this may be due to its comparatively small user base.

¶ 17 Leave a comment on paragraph 17 0 Disqus is a plugin for most content management systems that allows commenting using relatively similar features to these sites. In general, Disqus’s features are better than the commenting systems that come with most CMSes out of the box; they allow users to be signed into a single commenting system across multiple sites, which tends to encourage more commenting because there are fewer barriers to entry. Karma operates both inside a site that’s using Disqus and across all sites that the commenter is commenting on, reducing spam.

¶ 18 Leave a comment on paragraph 18 0 A different approach to the problem of fostering smart conversations is be the route taken by Metafilter, which presents links and comments, like Slashdot, Digg, Reddit, and Hacker News. Founded in 1999, it takes a different approach to the problem of discourse quality: from 2004, new users have been charged a one-time $5 registration fee. This weeded out a great deal of the trolling that other sites have to deal with; because the users are self-selected, there’s less problem with trolling and poor-quality commenting. Social norms and peer pressure tend to keep things clean here (as is the case with Hacker News), though there are human moderators as well. Metafilter notably differs from the above sites in that it uses no karma system. Because of the registration fee, however, it’s operating at a much lower volume; as with Hacker News, this might make it a more useful example.

¶ 19 Leave a comment on paragraph 19 0 All of these sites, of course, require an account; purely anonymous comments aren’t allowed, although there’s nothing to stop people from creating anonymous accounts.

¶ 20 Leave a comment on paragraph 20 0 A small jump over from the links-and-comments sites are question-and-answer sites. (A number of links sites also host questions in various forms – AskSlashDot, Ask MetaFilter, etc.) A basic template of this sort of site might be seen at Yahoo! Answers, started in 2005, which allows users to ask and answer questions; however, there’s no metric for authority, and the content isn’t especially good. StackOverflow began as a more specialized site, based around computer programming questions, in 2007; since then, it’s grown and begun to cover other areas. StackOverflow uses a reputation score: a good answer to a question (or other actions that affect the community and content positively) results in more reputation points. This has generally been successful. Quora is a more recent question-and-answer site; it’s similar to StackOverflow, but emphasizes real-world names and qualifications as experts (taking advantage of users’ social networks from Facebook or Twitter). It’s too soon to say whether this reputation model will be successful outside the technology/Silicon Valley world that is currently its primary focus; within that limited community, it seems to work.

¶ 21 Leave a comment on paragraph 21 0 Finally, it’s hard to avoid Facebook, which has liberally borrowed from all of these sites and introduced some of its own innovations in moderation. Users on Facebook can choose who of their friends they want to see more or less of; Facebook is internally using its own karma system which determines how important particular users’ updates are, though this is almost entirely opaque to the end user. Comments on posts don’t currently receive any internal moderation, though this seems like it may change so that the most liked or shared comments are shown most prominently.

¶ 22 Leave a comment on paragraph 22 0 What’s worth taking away from these examples? A couple of key points might be usefully remembered:

• ¶ 23 Leave a comment on paragraph 23 0
• Volume. The problems of moderation are directly related to the problem of site volume: a site with a million users has very different problems from a site with a thousand users. What works on Facebook isn’t likely to work on a site designed for 10,000 users. Smaller communities tend to have better conversations because users tend to know each other; this declines as the size of the community increases.
• Reputation. The problem of reputation is one that needs to be dealt with: not all commenters are equal. Karma might be part of a solution to this, though it tends to lead to users gaming the system; in the academic world (where the number of potential commenters is finite), there are probably more useful solutions.
• Identity. Quality in moderation and community appears to be directly correlated to consistent use of user accounts, though those may be comparatively anonymous. Recently a move has been made to correlate quality of community to lack of anonymity, but this correlation is more ambiguous.

2.2. Sites based around commented reading or presentation

• ¶ 24 Leave a comment on paragraph 24 0
• Bookglutton (http://www.bookglutton.com)
• NowComment (http://nowcomment.com)
• Eli (http://www.elireview.com)
• Highlighter (http://www.highlighter.com)
• Social Book (http://livemargin.com)

¶ 25 Leave a comment on paragraph 25 0 A number of sites have been constructed around the idea of social reading, generally without the concept of peer review. Digress.it, discussed in section 3.1, is primarily a WordPress plugin, but also functions as a platform for the distribution of commented texts; it could be seen as being part of this group.

¶ 26 Leave a comment on paragraph 26 0 Bookglutton was one of the first of the dedicated sites out of the gate, in 2008; it allows users to upload documents and to comment on them on a granular level. A social network allows for groups to be constructed around texts; integration with Facebook is also present. Books can be embedded in other sites (like blogs) as widgets. Rudimentary peer review might be possible with Bookglutton by constructing a group for reviewers and giving out anonymous accounts as necessary; no one seems to have done this yet.

¶ 27 Leave a comment on paragraph 27 0 Bookglutton only works with simple text files, which might rule out academic documents; this is also a downside to NowComment, a very similar project. NowComment seems to have more educational users than BookGlutton; BookGlutton’s interface is much better than NowComment’s. Both are self-contained sites, which complicates the possibility of extending them. While these sites function similarly to plugins like CommentPress, the modular nature of plugins means that they can be used in radically different environments; users of BookGlutton or NowComment need to use the site.

¶ 28 Leave a comment on paragraph 28 0 Eli works very similarly; the platform is tightly tied to the idea of teaching writing and teachers giving writing feedback to students. Highlighter takes the same idea and moves it to a predominantly mobile platform; there’s something more of a focus on analytics, so that publishers can see what readers are most interested in. Highlighter is also interesting in that they provide code that can be added to any website adding a social annotation layer. Neither of these is an especially good fit for peer review, though Highlighter seems to have more funding than the other sites and a bit more drive; they might be worth keeping an eye on. (Worth noting: they’ve hired people from the Sakai project.)

¶ 29 Leave a comment on paragraph 29 0 SocialBook is Bob Stein’s new platform for social reading, which hasn’t yet been released; it’s a publisher-centric model, allowing the sale of group access to texts. The focus is strongly on how different groups can use the same text; it seems possible that with some editorial structures, online peer review could be done using the software. This is not, however, something that it is designed for out of the box.

2.3. Sites based around collaborative writing

• ¶ 30 Leave a comment on paragraph 30 0
• Google Docs/Drive (http://docs.google.com)
• Google/Apache Wave (http://incubator.apache.org/wave/)
• various wikis, including Scholarpedia (http://www.scholarpedia.org)

¶ 31 Leave a comment on paragraph 31 0 Collaborative writing environments might present a path forward; it’s possible to imagine, for example, using Google Docs for rudimentary peer review. A document could be forwarded to a reviewer who adds comments to it; these come back to an editor, who could forward them. One problem is the lack of a way to anonymize comments; and while Google Docs does track changes, the user interface for this is too poor to consider as a serious solution. (It’s also difficult to tell what Google’s long term plan is for Google Docs, which they seem to be de-prioritizing.)

¶ 32 Leave a comment on paragraph 32 0 Google Wave presented what might have been a solution: as initially presented, it was an environment for conversations around documents. While full of promise, it was confusing, users were slow to pick it up, and Google killed the project; it’s been picked up by the Apache Software Foundation and rebranded Apache Wave. Scheduled for release in 2011, it is still not out. It seems possible that this could work as an environment for online peer review, but this can not be determined until release.

¶ 33 Leave a comment on paragraph 33 0 Google/Apache Wave is reminiscent of wikis in that it allows for the editing of documents by many users over time. While the use of wikis in peer review has been tried – Scholarpedia, based on MediaWiki, is an example – the form is hard to use and requires a great deal of editorial self-discipline to be useful.

¶ 34 Leave a comment on paragraph 34 0 ICE, a WordPress plugin, makes collaboration possible in editing WordPress content; it is discussed below.

2.4. Other sites

• ¶ 35 Leave a comment on paragraph 35 0
• Academia.edu (http://www.academia.edu)
• Scribd (http://scribd.com)
• Curēus  (http://www.cureus.com/, formerly Peer Emed)
• O’Reilly Rough Cuts (http://my.safaribooksonline.com/roughcuts)
• Philica (http://philica.com)

¶ 36 Leave a comment on paragraph 36 0 Academia.edu is a social network founded in 2008 based around the unit of the academic paper. Users can create accounts and follow each other like any social network; they can also upload papers. Despite the domain name, this is a for-profit corporation. There are currently no facilities for detailed commenting on documents; this probably isn’t that useful.

¶ 37 Leave a comment on paragraph 37 0 Scribd is immensely popular; the site allows users to upload text in a variety of formats (Word, PDF, EPUB) to be viewed online. While it’s not inconceivable that Scribd could move in the direction of collaborative reading (by allowing targeted comments, for example), this hasn’t happened so far. Users can comment on documents, but there’s only a single comment stream for documents. Scribd is notable in its integration into other social networks (especially Facebook) and its attempts to build communities around texts, but this hasn’t gotten far.

¶ 38 Leave a comment on paragraph 38 0 It is worth taking a look at PeerEmed, an attempt to make an online peer review system for medical papers based on the Scribd engine. This doesn’t seem to have much traction despite being started in 2010; peer review is limited to a single stream of comments and a single numeric score. The most interesting element of this is the editorial layer, which doesn’t seem tremendously engaged; it does point to the idea that a simple peer review system could be set up using a blog and embedding documents from somewhere like Scribd. (PeerEmed is currently in the midst of rebranding itself as Curēus; it’s difficult to tell where they’ll end up.)

¶ 39 Leave a comment on paragraph 39 0 O’Reilly’s Rough Cuts is an online book reading environment from O’Reilly, the technical publisher; it is designed to get books on new or developing subjects to readers before they have been thorough edited.  Readers – who pay to use the service – are invited to leave comments on what works and doesn’t work. Commenting is granular, so readers can highlight passages for commenting. On the whole, however, this doesn’t seem to have taken off online, even though this is aimed at a highly technical audience; most books have few if any comments. The same books are available in less interactive formats (EPUB, PDF), which may be compromising the online environment.

¶ 40 Leave a comment on paragraph 40 0 Philica is a somewhat quixotic attempt to make an online peer reviewed journal; articles are accepted on any topic. Qualified experts are allowed to write reviews for submitted articles. Comments aren’t granular. It’s a somewhat interesting implementation – they have specifically tried to reimagine peer review for the online world – but it’s hard to imagine this being more generally used or gaining much traction.

2.5. Prior online experiments with peer review

¶ 41 Leave a comment on paragraph 41 0 Plenty of these exist. A detailed list is outside of the scope of this report, but important for any consideration of the subject.

3. Potential pieces of an implementation

¶ 42 Leave a comment on paragraph 42 0 Nothing off the shelf is going to provide a perfect solution for the problem at hand. But some existing content management systems & plug-ins could conceivably be part of a solution. Both Drupal and WordPress are robust enough that they could perhaps be used as a basis for this project. Both are open source and have fairly robust communities; in general, WordPress is easier to use, while Drupal’s architecture is more powerful. It’s also worth keeping in mind Open Journal Systems, a CMS based around journal publication.

¶ 43 Leave a comment on paragraph 43 0 For both WordPress and Drupal, any solution is going to involve plugins (WordPress) or modules (the word Drupal uses for the same thing), chunks of code that enable new functionality. Plugins are available to do almost anything conceivable for this project. However, plugins can also complicate the development process: generally they’re made by different programmers and may not be updated as frequently as the main CMS is. Getting plugins to work together can also present problems: adding a new plugin can cause new problems which will need to be debugged.

¶ 44 Leave a comment on paragraph 44 0 For any one of these systems, user accounts are going to be important: the CMS needs to function like a social network, so that individuals using the site have an easily accessible history of their contributions. Drupal handles this out of the box; WordPress can be turned into more social software using BuddyPress, a popular plugin.

3.1. Commenting plugins

• ¶ 45 Leave a comment on paragraph 45 0
• CommentPress (http://www.futureofthebook.org/commentpress)
• Digress.it (http://digress.it)

¶ 46 Leave a comment on paragraph 46 0 CommentPress is a WordPress plugin developed by the Institute for the Future of the Book; Digress.it is a forked version of the project by original developer Eddie Tejeda that works similarly. Both allow paragraph-by-paragraph level commenting on text in WordPress. Digress.it also offers a hosting platform, so that potential users don’t need their own installation of WordPress; CommentPress is a pure plugin that the user must install on a WordPress installation. Both have been used online extensively (Cornell’s Regulation Room project is probably the most serious install of Digress.it); the differences between the two pieces of software aren’t extremely serious.

¶ 47 Leave a comment on paragraph 47 0 Either of these plugins could function as part of a peer review service. One drawback is the backend of these projects: getting text into them requires the editor to understand WordPress management. A better backend would be a useful avenue for further development. Versioning is also a problem: if a comment suggests a change in the text, the editor changes the text; there’s no way of marking the suggested change as having been done (or pointing back to the version of the text that the comment referred to). In addition, it’s difficult for reviewers to edit comments that they’ve made. These problems can be solved, possibly  by using other plugins.

3.2. Revisioning & document management plugins

• ¶ 48 Leave a comment on paragraph 48 0
• Document Revisions (http://wordpress.org/extend/plugins/wp–document–revisions/)
• Edit Flow (http://wordpress.org/extend/plugins/edit–flow/)
• Annotum (http://annotum.wordpress.com)
• ICE (http://nytimes.github.com/ice/demo/)

¶ 49 Leave a comment on paragraph 49 0 Document Revisions is a WordPress plugin designed to allow multiple revisions of files being used in WordPress. This is not, however, designed to be used with text entered directly in WordPress. Rather, it will work with documents (like PDFs or Word documents) that are being managed by WordPress. This could be used as part of a rudimentary peer review system if the documents being used were in Word format; Document Revisions can also be used in conjunction with Edit Flow, which is designed for collaborative editing and management of a WordPress site. Edit Flow allows internal editorial comments on different versions of documents; it brings fuller control over versioning to WordPress. It’s possible to imagine these two plugins being used together to make a peer review system, although the editors would be working on the backend of the system, not publicly, which seems like a drawback. These plugins are designed for web magazines, not really for peer review on journals; however, they could probably be put to use.

¶ 50 Leave a comment on paragraph 50 0 Annotum is a new WordPress theme (which includes a variety of plugins) designed to function as an open-access scholarly publishing platform, coming from the science world. It’s too early to say much about how well Annotum will work, as it doesn’t seem to have much actual use; however, it is designed to support an article review workflow and version comparison, as well as structured documents and smart citations. This is a project that should be kept in mind.

¶ 51 Leave a comment on paragraph 51 0 ICE is a new WordPress plugin developed by The New York Times that enables collaborative editing of posts akin to TrackChanges in Microsoft Word. This could provide basic versioning for editing.

3.3. Dedicated journal content management systems

• ¶ 52 Leave a comment on paragraph 52 0
• Open Journal System (http://pkp.sfu.ca/?q=ojs)

¶ 53 Leave a comment on paragraph 53 0 Open Journal System is a CMS for online journals, released by the Public Knowledge Project, a consortium of the University of British Columbia, Simon Fraser University, and Stanford, under the direction of John Willinsky. The software is open source and dedicated to open access; over 5000 journals use OJS. A variety of modules exist, among them one dedicated to publishing monographs (Open Monograph Press).

¶ 54 Leave a comment on paragraph 54 0 OJS does support online peer review, though not in an especially automated fashion. The submitter sends a Word file or PDF to the editor; for a blind peer review, the editor removes the submitter’s name and sends it (through OJS) to the reviewer. The reviewer edits the document in Word and sends it back; if approved, final changes are made in Word and the piece is published through OJS.

¶ 55 Leave a comment on paragraph 55 0 One strength of OJS is that it’s essentially a framework and is content agnostic. OJS can be used with something like CommentPress as a display front end; it’s conceivable that a module for more interactive peer review could be built on the OJS format.

¶ 56 Leave a comment on paragraph 56 0 A distinct advantage of OJS is how widely it’s used. There does not appear to be a huge number of OJS developers (as there are for WordPress or Drupal); however, the Public Knowledge Project will do work for hire. It’s worth keeping Open Journal System in mind.

¶ 57 Leave a comment on paragraph 57 0 A current weakness of OJS is that it is not particularly social: user accounts don’t form a major part of the interface, although it is possible that more development (or integration with another CMS) might solve this problem.

3.4. Other frameworks & ideas that could be useful

• ¶ 58 Leave a comment on paragraph 58 0
• Open Annotation Collaboration (http://www.openannotation.org)
• Hypothes.is (http://hypothes.is)
• PressForward (http://pressforward.org)
• PLoS article-level metrics (http://blogs.plos.org/plos/2009/09/article–level–metrics–at–plos–addition–of–usage–data/)
• DocumentCloud (http://www.documentcloud.org/; example use:http://www.propublica.org/article/why–cant–linda–carswell–get–her–husbands–heart–back )

¶ 59 Leave a comment on paragraph 59 0 Open Annotation Collaboration is developing a framework for sharing annotations across the web. While this isn’t specifically focused on the problem of peer review – and is generally operating at a fairly deep level – development of a system for open peer review should take this system into account.

¶ 60 Leave a comment on paragraph 60 0 Hypothes.is is a new and relatively prominent attempt at adding a reputation layer to the web, similar to the old semantic web idea. It is very hard to tell if this will get anywhere; historically, this sort of idea fails quickly and ignominiously, though the slate of people involved is impressive. The project essentially seeks to bring open peer review to the entire web; expertise and reputation factor into it. It might be worth meeting to see if there’s an intersection with the academic problem of peer review.

¶ 61 Leave a comment on paragraph 61 0 PressForward’s critical work on the peer review process (including sites like Digital Humanities Now) should certainly be taken into account.

¶ 62 Leave a comment on paragraph 62 0 PLoS provides article-level metrics for every article published in their journal; this is an example that should be taken into account. These metrics allow readers to see who’s cited the article and where mentions have been made. This is worth doing; if a new system is being built, building this in would not be especially tricky.

¶ 63 Leave a comment on paragraph 63 0 DocumentCloud is a new project for journalists based around the idea of making primary sources more accessible. As demonstrated at ProPublica, it enables dynamic linking to the documents that were the original sources for information used in news reports. It’s imaginable that a similar system, put into use the academic world, might function as a way of making citations more immediate. While such a system might be desirable, it is probably outside the scope of this project.

4. Suggestions for a solution

¶ 64 Leave a comment on paragraph 64 0 No perfect solution currently exists for comprehensive peer to peer review. However, the number of tools and platforms that could be used for more limited implementations is growing fairly rapidly. An increasing number of these tools and software are for-profit, limiting their utility in the academy; and none bring to bear everything that would be needed.

¶ 65 Leave a comment on paragraph 65 0 A handful of different strategies might be employed.

4.1. Building from scratch

¶ 66 Leave a comment on paragraph 66 0 It would be possible to build a bespoke platform for online peer review. This isn’t quite the immense job it would have been a few years ago; libraries exist for most of the major functionality that would be needed (a social network component, commenting, versioning, moderation). This would provide the benefit that all of the code could be managed by a single team, making changes relatively simple.

¶ 67 Leave a comment on paragraph 67 0 The downside, however, is when the software requirements change: the programmers who developed the system would be the ones most competent to make changes, which might lead to bottlenecks. Development time in general would be likely to take much longer than any other strategy.

¶ 68 Leave a comment on paragraph 68 0 A number of components of this project – for example, an automated system that will issue invitations to reviewers when a piece is ready to be reviewed, which then creates semi-anonymous accounts for them – will certainly need to be built from scratch, just because nobody is currently doing anything like this. This will be true in the next two scenarios as well.

4.2. Building from little pieces

¶ 69 Leave a comment on paragraph 69 1 A number of pieces are currently available that could be cobbled together to create a workable system. One can imagine, for example, a WordPress-based system that uses BuddyPress to provide a social network, Annotum to provide an article review workflow, CommentPress to enable paragraph-by-paragraph level commenting, and Disqus for superior comment moderation powers. Additional plugins might handle bibliographic citations (like Mendeley), audio or video content (like PodPress), or versioning (Document Revisions).

¶ 70 Leave a comment on paragraph 70 0 There is a large potential downside to this strategy: if many different pieces of software (eight, in this example) are being used, the system is dependent upon many different code bases. Updates to WordPress, for example, might break something in CommentPress. If relatively standard plugins are being used, it won’t be hard to find a programmer who knows what needs to be changed; but more obscure plugins may require dealing with the people responsible for their development. A competent WordPress programmer might be able to handle these changes over time; however, this is going to be an ongoing task, as software on the web is rarely static. (While Drupal is more protean than WordPress, using it is likely to exacerbate this potential problem: a Drupal installation will almost certainly need more modules than a WordPress installation will need plugins.)

¶ 71 Leave a comment on paragraph 71 0 It is also worth remembering that plugins almost certainly won’t work nicely together out of the box: some tweaking by a programmer will be necessary. This means that a programmer staffing the project is likely to be necessary. In addition, of course, designers will need to make a custom user interface that makes the editorial structures clear.

¶ 72 Leave a comment on paragraph 72 0 This strategy is worth considering.

4.3. Building from big pieces

¶ 73 Leave a comment on paragraph 73 0 A third strategy might be to split the difference between the first two by creating a system that uses existing software along with a fair amount of custom coding. For example, we might take Open Journal Systems, which takes care of most of the problems of editorial management for journal content as the back end, and use WordPress as a front end, to display the content and allow reviewers and authors to interact. This will require a fair amount of programming work: these systems are very different, and a good deal of custom functionality must be added. But this strategy does not have as many dependencies on other code bases as 4.2 does; and it would be considerably less work than 4.1. Because what is being done is very specific, forking the OJS codebase might be a reasonable solution, removing that dependency entirely; in effect, that would make this a separate piece of software.

4.4. What’s not in software

¶ 74 Leave a comment on paragraph 74 0 Finally, some words should be said about what software won’t do by itself. While software can certainly go a long way towards easing workflow, it won’t make this problem go away entirely. Nor is the role of the editor going to disappear. Editors will need to know how to guide content through the system; reviewers are going to need some instruction in how to interact with the system. And while automatic moderation can deal with spam, dealing with trolls will require the good will of the community.  Dealing with score-settling is likely to still be the job of the editor.

¶ 75 Leave a comment on paragraph 75 0 It’s also worth noting that deploying this software is likely to be iterative: when it’s actually being used, editors and reviewers will find that some of the software doesn’t work the way they’d like, and that changes should be made.

¶ 76 Leave a comment on paragraph 76 0 If the system is to be installed by other people, detailed instructions on both installing and maintaining the system will have to be provided; there will also need to be coordination on updating the system or plans to add new functionality. This work is crucial if the system is to be broadly used; without it, plenty of open source projects have foundered even though the software was suitable.