mediacommons and peer-to-peer review

Kathleen Fitzpatrick

30 August 2009

¶ 1 Leave a comment on paragraph 1 3 This need to focus on the communal aspects of peer-to-peer review — in particular, the review of the reviewers – led me and my colleagues, in the early stages of our planning, to start thinking about MediaCommons as less a digital scholarly press than a digital scholarly network. Though the social aspects of MediaCommons are not its primary product, we’ve increasingly come to believe that they’re a precondition for the success of the publishing aspects of the network. Too many digital publishing experiments, like Philica, have lagged due to an assumption that might be summed up as “if you build it, they will come.” In fact, such publishing experiments would often benefit from examining the relative success of MySpace in comparison with MP3.com, thus placing a greater focus on getting users to come in the first place, on drawing them in by demonstrating the ways that the network’s connections will benefit their work. For this reason, the first part of MediaCommons that we are building is the community, in order to create a network of trust between authors and reviewers. For our purposes, a more appropriate analogy between MediaCommons and other “web 2.0” systems, rather than “eBay for academics,” might well be “Facebook for scholars,” as we are focused on building a network structure that allows people, and not just texts, to interconnect. And the most salient point of that comparison is this: as some scholars have argued, the success of Facebook, as compared with earlier social networking systems such as Friendster and Orkut, derived in no small part from the decision its developers made in keeping the network relatively closed by limiting its use, in its early days, to students at a small number of colleges and universities and by focusing on the pre-existing connections among the members of those institutions.[1.40] The emphasis, in other words, was not on allowing users to create new social networks, but rather on helping them extend their offline social networks into digital environments. MediaCommons will similarly begin by facilitating the relationships among scholars who are already connected — who already attend the same conferences, publish in the same journals, and read one another’s work.

¶ 2 Leave a comment on paragraph 2 0 This text is being written while the MediaCommons systems are still very much in development, and therefore what follows retains a speculative, hypothetical tone; the actual functioning of the network may well wind up being a bit different than what I here project.[1.41] We are fairly certain, however, that the peer-to-peer system that will be the backbone of MediaCommons will involve the development and implementation of a number of interconnected modules, including:

¶ 3 Leave a comment on paragraph 3 0 * a networked user profiling system that will enable scholars to define their interests in taggable, complexly searchable ways;

¶ 4 Leave a comment on paragraph 4 0 * a portfolio system that will allow users to build and maintain a comprehensive record of their writing within the site and in other networked spaces, both formal and informal, allowing scholars both to maintain publicly accessible versions of their work and to receive some sort of academic “credit” for the kinds of work — including peer reviews and participation in online forums — that too often remain invisible;

¶ 5 Leave a comment on paragraph 5 0 * a sophisticated recommendations system that uses the information in a member’s profile, along with robust textual analysis of documents in the network, to present the user with frequently updated suggestions for texts to read, discussions to participate in, and collaborators to work with;

¶ 6 Leave a comment on paragraph 6 0 * a “reputation” system that will allow users of the network to review the reviewers, to assess the “value” of a particular scholar’s work within the network.

¶ 7 Leave a comment on paragraph 7 0 Many systems like these have been developed in isolation from one another, both in open-source and proprietary variants, but they have not as yet been brought together to create such a dynamic community structure, nor have they been put to the uses that scholars might make of them. A social networking system such as Facebook, for instance, allows its users to create profiles and join groups, but its publishing tools are limited in effectiveness.[1.42] Drupal, an open-source content management system, allows users to create limited profiles, and tracks user participation in a site, but that information remains relatively static rather than being used dynamically to help generate connections amongst users. BuddyPress, a WordPress plugin, uses this same profile and user tracking information in conjunction with the ability to form and publish with groups, but like Drupal, the system requires extensive customization to work in a complex publishing environment. Recommendation systems of varying stripes are in use at a number of commercial sites (usually extending the “customers who bought x also bought y” type to text recommendations), but they usually rely upon keywords rather than full textual analysis, and little use of such systems has been made in the organization and dissemination of scholarly research. Then there are “reputation” systems, such as that in use in a large-scale discussion forum like Slashdot, which have proven effective at filtering out unhelpful or nuisance commentary, but their potential use in a system of scholarly review has as yet gone unexplored.

¶ 8 Leave a comment on paragraph 8 0 MediaCommons intends to bring such systems together, providing scholars with a range of tools through which to connect with one another, to produce and publish networked, multi-modal texts, to review those texts, and then, most crucially, to review the work of the reviewers, enabling the community to determine its own standards and adjudicate their implementation. In a peer-to-peer reviewing system, “reputation” will be determined not simply through an assessment of the scholar’s own production but through an assessment of her reviewing practices. Reviews might, for instance, be rated on numerical scales that measure both their incisiveness and their helpfulness, resulting in a reviewer reputation score. Reviews written by scholars with better reputations would then be accorded more weight in determining the status of texts published through the network.

¶ 9 Leave a comment on paragraph 9 7 The emphasis in MediaCommons’s peer-to-peer reviewing system is thus not simply on being smart, but on being helpful — and I don’t want to underestimate the enormity of that shift; as my Twitter commenter pointed out, little in graduate school or on the tenure track inculcates helpfulness, and in fact much mitigates against it. However, for network-based publishing to succeed, the communal emphasis of network culture will have to take the lead over academic culture’s individualism. Again, this is not meant to paint a rosy picture of a community governed by consensus, in which we all just happily get along, but rather to suggest that our ethical commitment to one another requires an active participation in discussion and debate, particularly as listeners; “helpful” criticism avoids logrolling, but it also avoids snark, instead working to press both author and reader toward a deeper understanding of the questions involved. This open discussion will have to become the primary point of network members’ commitment, placing the advancement of the community as a whole alongside the advancement of their own work; only in that way can both the individual scholar and the field as a whole succeed. In order to promote such a commitment, MediaCommons will need to find a way to implement a pay-to-play system of sorts, requiring community members to become active participants in the network’s review processes in order to take advantage of its publishing capabilities. This might be done by constructing a point system, in which a scholar must earn credits by reviewing, which can then be spent on publishing, but it might also be done by linking the scholar’s reputation as a reviewer to her own published texts, encouraging authors to improve their “karma,” in Slashdot-speak, and thus the rankings of their work as a whole, by publishing more, and more helpful, reviews of texts by others.[1.43]