the reputation economy

Kathleen Fitzpatrick

30 August 2009

¶ 1 Leave a comment on paragraph 1 3 The notion of “peer-to-peer review” that I have been circulating in talks and articles for the last couple of years draws upon the convergence of the kinds of discussion many scholars would like peer review to produce and the decentralized peer-to-peer networks that have sprung up across the Internet. In fact, just as Biagioli suggested a shift, across the early modern development of the scientific academy, in the definition of the term “peer” — from a member of the royal court to a scholarly colleague — so Chris Anderson has argued that the term is once again being redefined in online communities: “In the Internet age, ‘peer’ is coming to mean everyman more than professional of equal rank. Consider the rise of peer-to-peer networks and the user-created content of ‘peer production,’ such as the millions of blogs that now complement mainstream media” (Anderson). Anderson uses this transformation in the notion of a peer to suggest that the academy might fruitfully find ways to open its review processes to “the wisdom of the crowds,” allowing new models of authority in online information distribution to augment more traditional review systems. For instance, Anderson’s reading of Wikipedia contradicts many of the conventional academic assumptions about the project, calling it “not so much anti-elitist as… ‘anti-credentialist’,” a distinction that indicates that site editors’ “contributions are considered on their merit, regardless of who they are or how they became knowledgeable. If what they write stands up to inspection, it remains; otherwise it goes” (Anderson).[1.29] Such systems of communal knowledge-production are thus far from the free-for-all that many have assumed — and, in fact, are at least in theory bringing into being a new mode of authority production; those editors whose work consistently “stands up” to community inspection may be accorded a kind of clout within the community that then affects assumptions about their future work.

¶ 2 Leave a comment on paragraph 2 0 I say “in theory” because one of the most important criticisms that has been leveled at Wikipedia is its acceptance of anonymous contributions, which hinders the ability of readers to assess particular Wikipedians’ work based upon their reputations. Reputation in this sense should be understood as separate from credentials; the point is not whether a particular Wikipedia editor has a degree in the appropriate subject area, but rather whether his or her work on the site has repeatedly stood up to community scrutiny.[1.30] There is, of course, no small irony in the fact that the academic outcry against the anonymous nature of much of Wikipedia’s production occurs at the very same time that we cherish our own anonymity as peer reviewers, and we might take the implications of this contradiction to heart.

¶ 3 Leave a comment on paragraph 3 0 In a recent experiment with community-based peer review, Noah Wardrip-Fruin published the manuscript of his book-in-progress, Expressive Processing, in a CommentPress-based format on his co-authored blog, Grand Text Auto, seeking review from the community of GTxA’s readers, at the same time that MIT Press sent the manuscript to traditional anonymous peer reviewers [see screenshot 1.3].

¶ 4 Leave a comment on paragraph 4 0 Though a number of articles, including most notably one in the Chronicle of Higher Education, represented this experiment as a “head-to-head” competition between open and closed peer-review systems,[1.31] Wardrip-Fruin was clear that such a contest was not his goal. The important aspect of the experiment was in getting feedback from a community he trusted:

¶ 5 Leave a comment on paragraph 5 0 In most cases, when I get back the traditional, blind peer review comments on my papers and book proposals and conference submissions, I don’t know who to believe. Most issues are only raised by one reviewer. I find myself wondering, “Is this a general issue that I need to fix, or just something that rubbed one particular person the wrong way?”…

¶ 6 Leave a comment on paragraph 6 0 But with this blog-based review it’s been a quite different experience. This is most clear to me around the discussion of “process intensity” in section 1.2. If I recall correctly, this began with Nick’s comment on paragraph 14. Nick would be a perfect candidate for traditional peer review of my manuscript — well-versed in the subject, articulate, and active in many of the same communities I hope will enjoy the book. But faced with just his comment, in anonymous form, I might have made only a small change. The same is true of Barry’s comment on the same paragraph, left later the same day. However, once they started the conversation rolling, others agreed with their points and expanded beyond a focus on The Sims — and people also engaged me as I started thinking aloud about how to fix things — and the results made it clear that the larger discussion of process intensity was problematic, not just my treatment of one example. In other words, the blog-based review form not only brings in more voices (which may identify more potential issues), and not only provides some “review of the reviews” (with reviewers weighing in on the issues raised by others), but is also, crucially, a conversation (my proposals for a quick fix to the discussion of one example helped unearth the breadth and seriousness of the larger issues with the section).

¶ 7 Leave a comment on paragraph 7 1 In the end, he notes, “the blog commentaries will have been through a social process that, in some ways, will probably make me trust them more” (Wardrip-Fruin). Knowing the reviewers’ reputations, and seeing those reputations as part a dynamic process of intellectual interaction, produces the authority of the comments, and will thus affect the authority of the book that Wardrip-Fruin finally publishes.

¶ 8 Leave a comment on paragraph 8 0 Given this, we might begin to posit an intimate relationship between reputation and authority in the intellectual sphere. This relationship has of course long existed within the academy, manifested in our various mechanisms of assessment and review, but digital networks give us new modes of determining reputation, as well as new requirements for such reputation-determining metrics. Not all networked publishing structures are concerned with reputation, of course: Wikipedia, for instance, only makes tangential use of a reputation-based system in assessing the authority of its entries. Other systems, most notably online retailers such as eBay, rely heavily on customer feedback in evaluating the reliability of service provided by individual merchants within the network. And the news and discussion forum Slashdot, most famously, uses a system of rating contributions to assess the reputations of individual contributors.

¶ 9 Leave a comment on paragraph 9 2 The Slashdot system evolved out of a more traditional system of comment moderation, in which 25 people weeded out the nonsense and highlighted the valuable; when the work became too much for those moderators, they selected 400 more moderators based on the reputations they’d developed as users of the site. However, this hierarchical moderation system, in which some users had power that others didn’t, quickly led to abuses, and the site’s owners began developing what they refer to as a system of “mass moderation.” In this system, nearly every active contributor to the site has the potential to receive, for a period of time, a degree of power to rate the site’s contributions, through being given a number of “points of influence”; each time the contributor rates a comment on the site, he expends one influence point.[1.32] These influence points expire rapidly if unused, and contributors cannot rank comments in threads in which they actively participate, thus preventing influence from becoming a currency within the system, and preventing moderators from controlling the discourse. The power to moderate, moreover, is only granted by the system based upon the contributor’s “karma” within the site, based upon the ways that the contributor’s own comments have been moderated, which is understood to be a community-based assessment of whether or not the contributor’s comments have been a helpful, positive addition to the community.[1.33]

¶ 10 Leave a comment on paragraph 10 0 There are, of course, weaknesses in a reputation system such as Slashdot’s, in which the value of a user’s contributions to the community can become subject to manipulation and attack, potentially replacing substantive discourse and engagement with a networked popularity contest. As noted by one user of Advogato, an online forum for free and open-source software developers,

¶ 11 Leave a comment on paragraph 11 0 If you believe that “in any sufficiently large crowd, the majority are idiots,” then this can be applied to Slashdot moderators too. All moderators have equal powers and the system is supposed to work as a kind of democracy. But if the majority does not think very much about what they are doing (because of lack of time, lack of interest, lack of intelligence or many other reasons), then it becomes easy to abuse the system…. I hope that something similar to the trust metrics implemented on Advogato could help. (Quinet)

¶ 12 Leave a comment on paragraph 12 0 Advogato’s “trust metrics” are intensively computational, evaluating each “node,” or user, within the network via its interconnections with the network’s many other nodes, certifying each node through three levels of trust (apprentice, journeyer, and master). One of the benefits of this system, as its developer writes, is its “resistance to catastrophic failure in the face of a sufficiently massive attack” (Levien). Reputation, in this implementation, cannot be hacked; on the other hand, it is entirely objectively calculated, leaving little to no room for subjective judgment.

¶ 13 Leave a comment on paragraph 13 0 However, while such “trust metrics” might seem inappropriate as a model for reconsidering peer review, they may nonetheless help point us in the direction of a more sophisticated, partially computational, partially review-based system for determining authority in networked scholarly publishing, the kind of model Michael Jensen imagines under the rubric of “Authority 3.0.” Such a system, whatever its particulars, must operate in accordance with three key principles. The first is that it must be as non-manipulable as possible, preventing the importation of in-group favoritism, logrolling, and other interpersonal abuses from traditional peer review into the new system. Second, the system must achieve a critical mass of participation, and thus will need to operate within an ethos of “quid pro quo”; in contrast with Slashdot’s system, in which users earn the right to become reviewers by publishing within the system, scholars must earn the right to publish within these new electronic publishing networks by actively serving as reviewers. And finally, and most significantly: the key activity of such a peer-to-peer review system must be not the review of texts, but the review of the reviewers. It is the reviewers, after all, that a reader within such a network needs to trust, and as Jonathan Schwartz, the COO of Sun Microsystems, has argued in numerous interviews, “trust is the currency of the participation age.”[1.34]

¶ 14 Leave a comment on paragraph 14 0 It’s no accident, of course, that trust is here defined through an economic metaphor; while the “currency” that reputation affords within the academy is far less spendable than is that within the corporate world, there’s nonetheless an economic reality at its root, and thus at the root of the peer-review mechanisms through which reputation is currently granted. Print-based publishing operates within an economics of scarcity, with its systems determined in large part by the fact that there are a limited number of pages, a limited number of journals, a limited number of books that can be produced; the competition among scholars for those limited resources requires pre-publication review, to make sure that the material being published is of sufficient quality as to be worthy of the resources it consumes. Electronic publishing faces no such material scarcity; there is no upper limit on the number of pages a manuscript can contain or the number of manuscripts that can be published, or at least none determined by available resources, as the internet operates within an economics of abundance. We might think, for a moment, of Cory Doctorow’s “Whuffie,” in Down and Out in the Magic Kingdom, a currency of sorts that measures the esteem one is held in, a system designed specifically for an economics of abundance.[1.35] As Doctorow explained in an interview, Whuffie becomes important in the digital sphere precisely because such a sphere “isn’t a tragedy of the commons; this is a commons where the sheep s*** grass – where the more you graze, the more commons you get” (Tweney). Such is the abundance of the internet, and given this abundance, imposing artificial scarcity through a gatekeeping model of peer review makes little sense.

¶ 15 Leave a comment on paragraph 15 1 However, in a self-multiplying scholarly commons, some kind of assessment of the material being published (or having been published) remains important, but not because of scarce resources; instead, what remains scarce are time and attention.[1.36] For this reason, peer review needs to be put not in the service of gatekeeping, or determining what should be published for any scholar to see, but of filtering, or determining what of the vast amount of material that has been published is of interest or value to a particular scholar. As Clay Shirky has argued, “Filter-then-publish, whatever its advantages, rested on a scarcity of media that is a thing of the past. The expansion of social media means that the only working system is publish-then-filter” (Here Comes Everybody 98). In using a computational filtering system, of course, the most important thing to understand is its algorithm – what criteria, in what balance, it’s using in making decisions to include or exclude various pieces of data.[1.37] Similarly, in using a human filtering system, the most important thing to have information about is less the data that is being filtered, that the human filter itself: who is making the decisions, and why. Thus, in a peer-to-peer review system, the critical activity is not the review of the texts being published, but the review of the reviewers.