The following e-mail exchange took place with Wikipedia co-Founder, Dr. Larry Sanger, during February 2006. As one of the leading experts in online, expert, collaborative communities, I turned to Dr. Sanger for his advice with the Community Patent Project. He raised several important critical comments in response to the draft of the Peer to Patent: Community Patent Reform proposal. To push the dialogue and the design forward, we are publishing the interchange here.
Sanger’s comments are numbered with my responses and clarifications interspersed. This provides a supplement to the “questions asked-and-answered” in the Peer to Patent draft proposal.
1. Granted, patent trolls are a nuisance and they should be stopped. But, on the other side, there are some quite serious ideologues who are strongly opposed to patents in general and software patents in particular. How would you propose to keep these ideologues from gaming an open system, thereby seeking to influence public policy at the enforcement level?
Let me state at the outset that we won’t know the answers to some of your insightful questions until we try. The goal of the project is to design a specification and prototype leading to a pilot for open, community peer review. As with any social networking application, there are inevitably going to be unanticipated “social hacks,” evolutions of the technology we cannot fully anticipate. To answer your question, an online peer review system need not be an unstructured system. We have to manage the system to ensure that input is useful and helpful to the examiner. This is addressed, in part, with a rating system to identify who is providing the input. At the same time, we can optimize the inputs by designing the screen to channel input appropriately and minimize the risk of this becoming a “soap box.” In addition, those who oppose software patents and who object to the disastrous state of current patent practice should want to support a system such as this one that improves patent quality.
2. More generally, whenever there is a community—as a collaborative patent review project would certainly produce—there arises an ethos that may or may not gel with the law. What you are contemplating here is that persons not trained in and not necessarily in sympathy with the law are engaging in the crucial discussions and recommendations that lead to legally-binding decisions. This isn’t necessarily a killer objection; but surely there is a huge need to make sure that those persons do not hijack the process.
Perhaps it is not clear from the draft that for many of the reasons that you mention we will preserve the role of the patent examiner as ultimate arbiter. While such a peer review system might, ultimately, be deferred to for final determination of patentability, we should begin by tapping the wisdom of scientific experts for what they know best: science. The examiner continues to make the legal determination based on that science. If the ethos of the community diverges from the law over time, then we will have to change the law. After all, the goal of the patent system is to reward those who have invented something that represents a significant advance over that which came before. The standards we apply are supposed to be those of the scientific community. So if consulting that community leads to a change in patenting practice, so be it. And if the legal rules we have do not give effect to those standards, we will need to change them.
3. For instance, why not think that the community of experts that gets involved will end up regularly ruling that there is “prior art” when, according to the rules more fairly applied, there isn’t? Mightn’t it be necessary to eliminate certain persons based on their failure to “predict” the correct outcome of a case—regardless of their expertise?
We are not asking scientists to be lawyers. We will preserve a role for the patent examiner to translate those scientific assessments into legal ones.
4. Social reputation software on its own cannot solve the problem in §2. Reputation reflects ethos. See Slashdot: granted, it works reasonably well, but what it works to do is to display those remarks that best reflect the leading biases and values of the community. If, then, the ethos of the patent review community does not reflect the requirements of the law, social reputation will not be a reliable indicator of the requirements of the law.
Absolutely right. Inevitably, Slashdot ratings reward the funniest or the best written postings but not necessarily the most accurate. This points to the need for directions and standards for the community. I am contemplating a system that rates both the person doing the posting and the contributions themselves.
5. “…we need to unleash the power of groups of self-identified experts.” The power of the actual experts, not the self-identified experts (a much broader class), is what has to be unleashed. Moreover, the power of those with respect for the law, too. Moreover, I have no doubt that this can be done.
We may diverge on this point. While we do need objective measures of expertise, these should not be based merely on U.S. News and World Report’s rankings of the school where a person teaches. In other words, academics may be easy to rate and rank but that is no indication of true expertise. There are amateur hobbyists with deep knowledge as well as professionals without grants or honorary degrees who are the leading experts in particular arenas. We can learn from ways in which litigators seek expert witnesses and go beyond narrow criteria of expertise as measured by one’s scientific peers, not only by lawyers.
6. Don’t think that expertise cannot be identified by objective measures. Mutual evaluations could be one metric, but I suspect that a formula based on educational background, number of patents awarded, number of publications, and specific recommendations (numerical awards) by specific trusted individuals will be much more reliable. Parts of applications can be posted publicly as well (as CVs are posted) and made subject to peer checking. (I see that you've later said much the same.)
Agreed. We should experiment with a combination of objective and subjective metrics. Need to figure out if there is useful data that can be scraped to automate some of this process.
7. One of the more exciting aspects of this is that we can retain our current patent examination staff and use them to manage the larger project. The role of current patent examiners must be considered carefully. In this way the project can differ extremely from Wikipedia and from the Digital Universe and other projects: it already has a large, trained staff.
Yes, yes, yes! And that staff has deep expertise. This makes patent examination a promising avenue for experimentation with the construction of large-scale social decisionmaking networks. It also gives hope for the idea of public involvement in governmental decisionmaking whereby we rethink the nature of bureaucratic expertise as something collaborative, joining scientific networks of fact-based expertise with legal decisionmakers rendering legal judgments.
8. This part needs considerable elaboration, I think: “In fact, overworked, unelected and unaccountable bureaucrats subject to a constant barrage of lobbyists make the rules. Even absent the problems of regulatory capture and political influence, the sheer volume of decisionmaking about complex scientific subject matter renders it challenging to make decisions in the public interest."
Well, this section could do with some re-writing, that’s true. The point is that centralized and insular bureaucrats are no longer in the best position to make decisions in the public interest. Science still matters.
9. "Patents are consequently both stronger, easier to get and more likely to be upheld without any concomitant guarantee of their quality. This has led to a system that sometimes rewards invention at the expense of innovation and has generated tremendous uncertainty with regard to the role patents play in the marketplace." What does this latter sentence mean, exactly?
In other words, the inventor in his garage or lab may receive the boon of the patent monopoly but the “useful arts,” as the Constitution puts it, are not advanced and social welfare is not promoted. The Constitution demands that society benefit from the reward to the inventor and that is not necessarily the case.
10. "Participants in a peer production system, unlike in a firm or other hierarchical organization, need to be able to self-assign their tasks or roles in the group based on their own estimation of expertise. Far better for me to designate what I am good at since I am in the best position to have this information." It seems to me that you are confusing two issues here. On the one hand, I would agree that it is absolutely necessary for people to be able to self-assign and to have open access to discussions throughout the patent system. On the other hand, it is not clear that just anyone should be let into the system; there is nothing wrong, in principle, with requiring commenters to have some sort of minimum requirements (though I'm not saying exactly what they should be). Moreover, there can be large categories of patent review, each with its own benchmark requirements for participation. In short: there is no conflict between openness or self-assignment within a system and having a (to some extent) closed system.
This is useful and important feedback. While the Patent Office should receive suggestions of prior art from whoever wants to propose it, the proposer should be identified. Secondly, with regard to advising the Patent Office as to obviousness and the ultimate determination of patentability, there should be gating mechanisms and way to ensure the statutory standard of judging based on the “person having ordinary skill in the art.” The question is whether the rating and reputation system is enough to accomplish this.
11. By the way, when I say that the system should be "closed" I mean only that for those not signed in it should be read-only—not that they shouldn't be able to see it at all.
Duly noted. By law and principle, it ought to be transparent.
12. I'm not impressed by folksonomies. The biggest (and well-known) problem with them is that people use different tags for the same things. They are useful personally, but not collectively. So, in this system, what exactly would they be good for? This is an excellent test case, actually; the one paragraph you devote to folksonomies does not clarify exactly what users would do (in terms of tagging) or what the tags would be used to do—and how, precisely, they would succeed in helping anyone do anything.
The Patent Office already has a Patent Classification system. Each claim of a patent is tagged with the relevant class and sub-class of art. But in order to publish patents and ensure that they are seen by the relevant experts, it strikes me that it is important to group those categories into “tag clouds” to which people can subscribe. Would it be problematical to have things double-labeled? Or to let scientific experts add tags in addition to the standard, Patent Office taxonomy as a way to ensure that other experts see the applications?
13. You really say nothing about how prior art and novelty review takes place, i.e., what tools and processes are used. I think that a wiki-based collaboratively-written summary of the arguments regarding prior art and novelty is a wonderful possible application of wiki technology. This should be supplemented by a moderated discussion forum.
First, I’ve added a section to the paper on the mechanics of the current process. Second, are you proposing that the legal standards/directions for review be supplied as a wiki? If so, we can collaborate with the Legal Information Institute/Wex project at Cornell Law School to build a wiki on patent review. My students would love this! Third, do you have in mind an asynchronous discussion forum along the lines of wikipedia? What’s the subject matter? The standards for legal review? Or discussion of specific applications? Please clarify.
14. "The novelty determination is ideally-suited to peer review because it enunciates a clear goal, requires only minimal participation to answer and lends itself to self-selection on the basis of expertise." You might be overestimating the degree to which people will be good at self-selection. Some safeguards will have to be in place. We should talk about this.
In theory, we should want anything that the public has to suggest as prior art. In practice, however, we need to make the inputs manageable and relevant so as not to overwhelm the examiner. What do you think about having experts rate the prior art? Or having mechanisms to identify the claim to which the prior art is responding? Or publishing a list of proposed prior art so that members of the public can “sign on” to a given piece of prior art and comment on it without adding yet more prior art?
15. "While a patent examiner might have to search for prior art for hours, an expert knows instantly whether an invention is reminiscent of earlier work or avenues of research. Designed right, the software can make participation for a network of scientific and innovation experts clear and easy." This suggestion and thus the whole system assume that, for any given patent, there will be enough participating experts. This seems unlikely to me, because particularly for specialized fields, there are often just a handful of people who know anything significant about them. Wikipedia greatly suffers from this problem. I also have worries along these lines for the Digital Universe. Generally, lots of people will show up to participate (one hopes). But that doesn't mean that there will be enough of the right people.
Agreed. We need a mechanism for one expert to invite another expert.
16. In general, I think Part III is good, but it needs rather more specifics in terms of how you expect it to work—and evidence that it will work that way. In several places I find myself saying, "That sounds nice, but what exactly does it mean?" The devil is definitely in the details when it comes to community design.
This is precisely where your comments are most helpful and why we’re having these workshops. However, even before all the workshops take place, I have already written a new draft which I plan to post in the next week.
I have a very nice Community Patent-Review System up and running for about one year now. Maybe you want to take a look at it to get some inspirations for this project. (I only have german and european patents though, but it should work to give you the idea)
Jan
PS: it's all GPLed, so feel free to reuse my complete system if you like ;-)
PPS: http://prevalent.de?lang=en
Posted by: Jan Kechel | March 17, 2006 at 01:10 PM