| • |
|
|
|
|||
|
||||
John Caron caron@ucar.edu
University of Colorado, Boulder
Dec. 15, 1997
The Internet is useful because it is a wide area communications network. Demand for this global network has exploded in the last few years into exponential growth and frenzied commercial development in every sector of the economy: business, government, educational, entertainment, and home. Sustained long term economic investment and commitment to the Internet is a reality.
There are several obvious functions that the Internet provides that prove compelling: electronic mail is the prosaic example that is the most useful current application. In the future, on-line commerce, especially between businesses, is likely to be a prime motivation for investing in Internet technology. These two examples provide faster and more convenient versions of already existing services. In contrast, wide-area collaboration, the initial motivation for the Internet [Berniers-Lee94], is not something that can be done within any other existing infrastructure. While video conferencing or even phone conferencing is possible, these require all participants to be available simultaneously, and become ineffective for more than a few participants.
In an exact parallel to wide-area collaboration, the growth of on-line communities has been made possible by the Internet and its precursors. The ability of geographically distant individuals to form social groups around common and perhaps narrow interests may be the most important sociological impact of the Internet.
This rapidly growing infrastructure has the possibility of making unique and profound contributions to human culture in the areas of group formation, collaboration, and knowledge sharing, among others. Despite the cacophony of media hyperbole, derisive comments on the shallowness of web content, frenzied and uninformed market investment in Internet companies, and general public confusion, the potential of this technology deserves sober attention.
In this paper I explore an idea for a Web-based wide-area collaborative
application for capturing and structuring knowledge. Wide-area collaboration requires that
interactions be asynchronous, and that will be the focus in this paper. Part two presents
further motivation and context. Part three contains summaries of related work. Part four
presents elements of the application requirements and high level design, including
proposed user roles. Part five has some modest contributions to engineering and
implementation issues, and part six summarizes and concludes with thoughts on future work.
"Wicked problems" are characterized as those in which the problem itself is not easily defined or agreed upon, and therefore there is no clear way to judge solutions [Rittel73]. These "require complex judgments about the level of abstraction at which to define the problem" [Shum97] and often involve moral or value judgments around which people disagree. Rittel concludes that they can only be tackled through an "argumentative" method. Shum asserts that "the fundamental way in which we tackle such problems is to discuss them. Consensus emerges through the process of laying out alternative understandings of the problem, competing interests, priorities and constraints."
Many, if not most, political and societal issues fit this characterization. As a motivating example, suppose that we wanted to estimate the "true" ecological costs of the products we consume, along the lines of Paul Hawken's proposals in "The Ecology of Commerce" [Hawken93]. To do so well, we need to understand the production process, the inputs, the way the product is used, the cost of disposal or recycling, etc. There needs to be long discussions on how to value any of this: some will try to reduce it to dollars, others will have less concrete metrics. There are a huge number of technical details from industrial and process engineers to assimilate, but the philosopher, ecologist, economist, etc. all have to be involved.
As a second motivating example, and a paradigmatic context to start from, consider Usenet, the Internet system for broadcasting text messages around the world. The messages are directed to topical newsgroups and are replicated by netnews servers across the Internet. Messages are kept by the servers for an average of 2 weeks for access by newsgroup readers before being discarded. Current estimates of total Usenet readership are hard to make accurately, but DejaNews, which archives most of the mainstream groups, estimates 24 million users, and reports that they collect 730,000 messages daily from 50,000 newsgroups, comprising 5 Gbytes of data [DejaNews97]. Furthermore, two thirds of the messages are "spam-related", meaning off-topic advertisements posted to many unrelated newsgroups. A number of systems to filter unwanted messages have been proposed and are being experimented with, some of which are outlined below. Web-based display front-end applications have also been implemented to provide more permanent archival of threaded discussions within newsgroups or email groups [HyperNews].
While some Usenet groups focus on political and social problems and values, they function as social and discussion groups rather than problem solving groups. However, in domains that are rapidly changing, such as technology, Usenet groups act as a very important source of up-to-the-minute information and opinion. What is striking is the presence of many knowledgeable users in these kinds of newsgroups, and their willingness to contribute valuable insights and advice. But other than the sporadic maintenance of a list of frequently asked questions ("FAQs"), there is no systematic capturing of the information that "streams" through a newsgroup and is discarded a few days later. The decision of what is worth saving and how to make it accessible in the future is value laden, and must be weighed against the effort of implementation. There are currently no tools to minimize the costs of those efforts.
Finally, the development of the Linux OS is a compelling example of wide area collaboration. Eric Raymond offers an important analysis of the history, psychology, and the lessons learned from this project in The Cathedral and the Bazaar : "Linux was the first project to make a conscious and successful effort to use the entire world as its talent pool... while coding remains an essentially solitary activity, the really great hacks come from harnessing the attention and brainpower of entire communities." While Raymond focuses on lessons about developing software, most of what he says is directly applicable to any wide area collaboration: "Linus (Torvald) was keeping his hacker/users constantly stimulated and rewarded -- stimulated by the prospect of having an ego-satisfying piece of the action, rewarded by the sight of constant (even daily) improvement in their work. Linus was behaving as though he believed something like this: Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone... The cutting edge of free software will belong to people who start from individual vision and brilliance, then amplify it through the effective construction of voluntary communities of interest." [Raymond97]. Admittedly, developing software even as complex as an entire operating system probably does not qualify as a "wicked problem". Nonetheless, the Linux project is a stunning example of the power of collaborative effort to produce extremely sophisticated and highly praised results, given the right tools and social conditions.
Here then are two problems which I think have interesting similarities and in fact
are related: value-laden "wicked" problems, and the capture of current knowledge
of a rapidly changing domain from a stream of comments that vary widely in content and
authority. In both cases, a wide diversity of opinion is allowed and even encouraged,
there is no controlling authority, and anarchy can be seen as an asset, not a liability.
These problems require, by their nature, collaborative solutions.
The design of an application to support wide-area collaboration can draw on the results from a very wide range of research and existing development, from the fields of CSCW (computer-supported collaborative work), CMC (computer mediated communication), groupware, user interface design and others. I present here some summaries of only a selection of relevant research and available systems.
3.1 MIT's Open Meeting System
Roger Hurwitz and John Mallory developed an asynchronous collaboration tool for Vice President Gore's "Open Meeting on the National Performance Review" to organize and structure comments of over 4000 participants on national policy [Hurwitz95]. They allowed comments to be linked to evolving hypertext documents with semantics of Agreement, Disagreement, Question, Answer, (propose an) Alternative, Qualification ("yes, but''), or (report a) Promising Practice. There were certain restrictions on the "link-type grammar" (rules that specify the ways in which comments can be linked based on the link type), such as disallowing comments on parts of the document considered as given, disallowing alternatives to alternatives, and disallowing alternatives or qualifications to a question. HTML pages were dynamically generated to show a simple indented list of hyperlinks to comments about a particular document. All comments were reviewed by a human moderator, who could accept, reject, defer to another moderator, or return a comment for revision. A text search facility was provided to users.
"The key idea is to reduce the volume of communications and increase the locality of communications in order to match information processing levels with people's ability to cope with complexity and with their commitment to the collaboration." Here, locality should be thought of as "specialization by interest, role or function" rather than geographic locality. They also point out the need to reduce dataflow by recognizing and removing redundant information, which becomes unmanageable as data volume increases. Structured annotation and assertions can make otherwise opaque comments capable of being automatically organized into traversable hypertext and semantic networks.
Message classification and taxonomy based on a domain theory allows clustering of information to match the needs of individual users. They also implemented bi-directional typed links as first-class database objects, turning the database into a semantic network with typed nodes. Moderators rated the quality of comments as low, average, high, or exceptional, although its unclear how those ratings were used. Documents could be organized by Boolean combinations of categories, which were used by moderators to create task queues for themselves and others.
"The success of the Open Meeting demonstrates the importance of taxonomic decomposition and meaningful link types in the organization of wide-area collaboration. The meeting showed that people can use typed links which they understand to create argument-structured discourse in a policy planning situation." They suggest next developing link grammars for decision processes, and for knowledge formation in scientific communities.
3.2 WEB4GROUPS
WEB4GROUPS is a European Commission funded project to build a web-based group-cooperation application . It supports web, email, and netnews messaging, along with "value added" services like joint editing, multilingual support, voting and rating support, audiotex and fax gateway. It creates "forums" that may be open or restricted to members, moderated or not, and allows annotation of any web page. It has a distributed architecture: users connect to a local server and activities are automatically replicated to any server with an active member of that forum.
3.3 Collaborative Argumentation
These are systems which focus on structuring discourse to allow computer support for group communication.
Simon Buckingham Shum (Knowledge Media Institute, The Open University, UK), does research on notation and tools to support argumentation, as a way of capturing organizational knowledge and design rationale [CSCA]. His work builds on the argumentative methods developed by Rittel [Rittel72] and Conklin and Bergman [Conklin88] called IBIS (Issue-Based Information Systems). He argues that "wicked problems" require opportunistic ideas and insights not overly constrained by precise semantic typing of ideas and their relationships, and that hypermedia is an ideal technology for capturing this type of partly formalizable knowledge [Shum97]. An alternative to IBIS is QOC (Questions, Options and Criteria), developed at Rank Xerox Cambridge EuroPARC, which "allows you to represent different design solutions within a graphical 'design space', and debate how they trade off against each other" [Shum97].
QuestMap [QuestMap] is a commercially available hypermedia system based on IBIS, supporting workgroup decision making. IBIS has as its primitives objects Questions, Ideas (proposed resolutions of a question), and Arguments (opinions for or against an idea). This structuring of the conversation prevents some of the habits of verbal argument such as not formulating the question before an answer is ready, and confusing ideas with a particular argument. QuestMap adds semantic links between objects: Specializes, Challenges, and Expands-on. Other object types are Notes (annotations), References (links to objects not in QuestMap), Decisions, and Views (graph representations of a conversation).
QuestMap distinguishes three stages in the process of resolution or decision making, called divergence, convergence, and decision. Divergence is the phase when the problem is freely explored and alternative points of view gathered. Convergence focuses on clarification and a narrowing of opinion. Decision is when solutions are agreed upon to a set of questions. Since decisions are often revisited, capturing the range of viewpoints and rationales allows decisions to be revisited without starting from the beginning. To support convergence and decision, QuestMap supports a form of voting called endorsement, and allows Ideas to be retired.
There are a number of rules that are offered in support of effective dialogues. These are not enforced or embodied in the ontology, and so constitute meta-rules of the system. In brief, these rules are that a statement of a question: 1) should be simple and concise, 2) should be open, so that any number of possible solutions can be offered as ideas, 3) should not contain elements of any potential solution or answer, 4) should not contain the word "other", as in "What other ...?", and 5) should not ask for the benefits or disadvantages of a situation or solution.
QuestMap designer Jeffery Conklin proposes that a tool to support "organizational memory" should provide a short-term, operational high-speed data store on a user's desktop, with fast transfer and indexing to retrieve subsets of information from the permanent, long-term storage that constitutes the organizational memory/archive [Conklin88]. QuestMap can be used in both synchronous and asynchronous mode. The details of its user interface are worth studying, although it doesn't obviously scale well to large groups.
Dan Suthers and coworkers at the University of Pittsburgh developed Belvedere, a prototype for teaching high school students collaborative reasoning and argumentation [Suthers95]. Belvedere provides a diagramming tool for "constructing representations of the logical and rhetorical relations within a debate". Objects have graphical shapes reflecting their epistemological status: Theory, Hypothesis, Fact, Claim, Principle. Links between objects have semantics such as supports, conflicts with, implies. Colors are used to distinguish viewpoints. Line thickness is used to express importance. The tools is used synchronously by groups of 2 or 3 students sitting together, with off-line discussion. They found evidence that "a strength of collaborative work is that alternate hypotheses and multiple perspectives may be considered" and also that "In an effort to be politely consensual throughout, a cooperative pair or group may fail to make the most of individual knowledge and judgment."
The Coordinator is a commercially available group communication tool [Flores88], based on "speech act theory" [Searle75], basically a form of structured discourse, with typed messages such as request, decline, promise, commit, etc.
3.4 Annotation
Annotation can be thought of in a general way as any system that allows third parties to add value to on-line content, such as comments, ratings, or links. The salient feature is that this information is provided and managed by someone other than the original document owner.
ComMentor, part of the Stanford Digital Library Project, uses servers and dynamic web page synthesis to construct virtual documents [ComMentor95]. In-line comments can be attached to any web document, using "string position trees" to maintain their document position even when the document changes. Hypertext links can be added for help in navigating: landmarks are reference points, tours are lists of links associated with a reference page, and trails provide multiple tours through related material. Dynamic page generation allows many user choices without creating confusion. Seals of Approval (SOAPS) are annotations which provide third-party ratings of web pages. These can be used to filter or recommend web content. Annotations are grouped into sets, which have world/group/owner read/write access controls, and users can activate or deactivate sets.
Other available services that allow annotation in some form include:
Aqui, from IBM, allows users to add and view links to any web page. You can also specify what type of user would like the link you are adding, which allows collaborative recommendation.
NetQ provides a facility for asking questions of authors of collected papers, and providing web access to accumulated questions and answers.
Early work on group annotation was done in Mosaic 2.1, and HyperNews; there was originally an annotation server at NCSA that has since been withdrawn.
3.5 Recommender Systems
Recommender systems make recommendations of web resources. These can be tailored to the requester's preferences, or be generic, and be based on implicit criterion (mentioning of a URL, time spent by a user reading a newsgroup message), or by an explicit rating by a user.
Fab, part of the Stanford Digital Library project, combines content-based (find items similar to those a given user has liked in the past) and collaborative (find items liked by other users who are similar to this user) recommendations [Fab97]. Fab starts with a small number of topics automatically generated to "track the changing tastes of the user population". Users ask for recommendations, are shown web pages and then asked to assign ratings on a scale of 1-7. Weights are assigned to the keywords from these pages, and this "relevance feedback" is incorporated into the user's profile. Fab can then tailor future recommendations for this user to the user's evolving profile, using comparisons of the content of new pages to the user's profile, and by computing distances between users and then using pages that were liked by close neighbors. Since the web page collection process is shared by all the users, this system scales well, since it is proportional to the number of topics, not the number of users. The success of the system depends "on the ability of the collection agents to specialize and learn profiles which do indeed represent areas where user's interests overlap", which then provides a good starting set of pages, which keeps people interested so that enough user ratings and relevance feedback are generated to sustain the process.
PHOAKS (People Helping One Another Know Stuff) attempts to automatically recognize recommendations of Web resources (URLs) contained in Usenet, by analyzing the text of Usenet postings [PHOAKS]. Usenet messages mention URLs 23% of the time; the designers claim to be able to extract which of those are recommendations with 88% precision and 87% recall. The system then provides Web pages listing the ranked recommendations from each newsgroup. The system recognizes role specialization in Usenet: "only a minority of people expend the effort of judging information and volunteering their opinions to others". They focus on (human generated) FAQs to validate their analysis and enhance the recommendations. For the future they plan on "exploring the issues of how to compute the credibility of recommenders and affinity between those who offer and those who seek recommendations".
Referral Web, from the AI Research Department at AT&T Labs, uses the co-occurrence of names in close proximity in Web documents to construct a graph of the "social network" of web users [Kautz97]. It uses AltaVista to do a search of the full Web. This allows searches based on connections to known experts in a field, to search for experts, to discover how distant two individuals are, etc. The system thus provides "referrals via chains of named individuals".
GroupLens, now a project at the University of Minnesota, attempts to provide collaborative recommendations of Usenet articles [GroupLens]. It provides modified newsgroup clients that allow readers to rate articles and receive predictions based on matching similar users. "Users generally consider only 5% to 30% of articles in typical newsgroups to be desirable". Along with the high volume of news, this implies that the value of correctly filtering uninteresting articles is high. The large number of Usenet readers implies that the value of interesting articles is also high. There is a need for article ratings almost immediately upon posting: a one-day delay means no prediction for about half the users. A major problem is with sparse ratings, since "efficiently reading high volume groups requires being highly selective", many articles are not rated at all. Scaling issues can be handled by partitioning the servers by newsgroup, and by clustering similar users. Also, "composite users" can be defined, which reduces the computational burden of examining lots of other users looking for a match. In the future the group will examine compensation systems, and implicit ratings based on time spent on an article, whether it got a response, was saved, etc.
Dozens of other recommender systems have arisen in the last few years, including: Alexa, which adds a background process to track web navigation, and measures time spent on pages visited. From this information, it derives recommendations of where users might like to go next. SiteSeer from Imana recommends web pages based on matching bookmarks with other users. Intel's Cool URL Recommender adds a browser button which users press when they want to recommend a web page; popup recommendations are also available based on what similar users have liked. Intel also offers Smart Newsreader, a newsreader that lets you rate news messages, and gives you predictions based on content-based filtering. Domain specific recommendation systems include Each Movie from DEC and Morse from British Telecom for movie recommendations, and Tunes for music recommendations.
3.6 Usenet filtering systems
NoCem ("no-see-em") was developed by the anti-spam group Cancelmoose. Anybody may issue a NoCem notice to cancel a Usenet posting. Individual users and/or news administrators decide whether to accept the notices from a particular person, based on their "net reputation". Specially enabled newsreaders delete or mark those messages as unread. Articles can also be marked as recommended: by asking to see only these articles, a user allows a NoCem "rater" to act as a moderator. All messages are PGP encrypted to ensure authenticated users. A working group of the Internet Society is considering format changes to newsgroup messages that may support this kind of filtering directly.
Spam Hippo, written by Zippo News Service, is software that runs as part of a netnews server, and is designed to automatically remove Usenet spam. The program identifies spam based on the number of identical postings or cross-postings.
DejaNews is a commercial system that archives most articles posted to the mainstream Usenet groups. They estimate that as much as two thirds of Usenet messages are spam related. They have recently (12/1/97) begun filtering their archives by using NoCem and Spam Hippo, as well as proprietary artificial intelligence filtering techniques.
Usenet2 is a proposed top-level
Usenet hierarchy (net.*) which requires any postings to those groups to follow rules to
prevent spam. These rules include well-formed messages, real return e-mail address,
identification of the host machine of the article, no excessive cross-posting or identical
postings, no cross-posting outside of net.*, and the banning of binary files. Usenet2 has
preliminary support from Panix Public Access Network in New York, Deja News, Inktomi,
universities such as Stanford and Penn State, and corporations such as IBM, Sun
Microsystems, and NASA.
3.7 Commercial Groupware
While the term "groupware" can generally have a wide meaning [Ellis91], I mean by "commercial groupware" a class of applications pioneered by Lotus Notes. These applications are seen as adding value to corporate intranets, and the major vendors are those competing in that market. I briefly survey this category since these are groupware applications with the most widespread use, and so there will be some user expectations and experience based on these systems.
All these systems provide client/server architectures to provide the basics of email and newsgroup services, as well as web intranet and extranet servers, usually within the context of a work organization. Messages between the members of the group are the basic unit of the workflow. The creation and use of "private newsgroups" within these workgroup settings is a form of shared document databases. Automatic filters and full-text searching can create "virtual newsgroups" that track, classify, and archive the group's workflow. All have integrated Web/Internet access and standard protocols to what were originally proprietary applications.
Lotus Domino is a collection of server processes providing database replicating, SMPT and POP3 mail servers, indexing, web server, and document-to-HTML conversion on the fly. Lotus Notes is the client that provides a document-oriented, flat-file message store, with calendaring and scheduling, and a web browser.
Microsoft Exchange is based on Messaging Application Interface (MAPI), and includes an NNTP newsgroup server, a directory based on the Lightweight Directory Access Protocol (LDAP), and a POP3 mail server, and can access MIIS (Microsoft Internet Information Server) and Windows NT's Domain Name Service (DNS) server. Outlook is the client with integrated messaging, calendaring, contact, and task management application.
Netscape SuiteSpot is the collection of email, web, LDAP, and security servers, including Collabra, an NNTP news server which adds Secure Sockets Layer (SSL) encryption, client authentication, and integrated full-text search to standard news server functionality. Users can add new (private) discussion groups, with names not limited to netnews dotted-name hierarchies. Communicator is the new name for the client web browser/email/news client with calendaring, real-time conferencing, and IBM mainframe access. The calendar server has people and resource scheduling, notifications, proxy users (for allowing someone else to manage your calendar), and personal tasks. Enterprise Server is Netscape's high-end Web server platform, which allows building applications that integrate the features found in Mail, Collabra, and Calendar, and provides a shared filesystem.
Novell
Groupwise has integrated e-mail, including threaded discussions, calendaring and
group-scheduling, task management, and shared document management. Shared folders provide
a repository where multiple users can have discussions or store items relevant to the
group, and access control, versioning, check-in/check-out, and tight application
integration functionality is supported.
My goal is to envision applications that will support the capturing and structuring of knowledge from a widely distributed and likely heterogeneous group of users. The supporting infrastructure for such an ambitious venture is so new that only very recently have we been able to imagine the shape of possible solutions and consider what designs are implementable. Both the benefits and costs of building a successful system are likely very high. The number of ways of making a wrong turn in design and implementation are large, so it is imperative to examine in depth the assumptions, context, and process for creating this system.
Let me start by listing my assumptions about the needs and purposes of such an application:
- The generation of knowledge has to be done by humans for the foreseeable future.
- There are domains where no one person or group can summarize the field, because the domain is changing rapidly or it is a "wicked" problem with conflicting values. A collaborative summary of the state of knowledge/opinion would be useful to many people.
- There are domains that have sufficient number of knowledgeable people who will be willing and motivated to contribute to building these summaries. The value of such knowledge is sufficient to induce organizations to invest their human and computer resources.
- Supporting technology is necessary to create such collaborative summaries; that technology is feasible now or within a year or two
What follows are a number of specific ideas and scenarios that lie somewhere between assumptions, requirements and design. My intention is not to create neat bundles of requirements, designs and implementation decisions as in the classic waterfall method of software development. Rather, I hope to describe a plausible and implementable application that might inspire others to join in a collaborative effort to design the supporting protocols necessary to build some similar system. The actual implementation of such a system might be done through a collaborative engineering effort similar to the Linux OS project. Raymond characterizes the beginning of a collaborative software effort in this way: "When you start community-building, what you need to be able to present is a plausible promise. Your program doesn't have to work particularly well. It can be crude, buggy, incomplete, and poorly documented. What it must not fail to do is convince potential co-developers that it can be evolved into something really neat in the foreseeable future." [Raymond97]
My current understanding of the requirements of such a collaborative system are that:
- It must support the various ways that people want to create meaning and structure out of their discourse. Filtering and structuring is needed to keep the discussion intelligible; hierarchy and summary is needed to orient users and prevent information overload.
- The roles and responsibilities of the system must be distributed among the participants. Tools are needed for dividing the work among many participants. No centralized authority can be assumed. Commitment and abilities will vary widely among people and will change with time.
- The most valuable resource we have is our own time; every effort must be made to make efficient use of it, and to enhance the perception of users that their time investment is worthwhile.
- People are motivated by a combination of peer approval and recognition of their individual contributions, and by participation in a group project with tangible results. The system should support feedback for both kinds of motivation.
- Information needs to be persistent as long as it is deemed relevant.
- There must be a critical mass of people willing and able to participate.
One of the critical pieces of creating software is getting people to use it, known in commercial contexts as marketing. A collaboration system requires a critical mass of users before it can even start to function correctly; therefore I propose to take Usenet as the starting point for design, and position the system as an extension of Usenet culture, infrastructure, readership, and purpose. To emphasize this as our marketing plan, I will provisionally name the system CUsenet for "Collaborative Usenet". I do not mean to imply, however, any implementation strategy such as using NNTP (network news transport protocol).
4.1 Roles
An important way of specifying what functions the system should provide is to create roles that a user may be willing to assume. A user role is a coherent delineation of responsibility, focus, motivation, and integrated system support for an individual user. Role specialization allows supporting tools to efficiently divide the information "into meaningful and coherent chunks that match cognitive capacity and motivational level of participants." [Hurwitz95]. Here I briefly describe each role, and in section 4.3 I propose some designs to support those roles:
Filterer: Read raw postings and remove inappropriate messages, and make certain kinds of message classifications such as question, rating, etc. on messages that are misclassified.
Rater: Evaluate the content of articles on two separate scales: 1) usefulness of the article, and 2) personal agreement with the article.
Contributor: Respond to other articles, add new ideas, comment, annotate, suggest links to other resources. This is what a current Usenet contributor does now.
Discussion Organizer (doer): Structure a discussion by filtering and editing, summarize arguments, add semantic links, highlight important contributions and answers, delineate alternatives, annotate, and generally create structure and organization out of a discussion thread.
Topic Organizer: Similar to a doer, but works at a higher level. Organizes and summarizes related discussions and results for an entire topic.
Site Organizer: Creates a high level view of the entire set of useful results for this newsgroup.
Index Librarian. Add indexing tags to messages to allow search capabilities. Propose tags to be added to a standard, domain-specific set. Propose items to be added to a glossary.
FAQ Librarian. Create "frequently asked questions", and summarize answers. Provide "see-also" links to more detailed and related answers, or to alternative answers or viewpoints. Generally, maintain and enhance FAQs.
Moderator. In Usenet, a moderator is someone who filters noise from a newsgroup, and often responds to messages that are off-topic or otherwise violate Usenet norms. Any rater can play the role of a moderator if the system allows a user to filter messages based on a specific rater's evaluations, rather than on a collaborative scoring of messages.
Sysadmin. A system administrator for a cnewsgroup is needed to perform various administrative and monitoring functions. While this role in Usenet is normally played by a technical person who administers the hardware and software, in theory it could be a non-technical person who has some named rights and responsibilities for the system.
In general, a user may play more than one role, perhaps simultaneously, since many of these roles overlap and their boundaries are not always sharp. For example, filter and rate are obvious tasks to combine, and the system should allow a user to efficiently do both at once. The distinction between the roles of discussion organizer and topic organizer may be mostly useful in the abstract, since discussions often wander all around a topic area.
A very important feature is that roles can be played by more than one, even many, users. This makes the question of ownership and editing rights critical. It is a requirement that multiple viewpoints are supported, so it is possible to have multiple discussion or topic summaries, or even site maps or FAQs. The goal is to capture the diversity of opinion and to allow a user to see the information from various viewpoints, through various explicit and implicit filters. While the focus here is on the collaborative sharing of roles, it is possible that some groups may decide to restrict some of these roles to named individuals.
The set of people willing to assume a specific role is an important subgroup, since it allows certain types of messages to be dealt with only by that group. For example, a discussion on whether a question should be added or deleted from the FAQ could be directed at the FAQ Librarian subgroup. This would constitute a very valuable kind of filtering.
4.2. Definitions and Ontology
A cnewsgroup ("collaborative newsgroups") is a CUsenet group with a specific subject matter (as in Usenet), which may be open to all or be restricted to registered users only. A cnewsgroup has a charter, which states the subject matter and the scope of the group, whether open or closed to readers and writers, as well as any ground rules, restrictions, decisions, or other agreements expected to be followed.
A message is the basic unit of communication to a cnewsgroup. A message has a unique message type and any number of attributes specified as keyword-value pairs. A message stream is a set of messages that are passed to a user. The raw stream is all the messages posted to a cnewsgroup. The filtered stream is all messages that pass the collaborative filter algorithm, which removes spam and other inappropriate articles. A substream is a subset of the filtered stream, consisting of only certain types of messages, for example only ratings, or only FAQs. A rated stream is a set of messages passing some rating criterion, such as those with a usefulness score greater than zero.
Whereas a message has a limited lifetime, similar to Usenet articles, a document is intended to be a persistent object, similar to web pages. Messages that are referenced from a document are given the same persistence as the document. Again following the Usenet and Web analogies, messages once posted cannot be changed (although they can be canceled), while documents may be changed at will.
A discussion thread is all the messages with the same discussion attribute. This is separate from the In-response-to attribute that means that this messages is in response to another specific message. A topic is a general area of interest to the cnewsgroup, created by sending a topic-create message containing the topic name and a description of the question or scope of the topic. Each cnewsgroup has a "general" topic and each topic has a "general" discussion which are the default attributes for a message. Every message is therefore always contained in a hierarchy cnewsgroup topic discussion message.
A user is anyone who reads or writes to a cnewsgroup. A participant is a human person who has registered with some authority, whose messages are authenticated, and who may write documents. A contributor is a human person who has not yet registered, or who chooses not to register; a contributer can write messages but not documents. A contributor is not necessarily anonymous, since they might sign their messages with a real address. A reader is a user who has read-only permission for a group. An agent is a non-human reader, including corporations and software programs. A cnewsgroup that disallows contributors is a private-write cnewsgroup, and if it disallows readers, it is a private-read, or simply private cnewsgroup.
A rating is a numerical score assigned to a message or document by a user. Documents can always be rated, but not all messages can be rated, e.g. a message of type rating cannot itself be rated. Anything that can be rated is called a ratable. The usefulness rating gives the user's opinion on how useful a ratable is, while the agreement rating specifies how much a user agrees or disagrees with it. Ratings have the same persistence as the ratable they are attached to, and can be changed as long as the ratable exist.
A score is a numerical value assigned to a ratable by a scoring algorithm, always a function of the ratable's usefulness ratings, not agreement ratings. A distance is a numerical value assigned to a pair of ratables by a distance algorithm, always a function of the ratables' agreement ratings. Distances can also be assigned among pairs of ratables, users and viewpoints. A ratable may have multiple scores and distances based on different algorithms.
A user's weight is a numerical value assigned to a user by a weighting algorithm, typically a function of the scores of the ratables owned by the user. A user may have multiple weights based on different weighting algorithms. The system tracks the weight only of participants, that is, registered and authenticated users. A viewpoint is a composite user generated by a viewpoint algorithm, for the purpose of analyzing user distances and clustering behaviors.
4.3 System Design for User Roles and Tasks
One of the benefits of creating well-defined roles is to identify the goals of a user when s/he is in that role. Understanding those goals is key to creating a good system design [Cooper95]. Here I examine each role and suggest specific system functions that are required, with an emphasis on the efficient use of the user's time.
Filterer: A filterer needs to view the raw message stream sequentially and to give each a pass/fail rating. It may be feasible to provide text analysis algorithms that could automatically detect inappropriate messages and present them to a filterer for fast confirmation. A filterer also checks message types, and makes corrections if needed, so support for private email to the poster who misclassified the message would be helpful in educating new users.
Rater: A rater needs to view message streams and give individual messages numerical ratings of usefulness and/or agreement. When the filter and rate roles are combined, the user operates on the raw stream; when separated, the rater operates on filtered streams.
While the scale used for rating is essentially arbitrary, something like [-3,-2,-1,0,1,2,3] allows three levels of positive or negative utility/agreement, with 0 meaning "neutral". A separate option of "not rated" must be the default, since "neutral" means that the article was evaluated and the user doesn't know or care.
A more complex function would be to allow the rater to divide a message into sections, and separately rate each piece. There are a number of ways to design this. One is to treat the separated sections as new messages, with a link to the original article for context clarification. The other is to treat the separate ratings as annotations on the original article. This function starts to blur the rater role with the doer role.
Contributor: A contributor needs to write messages that are correctly typed, with message attributes (such as group, topic, discussion, In-response-to) inferred by the system, with optional override. Creating annotations and links to other articles should be as simple as drag-and-drop.
Discussion Organizer: A doer needs to cut and paste parts of messages into a discussion summary document, inline an entire message by reference, or create a typed hyperlink to another message or document. The component messages should be displayed in a way to indicate that they are quotes from other documents, and the system should allow a user to view the original owner's name and a link to the original message. The summary document may be monolithic, or may be a collection of linked documents. It should also consist of nested sections, so that a user can automatically see an outline view of the document. The semantic links among the sections of the document should allow an automatic generation of a graphical representation of the document called a graphical view, or node-and-link view, similar to gIBIS [Conklin88]. The doer may edit any of the views, and the other views are automatically changed, i.e. there is only one document.
Topic Organizer and Site Organizer: These roles require the same functions as a discussion organizer, but generally do not cut and paste or inline documents, although they can add annotation and comment, and general orientation remarks.
The topic organizer creates a topic summary document using references to discussions and adds comments and semantic links between the discussions. A topic organizer is free to include or exclude discussions, and may annotate discussions, but may not modify a discussion. The graphical view of the topic is automatically generated based on the discussions themselves and the links between them.
A site organizer creates a site map document which is a summary of the entire cnewsgroup, using topic summaries and adding links and comments.
Indexer. An indexer creates annotations of a message or document by specifying a word or phrase that indexes its content. These annotations have message type index, and are sent to the index message substream, normally not viewed by readers. An index may refer to a section, or range of a message or document. An indexer should be able to quickly specify one or more index words for a message or document, optionally mark the section that it applies to, and have the system generate the message(s).
Index searcher: An index searcher posts messages of type index-search, consisting of a phrase to be searched for in the index. These messages are automatically answered by the system via a web interface or private email to the index searcher.
Index Librarian: An Index Librarian performs the task of systematically adding indices to a message stream. Since index messages can be sent by anyone, the Index Librarian may also need to filter or rate index messages. Indices are used to build a cnewsgroup search index document, so there might be tools to assist Librarians to monitor the quality of the search index.
There might be a domain specific standard set of indices that the Index Librarian maintains; these might be sorted by frequency of use to allow quick generation of indices with minimal typing (e.g. with keyboard accelerators). There might also be a glossary of terms that the Index Librarian maintains.
FAQ contributor: A frequently asked question is a message of type FAQ, consisting of a question, an answer or a link to an answer, and optional comments or other links of type see-also. Any contributor may post a FAQ, and these messages are sent to the FAQ message substream, normally not viewed by readers. The system stores FAQs for automatic searching in the collaborative FAQ document.
FAQ searcher: A FAQ searcher posts messages of type FAQ-search consisting of a question which is automatically compared against existing FAQs, and the best answers are returned to the user by private email or Web interface. Optionally a message of type FAQ-answer is generated, for use by FAQ Librarians. If the answer is unsatisfactory, a FAQ searcher may generate a message of type FAQ-request which is a request for human help to answer a question.
FAQ Librarian: A FAQ Librarian may filter or rate the FAQ message substream, and may monitor the FAQ-answer and/or FAQ-request substreams in order to improve the automatic FAQ responses. They can annotate FAQs by adding alternate question formulations for the same answer, add see-also links to alternative answers and add for-more-detail links to more detailed answers or discussions. The system should generate periodic statistics of FAQ usage, hits, requests, etc.
FAQ Organizer: A FAQ Organizer uses the same tools as discussion and topic organizers to generate a FAQ summary document, using similar structuring and linking. This is separate from the collaborative FAQ document itself, which is owned and maintained by the system: a FAQ summary document is maintained and owned by an individual. FAQ searches can be generated against one or all FAQ summary documents, in addition to the collaborative FAQ itself.
Reader: A reader can read any of the messages or documents, and post messages of type Index-search, FAQ-search or FAQ-request.
4.4 System Design for Collaborative Features
Many of the design features described so far are straightforward for single user document editing or for small-group settings, but are less clear how they will work in a large-group Usenet-like environment. For example, what does it mean to have multiple FAQ librarians? Who gets to decide what answer gets generated to a question? This section outlines some of the features necessary to deal with these issues.
Co-ops. We have already distinguished several classes of users: readers with read-only privileges, contributors with message writing privileges, and participants who are registered and authenticated, and can create persistent documents. Now we define a co-operating group, or co-op as a named group of participants in a relationship of mutual trust. All members have equal rights within the co-op, and all messages and documents from the co-op are assumed to be written by consensus. Messages and documents are owned by the co-op, but all members have the write privileges to them.
A co-op may give itself a pseudonym, but membership information is publicly and easily available. Formation and changes of membership requires authentication from all members. A co-op is dissolved when any member requests to do so. Membership in a co-op does not preclude or affect participation as an individual (except for scoring, see below).
Other kinds of groups might also prove useful, which share a different mix of rights and responsibilities among its members
Ownership. Every message or document is owned by the user or co-op who created it. Only owners have write privileges for a message or document. Ownership of a document may be transferred to another participant. (It may be useful to define derivations and inheritance relationships among documents, but for now I'll forgo those possible complications.)
Document Ownership. Discussion Organizers are supposed to structure a discussion into an intelligible form, clarifying and focusing the arguments and information. Even with the intention of neutrality, it is impossible in general to structure a controversial discussion in an unbiased way. The question of what is relevant to a discussion is a form of bias. Therefore the ownership and control of the discussion is made explicit; a doer is encouraged to state their point of view as a standard attribute of the discussion. Any participant can create an alternative discussion on the same question. Since discussions are rated by participants, readers can filter what discussions they wish to view or participate in. These comments apply equally to Topic, Site, and FAQ Organizers.
Scoring. Scoring is a way of filtering and ranking the usefulness of information. Scoring does not directly measure agreement or closeness of values of participants, although usefulness ratings will be affected by agreement in many cases. The system tracks only weights for participants, but not of contributors. It might be the choice of a newsgroup to disallow contributors, to give them a weight of zero, or give them a weight of one, and use that as a reference value when calculating participant weight.
Message scores and user weights depend on the scoring and weighting algorithms used, and so there may be multiple scores and weights if multiple algorithms are available. At a minimum, the system always provides the default consensus algorithms, whose intent is to reflect the authority and contributions of participants over the life of the group, using some measure of "majority opinion" like the accumulated weights of participants.
Another suggested algorithm might calculate scores by giving participant X a weight of 1, and weighting other ratings by their owner's distance from X. This algorithm might be called scoring from X's viewpoint.
Messages posted from co-ops are owned by the co-op, and weight derived from those messages accrues to the co-op. Co-op members may not rate postings from co-ops to which they belong. Co-ops themselves cannot rate anything. While these kinds of rules need to re-evaluated in light of possible weighting algorithms, the intent is to be careful about how power can be accumulated by a group.
Viewpoints: The goal of viewpoints is to allow minority opinions to be found. There are three possible aspects to this. The first is to calculate scores based on an individual's ratings, as discussed above. The second is to identify clusters of users automatically using distance algorithms. The third is to allow users to form self-identified clusters by allowing them to claim allegiance with a named viewpoint. The idea is to let users see representations of a site/topic/discussion from any of these viewpoints. A user who finds him/herself agreeing with a viewpoint would add weight to the viewpoint. In this way a minority or even an individual viewpoint could attract attention and support.
Voting. A question can be put to a vote by framing it in a yes/no or numeric range, and specifying the closing date to vote by. Any participant can vote, votes are authenticated, and a participant can vote only once. Results are displayed both weighted and unweighted, and are always public, that is cannot be withdrawn from public inspection. Votes are by default public, so that tallies of the participant's votes (and their weights) are publicly available. Votes may also be declared anonymous, so that a participant's vote is not public. The form for voting has a comment field that a voter can use to annotate his/her vote.
4.5 Message and Link Types .
The classification of messages is critical to the structuring of information. Standard message types should be very easy for users to understand and use to classify their own messages. These might include:
Question for Discussion: Pose a question that you would like to discuss.
FYI ("For your Information"): An (alleged) fact or piece of information.
Response: a response or comment on another specific message or thread. In-line annotations are possible using this message type.
Summary: an attempt to summarize a discussion, question, or issue.
Graph: Node and link graphs may be attached to a discussion, a topic, or the entire site.
In addition, we have already identified a number of specialized messages, that are filtered out of the general message stream such as topic-create, FAQ, FAQ-search, FAQ-answer, FAQ-request, index, index-search, rating, and vote.
Standard link types that should be understood easily by users include: Re (in reference to) and See Also (additional information at). Other semantics that might be used by the original authors, or by discussion organizers might include Agrees, Conflicts, Qualifies, Summarizes, and Is an Answer to. Many others types are possible. It will likely be important to allow groups and even individual users to define their own link types.
4.6 Security
In a controversial cnewsgroup where there may be fear of vote or algorithm tampering or other illegal behavior, the role of the sysadmin must be perceived as neutral, and his/her actions even be monitored and reviewable. This role might be shared by a named group of individuals whose decisions must be explicitly ratified by all members. We might call this a check group or checkpoint, and note it assumes a relationship of mistrust, whereas a co-op assumes a relationship of trust.
It is important that each participant have only one rating and one vote, although participants obviously accrue different weights. Therefore, participants may not assign their authentication to other users for any reason, so a single human person cannot be more than one participant. An agent is a non-human user, with read-only rights. In the future, this restriction might be relaxed under suitable safeguards. It is not yet clear how to prevent participants from registering more than once, or agents from registering at all.
Given the requirement to register and authenticate ratings, it seems likely that scores and user weights can remain free from outside attack. The possibility of saboteurs joining a cnewsgroup to disrupt it, for example by introducing random noise in the ratings, is harder to defend against. A group of saboteurs might be called a cabal; a large enough cabal attacking the filtering or rating substream could be very disruptive. Tools will have to evolve to deal with these situations. A cluster analysis should reveal the participants, and an accusation of sabotage might result in triggering an optional filter that users could enable to remove any messages owned the cabal.
4.7 Miscellaneous design ideas
There is another paradigm from which to design CUsenet: we can think of it as a wide-area, multi-user game. This reflects the centrality of the rating of the articles. Users "score points" by posting articles that are positively rated. Their cumulative score is reflected in their "weight". Exposing this facet of collaborative work is likely to have interesting consequences. Can the game rules be designed to encourage participation and so that the results are useful collaborative summaries of knowledge? There may be valuable insights from the "dungeon and dragons" type adult fantasy games. These games have cumulative scoring (possibly accumulating over many years), create many roles that users can take on, and are accessible to novices even while the rules become very complex for experienced players.
Following the example of various Web services that have proven popular, other
services that might be included are: 1) the ability to do a full text search on the
message stream. 2) buddy list: who is currently on-line. 3) email groups to send private
messages, like "I'll do this and you do that" to minimize collisions among
groups of close collaborators.
Object Implementation
The likely choice for document encoding is with Extensible Markup Language (XML), since this includes all of the functionality of HTML, and has the extensibility required. It seems likely that messages should also be XML documents.
Bi-directional typed links appear adequate to express the various relationships among messages [Hurwitz95], as well as provide the hyperlink navigational anchor. A link then consists of two addresses and a type. From the system level, the link type is simply a short display phrase, an explanatory help paragraph, and a set of rules or grammar specifying what types of messages the link can point to
It should be possible for users themselves to define new attribute tags and link types, with the application automatically incorporating them. Where the semantics of a new type require the application to process or display them differently will obviously require new coding. In an ideal world, user contributed applets (properly authenticated, perhaps distributed only as source code) could add the new functionality.
Client Design
I assume that the look and feel and functionality of Web browsers and news readers is the starting point for the user interface. Whether it is a plug-in/add-on to an existing browser or a separate application is unclear, though I suspect that only a separate application can give the fully integrated functionality necessary to insure optimum efficiency of a user's time. More broadly speaking, subsets of the functionality can be made accessible from many platforms.
The user interface design seeks to make the capture of collaborative knowledge efficient and even enjoyable from the point of view of an individual user [Lewis93]. While the task of capturing the full range of discourse semantics is impossible, I assume that capturing a small but powerful subset of those semantics is feasible. Still, even the limited set of roles and functions outlined in Part 4 will create an impossibly complex and confusing application unless some underlying simplifying implementation principles are found.
From the user's point of view, I theorize that the client manipulates only four different types of objects: message, document, link, and graph. Messages and documents have structured attributes that represent their semantics. Typed links represent the relationships between messages and documents. A graph is a graphical representation of the messages and links between them. Messages and documents may actually differ only in their persistence, and can be composed of any of the objects that a web browser can deal with: text, images, multimedia, etc.
Server Design
GroupLens estimates that for their application, a single workstation can serve the needs of 10,000 users and 10-20 newsgroups. Usenet, on the other hand, has data volume much higher, and so replicates data to thousands of servers. It is likely that a single web server can meet the needs of a single cnewsgroup, at least initially, which would simplify greatly the server design. I theorize that client pull or server push of new messages at a user settable period will be adequate; this is an asynchronous application, after all. However, scaling to large numbers of users or data volume may require a distributed or replicated server design. It will be important to study what limits scalability, and how the user can be shielded from perceived latency.
Collaborative filtering implies that messages go to filterers before being distributed to others. The question is then how long the system should wait for filterers to respond before sending the message on to other users. A possible solution is to allow each client to set their own fetch policy, for example the options might be: 1) immediate push (for filterers), 2) wait fixed time and apply collaborative filter, or 3) wait until algorithm determines some probability of message being good (likely a function of number and agreement of filtering messages).
User Role Support
The filtering of messages based on message type is a simple and effective way of supporting role specialization. While each user role might have a default setting of message types to be filtered, the user should be allowed to override those defaults at the granularity of message type. This allows users to choose their own mixture of roles. The user might also want to separate the roles s/he plays at different times, and so the system should support configuring any number of roles, and allow easy switching between them.
Scoring
The user-visible rules of scoring and viewpoints should have the nature of the rules of a board game. The underlying algorithms might be genetic or otherwise mutable. The scoring system should reward cooperation, without suppressing dissent. Alternative scoring algorithms might be user-contributed, with safeguards like authenticated source code distribution only.
Synthetic datasets should be developed against which algorithms can be tested. One
possible method is to define a synthetic set of messages as "true", then test
algorithms using a stream of ratings with known noise. Another is to create a
"true" set of user weights and viewpoint distances, then simulate those users
sending messages to each other with varying degrees of ratings sparsity, noise, accuracy,
etc. An algorithm can then be characterized by its time evolution towards
"truth". Its possible that there are results from game theory that would be
useful starting points.
I have presented motivation and design goals for a Web-based collaboration application that significantly extends the functionality of Usenet newsgroups to allow collaborative knowledge capture. The key assumptions are that only the human mind can create this knowledge, that a critical number of people in certain domains will be motivated to invest their time to do so, and that the Internet and supporting software such as proposed here are enabling technologies.
It's important to acknowledge that the features presented here have the nature of brainstorming ideas, rather than concepts of proven utility. Many iterations of design and implementation will be needed to make such an application. Nonetheless, all of these features have been inspired by existing systems. Most importantly, all are implementable within existing technology and infrastructure. Of course, how to manage the complexity of a large set of features is always the challenge and art of application design. But my intent here has been to envision a set of implementable features that, taken together, create a tool for collaboratively tackling problems that heretofore have seemed unsolvable.
The first group to use such a tool might be one established to collaboratively design the tool and the protocols needed to implement the tool. That group would learn what works in a collaboration tool by using the tool themselves. By designing a framework from the start to allow rapid prototyping, software components, version control, and source code contributions (possibly from co-ops?), it is possible that the group could rapidly explore design alternatives. In any case, the group's real goal would be to develop communication protocols, message/document formats, and other standards that would enable such an application to be built. The Internet Society might be an appropriate place to establish a working group for this purpose. Implementations eventually would be done by other groups, possibly by commercial companies who want to include some or all of the functionality in their web or groupware applications.
In order to be successful, a critical mass of users must be attracted to use such a system. I theorize that the users of Usenet technology newsgroups should be the target users for a beta version of the application. These users are more focused, highly motivated, and tool literate than average newsgroup readers, and would be tolerant of new and untested systems, sympathetic to efforts to improve the quality of newsgroups, and appreciative of the results of consensual and persistent knowledge bases. Any user will only be motivated to use an application whose value is higher than the cost of learning and using it, so the application must be perceived as making efficient use of a user's time, give positive feedback to those willing to contribute, and produce a result that is valuable. The initial design of the system should focus on this group of users and their needs.
This paper has lumped together features intended to create a "better Usenet" with the more difficult requirements of "wicked problems" and their social conflicts and likely irreconcilable values. My intent has been to emphasize the similarities of these two types of problems, or at least suggest the existence of a continuum between problems that admit to definition and solution, and those that resist consensual formulation. As a design and implementation strategy, however, it might prove useful to delay the complex features needed for wide-area structured discourse until some experience is gathered in deploying a tool for capturing less controversial, possibly technical, knowledge that is undergoing rapid change.
The clearest research need is to develop scoring, weighting and viewpoint algorithms. Synthetic datasets and ways to characterize algorithmic properties need to be developed. Methods of efficiently visualizing information using multidimensional metrics are needed in order to support multiple viewpoints. Authentication of participants as unique and human is also needed, as are other security concerns about deliberate sabotage or rating skewing. Architecturally, the biggest question is how to scale the system to large groups.
Hurwitz and Mallory concluded their work on the Open Meeting collaboration tool by noting that "the World Wide Web offers unprecedented opportunities for wide-area collaboration at a time when nothing less seems likely to cope with endemic and emergent global problems. We have argued that collaboration systems can begin to manage the complexity by supporting the specialization and localization of knowledge, planning and evaluation."
The development of the Linux OS not only demonstrates that the Internet has enabled successful wide area collaboration, but also gives us a blueprint for how to create the software necessary to extend collaborative methods to previously intractable "wicked" problems. The successful development of these collaborative tools may significantly contribute both to the sum total and intelligibility of human knowledge.
Present and future versions of this paper are available at http://acd.ucar.edu/~caron/wa_collab.html
[Berniers-Lee94] Tim Berniers-Lee, Robert Cailliau, Ari Luotonen, Henrik Nielsen, and Arthur Secret, The World Wide Web, Communications of the ACM, Vol 37, No. 8, Aug 1994.
[ComMentor95] Martin Röscheisen, Terry Winograd, Andreas Paepcke, Content Ratings and Other Third-Party Value-Added Information: Defining an Enabling Platform, D-Lib Magazine, August 1995, http://www.cnri.reston.va.us/home/dlib/august95/stanford/08roscheisen.html
[Conklin88] (2) Conklin, J. & Begeman, M. L. (1988). gIBIS: A Hypertext Tool for Exploratory Policy Discussion. ACM Transactions on Office Information Systems, 6, 4, 303-331. also see E. Jeffrey Conklin, Designing Organizational Memory: Preserving Intellectual Assets in a Knowledge Economy. http://www.gdss.com/DOM.htm
[Cooper95] Alan Cooper, About Face : The Essentials of User Interface Design, IDG Books Worldwide, ISBN 1568843224 1995
[CSCA] Computer-Supported Collaborative Argumentation Resource Site (Simon Buckingham Shum) has a summary, and pointers to papers and other sites. http://kmi.open.ac.uk/
[DejaNews97] Press Release from DejaNews, Dec 8, 1997 Deja News Launches Major Assault On Spam http://emarket.dejanews.com/emarket/about/pr/dnpr_971208.html
[Ellis91] Ellis, C.A., Gibbs, S.J., Rein, G.L., Groupware: Some Issues and Experiences, Communications of the ACM, 14 (1), Jan 1991.
[Fab97] Marko Balabanovic and Yoav Shoham, Fab: Content-Based, Collaborative Recommendations, Communications of the ACM, March 1997, Vol 40, No 3. http://fab.stanford.edu.
[Flores88] Flores, Graves, Hartfield and Winograd, Computer Systems and the Design of Organizational Interaction, ACM Transactions on Office Information Systems, vol 6, no 2, 1988. Commercial system available at http://www.actiontech.com.
[GroupLens] Konstan, J., Miller, B., Maltz, D., Herlocker, J., Gordon, L., and Riedl, J. GroupLens: Applying Collaborative Filtering to Usenet News. Communications of the ACM 40,3 (1997), 77-87 System available at http://www.cs.umn.edu/Research/GroupLens/
[Hawken93] Paul Hawken , The Ecology of Commerce: A Declaration of Sustainability, HarperBusiness, NY, NY) 1993.
[HyperNews] (http://union.ncsa.uiuc.edu/) also see Hypermail (http://www.eit.com/software/hypermail/hypermail.html) and WebNotes (http://tucknt2.dartmouth.edu/).
[Hurwitz95] (2) (3) Roger Hurwitz and John C. Mallory The Open Meeting: A Web-Based System for Conferencing and Collaboration , Proceedings of The Fourth International Conference on The World-Wide Web, Boston: MIT, December 12, 1995. http://www.ai.mit.edu/projects/iiip/doc/open-meeting/paper.html
[Kautz97] Henry Kautz, Bart Selman, Mehul Shah, Referral Web: Combining Social Networks and Collaborative Filtering, Communications of the ACM, March 1997, Vol 40, No 3.
[Lewis93] Clayton Lewis and John Rieman, Task Centered User Interface Design, 1993
[QuestMap] Jeff Conklin, The IBIS Manual: A Short Course in IBIS Methodology. http://www.gdss.com/IBIS.htm
[PHOAKS] Loren Turveen, Will Hill, Brian Amento, David McDonald, and Josh Creter, Phoaks: A System for Sharing Recommendations, Communications of the ACM, March 1997, Vol 40, No 3. The system is available at http://www.phoaks.com/
[Raymond97] (2) Eric S. Raymond, The Cathedral and the Bazaar. first presented at Linux Kongress 97. http://www.ccil.org/~esr/writings/cathedral.html
[Rittel72] Rittel, H. W. J. (1972). Second Generation Design Methods. Interview in: Design Methods Group 5th Anniversary Report: DMG Occasional Paper, 1, 5-10. Reprinted in: Developments in Design Methodology, N. Cross (Ed.), 1984, pp. 317-327, J. Wiley & Sons: Chichester
[Rittel73] Rittel, H. W. J. & Webber, M. M. (1973). Dilemmas in a General Theory of Planning. Policy Sciences, 4, 155-169
[Searle75] Searle, J.R. A Taxonomy of Illocutionary Acts. In Gunderson, K. (Ed) Language, Mind, and Knowledge, Minnesota Studies in the Philosophy of Science, Vol 11, University of Minnesota Press, 1975.
[Shum97] (2) Buckingham Shum, S. (1997) Representing Hard-to-Formalise, Contextualised, Multidisciplinary, Organisational Knowledge. AIKM'97: AAAI Spring Symposium on Artificial Intelligence in Knowledge Management(Mar. 24-26, 1997), Stanford University, Palo Alto, CA (AAAI Press). http://kmi.open.ac.uk/kmi-abstracts/kmi-tr-40-abstract.html
[Suthers95] D. Suthers and A. Weiner. (1995). Groupware for developing critical discussion skills. CSCL '95, Computer Supported Cooperative Learning, Bloomington, Indiana, October 17-20, 1995. http://www-cscl95.indiana.edu/cscl95/suthers.html
| Contact Us • Site Map • Search • Terms and Conditions • Privacy Policy • Participation Policy | |||||
|
|||||
