Computer users have become accustomed to the writing of documents being regarded as a separate activity from the reading of documents. We believe that this division is unnecessary and limits the effectiveness of virtually every computer user. It is time for a rethink of underlying concepts. A key concept for integrating reading with writing is a general mechanism for annotation. This general mechanism can be combined with hyperlinking to create a single unifying super-concept that provides a base for integrating reading and writing. The paper explains the underlying ideas, and describes the results of a small experiment that supported the viability of the super-concept. We believe that the super-concept might possibly provide the foundations for a revolution in thinking about documents, which would benefit everyone.
Currently we have a dichotomy between:
Computer applications designed primarily for reading documents. The most prevalent of these are Web browsers.
Computer applications designed primarily for writing documents. These include editors and formatters; some, like HTML editors, are designed for creating a particular class of document.
The dichotomy started when the first document-manipulation applications were produced, and now seems to be engrained in our thinking. It is time for a radical change: reading documents and writing documents should be regarded as two integrated activities, not as separate ones to be covered by different types of software.
Integration of reading and writing meets a human need. It applies both to writing-while-reading and to reading-while-writing. We will start with the former.
When reading paper documentation we have always been able to write while we read: in particular we can make annotations. We write these in the margins or between lines — indeed the use of double-spacing is specially geared to making annotation easier. Annotation of paper documents is a standard activity in many fields, particularly scientific research. Any document can be annotated — though if the document is read-only, as a library book should be, a copy needs to be made first. Note that annotation captures a degree of physical integration between the underlying document and the annotation itself: each annotation is attached to a particular fragment of the underlying document.
This simple paper facility for writing-while-reading has not yet been carried over to electronic documents. Certainly electronic annotation systems exist, often being related to existing applications such as browsers, e.g. Annotea ( Kahan et al. 2001), iMarkup and the "note" facility in the Opera browser. Annotations are also commonly available in many word processors. Such systems are not, however, close to being available universally to meet annotation needs. To build really good annotation systems we need to rethink our basic concepts and integrate reading and writing from the start. If we do this, electronic annotation can be more powerful than paper annotation, since it has all the extra computer power for organising annotations so that they can be systematically searched, edited, transformed, classified by attached properties, etc.
We now move on to the mirror-image of writing-while-reading: reading-while-writing. While the need for writing-while-reading is obvious, the mirror-image need is less so. Currently the user might see it as "I did not realise I had a need for this". Clearly in the paper world when one is writing a document one sometimes pauses to read a second document, and possibly copy a piece of this second document as a quotation, or include it as a reference. Copying is mirrored and indeed bettered in the electronic world: cut-and-paste is much easier than copying material by hand, especially when the material is not textual. Nevertheless the promise of reading-while-writing goes well beyond this facility. The promise is embodied in the work of Rhodes and Maes (2000) at MIT, and in particular their Remembrance Agents. When a user is writing a document, the Remembrance Agent proactively pops up suggestions of other documents that might be relevant to what is being written. Thus it might pop up a link to a recent e-mail from a person who has just been mentioned in the document being written, or it might pop up a link to a paper that appears to cover the same topic as the current paragraph. Each suggestion is effectively an annotation that is anchored to a certain fragment of the currently-written document. If the user is interested, she expands the full annotation. The Remembrance Agent concentrates on what has most recently been written, e.g. the current paragraph.
Obviously any proactive agent has a danger of becoming an annoyance rather than a help to the user. For the agent to be successful its user interface needs to be discreet. The Remembrance Agent uses a ramping interface, where proactive suggestions are first presented in highly abbreviated form, but can be ramped up (in gradual stages) if the user thinks she may be interested in a particular suggestion. Overall, this proactive approach fits well with the current notion of users employing proactive agents to sift out information that meets their needs. However, the user needs good controls — and a good user interface — to prevent such agents becoming an annoying distraction.
Overall, we believe that reading-while-writing is a technology that has future potential remaining to be exploited.
An important parallel between writing-while-reading and reading-while-writing, as exemplified by Remembrance Agents, is that both are based on an annotation mechanism. What the Remembrance Agent does is to provide proactive annotations for the document the user is writing — these annotations, as we have said, suggest relevant documents that can usefully be read. Typically an annotation produced by a Remembrance Agent is an ephemeral one, but the user might choose to make an ephemeral annotation permanent, e.g. by accepting a suggestion that a certain paper should be cited.
Another creation from the MIT work, its Margin Notes ( Rhodes 2000) agents, shows that proactive annotations are also useful for reading documents via a browser. A Margin Notes agent rewrites Web pages as they are loaded, automatically adding annotations that are relevant to the user's current needs. Proactiveness, therefore, can be a useful property in any annotation system.
A conclusion from the above is that a general annotation mechanism is a prime aid to integrating reading with writing. The mechanism must be general in the sense that:
Annotations can be applied to all documents — including, of course, other annotations.
Annotations can be ephemeral or permanent.
Annotations can be created proactively by outside agents or directly by the user, and in general are suitable both for writing-while-reading and reading-while-writing.
Annotations can be applied to all types of media.
Current annotation systems support some of the above, but to support all four properly we believe a fundamental rethink is needed.
A characteristic of many advances in computer technology has been that they are used in ways that their designers never imagined. By their very nature these uses cannot be forecast. However, one property of a general annotation system, with a good mechanism for data types (see below), gives a hint of its extra possibilities. When the user annotates a document, he tells the computer, as a free side-effect, what parts of the document interest him, and perhaps what parts do not. This is particularly true if each annotation has a data type that registers its nature and importance. Thus the computer gains an ever closer knowledge of the user's current interests, and can adapt its behaviour accordingly, e.g. by refining those proactive suggestions it makes. Carrying this further, a user may make an annotation with the sole purpose of telling the computer that a certain item reflects their interests. Maybe this gives a flavour of a new market for applications.
If a general annotation mechanism is to be a key to the document systems of the future it must be coupled smoothly with the most important mechanism of currently-used systems: the hypertext link. Our view is that this should be easy to do because the concepts are already close to one another.
In the simplest form of annotation a comment is attached to a fragment of an existing document. (With paper annotation, the annotator often underlines or circles this fragment.) We call the comment the annotating document — typically it is a very short document. Following hypertext terminology we call the fragment of the existing document the anchor. With this model, the annotation mechanism mirrors the hypertext linking mechanism found in the Web. The difference is one of ownership:
An annotation is a link from an existing document (which may be owned by someone else) to a document created by the author of the annotation — we have termed this the annotating document.
A Web link is a link from a document created by the author to another document, which might be owned by someone else.
To put this another way, the difference is whether the author's document is created as the source or the destination of linking.
This analysis is, of course, somewhat simplistic, but more detail is given in Brown and Brown (2003a). In particular, annotations can usefully be generalised beyond the model of the simple linking mechanism found in the Web to mirror a more general linking mechanism, as provided, for example, by Microcosm ( Davis et al. 1992) or as analysed by DeRose (1989). Similarly, annotations can be enhanced by using data types that capture the different types of annotation, just as hyperlinks may have data types.
The computer science graveyard is full of great ideas that died. Many of the dead are attempts to generalise apparently similar concepts; the cause of death was that at the detailed level the concepts had a lot of minor differences, and the patient died of complexity. The mantra 'the devil lies in the detail' applies strongly.
Thus to test the practicality of combining hypertext linking with annotation we performed a small in-house experiment ( Brown and Brown 2003b). This involved two examples of annotation needs, both real cases that had already been done on paper. The first example was a set of annotations representing suggested edits to a document. These suggestions covered insertions, deletions, replacements and general commentary ('This paragraph is long-winded'). The second example was a set of annotations, written by a research scientist, attached to a research paper. These annotations related certain sentences or longer extracts in the paper to the annotator's current projects; they also contained some opinions on parts of the paper. One of the annotations related to the markup of the paper rather than its textual content; the markup in question was a comment saying that the author thought certain work was irrelevant — something the annotator disagreed with.
The experiment involved trying to implement these annotations just by using existing hypertext linking facilities, as provided by an existing hypertext system — we did not change this system in any way. The experiment was made more realistic by attaching a set of properties to each annotation (its data type) in order to make selective retrieval easier at a later date. Possible data types might be for-my-thesis and memorable-quote.
Clearly the experiment was a limited one — we are now doing more comprehensive studies — but at least it represented real tasks. The aims of the experiment were:
To show the similarity of hyperlinking and annotation by implementing annotation just by using hyperlinking.
To show some of the devils that come out when one works at the detailed level.
The first aim was fully met in that all the annotations were successfully implemented, though the interface was clumsy. The conclusion to (2) is that we did find some devils, but we also found a few angels — i.e. cases where the detail revealed gains from combining hypertext linking and annotation.
Some detailed points to come out of the experiment were:
Sequential viewing: when looking at a set of annotations, a user will often want to go through them sequentially. Hence a Find Next command is needed. Such a command can also be useful for hypertext links, e.g. in conjunction with systematic checking — indeed the facility was already present in the hypertext system we used. Thus this is an example where detailed examination brings out common needs rather than differences — an angel more than a devil.
Advantages of paper: even if the user interface had been perfect, annotation online is still tedious compared with paper annotation. If the document to be annotated is long, most of us would much prefer to print a paper version and annotate this. Our experiments therefore stretched the user's patience. The benefits from capturing annotations online need to be great if users are to be convinced that the extra initial effort is worth it. We believe that the benefits really are great. For the electronic annotations captured in our experiment, these benefits would particularly come from the searchable and processable nature of the annotations on the research paper — benefits that, perhaps years later, could amply reward the initial creation effort. We also found benefits where our annotations were sparse and hard-to-find — see Find Next above. Overall this represented a powerful devil, albeit an already well-known and wide-ranging one.
Carelessness: as Marshall (1998) has observed, and as we found ourselves, users are often careless in setting the anchors of their annotations. For example, if the user indicates an annotation anchor by underlining it on paper (or dragging over it electronically), they might carelessly miss out the last letter of the last word. This does not matter if the annotation is to be processed by a human, who can make the necessary adjustment: 'This is what was really meant'. If annotations are electronic and subject to electronic operations (e.g. find all anchors containing the word X) carelessness can lead to wrong results. Since in hypertext it is rarely a problem if anchors are inaccurately specified, this represents a devil.
Annotation properties: we have said that the experiment used different data types: these helped to distinguish different kinds of annotation and also to distinguish different parts of the body of an annotating document (e.g. (Part 1) a suggested re-wording and (Part 2) a comment explaining why). The simple model we used proved, however, to have limitations: it was desirable in some cases for an annotation to have two different data types (e.g. both "for my thesis" and "for my anthology of pithy sayings"). We need a better mechanism to attach properties to annotations, and also to find a compromise between freedom (anyone can create a new data type on a whim) and discipline. We have a devil, but have a chance of killing him.
Overlapping anchors: sometimes annotation anchors may overlap. For example an annotation may be attached to a whole paragraph and another annotation may be attached to a word within that paragraph. Worse still, two anchors may overlap in a non-nested way, e.g. one anchor on the first two words of a paragraph and another on the second and third words. Overlapping anchors are not a concept familiar in hypertext. In fact the examples in our experiment did not happen to contain any overlapping anchors. If they had, our experiment would have partly failed: we were lucky. The realisation of a more general anchoring mechanism needs work, and this devil represents a significant enemy.
Multiple anchors: some annotations may have multiple anchors, e.g. an annotation that relates to two disjoint paragraphs ('These appear to contradict'); again, by luck no examples of this occurred in our experiment. Multiple anchors are, however, a more familiar concept in hypertext than overlapping anchors. This applies particularly to generic links and more generally to all instantiations of many-to-many links. Thus, again, we have a conceptual similarity rather than a conceptual difference with hypertext — another angel.
Symbiosis: it proved valuable to use, in tandem, annotations and conventional hypertext links to outside documents — an obvious angel.
One further caveat is that the experiments used what is today an off-beat hypertext system, Guide. Guide was chosen primarily because it has the advantage of offering a built-in authoring mechanism, which is good for supporting a writing-while-reading interface. In spite of this caveat and the devils noted above, the experiment increased our overall confidence that death-by-devils-in-the-detail was not imminent, and thus we moved on to the next stage.
Given that annotation is a key to integrating reading and writing, and that annotation and hyperlinking are similar concepts, our quest for the future is a super-concept that embraces existing ideas of both annotation and hypertext linking. This super-concept should also capture other existing annotation-like practices, such as bookmarks, history and paths. We call this super-concept hypernation — a horrible word the only merit of which is that it is no less horrible than the obvious alternatives. Hypernation is our means of realising the read/write document.
This last part of the paper looks at issues concerning possible future realisation of the ideas. We claim that hypernation will benefit every user of electronic documents. However, many failed dreams have had the extra claim '... and this will be the end of the paper document'. We are well aware that reading a document on paper, and annotating its paper form, will always be more convenient than using even the best electronic reading/annotation system. As we mention above, our experiment reinforced the tedium of creating electronic annotation, though it did show the benefits of subsequently using them. We therefore expect electronic annotation to represent only a proportion of the existing market, especially that where annotation can have a long-term use. In addition we expect it to create new markets; we have already guessed at where these might lie, i.e.
Proactive suggestions based on a knowledge of a user's true interests gleaned from the annotations they make.
Effective support for all the document work of a research scientist.
When taking hypernation in general, an integration of previous diverse mechanisms.
The computer's big advantage is that it can store all annotations (and all authored hypertext documents?) in a repository, thus allowing annotations to be selectively retrieved at later dates. A real pipedream would allow a user to retrieve any annotation made in her lifetime ("Find everything I ever marked with an annotation containing the words 'ULTRA project'" or, better still, "Find those whose importance is above a certain threshold", as in the work of Röscheisen et al. 1995). This would help improve the current extremely inefficient use of document resources such as Web browsers ('I knew I saw something about that a few months ago, but now I cannot find it'). The computer also has an advantage when the document being annotated would normally be read online (e.g. it is not a long textual document) and thus paper is a weaker competitor, and when annotation can be smoothly integrated with other electronic mechanisms, as the hypernation super-concept envisages.
Overall, we have great confidence in the advantages to be derived from realising our dream. One field in which we feel the advantages are particularly compelling is for research scientists reading documents. Our particular experience is in bioinformatics. A researcher investigating, say, genetic links to cancer has a needle-in-a-haystack problem. If, when the researcher finds a straw (say an existing paper, on the Web, reporting a study of patients in 1980) that may be close to the needle, then it is invaluable to have a facility to mark it (make an annotation that comments on why the paper relates to a certain genetic hypothesis) so that it can retrieved at any later date. More generally, if the user has tools for processing the repository of hopeful straws that have been recorded in order to find overall patterns, then the promise is multiplied. We hope our ideas represent a better way forward than current approaches.
We feel that implementing software to realise the dream should have two guidelines:
Orient it to use in a particularly favourable area. In one respect document systems are already a favourable area for experimentation because errors are not usually catastrophic. If a user fails to find a document or loses an annotation, no bridges will fall down. To take another example, many Web pages are full of HTML syntax errors, but they are still useful; on the other hand a computer program containing a single syntax error is unusable. Thus we already have one favourable property. We think, however, that this guideline should be carried further by choosing a limited sub-area where potential advantages to users are especially great. We believe, as we outlined in the previous section, that scientific research is one such sub-area.
Do not try to solve all the problems of the world in one go.
The second guideline needs amplification. The original designers of the Web might have said "we have found a flaw in our design: there is no mechanism to deal with dangling links. We must stop development until we have produced a new design to remove this flaw, perhaps using global registration of all links". They did not: dangling links remain a very serious problem, and yet the Web is a gigantic success. People live with the problems. The same is true of annotations: the dangling link is mirrored by the orphan annotation, which is an annotation attached to a document that no longer exists, or the content of which has changed, thus invalidating the annotation anchor. Orphan annotations will always be with us; indeed, Bernheim Brush et al. (2001) suggest that users prefer to live with them rather than employ an "intelligent" system to resolve them. Likewise, legacy problems will always be with us: if a document and its annotations are saved in a repository, then five years later the document may be useless because the software to render it is no longer available.
The above philosophy of ignore-all-hard-problems-if-you-possibly-can can be carried to a higher level: documents themselves. It is therefore unwise to tackle all the current issues in document use. For example, although capturing the semantics of documents is an important issue (and could help greatly with intelligent proactive annotations), it is not one we plan to encompass.
The key to our dream is building and evaluating hypernation systems. We believe that our approach should have two parallel strands:
Today strand: based on using today's systems, notations and standards. We have already started work on this.
Tomorrow strand: developing the ideal framework, architecture and notations, and building a prototype system to realise this. 'Tomorrow' will involve a timeframe of 5-10 years. For this reason there must still be elements of pragmatics: (1) ignoring hard problems if we can, and (2) not saying 'we will change every aspect of the document world in order to meet our needs' — thus destroying all compatibility with existing systems.
Our first step to realising hypernation must be to flesh out the concept so that it is as general as possible, has motherhood properties (simplicity, ease of communication, robustness over change, a good user interface (see, e.g. Zellweger et al. (2001), etc.) and provides a good foundation for implementation. This is important for both the 'today' and 'tomorrow' strands, though the former will only be able to provide a subset of the features.
The second stage is to define a notation for capturing hypernation documents. Three choices are:
the ideal notation for the job, with no concessions to existing practice
the XML family
Given the need for some degree of pragmatism, we believe (2) is suitable for the 'tomorrow' strand. The 'today' strand will use (2)/(3).
The third stage is to build prototype systems to realise hypernation. These are probably best presented to the user as enhanced browsers. Our first PhD. supervisor, David Barron, offered the wise advice: 'if you want to sell a new concept, relate it to something the user already understands'.
The fourth stage is to evaluate usability. It is notoriously difficult to get large numbers of outside users to try out prototype systems for real tasks. However, we believe that the benefits of hypernation can be demonstrated using single users and reasonably short-term tasks — e.g. three-month tasks. Thus there is hope. If evaluation is successful there can be a stage where success quickly builds on success. Experience shows, however, that success is only partly due to a good idea, a good realisation, and determined exploitation (see Milner and Wand 1996). The biggest contributor to success is likely to be the lucky break, e.g. the head of Microsoft happening to see a demonstration. 'I returned, and saw under the sun, that the race is not to the swift, nor the battle to the strong, nor yet riches to men of understanding, nor yet favour to men of skill; but time and chance happeneth to them all' Ecclesiastes.
Few would argue against the desirability of integrating reading documents with writing them. Few would also argue against combining hypertext linking with annotation as one aid to implementing this. The doubts come when one considers feasibility. This paper, though its main aim is to present the ideas, has also made a small contribution to showing feasibility. Our dream is to see a continued advance to proving feasibility. If successful the outcome will be a new breed of software products that combine increased conceptual simplicity with increased power.
We are grateful to the University of Exeter for support of this work. We would like to thank referees for their well-considered comments.
Bernheim Brush, A. J., Bargeron, D., Gupta, A. and Cadiz, J. J. (2001) "Robust annotation positioning in digital documents". Proc. SIGCHI '01, Seattle, WA, pp. 285-292
Brown, P. J. and Brown, H. (2003a)
Annotation: a step towards the read/write document
Brown, P. J. and Brown, H. (2003b) Is
annotation a form of hypertext linking?
Davis, H. C., Hall, W., Heath, I., Hill, G. J. and Wilkins, R. J. (1992) "Towards an integrated environment with open hypermedia systems". Proceedings of the ACM Conference on Hypertext: ECHT92 (ACM Press), pp. 181-190
iMarkup (2003) iMarkup: annotate, organize and
collaborate on the web
Kahan, J., Koivunen, M-R., Prud'Hommeaux, E. and Swick, R. R. (2001) "Annotea: an open RDF infrastructure for shared web annotations". Proc. WWW10 International Conference, Hong Kong, May, pp. 623-632
UKCRC (2002) Grand Challenges for Computing
Peter and Heather Brown are both Visiting Professors in the Computer Science Department of Exeter University. Both have been interested in document systems since the early 1980s when, while at Stanford University, Heather worked on the display of TeX documents and Peter worked on embryo hypertext systems. The work described here came out of the UK initiative for Grand Challenges for Computing Research.