Experimental User-Centered Evaluation of an Open Hypermedia System and Web Information-Seeking Environments

Michail Salampasis and Konstantinos I. Diamantaras
Department of Informatics, Technology Educational Institute of Thessaloniki, Greece
Email: {cs1msa, kdiamant}@it.teithe.gr
Web: http://aetos.it.teithe.gr/~cs1msa/; http://aetos.it.teithe.gr/~kdiamant/


This paper presents an experimental user-centered evaluation of two hypermedia system architectures, each representing two different interaction models and information-seeking environments. The first system is a hypermedia digital library based on the World Wide Web. This system represents an interaction model in which information seekers consistently use a single interface (i.e. a Web browser) to access different information seeking strategies (ISSs). The second system is a similar library (in terms of content and organisation) which is based on an agent-based Open Hypermedia System (OHS). This library encourages an interaction model in which multiple user interfaces and information-seeking strategies may be used in a more parallel fashion. Several researchers have suggested that information seeking may be more effective in systems that allow the parallel use of multiple information-seeking strategies. On the other hand, the ease of use of the simple click-and-go-to interaction model introduced by the Web and the consistency of its interface appears to be more attractive for most information seekers. The aim of this paper is to examine and discuss these hypotheses critically. Although general conclusions cannot be drawn from the experiment, the results present some useful indications. A first indication is that information-seeking environments that support multiple seeking strategies through multiple interfaces may be more effective and efficient for some information-seeking tasks. Also, results taken from a questionnaire given to users of the OHS indicate that complex interaction models may not be prohibitively difficult to use, even for inexperienced information seekers.

1 Introduction

The World Wide Web has attracted much criticism, especially from the Open Hypermedia Systems (OHSs) research community, despite its tremendous success as an underlying platform for developing a large-scale informational environment. Most of the criticism is focused on the simple linking mechanisms that are inherently supported in the Web and the lack of separation between the links and the data (i.e. the documents) being linked. Projects such as Hyper-G (Andrews et al. 1995) and theDistributed Link Service (Carr et al. 1995) are early examples of efforts to enrich the Web with non-embedded links. Since 1996, when the wide acceptance and predominance of the Web became apparent, other research efforts have been also presented in the hypertext literature suggesting methods for enhancing the simple linking mechanisms of the Web (e.g. Groenbaek et al. 1997; Anderson 1997; De Roure et al. 2001).

Another issue that concerns the hypertext community is that the Web does not inherently support advanced structures as first-class objects. The Web supports only the basic hypermedia directed graph structure, so it can only give an illusion of other structures such as hierarchies or clusters of documents (Nurnberg and Ashman 1999). This illusion of structure can be given in the Web only using specially developed tools. In other words, in the Web �structure awareness� can be implemented and perceived only at the application or user level, but it is not inherently supported at the system level. In contrast, because in OHSs links are stored externally and separately from data, and may be based on more advanced data models, e.g. the Dexter model (Halasz and Schwarzt 1994), they can inherently support more organisational entities, e.g. sets, hierarchies, which can be manipulated as first-class objects.

These concerns and efforts to distinguish OHSs from other hypermedia systems such as the Web were largely based on considerations that examine OHSs and the Web from the perspective of architecture and linking mechanism. Our aim in this paper is different. In examining the two different architectures (i.e. OHSs and the Web), we wish to emphasise an information-seeking and digital library [1] perspective. Undoubtedly, the Web is an environment mostly used for information seeking. On the other hand, despite the fact that OHS research has focused on viewer integration and linking mechanisms methods and protocols, OHSs should be clearly regarded as informational hypermedia systems which are suitable as platforms for developing digital libraries. This is mainly because OHSs inherently satisfy architectural characteristics (e.g. distribution, extensibility, scalability, heterogeneity, interoperability) that are desirable in developing digital libraries.

Generally, the term Open Hypermedia Systems describes systems delivering hypermedia functionality in an open and flexible manner. In an OHS the linking mechanism and the integration of different autonomous tools in user�s desktop is considered as a back-end process (Hall et al. 1996, pp. 28). An OHS also provides one or more protocols to allow any external application to access linking services. Osterbye and Wiil (1996) presented the flag taxonomy to define and classify OHSs and to distinguish them from closed hypermedia systems. The flag taxonomy (Figure 1) distinguishes between storage and runtime aspects as well as between content and structure. Four functional modules are defined and protocols exist to enable communication between these modules.

figure 1

Figure 1. Flag taxonomy specifies four functional modules: storage manager which handles content, data model manager which manages structure, session manager which enables runtime structure, and collaboration and viewer (actual representation of content and structure)

From an information seeking perspective, the central issues are quite different from the architectural issues. For example, the openness that characterises an OHS allows a set of different interfaces to coexist in an information seeker�s desktop. This may be seen as a desirable feature, but it could also easily lead to �information overload� on the user. Therefore, OHSs and the Web should be carefully considered within the context of hypermedia information-seeking environments. Information seeking in very large information spaces such as hypermedia digital libraries (HDLs) is an interactive and complex process that must be performed efficiently and effectively. HDLs are characterised by the existence and potential parallel use of multiple strategies for information seeking. Since they are based on a hypermedia paradigm, they will support opportunistic browsing strategies (e.g. across-document browsing). In addition to browsing, HDLs also support analytical (i.e. query-based) strategies because they are usually more effective in large electronic environments.

Besides these two fundamental information-seeking strategies (ISSs), several other tools and interfaces may be used (e.g. tools for source selection, tools visualising the information space under searching, interfaces that allow navigation of meta-information such as a hierarchical index or clusters of documents, tools for using a bookmark collection, and tools that can perform collection fusion or information filtering). This paper presents an experimental study with 24 users (12 users allocated in each of two experimental environments) using multiple search strategies (e.g. browsing and searching). The concept of multiple ISSs has been known for some time, so we review work in this area that was largely based on a highly user-centered theory of information seeking known as Anomalous State of Knowledge (ASK) (Belkin 1980).

Based on this highly interactive view of information retrieval (IR), several works published in the IR and hypertext research literature have suggested the idea of using multiple ISSs. Pejtersen (1989) implemented the idea of using multiple strategies in a system called Bookhouse. This system had an integrated spatial metaphoric representation that included database content storage and structure as well as support for information processing and retrieval. It is useful to note that Bookhouse clearly separates content and structure (as in OHSs), as well as between these information objects and the retrieval and processing functionality. Also of interest in the context of this paper is that Bookhouse may be used as a universal metaphor because it can accommodate different databases in terms of content and structure while the basic search facilities remain essentially the same. This feature is analogous to the capability of OHSs to host different autonomous viewers (in the viewer functional module as it is presented in the flag taxonomy). The highly interconnected nature that distinguishes hypertext from conventional text databases has also been used to design information-seeking environments supporting both browsing and analytical searching (examples of older and more recent works are: Croft and Thomson 1987; Frisse 1988; Rada and Murphy 1992; Golovchinsky 1997).

The concept of using multiple strategies was discussed and formalised by Belkin et al. (1993) in the BRAQUE interface. The underlying framework in the design of BRAQUE was a model of ISSs classified in 16 categories as a result of applying four dichotomous factors (method of interaction, goal of interaction, mode of retrieval, resource considered). Besides the categorisation of multiple ISSs, Belkin et al. emphasised the observation that in systems not constrained to a single ISS people often change from one ISS to another during an information-seeking episode. They also emphasised that few studies have investigated information-seeking behaviour from the point-of-view of changing from one ISS to another, although there is empirical evidence that it takes place (Hancock-Beauliue 1990; Belkin et al. 1990). This aspect of users changing from one ISS to another is also observed in the experimental study presented in this paper. The idea of multi-dimensional ISSs was further developed and combined with the concepts of dialogue structures for information seeking, cases of specific information-seeking dialogues and scripts as distinguished prototypical cases (Belkin et al. 1995).

To understand the nature of interaction in IR, Lalmas and Ruthven (1999) suggested a formalised framework based on channel theory. Formalised frameworks that can provide a certain degree of abstraction are desirable in studying the complex process of information seeking. We also believe that it is of special interest not only to investigate how and why changes from one ISS to another occur during an information-seeking episode, but additionally to investigate patterns of ISSs. More precisely, we think it may be of special importance to investigate the distribution of different search states (ISSs), what ISSs are more likely to follow another ISS, the probability of one ISS occurring after another, if and how search tasks influence the selection of ISSs. Qiu (1993) presented a mathematical approach based on the Markov model to study search state patterns in a hypertext IR system. In experiments, Qiu found that there were significant differences between different groups (specific search task versus general search task) in terms of the search state patterns produced by each group.

The study presented in this paper is based on a less formalised framework in comparison with the works above. Another difference is that the concept of multiple ISSs is examined within the context of OHSs and therefore reveals some interesting questions that require careful consideration of OHSs and the Web. For example, to what extent and what actions should be taken into the design of such systems in order to support a rich information seeking environment? Another question is whether such a rich information seeking environment will actually be more effective and efficient than one that supports just one or a few tools/interfaces. It may legitimately be suggested that, in terms of cognitive overhead, it may be expensive to use multiple strategies in parallel, and it may also lead to user disorientation. Carr et al. (1998) report in an informal evaluation of the Open Journal project that �many users were confused by the use of multiple windows��. Thus, a simple interaction model using few methods serially may produce more efficient and/or effective searches. On the other hand, others may think that more research is required to design systems and interfaces that can support multiple ISSs without overloading or confusing the information seekers.

Another crucial issue in considering OHSs and Web from an information seeking perspective is how these systems can support efficient movement from one ISS to another. If we accept that movement between ISSs will happen during an information seeking episode as a result of a change in the user�s goals or a change in her/his state of knowledge, then it is better to use systems that can provide support for such a movement in a principled fashion.

Some works in the IR literature relate to the first question of designing IR-extensible system architectures for supporting rich information seeking environments like HDLs. Rao et al. (1992) presented an application framework called InfoGrid, which enables the design of rich information seeking environments. Hendry and Harper (1996) report another effort to increase effectiveness by designing an extensible user interface architecture called FireWorks. Hendry and Harper (1997) suggest that flexible interfaces are useful in developing extensible information seeking environments. These reports and suggestions are contrasted with system architectures and user interfaces that are inflexible and use a consistent interaction model and are therefore characterised as "over-determined" and "over-engineered" (Vickery and Vickery 1993).

Other works consider whether a rich information seeking environment will actually be more effective and efficient than an environment which supports just one or a few tools/interfaces. Rao et al. (1995), in a work based on InfoGrid, reported that performance of information seekers could be increased if they interact in rich information workspaces. Bonder et al. (2001) report TREC experiments investigating the impact of text browsing on the effectiveness of text retrieval, finding that recall is improved using browsing in addition to query-based searching, probably as a result of users viewing more documents. Other interactive experiments, however, do not report a significant increase in performance by the combined use of multiple strategies. For example, Allan et al. (2001) evaluated a system combining ranked search results with visualisations of document similarities. They found that although users liked the combination, there is no significant improvement in effectiveness in comparison with using only the traditional ranked list.

The basic aim of this paper is to examine two different system architectures, each representing a different philosophy and interaction model for information seeking. The first system under evaluation is a digital library based on the Web. This system will usually be accessed through a single Web browser, which represents a relatively "pre-engineered" and consistent interface. The second system is an OHS calledNIKOS (Salampasis 1997). NIKOS is an agent-based OHS in which different objects called hypermedia agents can coexist, coordinated and examined by the information seekers in a more parallel fashion.

The aim of the experiment is not to criticise one or other system or to find a 'winner'. The systems used are not the main subject of the experiment; they are only meant to act as typical representatives of the information seeking environments and interaction models under consideration. The experiment aims to illustrate two different approaches for designing information seeking environments and to measure their performance in a similar (in terms of content and structure) hypermedia digital library. It must also be noted that the results presented in this paper cannot be generalised to all information seeking environments or to all information seeking tasks. However, we believe the experiment and the results are useful in two dimensions. First, they may shift some attention in the comparison between OHSs and the Web from a linking mechanism to an information seeking perspective. Second, it presents some results produced from real world experiments. User-centered experiments are difficult and time consuming, but they usually reveal issues and produce results that cannot be found in less formal evaluations.

The rest of this paper is structured as follows. In Section 2 describes the two systems used in the experiment and examines them in detail from an information seeking perspective. This is a long section to help the reader judge the results adequately and to allow replication of a similar experiment. In Section 3, we present the experiment and  discuss the results critically. Finally, Section 4 considers the limitations of the experiment and attempts to draw some initial conclusions.

2 Two different Information Seeking Environments

2.1 Digital Libraries based on OHSs

The most important characteristic of OHSs from an information seeking perspective is that because of their open architecture they can provide different interfaces and tools on the user�s desktop. These tools and interfaces may use the linking mechanisms that are the essential part of an OHS. We should make clear that the use of multiple information seeking strategies can be implemented in any hypermedia digital library system, not only in HDLs based on OHSs. However, the basic architectural characteristics of OHSs make it easier and more natural to develop HDLs supporting �rich� interactions. A detailed discussion of OHSs can be found in Hall et al. (1996) and in the series of OHS workshops. In this paper we use an OHS system called NIKOS as an example of an OHS-based information seeking environment. The NIKOS system is based on an agent-based distributed OHS architecture and is composed of different programs that are engineered as hypermedia agents (HAs). HAs are software agents (Nwana 1996) exchanging messages in NIKOS' commonly agreed hypermedia agent communication language.

This paper is mostly concerned with NIKOS from an information seeking and HCI perspective. The defining feature of the NIKOS OHS as an information-seeking environment is that it emphasises and advocates the mixed use of different information seeking strategies. In NIKOS, information seekers use multiple HAs that may interact in a parallel, coordinated fashion. From an HCI perspective, NIKOS aims for maximal interactivity, which can keep users engaged in a concurrent viewing and decision making process rather than batching the decision making first and then viewing as it may happen in the Web.

This interaction model is possible because multiple HAs can coexist in the NIKOS system and each HA supports a different information seeking strategy:

  1. simple across-document browsing, which is supported by atom and viewer HAs;
  2. clustered browsing: this type of browsing is supported by primitive HAs which do not display raw data, but display clusters of raw information objects;
  3. hierarchical browsing: supported by composite HAs which display hierarchies of other composites and clusters;
  4. browsable tables of content, which is supported by a special HA called library hypermedia agent and provides an overall view of the information space being searched;
  5. single collection and multi-collection query-based searching supported by a specialised hypermedia agent called Information Retrieval (IR) HA.

Note that the use of multiple browsing strategies (e.g. simple, clustered and hierarchical browsing) is possible because NIKOS uses a more advanced data model [2] which utilises the notion of sets (primitive and library HAs in NIKOS) and hierarchies (composites in NIKOS) for organising information.

HAs, which represent the different strategies, coexist in the user's desktop and can cooperate by exchanging messages. Hence, they can support the information seeker in moving from one tool (i.e. a HA) that implements a particular search technique and uses a particular interface, to another tool that supports a different strategy and implements a different interface. Of course, the information seeker must have control of this process and should be able to c-ordinate multiple HAs. Thus, to seek information in NIKOS is to co-ordinate, manage and interact with HAs.

From the user�s point of view, as an information seeking environment NIKOS is seen as a set of autonomous cooperating tools which are available to assist searching (Figure 2). Each tool can assist information seekers in a different way during an information seeking episode. For example, the composite HA assists users doing clustered browsing, and the IR HA helps in the execution of queries. Other HAs assist information seekers in examining search results using different views of the information space being searched (e.g. simple networked, clustered, hierarchical).

figure 2

Figure 2. Overview of NIKOS as an information seeking environment

Figure 3 illustrates the manifestation of the sketched view shown in Figure 2. It presents a snapshot of an information seeking episode with several HAs activated. Each HA presents a different view of the information space . In Figure 3, different HAs are active and offer the user the opportunity to view different, but possibly related, views of the information space. For example, the user views the actual document in the Text Atom HA (on the right of the screen), while at the same time s/he can view the cluster to which this document belongs (Primitive Agent; top left) and the hierarchy of documents (Composite Agent; bottom left). At the bottom of the screen (task bar) the information seeker can view other HAs (e.g. a library HA and an IR HA) that can be activated if required. This interaction model is quite different from the usual practice of information seekers using the Web who interact with one Web browser and have a single view of the information space.

figure 3

Figure 3. Screenshot showing the parallel use of several interfaces/HAs to search information in NIKOS

During the information seeking process, interoperation between different HAs is possible using the commonly agreed language. A message usually informs other HAs about a change in the information seeking environment (e.g. a text HA may inform other HAs that the information seeker has changed the document under examination). This information may be used by other HAs to change or adapt the information they are presenting to the user. For example, a Primitive HA may change its view to the cluster that has as a member the new document under examination.

2.2 Information seeking environments based on the Web

The Web can be regarded as an information-seeking environment that supports at least the two fundamental information seeking strategies, i.e. browsing and query-based analytical searching [3]. On the Web, however, both browsing and analytical strategies are usually accessed through the same interface, i.e. a single Web client. Consider for example a typical information seeker using the Web. The user will click on a link and s/he will be presented with a new search artefact that should be examined in isolation. Based on examination of the search results (i.e. a Web page), s/he will click on another link to produce another search artefact that must be examined, again in isolation, and so on.

Usually an information seeker on the Web will initiate the searching process using an analytical (i.e. query based) strategy. In that case, s/he will access a search engine through a Web client to write and run a keyword-based query. The results of the query will be presented to the user as a new Web page that must be examined before browsing will continue or a new search is initiated. This process is mostly "pre-engineered" into the design of the Web. On the other hand, the simple interaction model used on the Web has proved to be easy to understand and use even by inexperienced users.

Besides the two fundamental strategies, other methods can be used to explore the Web. For example, one method is thematic, clustered browsing provided as an externally developed add-on service to the Web (e.g. Yahoo). Using thematic exploration someone can use a Web browser to search in specially prepared Web pages presenting different hierarchically organised subjects. At the bottom of this hierarchical structure one can find clusters of document-leafs that are all relevant to a common theme. Another example of a method for Web exploration is WEBSOM, which uses Self Organising Maps (SOMs) or other visualisation techniques (e.g. Chen 1999). SOMs is a clustering technique which pre-processes and organises collections of text documents and prepares a visual map to facilitate the retrieval of information (Kaski et al. 1995, Rizzo et al. 1999). Using a single Web browser someone can navigate through this specially prepared map and find a relevant document using explorative search.

It should be noted, in response to the idea of using multiple searching strategies discussed in Section 1, that an information seeker on the Web can use multiple strategies for searching relevant information (e.g. browsing, query-based searching, clustered and thematic browsing, WEBSOM). However, on the Web these different strategies are accessed and used through the same Web client interface. The consistency of the interfaces and the serial "click and go to" interaction model may be one of the reasons that most people find it natural and easy to use the Web. On the other hand, the existence of a single tool (i.e. a Web browser) to access many different information seeking strategies may have an effect in reducing the overall interactivity of the environment. Although multiple strategies exist on the Web, due to the use of a single interface these strategies are mostly used in isolation and not in parallel.

It is true that in a Web based information seeking environment, multiple Web browsers may be activated to search using different information seeking strategies. However, few information seekers use this method. The first author of this paper remembers many teaching efforts to demonstrate to new users of the Web how multiple browsers may be used in parallel for information seeking. Most users preferred to switch quickly to a  single browser. This is probably because the use of multiple browsers may be confusing for users who are primarily looking for simplicity in their interfaces. It could also be explained by the lack of understanding of most users of how to use multiple interfaces or how to coordinate multiple strategies in parallel. Undoubtedly, the consistency experienced using a single Web browser is a factor that discourages most users from adopting a multiple browser strategy.

If someone using the Web utilises multiple Web browsers to employ multiple information seeking strategies in parallel, coordination between these multiple browsers is difficult. Consider for example that multiple browsers are used to access a Web-based digital library, e.g. one browser to access Web pages of a digital library and another browser to access a WEBSOM view/map of the same library. The Web does not provide any protocol for interaction and coordination between the two browsers. Coordination may be useful in exchanging information between the two browsers about the Web page under examination. For example, a Web browser showing a WEBSOM may change its view to a map area that includes the Web page under examination in a second Web browser. Using this method, a user may navigate and examine Web pages in the first browser, and at the same time would be able to view in parallel (using the second WEBSOM browser) the �neighbourhood� of the document under examination. Unfortunately, the Web is not open enough to allow such interoperation between browsers.

2.3 Web system used in our experiment

Before we describe the Web system used in our experiment in detail, we explain the document collection. The CACM standard test collection [4] (Fox 1983) was used to build the HDLs under evaluation. CACM is a typical medium-sized collection (3204 documents containing title, author, abstract, keywords and links between documents) used in the past to conduct hundreds of experiments in information retrieval. In the results reported in the rest of this paper a subset of the CACM collection has been used (CACMB) which contains all the documents that have links to other documents in the collection. Some statistical information about the test collection is given in Table 1.

Table 1. Basic characteristics of the CACM collection

table 1

Since one of the elements that we wanted to investigate in this study was the user�s information seeking behaviour in a distributed environment, we had to simulate a distributed environment. To produce a distributed electronic environment the single collection of CACM documents was clustered into eight sub-collections using the complete-link hierarchical clustering method (Voorhees 1986). Content similarity between documents was used to create the between-documents similarity matrix. The result of the clustering process is a well organised HDL in which several information seeking strategies may be utilised. Table 2 shows the �distributed� version of CACM used in our experiment.

Table 2. �Distributed� version of the CACM collection

table 2

The CACM documents originally come as text files and therefore had to be pre-processed and converted into HTML. Figure 4 shows a CACM document in HTML format, as it was used in our experiment. Overall, eight different types of Web pages have been produced with each page corresponding to a specific ISS/interface within the Web system. Besides the �Back to previous screen� capability that is provided by the Web client, several movements were defined explicitly between one page and another. These movements became possible by activating links properly engineered in every page (see for example the top of the Web page in Figure 4).

figure 4

Figure 4. Example CACM document as used in our experiment

Table 3 and Figure 5 explain all the possible information seeking strategies that were available in both systems and all the possible movements from one strategy to another.

Table 3. Information seeking strategies (states), descriptions and possible movements

table 3
figure 5

Figure 5. States during an information seeking episode and possible movements

3 Experiment

3.1 Aims

This experiment aims to evaluate the performance of two different information-seeking environments. The first system under evaluation is a digital library based on an OHS. The second system is a similar, in terms of content and organisation, digital library based on the Web.

3.2 Systems used

Two different HDL systems have been implemented for the experiment. These two HDLs were developed over exactly the same CACM collection.

  1. Based on the NIKOS OHS, described in Section 2.1.
  2. Based on the Web. This system is described partially in Section 2.2 and in more detail in Section 2.3. Subjects were told they could use more than one �open� browser simultaneously to search for information.

In summary, the same information seeking strategies were supported in both HDLs, but each HDL uses a different interaction model and represents a different information seeking environment.

3.3 Method

Twenty-four subjects voluntarily participated in the experiment, with 12 subjects randomly allocated to each condition. All the participants had experience of using the Web on a daily basis. None of the participants had any experience of using the NIKOS OHS. A 15 minute presentation outlined the NIKOS OHS to the subjects before the test. Also, a 15 minute training session was conducted with each subject before testing, to ensure that s/he could search the HDL and also s/he understood the nature of the task that s/he will be asked to perform. No formal training in information seeking strategies was given to the subjects.

Subjects were tested individually. Each subject was given an information problem (i.e. a query) and asked to find as many relevant documents as possible in 30 minutes using their preferred strategy or combination of strategies from those that were available. The subjects were also asked to write in a special prepared form the numbers (i.e. the IDs) of the documents they viewed and that they judged as being relevant to their query. This list of documents is called the "judgement list" of a subject. At the end of the search session each subject using the NIKOS system was given a questionnaire and was answered anonymously. Each question could be answered using a five-scale answer list where the middle response was neutral. If a subject did not experience the subject of a question, s/he could choose the sixth "No Opinion" response.

Each search session was logged and the data were analysed. Judged Recall (JR), which is the proportion of relevant material retrieved, and Judged Precision (JP), which is the proportion of retrieved material that is actually relevant, were used to measure the performance of information seekers. These are traditional measures of the effectiveness of information retrieval. The metrics JR and JP refer to �judged� recall and precision. In other words, these metrics refer to the judgement list that subjects produced during the experiment. Of course, it is possible that a subject has viewed a relevant document without recognising/judging it as relevant. So, we also measure the viewed recall (VR). It is also possible that a document is retrieved from an analytical search but never examined. Therefore we also measure the retrieved recall (RR). Table 4 defines JR, VR and RR. Also measured were the total number of different states (movements) that a subject produced during his/her session.

Table 4. Definitions for JR, VR and RR

table 4

3.4 Results and Discussion

3.4.1 Recall and Precision Results

Table 5 illustrates the effectiveness (R and P) results obtained using the NIKOS-based HDL (condition 2) and the Web HDL (condition 1).

Table 5. Effectiveness results of subjects using the NIKOS-based HDL (condition 1) and the Web-based HDL (condition 2)

table 5

Table 5 shows that subjects using the NIKOS OHS performed more effectively than subjects using the Web for JR and JP. It also performed better in respect to RR, while it performed worse in terms of VR. A difference can be regarded significant in information retrieval if it is larger than 5%. Due to the large standard deviation values, the differences shown in Table 2 are not statistically significant, however. Although the results cannot be validated statistically, they are quite encouraging, given that subjects in the NIKOS condition did not have any experience of using this system, while subjects in the Web condition had great experience.

Another useful result is the large difference in the states that searchers produced in the two different conditions (Table 6). On average searchers using the Web produced 129.20 states per session while subjects using the OHS produced on average 75.75 states. During each session all the movements from one Web page (hypermedia agent in the case of the OHS) to another were captured. Each movement was considered as producing a new state. T-test shows the differences are statistically significant (p < 0.007). These may be significant results because the different states (movements) that an information seeker makes during a search session to some extent determines the information load and the cognitive overhead that the searcher experiences. It is anticipated that information environments that can help users move through fewer states during a search session will be substantially more efficient. Normally fewer states will usually mean that users become less disoriented and less overloaded since they receive less information. It is also important to note that subjects using the OHS system moved through fewer states (i.e. were more efficient), while at the same time they produced at least equivalent results in terms or R and P.

Table 6. Efficiency results of subjects using the NIKOS-based HDL (condition 1) and the Web-based HDL (condition 2)

table 6

It must be said here that the way �new� states are produced in the two systems is quite different. In the Web-based HDL, users mostly searche using a single browser, so every movement from one page to another was producing a new state. Of course, in the first stages of their searching episode most Web users opened multiple browsers to have multiple views of the DL at their disposal, but after some point they usually �focused� on a single browser, probably due to the difficulty of coordinating multiple views (browsers). Similarly, in the OHS system a state was produced for each movement from one HA to another, but less movement was produced, probably because of HA coordination. We believe that fewer states can be attributed to providing multiple views in a more automatic fashion in the OHS condition, but to what extent is something that clearly needs further investigation. Nonetheless, the study and the results presented are indicative and encouraging for designs that could provide �multiple views� to information seekers in a more proactive manner. These designs may relieve information seekers from some tasks during their interactions and thus may result in more efficient digital libraries.

A careful examination of the �states� results in Table 6 reveals other interesting issues. It can be observed that subjects in the Web condition produced more of their movements in the second 15 minute period (55%) than in the first 15 minutes of the search session (45%). This searching behaviour may be partially explained by the fact that Web users at some point switched to �a single browser mode� and therefore produced more states. It also probably shows that subjects in the Web condition gradually became more opportunistic as they were searching for relevant documents and as they were acquiring more experience of the digital library under examination.

Users in the OHS condition, on the other hand, seem to become more eclectic and careful in the second 15 minutes of their search. They produced fewer states in the second 15 minutes of their search session than in the first 15 minutes. In general the authors believe it is a positive sign that users in the OHS condition became less opportunistic as their searching session progressed and they acquired more knowledge of the information space. We hypothesise this result may be a side-effect of the information seeking environment which allows parallel use of multiple viewers/tools and possibly examination of multiple search artefacts.

Finally, the fact that subjects in the OHS condition produced significantly fewer states/movement that usually indicates more efficient and less disorienting searches, is positive in an additional way. One of the main considerations in information seeking environments that promote the parallel use of multiple strategies, is the difficulty that users may find in using and coordinating such an environment. This experiment indicates that the parallel use of multiple strategies and interfaces may not necessarily be confusing or difficult to apply. The  opposite may be true, that it may provide some quality and rationalisation to an information seeking process.

3.4.2 Results of the Questionnaire

As has been stressed in the previous sections, the difficulty in using more complex interaction models is one of the main considerations of this work. This section presents the results of a qualitative evaluation of this issue. Subjects using the OHS (i.e. 12 users), after they had completed their search session, were asked to answer a questionnaire aimed at examining the user�s experience. The questionnaire was given only to users of the OHS digital library and not to the subjects that used the Web digital library, because the aim was to investigate how users felt about their experience in using a quite different information seeking environment (e.g. NIKOS) from that they normally use (i.e. the Web). Figures 6 - 11 present the results of the questionnaire. Each figure presents the question and the two marginal responses in the scale (i.e. the most positive has code 1 and the most negative has code 5). Neutral responses have the code 3 and N/O (no opinion) responses are presented with code 6.

One could say that the interaction model found in the OHS system is more complicated than that found on the Web and, therefore, users might be confused or they might find it difficult to use the system, or they may even become cognitively overloaded. It must be said that the results of the experiment reveal quite the opposite. Also, the practical experience gained from the experiment shows that a 15 minute presentation was enough for subjects to start using the system. By answering the questionnaire the subjects had the opportunity to express their experiences directly.

The first question explores whether subjects had any difficulty in understanding the data model of NIKOS OHS (Figure 6). This is a quite important question since the NIKOS system inherently supports a more advanced data model. It may be an important problem if users find it difficult to work in an environment organised using a data model more complex than the simple graph data model found on the Web. The responses shown in Figure 6 indicate that comprehension of the data model was not a problem for most subjects.

The next question (Figure 7) examines if the process model that the NIKOS OHS suggests for information seeking was difficult for subjects to understand. NIKOS� interaction model is based on the use of multiple HAs and is explained in Section 2.1. Information seekers can open and use multiple hypermedia agents concurrently, and they can �switch� from one hypermedia agent/strategy to another. The responses to this question show that almost no subject felt it was difficult to understand the information seeking process and interaction model promoted by NIKOS.

Question 3 (Figure 8) aimed to examine the general impression of the usability of NIKOS. Responses to this question show that most users found it easy to use the NIKOS system, despite its more demanding interaction model. One of the main considerations in using information seeking which utilises a more complex interaction model and multiple strategies is whether information seekers can cope with the difficulty of using and coordinating multiple interfaces. It is generally true that a user may become overloaded and may experience high cognitive overhead if s/he has tp use and coordinate multiple tools and interfaces during an information seeking process. This is especially true if we consider that besides the effort to use multiple tools and interfaces, an information seeker should effectively manage and organise the information seeking process and find all the relevant information which satisfies her/his information need.

The next question (Figure 9) refers to the difficulty that subjects could have in coordinating different programs (i.e. hypermedia agents) during their information seeking activities. Subjects in the NIKOS condition, according to responses in Figure 9, found it easy to coordinate and manage multiple hypermedia agents. Again, this result is indicative that the interaction model proposed by the NIKOS system is easy to use.

We believe that information seekers generally found the system easy to use and coordinate, basically because the purpose of each HA in the information seeking environment was quite clear, since each HA implements and represents a different information strategy. Information seekers may find such an environment enabling and useful because they view each different HA (i.e. interface) as complementary to the others available. Also, they may find the environment easy to use because during the information seeking process some of the coordination may happen automatically since HAs send messages to other HAs.

The next question, shown in Figure 10, aimed to explore the difficulty that subjects had in using and "switching" from one HA to another. Each HA represents a different information seeking strategy and therefore information seekers might find mentally or cognitively demanding an information seeking environment that promotes the use of multiple strategies and the switch from one strategy to another. Subjects unanimously expressed their opinion that it was easy to "switch" and use multiple strategies. Most users (66%) had a positive opinion about the combined use of multiple strategies in comparison with a single strategy (Figure 11).

figure 6

Figure 6. Was the data/organisational model easy to understand?

figure 7

Figure 7. Was the process model/information seeking process difficult to understand?

figure 8

Figure 8. Was the system easy to use?

figure 9

Figure 9. Did you find it difficult to coordinate the different tools/hypermedia agents?

figure 10

Figure 10. Was it difficult to "switch" from an information seeking strategy to another?

figure 11

Figure 11. What do you think for the following statement: "the combined use of multiple strategies is more effective than using a single strategy"?

Figures 6 -11. Results of the questionnaire from the 12 users of the OHS system

4 Limitations of the Experiment and Conclusions

Conducting user-centered experiments is difficult and usually bounded with limitations. First, because there are many different variables that must be taken into account. Second, it requires a large effort to organise a user-centered evaluation. The experiment described in this paper is large enough (24 subjects were involved conducting 24 search sessions), but clearly further experiments are required to reach statistically valid conclusions. Because the results cannot be validated statistically, the views and statements reported in this paper should be regarded as indicative and tentative. Another limitation of the experiment is the artificial method (i.e. using an agglomerative clustering method) used to produce and simulate an information-seeking environment of a distributed hypermedia digital library. Also the CACM standard collection used in the experimental study is relatively small (1751 documents). We intend to conduct larger experiments using larger collections (e.g. the TREC collection) to further investigate the issues presented in this paper.

Despite the limitations of our study, we do not consider this a primary problem because the aim of this paper was not to provide conclusive results. It is generally difficult to reach conclusive results in any user-centered experiment given the limitations in the number of users that can practicably participate. Our work in this paper was driven by the need to examine two different interaction models for information seeking environments. It was also driven by the need to examine and compare OHSs and the Web from an information seeking perspective. Advocates of rich and flexible information seeking environments hypothesise that these environments will be more effective and efficient for information seekers. Others believe that the simplicity and consistency found in environments like the Web can help users to produce better results. The experiment presented in this paper is a small contribution to this debate between flexible and more open user interfaces in comparison with consistent and �over-engineered" interfaces. We believe the experiment could give some insights into the design of information seeking environments, and has revealed issues thta need further consideration and closer examination.

Another potential contribution of this paper may be to shift attention to OHSs from an information seeking perspective. As was mentioned in Section 1, OHS research has focused on linking mechanisms and architectures. We believe it would be interesting to investigate OHSs as information seeking environments. It may also be useful to examine how the design of the Web may benefit from the information-seeking paradigm presented in most OHSs (i.e. multiple tools/interfaces can coexist).

Besides the issues mentioned above, several other aspects could be considered based on the results produced from the experiment. First, the effectiveness results (i.e. JR, JP, VR and RR illustrated in Table 5) indicate that environments in which information seekers use multiple strategies in a more parallel fashion may be at least equally effective as environments that follow a more serial and less parallel approach. The differences observed have not been statistically validated so more experiments are required to identify if and to what extent the use of multiple strategies has any effect (negative or positive) in information-seeking performance.

Second, the number of states/movements that information seekers produced using the two different environments (Table 6), suggests that information seeking may be more efficient in an environment that promotes parallel use of multiple strategies. This is a quite important result because the number of movements partially determines the cognitive overhead that an information seeker experiences. It is also important because information seekers moved through fewer states while at the same time they demonstrated at least equal effectiveness in comparison with the Web system. Results also indicate that the use of a single interface, as in the Web, drives information seekers to become more opportunistic and to change easily from one �information view� to another.

Third, the results of the questionnaire from the users of the OHS indicate that most users agree with the combined use of multiple strategies. As has been reported in similar studies, there was a small proportion of users which found it difficult to use many strategies in parallel. However, most users found it quite easy to comprehend and use a system that advocates a more parallel interaction model and requires some coordination during the information seeking process. We should say here that most of the subjects who participated in the experiment were computer literate and therefore that issue should be further examined by considering a less experienced group of users.

It could be said that from an information seeking and digital library perspective, OHSs are closer to the idea of rich information workplaces than the Web. This is mainly because of their open architecture, which makes them more flexible and extensible. Alternatively, there is a powerful mechanism to extend the Web using server-side or client-side scripts (i.e. CGI scripts). However, CGI scripts are basically used in a proprietary manner, and therefore in principle may be less effective than the type of extensibility promised by most OHSs.

The discussion and the experiment presented in this paper may be useful to the designers of digital libraries based on the Web because it indicates that information seekers in a Web-based digital library may benefit if it:

  1. Could offer many information seeking strategies . This should be a primary goal for any information system since it allows the development of rich information workplaces.
  2. Offers multiple strategies, but it also supports parallel, interleaved use.
  3. Supports incremental learning and selective utilisation of different tools/Web clients in the electronic environment.
  4. Allows interoperation and synchronisation between different information seeking strategies/Web clients.

In conclusion, we have illustrated that �open� and parallel information seeking environments could be at least equally effective and potentially more efficient than other designs based on more serial and �over-engineered� interaction models. More and larger experiments are required before we can reach a general conclusion, but our experiment has produced indications advocating the development of an information seeking environment that may be based on similar principles to those that inspired the development of open hypermedia systems.


We would like to acknowledge Professor John Tait of the University of Sunderland, UK, for his comments on an earlier draft of this paper, and our anonymous referees for their help in the preparation of this paper in its current form.


Allan, J., Leuski, A., Swan, R. and Byrd, D. (2001) "Evaluating combinations of ranked lists and visualizations of inter-document similarity". Information Processing & Management, Vol. 37, No. 3, 435-458

Anderson, K. (1997) "Integrating Open Hypermedia Systems with the World Wide Web". In Proceedings of the Eighth ACM conference on Hypertext, Southampton, UK, pp. 157-166

Andrews, K., Kappe, F. and Maurer, H. (1995) "The Hyper-G Network Information System". JUCS 1, 4, 206-220

Belkin, N. (1980) "Anomalous states of knowledge as a basis for information retrieval". Canadian Journal of Information Science, 5, 1, 133-143

Belkin, N., Chang, S., Downs T., Saracevic, T., and Zhao, S. (1990) "Taking account of user tasks, goals and behaviour for the design of online public access catalogs". In Proceedings of the 53rd annual meeting of the American Society for Information Science, pp. 69-79

Belkin, N., Marchetti, P., Cool, C. (1993) "Braque: Design of an Interface to Support User Interaction in Information Retrieval". Information Processing and Management, Vol. 29, No. 3, 325-344

Belkin, Nicholas J., Cool, Colleen, Stein, Adelheit, and Thiel, Ulrich (1995) "Cases, Scripts and Information Seeking Strategies: On the Design of Interactive Information Retrieval Systems". Expert Systems and Applications, Vol. 9, No. 3, 379-395

Bonder, R., Chingell, M., Charoenkitkarn, N., Golovchinsky, G. and Kopak, R. (2001) "The impact of text browsing on text retrieval performance". Information Processing & Management, Vol. 37, 507-520

Carr, L., Hollom, R., Hall, W. and Davis, H. (1995) "The Distributed Link Service: A Tool for Publishers, Authors and Readers". In Proceedings of the fourth international World Wide Web conference, Boston, USA, pp. 647-665

Carr, L., Hall, W. and Hitchcock, S. (1998) "Link Services or Link Agents?" In Proceedings of the ninth ACM conference on Hypertext and Hypermedia, Pittsburgh, USA, pp. 113-122

Chen, C. (1999) "Visualising semantic spaces and author co-citation networks in Digital Libraries". Information Processing & Management, Vol. 35, No. 3, 401-420

Croft, B. and Thomson, R. (1987) "I3R: A new approach to the Design of Document Retrieval Systems". Journal of the American Society for Information Science, Vol. 38, 389-404

De Roure, D., Hall, W., Reich, S., Hill, G., Pikrakis, A., Stairnand, M. (2001) "MEMOIR � an open framework for enhanced navigation of distributed systems". Information Processing and Management, Vol. 37, 53-74

Dunlop, M., Johnson, C. and Reid, J. (1998) "Exploring the layers of information retrieval evaluation". Editorial for second special issue on HCI and IR of Interacting with Computers, Vol. 10, No. 3, 225-236

Fox, E. (1983) "Characterization of Two New Experimental Collections in Computer and Information Science Containing Textual and Bibliographic Concepts". Technical Report TR83-561, Cornell University, September

Frisse, M. (1988) "Searching for Information in a hypertext medical book". Communications of the ACM, Vol. 31, No. 7, 880-886

Golovchinsky, G. (1997) "What the query told the link: The integration of hypertext and information retrieval". In Proceedings of Hypertext '97, Southampton, UK, April (ACM Press), pp. 67-74

Groenbaek, K., Bouvin, N. and Sloth, L. (1997) "Designing Dexter-based hypermedia services for the World Wide Web". In Proceedings of the eighth ACM conference on Hypertext, Southampton, UK, pp. 146-157

Hancock-Beauliue, M. (1990) "Evaluating the impact of an online library catalogue in subject searching behaviour at the catalogue and at the shelves". Journal of Documentation, Vol. 46, No. 4, 318-338

Halasz, F. and Schwarz, M. (1994) "The Dexter Hypertext Reference Model". Communications of the ACM, Vol. 37, No. 2, 30-40

Hall, W., Davis, H. and Hutchings, G. (1996) Rethinking Hypermedia: The Microcosm Approach (Kluwer Academic: Boston)

Hendry, D. and Harper, D. (1996) "An architecture for implementing extensible information-seeking environments". In Proceedings of the Nineteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Zurich, Switzerland, pp. 94-100

Hendry, D. and Harper, D. (1997) "An informal information-seeking environment". Journal of the American Society for Information Science, Vol. 48, No. 11, 1036-1048

Kaski, S., Honkela, T., Lagus, K. and Kohonen, T. (1986) "Creating an order in digital libraries with self-organizing maps". In proceedings of WCNN�96, World Congress on Neural Networks, San Diego, September, pp 8-14

Lalmas, M. and Ruthven, I. (1999) "A framework for investigating the interaction in information retrieval". In Proceedings of the 9th European-Japanese Conference on Information Modelling and Knowledge Bases (EJ-IMKB), Iwate, Japan

Nurnberg, P. and Ashman, H. (1999) "What was the question? Reconciling open hypermedia and World Wide Web research". In Proceedings of the tenth ACM conference on Hypertext and Hypermedia, Darmstadt, Germany, pp. 83-90

Nwana, H. (1996) "Software Agents: An Overview". Knowledge Engineering Review, Vol. 11, No. 3, 1-40

Osterbye, K. and Wiil, U. (1996) "The Flag Taxonomy of Open Hypermedia Systems". In Proceedings of the seventh ACM conference on Hypertext, Washington, DC, USA, pp. 129-139

Pejtersen, A. M. (1989) "A Library System for Information Retrieval based on a Cognitive Task Analysis and Supported by an Icon Based Interface". In ACM, SIGIR Conference Proceedings, Boston

Qiu, Liwen (1993) "Markov Models of Search State patterns in a Hypertext Information Retrieval System". Journal of the American Society for Information Science, Vol. 44, No. 7, 413-427

Rada, R. and Murphy, C. (1992) "Searching versus Browsing in Hypertext". Hypermedia, Vol. 4, No. 1, 1-10

Rao, R. et al. (1992) "The Information Grid: a Framework for Information retrieval and Retrieval-Centered Applications". In Proceedings of the fifth annual ACM symposium on User Interface software and technology (UIST 92)

Rao, R., Pedersen, J., Hearst, M., Mackinlay, J., et al. (1995) "Rich Interaction in the Digital Library". Communications of the ACM, Vol. 38, No. 4, 29-39

Rizzo, R., Allegra, M. and Fulantelli, G. (1999) "Hypertext-like structures through a SOM Network". In Proceedings of the tenth ACM conference on Hypertext and Hypermedia, Darmstadt, Germany, pp. 71-72

Salampasis, M. (1997) "An Agent-Based Hypermedia Digital Library". In Proceedings of the third workshop on Open Hypermedia Systems, Hypertext 97, Southampton, England, April, pp. 116-125

Vickery, B. and Vickery, A. (1993) "Online search interface design". Journal of Documentation, Vol. 49, No. 3, 103-121

Voorhees, H. (1986) "Implementing Agglomerative Hierarchic Clustering Algorithms for Use in Document Retrieval". Cornell University, Computer Science Department Technical Report TR86-765, March

Wiil, U. K., Leggett, J.J. (1996) "The HyperDisco Approach to open hypermedia systems". In Proceedings of the 7th ACM Conference on Hypertext, pp. 140-148


[1] The term Digital Library (DL) is used to designate very large, highly interactive, highly dynamic and distributed electronic environments (Fox et al. 1995). Hypermedia Digital Libraries (HDLs) are digital libraries based on a hypermedia paradigm. The Web can be regarded as the first large-scale HDL ever used.
[2] In addition to the simpler directed graph data model which is used in the Web.
[3] We assume that the analytical searching services that are provided by search engines (e.g. Altavista) are part of the Web.
[4] A test collection is a document collection that comes together with a set of information problems (i.e. queries) and relevance assessments. More information about evaluation of information retrieval can be found in Dunlop et al. (1998).
[5] The study that is presented here is part of a larger experiment. This paper presents the results of the questions (6 out of 17) that are more relevant to the subject of this paper.