Adaptive Hypermedia System Interoperability: a 'real world' evaluation

Alexandra Cristea
Information System Department,
Faculty of Mathematics and Computing Science,
Technical University Eindhoven,
Den Dolech 2,
PO Box 513,
5600 MB,
Eindhoven,
The Netherlands,
a.i.cristea@tue.nl

Craig Stewart
Department of Electronic Engineering
Queen Mary, University of London
Mile End Road
London
E1 4NS, UK
craigs@elec.qmul.ac.uk

Tim Brailsford
School of Computer Science and Information Technology,
University of Nottingham,
Jubilee Campus,
Wollaton Road,
Nottingham,
NG8 1BB, UK
tjb@cs.nott.ac.uk

Paul Cristea
Digital Signal Processing Laboratory,
"Politehnica" University of Bucharest,
Spl. Independentei 313,
Bucharest 77206,
Romania,
pcristea@dsp.pub.ro

Abstract

Adaptive Hypermedia (AH) authoring is widely acknowledged to be complex and time consuming, yet this vital process is rarely evaluated. Recent research has approached the authoring problem by ensuring that previously created materials can be converted from one system to another. This paper evaluates the results of this research, specifically the creation of adaptive materials in MOT and their conversion and subsequent delivery in WHURLE. A group of technically experienced IT users who are novice AH authors were exposed to MOT and WHURLE during an introductory week long course. This paper interprets the results of these authors using a "write once, deliver many" paradigm of adaptive hypermedia creation.

1. Introduction

In recent years Adaptive Hypermedia has arisen as a response to the almost ubiquitous dominance of the "one size fits all" approach to hypermedia on the Web. Whether a person is surfing the web for a product (as a customer), medical information (as a patient), educational material (as a student) or information about governmental bureaucracy (as a citizen), web sites usually offer the same content to each visitor. Thus a person has to search through potentially large amounts of material much of which is often irrelevant to him or her.

Adaptive Educational Hypermedia (AEH)(Brusilovsky, 2002) deals with the issue of providing a personalized educational experience. Rather than a learner having to sift through every piece of content whether or not it is appropriate, an AEH system will adapt its presentation of content to the learner's needs - this adaptation being informed by a User Model.

However the creation of adaptive content can be a complex and time consuming task. Imagine an AEH system that adapts around the learner's Visual::Verbal Learning Style (Felder & Soloman, 2004). At its simplest an author would have to create two instances of the same lesson, one for those students who learn more effectively from primarily visual information and one for those who are more inclined to textual or verbal learning. As most AEH systems adapt to many more learner characteristics than this, it is easy to understand why, notwithstanding the global stride for customization, AEH have not yet been widely adopted in educational institutions.

One of the issues that aggravate the problem of AEH authoring is the fact that often each AEH has its own authoring system. Materials authored for a given system are only viewable within that system. As AEH systems undergo research and improvement (Brusilovsky, 2002), it can be almost impossible for a non-technical author to stay up to date. Also the system they are currently using may lose its support and development team, for example, as academic project funds dry up or the team moves onto new areas of research.

In response to this we have proposed a new paradigm for AEH authoring, "write once, use many" (Stewart et al, 2005), whereby an author only has to learn to author for a single AEH system, and the materials from this system can be converted into any other AEH system. This is of course only a single step towards a greater change: ideally all systems should be interoperable using a "many to many" methodology.

So far we have used the AEH system MOT as an authoring system for three other Educational Hypermedia systems, AHA! (Cristea & DeMooij, 2003a), WHURLE (Moore et al, 2001) and the commercial system Blackboard (Blackboard, 2005). It is hoped that each conversion will reveal the fundamentals required for a more generic conversion system. However each conversion system needs to be tested with the audience it has been designed for, the AEH authors themselves, as the aim of this work is not only to reduce author load but to ensure that the authoring process itself is as easy and trouble free as possible.

In this paper we describe an experiment in which we examine the authoring process. A class of 31 students has been introduced to authoring for MOT & WHURLE in an intensive short course; their progress and responses to this new methodology were recorded in a series of questionnaires.

The remainder of this paper is structured as follows: Sections 2 & 3 introduce the two AEH systems involved (MOT as an authoring environment and WHURLE as the delivery environment), with Section 4 briefly describing the conversion system (Stewart et al, 2005). Section 5 details the experimental settings, and Section 6 the hypotheses we evaluated. Section 7 presents the results. Finally we conclude and draw inferences for future work in this area in Section 8.

2. MOT Presentation

MOT (Cristea & DeMooij, 2003b) is a web-based generic adaptive hypermedia authoring system based on the LAOS framework (Cristea & DeMooij, 2003a). For the purpose of our current paper, this means that MOT allows the creation of domain concept maps (DM), containing the actual resources clustered as content alternatives, and the creation of lessons (GM), based on these domain maps, that allow a restructuring and filtering of the contents. These contents are stored in a MySQL database, which means that MOT's adaptation is based upon the queries sent to the database from the MOT delivery engine.

One of the interesting features of the lesson layer, which was of use to the students during the tests presented in this paper, is the functionality of pedagogical labelling of previous concept (attributes) from the domain maps. As Figure 1 shows, attributes (containing concept alternatives, such as text, figure, etc.) can be pedagogically labelled (e.g., the figure attribute is labelled "vis" for visual, and the text attribute is labelled "ver" for verbal, as according to the ILS learning style questionnaire (Felder & Soloman, 2004)).

figure 1: Weights and labels for the attributes of a MOT concept
Figure 1: Weights and labels for the attributes of a MOT concept

MOT was the content creation environment used by the students. For more information on MOT, refer to Cristea & DeMooij, 2003b.

3. WHURLE Presentation

WHURLE is an XML based, on-line integrated learning environment, which is designed to deliver content that is personalised to the needs of the learner. The learner is presented with a lesson, which is constructed from a collection of underlying educational resources. The basic lesson structure is defined in a Lesson Plan and filtered according to rules specified in the WHURLE user model (Moore et al, 2004).

The underlying content objects in WHURLE are resources called chunks. Each chunk is a single text file describing a conceptually discrete piece of information with no links to other resources (such as external web pages), written in the WCML (WHURLE Chunk Markup Language, an XML application). Owing to the flexibility provided by WHURLE's use of chunks (through the conditional transclusion of chunks appropriate to each learner (Moore et al, 2001)), adaptation may be implemented at the content level to determine which chunks are made available to each class (group or "stereotype") of learner.

Authoring materials for use in WHURLE is a time consuming task. Authors write material using standard XML editing tools (facilitated by preview stylesheets) - a daunting task for authors not well versed in XML.

4. MOT to WHURLE conversion

Authoring materials in MOT (section 2) and delivering these materials in WHURLE are both simple tasks. In the former an intuitive web-form based process is used to create and order materials; with the later, the learner only has to register for a given lesson and the relevant materials will be adapted for a personalized delivery.

The conversion itself is currently slightly more complex than either of these processes. Initially we must understand the similarities between the two systems. The GM (Goal and Constraints Model, describes lessons) used by MOT is a hierarchical structure organised by "concept". The Lesson Plan used in WHURLE is also a hierarchical structure, organised by "level". Each MOT concept has "attributes" and each WHURLE level has "chunks" (collected into a "page") that define the actual content. From this basic description we can begin to derive an initial conceptual mapping of MOT to WHURLE. MOT has several default standard attributes, of which "title" and "keywords" are common to WHURLE chunks. Therefore, in any conversion process it is necessary that these common elements are included in every chunk created.

The conversion system uses the MOT "weights" (shown in Figure 1) to map groups of MOT concept-attributes to a single WHURLE Chunk. Thus initially the correct weights have to be applied to each MOT concept-attribute. It is by using these that the MOT to WHURLE converter can identify which concepts are to be delivered to a given learner. Whilst there are a range of weights and labels that can be used during the conversion the students in this exercise were told to use the weights and labels in Table 1 to keep the authoring process straightforward. NOTE: a Concept Weight value of "0" defines the "common" elements (such as "title" and "keywords"), and as such defines which elements will be available to everyone.

Table 1: MOT weights & labels (Figure 1) used to identify which concepts would be appropriate for learners with a given preference for the ILS learning style (Felder & Soloman, 2004).

MOT Concept Weight MOT Concept Label WHURLE learner group
0 n/a Everyone
35 Vis Visual preference
75 Ver Textual preference

Using these weights and labels to group concepts for similar types of learners, the conversion system maps the MOT structure into a similar structure in WHURLE. Each of MOTs lesson concept groups are associated and transformed into WHURLE chunks, with each chunk being appropriate for a given learning preference. Table 2 shows how a series of MOT concepts (as described in Figure 1) can be associated using their weights & labels to create WHURLE chunks.

Table 2: Creation of WHURLE (white section) chunks from MOT (grey section) concepts.

MOT components WHURLE components
Attribute Weight Label "everybody" chunk Visual prefs chunk Textual prefs chunk
Title 0   x x x
Keywords 0   x x x
Introduction 0   x x x
Text 75 Ver     x
Conclusion 0   x x x
Figure 35 Vis   x  
Chunks: C1 C2 C3

Table 2 shows how using the concept labels as described in Figure 1, the conversion process will create three WHURLE chunks (C1, C2 & C3) by associating: all of the weights of "0" (C1), associating all of the weights of "0" and "35" (C2) and associating all of the weights of "0" and "75" (C3). Note that the "common" elements are placed in every chunk, therefore C2 and C3 contain the same elements as C1 - as "0" weight elements are common to every chunk being created. Once these weights and labels have been established, the conversion program will automatically convert the MOT lesson into a WHURLE LP with chunks, and register the LP within the WHURLE mysql database. In this way, for each concept, three types of possible display will result (Table 2): a chunk to be seen by all students ("everybody" chunk), one for students with visual preferences (visual prefs chunk), and one for students with textual preferences (textual prefs chunk). More details on this conversion process can be found at (Stewart et al, 2005).

5. Experimental Settings

A class of 31 students, in the 4th year of study for a technical Masters degree at the University of Bucharest, Romania, was required to attend an intensive week long course on Adaptive Hypermedia, within a Socrates mobility exchange course. The students were supposed to have a combined course of theory and hands-on experience. The week started with two half day lectures, whereby the basics of the subject were introduced, before moving onto discuss the specifics of the used systems. Specifically, students:

The assignment was performed by breaking the class down into six groups (of 5-6 students); in the last three days of the course each group was asked to:

  1. Create 2-3 MOT Domain Concept Maps, with approximately 5-10 concepts on the http://e-learning.dsp.pub.ro/mot/ MOT server
  2. The attributes of each concept were: title; keywords; introduction; text; conclusion and figure. With limits placed on the type and amount of content in each one (this was done so as that each group would not spend their limited time creating a vast corpus of information).
  3. Create a single MOT Lesson (Goal & Constraints Map) using their Concepts maps.
  4. Alter the lesson so that the weights and labels of each concept agreed with those described in Table 2.
  5. Run the "mot2whurle" conversion program and copy the files to WHURLE.
  6. Check that the WHURLE XML files are well-formed.
  7. Run and login to WHURLE to check that the lesson matches their design and make any necessary changes.
  8. Finally at the end of the week, each student was asked to complete a series of questionnaires: three generic SUS (System Usability Scale (Brooke, 1996)) questionnaires, one for each system (MOT, mot2whurle and WHURLE) and a single specific questionnaire designed to determine their level of knowledge about each system, as well as to gather non-statistical information.

The students were told from the very beginning that their response to the questionnaires will not affect their mark. In fact, the marks were given to them before the questionnaires were processed. Further details of the task each group undertook can be found at: http://wwwis.win.tue.nl/~acristea/AH-Ro/

During the course the students had access to support mechanisms, in the form of the course moderator (either in person or via email), and their own peer support mechanism within each group.

6. Hypotheses and Evaluation Goals

We have decided to evaluate different aspects of the experimental setup as follows. We wanted some generic information about the students' experience with MOT and WHURLE, although our experimental focus was on the conversion program, MOT2WHURLE, which has never been tested before. The reason why we also tested MOT and WHURLE separately is, on one hand, the possibility of letting the students express their opinions fully and unrestrictedly about all the separate parts of the experiment, and also because WHURLE was not previously evaluated via a SUS questionnaire.

For pure usability we used the SUS questionnaire three times. SUS is a simple yet flexible usability scale consisting of 10 questions, ideal for the generic assessment of a system's usability. The results of each question can not be considered on its own but should be summed to form a final SUS Score (from 0 - 100), the higher this score the more "usable" the system.

However, the SUS questionnaire did not answer some specific issues we wanted treated, and therefore we also added a generic questionnaire on these issues, such as the level of student understanding, the type of tasks performed, their specific difficulties, etc.

Some of the hypotheses that we checked with the help of the experiments (student work and questionnaires) are as follows:

  1. The systems (MOT, mot2whurle, WHURLE) are simple and intuitive to use, with a minimum amount of explanation.
  2. The students understood the theoretical background (Adaptive Hypermedia, LAOS, Adaptive Strategies) of these systems.
  3. The students understood the connection between LAOS and MOT.
  4. The students used MOT purely for authoring adaptive hypermedia, and perceived it as such.
  5. The students used WHURLE solely for delivering adaptive hypermedia, and perceived it as such.
  6. Students consider automatic conversion between one-to-many or many-to-many adaptive hypermedia systems useful.

Beside these hypotheses, the aim of the evaluation and testing was to gain information for further development of the conversion system in particular, and the other two systems.

7. Experimental Results

First let us analyze the numerical results, from the point of view of validating or refuting our hypotheses. Figure 2, Figure 3 and Figure 4 show the average SUS results in the form of a radar chart. As a simple interpretation, the more the chart resembles a star, the greater the students' conviction that the system is readily usable. Following each chart are the actual SUS scores.

figure 2: MOT SUS results.

Figure 2: MOT SUS results.

The SUS score for the MOT usability is 75%. There were 29 students answering, with an average variance of their overall usability estimation of 15%.

figure 3: WHURLE SUS results.

Figure 3: WHURLE SUS results.

The SUS score for the WHURLE usability is 66.6%. There were 28 students answering, with an average variance of their overall usability estimation of 19.1%.

figure 4: MOT2WHURLE SUS results.

Figure 4: MOT2WHURLE SUS results.

The SUS score for the MOT2WHURLE usability is 60.7%. There were 29 students answering, with an average variance of their overall usability estimation of 19%.

The scores above sustain to some degree hypothesis 1 with empirical numerical data. We can also draw the conclusion that, as the author-centred MOT received the highest SUS score (75%), we have chosen a comparatively simple and "usable" environment for authors to create adaptive materials within. WHURLE received a slightly lower SUS score (67%), which supports our view that using MOT as an authoring environment is advantageous for the author concerned. Finally the MOT2WHURLE conversion system had the lowest SUS score (61%) and as such requires, comparatively, the most work to ensure ease of use for AEH authors.

Hypothesis 1 is also sustained by the qualitative fact that the students were able, with minimal explanation, during a one-week period, to produce content with MOT, convert and visualize it in WHURLE.

Hypotheses 2 to 5 have to be extracted from the extended, specific questionnaire. Hypothesis 4 is validated by a vast majority of 25 students out of 29 selecting MOT to be an adaptive hypermedia authoring system (one student classified it as an adaptive hypermedia system, and the rest did not answer). Hypothesis 5 is validated by 21 students selecting WHURLE to be an adaptive hypermedia system.

However, the hypotheses 2 (average of about 70% with a dispersion over 24%) and 3 (average of 70% with 21,7% dispersion) are not validated conclusively. It seems that students were much more confident in their system usage, than in their comprehension of the theory behind it. Although this might be unfortunate from academic point of view, from the point of view of system testing, this means that the systems could be used even with a vague understanding of the theoretical background.

Hypothesis 6 is confirmed by the students' reply to the generic questionnaire. The students' replies were very positive to this question, with an average of 4.57 (out of 5), and variance of 0.77.

Furthermore, from the specific questionnaire we were able to gather some points of possible improvements of the systems that the students were asked to use, as free-style textual answers. By far the greatest number of suggestions and comments were made for the MOT2WHURLE conversion program, for example:

Question: "What are the major difficulties that you encountered when working with the MOT to WHURLE conversion program, in your opinion?"

Sample Answers:

Comments: The majority of these responses alluded to the problems with the conversion interface - indeed as the conversion program is an offline, command line system, the requirement for a more advanced interface was already suspected. From these comments it would seem that integrating the conversion system into MOT as an online process would be desirable. The comment addressing the correction of the "*.wcml" files alludes to a bug in the conversion program which is being addressed. There were also a few comments concerning MOT, however these were matters of additional functionality rather than fundamental operating principles (as suggested for the MOT2WHURLE program) - as would be expected from its comparatively higher SUS score.

Finally we asked our novice authors to suggest any other adaptation strategies that they would like to author in MOT and see used in WHURLE. They answered that "... sequential versus global adaptation strategy would seem interesting for me." and "Some combinations of visual and audio would make it more attractive". As the MOT system can author pedagogically flexible materials and WHURLE can deliver different types of adaptation (by use of different adaptation filters), these two suggestions could easily be implemented using these systems.

8. Discussion & Conclusion

This paper is, according to our knowledge, the first attempt to empirically test the conversion process between two completely separate adaptive hypermedia systems, by using one for authoring and the other for delivery of AEH materials. We have gathered data about the process, some of it validating our efforts into:

The comments gathered from our test authors, supports our primary and most important hypothesis that using a single authoring platform to write materials for multiple delivery systems is indeed greatly desired by authors of AEH contents.

Author comments however also indicate the great deal of additional work required to improve the user interface and connection between these systems, with many areas for improvement (or areas of misunderstanding) being highlighted. For future work, we are going to integrate the students' comments into improving the MOT2WHURLE conversion. In the meantime, MOT2AHA was also tested, so it will be interesting to extract commonalities between these processes.

Another important lesson learned from our experiments is that to create an AEH interoperability tool there must be either

The conversion between MOT and WHURLE alone does seem to imply that some similarity does exist. Both systems were developed entirely independently, and yet they employ a similar conceptual approach in their development. Current work upon MOT to AHA! and MOT to Blackboard conversion tools suggests however a degree of generality in MOT that extends beyond the limits of MOT & WHURLE similarity. Hence the goal of a "write once, use many" authoring and delivery system interoperability would seem eminently feasible, starting with a MOT-like common description of authoring content and dynamics. Of course, to have any hope of achieving this goal on the long term, the interoperability process and/or common description would require standardisation. At the moment there are standards for the structure of learning data (e.g. LOM (LOM, 2005) for metadata), however there are none for the dynamic aspects of an adaptive system. IMS LD (IMS LD, 2005) and IMS SS (IMS SS, 2005) both fall short of being able to fully describe the flexibility of an AEH.IMS LD is not a hypermedia design model, but a high-level framework for educational activities specification, therefore not dealing with specifics of adaptive hypermedia (reordering, hyperlinks, etc.). IMS SS does not deal with adaptive content or adaptive presentation, and its adaptive navigation model uses preconceived manifests. Therefore an author has to describe every outcome in the manifest, they can not automatically generate any lesson their learners require at run time - unlike most adaptive systems.

9. Acknowledgements

This work is supported by the Minerva Socrates project ADAPT (ADAPT, 2004) (101144-CP-1-2002-NL-MINERVA-MPP), and the Socrates mobility exchange program. SUS was developed as part of the usability engineering program in integrated office systems development at Digital Equipment Co Ltd., Reading, United Kingdom.

10. References