Café Jus: an Electronic Journals User Survey

Hazel Woodward,* Fytton Rowland, Cliff McKnight, Carolyn Pritchett and Jack Meadows
Department of Information and Library Studies, Loughborough University,
Loughborough, Leicestershire LE11 3TU, UK
*Pilkington Library
Email contact:


During 1996, the number of scholarly periodicals available in electronic form increased rapidly. The Café Jus project took advantage of this critical mass of electronic journals to mount a major user study with taught postgraduate students, research students and staff in various disciplines at Loughborough University. The main conclusions were that low-level technical problems are still a deterrent to the use of electronic journals; that people prefer not to read at length on screen, but printing out is slow; that commercial publishers tend to follow the lead of technology rather than consider the convenience of their users; that at present there is a significant need for user training, exacerbated by the variety of publishers' interfaces and their speed of change; and that free journals using HTML are preferred to commercial journals using PDF for convenience of reading, but they are likely to be regarded as of lower academic quality. The implications of these results for publishers and for the future of electronic journals are discussed.

1 Introduction

Electronic scholarly journals have increasingly become the focus for research and development in recent years (see, e.g. Rowland 1995). Most work has concentrated on three topics: technical developments in networking, browsers, search engines and interfaces to provide an acceptably user-friendly electronic environment; the conversion of print-only journals to a dual format (print and electronic); an examination of input and handling requirements. User studies have necessarily lagged behind, since they could not readily be carried out until electronic journals became more generally available. However, one major user study has been reported by Stewart (1996). Much of the early research in this area (e.g. Shackel 1991) simulated, so far as possible, real usage; but the people involved were typically electronic-journal enthusiasts, rather than a cross-section of ordinary journal users.

Alongside and, in a number of cases, preceding the development of dual print/electronic versions of existing journals by commercial publishers, scholars in a range of academic fields have begun to found new, usually electronic-only, journals. Unlike commercial electronic journals, which are normally priced along similar lines to their printed counterparts, or even tied to the print subscription, most of these journals are distributed free of charge over the Internet. In a number of cases, the founders of these journals see them as offering an alternative publication route, in place of some of the increasingly costly specialist products of commercial publishers (e.g. Harnad 1994, Odlyzko 1996). However, it has been observed that most of the free electronic journals have so far attracted rather few submissions, and therefore publish relatively few articles per year (Harter and Kim 1996). It should also be noted that the two models discussed here -- free journals on the Internet, and electronic versions of existing commercially published journals, sold on subscription, under site licensing arrangements, or by 'pay-per-view' -- are not the only possible models of scholarly communication systems. A full discussion of many possibilities is given in Okerson and O'Donnell (1995). The two discussed here actually exist, however, and are therefore susceptible to user studies.

The rapidly expanding numbers of both commercial and free electronic journals have emphasised the need for user studies, whilst also increasing the possibility of carrying them out. Many libraries are now taking steps to provide their users with access to relevant electronic journals. One pressing question, as discussed by Kidd (1997), is how best to do this. In 1995, Loughborough University Library began to make available, via its World Wide Web pages, a collection of links to electronic journals. Titles were selected initially from the free journals found on the Internet. The criteria for choice were that the journals were, first, peer-reviewed and, second, related to the teaching or research interests in the University. At the same time, it became clear that there would be a major growth in the availability of electronic versions of commercially produced journals during 1996 (Woodward 1997). The Library began negotiations with publishers to allow trial use of some of their electronic journals, but this initiative was overtaken, in part, by a new project started by the Higher Education Funding Councils (HEFCs) in the UK. Their Pilot Site Licence Initiative (PSLI) was designed to allow access by universities to the electronic versions of journals produced by the Institute of Physics and Academic Press (and, subsequently, the two Blackwell imprints) on favourable terms (Bekhradnia 1995). In addition, Chapman & Hall allowed Loughborough University free access to its electronic journals for the duration of the Café Jus project.

As a consequence of these developments, a total of some 300 refereed electronic journals were available to staff and students at Loughborough by mid-1996. The Library restructured its Web electronic journal pages to take account of this increased number and diversity of types. Separate pages were created for free electronic journals, free sample issues of electronic journals to which the University did not subscribe, the prototype Blackwell's Navigator service, and for the commercial electronic journals to which the University did subscribe. The last-named page had links to the home pages of each of the publishers involved. (This was because each commercial publisher had its own access control system which required entry via its own home page.)

By 1996, conditions at Loughborough were appropriate for implementing a detailed study of users' reactions to electronic journals. A range of titles was now available across a variety of subjects, with easy access from anywhere on the campus. In anticipation of this development, the Café Jus (Commercial and Free Electronic Journals User Study) project was initiated in 1995, with funding from the British Library Research and Innovation Centre. Its main aim was to examine what problems readers with differing subject and computing backgrounds experienced when using electronic journals. A supplementary aim was to compare readers' reactions to free and commercial electronic journals. The freely available journals can generally be accessed by a direct hypertext link from the library's Web page to the journal's own home page, and they are usually in HTML; commercial journals require registration on the publisher's home page, and the full texts are generally in PDF. Were the former considered as on a par with the latter by academic readers?

2 Methodology

In carrying out the study, it was decided to concentrate primarily on students taking master's courses. The rationale was that they formed a group which was expected to make considerable use of journals, but had not yet developed set ways of acquiring information from them. In addition, volunteers from these courses were expected to provide a sufficient number of responses to make inter-subject comparison viable.

A structured questionnaire was developed which could be used in conjunction with hands-on access to an electronic journal. It is shown in Appendix 1 (**if you have Acrobat Reader, go directly to Appendix 1 ). The questions were arranged under three headings - journal content, journal appearance and facilities offered by the journal. This questionnaire was tested with volunteers from the 1995/96 cohort of master's students in four departments - Computer Studies, Human Sciences, Information and Library Studies, Physical Education & Sports Science. The students were introduced to the electronic journals in groups, which met in the computer-based teaching laboratories of the Department of Information & Library Studies. An initial presentation on the project together with documentation was provided; then each student accessed an electronic journal, noting down responses to the questionnaire as their reading progressed. If they had time, they could move on to another journal. Dealing with the students in groups made it possible to provide advice and help in real time, and also ensured completion of the questionnaires.

The pilot study indicated that the planned organization of the investigation was acceptable, but that there were minor ambiguities in some of the questions (more especially those dealing with journal facilities). After modifications to the questionnaire, the main study was launched with master's students from the 1996/97 cohort. The same four departments provided students, along with two more - Civil Engineering and Economics. Appendix 2 shows the full lists of titles accessed by at least one student during the pilot and main studies.

For comparison, it was decided to obtain feedback on electronic journal usage from research students and academic staff. In this part of the project, users were given individual instruction in the use of electronic journals, and then left to use them on their own at their own desks. The method of obtaining data here was via a log sheet on which participants were asked to keep a record of their usage of electronic journals. Volunteers were sought mainly via electronic mail messages inviting participation, but this proved to be a good deal less effective as a recruiting tool than the group approach used with master's students.

3 Results

The pilot study involved 47 master's students, who returned 55 questionnaires. There was a fairly even gender balance (male - 27; female - 20) and UK/overseas geographical distribution (UK - 27; overseas - 20). The total number of electronic journal titles accessed was 13, seven of which were commercial and the remainder free. The main study involved a further 75 master's students, who returned 85 questionnaires. The gender balance was again fairly equal (male - 40; female - 35), but there was a lower proportion of overseas students (UK - 56; overseas - 19). Altogether 34 electronic journal titles were accessed, of which 24 were commercial. The shift towards commercial titles reflects primarily the growth in the number of these available between the pilot and the main study. In the pilot study, some 60% of the respondents did not regard the electronic journal they had scrutinized as being particularly relevant to their interests. By the time of the main study, more commercial titles had come on-stream, and only about 20% of the respondents now regarded the journals they looked at as not being particularly relevant to their interests.

One change made between the pilot and the main study was to introduce a question about the computer expertise of the respondents. This was to help identify any major differences between groups of students that might affect their handling of electronic journals. The overall distribution of expertise in the main study was found to be: limited expertise - 20%; moderate expertise - 50%; considerable expertise - 30%. There was no significant difference in this distribution for male/female or UK/overseas groups. In subject terms, only Computer Studies - where all students had considerable expertise - differed from the norm.

Apart from the master's students, feedback was obtained from eight staff on 38 attempts they made to access electronic journals, and on 29 attempts made by four research students. In both cases, the great majority of the accesses were to commercial journals. Overall, staff and research students were more experienced than the master's students both in using journals and in handling information technology.

3.1 Results: master's students

3.1.1 Relevance and quality

Respondents were asked to compare the overall quality of the articles in the commercial and free electronic journals they were examining with articles in the printed journals they knew. In the pilot study, the standards of commercial and free journals was judged to be much the same. However, in the main study, about 40% of the respondents thought articles in free journals to be of a somewhat lower quality than those in printed journals (as compared with only 7% saying this of commercial electronic journals). These figures have to be treated with caution. In the pilot studies, 58% of the responses related to commercial journals, compared with 42% related to free journals. In the main study, the imbalance was considerably larger: 80% related to commercial and only 20% to free journals. Consequently, the differences as regards quality are less well established in statistical terms than the figures might suggest. In addition, judging from student comments, a particular difficulty with free journals related to the range of their contents. Free electronic journals might include - along with research articles - news items, discussion, etc. The students tended to group these together with the articles, leading them to a lower estimate of the overall quality of the journal.

It is not surprising that the contents of commercial online journals were thought to be of similar standing to those of printed journals, since they were essentially parallel publications of the same material. The likelihood of being acquainted with the electronic journal increased between the pilot and the main study, in step with the increasing number of titles becoming available. The number of respondents who already read the printed version of the electronic journal they were examining almost doubled from the pilot study (18%) to the main study (32%).

3.1.2 Access

There were continuing problems in gaining speedy access to electronic journals. These almost entirely concerned the commercial titles. In the pilot study, 9% of the respondents experienced problems in accessing free journals compared with 48% of those accessing commercial journals. The corresponding figures in the main study were 13% and 33%. In the pilot study especially, though not exclusively, the IDEAL system (used for Academic Press' electronic journals) proved to be far from ideal. There could be considerable delays in gaining access at any time of the working day, but it was particularly evident for American connections after lunch in the UK. The existence of alternative sites did not seem to improve matters. For example, one student was told to log into the US San Jose site because there were too many connections to the UK Bath site. When he tried, he was refused entry to the San Jose site and told to log into Bath.

Most students expected to be able to download full-text articles that interested them to the screen. This often proved to be time-consuming. Some longer articles took so long to download that the session timed out, and students had to log in again. In its favour, IDEAL gives file sizes, and this assisted students in deciding whether or not to download. Not all the publishers provide this kind of help.

The following comments by students, in answering a question about whether they encountered difficulties, indicate the type of access problems that were found.

These and similar comments can be generalized into the following complaints:

3.1.3 Layout and navigation

The most frequent queries under this heading related to legibility, followed by the comprehensibility of the graphics. There was also only muted enthusiasm for the ease of examining tabular material and for the way colour was used to assist comprehension. (Generally similar results were found in the pilot survey.) In effect, less than half of the respondents felt that deciphering the journal contents was made as easy as possible. The problems were partly related to the Acrobat reader and its relationship with Netscape. Some students commented on this directly. For example, a number tried to improve legibility by enlarging the text. One student commented that this worked, "if you understand the [Acrobat] zoom function, which is definitely not intuitive!"

Students clearly had some difficulty in understanding how some features operated. To look at this further, it was necessary to look at titles which had been examined by a number of readers independently. The two titles selected for this purpose were Construction Management and Economics and Journal of Sports Sciences. An analysis of these showed that most confusion reigned over the question of whether hypertext links existed within articles. There was also some uncertainty in replies relating to the presence of hypertext links to other documents, whether the journal allowed browsing through tables and diagrams, and whether it supported searching across articles.

Although most students regarded the various icons as easy to use, there was often some confusion until they became accustomed to them. One problem was that the icon might not immediately call to mind the operation concerned. For example, some failed to guess that a man with a spanner represented 'services'. Another problem was misinterpretation of the activity involved. For example, some expected the 'back' button to return them to the contents page rather than the previous page of text. It was also not immediately clear to many how to exit from an article in Acrobat. In addition, the help facilities were not found to be very helpful. Indeed, a number of the students did not manage to find them. As one commented, "Could not find them - they should be obvious to novice users!"

The various problems encountered by groups of users trying to read the electronic journals can best be indicated by the account appended below (in abbreviated form) of a session undertaken with a group of Computer Studies students. The session was more fraught than most, even though the students had considerably more computer expertise than the average.

The first problem in this session was that one of the commercial publishers had changed the layout and appearance of its site (without prior notification). One student was thrown off the system completely with the message, 'System error - Netscape Type 2.02', and had to log in again. Another student said he had been going round and round trying to find the search engine. The alphabetical list of journals was squashed into the bottom left-hand corner of the screen, and was almost too small to be read. In addition, the green lettering on a white background proved difficult to decipher. Another student got into the Home page of the selected journal, but could not find the contents pages, nor could he find a way of accessing a document. This problem was finally resolved by a telephone call to the publisher. A further student received the message, 'The file ATM Font Database must be in the system folder. Please re-install ATM. Acrobat Reader will now quit.' He thought this meant he should move to another machine, but it was suggested he should try another document on the same machine. He got into that satisfactorily, but then received the message, 'An error occurred and has been logged.' He then found that he could not retrieve the article he wanted. After several further attempts, he commented, 'This is getting on my nerves.' A fellow student encountered the same error message, and tried every journal on the list. The same thing happened each time. He tried logging in again, and failed to gain access at all. Several students wanted to print articles, but had to wait so long that they eventually gave up.

3.1.4 Perceived advantages to the electronic journal

3.1.5 Perceived disadvantages to the electronic journal

The predominant view was that electronic journals were already useful for reading from a site with no nearby library, or for quick scanning and searching. However, they still need appreciable improvement if they are to compete with paper journals more generally. As one student commented: "Need more user trials before implementation. Avoid the 'Big Bang' introduction of the system. Patience!"

3.1.6 Comments from discussions in sessions

There were a number of comments relating directly to the publishers' approach and particularly relevant to the present project.

Some of the colours used, especially for hypertext links, were difficult to see on-screen (e.g. a combination of green and brown, which was particularly unsuitable for readers with colour deficiencies).

Apart from these, the overwhelming comment was that accessing and using many of the electronic journals (primarily the commercial ones) would have been difficult, or even impossible, without the handouts and oral guidance given during the sessions.

3.1.7 Comparison of electronic and printed journals

Two-thirds of the respondents saw the electronic versions as offering, in general terms, easier access to journals. They meant by this that all the journals they wanted could, in principle, be accessed from the same terminal. This removed the need to set aside special times for visiting the library, when they might find that the journals they wanted were already in use. When asked which type of journal they found it easier to use, a somewhat smaller proportion (59%) chose the printed journal in preference to the electronic journal. Computer-oriented students, especially in Information and Library Studies, claimed to find electronic journals easier to handle. (These figures are derived from the main study, but the responses in the pilot study were very similar.)

3.2 Results: research students and staff

For the most part, the assessment of electronic journals by research students and staff proved to be very similar to that derived from master's students. The following points were either new or more strongly emphasised by this group.

3.2.1 Access

On several occasions, the potential reader either failed to gain access, or did so only after considerable effort. As one respondent commented: "Online access remains inefficient and slow. It would often be quicker to go to the library, indeed to send a carrier pigeon, than to spend countless hours watching a picture of an egg-timer, especially when the end result is an error message, one of an infinite variety of reasons material cannot be accessed, or gibberish." There was a strong feeling that the average first-time user would be unlikely to persevere unaided. Sometimes, even identifying the contents could be something of a feat. As another respondent remarked: "Without obtaining help from the Library, I think I would have given up before even finding the contents page." In addition, readers were occasionally ejected unexpectedly from the journal they were reading.

The provision of a list of journals grouped into subject categories was not found very helpful. Complaints ranged from an excessive number of subject headings to the inclusion of irrelevant titles. It was felt that access requirements overall were far from intuitive (this included use of Adobe Acrobat).

3.2.2 Format

Most complaints related to illegibility. For example, the typeface was often described as too small, and, especially on a lower-resolution screen, as indecipherable. One particular point was the difficulty of reading mathematical formulae on-screen. The use by publishers of their Web site for trying to market other products along with providing access to their journals was described as irritating. One reader who had thought of embedding his own hypertext links in an article decided that the PDF format precluded this. Slow response rates were generally remarked. For example, graphics in headers obviously slow transfer. If publishers insist in including such material, they should at least make it clear that it can be switched off.

One summary comment on access and format may be quoted: "I wish that [a specific commercial publisher] would go away and die somewhere - or at least their e-journal Web designers!"

3.2.3 Printout

One of the points made about legibility was that, if the typeface was too small, the natural response of the reader was to enlarge the portion of the text concerned. However, this left too little text on the screen to be easily readable. Consequently, the natural response was to read a print-out instead. All the staff involved said they would normally print out material for continuous reading. The exception was if the article contained important hypertext links. One reader found an issue of a journal where most of the articles were of interest. She did not have the time to read them all at work: at the same time, there were too many to print. She commented that this was an instance where a printed version of the journal would have been preferable.

In almost all cases, the time required for downloading and printing material was regarded as excessive. The explanation for this seems to lie in the large size of PDF files -- for most papers , 0.5-1.0 Mbyte -- and congestion on the network. One respondent noted that he allowed for this by making the process a background activity, while he continued with all his other computer activities.

Research students essentially echoed the comments of the staff, but with two particular emphases of their own. One related to back issues. Research students are particularly concerned with in-depth coverage of the literature. Hence they were frustrated by the paucity of back issues, and by the problems of searching across back issues where they did exist. The other query concerned the introduction of non-essential links, often to general Web sites, in the text of an article. A conscientious student who worked systematically through these would, it was noted, soon become irritated by the waste of time. One example quoted was an article about archival materials in the White House. The author had written: 'My day job is at the University of Michigan.' The link related neither to the subject of the article, nor in any specific sense to the author.

4 Conclusions

Though computer experts may be at some advantage in coping with the idiosyncrasies of electronic journals, none of the factors - computer expertise, gender, status and home/overseas background - had any relation with the identification of problems. This similarity of opinion of all groups in their assessment means that firm conclusions can be drawn on the basis of the present survey. The major points to be made are as follows.

It is possible to look at these comments in a different way: they highlight the actions that publishers of electronic journals need to take in order to improve the acceptability of their product among readers. The survey results are therefore reconstructed below in terms of guidelines to publishers.

  1. Do not launch an electronic-journal service publicly without testing it with a pilot group of real users first; take note of their experiences in using it to redesign it before public launch. Do not allow yourself to be pushed by the technologists; remember the user.
  2. Do not have too many graphics in your Web page design. Although it is possible for users to disable graphics in their Web browser, näive users will not know that, and will suffer low delivery speeds owing to the graphics.
  3. Do not have unnecessary extra steps in the chain of Web pages leading from your home page to the actual full-text articles. At busy times each page in the chain may take several seconds to deliver.
  4. Once the service is launched, do not make frequent changes to the interface or software. If you must change them, test the new versions again as in (1) above, give plenty of warning on your Web pages that they are going to change on a certain date, and leave an explanatory note about the changes on your home page for some time after making the changes.
  5. Make journals available through all the various 'one-stop shop' vendors now being developed. In this way, each university can choose one such vendor as standard, and then be able to present all its electronic journals to its users with a standard interface and a single password. If some publishers boycott the one-stop shops, this advantage is lost and the progress of electronic journals in general will be retarded, to the detriment of all publishers.
  6. Think seriously about the format in which full texts are presented. Adobe Acrobat/PDF is the current de facto standard, but it is slow to deliver articles, its interface is not as intuitive as it should be, and page integrity means that the article pages are the wrong shape for a VDU screen. Double-column layouts are particularly troublesome for users. PDF is a good document delivery system, and will probably remain standard during the transitional dual-publishing era, but for real electronic publishing, HTML, SGML, HyTime or XML will ultimately be used.
In terms of future development, the results of this study suggest that publishers face three categories of problem depending on the factors involved.


We wish to thank the British Library Research and Innovation Centre for a grant supporting this investigation.


Bekhradnia, B. (1995) "Pilot national site licence initiative for academic journals". Serials, 8(4), pp 247-250.

Harnad, S. (1994) "Publicly retrieval FTP archives for esoteric science and scholarship: a subversive proposal" In Scholarly Journals at the Crossroads: A Subversive Proposal for Electronic Publishing, edited by Ann Okerson & James O'Donnell, June 1995 (Washington, DC.: Association of Research Libraries) http://cogsci.soton.

Harter, S. P., and Kim, H. (1996) "Electronic journals and scholarly communication: a citation and reference study". American Society for Information Science (ASIS) Midyear Meeting, San Diego, CA, May http ://

JSTOR (1998) "About JSTOR".

Kidd, T. (1997) "Are print journals dinosaurs?" Ariadne, issue 12, November, pp 6-7

Odlyzko, A.M. (1996) "Tragic loss or good riddance? The impending demise of traditional scholarly journals". In Scholarly publishing: the electronic frontier, edited R.P. Peek and G.B. Newby (Cambridge, Mass.: MIT Press) pp. 91-101

Okerson, A.S. and O'Donnell, J., eds (1995) Scholarly Journals at the Crossroads: A Subversive Proposal for Electronic Publishing (An Internet Discussion About Scientific And Scholarly Journals And Their Future) (Washington, D.C.: Association of Research Libraries)

Rowland, F. (1995) "Recent and current electronic journal projects". In Project ELVYN: An Experiment in Electronic Document Delivery, edited by F. Rowland, C. McKnight and J. Meadows, (East Grinstead, UK: Bowker-Saur) pp. 15-36

Shackel, B. (1991) BLEND-9: Overview and Appraisal, British Library Research Paper 82 (London: British Library)

Stewart, L. (1996) "User acceptance of electronic journals: Interviews with chemists at Cornell University". College and Research Libraries, 57(4), pp 339-349

Woodward, H. (1997) Electronic journals in an academic library environment. Serials, 10(1), pp 53-57

Appendix 1: Commercial And Free Electronic Journals: User Studies: user questionnaire

Presented in PDF to preserve page layout of original questionnaire.

Download latest PDF viewer

Appendix 2: e-journals accessed by students during pilot study

Table A1. E-journals accessed by students during pilot study

Journal [Publisher/free]

No. of uses

Appetite [Academic Press] 3
Ariadne: magazine of Internet issues for librarians and information specialists [Free] 2
Computer Speech and Language [Academic Press] 1
Collaborative Computing [Chapman & Hall] 3
International Information and Library Review [Academic Press] 2
International Journal of Human Computer Studies [Academic Press] 8
Journal of Network & Computer Applications [Academic Press] 1
Journal of Sports Sciences [Chapman & Hall] 13
Journal of the International Academy of Hospitality Research [Free] 3
Library & Information Science Research Electronic Journal [Free] 3
PACS Review [Free] 3
Sports: Coaching Science Abstracts [Free] 8
Swimming Science Journal [Free] 4

Table A2. E-journals accessed by students during main study

Journal [Publisher/free]

No. of uses

Appetite [Academic Press] 1
Applied Economics [Chapman and Hall] 1
Applied Economics Letters [Chapman and Hall] 1
Applied Financial Economics [Chapman and Hall] 2
Ariadne: magazine of Internet issues for librarians and information specialists [Free] 2
Building Research and Information [Chapman and Hall] 1
Construction Management and Economics [Chapman & Hall] 9
Gynecologic Oncology [Academic Press] 1
ELH (English Literary History) [Project Muse] 1
Gender and History [Blackwells] 1
International Information and Library Review [Academic Press] 2
International Journal of Human Computer Studies [Academic Press] 5
Journal of Architecture [Chapman & Hall] 1
Journal of Artificial Intelligence Research [Free] 1
Journal of Business Ethics [Blackwell's] 1
Journal of Environmental Psychology [Academic Press] 2
Journal of Experimental Social Psychology [Academic Press] 1
Journal of Financial Intermediation [Academic Press] 1
Journal of Information, Law and Technology [Free] 1
Journal of Programming Languages [Chapman & Hall] 1
Journal of Sports Sciences [Chapman & Hall] 18
Journal of the International Academy of Hospitality Research [Free] 3
Journal of Visual Communication and Image Representation [Chapman & Hall] 1
Leisure Studies [Chapman & Hall] 1
Library & Information Science Research Electronic Journal [Free] 1
Managing Leisure [Chapman & Hall] 1
Network: Computation in Neural Systems [Institute of Physics] 1
Organizational Behavior and Human Decision Processes [Academic Press] 1
PACS Review [Free] 1
Postmodern Culture [Free] 1
Psycoloquy: a refereed journal of peer commentary in psychology [Free] 1
Swimming Science Journal [Free] 2