Abbreviations

A2A Access to Archives programme (the English strand of the UK archives network) – www.a2a.pro.gov.uk

AIM25 Access to the archives of higher education institutions and learned societies in London and the M25 Area project – www.aim25.ac.uk

Archives Hub Access to the archives of UK universities and colleges project – www.archiveshub.ac.uk

COPAC union catalogue providing access to the merged online catalogues of members of the UK Consortium of University Research Libraries (CURL)

FLC Forum Learning Centre, Wythenshawe Library

HMC Historical Manuscripts Commission

IHR Institute of Historical Research, University of London

NANURG National Archives Network User Research Group

NLS National Library of Scotland

NCA National Council on Archives

PRO Public Record Office

PROCAT Public Record Office’s Online Catalogue

SCAN Scottish Archive Network – www.scan.org.uk

UCL University College London

List of Figures

Figure 1 – Usefulness and Information Value of Sites’ Content............. 16

Figure 2 – Ease of Navigation & Return to Home Pages....................... 21

Figure 3 – Visual Appeal and Layout of Sites.......................................... 39

I am grateful to the staff of all the archives who participated in the NANURG team and collaborated with me on this project. Special thanks are due to Joanna Baird (SCAN), Robert Baxter (AIM25), Michelle Bell (Archives Hub), Alan Borthwick (SCAN), Sarah Flynn (A2A), Stuart Low (SCAN), and Julie Tancell (AIM25), for their valuable guidance into the archival world and their commitment and enthusiasm to explore new ideas and methods in user evaluation, as well as from their meticulous note-taking and observations.

I would like to thank all the contributors to the study (listed in Appendix I) for the constructive feedback they offered during the evaluation session and for taking the time to think, discuss, and write about various issues related to the use of the archive sites tested during this project.

I would also to thank Dr Seamus Ross (Humanities Advanced Technology and Information Institute, University of Glasgow) for introducing me to this project and for his continuous support of my work.

1. Summary

1.1 Formative qualitative evaluation of the four archive strand websites, A2A, AIM25, Archives Hub, and SCAN, which are part of the National Archive Network and represent work in progress, showed that the websites appeal to a diverse audience, were received positively, and offer great potential for further development and use.

1.2 At the same time, all the users who participated in the study, including the ones more experienced in archival research, had problems with navigation and use of the sites.

1.3 Based on observation of participants’ use of the sites, analysis of the questionnaires and the focus group discussions, a number of recommendations are made for further development and improvement of the tested four sites in order to address the needs of a diverse range of users, from school pupils and the general public to specialised researchers and academics. These are detailed as they relate to each site at the beginning of every relevant section (sections 4 to 9), while general observations and key points are summarised here.

1.4 At an appropriate moment in their development (which for some sites might be after more material has been added or more work on the design has been completed), the programmes would need to consider wider publicity and advertising at different levels to increase the profile of the sites among the various targeted users, as even some archive professionals or researchers had not used the sites before or were not aware of their existence and the area that these covered.

1.5 Focusing on improving usability, searching, navigation and on making the sites simpler and more user-friendly would satisfy not only the more general part of the audience, but also specialised researchers who appeared to have difficulties with some of the more advanced search features. It would be more effective to layer the different searching options, presenting in the first pages the more simple ones, while providing clear links to other pages for further exploration and more sophisticated searching for those interested.

1.6 For specialised users, key features that they would like to see in the future include further work on adding more content and making the sites more comprehensive; making clearer each site’s remit, while trying to avoid areas of overlap; and greater consistency in the design of all four strands.

1.7 Several users who were positive about the potential of the sites were hindered by their limited understanding of the function of archives, catalogue descriptions, and the language used. As an example, from the two Manchester groups who represented the least specialised part of the audience: (i) more than a quarter who rated highly the usefulness and information value of the content of A2A, (ii) more than a third who rated AIM25 and the Archives Hub highly and (iii) over seventy percent who rated SCAN highly, had difficulties understanding the nature of archival records and catalogues. In order to appeal to this audience, attractive and effective ways of explaining archival work on the sites are needed. Beginner guides or introductions might also be useful, as well as more assistance in interpreting search results. More work on the design of the webpages, greater use of colour and graphics, and use of interactive elements such as comments pages and notice boards would appeal to younger users. SCAN is currently more effective in this area, employing different techniques. It is important to test further with a range of users from these groups any initiative in this direction.

1.8 The sites currently meet the needs of users with disability and minority interest groups to a varying degree. More work would need to be carried out in this direction.

1.9 The design of the home pages needs to be considered further by all sites. It is important to offer brief, succinct descriptions of the aims of each programme and the material it covers, while explaining the acronyms used.

1.10 All programmes, and particularly A2A, AIM25 and the Archives Hub, need to consider greater use of images and visual information. However, these need to be employed only if they serve each site’s aims, are cost effective, and do not distract from making the material more comprehensive. Providing audio has potential for the future for specific applications, but should not be seen as a priority at this stage.

1.11 Users welcomed the envisaged development of a common gateway linking all strands. This would help answer a lot of users’ points about providing common interface and search mechanisms between the four strands. It would also help to pool the resources which would be required for developing the sites further for novice users. It would be desirable to design this so that users can have the option to also search the individual strands separately. The interface of the gateway would need to be carefully thought out to maintain flexibility and ease of use, while work on the technical side should ensure that searching across all sites would not be slower than is currently at the participating sites.

1.12 As the more specialised archive users who tested the HMC/NRA links generally found them useful, it is recommended that these are extended to the other network strands. Suggestions for the design of the user interface of these pages are made in paragraph 7.14.

1.13 Finally, it should be noted that all participants, even those who had serious problems during the exploration of the sites, were generally complimentary and supportive of the initiative and offered constructive suggestions.

2. The Brief

2.1 The report addresses the ‘Brief for evaluation research’ issued by the National Council on Archives and the National Archives Network User Research Group in October 2001.

2.2 This outlined the terms of user evaluation research (of mainly qualitative nature) to study the websites of four main strands in the UK archives network:

· A2A Access to Archives programme (the English strand of the UK archives network) – www.a2a.pro.gov.uk

· AIM25 Access to the archives of higher education institutions and learned societies in London and the M25 Area project – www.aim25.ac.uk

· Archives Hub Access to the archives of UK universities and colleges project – www.archiveshub.ac.uk

· SCAN Scottish Archive Network – www.scan.org.uk

2.3 The idea of a National Archives Network was first set out in 1998. Two reports which were published that year by the National Council on Archives, 'Archives On Line: The Establishment of a United Kingdom Archival Network' and 'British Archives: The Way Forward', were significant for the development of the Network[1]. At that time archive collections in the UK were difficult to access, as there was little way of discovering what existed other than to contact individual archive repositories. The four archive strands covered in this evaluation brief were among several archives projects launched in order to make descriptions of archive collections held within the UK freely available via the Internet. Essentially, each project is a searchable database with its own unique interface and searching facilities. All four websites tested represent work in progress.

2.4 The user evaluation study was conceived in order to provide recommendations in the following areas:

· Design of project websites

· Content of project websites

· Development of the National Archive Network in general

2.5 The evaluation study focused on the formal evaluation of all four projects on the same terms (with users exploring them in alphabetical order during all sessions). It consisted of focus group sessions for users from various communities (school pupils, library users, students, family and local historians, academics, professional researchers and others) held in three venues across the UK. At this stage in the sites’ development, it was felt that qualitative rather than quantitative analysis would be particularly useful for informing further developments, offering a rich layer of data from diverse user groups.

2.6 The study was funded after an award of £4,000 from Re:source, the Council for Museums, Libraries, and Archives. This covered a consultant’s fee to Dr Maria Economou (half-time Lecturer in Art Gallery and Museum Studies at the University of Manchester who also works as independent consultant in the museum and heritage field), and administration costs including venue hire.

2.7 The project was overseen by the National Archives Network User Research Group (NANURG) team and was co-ordinated by Sarah Flynn (A2A).

3. Methodology

3.1 After the initial meeting with the NANURG team in October 2001, the brief set the terms for a programme of user evaluation with various sessions taking place in December and analysis of the results in the following month.

3.2 During this period, six evaluation sessions were held on three dates in December 2001 at various venues in London, Manchester and Edinburgh. The users attending the sessions included school pupils, postgraduate students, academics, professional researchers, library users and members of the general public.

3.3 The scope of the study did not permit nor necessitate a large quantitative audit of the use of the four strand sites, while these are still being developed. The report focuses on the analysis of in-depth focus group discussions, comments and observations, rather than large-scale statistical analysis.

3.4 During all these sessions, after a brief introduction by the archive staff about each project, users were able to browse and search the four strand websites, complete a written questionnaire, and provide feedback and comments during a focus group discussion. The sites were explored in alphabetical order during all sessions (with A2A first and SCAN last), which might have affected users’ perception of the sites, their interface design and ease of use (particularly some of the lower ratings for ease of navigation and usefulness that A2A received). Some users reported on this explicitly, commenting on how increased familiarity might have affected their perception of the websites (e.g. paragraphs 8.6 and 8.43).

3.5 The evaluation sessions tested also the use of the links between the National Register of Archives and the National Archives Network.

London sessions – 5 December 2001

3.6 There were nine participants in the morning evaluation session at the Institute of Historical Research, University of London. The group comprised of students (most at PhD level) and researchers. Their interests varied from Anglo-Saxon literature, early modern English history, black women’s history, war studies in the twentieth century, Tudor diplomacy, and Caribbean studies.

3.7 The afternoon session in London took place at the Imperial College and included four participants, three PhD students and a history teacher in the City of London. Their interests were mainly in the history of photography and the history of science, particularly eighteenth century science and religion. The atmosphere in both sessions was relaxed and friendly, with most users already familiar with the AIM25 staff and the space used. The equipment used was of average specification and Internet connection speed, but there were no major technical problems. Most users were familiar with Internet searching and browsing and were regular archive users.

Manchester sessions – 11 December 2001

3.8 The participants in the morning evaluation session were ten Lower Sixth Form IT students from the Parrs Wood School. The atmosphere was relaxed, but the participants took the activities seriously and were motivated in attempting to work through the websites and questionnaire. About half made valuable contributions during the focus group, while the remainder seemed unwilling to be drawn into the discussion. They were well informed about the web technology, the relevant jargon, and appeared to have a good eye for design and functionality. The IT equipment in the room was very modern and to a high specification, with fast network connections for all the websites.

3.9 The afternoon session in Manchester took place at the Forum Learning Centre, Wythenshawe Library and was attended by thirteen adults, mainly older, retired participants (but with one woman in her twenties), and one library professional. Many had taken or were taking part in IT access courses at the Wythenshawe Forum Learning Centre, including one on the Internet, but were not experienced Web users. The session was friendly and lively with a lot of participants volunteering comments and opinions during the discussion to the point that it was hard a few times to keep the participants’ attention focussed on specific points. This was the session where participants needed the most help from NANURG members to get started and browse the sites. The computers available in the room were of average specification, with fairly slow web connections. The sites, and particularly SCAN, were generally slow to load. The participants in this session were taking longer to go through the sites and complained that they did not have enough time to complete the tasks. They were generally very task oriented, took the ‘ideas to get you started’ and the questions on the questionnaire literally and did not explore the sites further to follow their own interests. Users appeared to start grasping what the sites were offering only by the end of the session.

Edinburgh sessions – 13 December 2001

3.10 The sessions took place in a very well laid-out room in a commercial training centre with high-specification equipment, refreshments and free lunch for participants. They also differed from the London and Manchester sessions in that they were also attended by archivists and librarians, apart from archive users. There were nine and eight participants respectively in the morning and afternoon sessions. These sessions ran very smoothly, without any technical problems and with participants paying close attention to the sites they were testing, quietly working through the questionnaire and browsing the sites. The archivists and librarians who participated were asked to approach the sessions from a user’s rather than a professional perspective. Although all of them had used the Internet, some were not as familiar with it as others. Almost all of the users were based in Scotland.

3.11 A list of contributors to the study is given in Appendix 1.

REPORT OF KEY FINDINGS AND ISSUES RAISED

4. Familiarity and previous experience with the sites

Key issues and recommendations

- The degree of familiarity with the tested websites varied among the different user groups, but was generally relatively low (paragraphs 4.1-4).

- It is recommended that wide publicity is planned for all four sites at an appropriate stage in their development. This would need to be nation wide[2] and cover various different routes beyond channels related to the academic and archival community. Publicity information would need to clearly describe the remit of each site and the material it covers.

4.1 The degree of users’ existing familiarity with the four evaluated sites varied between the different groups. It was generally relatively low, even among the more experienced archival users, with no user having had previous experience of all the strand sites.

4.2 A small number of the London researchers and students had used the archive sites evaluated before, approximately one participant for each site in the two sessions.

4.3 None of the users in Manchester had used any of the sites before and most, particularly the younger evaluators, did not have any particular interest in the content of the sites (with the possible exception of one student and several members from the Wythenshawe Library group who were interested in family history). In fact, most users in these sessions did not have any real concept of what archives were. Several were obviously struggling to gain any idea of what they were searching, though only a few appeared to lose concentration and give up.

4.4 In the Edinburgh sessions, as was expected, several of the users were familiar with the SCAN site, but only a few had used the A2A and the Archives Hub, while no one had used AIM25.

4.5 Other similar websites which the London participants said that they use include the site of the Historical Manuscripts Commission (HMC) and the National Register of Archives, the Public Record Office site and its catalogue (PROCAT), COPAC, the Arts and Humanities Citation Index, specific library sources, such as the Wellcome Library, the History website of the Institute of Historical Research, and the British Library catalogue

4.6 The Manchester Sixth Form students mentioned that resources that they tend to use on the Internet for schoolwork and related projects are general search engines, like Google and AltaVista, for which they were generally complimentary because they saw them as very easy to use, simple, and retrieving usually relevant information.

4.7 The adult group at the Wythenshawe local library session mentioned also apart from the major web search engines, other general sites that they use, like the BBC one, whose constant updates were appreciated, and route planning and shopping sites. As they were novices of the web, they had been introduced to general interest sites and search engines during their classes. Some participants referred to family history sites, some of which they found difficult to use, and to the PRO site, which was seen as helpful. One participant talked about ‘Mormon CD-ROMs’ (i.e. the International Genealogical Index or similar at Church of Latter Day Saints family history centres), which participants thought were helpful but contained information only on the CD-ROM disks, without any online links. While they showed some tentative experience of having used archival sources for research, there was little other evidence of archive awareness or use. In general, they found the archive sites which were evaluated that day more difficult to use, but liked the ‘ideas to get you started’ in the questionnaire, as they gave them a clearer idea of what to do. One person also added that for specific project information, the archive sites would probably be more useful than Google.

4.8 Similar sites which the Edinburgh participants had used before included PRO, which was mentioned by several, ‘Tomorrow’s History: Images of the North’[3] referred to by one user, and library sites, such as the National Library of Scotland, the British Library, and the Library of Congress. These sites were all felt to be quick, useful, easy to load, search, and navigate, and offering good basic information. These users raised the point that usability depended on the level that the user was at and the level of information required, finding that library sites were useful for finding individual books, but perhaps not for general guidance on subjects or records.

5. Previous impressions and expectations from the archive sites

Key issues and recommendations

- Most users did not have any specific expectations from the sites before using them at the evaluation sessions. This is related to limited previous knowledge of archives in general (for the general public), or in the case of the more experienced participants, no previous use of the evaluated sites or the fact that they represented work in progress (paragraphs 5.1-5.7). Users did not have a clear idea of each site’s remit and contents either, while they found that the acronyms used were not memorable (paragraphs 5.8-5.9).

- It is recommended that each site offers a brief, succinct description of its aims and remit at its home page. The sites are currently attempting this to different degrees, but more effort should be placed in making clear to new users what each site is about. Wherever acronyms and logos are used, they should be explained. It would be useful to test further changes in the language, design, and layout of each homepage with a small, but varied sample of targeted users.

5.1 In all sessions users reported that they did not have a clear idea what each site contained before starting to browse in greater depth, although some said that it was becoming clearer with the last few sites as they were proceeding with their navigation. One user reported that it was easier to get a clear idea of SCAN’s content compared to the other sites, but that as the others had more specific use it was more likely that users would access them intentionally. Some users thought that it was clear that AIM25 related to London and the M25 area. Several participants from all user groups reported that they went straight to search without reading any of the introductions. They thought that most users would want results immediately, which is why they would like the search button(s) to be immediately visible high up on the page.

5.2 Most of the London users also agreed that the acronyms used for the site titles were not memorable and did not give an idea of what the sites were about. One researcher said that he could not remember the difference between the Archives Hub and A2A, suggesting that a stronger identity was needed.

5.3 In general, all user groups found this question about previous expectations difficult and appeared to not have had any specific expectations from the sites before. As some wrote in the questionnaires, they had come to the sessions ‘with an open mind’. Even those who had used the sites before, did not appear to be frequent users with great familiarity with them.

5.4 Some of the London participants found that their expectations from the archive sites were met, although a small number of users said that they found the sites more difficult than they had expected and referred also to the artificial nature of the evaluation exercise.

5.5 Both of the Manchester user groups had difficulty describing what their expectations of the archive sites were, mainly because they had very limited knowledge of archives and no previous expectations of the websites.

5.6 The Edinburgh users confirmed what was said in the other sessions about the lack of expectations from the sites and they reported that as the projects were not completed, users had not yet formed predetermined ideas about them. In terms of content and coverage, one user reported that there is a subconscious expectation about finding things in their area of interest, admitting that it is in fact unreasonable to expect English projects to cover Scottish local history.

5.7 In terms of usability, one of the users in the Edinburgh sessions was pleasantly surprised by the usefulness of the sites for family history research and said that the sites were a revelation for people like her who were novices in the use of online catalogues and the Internet. Others felt that the sites should be simpler in their design and easier to use for those inexperienced in sophisticated searching at different levels. Another user had expected the sites to be difficult to use at first, but that after half an hour or so he would get to grips with each of them. This indicates that the underlying assumption that these more experienced users had about these type of sites was that they were not an intuitive tool which you could use straight away without any problems, but something that needed some investment of time and effort to yield useful results.

6. Satisfaction of research and information needs

Key issues and recommendations

- Most users could see the potential of the sites for satisfying, at least some of their research and information needs. This was more obvious with the more experienced users from the London and Edinburgh sessions (paragraphs 6.2-3 & 6.6-8). However, even the Sixth Form students (who were the least likely to use the sites again) and the adult Manchester users could see the sites’ potential for supporting family history research (paragraphs 6.4-5).

- In order to support the needs of all these groups more effectively, the sites would require: more user-friendly interface and navigation, as well as a ‘guide’ or pointers on how to get started for Internet and archive novices; for specialists and archive professionals, a clearer indication of the material included, any gaps in coverage, and areas of overlap between the different projects/strands, as well as sufficient level of description and amount of information to enable users to determine whether a visit to the relevant archive would be useful.

6.1

Users generally rated highly the usefulness and information value of the content of the sites (figure 1). The variations between the different types of users reflect their different profiles and interests. The less specialised Manchester users rated SCAN higher than the other three sites, appreciating its design and appeal for a more general audience (with the Parrs Wood students giving A2A the lowest rating, which might be affected by the fact that this was visited first). The London participants gave the highest rating to AIM25, the site they were most familiar with and which was more closely related to their interests, with high ratings also for A2A and the Archives Hub, finding SCAN the least informative and useful for them. The Edinburgh users rated all four sites highly with very small variations between them.

6.2 Some of the London users were positive about the potential of the archive sites for supporting their research and information needs. One user interested in nineteenth century history said that she found ‘intriguing stuff’ on A2A and AIM25 that would be useful for her research. Another researcher interested in Caribbean studies found that AIM25 had the greatest potential for discovering information about ethnic groups because of its subject index headings, and SCAN because of its Knowledge Base.

6.3 Some others found it difficult to judge the sites’ potential for meeting their research needs because of the short time available to explore all four of them and felt that they would need to revisit them. Both London and Edinburgh users felt also that it was difficult to answer this question when the sites were incomplete with a lot of the material still unavailable. Another problem raised was that it was not clear how the projects related to each other within the country. Finally, one user thought that the A2A catalogues were too brief and had not realised that these were reliant on the repositories’ own catalogues and the level and standard of information that they provided.

6.4 The Manchester Sixth Form students were not familiar with the concept of archives and would be unlikely to use these sites again unless they found them through a general search engine such as Google when they were carrying out specific research for a school-related topic. It would be unlikely for them to use the archive sites outside school. However, the group could see potential future use of the sites for family history, place name and surname research.

6.5 The older Manchester users, most of whom were also unfamiliar with archives, could see how the sites could be useful for them for researching family history and for providing information on the archival institutions themselves, although they would require an easier way for beginners to get started.

6.6 The Edinburgh groups had in general similar responses to those in London, with some users stating that the sites would meet their research and information needs. They could see how online access allowed them to assess documents, which would help them to prepare for visits to archives, and decide whether it would be worth visiting them at all. Also, they thought that the evaluated sites would be useful tools for regular users of archives, in the same way, for example that the PRO site is useful for providing basic information to PRO users and not hard to use. These groups, which included regular users of archives and professionals, stressed that the key was for a site of this type to allow them enough information to decide whether they should go to an archive, and felt that the four strand sites did that to varying degrees.

6.7 Some reported that the sites were useful as they included records relating to their area of interest held in repositories where they would not have normally searched for information in the first instance. As one user said, ‘it was a surprise to find Scottish material in the other sites’. For example, the Archives Hub helped them find collections relating to Edinburgh, but held outside Scotland, in this case, in the Modern Record Office in Warwick. For others from the Scottish groups, some of the sites, like A2A and AIM25 appeared less relevant to their interests.

6.8 Other users raised the issue of the level of description and the amount of information available in the sites. They thought that individual archive descriptions should aim for a useful level of detail that would help users to avoid a wasted journey. In this sense, one user said that SCAN’s site will be good for wills, as it will enable users to see digitised versions of the exact document. Another user said that he was impressed with the amount of information available for each record in the Archives Hub, and looked forward to seeing more information on the SCAN site, while another assumed that eventually all the sites would provide an expansion of the levels of description. In a few cases however, the users’ perception of the amount of information available was restricted due to the way they had searched the site, for example one user who did not find much information of interest in A2A, had been using focussed index searches, rather than general ones.

7. Method of navigation and use of the sites – General points for all sites

Key issues and recommendations

- The method of navigation and way of using the sites varied between users. Although most users found the sites quite user friendly, related problems were observed and reported in all sessions, particularly with the general public group at Wythenshawe Library, which was the least experienced in Internet use (paragraphs 7.1-4). The main problem with the two Manchester groups was their limited understanding of archives (7.5).

- The more experienced users raised issues which would make the sites easier to use at all levels: more standardised and simpler interface, search devices, and other tools used between the four strands; and avoidance of areas of overlap (paragraph 7.7-8). (More detailed recommendations in these area are presented in the following section 8). Users felt that the sites did cater to some extent and to varying degree, for disabled users and for minority interest groups (paragraphs 7.9-10).

- The links between the HMC/NRA were used by some users during the London and Edinburgh sessions who found them useful, while the Manchester groups did not use them due to lack of understanding of their function and limited time (paragraphs 7.11-12). Recommendations for improving the usability of the links are offered in paragraph 7.14.

- It is recommended that all sites, and particularly A2A, AIM25, and the Archives Hub, consider creative and effective ways of presenting the whole idea of archives and of interpreting catalogue searches and results to users who are complete novices to archival research. These users were perplexed and frustrated by seeing their searches resulting in catalogue references, rather than encyclopaedic or general search engine’s type of information. If the sites currently offering online access to their catalogues want to cater for this type of audience more effectively, they would need to think of innovative and even entertaining ways of introducing their subject. The design, language, content, and layout of any prototype material in this area should be tested with the relevant user groups. SCAN has already made successful steps in this direction, which were appreciated by the users. It would be useful to share the lessons learned from this experience with the other sites.

- All the sites should consider ways and follow generally accepted guidelines (such as those proposed by the World Wide Web Consortium (W3C)) in order to address users with disabilities, such as the provision of the ‘no frames’ searching option currently employed by the Archives Hub, the key access points at the top of each page in SCAN, or the highlighting of the keyword or phrase searched in A2A and AIM25.

- The sites should also consider providing more and/or more easily accessible information on other sites or organisations where minority interest groups might be able to find information not covered in the sites.

7.1 General observation showed that in all sessions some of the users navigated through each site slowly and methodically, while others went through them fairly quickly and then hopped back and forward between the sites, searching for particular subjects.

7.2 Most users were familiar with the use of computers, the Internet, and interrogating databases (except the ones at the Wythenshawe Library), and a few were observed carrying out sophisticated searches (such as using the ‘Search In Current Document’ feature of the Archives Hub). In general, none of the London users had any serious difficulties with navigation. They thought that the sites had good links, which were simple and clear to use and that it was easy to find contact details for all sites.

7.3 On the other hand, a small number of users, particularly in the Manchester and Edinburgh sessions, had difficulties using the web browser and navigating the archive sites. Some users from the Manchester session at the Forum Learning Centre of the Wythenshawe Library (which as we mentioned earlier, included the least experienced computer and Internet users of all evaluation sessions), were observed scrolling rapidly through results’ screens without really looking at them. In one instance, one participant was found to be searching via a search engine without realising that he had exited the A2A site.

7.4 When asked about ease of use, the older Manchester group reported that the archive sites were quite difficult to use (even though they gave all four sites relatively high ratings for ease of navigation in their questionnaires, as is shown in figure 2). They would have liked more time to use them, as they were Internet ‘learners’. They thought that background knowledge was assumed about the users of the sites. Nevertheless, they stated that they had enjoyed the session and were sufficiently interested to return to these web-sites in the future.

7.5 The main problem that the Manchester users had was related with their understanding of archival material and the way it was organised. As their interest was a general one, they expected the catalogue links to give them a brief introduction to the topic they searched for, rather than the catalogue description of the particular holdings. They would have preferred to use the sites as a kind of encyclopaedia. This is a recurrent theme in the detailed reports from most sites below.

7.6 As figure 2 shows, users generally rated the ease of navigation and return to the home page for all sites relatively high. Users’ ratings for the four sites did not generally show considerable variation, except those of the London users (who rated AIM25 as the highest and SCAN as the lowest) and the Manchester six-formers who found A2A less easy to navigate. More detailed comments related to navigation of each site are included in section 8.

Standardisation of interface, search devices and tools

7.7 Another point raised by the more experienced users related to the lack of standardisation between the four strand sites in terms of the language used in search devices, buttons, tools, etc. Although archival language used for the records and descriptions is generally becoming standardised, the sites tested have not followed that route yet, which makes their use more confusing and difficult for the users. One user commented that the range of search devices between the sites was confusing and perhaps unnecessary, requiring standardisation. Several users thought that the layout and design of these features should be similar, with the positioning of the buttons, for example, at the same place across all sites. In this direction, they suggested that it would be useful to examine existing guidelines, for example the ones IFLA produced for search engine buttons, or similar practice in the library world. This point relates to the desirability of a common gateway for searching all strand sites, which is examined later in the report (section 10). Comments from all sessions, and particularly from the more specialised users at the London and Edinburgh ones, indicated also that users felt that overlap should be avoided, as it was confusing for users.

Target audience

7.8 Specialist users raised the question of who the sites were aimed at, questioning the ability of all the tested sites to appeal to complete beginners without the need for a personal interface with an information professional, like an archivist or librarian, to help them formulate and refine their searches and introduce them to the sites. Several users thought that the sites should address all levels of expertise and that the web designers should aim for the lowest common denominator. On the other hand, another user, a professional from the archival world, thought that the projects should not concentrate too much on first-time users. He felt that all the sites should be aiming for users who come back often and get to know them.

Users with disabilities / minority interests user groups

7.9 In terms of provision for users with disabilities, one partially sighted user from London commented positively on SCAN. Its use of the cursor lighting up and flashing and the provision of key access points at the top were seen as very useful. Another user reported on the headers used by several of the tested sites; in some instances, these might be lost in the navigation through pages, making it very confusing for the users to see where they are. Providing a ‘no frames’ option, the way the Archives Hub does, is a positive step congratulated by users.

7.10 In terms of minority interests, the Scottish users felt that the three non-Scottish sites were doing quite well, although one user thought that A2A and AIM25 did not represent minority interests sufficiently. Others felt that the projects needed to take advantage of the energy and labour force of family historians and related societies and their indexing and online provision. Participants also thought that the sites needed to provide more information on where users should go if they could not find what they are looking for in the strand sites.

Links between the HMC/NRA and the AIM25 sites

7.11 There was general agreement by the London users that the HMC/NRA links to AIM25 were useful. One user commented on the extra detail available, which she found really useful, while another one said that it was useful to know that she had covered as much ground as possible. Another user said that the links to the catalogue entries and to the repository details looked similar, which could be confusing.

7.12 The HMC/NRA links were not explored at either of the Manchester sessions. During the morning one, students did not appear to have any understanding of the role of the HMC and the NRA. In the afternoon session, there was not enough time for this more inexperienced group to explore the links, since a lot of their energy was consumed by browsing the other sites. As with the Parrs Wood College group, there was little evidence of prior knowledge of the HMC and the NRA.

7.13 In Edinburgh at least one person used the links between HMC and AIM25 and found them easy to use. Another user commented that these worked at a detailed level, adding that at general browsing level it was easy to identify links. ‘Found it easy because it was step by step – casual browse might be more difficult’.

7.14 Usability of the HMC/NRA links could be improved if the explanatory text related to search results (for example, phrases such as ‘x number of records noted’, ‘x number of records found’, ‘Where reference is made to an NRA number, a catalogue is filed in the National Register of Archives and may be consulted in our public search room’) is placed at the top of the screen. Users at all sessions generally commented on the importance of including search links and other important information at the top of the page (for example, for the A2A search button). A clearer indication of what each type of HMC/NRA link signifies, what the ‘reference’ caption refers to and perhaps colour coding the links to the AIM25 description would help to distinguish them from the ARCHON entries showing the contact details and would prepare users better for the transition to the AIM25 site.

8. Search mechanisms and usability issues related to each site

Key issues and recommendations

This chapter covers separately the detailed comments that users of the London, Manchester, and Edinburgh sessions made about each site (A2A: paragraphs 8.1-22; AIM25: 8.23-35; Archives Hub: 8.36-46; SCAN: 8.47-8.62). A general observation is that there were positive remarks from the different types of users about all the sites. On the other hand, even the users most experienced with archival research and Internet searching had problems navigating and searching the sites.

From the users’ suggestions and comments, the following recommendations and issues arose:

A2A

- It is recommended that a more obvious option for simple browsing is considered, particularly for new users.

- The redesign of the focussed index search to allow users to qualify their searches should be considered.

- The use of example search terms in the search page would help users to use these appropriately and effectively, e.g. to avoid entering words and phrases, instead of reference numbers in the catalogue reference search.

- As it appears that few users clicked on the help option, some of this information might need to be incorporated in the main search page.

- Clarification of the options in the focussed indexes and of the use of authority controlled terms, together with the use of examples, would help users unfamiliar with archives, but this needs to be kept brief and succinct.

- Error messages for unsuccessful searches could be redesigned to provide more useful feedback and to suggest how a search could be improved or refined. This is particularly needed for the Focussed Indexes, as poor use of the searching tools and unsuccessful search results led users to believe that there was little content in the site.

- There was unanimous agreement by all users that the search button in the home page should be positioned higher up in a more obvious location.

- The text on the home page could be reduced, keeping the page simpler and less cluttered, with buttons and clearer links leading to separate pages with further information.

- It would be useful to redesign the search page (e.g. using different colours to code the function of different buttons, separating search buttons from other functions or layering the different search options), so as to make it more obvious to users what to select. This page could become simpler and less cluttered by leaving only the keyword or phrase search in the first search page, and adding a clear link to a separate page for those interested in advanced search options.

- Further development of the site should reconsider navigation issues, providing users with easier options for returning back without getting lost. A clearer warning should be provided to users when they are about to leave the main A2A site and be linked to the site of one of the repositories.

- As the timeout feature seems to have created some problems for the users, this should be tested further, and perhaps be set up to start after a longer period.

- When search results are presented, there should be a clear indication of how these are sorted, e.g. at the top of the list, adding to the existing phrase ‘shown below are catalogues 1 to x’, a phrase like ‘sorted by repository name’. Adding also a phrase like ‘included the keyword or phrase you searched for’ to the number of catalogues found, might help the users who were confused with the reason why hits came up.

- As the use of frames causes printing problems for the users, the addition of a ‘printer friendly’ option on every page should be considered.

- The font size used in several pages of the site should be increased and a sans serif font could be used instead, e.g. in the main text of the home page.

- It would be useful to add a facility for ‘emailing your results’, as users appreciated this feature provided by the Archives Hub.

AIM25

- The subject thesaurus and name indexes should be redesigned with the ‘search’ button placed in a more obvious place and with clearer instructions given to users to click on this after selecting their multiple terms from the pick lists.

- In the name indexes, the drop-down menu showing either the option ‘ANY’ or ‘ALL’ that users can select in the ‘Find description indexed with ANY or ALL of the names selected below’ would be more effective if both options appeared clearly at the same time, for example as check boxes.

- A link to an introduction on the way the subject thesaurus was designed and used might help users understand its function better. Novice users might also benefit from a brief explanation of what a thesaurus actually is.

- Making the ‘Browse’ or the ‘Free Text’ search option more distinguished than the other search options would help give novice archive users a clearer idea of where to start.

- A brief sentence introducing the ‘List of Repositories’ and suggesting where to click to proceed at the ‘Browse’ section would help new users unfamiliar with archives.

- Expand the highlighting of search terms also in the abstracts and titles of the summary list of retrieved results, and not only in the detailed view of each record.

- Provide a clear summary and introduction to the remit of subjects and materials the site covers. Consider providing also links and suggestions of other sites and organisations that users could approach for material not included in AIM25.

- It would be useful to add a facility for ‘emailing your results’, as users appreciated this feature provided by the Archives Hub.

Archives Hub

- Unless most records show a marked difference between summary and detailed view of the descriptions at collection level, the site should consider providing only one.

- Consider more obvious use of highlighted text for the searched terms in the results pages, as users commented favourably on its use in A2A and AIM25, while some had not realised that it was possible to use this feature as an option in the Archives Hub.

- In general, the use of ‘search in current document’ and other ‘Search In:’ options, although a useful feature liked by some users, should be tested further as it appeared to either not always be working properly or to not be obvious to new users.

- Make clearer connection between the search terms and the resulting retrieved entries. Also, make clearer when it would be more effective to use an ‘advanced search’ option.

- In the results page, make clearer what ‘Title’ refers to. This is currently in blue letters and underlined, although it does not appear to link to anything, which is confusing.

- It is recommended that an option to browse by institution is also provided.

- The design of a ‘printer friendly’ option on every page should be considered.

- It would be useful (especially for users unfamiliar with the use of frames) to indicate how to resize the frame windows to show more of the summary or detailed description of the document.

- It is recommended that a ‘no frames’ option is provided not only at the search pages, but also on every results page.

- The home page should be redesigned so as to be less cluttered. Reduction of the size of the logos at the bottom of the page might be useful, as they currently seem to dominate it.

SCAN

- Make clearer that online searching of the catalogues is not available at the moment and that the site represents work in progress.

- Make clearer the site’s remit and the range of materials it covers, indicating where possible the reasons for exclusion (e.g. wills after 1876)

- Consider adding a facility for ‘emailing your results’, as users appreciated this feature provided by the Archives Hub.

- In the design of the online catalogue pages, consider using highlighted text for the searched terms in the results pages, as users commented favourably on its use in A2A and AIM25.

- Test navigation and backtracking options further.

- It would be useful to be able to narrow down and combine different options from the drop-down menus at the Knowledge Base searches.

- As the Knowledge Base is a very popular option, it should be placed at a more prominent option.

- Research tools, such as the Glossary and the ‘Scottish Handwriting’, should also be placed in a more prominent position with links leading to them from more pages.

- Contact details for SCAN staff should be made more obvious, as the ‘Team profiles’ is currently hidden under the not immediately obvious section titled ‘Press Box’.

A2A

London Sessions

8.1 The London users generally found A2A easy to use, although some reported that they ‘struggled a little when searching’. Users commented favourably on the date searching feature, while another user reported positively on the highlighted text which helped her not to get lost. Some of the features that the users from the two London sessions would like to see improved included:

8.2 – More sophisticated searching capabilities or alternative and easier ways of searching:

o being able to browse the subject list (‘only annoying feature was you had to get to the subject lists to browse’);

o being able to qualify the focussed index search (as one participant commented he ‘would have liked to be able to narrow down the search’). Another user was observed wanting to use the drop-down menus (for Archives Location and English Region) to qualify a search of the focussed indexes, but this option is only available to qualify a Keyword (or Phrase) search.

However, it is worth noting here that some users appeared to have difficulties with the more sophisticated searching features provided by the site. For example, one participant was observed trying to use the catalogue reference search by entering words and phrases, not reference numbers.

8.3 – Layout and design of the search and navigation buttons:

o position and design of the search button. A lot of users thought that this is hidden at its current position at the bottom of the page and that it should be placed higher up in a more obvious place, as most people want to see results immediately without having to scroll down to locate it. Others were also not clear about its function (‘search button was confusing; I spent some time trying to figure out what it does’);

o layout and design of all buttons. Some users commented on the lack of impact and the look of sameness of all buttons of the sites, which they thought did not make obvious what to select;

o lack of going ‘back’ option hindered navigation. Several users commented that a ‘back’ button would have been useful, as they found themselves ‘going down a blind alley once or twice’. Getting ‘back’ was difficult and users found that they often had to leave the search screen entirely.

8.4 – Printing facility:

o because of the use of frames, users might only get the header printed on the page, unless they select the text they want printed. As one user commented, ‘it is important for the sites to be in a format that is easily printed, as quite often users will print the results from their searches to go back to later’.

8.5 – Timeout feature in A2A:

o in general, it seems that users found that the A2A searches time out quite quickly;

o the ‘timeout’ error message came up for one participant and kept repeating, despite him reading the error message, which tells users how to start using the site again after timeout; (this also appears when users do not carry out the two steps required to search the focussed index, which may have been what the participant was doing).

Manchester sessions

8.6 The Sixth Form students appeared to navigate the site without too many major problems, although some difficulties with searching and understanding the search results were observed. The older users from the Wythenshawe Library session appeared to have more problems using the site, but as one user put it, ‘the sites were easier to use as we went along, so A2A suffered as it was the first we looked at’. Both groups made similar comments overall about the site’s usability.

8.7 Some students spent an extremely long time on the home page and the links from it (one student seemed to never get past these), while some users from the Wythenshawe group got bogged down in the Contributors List linked to the home page. Several students were observed becoming lost using the links from the home pages and attempting to do their searches in linked sites (e.g. HMC), while a few seemed reluctant to click on the links to search results available from a results ‘hit list’. This might be related to lack of interest in the relevant subjects or unfamiliarity with archives and related searches.

8.8 A common problem with both user groups in Manchester, which represented non-specialist audiences (and one which applied also to AIM25 and the Archives Hub), was their lack of understanding of archives and of the characteristics of archival catalogues. For example, one student was baffled at not finding ‘information’ [of encyclopaedic type] on Florence Nightingale, but rather ‘references to stuff you had to go elsewhere to see’ (referring to the catalogue entries). The older users made similar comments and several would have like the history of Florence Nightingale, not a ‘list of books’. However, the Access Assistant at the Forum Learning Centre who participated in the afternoon session at Wythenshawe pointed out that catalogue information could also be interesting and another participant said that ‘it is good for students to have to do more research instead of just lifting excerpts from internet sites’.

8.9 The participants from both groups made comments about features which they liked and offered useful suggestions about others that could be improved, including several points which had also been brought up by the London users:

8.10 Searching features:

o in general, users found that there were too many search options on the ‘Search Page’;

o others reported that the site did not have any obvious ‘browse’ option;

o a few users reported that they got lost on the focussed index of A2A (‘not clear what these options where [on the search page] or how to use them – especially focussed indexes’). Students generally seemed unfamiliar with the concept of authority controlled terms;

o there was a problem with searching the focussed index for at least one student, with two indexes being displayed simultaneously which is not meant to happen. It was not clear whether this was an A2A style sheet problem or some other problem;

o a student reported that ‘the information on how to search was too long to be practical’;

o another student could not find her way to the search screen from the home page and had become lost in the HMC links from the A2A homepage;

o users found the Suffolk Record Office search that was suggested in the questionnaire confusing and with some links not working. An older participant was frustrated by the fact that she could not look at information from all the branches of the Suffolk Record Office at once [i.e. that she could not select more than one repository at a time to search/browse];

o one student made the suggestion that the error message [at the searches] would be better used to signify a wrong search and to then enable users to go back to their original search with suggestions of how to refine it;

o older users were confused by the logic behind the retrieval of information and sorting of hits (‘some idea of why hits have come up would be helpful’), indicating that it was probably not clear that users could sort the results by number of hits or by repository name, which is the system’s default;

o students liked the highlighted search term displayed in the search results.

8.11 – Layout and design of the search and navigation buttons:

o users at both sessions agreed that the A2A search should be at the top of the screen, as it was easy to miss. (‘It is much better to have all [search-related] screen information visible at once rather than logos and sponsors names. The first page is the most important’);

o students found the date searching also easy to miss due to the need to scroll down (‘The dates at the bottom were in the wrong place', referring to the date entry boxes on the search page)

o participants from both session reported that (due to its design) they were easily led away from the site and that it was ‘too easy to get side-tracked by other sites’.

8.12 – Overall design:

o some Sixth Formers reported about A2A (and the other sites) that some pages ‘were not so appealing – just white screens with lots of text’, while ‘some information [in the search results] was too long and not user friendly’ (as is also indicated by their rating of the site in figure 2 and 3). This confirmed that this Internet-savvy younger audience is demanding in terms of graphic design and visual appeal, and is not easily satisfied simply with rich textual content;

o some retired users also found that there was ‘too much text on the initial page, which was off-putting’.

8.13 – Timeout feature:

o the timeout error message came up again in both the morning and afternoon session (but it was unclear whether this related to true timeout or to a misunderstanding of the use of indexes)

Edinburgh sessions

8.14 Several users during these sessions were observed reading through the text on the opening screen and other pages. One person also looked at the pages of information on the A2A programme. Some users in this group explored some more advanced features, and there was noticeable use of the drop-down lists and the Help screen for keyword searching (although it was not clear whether any user noticed the clickable help available from the field labels).

8.15 There appeared to be some difficulties with navigation and searching, even with these groups which represented experienced archive users and professionals, but which also included some Internet novices. Some users found this site the most difficult to use, but as was mentioned by a Manchester user, this view might be affected by the fact the A2A was the first one which users explored.

8.16 Users comments and observation indicated the following points in relation to usability and searching:

8.17 – Simpler and easier searching options:

o some users found A2A confusing with too many options in the search menu and the home page, and too much text in the Boolean options (‘the two-layer search is unwieldy’);

o some commented positively on the keyword/phrase search which they felt worked well and was useful. One user noticed how ‘limiting the number of letters in the search word paid dividends, but perhaps only an experienced researcher would know this’;

o one user was observed being completely lost in his attempt to search on ‘Florence Nightingale’. He had made his way through to the HMC site from the A2A home page and was trying to find information on her in the women’s history section. Perhaps simpler and clearer search options and fewer links on the home page would have helped to prevent this.

8.18 – More effective searching:

o Focussed Index needs a clearer explanation or simpler design: one user had difficulty retrieving any useful information when searching for English Civil War but had not realised that the most suitable searching method would be the Focussed Index. Another user who started using the Focussed Index immediately (and seemed to ignore the Keyword or Phrase search), appeared to have problems selecting suitable index terms as error messages kept appearing.

o unsuccessful search results led users to believe that there was little content in A2A (as was the case with one user, for example, whose searches on ‘window tax’, ‘turnpikes’ and personal names beginning with ‘Mac’ had been unsuccessful).

8.19 – Layout and design of the search and navigation buttons:

o position and design of the search button. Confirming what the London and Manchester users had expressed, several participants in these sessions thought that the search button should be in a more prominent place high up on the page.

8.20 – Overall design:

o several users thought that the font size on the web site is too small. One also commented on the poor impact of a serif font, like Times New Roman, suggesting that a sans-serif one would be more suitable;

o another user suggested the use of colour coding in the site;

o some users thought that there was too much text in the help screen.

8.21 – Use of themes:

o one archivist expressed concern with the use of themes and their limitations, as ‘many repositories/archives might fall between two different themes and so may get lost’.

8.22 – Overlap of coverage with other sites:

o one archivist commented that she felt more inclined to go straight to ARCHON to find information about Suffolk Record Office (one of the suggested ideas in the questionnaire) rather than use the A2A links.

AIM25

London Sessions

8.23 Most users in these sessions found the AIM25 site easy to use. One said that he would rank it as the best (reflected in the site’s ease of navigation rating by London and Edinburgh users shown in Figure 2), but added that this might be because he had used it before. Users commented on its clear and straightforward design, giving it a high rating for visual appeal and layout.

8.24 Some of the comments they made on specific aspects of usability, searching and design were:

8.25 – Subject thesaurus:

o some users were confused with the way the subject thesaurus was designed. Apparently, it took them some time to work out that ‘multiple clicks on terms did not lead to search results and that you had to click “search” at the bottom after checking the boxes’;

o another user commented on the fact that use of the subject thesaurus also implies a degree of familiarity with the terms and that otherwise users have to browse to find what they want; another researcher found the subject hierarchy a bit confusing.

8.26 – Name indexes:

o several users commented positively on the name indexes which they found useful;

o when asked about the pick lists available in the indexes, a few said that they were easy to use, but one user thought that it was not obvious that you could pick more than one name. This suggests that the drop-down menu showing either ‘ANY’ or ‘ALL’ that users can select in the ‘Find description indexed with ANY or ALL of the names selected below’ might be more effective if both options appeared clearly at the same time as check boxes.

8.27 – Choice of fonts:

o one user commented on the font of the summary guide, which he found difficult to read, and suggested a sans serif font which might be easier.

Manchester sessions

8.28 The Sixth Form students did not appear to have difficulties navigating this site which some found easy to use. Unfamiliarity with archives and lack of interest in the subject meant that the users in both Manchester sessions did not fully understand the various searching options and their advantages. One older participant who thought that AIM25’s content was just related to the M25 area, could not see ‘what’s it got to do with [him] in Manchester’, although he added that browsing the site was interesting. Users from this group appreciated the site’s clear indication that this was ‘work in progress’, something which they felt was missing from the other sites.

8.29 The student group made several comments on usability and design, while a few points were also made by the Wythenshawe Library users:

8.30 – Searching features:

o most students found the searching good overall, but made specific points about aspects of it;

o new users found that it was not clear which search option you should use, despite the descriptions on the search page, and suggested adding a clearer search box on the home page; one student reported that he was unsure ‘which was the general search engine’. For example, it appeared that it was not immediately clear to all new users what ‘Browse’ and ‘Free text search’ were most suitable for, and one user was observed attempting to find collections held by Imperial College [the aim was to do this using browsing], and finding that entering 'Imperial College of Science, Technology and Medicine' brought no results in a Free text search.

o some users thought that the way you accessed the actual records was not obvious;

o observation showed that one student was baffled by search on 'Einstein' returning the Bohm collection, as she could not understand the relevance;

o some users said that they liked the ability to ‘Browse’ records, although one student ‘couldn’t understand what was happening in browsing’ and another found the Browse page quite hard to use;

o users appeared unsure about the ‘targeted search’ option;

o there were general expressions of confusion among the student group during subject searching (‘clicking on the links only took you around and around the lists’) and students stated that they were not sure what the Subject thesaurus was supposed to do. Several users were observed navigating around the thesaurus, seemingly not understanding what they were doing and unaware that they needed to click the boxes in order to link to the records.

8.31 – Overall design:

o some users found the framed layout of the pages confusing.

Edinburgh sessions

8.32 The users from both these sessions made some very positive comments about AIM25, which they felt was clear and easy to use. One user said that he would rate this as the highest of the four strands, while another one felt that for ‘general usability, the AIM25 site was the clearest’ and that it was generally very quick to get into.

8.33 – Search features:

o one user reported that they would have like the search box at the top of the page. [It was unclear whether this referred to the Named Indexes or whether he was confusing AIM25 with the A2A home page];

o another point raised was that the search terms do not necessarily appear in the resulting descriptions, making it confusing for users to interpret the results. [Search terms appear in red in the detailed catalogue record description, but are not highlighted in the initial list of descriptions which shows titles and abstracts];

o an archivist reported that he liked the site because of the hierarchical levels it provided.

8.34 – Effectiveness of searches:

o some users were disappointed with the results when searching for names other than the ones suggested in the questionnaire. However, this might be an indication that they did not have a clear understanding of the remit of the subjects covered by the site. For example, one user did not have any luck searching for references to ‘jewellers’ or ‘water supply’ in London. This suggested that it was not clear to the users what the remit of the site was and what was not covered, as they had not realised that the site does not contain, for instance, local authority records. This is a point which applies to all the strand sites;

o one user suggested that in order to cut down the frustration level, if the results were negative, the results page should have a suggestion about where to go.

8.35 – Overall design:

o some users commented positively on the use of colour and highlighting of terms in the site.

Archives Hub

London Sessions

8.36 The London users were generally very positive about Archives Hub, which a few found most ‘ready to go’ out of all of them. They found it easy for anyone to use and one researcher suggested that it looked like a search engine, which might indicate that he found how to start searching the site obvious and clear.

8.37 – Search features:

o one person did not understand what the difference was between the summary view and the full view of the descriptions at collection level, and felt that only one was needed;

o another user thought that it was not clear how the searching within the text of descriptions worked, which should have been self-explanatory;

o one user reported problems using the phrase search;

o one participant who had liked AIM25’s feature of highlighting the search term, had not realised that it was possible to do this in the Archives Hub.

8.38 – General design:

o some users thought that the presentation was generally not clear and the design should be simpler and self-explanatory;

o one user did not like the frames aspect, as he was not clear why it was used and thought that it made the site look too cluttered. As others mentioned during all sessions, he felt that users should not need to read so much before using the site.

Manchester sessions

8.39 The Manchester users responded generally positively towards the Archives Hub, with one further education student stating that she liked this site the best. As with the other sites, these more inexperienced archival users appeared in some cases to have difficulty understanding the results they were getting from their searches. More detailed comments included:

8.40 – Search features:

o users liked the home page and the ‘Quick Search’ option;

o they thought that the other search options were easy to use;

o one student mentioned that he liked seeing all the records, but would have liked the option to maximise the screen;

o some users would have liked to be able to browse by institution;

o as with the London sessions, the ‘Summary information’ feature was again perceived as pointless, as it seemed similar to the full information.

8.41 – General design:

o as with the London sessions, some Manchester students did not like the frames layout;

o another user made the suggestion that a different, stronger colour was required for highlighting the terms. He had not noticed that you were able to do that through the ‘search in current document’ feature.

8.42 – Email and printer-friendly options:

o users in both sessions commented favourably on the ‘e-mail your results’ facility and the printer-friendly options and suggested that these should be a feature in more sites. [The former is a new option that was not available at the London sessions].

Edinburgh sessions

8.43 The Edinburgh users generally liked the Archives Hub, finding it easy to browse and offering quick results. They also added however, that their view of the site might be influenced by the fact that they were getting used to the sites by that time [as the Archives Hub was the third site that they explored]. One participant preferred it to A2A, finding it easy to get into and having good search terms. Several users thought that the site contained an ‘impressive amount of information’, while one participant thought that ‘on the whole, the search does what you want it to’.

8.44 More specific points raised were:

8.45 – Search features:

o one user (a Higher Education archivist) felt that ‘browsing’ was really a structured search and that it was necessary to explain the difference between browsing and structured search better;

o another comment made was that it was not clear what the ‘title’ link signified;

o success of retrieval and satisfaction with results: Several users expressed puzzlement and frustration at the search results they were getting, a lot of which appeared to be extraneous references (‘it was annoying that it showed results for ‘Ecuador’ when searching for ‘Edinburgh’, while Aberdeen search brought up results for Aberdare’). Users had not realised that the ‘advanced search’ would be more suitable in these cases. One user thought that A2A was better in this respect, making clearer connection between the search term and the resulting retrieved entries. It appears that most users had not realised that the site offered the ‘search in current document’ option.

o one user who had observed and used the ‘search in current document’ option, was positive about it, although he added that it was not very obvious. Another user felt that it was not clear how to use this option from the screen.

o another user, a local history researcher and retired librarian, felt that the site was generally good for looking for ‘big names’, such as ‘Winston Churchill’, but that the results it retrieved would not be of interest to schoolchildren wanting information. She thought that the site would be better for local or family history, but felt that none of the archive sites would be sufficiently user-friendly for complete beginners, as they would not normally know how to phrase their search.

8.46 – General design:

o one participant complained that the home page was too busy, with too many logos.

SCAN

London Sessions

8.47 The London users reported that this looked like the most appealing site of the four tested. They felt that the layout was nice and that the site was easy to browse. One user said that it ‘looked dark and theatrical in relation to the other sites, visually projecting a different image’. Several commented positively on the site’s more general approach, which they thought helped users ‘get at knowledge’, providing a good way to get started with Scottish archives and to get information on Scotland. They also observed that this was the most technologically complicated site, which sometimes did not work, but which was ‘entertaining’ when it did. More specific comments were:

8.48 – Search features:

o one user thought that it was not clear that you can not search for records at the moment; for this reason, and because several sections are incomplete, another user found the site difficult to use in general;

o others ‘got stuck now and again’ and could not get the site to work at some points;

o a few users had problems backtracking and felt that the site did not provide enough ‘back’ options;

o some users were positive about the knowledge base;

o another feature which users liked was that the text was highlighted when the cursor hovered over it;

o one user thought that it was difficult to find contact details for repositories in the SCAN site.

8.49 – Content coverage:

o one user was puzzled by the fact that the site’s coverage of wills stops at 1875. When he was told by the SCAN staff that this is because a good list already exists for 1876 onwards, he made the recommendation that this should be made clear on the website.

Manchester sessions

8.50 The Manchester users made similar comments as the London ones about SCAN’s visual appeal. The Parrs Wood six-form students found SCAN particularly appealing, as is obvious from figure 3. Most users from both sessions rated it the highest of the sites tested, but found it complicated and with some problems with navigation and searching. Several users found it the easiest because it was the last site that they went through during this session, as well as being different from the rest.

8.51 – Search features:

o users felt that it took a long time to search, but this might have also been affected by the fairly low configuration of the computers in the Wythenshawe Library;

o as was obvious from several comments, users of these sessions did not realise that you could not search for catalogue records yet or that large parts were incomplete: ‘it was good that it went straight to records’, ‘the search option could be more obvious’, ‘after entering a search term, using the return key did not work’, ‘Search results should be highlighted in actual pages’.

8.52 – Level and amount of information:

o because of the greater amount of introductory and general information, several users formed the impression that SCAN actually contained more than other sites. As one Sixth Former said ‘SCAN has more content compared with A2A where you would have to send off to actually get the information’;

o In general, users felt that the site offered a lot of information, which was sometimes difficult to assimilate and read from the screen and that more clarity was needed.

8.53 – Knowledge Base:

o one of the Sixth Form students expressed the rather discouraging view that ‘the Knowledge Base didn’t tell you anything you wanted to know’. As was mentioned above however, this group had no particular interest in history. This student added that he would have preferred an open-ended search rather than a drop down list;

o another student commented that [when searching in the Knowledge Base] ‘it would be good if you could limit or combine the search, for example Mary Queen of Scots limited by personal letters’.

o several older users in the afternoon session made positive comments about the Knowledge Base, which one called the ‘Frequently Asked Questions’. They enjoyed this and found it easy to use.

8.54 – General design:

o users from both sessions liked the ‘tab’ buttons along the top of the pages;

o others commented favourably about the layout and buttons in general;

o another one liked the ‘Click to print’ option on the pages.

Edinburgh sessions

8.55 As with the previous sessions in London and Manchester, users in Scotland noted the ‘completely different’ character of the SCAN site. They thought that with this site, ‘the expectations were more easily matched’, but that more advanced searching would eventually be needed. Phrasing in a more archive-related way the comment made by the Manchester general audience groups, these users also made the point that one of SCAN’s key differences is that it ‘focuses on data retrieval, rather than data description’ (which is what the other three sites offer). Another one stated that SCAN is ‘a learning opportunity and not a “what is where”’, which is what the other three sites are supplying, aiming more at academics/researchers’.

8.56 The users during these sessions were more familiar with the subjects that SCAN covers and in several cases had already explored the site before (see 4.7 above). It is probably for these reasons that they were the most critical reviewers of the site, offering several suggestions for improvement and further development:

8.57 – Search features:

o users felt that a more sophisticated search facility would be needed eventually, although not necessarily on the opening screen. Some expressed the view that a subject search would be useful. In general, several users in this group thought that the site would be useful once it became searchable [i.e. providing access to catalogue records].

o another user suggested that there should be separate ways into SCAN, improving on the current home page where it wasn’t clear what to do and that the site would ultimately need more elaborate search options than just a simple text search facility;

o another user, on the other hand, would prefer a simple opening screen and the option to go straight to search without waiting for the images and buttons to load.

8.58 – Knowledge Base:

o several users were positive and complimentary about the Knowledge Base;

o one user thought that in the absence of other content, the Knowledge Base predominated the site. Following on from this point, another user felt that the Knowledge Base should be a more prominent option, and that it required a much clearer opening screen;

o one participant thought that the whole site and particularly the Knowledge Base section was generally ‘incredibly wordy’ with two pages of text to introduce the testaments which he felt was too much to read. However, another user strongly disagreed, thinking that the content and layout were good and that content was generally the most important thing. As he put it ‘the more the better; after all you can always print it. For example the new entry on money and banking is long, but it is very useful and probably as succinct as it could be’.

8.59 – Glossary:

o one user commented on the SCAN glossary [in the ‘Research Tools’ option] which she thought that while it was good and easy to use, it was difficult to find.

8.60 – ‘Scottish handwriting’ section:

o one participant, a historian, was very positive about the ‘Handwriting section’, which he felt was excellent and had potential for commercialisation.

8.61 – Contact details:

o another user found it difficult to find contact details for SCAN staff.

8.62 – General design:

o some users thought that the current opening page was ‘too fancy and cluttered’ and that a simpler one was required.

9. Desirable features and facilities

Key issues and recommendations

- This chapter summarises the features and facilities that users at all sessions would like to see in the future. The suggestions of specialist users concentrated on better, more user-friendly and effective search options, and greater consistency in the design of all four strands (paragraphs 9.1 and 9.3). The Sixth Form students added also suggestions for more attractive design, greater use of graphics and images, wider support for easy free text searching, more assistance in the explanation of retrieved items, and more interactive elements (paragraph 9.2).

- Users suggested that home pages should have limited text and images and provide a clear idea of what each site offers (paragraphs 9.4-5).

- Participants were positive about the use of images and could see the potential they offered, as long as they were used appropriately to advance the sites’ aims and were cost effective (paragraphs 9.6-8). Audio was not seen as a high priority (paragraph 9.9). More experienced archive users and older participants thought that comprehensiveness of the material was more important.

9.1 The London user groups reported that they would like to see consistency in the way links are made from each strand website to participating institutions and to other strand websites, which they thought should be made more obvious. They would also like to see some kind of common practice between the sites. Free text searching and also by periods and dates is important to them and is not always available or does not work effectively in all sites. Good search help pages were also seen as a desirable feature, as the current ones were not always very helpful. They would also have liked to see a scroll bar and buttons or headers at the top of each page, so that users would know which site they were in. This was not available for all the archive pages that they looked at. An A-Level teacher said that a good biographical side to the sites would be very good for his students. Some of the London users reported that it would be useful for other sites used by overseas user groups, such as North American repositories including the US National Archives and Records Administration, to link with the British archive sites tested.

9.2 The Manchester Sixth Form students said that they would like to be able to click on images linked to the relevant text they were looking for. Printer-friendly options were seen as important for all sites. In general they also thought that there was a need for sites to be more user-friendly and look more appealing, while maintaining simplicity and ease of use. Site maps should be extended to more sites, as they can assist in navigation. Free text search should be expanded to all sites, as it was preferable to this user group, who found the featured indexes difficult to use. Clearer description of each record would be useful for giving them a better idea of what was listed. A link to information for beginners was seen as a good way to avoid information overload. A comments page and a notice board were also suggested. Finally, one participant reported that it would be very useful to be able to access the archives through the Internet, referring to the possibility of general web engines incorporating the archive records in their search and retrieval.

9.3 The Edinburgh users expressed the wish for the websites to lead to the search page immediately, as they find it irritating when anything hinders their exploration. Slow page loading was another problem for some users. Using thumbnail images of retrieved documents was mentioned by one user as a desirable feature, which he felt was used to good effect at the site of the National Maritime Museum helping users determine if the retrieved documents are useful. In general, users felt that the sites did not seem to appreciate how much of a novice the people searching the sites could be, especially family historians. They would like to see site design that is simple and straightforward with user-friendly search options, since terms such as ‘focussed searching’ can be meaningless to many users. They thought that a step by step guide, rather like the HMC handout, would be useful for each site. The use of sample searches to show likely results was another suggestion made.

Introduction to the sites and homepages

9.4 All users agreed that a good first page was important. This should not have too much text, and some would not like too many pictures either (thinking of modem speeds and related costs). The pages should offer a clear idea of what each site offered and how to use it, while other information can be linked to it, including for example, a single link for new users. The younger student group in Manchester did not think that it was necessary to put explanatory information about the actual project on the Homepage of each website, as long as a link to it was provided somewhere in the site for those who wanted to know more (e.g. ‘About Us’). An Edinburgh participant confirmed this point, suggesting that ‘introductory information on the sites should be a linked option, so you didn't need to read about each time you visited’.

9.5 Users generally felt that having an overview or a summary of each site’s contents at the homepage, might not have helped to give a clearer indication of its contents and remit, since most user normally like to get into a site immediately. Many users normally do not read any introductory text, but try to get an idea of what it is about at the first glance. Older users agreed also that having a whole page of instructions would not have helped, as people want immediate results. Some users felt that a bold, single line statement would be more effective than a ‘new user guide’, although others, particularly older users, thought that this would be useful.

Visual and aural information

9.6 All user groups, and particularly the Manchester ones, showed appreciation for visual information and commented favourably, for example, on the visual elements and graphic design of the SCAN site. They were in general in favour of greater use of images, some preferring them to the written word.

9.7 The London users reported that digital images of documents would be useful if they were legible, citing the example of the Booth Diaries at London School of Economics[4]. They saw this as the ‘next great step’ (after, for example the provision of images on microfilm that the PRO offers) that would provide exciting possibilities for printing and manipulating the documents. On the other hand, there was general agreement in all sessions with one user who reported that the priority should be access to all descriptions of the archives. Comprehensiveness was seen as more important than a ‘few digital images’ or sound files.

9.8 In Edinburgh, users added that digitised images was what they were all waiting for with SCAN, referring not to the online provision, but to the CD-ROM which they can buy at low cost. Another user added that images could be used for ‘hot spots’, frequently requested items in archives, following an educational direction after finding out what people would like to see. Others thought that CD-ROMs might be a better option for specific types of records and for audio files (as the Guildhall Library provides) as web-enabled sound files might not work for different end-users and their varying equipment.

9.9 Most users reported that audio files would not be of great importance to them and were against their gratuitous use. Some users (including one with sight disability) could see however, how the provision of audio could help those with poor eyesight, by providing for example ‘voiced descriptions’. They also thought that it would be good to have samples of manuscripts read out to give an idea especially to schoolchildren, of how people spoke at the time, while access to sound archives would be another improvement. Several users stated that it was important to look carefully at how these sites expanded and that they must concentrate on getting the basics right first, trying to fulfil each site’s main objectives. Other users raised the issue of the cost effectiveness of audio-visual material, as concentrating on it could be disadvantageous in terms of the main objectives of the site, diverting from the core work of the archive.

10. Single portal or gateway about UK archives

Key issues and recommendations

Most users supported the idea of a common gateway linking all the strand sites, with some hesitation about speed of searching or level of detail which might be lost. Several participants suggested having the option to also search separately the individual sites (paragraphs 10.1-4).

10.1 Almost all users reported that they would prefer to have a single portal or gateway linking the information about UK archives, rather than the current multi-site approach.

10.2 As a London user put it ‘a one-stop shop would be good’, but in this case, it would be important to indicate what was covered in each database and also have the option to search specifically through the archives you wanted. Another said that a single gateway would be desirable, especially if it provided links to other databases or the ability to cross search or both. Another issue raised was whether general web search engines would find information in each of the websites, performing the task of searching across all sites (which is not currently the case for all sites due to the technology used). Another suggestion made was creating a ‘Knowledge Base’ on each of the four sites, as a good stop-gap, before one portal containing detailed catalogues could be established.

10.3 Manchester users were also favourable to the idea of a common gateway, ‘perhaps with four buttons leading to the four sites, but incorporating common, quick searching across all four’ and allowing users to narrow down their searches. The users from the Forum Learning Centre thought that this would be a lot easier, but wondered whether this would take longer to load and be harder to use. A few thought that keeping all four separate was acceptable, as regular use and familiarity would help. Four separate sites might also give more information, which could be missed if combined.

10.4 Edinburgh participants agreed that it would be much better to have one gateway with the option of single searching, as it can currently be confusing for people who were not aware of different sites/funding streams. Some expressed reservation about how this would work with different search methods and different types of records. As some archives would appear in more than one site, and their catalogues would eventually be covered at different levels, they thought that this might be confusing. Another user thought that the Scottish collections should be on the SCAN site, since this represented the national network for Scotland (‘why have AIM25 and the Hub, when A2A and SCAN should be nation-wide networks for England and Scotland?’). Other users were doubtful whether users could search across all sites simultaneously, as there were too many different markets. Another user suggested that cross searching would be useful, but that the possibility of single searching should still be maintained.

11. Concluding remarks

11.1 At the end of each evaluation discussion, users at all sessions offered some general observations, words of encouragement, and final comments. One archivist from the Edinburgh group said that there was a tendency to be negative in the archive world, but that he thought that what was happening with online catalogues was a revolution. Others were very glad to be involved in the evaluation process, which they felt was valuable. One added that the sites were a definite thing to be pursued, even though there was still a lot to be done in ‘the cleaning up process’. Feeling perhaps that they had been too critical, users added that they thought that searching was becoming easier overall and noted that their specific criticisms were expressed in a background of a generally positive perception of the sites. Other professional users were optimistic that a clear system would eventually come, as what they saw as the ‘donkey work’ was being done now. They felt that this has radically changed the work of family historians and found that the rate of progress was encouraging.

Appendix 1 - Contributors to the study

London morning session – 5 December 2001,

Institute of Historical Research

Dr Geoff Browell professional research in early modern history

Caroline Brassey PhD student on Black women’s history, based in Department of Geography, UCL)

Derek Clear PhD student on the writing of history in the nineteenth century, Queen Mary College)

Rob Hodgins-Vermass PhD student on modern conflict strategy, King’s College London)

Dr Luke Macmahon professional research in Tudor military and diplomatic history; sub-editor for new Dictionary of National Biography)

Peter Robson final year English Literature undergraduate, King’s College London, researching Anglo-Saxon literature)

Roiyah Saltus-Blackwood Project researcher, CASBAH; interested in Caribbean studies)

Vanessa Taylor PhD student in 19^th century history)

Ayako Towatari PhD student, Institute of Contemporary British History)

London afternoon session – 5 December 2001

Imperial College

Clinton Chaloner PhD student on the history of photography, Imperial College

Lionel Knight History teacher, 6^th form level, City of London School

Cesare Pastorini PhD student on the history of science, Imperial College

Terence Banks PhD student on the history of science (18^th c. science and religion), Imperial College

Manchester morning session – 11 December 2001

Parrs Wood Technology College Sixth Form Students

Naomi Adelgholamzadeh

Sadia Akram

Amina Ali

Laurence Duric

Alison Ellery

Abdulkarim Fallaha

Louise Jervis

Kushwant Randhawa

Natasha Rashid

Nadia Saleem

Robert Sharples

David Silcock

Andrew Rawson

Nathan Gulum

Manchester, afternoon session – 11 December 2001

Wythneshawe Library, Forum Learning Centre

Gemma Beesley

John Buckley

Geoff Corbett

Pat Corbett

Graham Forshaw

Margaret Hobson

Ada Killip

Andrea Merrall

Pauline Rashid

Anne Sheridan

Ann Squibbs

Jayne Wainwright

Barbara Williams

Charles Williams

Edinburgh participants, morning session – 13 December 2001

Frank Bigwood Genealogist/Record Agent/Individual history researcher

Sheila Duffy Family Historian/Family History Society Official/Retired Journalist

Olive Geddes Manuscripts Librarian/Individual history researcher

Anthea Lang Local Studies Librarian

Margaret McBryde Education Officer in Archive

David McClure Individual history researcher

Bruce Read Academic Research Assistant

Helen Taylor Local Authority Archivist

Arnott Wilson University Archivist

Edinburgh participants, afternoon session – 13 December 2001

Jean Crawford Archivist

Michael Dun PhD student

Eric Graham Professional researcher

Graham Holton University Librarian/Genealogist

Marjory Howat Retired Librarian/Local History Researcher

Iain McKenzie Genealogist/Official of Family History Society

Michael Nix Individual Historian

Marion Stavert Record Agent/Individual researcher

[1] 'Archives On Line: The Establishment of a United Kingdom Archival Network' (1998) (see http://nca.archives.org.uk/ONLINE01.HTM) and 'British Archives: The Way Forward' (Nov 1998) (see http://nca.archives.org.uk/brarchs.htm).

[2] Although the international community would also benefit from use of the sites, this type of audience was beyond the remits of the study.

[3] This is a project currently under development, aiming to offer access to images from local studies collections of the Northern Region’s public libraries and other sources (www.thenortheast.com/northlib/).