Evaluation Process

A two-part evaluation of the IIS Thesaurus system was conducted in late April 2004 and included an indexing test and expert analysis. Due to time constraints imposed by the academic calendar, a formal and more comprehensive evaluation program has been postponed until further notice. Results from the indexing test are outlined below. Qualitative expert analysis will be completed on the 4th of May 2004 and a summary added to this section of the Evaluation Summary as soon as it is available.

Results from All Testers:

90.2% of all terms found were "good enough" or better at representing the desired concepts
65.4% were a "close fit," "exact" or "better/more specific" for the concept

Results from Student Indexers Only:

87.4% of all terms found were "good enough" or better at representing the desired concepts
68.4% were a "close fit," "exact" or "better/more specific" for the concept

While this quantitative analysis is not precise given a variety of variables and inconsistencies, it is enough to consider the IIS Thesaurus system a functional tool. Detailed review of both the quantitative analysis and qualitative feedback provide direct guidelines for how to improve the thesaurus.

Considering that the IIS Thesaurus tested was the very first draft of the very first thesaurus ever produced by the authors, the fact that roughly two thirds of the concepts sought by indexers were represented adequately in the thesauri is considered a great success. Details of the testing process are outlined below.

Indexing Test Procedures

A convenience sample of testers was selected from among the personal contacts of the authors and colleagues at the Graduate School of Library and Information Science of Simmons College. Testers were provided with 7 citations, which were accompanied by personal reference notes written about each book/article to aid in subject analysis (formal abstracts were not available). The testers were asked to index these citations using the IIS Thesaurus system (the Thesaurus of ERIC Descriptors and the IIS Thesaurus), rank their satisfaction with the terms they found, and provide feedback on the process, the thesauri and the materials available on the Web to support their testing.

Testing Instructions as well as the Citations for Testing were made available via the IIS Thesaurus Web site and will remain available as part of the project site (see Thesaurus Testing). Testers were also given an Evaluation Form which provided space to list concepts, terms, rank, and comments.

Testing occurred between the 19th of April and the 1st of May 2004. Because all materials were available via the Web, testers completed their evaluations independently, at times that were convenient for them. The authors were available by e-mail and phone for questions, and provided minor assistance to some of the testers.

Top

Tester Demographics

A total of six individuals completed the indexing test process. Two individuals were MLIS students, trained in indexing. The remaining group of 6 testers included 1 PhDs, 1 JD, and 2 Masters.

Variables Affecting Results

Variables and inconsistencies which make the results data difficult to compare are outlined below. Quantitative analysis conducted under these conditions cannot be considered precise, but general trends are still identifiable.

Degree of Test Completion - Four testers provided results that included concepts, terms and ranking. One tester provided concepts alone. In this case, thesaurus terms were then retrieved by one of the authors and ranked for their ability to represent the concept listed by the tester. One tester provided only thesaurus terms with narrative descriptions of their satisfaction and problems.

Variable Skills Limitations - All the testers had variable familiarity with indexing and working with thesauri. For example, untrained indexers looked more often for very specific or proper names that a trained indexer would know would never be found. In other cases, testers selected non-preferred terms instead of the preferred terms in spite of instructions to be vigilant about this issue. Each tester also would also have had different criteria for ranking their experience, in spite of the scale description given in the testing instructions.

Structural Problems in Thesauri - The challenge of not having all terms in a single place cannot be underestimated. The test results show that there were many occasions when testers settled for a "good enough" ERIC term when in fact there was a more specific IIS term available. But, without interlinking between thesauri, locating the IIS term would have required far greater effort.

Citation Problems - The selection of citations was based on convenience, an attempt to cover the range of IIS topics, and a desire to keep the notes to a manageable length. As such, they may not have been the most effective citations for testing the thesaurus. Future formal testing will involve a more deliberate citation selection process which will include the provision of full articles and/or formal abstracts rather than notes to aid in subject analysis.

Top

Quantitative Results

All Testers - A total of 336 concepts/term pairs were provided by 6 testers for 7 citations. Of these, 278 terms were ranked by satisfaction with the ability of the term found in ERIC or IIS to represent the concept the tester had identified as central to the article they were reviewing. 182 terms (65.5%) were ranked at level 4 or 5, meaning that the tester found a term that was "a very close fit" or the "exact term" or "found something even better/more specific." 24.8% (69 terms) were deemed "good enough" and 17 terms (9.7%) were declared unsatisfactory for one reason or another.

Student Indexers - A total of 79 ranked concept/term pairs were provided by 3 testers for 7 citations. 54 terms (68.4%) were ranked at level 4 or 5, meaning that the tester found a term that was "a very close fit" or the "exact term" or "found something even better/more specific." 19.0% (15 terms) were deemed "good enough" and 12 terms (12.6%) were declared unsatisfactory for one reason or another.

Top

Qualitative Results

Feedback from Indexers - Comments from student indexers praised the overall thesaurus system and commented on the challenges of using two thesauri (not easy to discover better terms in the other thesaurus). They also made useful suggestions for improvement, including better scope notes on the key IIS terminology, and some clearer and more consistent language in the instructions. Indexers also helped to identify spelling and typographical errors.

"My sense is that I may not have gone to the lowest level when picking terms... came across a lot of direct matches in ERIC, but it seemed like there was a bit more specificity in IIS."
- GSLIS Student

Feedback from Untrained Testers - Comments from untrained testers revealed that the thesaurus introduction needed to be clearer so that they could more easily find explanations when they needed them (e.g. of the abbreviations, of why there were ERIC terms in the IIS thesaurus, of why the key IIS terms were so important).

"God help the professional catalogers who are really going to do this. Doing this all day would tie my brain in knots that could never be undone."
- Untrained Tester

Top

Thesaurus Improvement Recommendations

Adding Preferred and Non-Preferred Terminology - The concepts and terms which received lower rankings (1-2) provide a quick list of candidates to be reviewed as potential non-preferred terminology. For example, "matrix" related concepts were searched for by only one tester, and it is unlikely that a preferred term would be necessary, but a non-preferred term might be recommended to lead the user to a more general statistical analysis concept. Some terms not found at all by testers do actually exist in ERIC or IIS. This suggests that entry vocabulary on these topics is not good enough to lead the user to the proper term.

Quite a few terms in the 3 level ranking received individual comments and suggest the possible addition of non-preferred terminology or perhaps even a preferred term. For example, the term "Collaborative Research" is included in IIS as non-preferred terminology (the user is instructed to use "Cooperation (ERIC)" and "Research (ERIC)"). This test revealed, however, that almost every user listed "collaborative research" as a concept in more than one article and was only moderately satisfied with how they were able to represent it. This suggests that the term might be more important to the field of Integration and Implementation Sciences than originally thought and should be a preferred term.

Distinguishing Non-Preferred Terms - The fact that some testers used non-preferred terminology in spite of instructions to the contrary suggests that perhaps they were not actually reviewing the details of the term itself (most likely in ERIC, which provides a preliminary term list in response to a search which offers no distinction between preferred and non-preferred terms). Using a different font, style (e.g. italic), or other visual designation with the word to distinguish non-preferred terms might help prevent this problem.

Updating the Thesaurus Introduction - The questions and frustrations received by many testers indicate that the thesaurus introduction should be more extensive, clearer, and more effectively organized. Given the need to begin testing as soon as possible, the introduction made available to testers was preliminary. It was updated during the testing process and provided to some testers for review. Feedback on the updated version was positive and it has been incorporated into the IIS Thesaurus Project Web site as of 3 May 2004.

Expert Review
Dr. Gabriele Bammer, the key scholar developing Integration and Implementation Sciences, reviewed the IIS thesaurus generally and also completed the test. Overall, she found the thesaurus to be a good representation of the field as it is currently defined and organized and was very pleased with the product.

"I am just delighted with what you have achieved and with the vision you have set for what is possible and desirable. You have done a highly commendable task of accurately reflecting what Integration and Implementation Sciences is about."
- Dr. Gabriele Bammer

The primary purpose of a thesaurus usually is to serve as a controlled vocabulary for indexing resources within a specific field of study or application. Integration and Implementation Sciences, however, is a very new field. It is difficult to build a thesaurus when the important concepts of the field itself are still being hammered out. Even so, Dr. Bammer’s test of the thesaurus indicated that the IIS thesaurus system was highly successful in representing key concepts from seven different citations (95.3% of terms found were ranked “good enough” or better by Dr. Bammer).

There are other puposes of thesauri, however, that are particularly relevant for growing fields like Integration and Implementation Sciences. Two of the four uses identified by Booth (see the Annotated Bibliography) demonstrate the value of a thesaurus as a source for subject familiarization and to provide meaning and authenticity checks. It is in this context that Dr. Bammer discussed the great value of this first draft of the IIS Thesaurus. Bammer found the thesaurus provided an essential framework for further discussions with colleagues. These discussions will help to clarify the field by cementing definitions and relationships between key concepts.

"One of the many benefits of the thesaurus development exercise is that it has helped me clarify my thinking about Integration and Implementation Sciences, and your very accurate work has made me acutely aware of the short-comings of existing descriptions."
- Dr. Gabriele Bammer

After discussion of the quantitative results of Dr. Bammer’s testing and the language of the thesaurus, the following sections offer a brief summary of Dr. Bammer’s thoughts about how to proceed with the IIS Thesaurus. These reflections are included in order to provide readers with insight into both how a new disciplinary specialization develops, and the great value of thesaurus construction beyond the production of an indexing tool.

Quantitative

A total of 87 concepts/term pairs were identified by Dr. Bammer for 7 citations. Of these, 78 terms were ranked by satisfaction with the ability of the term found in ERIC or IIS to represent the concept Dr. Bammer had identified as central to the article she was reviewing.

95.3% of terms found were ranked "good enough" or better at representing the desired concepts (74 terms)
85.9% were a "close fit," or "better/more specific" for the concept (67 terms)

Language

Much to the joy and chagrin of the authors, Dr. Bammer has relieved the IIS Thesaurus of its mandate to use British English. While it is unfortunate that the authors did not inquire directly about this issue sooner, this edict will make reconciling the IIS Thesaurus with ERIC much easier in subsequent stages.

Classification

Dr. Bammer indicated that the current structure of the thesaurus (as evident in the TREE display) does adequately represent the current state of the field as evidenced in available literature and other resources (i.e. the IIS Web site). The act of viewing the structure in this way, however, revealed to her that perhaps there were better ways for the field to be organized. The current structure does not make the relationships between key elements and functions as clear as they ought to be (e.g. the contributions of the key research areas, the unique theory and methods of IIS itself, the conditions under which IIS theory and methods are developed and applied, the necessary tools for these processes).

She agreed that as further clarification of terminology occurs, a more effective structure of classification and organization will undoubtedly emerge.

Value in clarifying thinking for the field

Dr. Bammer discussed the ongoing challenges she has encountered when discussing meaning and definitions with various colleagues. She is particularly averse to the creation of new terms for every new concept, but without them it becomes exceptionally important to be clear about the use and context of existing terms. The IIS thesaurus has provided her with a focal point to more effectively guide those discussions, and she plans to include terminological review among the upcoming tasks for the field.

"I am also very pleased to have a framework in which to conduct debates about terminology. As the field grows, agreed definitions of terms and concepts are crucial, but such debates can often be self-referential and sterile. Having a real world use context to guide those debates is of immeasurable value."

One example where the IIS thesaurus has stimulated action involves the concepts of transfer, translation and transformation. The study and application of these concepts comprises a 7th key research area missing from the thesaurus. The thesaurus remains accurate based on what was presented on the IIS Web site at the time the thesaurus was constructed, but as these concepts have grown more important, the currently weak treatment in the IIS thesaurus has motivated Dr. Bammer to clarify definitions and roles for these areas so that they may be included at their full weight in the next draft.

Multi-thesauri system and Next Steps

The single greatest challenge for immediate use of the thesaurus for review and discussion by IIS scholars is the fact that it is distributed among three thesauri (ERIC, IIS and UNESCO). While this distribution is functional (if annoying) for indexers, it is completely inadequate for those attempting to review the terminology of the entire discipline as a whole.

Dr. Bammer considered the cumbersome task of trying to import or enter all relevant ERIC terms into the IIS Thesaurus, and suggested that perhaps it would be easier to embark on a campaign to get ERIC to include the IIS terms from this thesaurus. The authors advised her that the process of proposing new terms to a thesaurus as established as ERIC was not feasible. The next suggestion was to break the thesaurus into component disciplinary parts, so that the work to further develop a single IIS thesaurus (by incorporating ERIC terms and possibly others) would be more likely to receive funding.

For example, IIS draws on the scholarship of different research areas to varying degrees. While virtually all of the study of Participatory Methods is relevant to IIS, only part of Complexity Science is relevant. It was suggested that the development of a Participatory Methods glossary/thesaurus might be useful to that field (and therefore fundable by mediation, facilitation or other public participation organizations) and would ultimately provide a fully inclusive chunk of the IIS Thesaurus.

Any parties interested in sponsoring such a project are encouraged to contact either Dr. Gabriele Bammer or Caryn Anderson.

Overall the IIS Thesaurus was considered by Dr. Bammer to be a great success, if more for its motivational use than for immediate indexing. The authors are deeply grateful to Dr. Bammer for her time and feedback.

Top