ERIC Number: EJ749109
Record Type: Journal
Publication Date: 2006-Oct
Pages: 17
Abstractor: Author
ISBN: N/A
ISSN: ISSN-1093-023X
EISSN: N/A
Improving Performance Support Systems through Information Retrieval Evaluation
Schatz, Steven
Journal of Interactive Learning Research, v17 n4 p407-423 Oct 2006
This study examines existent and new methods for evaluating the success of information retrieval systems. The theory underlying current methods is not robust enough to allow testing retrieval using different meta-tagging schema's. Traditional measures rely on judgments of whether a document is relevant to a particular question. A good system returns all the relevant documents and no extraneous documents. There is a rich literature questioning the efficacy of relevance judgments. Such questions as, Relevant to whom? When? and To what purpose? are not well-answered in traditional theory. In this study, two new measures (Spink's Information Need and Cooper's Utility) are used in evaluating two search tools (tag-based and text-based), comparing these new measures with traditional measures and each other. The open-source Swish text-based search engine and a self-constructed tag-based search tool were used. Thirty-four educators searched for information using both search engines and evaluated the information retrieved by each. Construct measures, derived by multiplying each of the three measures (traditional, information need, and utility) by a rating of satisfaction were compared using two way analysis of variance. This study specifically analyzes small information systems. The design concepts would be untenable for large systems. Results indicated that there was a significant correlation between the three measures, indicating that the new measures provide an equivalent method of evaluating systems and have some significant advantages, which include not requiring relevance judgments and the ability to use the measures in situ.
Descriptors: Statistical Analysis, Search Engines, Information Needs, Information Systems, Engines, Information Retrieval, Evaluation
Association for the Advancement of Computing in Education. P.O. Box 1545, Chesapeake, VA 23327-1545. Tel: 757-366-5606; Fax: 703-997-8760; e-mail: info@aace.org; Web site: http://www.aace.org
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A