ERIC Number: ED027909
Record Type: RIE
Publication Date: 1967-Oct
Reference Count: N/A
A Field Experimental Approach to the Study of Relevance Assessments in Relation to Document Searching. Final Report to the National Science Foundation. Volume I.
Rees, Alan M; Schultz, Douglas G.
An empirical study of the nature and variability of the relevance judgment process was conducted from July 1, 1965 to September 30, 1967. Volume I of the final report presents a literature review and statement of the theoretical framework of the study, a discussion of the experimental design and a summary of data analyses. The study had two objectives: (1) to construct and test a methodology for the investigation of the relevance judgment process and (2) to gather data descriptive of this process. The experimental design focused on four independent variables: judges, research stages, documents and document representations, in addition to the primary dependent variable, the relevance rating. The analyses investigated the effect of the primary variables and their interactions upon the relevance rating. Biographical characteristics of the judges were also studied, and data suggest that the individual variability which characterized the relevance ratings assigned by judges was apparently determined by a complex of biographical characteristics. Data also indicate that relevance ratings were influenced by the rating scale employed and by the incremental information given to the judge about the information need and document. Findings suggest that the use of relevance judgments as an evaluative measure for the output of information retrieval systems must be a complex, well-controlled procedure. (Author/JB)
Descriptors: Comparative Analysis, Content Analysis, Decision Making, Evaluation Methods, Individual Characteristics, Information Needs, Information Systems, Input Output Analysis, Medical Libraries, Medical Research, Medical Students, Operations Research, Performance Criteria, Physicians, Rating Scales, Relevance (Information Retrieval), Reliability, Scientists
Clearinghouse for Federal Scientific and Technical Information, Springfield, Va. 22151 (PB 176-080, MF $0.65, HC $3.00).
Publication Type: N/A
Education Level: N/A
Sponsor: National Science Foundation, Washington, DC.
Authoring Institution: Case Western Reserve Univ., Cleveland, OH. Center for Documentation and Communication Research.; Case Western Reserve Univ., Cleveland, OH. Dept. of Psychology.
Note: LI 000 435 is related document, the appendices to this report.