ERIC Number: EJ1144763
Record Type: Journal
Publication Date: 2017-Jun
Pages: 18
Abstractor: As Provided
ISBN: N/A
ISSN: EISSN-1368-1613
EISSN: N/A
Document Level Assessment of Document Retrieval Systems in a Pairwise System Evaluation
Rajagopal, Prabha; Ravana, Sri Devi
Information Research: An International Electronic Journal, v22 n2 Jun 2017
Introduction: The use of averaged topic-level scores can result in the loss of valuable data and can cause misinterpretation of the effectiveness of system performance. This study aims to use the scores of each document to evaluate document retrieval systems in a pairwise system evaluation. Method: The chosen evaluation metrics are document-level precision scores against topic-level average precision (AP) scores, and document-level rank-biased precision scores against topic-level rank-biased precision at cut-off k (k = 100) scores. Analysis: An analysis of the results of paired significance tests with the use of document-level and topic-level scores are compared to determine the agreement in the obtained numbers of statistically significant information retrieval system pairs. Results: The experiment results at document-level are an effective evaluation unit in the pairwise evaluation of information retrieval systems, with higher numbers of statistically significant (p = 0.01) system pairs, compared with the topic-level results and a high percentage of statistically significant agreement with topic-level. Conclusion: This study presents an original viewpoint on measuring the effectiveness of document retrieval systems through pairwise evaluation by using document-level scores as a unit of evaluation in the significance testing instead of the traditional topic-level scores (which involve averaging document scores).
Descriptors: Information Retrieval, Documentation, Scores, Information Systems, Measurement, Statistical Analysis, Hypothesis Testing, Statistical Significance, Evaluation Methods, Accuracy, Comparative Analysis, Classification
Thomas D. Wilson. 9 Broomfield Road, Broomhill, Sheffield, S10 2SE, UK. Web site: http://informationr.net/ir
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A