Rajagopal, Prabha and Ravana, Sri Devi and Koh, Yun Sing and Balakrishnan, Vimala (2019) Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment. Aslib Journal of Information Management, 71 (1). pp. 2-17. ISSN 2050-3806, DOI https://doi.org/10.1108/AJIM-04-2018-0086.
Full text not available from this repository.Abstract
Purpose: The effort in addition to relevance is a major factor for satisfaction and utility of the document to the actual user. The purpose of this paper is to propose a method in generating relevance judgments that incorporate effort without human judges’ involvement. Then the study determines the variation in system rankings due to low effort relevance judgment in evaluating retrieval systems at different depth of evaluation. Design/methodology/approach: Effort-based relevance judgments are generated using a proposed boxplot approach for simple document features, HTML features and readability features. The boxplot approach is a simple yet repeatable approach in classifying documents’ effort while ensuring outlier scores do not skew the grading of the entire set of documents. Findings: The retrieval systems evaluation using low effort relevance judgments has a stronger influence on shallow depth of evaluation compared to deeper depth. It is proved that difference in the system rankings is due to low effort documents and not the number of relevant documents. Originality/value: Hence, it is crucial to evaluate retrieval systems at shallow depth using low effort relevance judgments.
| Item Type: | Article |
|---|---|
| Funders: | Universiti Malaya and the Ministry of Higher Education, Malaysia: UMRG Programme RP059A-17SBS |
| Uncontrolled Keywords: | Information retrieval; Information system; Large-scale experimentation; Relevance judgments; System-oriented evaluation; TREC |
| Subjects: | Q Science > QA Mathematics > QA75 Electronic computers. Computer science Q Science > QA Mathematics > QA76 Computer software |
| Divisions: | Faculty of Computer Science & Information Technology |
| Depositing User: | Ms. Juhaida Abd Rahim |
| Date Deposited: | 15 Jan 2019 03:06 |
| Last Modified: | 15 Jan 2019 03:06 |
| URI: | http://eprints.um.edu.my/id/eprint/20008 |
Actions (login required)
 |
View Item |
