Muwanei, Sinyinda and Ravana, Sri Devi and Hoo, Wai Lam and Kunda, Douglas and Rajagopal, Prabha and Sodhi, Prabhpreet Singh (2022) Correlation and prediction of high-cost information retrieval evaluation metrics using deep learning. Information Research-An International Electronic Journal, 27 (1). ISSN 1368-1613, DOI https://doi.org/10.47989/irpaper920.
Full text not available from this repository.Abstract
Introduction. To reduce cost of the evaluation of information retrieval systems, this study proposes a method that employs deep learning to predict the precision evaluation metric. It also aims to show why some of existing evaluation metrics correlate with each other while considering the varying distributions of relevance assessments. It aims to ensure reproducibility of all the presented experiments. Method. Using data from several test collections of the Text REetrieval Conference (TREC) we show why some evaluation metrics correlate with each other, through mathematical intuitions. In addition, regression models were used to investigate how the predictions of the evaluation metrics are affected by queries or topics with variations of relevance assessments. Lastly, the proposed prediction method employs deep learning. Analysis. We use coefficient of determination, Kendall's tau, Spearman and Pearson correlations. Results. This study showed that the proposed method performed better predictions than other recently proposed methods in retrieval research. It also showed why the correlation exists between precision and rank biased precision metrics, and why recall and average precision metrics have reduced correlation when the cut-off depth increases. Conclusions. The proposed method and the justifications for the correlations between some pairs of retrieval metrics will be valuable to researchers for the predictions of the evaluation metrics of information retrieval systems.
| Item Type: | Article |
|---|---|
| Funders: | University of Malaya RU Grant [Gerant No; GPF001D-2018], University of Malaya Interdisciplinary Research Grant Programme by Ministry of Higher Education, Malaysia [Grant No: IIRG005B-2020SAH] |
| Uncontrolled Keywords: | Judgments; Topics |
| Subjects: | Q Science > QA Mathematics Q Science > QA Mathematics > QA75 Electronic computers. Computer science Q Science > QA Mathematics > QA76 Computer software |
| Divisions: | Faculty of Computer Science & Information Technology Faculty of Computer Science & Information Technology > Department of Computer System & Technology |
| Depositing User: | Ms. Juhaida Abd Rahim |
| Date Deposited: | 17 Oct 2023 01:37 |
| Last Modified: | 17 Oct 2023 01:37 |
| URI: | http://eprints.um.edu.my/id/eprint/42045 |
Actions (login required)
 |
View Item |
