Publication

The role of negative results for choosing an evaluation approach - a recommender systems case study

Heitmann, Benjamin
Hayes, Conor
Loading...
Thumbnail Image
Repository DOI
Publication Date
2015-06-01
Type
Workshop paper
Downloads
Citation
Heitmann, Benjamin, & Hayes, Conor, (2015). The Role of Negative Results for Choosing an Evaluation Approach - A Recommender Systems Case Study Paper presented at the NoISE 2015 : Workshop on Negative or Inconclusive Results in Semantic Web, Portoroz, Slovenia, 01 June.
Abstract
We describe a case study, which shows how important negative results are in uncovering biased evaluation methodologies. Our re- search question is how to compare a recommender algorithm that uses an RDF graph to a recommendation algorithm that uses rating data. Our case study uses DBpedia 3.8 and the MovieLens 100k data set. We show that the most popular evaluation protocol in the recommender sys- tems literature is biased towards evaluating collaborative filtering (CF) algorithms, as it uses the rating prediction task. Based on the negative results of this first experiment, we find an alternative evaluation task, the top-k recommendation task. While this task is harder to perform, our positive results show that it is a much better fit, which is not biased to- wards either CF or our graph-based algorithm. The second set of results are statistically significant (Wilcoxon rank sum test, p
Publisher
CEUR Workshop Proceedings
Publisher DOI
Rights
Attribution-NonCommercial-NoDerivs 3.0 Ireland