TY - GEN
T1 - What deliberately degrading search quality tells us about discount functions
AU - Thomas, Paul
AU - Jones, Timothy
AU - Hawking, David
PY - 2011
Y1 - 2011
N2 - Deliberate degradation of search results is a common tool in user experiments. We degrade high-quality search results by inserting non-relevant documents at different ranks. The effect of these manipulations, on a number of commonly-used metrics, is counter-intuitive: the discount functions implicit in P@k, MRR, NDCG, and others do not account for the true relationship between rank and value to the user. We propose an alternative, based on visibility data.
AB - Deliberate degradation of search results is a common tool in user experiments. We degrade high-quality search results by inserting non-relevant documents at different ranks. The effect of these manipulations, on a number of commonly-used metrics, is counter-intuitive: the discount functions implicit in P@k, MRR, NDCG, and others do not account for the true relationship between rank and value to the user. We propose an alternative, based on visibility data.
KW - Metrics
KW - Result set manipulation
UR - http://www.scopus.com/inward/record.url?scp=80052129492&partnerID=8YFLogxK
U2 - 10.1145/2009916.2010072
DO - 10.1145/2009916.2010072
M3 - Conference contribution
SN - 9781450309349
T3 - SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval
SP - 1107
EP - 1108
BT - SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval
PB - Association for Computing Machinery
T2 - 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2011
Y2 - 24 July 2011 through 28 July 2011
ER -