A little competition never hurt anyone's relevance assessments

Yuan Jin; Mark J. Carman; Lexing Xie

A little competition never hurt anyone's relevance assessments

Yuan Jin
, Mark J. Carman
, Lexing Xie

Research output: Contribution to journal › Conference article › peer-review

2 Citations (Scopus)

Abstract

This paper investigates the effect of real-time performance feedback and competition on the accuracy of crowd-workers for a document relevance assessment task. Through a series of controlled tests, we show that displaying a leaderboard to crowd-workers can motivate them to improve their performance, providing a bonus is offered to the best performing workers. This effect is observed even when test questions are used to enforce quality control during task completion.

Original language	English
Pages (from-to)	29-36
Number of pages	8
Journal	CEUR Workshop Proceedings
Volume	1642
Publication status	Published - 2016
Event	3rd International Workshop on Gamification for Information Retrieval, GamifIR 2016 - Pisa, Italy Duration: 21 Jul 2016 → …

Cite this

@article{c02b8717a1a340b0ac602113522929ce,

title = "A little competition never hurt anyone's relevance assessments",

abstract = "This paper investigates the effect of real-time performance feedback and competition on the accuracy of crowd-workers for a document relevance assessment task. Through a series of controlled tests, we show that displaying a leaderboard to crowd-workers can motivate them to improve their performance, providing a bonus is offered to the best performing workers. This effect is observed even when test questions are used to enforce quality control during task completion.",

author = "Yuan Jin and Carman, \{Mark J.\} and Lexing Xie",

note = "Publisher Copyright: Copyright {\textcopyright} by the paper's authors.; 3rd International Workshop on Gamification for Information Retrieval, GamifIR 2016 ; Conference date: 21-07-2016",

year = "2016",

language = "English",

volume = "1642",

pages = "29--36",

journal = "CEUR Workshop Proceedings",

issn = "1613-0073",

publisher = "CEUR-WS",

}

TY - JOUR

T1 - A little competition never hurt anyone's relevance assessments

AU - Jin, Yuan

AU - Carman, Mark J.

AU - Xie, Lexing

PY - 2016

Y1 - 2016

N2 - This paper investigates the effect of real-time performance feedback and competition on the accuracy of crowd-workers for a document relevance assessment task. Through a series of controlled tests, we show that displaying a leaderboard to crowd-workers can motivate them to improve their performance, providing a bonus is offered to the best performing workers. This effect is observed even when test questions are used to enforce quality control during task completion.

AB - This paper investigates the effect of real-time performance feedback and competition on the accuracy of crowd-workers for a document relevance assessment task. Through a series of controlled tests, we show that displaying a leaderboard to crowd-workers can motivate them to improve their performance, providing a bonus is offered to the best performing workers. This effect is observed even when test questions are used to enforce quality control during task completion.

UR - https://www.scopus.com/pages/publications/84985914355

M3 - Conference article

AN - SCOPUS:84985914355

SN - 1613-0073

VL - 1642

SP - 29

EP - 36

JO - CEUR Workshop Proceedings

JF - CEUR Workshop Proceedings

T2 - 3rd International Workshop on Gamification for Information Retrieval, GamifIR 2016

Y2 - 21 July 2016

ER -

A little competition never hurt anyone's relevance assessments

Abstract

Other files and links

Fingerprint

Cite this