AI-Assisted Human Labeling: Batching for Efficiency without Overreliance

Zahra Ashktorab; Michael Desmond; Josh Andres; Michael Muller; Narendra Nath Joshi; Michelle Brachman; Aabhas Sharma; Kristina Brimijoin; Qian Pan; Christine T. Wolf; Evelyn Duesterwald; Casey Dugan; Werner Geyer; Darrell Reimer

doi:10.1145/3449163

AI-Assisted Human Labeling: Batching for Efficiency without Overreliance

Zahra Ashktorab, Michael Desmond, Josh Andres, Michael Muller, Narendra Nath Joshi, Michelle Brachman, Aabhas Sharma, Kristina Brimijoin, Qian Pan, Christine T. Wolf, Evelyn Duesterwald, Casey Dugan, Werner Geyer, Darrell Reimer

Research output: Contribution to journal › Article › peer-review

24 Citations (Scopus)

Abstract

Human labeling of training data is often a time-consuming, expensive part of machine learning. In this paper, we study "batch labeling", an AI-assisted UX paradigm, that aids data labelers by allowing a single labeling action to apply to multiple records. We ran a large scale study on Mechanical Turk with 156 participants to investigate labeler-AI-batching system interaction. We investigate the efficacy of the system when compared to a single-item labeling interface (i.e., labeling one record at-a-time), and evaluate the impact of batch labeling on accuracy and time. We further investigate the impact of AI algorithm quality and its effects on the labelers' overreliance, as well as potential mechanisms for mitigating it. Our work offers implications for the design of batch labeling systems and for work practices focusing on labeler-AI-batching system interaction.

Original language	English
Article number	89
Journal	Proceedings of the ACM on Human-Computer Interaction
Volume	5
Issue number	CSCW1
DOIs	https://doi.org/10.1145/3449163
Publication status	Published - 22 Apr 2021
Externally published	Yes

Access to Document

10.1145/3449163

Cite this

Ashktorab, Z., Desmond, M., Andres, J., Muller, M., Joshi, N. N., Brachman, M., Sharma, A., Brimijoin, K., Pan, Q., Wolf, C. T., Duesterwald, E., Dugan, C., Geyer, W., & Reimer, D. (2021). AI-Assisted Human Labeling: Batching for Efficiency without Overreliance. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), Article 89. https://doi.org/10.1145/3449163

@article{ec2e1d45e3f44f00b081e4ff98ff80ac,

title = "AI-Assisted Human Labeling: Batching for Efficiency without Overreliance",

abstract = "Human labeling of training data is often a time-consuming, expensive part of machine learning. In this paper, we study {"}batch labeling{"}, an AI-assisted UX paradigm, that aids data labelers by allowing a single labeling action to apply to multiple records. We ran a large scale study on Mechanical Turk with 156 participants to investigate labeler-AI-batching system interaction. We investigate the efficacy of the system when compared to a single-item labeling interface (i.e., labeling one record at-a-time), and evaluate the impact of batch labeling on accuracy and time. We further investigate the impact of AI algorithm quality and its effects on the labelers' overreliance, as well as potential mechanisms for mitigating it. Our work offers implications for the design of batch labeling systems and for work practices focusing on labeler-AI-batching system interaction.",

keywords = "agents, ai, collaboration, data labeling",

author = "Zahra Ashktorab and Michael Desmond and Josh Andres and Michael Muller and Joshi, {Narendra Nath} and Michelle Brachman and Aabhas Sharma and Kristina Brimijoin and Qian Pan and Wolf, {Christine T.} and Evelyn Duesterwald and Casey Dugan and Werner Geyer and Darrell Reimer",

note = "Publisher Copyright: {\textcopyright} 2021 ACM.",

year = "2021",

month = apr,

day = "22",

doi = "10.1145/3449163",

language = "English",

volume = "5",

journal = "Proceedings of the ACM on Human-Computer Interaction",

issn = "2573-0142",

publisher = "Association for Computing Machinery (ACM)",

number = "CSCW1",

}

TY - JOUR

T1 - AI-Assisted Human Labeling

T2 - Batching for Efficiency without Overreliance

AU - Ashktorab, Zahra

AU - Desmond, Michael

AU - Andres, Josh

AU - Muller, Michael

AU - Joshi, Narendra Nath

AU - Brachman, Michelle

AU - Sharma, Aabhas

AU - Brimijoin, Kristina

AU - Pan, Qian

AU - Wolf, Christine T.

AU - Duesterwald, Evelyn

AU - Dugan, Casey

AU - Geyer, Werner

AU - Reimer, Darrell

PY - 2021/4/22

Y1 - 2021/4/22

N2 - Human labeling of training data is often a time-consuming, expensive part of machine learning. In this paper, we study "batch labeling", an AI-assisted UX paradigm, that aids data labelers by allowing a single labeling action to apply to multiple records. We ran a large scale study on Mechanical Turk with 156 participants to investigate labeler-AI-batching system interaction. We investigate the efficacy of the system when compared to a single-item labeling interface (i.e., labeling one record at-a-time), and evaluate the impact of batch labeling on accuracy and time. We further investigate the impact of AI algorithm quality and its effects on the labelers' overreliance, as well as potential mechanisms for mitigating it. Our work offers implications for the design of batch labeling systems and for work practices focusing on labeler-AI-batching system interaction.

AB - Human labeling of training data is often a time-consuming, expensive part of machine learning. In this paper, we study "batch labeling", an AI-assisted UX paradigm, that aids data labelers by allowing a single labeling action to apply to multiple records. We ran a large scale study on Mechanical Turk with 156 participants to investigate labeler-AI-batching system interaction. We investigate the efficacy of the system when compared to a single-item labeling interface (i.e., labeling one record at-a-time), and evaluate the impact of batch labeling on accuracy and time. We further investigate the impact of AI algorithm quality and its effects on the labelers' overreliance, as well as potential mechanisms for mitigating it. Our work offers implications for the design of batch labeling systems and for work practices focusing on labeler-AI-batching system interaction.

KW - agents

KW - ai

KW - collaboration

KW - data labeling

UR - http://www.scopus.com/inward/record.url?scp=85125031808&partnerID=8YFLogxK

U2 - 10.1145/3449163

DO - 10.1145/3449163

M3 - Article

SN - 2573-0142

VL - 5

JO - Proceedings of the ACM on Human-Computer Interaction

JF - Proceedings of the ACM on Human-Computer Interaction

IS - CSCW1

M1 - 89

ER -

AI-Assisted Human Labeling: Batching for Efficiency without Overreliance

Abstract

Access to Document

Other files and links

Fingerprint

Cite this