Search
Now showing items 1-3 of 3
Mix and match: Collaborative expert-crowd judging for building test collections accurately and affordably
(
CEUR-WS
, 2018 , Conference Paper)
Crowdsourcing offers an affordable and scalable means to collect relevance judgments for information retrieval test collections. However, crowd assessors may showhigher variance in judgment quality than trusted assessors. ...
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to Ensure Quality Relevance Annotations
(
AAAI Press
, 2018 , Conference Paper)
While peer-agreement and gold checks are well-established methods for ensuring quality in crowdsourced data collection, we explore a relatively new direction for quality control: estimating work quality directly from ...
The many benefits of annotator rationales for relevance judgments
(
International Joint Conferences on Artificial Intelligence
, 2017 , Conference Paper)
When collecting subjective human ratings of items, it can be difficult to measure and enforce data quality due to task subjectivity and lack of insight into how judges arrive at each rating decision. To address this, we ...