Many computer scientists use the aggregated answers of online workers torepresent ground truth . Prior work has shown aggregation methods such asmajority voting are effective for measuring relatively objective features such as semantic connotation . We propose a quality-aware semantic data annotation system . We observe that with feedback on workers’ performance quantified by quality scores, betterinformed online workers can maintain the quality of their labeling throughout an extended period of time . We validate the effectiveness of the proposed system through evaluating performance based on an expert-labeleddataset, and demonstrating machine learning tasks that can lead to consistent learning behavior with 70%-80% accuracy . Our results suggest thatwith our system, researchers can collect high-quality answers of subjectivesemantic features at a large scale .

Author(s) : Jiele Wu, Chau-Wai Wong, Xinyan Zhao, Xianpeng Liu

Links : PDF - Abstract

Code :

Keywords : quality - system - workers - semantic - features -

Leave a Reply

Your email address will not be published. Required fields are marked *