A crowdsourcing engine for mechanized labor


Ustalov, Dmitry



DOI: https://doi.org/10.15514/ISPRAS-2015-27(3)-25
URL: http://www.mathnet.ru/links/f485efaa5210ed813fd056...
Additional URL: http://www.ispras.ru/en/proceedings/isp_27_2015_3/...
Document Type: Article
Year of publication: 2015
The title of a journal, publication series: Trudy Instituta sistemnogo programmirovanija RAN = Proceedings of the Institute for System Programming of the RAS
Volume: 27
Issue number: 3
Page range: 351-364
Place of publication: Moskva
Publishing house: Institut sistemnogo programmirovanija
ISSN: 2079-8156 , 2220-6426
Publication language: English
Institution: School of Business Informatics and Mathematics > Wirtschaftsinformatik III (Ponzetto 2016-)
Subject: 004 Computer science, internet
Abstract: Microtask crowdsourcing implies decomposing a difficult problem into smaller pieces. For that a special human-computer platform like CrowdFlower or Amazon Mechanical Turk is used to submit tasks for human workers motivated by either micropayments or altruism to solve. Examples of successful crowdsourcing applications are food nutrition estimation, natural language processing, criminal invasion detection, and other problems so-called “AI-hard”. However, these platforms are proprietary and requiring additional software for maintaining the output quality. This paper presents the design, architecture and implementation details of an open source engine for executing microtask-based crowdsourcing annotation stages. The engine controls the entire crowdsourcing process including such elements as task allocation, worker ranking, answer aggregation, agreement assessment, and other means for quality control. The present version of the software is implemented as a three-tier system, which is composed of the application level for the end-user worker interface, the engine level for the Web service controlling the annotation process, and the database level for the data persistence. The RESTful API is used for interacting with the engine. The methods for controlling the annotation are implemented as processors that are initialized using the dependency injection mechanism for achieving the loose coupling principle. The functionality of the engine has been evaluated by both using unit tests and replication of a semantic similarity assessment experiment.

Dieser Datensatz wurde nicht während einer Tätigkeit an der Universität Mannheim veröffentlicht, dies ist eine Externe Publikation.




+ Citation Example and Export

Ustalov, Dmitry (2015) A crowdsourcing engine for mechanized labor. Trudy Instituta sistemnogo programmirovanija RAN = Proceedings of the Institute for System Programming of the RAS Moskva 27 3 351-364 [Article]


+ Search Authors in

+ Page Views

Hits per month over past year

Detailed information



You have found an error? Please let us know about your desired correction here: E-Mail


Actions (login required)

Show item Show item