We are CrowdLab in the Department of Computer Science & Engineering at the University of Washington.

Our group is excited by combining machine and human intelligence to accomplish more than either could alone.

We use techniques from decision theory, machine learning, and human-computer interaction to leverage the power of the crowd.

Projects

MicroTalk is a crowdsourcing system that uses argumentation to boost worker accuracy. MicroTalk requires some workers to justify their responses and asks others to Reconsider their decisions after reading counter-arguments from workers with opposing views. Argumentation improves worker accuracy by 20% and achieves a high accuracy on a hard relation extraction task.

Guru is a reinforcement learning agent that optimizes crowdsourcing quality by optimally testing and teaching workers. By optimally testing and replacing workers, Guru can improve output results by up to 111% for the same cost.

Deluge is a crowdsourcing system for multi-label classification. By learning a decision-theoretic model and label co-occurrence probabilities, it asks the crowd about labels which provide maximum information gain toward a joint classification. Deluge uses less than 10% of the labor required by a round-robin labeling approach.

[Bragg, Mausam, Weld. Crowdsourcing Multi-Label Classification for Taxonomy Creation. In HCOMP 2013.]

Bootstrapping an online community is challenging because there is not enough content to attract a critical mass of active members. This project is examining how a system can use a decision-theoretic optimization model to address this “cold-start” problem by efficiently attracting non-members to contribute to the community.

[Huang, Bragg, Etzioni, Cowhey, & Weld. Toward Automatic Bootstrapping of Online Communities Using Decision-theoretic Optimization. In CSCW 2016.]

Reactive Learning - One of the most popular uses of crowdsourcing is to provide training data for supervised machine learning algorithms. When we wish to train the best classifier possible at the lowest cost, active learning algorithms can intelligently pick new examples for the crowd to label. However, because the labels are noisy, algorithms should also be able to pick examples to relabel, in order to decrease the overall noise of the training data. In our study of reactive learning, we seek to understand the difference in marginal value between decreasing the noise of a training set, via relabeling, and increasing the diversity of the training set, via labeling new examples. The ultimate goal is an end-to-end decision theoretic system that, given a new learning problem, can dynamically make these optimal tradeoffs in pursuit of the most accurate classifier obtainable for a given labeling budget.

[Lin, Mausam, Weld. To Re(label), or Not To Re(label). In HCOMP 2014.]

Decision-Theoretic Control for Crowdsourced Workflows is an ongoing project that applies AI techniques, such as reinforcement learning and Markov decision processes to problems in social computing. Our methods have been successful on a variety of workflows, including (1) control of voting to answer a binary-choice question, (2) control of an iterative improvement workflow, and (3) control of switching between alternate workflows for a task.

Publications

Conference publications

R. Drapeau, L. B. Chilton, J. Bragg, D. S. Weld. 2016. MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy. In Fourth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2016). AAAI Press.

A. Liu, S. Soderland, J. Bragg, C. H.. Lin, X. Ling, D. S. Weld. 2016. Effective Crowd Annotation for Relation Extraction. In 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2016).

J. Bragg, Mausam, D. S. Weld. 2016. Optimal Testing for Crowd Workers. In 15th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2016).

S. Huang, J. Bragg, I. Cowhey, O. Etzioni, D. Weld. 2016. Toward Automatic Bootstrapping of Online Communities Using Decision-theoretic Optimization. In Computer Supported Cooperative Work (CSCW). ACM.

C. H.. Lin, Mausam, D. S. Weld. 2016. Re-active Learning: Active Learning with Relabeling. In AAAI Conference on Artificial Intelligence.

C. H. Lin, Mausam, D. S. Weld. 2014. To Re(label), or Not To Re(label). In AAAI Conference on Human Computation and Crowdsourcing.

J. Bragg, A. Kolobov, Mausam, D. S. Weld. 2014. Parallel Task Routing for Crowdsourcing. In AAAI Conference on Human Computation and Crowdsourcing.

J. Bragg, Mausam, D. S. Weld. 2013. Crowdsourcing Multi-Label Classification for Taxonomy Creation. In Conference on Human Computation & Crowdsourcing (HCOMP). AAAI Press. Awarded Best Paper .

C. H. Lin, Mausam, D. S. Weld. 2012. Crowdsourcing Control: Moving Beyond Multiple Choice. In Conference on Uncertainty in Artificial Intelligence.

C. H. Lin, Mausam, D. S. Weld. 2012. Dynamically Switching between Synergistic Workflows for Crowdsourcing. In AAAI Conference on Artificial Intelligence.

D. Weld, Mausam, P. Dai. 2011. Execution Control for Crowdsourcing. In UIST.

D. Weld, Mausam, P. Dai. 2011. Human Intelligence Needs Artificial Intelligence. In Human Computation Workshop.

P. Dai, Mausam, D. Weld. 2011. Artificial Intelligence for Artificial Artificial Intelligence. In AAAI Conference on Artificial Intelligence.

P. Dai, Mausam, D. Weld. 2010. Decision-Theoretic Control for Crowdsourced Workflows. In AAAI Conference on Artificial Intelligence.

Journal publications

P. Dai, C. Lin, Mausam, D. S. Weld. 2013. POMDP-based control of workflows for crowdsourcing. In Artificial Intelligence.

Workshop publications

C. H.. Lin, Mausam, D. S. Weld. 2015. Reactive Learning: Actively Trading Off Larger Noisier Training Sets Against Smaller Cleaner Ones. In ICML Workshop on Crowdsourcing and Machine Learning and ICML Active Learning Workshop.

J. Bragg, Mausam, D. S. Weld. 2015. Learning on the Job: Optimal Instruction for Crowdsourcing. In ICML ’15 Workshop on Crowdsourcing and Machine Learning.

C. H.. Lin, Mausam, D. S. Weld. 2013. Towards a Language for Non-Expert Specification of POMDPs for Crowdsourcing. In Conference on Human Computation & Crowdsourcing (HCOMP) Works in Progress Track. AAAI Press.

A. Kolobov, Mausam, D. S. Weld. 2013. Joint Crowdsourcing of Multiple Tasks. In Conference on Human Computation & Crowdsourcing (HCOMP) Works in Progress Track. AAAI Press.

D. S. Weld, E. Adar, L. Chilton, R. Hoffmann, E. Horvitz, M. Koch, J. Landay, C. Lin, Mausam. 2012. Personalized Online Education — A Crowdsourcing Challenge. In 4th Human Computation Workshop (HCOMP-12).

Book chapters

D. S. Weld, Mausam, C. Lin, J. Bragg. 2015. Artificial Intelligence and Collective Intelligence. In The Collective Intelligence Handbook (Thomas Malone & Michael Bernstein, eds.). MIT Press.

Theses

C. H. Lin. 2017. The Intelligent Management of Crowd-Powered Machine Learning.

People

Faculty

Dan Weld

Professor Computer Science & Engineering, University of Washington

Mausam

Affiliate Associate Professor Computer Science & Engineering, University of Washington

Ph.D. Students

Jonathan Bragg

Computer Science & Engineering, University of Washington

Quanze Chen

Computer Science & Engineering, University of Washington

Masters Students

Daniel Gorrie

Computer Science & Engineering, University of Washington

Alumni

Andrey Kolobov

Microsoft Research

Christopher Lin

Microsoft

Lydia Chilton

Columbia University

Peng Dai

Airbnb

Ryan Drapeau

Facebook

Shih-Wen Huang

Jizhi.co