Work Queue: A Flexible Master/Worker Framework

Work Queue: A Scalable Master/Worker Framework

Work Queue is a framework for building large master-worker applications that span many computers including clusters, clouds, and grids. Applications are written using a simple C API that allows users to define tasks, submit them to the queue, and wait for completion. Tasks are executed by a standard worker process that can run on any available machine. Each worker calls home to the master process, arranges for data transfer, and executes the tasks. Together, the master and worker handle a wide variety of failures, allowing for dynamically scalable and robust applications.

Work Queue has been used to write many applications that scale up to hundreds or thousands of processors. These include the SAND genome assembler, the Makeflow workflow engine, and the All-Pairs and Wavefront abstractions. The framework is easy to use, and is currently use to teach parallel programming techniques in the Parallel Programming, Distributed Systems, and Programming Paradigms class at Notre Dame.

For More Information

Work Queue User's Manual

Work Queue Tutorial Slides

Work Queue API

Work Queue Example Program

Download Work Queue

Getting Help with Work Queue

Publications

Irena Lanc, Peter Bui, Douglas Thain, and Scott Emrich,
Adapting Bioinformatics Applications for Heterogeneous Systems: A Case Study,
Emerging Computational Methods for the Life Sciences Workshop at ACM HPDC, pages 7-13, June, 2011. DOI: 10.1145/1996023.1996025

Li Yu, Christopher Moretti, Andrew Thrasher, Scott Emrich, Kenneth Judd, and Douglas Thain,
Harnessing Parallelism in Multicore Clusters with the All-Pairs, Wavefront, and Makeflow Abstractions,
Journal of Cluster Computing, 13(3), pages 243-256, September, 2010. DOI: 10.1007/s10586-010-0134-7

Douglas Thain and Christopher Moretti,
Abstractions for Cloud Computing with Condor,
Syed Ahson and Mohammad Ilyas, Cloud Computing and Software Services: Theory and Techniques, pages 153-171, CRC Press, July, 2010. ISBN: 9781439803158

Peter Bui, Li Yu and Douglas Thain,
Weaver: Integrating Distributed Computing Abstractions into Scientific Workflows using Python,
Challenges of Large Applications in Distributed Environments at ACM HPDC 2010, June, 2010. DOI: 10.1145/1851476.1851570

Christopher Moretti, Michael Olson, Scott Emrich, and Douglas Thain,
Highly Scalable Genome Assembly on Campus Grids,
Many-Task Computing on Grids and Supercomputers (MTAGS), November, 2009. DOI: 10.1145/1646468.1646480

Christopher Moretti, Michael Olson, Scott Emrich, and Douglas Thain,
Scalable Modular Genome Assembly on Campus Grids,
University of Notre Dame, Computer Science and Engineering Department, Technical Report 2009-04, July, 2009.

Li Yu, Christopher Moretti, Scott Emrich, Kenneth Judd, and Douglas Thain,
Harnessing Parallelism in Multicore Clusters with the All-Pairs and Wavefront Abstractions,
IEEE High Performance Distributed Computing, pages 1-10, June, 2009. DOI: 10.1145/1551609.1551613