| The Cooperative Computing Lab | |||||||||||
Research | |||||||||||
| Projects | Papers | ||||||||||
| People | Jobs | ||||||||||
| Events | REU | ||||||||||
| News | Blog | ||||||||||
Software | |||||||||||
| Chirp | Downloads | ||||||||||
| Parrot | Mailing List | ||||||||||
| Makeflow | Manuals | ||||||||||
| Work Q | HOWTO | ||||||||||
| More... | API / SVN | ||||||||||
Operations
| |||||||||||
Recent News in the CCL
Two Teaching Fellowships
Two graduate students in the CCL have received competitive teaching fellowships for the coming year:Tuesday, November 17, 2009 - Permalink
Genome Assembly at MTAGS 2009
Christopher Moretti and Michael Olson will present their most recent work on Scalable Genome Assembly at the MTAGS Workshop held at Supercomputing 2009.Their (unnamed) scalable assembler allows the end user to plug in a variety of custom algorithms for the computationally intensive phase of sequence alignment, using the Work Queue software to manage a workforce of hundreds of computers harnessed via Condor.
Our largest run so far used over 1000 nodes at three different institutions, reducing the time to perform alignments from over nine days to less than one hour.
Friday, October 30, 2009 - Permalink
CCTools 2.5.5 Released
We are pleased to announce release 2.5.5 of the Cooperative Computing Tools, including Parrot, Chirp, Work Queue, Makeflow, and other tools which may be downloaded here:http://www.cse.nd.edu/~ccl/software/download.shtml
This release includes a number of bug fixes, particularly relating to handling of symbolic links in Parrot and Chirp. Thanks to Yushu Yao, Francesco Prelz, Peter Bui, Hoang Bui, Michael Albrecht, and Christopher Moretti for their contributions to this release.
Wednesday, October 21, 2009 - Permalink
Energy Management at IEEE Grid
Recent graduate Michael Lammie presented his work on managing energy in multicore clusters at the IEEE Grid conference in Banff, Canada. His paper titled Scheduling Grid Workloads on Multicore Clusters to Minimize Energy and Maximize Performance, describes how to reduce the energy consumed by a large multicore cluster through the careful application of node scaling, voltage scaling, and job assignment. This work was done in collabration with Paul Brenner in the Notre Dame Center for Research Computing.Tuesday, October 20, 2009 - Permalink
Ph.D. Defense: Kyle Wheeler
Congratulations to Dr. Kyle Wheeler, who successfully defended his dissertation titled Exploiting Shared Memory Topology with QThreads for Portable Parallel Performance. Dr. Wheeler will shortly take a postdoctoral position at Sandia National Labs.Monday, September 28, 2009 - Permalink
Talk at Clemson University
Prof. Thain gave a guest lecture at Clemson University titled Scaling up Data Intensive Science to Campus Grids.Saturday, September 26, 2009 - Permalink
Talk at GeoClouds Workshop
Prof. Thain gave the opening talk, Science in the Clouds: History, Challenges, and Opportunities, at the GeoClouds Workshop in Indianapolis.Thursday, September 17, 2009 - Permalink
NSF Grant to Support Open Source Engineering
A team of researchers at the University of Notre Dame has received a $1.4M grant from the National Science Foundation titled Open Sourcing the Design of Civil Infrastructure. This project will create a virtual organization and online collaborative facility that will enable new ways of designing and evaluating civil infrastructure by applying concepts from the open source software community. The faculty team leading the project consists of civil engineers Dr. Tracy Kijewski-Correa and Dr. Ahsan Kareem, computer scientists Dr. Greg Madey and Dr. Douglas Thain, and social scientist David Hachen.Monday, September 14, 2009 - Permalink
NSF Grant to Build Collaborative Storage
We have received a Collaborative Research Infrastructure grant from the National Science Foundation to build a wide area testbed for data intensive computing. The Distributed Research Testbed will establish interconnected computing nodes at the Universities of Chicago, Florida, Hawaii, Notre Dame, and Mississippi. This testbed will provide an infrastructure for creating and evaluating new mechanisms for cloud and grid computing.Monday, September 14, 2009 - Permalink
Talk at HEC-FSIO
Prof. Thain gave a talk titled "Getting Beyond the Filesystem" at the NSF/DOE High End Computing File Systems and I/O Workshop in Washington, DC.Wednesday, August 12, 2009 - Permalink
Ph.D. Defense: Jeffrey Hemmes
Congratulations to Dr. Jeffrey Hemmes, who successfully defended his dissertation titled Improving Data Availability in Mobile Applications Through Enhanced Cooperative Localization. Dr. Hemmes will return to a faculty position at the Air Force Institute of Technology.Friday, July 31, 2009 - Permalink
MAJ Hemmes Returns Home
Major Jeffrey Hemmes, USAF, recently returned to the United States from duty in Iraq. He is currently a PhD candidate in the CCL, and will assume teaching duties at the Air Force Institute of Technology in the fall. Welcome home, Jeff!Monday, July 06, 2009 - Permalink
CCTools 2.5.3 Released
We are pleased to announce release 2.5.3 of the Cooperative Computing Tools, including Parrot, Chirp, and other tools which may be downloaded here:http://www.cse.nd.edu/~ccl/software/download.shtml
This release includes two new pieces of software: the Work Queue library and the Makeflow workflow engine.
Friday, July 03, 2009 - Permalink
CCTools 2.5.2 Released
We are pleased to announce release 2.5.2 of the Cooperative Computing Tools, including Parrot, Chirp, and other tools which may be downloaded here:http://www.cse.nd.edu/~ccl/software/download.shtml
This version includes a number of small fixes to bugs in Parrot that occur when running 32-bit executables on a 64-bit machine.
Friday, June 26, 2009 - Permalink
Talks at HPDC 2009
Li Yu presented our work on Harnessing Parallelism in Multicore Clusters with the All-Pairs and Wavefront Abstractions at HPDC 2009 in Munich. Prof. Thain gave the keynote talk at the associated LSAP workshop, Scaling up Data Intensive Applications to Campus GridsThursday, June 11, 2009 - Permalink
Grid Heating Wins Green IT Award
Paul Brenner's paper, Grid Heating Clusters: Transforming Cooling Constraints into Thermal Benefits won a "Green IT Award" from the Uptime Institute. Read more about grid heating here.Wednesday, June 10, 2009 - Permalink
CCTools 2.5.0 Released
We are pleased to announce release 2.5.0 of the Cooperative Computing Tools, including Parrot, Chirp, and other tools which may be downloaded here:http://www.cse.nd.edu/~ccl/software/download.shtml
This version includes a technical preview of our recent published work on abstractions for distributed computing, as well as a number of minor bug and portability fixes. Thanks to Peter Bui, Rashid Mehdi, Chris Moretti, Francesco Prelz, and Li Yu for their contributions.
Friday, June 05, 2009 - Permalink
BXGrid Article in JCC
Our article on the Biometrics Research Grid, Experience with BXGrid: A Data Repository and Computing Grid for Biometrics Research has been accepted to the Journal of Cluster Computing in a special issue on e-Science topics.Monday, June 01, 2009 - Permalink
Parrot Flies on the LHC Computing Grid
In a paper presented at CHEP 2009, a group of physicists describes how Parrot is used to distribute a large software package hosted at Fermilab in the United States to thousands of CPUs harnessed via the LHC Computing Grid across Europe.Friday, May 15, 2009 - Permalink
Honors Defense: Patrick Braga-Henebry
Patrick Braga-Henebry successfully defended his B.S. honors thesis title "Biocompute: Providing a Distributed Computing Model for Searching Genome Datasets." The Biocompute facility that Patrick constructed is used to carry out data intensive genome queries, parallelized across on a 64-core cluster. Congratulations, Patrick!Thursday, May 07, 2009 - Permalink
Presentations at Condor Week 2009
Chris Moretti and Hoang Bui gave presentations at Condor Week in Madison. Chris presented Abstractions for Data Intensive Computing on Condor and Hoang presented BXGrid: A Data Repository and Computing Grid for Biometrics Research.Wednesday, April 22, 2009 - Permalink
Multicore Abstractions at HPDC 2009
A paper by Li Yu and Christopher Moretti on our newest developments in distributed computing with abstractions has been accepted to HPDC 2009 in Munich. Harnessing Parallelism in Multicore Clusters with the All-Pairs and Wavefront Abstractions describes how to extend two abstractions to distributed systems that consist of multicore computers. With our collaborators Dr. Scott Emrich at Notre Dame and Dr. Kenneth Judd at Stanford, we demonstrate applications of the Wavefront abstraction to problems in bioinformatics and genomics.Wednesday, April 01, 2009 - Permalink
Article on All-Pairs in TPDS
Our most recent article on All-Pairs has been accepted to the IEEE Transactions on Parallel and Distributed Computing. This article presents new developments in data distribution, output management using really large matrices (60k by 60k elements), and a record breaking biometrics experiment.Wednesday, April 01, 2009 - Permalink
Chirp on the Blue Gene/P at Supercomputing
In a recent paper at IEEE/ACM Supercomputing, researchers at Argonne National Lab deployed our Chirp filesystem on hundreds of intermediate nodes to support applications running on tens of thousands of processors.Tuesday, February 17, 2009 - Permalink
BXGrid Featured in ISGTW
Our work on the Biometrics Research Grid (BXGrid), was the feature story in this week's issue of International Science Grid This Week.Thursday, February 12, 2009 - Permalink
BXGrid at IEEE e-Science 2008
At the IEEE e-Science conference held in Indianapolis in December 2008, Hoang Bui presented this poster on BXGrid, the Biometrics Research Grid. Prof. Thain gave a talk on Using Small Abstractions to Program Large Distributed Systems (and multicore computers) at the Workshop on Distributed Programming Abstractions.Monday, January 05, 2009 - Permalink
CCL in the Indiana Diagrid
Our 600-CPU Condor pool at Notre Dame forms a small part of the Indiana statewide DiaGrid, which exploits about twenty thousand CPUs all managed by the Condor distributed computing software. Here is more information about Condor at Notre Dame.Monday, January 05, 2009 - Permalink
CCTools Release 2.4.6
We are pleased to announce release 2.4.6 of the Cooperative Computing Tools, including Parrot, Chirp, and other tools which may be downloaded here:http://www.cse.nd.edu/~ccl/software/download.shtml
This version rolls up a number of minor bug fixes in Parrot and Chirp. Thanks to Ismail Ataturk, Karen Hollingsworth, Nathan Regola, Yushu Yao, for their contributions.
Friday, October 31, 2008 - Permalink
Abstractions at CCA08
Prof. Thain gave a talk titled "Programming Distributed Systems with High Level Abstractions" at the Cloud Computing and Applications Workshop held at the University of Chicago on October 23.Sunday, October 26, 2008 - Permalink
ENAVis at LISA 2008
Qi Liao will present a paper on ENAVis, a dynamic visualization of user, program, and network data collected by the Lockdown enterprise system management tool.Thursday, October 23, 2008 - Permalink
Abstractions for Data Mining at ICDM
Chris Moretti and Karsten Steinhauser recently had a paper Scaling Up Classifiers to Cloud Computers accepted at the International Conference on Data Mining in Pisa, Italy. This paper describes a high level abstractions for running stanrdard data mining algorithms on systems of hundreds of CPUs.Wednesday, October 22, 2008 - Permalink
CCTools 2.4.4 Released
We are pleased to announce release 2.4.4 of the Cooperative Computing Tools, including Parrot, Chirp, and other tools which may be downloaded here:http://www.cse.nd.edu/~ccl/software/download.shtml
A recent version of the patched Red Hat kernel installed by up2date inhibits access to the special file /proc/X/mem, which caused previous versions of Parrot to stop functioning with the error "Permission denied."
This release works around that bug in the kernel.
Monday, August 25, 2008 - Permalink
Troubleshooting at Grid 2008
David Cieslak will present a paper titled Troubleshooting Thousands of Jobs on Production Computing Grids Using Data Mining Techniques at Grid 2008 in Japan. This work demonstrates techniques for drawing conclusions such as "Your jobs fail on Linux 2.4 with less than 16GB RAM" from complex workloads of thousands of jobs.Wednesday, June 25, 2008 - Permalink
Datalab at HPDC 2008
Brandon Rich presented a poster on DataLab: Active Storage for Data Drive Scientific Computing at High Performance Distributed Computing in Boston. DataLab is a system for robustly performing large data parallel workloads on hundreds of active storage nodes, using distributed transaction concepts to create a robust system.Wednesday, June 25, 2008 - Permalink
CCTools 2.4.3 Released
We are pleased to announce release 2.4.3 of the Cooperative Computing Tools, including Parrot, Chirp, and other tools which may be downloaded here.Major items in this release:
- New documentation for the Chirp APIs
- Improvements to the Chirp server:
- New support for streaming I/O and server-side FIFOs
- New support for complex path names using RFC 2396 encoding.
- New P right that allows a user only to put new files.
- Improvements to the Chirp clients:
- Strided I/O routines for array access.
- Client interfaces for streaming I/O.
- Improvement to the build flags that make it easier for other programs to compile against Chirp.
- Miscellaneous bug fixes:
- Fixed case where server improperly returns "permission denied" instead of "file not found".es in the server.
Wednesday, June 11, 2008 - Permalink
NSF Summer REU Grant
The CCL has received a grant from the National Science Foundation which will support two undergraduates in summer 2008 to participate in the construction of a novel repository for biometric data.Researchers at Notre Dame have collected tens of thousands of such images and videos, and design new algorithms for identifying and matching people based upon these measurements. Answering these questions is very computation and data intensive. A large scale study of a new matching algorithm could take many CPU years to complete. To attack these problems in a reasonable amount of time, we must enlist hundreds of CPUs to work on different portions of the problem. While we have demonstrated the practicality of this idea with some custom programming, the overall system is not (yet) easy to use for end researchers. To solve this problem, the participants in this program will construct a well-organized repository of biometric data, connect it to our campus distributed computing system, and create an interface that makes it easy to specify and execute large biometric jobs.
Sunday, June 01, 2008 - Permalink
CCL to Participate in Google/IBM Cluster Pilot
In the 2008-2009 school year, junior and senior students in the CSE department will have the opportunity to learn techniques for large scale computing on clusters used by large internet service providers. Google and IBM have announced the 2008 Academic Cluster Initiative, which will provide a 1000-node machine for use by university students around the country. Students in Professor Douglas Thain's distributed systems and operating systems classes will learn how to write large data intensive programs in languages such as Map-Reduce on this cluster.Tuesday, May 20, 2008 - Permalink
Papers at IPDPS 2008
At IPDPS 2008 in Miami, Chris Moretti presented All-Pairs: An Abstraction for Data Intensive Cloud Computing, and Kyle Wheeler presented QThreads: An API for Programming with millions of Lightweight Threads.Tuesday, April 01, 2008 - Permalink
Parrot Flies at Fermilab
Parrot and the GROW filesystem are in production use at Fermi National Laboratory. The CDF experiment exploits the open Science Grid to run a large number of monte carlo simulations. Because the simulation code is highly complex and not practical to install at all sites, Parrot and the GROW filesystem are used to access the software on demand from Fermilab. More details here.Tuesday, January 01, 2008 - Permalink
Recent News
See More News