This marks the first beta release of Pydra, a distributed computing framework for python. While most core features are implemented, at least partially, there are many open tickets in the tracker for bugs and improved features. While this is not a stable release we felt the project was far enough along that there should be an installable package out there for people to experiment with. The coming months will bring more stability and features as we transition to a scheduled release cycle.
Many thanks to Jakub Gustak, Yin Qiu, and Almir Karic who have contributed code vital to this release.
- Task API with:
- Basic building blocks which can be combined to build complex, reusable Tasks. Base classes include a basic task and containers for sequential work, parallel work, and MapReduce.
- Datasource API for managing connections and input efficiently.
- Forms for defining, processing, and validating data.
- Task Packaging system for packaging tasks, supporting files, and libraries
- Synchronization System to automatically version and deploying Tasks Packages to the cluster.
- Scheduling System to track and run tasks.
- Task History recorded for debugging and future profiling components.
- Web Frontend for managing and running tasks.
- Controller API for controlling Pydra from within other programs via REST. This includes a client object to simplify dealing with Pydra’s strict authentication system.
- Security for all connections within the cluster.
- Module System for enabling or disabling features.
We welcome and encourage feedback, whether your love Pydra or Hate it. We want to shape this project to make it more useful for people, and for it to be easier to use. If something doesn’t work well we want to know. If you have a great idea that isn’t on our roadmap, we’ll determine out how to fit it in.