Many thanks to Jakub Gustak, Yin Qiu, and Almir Karic who have contributed code vital to this release.

Major Features

• Task API with:
  • Basic building blocks which can be combined to build complex, reusable Tasks.  Base classes include a basic task and containers for sequential work, parallel work, and MapReduce.
  • Datasource API for managing connections and input efficiently.
  • Forms for defining, processing, and validating data.
  • Task Packaging system for packaging tasks, supporting files, and libraries

• Synchronization System to automatically version and deploying Tasks Packages to the cluster.
• Scheduling System to track and run tasks.
• Task History recorded for debugging and future profiling components.
• Web Frontend for managing and running tasks.
• Controller API for controlling Pydra from within other programs via REST.  This includes a client object to simplify dealing with Pydra’s strict authentication system.
• Security for all connections within the cluster.
• Module System for enabling or disabling features.

Feedback

We welcome and encourage feedback, whether your love Pydra or Hate it.  We want to shape this project to make it more useful for people, and for it to be easier to use.  If something doesn’t work well we want to know.  If you have a great idea that isn’t on our roadmap, we’ll determine out how to fit it in.

Download

Pydra 0.5

There are no releases yet but we’re getting ready to use it in production:

• We’re close to deploying Pydra with the Protein Geometry Project. I’ve been working bugs out of Pydra while implementing their protein data import tool. The main issues are related to the protein parser rather than Pydra itself.
• I’ve also deployed a small cluster of a few blades and other random desktop machines we had lying around. We’re throwing it at the Engine Yard Contest just to try out Pydra. The trial attempts have been good, but this was more about kicking the tires on Pydra than winning the contest.

You’re using it but, why are there no releases yet?

I’m a big fan of “release early, release often” but the project still needs have a certain degree of usability.  There’s still missing features that will dramatically affect how easy Pydra is to deploy and use. Sure you can get by without them but it may require a bit of hacking and manual setup to deal with. The type of person who can make use of Pydra right now isn’t phased by checking out directly from a git repository.

Whats being worked on right now?

We’re updating the project on a regular basis to get it ready for an initial release. Here are the highlights:

Module System

I’m refactoring the core of pydra to use a module system. Modules will be loosely coupled using the observer pattern. It will allow us greater flexibility with our components and make it easier to implement features such as replication. It will also allow us to make functionality modular or optional.

The module system is intended to allow long term growth of the project. It wasn’t in the original plan because the core isn’t overly complicated right now. I’m implementing modules now, to prevent a jumbled mess of code a year or more down the road.

Scheduler

The task scheduler handles assigning work to workers in the cluster. The new task scheduler will fix quite a few bugs in our current scheduler and make it easier to extend.

Datasources

We’re designing an abstract datasource API that will abstract interactions with storage systems. Datasources will simplify distribution of connection information along with your tasks. Datasources will reduce the amount of setup you have to do on a Node before you can run a task on it. They will also make it easier to slice your data into pieces that can be distributed amongst Pydra Nodes.

Want to know more, or better yet hack on Pydra?

I’ll be at OSCON all week.  Follow me on twitter, here.

I was able to get growl-net.pl and mumbles to work using the growl protocol.  It had two major flaws, we couldn’t automatically detect the IP for our laptop and it wouldn’t work behind a NAT.  Greg Lund-Chaix and myself hacked on different schemes for tunneling IPs using ssh reverse tunneling, and nc to deal with the fact that growl is UDP only.  We had some minor successes but the connections using nc would disconnect frequently.

The tunneling just wouldnt work and was too fragile so I changed plans and worked on using XMPP instead.  Sending messages via XMPP would not require tunneling at all even if you are behind a NAT.  To accomplish this I rewrote growl-net.pl into jabber-notify.pl and hacked up a quick python program that listens for jabber messages and forwards them on to libnotify (via pynotify), the standard notification library for ubuntu.

Install Jabber-Notify.pl

1. Install Net:Jabber -  sudo cpan -i Net:Jabber
2. Download jabber-notify.pl
3. Deploy it to .irssi/scripts on the server you run irssi on
4. Load script /load jabber-notify.pl (You’ll get an error unless your jabber server is localhost)
   
5. Configure settings using /set (option) (value)
   • xmpp_notify_server - jabber server to send messages to
   • xmpp_notify_user - jabber account to send message from
   • xmpp_notify_pass - password for the sending user
   • xmpp_notify_recv - jabber account which will receive the messages.  This can be the same as the sender.
   • xmpp_show_privmsg - notify about private messages
   • xmpp_show_hilight - notify when your name is highlighted
   • xmpp_show_notify - notify when someone on your away list joins or leaves
6. reload script with new settings /load jabber-notify.pl
7. Test using /xn-test

Install Jumbles.py

2. Download jumbles.py
3. Edit jumbles.py to set your values
   • JABBER_HOST - Hostname/IP for the jabber server
   • JABBER_PORT - Port for the jabber server.  Default is 5222, 5223 for ssl/tls
   • JABBER_USER - User to receive messages
   • JABBER_PASS - Password for user
4. start jumbles.py

Some Notes

• Jumbles is only going to receive limited support by me for the near future.  While i’d like to add things like a tray icon, settings menu, and better reconnection logic, i just don’t have time right now.  If you’re interested in any of these things feel free to contribute.
• Jumbles was hacked together fairly quickly from xmppy example code.  Its name is a reference to this and mumbles which i was trying to emulate.
• The Sound used for notification is from open office, and can be channged on line 27
• If you dislike sound you can comment out line 75

What

Part of the non-partisan Sunlight Foundation, Sunlight Labs is an open source development team that builds technology to make government more transparent and accountable.  They have been building the Fifty States Project with the help of volunteers around the country. The project aims to provide parsers and an API for legislative data from each of the fifty state governments in the U.S.  This is a local version of their already successful OpenCongress.org which provides the same data from the federal government.

Where

The hackathon will take place at the Open Source Bridge conference. Open Source Bridge is a new conference for developers working with open source technologies and for people interested in learning the open source way.  It is an entirely volunteer organized conference.  Open Source Bridge is providing a Hacker Lounge, providing a great environment to hack in 24 hours a day during the conference.

The hacker lounge is located at the Hilton Portland & Executive Tower

      921 SW Sixth Avenue
      Portland, Oregon

When

The hacker lounge opens at 9am on june 17th and is open round the clock until 12am june 19th.

• Project Intro: Anytime between 3pm - 8pm June 17th
• Coding: any time till the end of the conference

Who

We’re looking for any programmers of any level, interested in promoting openness and transparency in government. No prior experience with the project is needed, but feel free to check out the project wiki.  Fifty States is written in python but other languages are welcome as well.

RSVP

If you’re interested in attending RSVP at Upcoming

Pydra

Pydra is a distributed computing or cluster computing framework for Python.  Pydra seeks to provide a framework that is easy to deploy, manage, use.  This is on top of providing standard features such as fault tolerance.  There are two students working on Pydra for Summer of Code 2009.

MapReduce

Jakub Łukasz Gustak is a 5th year student of graduate studies at Wrocław University of Technology (Poland), Computer Science major. Recently interested in concurrent, parallel and distributed programming and languages. Also interested in programming languages design and its expressiveness. His master thesis is addressing problems of parallel and distributed computation (title: Parallelization in Scheme language).

Jakub’s project is to implement MapReduce support for Pydra.  MapReduce is a concept for simplifying parallel computing jobs.  Jakub will also be implementing input/output helpers, improved WorkUnit Slicing, and other features.

Task Scheduler and Management

Yin Qiu is a first year master student from Nanjing University, China, majoring in software engineering. He interned at IBM China Development Lab for 6 months, and at Microsoft Research Asia for another 4 months. He has over 5 years of experience in Java and 2 years of experience in Python. In 2007.  Yin is passionate about building distributed systems.

Yin’s project is to implement a Task Scheduler for assigning workunits to workers and managing workers within a Pydra cluster.  He will also be implementing and improving management for task objects.

Unify

Luis Arajo has been a Gentoo developer since the beginning of 2005 , being part of the Haskell and Scheme team. He also cooperates with many packages inside the dev-lang/ category and is the current leader of the Gentoo-GUIs sub-project. Luis also
successfully participated on the Google Summer of Code of 2007 developing a graphical framework for the Gentoo project, and in Google Summer of Code of 2008 designing and implementing the Unify project.

Luis will be continuing his work on the Unify Project.   Unify is a tool for creating packages across different distribution formats (deb , rpm, solaris packages) using standard portage Gentoo ebuilds as the specification for building the packages.  Luis will be continuing work on features and improving the usability of the application.

Android IRC

Tianyi Gao is a first-year college student with a 3.94 GPA, triple majoring in Computer Science, Software Engineering, and Mathematics at Rose-Hulman Institute of Technology in Terre Haute, Indiana, and hopefully graduating in three years with the class of 2011. He’s been programming for six years, with three years of Java experience

Tianyi will be implementing an IRC client for the Android mobile operating system.  Our IRC client will differ from existing IRC clients for android in that it will utilize a proxy service to keep connections alive even when the client is not running.  This model will allow connections to be maintained but preseve battery life by only using data connections while the application is in the foreground.

I built a category plugin that uses the built in query & display functions to list all posts from a category.  It should be compatible with any other plugin that modifies the links for articles.

Building the Plugin

After doing some research I found the query_posts() function.  This function allows you to query any wordpress articles and then use the standard functions, such as have_posts(), the_post(), and the_title(), to output the articles.  50ish lines of code to generate a sql statement could be reduced to this:

query_posts('cat='.$atts['id'].'&posts_per_page='.$atts['count']);)

and since it uses the same functions for outputing the articles, it works with other plugins like FeedWordPress

Plugin Properties

• id - id for the category to display
• title - title to display at the beginning of the list, default to “category posts”
• count - how many articles to display, defaults to 5
• href - optionally turn title into a link to this url
• class - optionally set the class attribute, can be multiple classes, defaults to “cat_list”

download

Here is the plugin: categories.tar.gz

it is free to use and modify.

What

RepresentMe (Repme.org) focuses on how well your representatives and senators represent you.  It takes your opinions on bills and issues and compares them to how your elected officials voted.  It applies statistics, accounting for how strongly you feel about specific issues and bills.  The end result is a detailed breakdown of how well you are being represented.

Repme.org is meant to compliment OpenCongress.org, a sunlight foundation sponsored project.  OpenCongress.org is a great site for learning about legislation passing through congress.  It gives you a very detailed view of everything happening in congress.  What it doesn’t have is a detailed rating system for elected officials.

Why

Your House Represenative and Senators are the only three people in congress you get to vote for.  Getting information on what these three people are doing should be quick, easy, and detailed.  For this reason I’ve focused on things directly relevant to the user and their representation. My hope is to affect people opinions, positively and negatively.  To expose how well people are represented and on what issues.  Voters will then know what to applaud them for, and which areas need improvement.

How

RepresentMe uses a user driven weighting system to factor in personal preferences and the subtleties of politics.  Some issues are more important to one person, than another.  Sometimes people feel more strongly for or against a bill.  RepresentMe does it’s best to capture this providing a more accurate view of user preferences.

Votes

RepresentMe collects user votes on a scale from -100% to +100% and compares that to the lawmaker vote which can only be Aye (+100%) or Nay (-100%).  Presents, and No Votes are also treated as Nays, because these are just a political move that results in the same thing.  The two votes are compared to create a match percentage.

Lawmakers only get to vote in absolutes, but RepresentMe users get to vote the intensity of their support or opossal.  This is where the user element comes into play.  A RepresentMe user can vote things like “I wanted TARP to pass, but I couldn’t fully support it”.

Issues

Users can select the issues most important to them and rate their relative importance.  These ratings are used to group and weight the results.  This allows users to adjust issues to have a bigger or smaller impact on the overall rating.

Ratings

Personal ratings for each user and their lawmakers are calculated.  The ratings are a weighted average derived from their vote matches and issues.  Aggregate ratings for lawmakers are also provided.  All ratings include a details of where the numbers came from.

Sharing Data

My hope is that RepresentMe will produce a set of useful data. I don’t want the data to just remain on repme.org, I want it to be used wherever possible. Sharing the data can bring about conversation and ultimately change. To that end I’ve provided several ways to share data from RepresentMe.

Facebook

Facebook has 175 million active users around the world, Its one of the most popular social media websites in the U.S. The RepresentMe Facebook Application allows you to share data from RepresentMe by automatically publishing your votes to your Facebook Status, and allows you to add your lawmaker ratings to your profile page.

Learn more about the application here

Widgets

A widget is a portable chunk of code that can be installed and executed within any separate HTML-based web page. By providing widgets, owners of other websites can display a small portion of your website, in theirs.  RepresentMe provides an initial set of six widgets and a Generic Widget Builder to make it simpler to add new widgets

View the widgets here

XML

An XML feed is provided for every lawmaker. This can be used by developers who want tighter integration within their own applications.

View an example feed here

Future Plans

The road doesn’t end here.  There are future plans to improve the calculations RepresentMe can produce.  The ratings system is a fairly simple one as far as statistics go.  I feel fairly confident with user’s personal ratings but aggregate ratings need more studying.  A single user casting votes inherently weights itself correctly.  Different users won’t weight their votes the same way and that needs to be accounted for.  More data is needed to determine how to properly adjust the aggregate lawmaker ratings to account for different user voting patterns.

Pydra was born out of a necessity.  Other projects being developed by the Open Source Lab required a large amount of processing.  Rather than implementing parallelism specific to our application, we chose to build a generic distributed computing framework with the features missing in other solutions.  We see Pydra as a useful tool for future projects at the lab.

Management and Security First

A cluster is an inherently unstable and insecure application.  As more hardware is added, the chance for failure increases.  A distributed computing cluster is by definition provides remote code execution.  For these reasons management and security were top priorities from the very beginning.  There are several solutions for parallel computing within python, but none include these features.

Pydra will provide a management interface to control every aspect of the cluster.  It will allow you to configure the cluster, monitor its health, and in some cases recover from failures.  It will also allow you to manage the task queue, and track task progress.  Some of these features have already been implemented, the rest are high up on the roadmap.

Security within Pydra extends beyond a simple password.  Even when hashed,  a password offers little security.  Sniffing the hashed password is all that is required to bypass the authentication.  Instead Pydra uses encryption key pairs within a handshake to ensure connections within the cluster are extremely secure.  Even if an intruder is able to view traffic within the cluster, they will be unable to obtain login credentials.

Job Oriented Programming

Pydra uses a Job or Task oriented programming model.  users can write tasks that logically encompass any task they need to perform with a predefined set of base classes and containers.  The base classes include the ability to write both sequential and parallel processes as Tasks, and any combination therein.

A MapReduce base class will also be included by 1.0.  The task framework is a more generic form of MapReduce and this style of task can be written already.  The MapReduce base class will just simplify the process.

Technologies

Pydra makes use of other APIs where possible.  I’d like development to be as abstracted from underlying technologies as much as possible.  We’d like to use the best tool for the job, and that may change over time.  RPC and distributed computing is a hot topic right now and there are some relatively new projects such as the recent multiprocessing module added in python 2.6.

• Python 2.5 is used for now, until dependencies are upgraded to python 2.6/3.0.  We are watching the time-lines closely.

• Twisted is used for all of the Remote Procedure Calls (RPC) and networking.  Twisted was chosen because it is a mature, well written API that has good cross platform support.  It also was works well for fault tolerance and security.

• Django is used for the management interface as well as Object Relational Mapping(ORM) tool.

RoadMap

Pydra is still in its infancy, though much of the application has at least basic functionality.  Clusters can be configured and tasks run on them, but there is still a lot of work to do.  The task tracker contains a more detailed roadmap and version release plan.

I would like to have a usable beta by the end of the summer and a 1.0 release in late fall / early December.  Its ambitious but we’ll hopefully have a summer of code student working on the project, potentially a student intern, and myself.  We’d love to have anyone from the community help out.  Contact peter(at)osuosl.org if you are interested in the project.

Building widgets is very easy, but building a widget editor and preview tool for each widget can be time consuming. I wanted to build at least a half dozen widgets for RepresentMe so I came up with a Generic Widget Builder. This tool will allow you to easily add new widgets each with their own editor and preview, requiring very little code.

Overview

This example will use the Lawkmakers By Zipcode widget.  It accepts a zipcode and displays the lawmakers for that zipcode.

Step 1 - BuilderForm

The BuilderForm is a Django Form that contains some common fields that most widgets will want, such as color and background.  Fields you add to the form will automatically be added to the preview page.

from repme_widgets.builder.forms import WidgetFormBase
 
class LawmakersWidgetForm(WidgetFormBase):
    """
    Form for a single lawmaker with a single issue
    """
    zipcode     = forms.CharField()

Step 2 - Builder Tempate

The builder template is a django template used to render the javascript or html code that will display the widget.  By defining it as a template we can take input from a form and give the user both the code neccesary for their customized widget, and a preview

<script type="text/javascript">
repme_host_url = "{{host}}";
repme_zipcode = "{{zipcode}}";
repme_frame_height = "340";
repme_frame_width = "220";
repme_bgcolor = "{{background_color}}";
repme_textcolor = "{{text_color}}";
repme_bordercolor = "{{border_color}}";
</script>
<script type="text/javascript" src="http://{{host}}/widgets/js/lawmakers.js">
</script>

Some values are not included in the form:

Host - this comes from the settings file, this is required because the code must include an absolute url including hostname.

Height/Width - The height and width for a widget is often fixed so we haven’t included it in the form.

Step 3 - URL Handler For This Widget’s Builder

All widget’s built with the Generic Widget Builder use the same handler, passing the form class and widget template as parameters.  These values will be used to customize the preview page to your widget.

from repme_widgets.forms import LawmakersWidgetForm
 
(r'^builder/lawmakers/$', builder, {'form_class':LawmakersWidgetForm, 'widget_template':'widgets/builder/lawmakers.html'}),

Step 4 - Rendering The Widget

There are a few ways to build a widget.  They all result in an Iframe that loads a small portion of your website.  You can either call an Iframe directory, passing in your properties or use javascript to write the iframe to the document.  RepresentMe uses the latter because the code the users must cut-n-paste is simpler and easier to read or modify outside the builder.

Here is the javascript loader:

var fHeight = parseInt(window.repme_frame_height);
if (window.navigator.userAgent.indexOf("MSIE")) {
fHeight = fHeight + 7;
}
document.write('<iframe name="repme_lawmaker" width="'+window.repme_frame_width+'" height="' + fHeight + '" scrolling="auto" frameborder="0" style="border-style: solid; border-width: 1px; border-color:' + window.repme_bordercolor + ';" allowtransparency="true" hspace="0" vspace="0" marginheight="0" marginwidth="0" src="http://' + window.repme_host_url + '/widgets/lawmakers/?zip=' + window.repme_zipcode + '&color='+window.repme_textcolor+'&bg_color='+window.repme_bgcolor+'"></iframe>')

The view called by this widget is just another django view, except its output is a much smaller window.

Wrap Up

You should now be able to easily build an editor and preview tool for your widget.  Here is the widget I discussed in action.  The rest of the widgets for RepresentMe are here.

RepresentMe is released under GPLv3, feel free to reuse the Generic Widget Builder with your site.

This tutorial is part 2 of a series that will focus on connecting your existing web application to Facebook. In part 1 we connected your website to a Facebook app.  In this post I’ll show you how to update a users status and profile. I’ll be using examples from RepresentMe a project built for the Apps For America contest. This is a standalone webapp that leverages Facebook to deliver its message to a wider audience.

Things you can update

Once you have an account linked to a Facebook uid you can send several different forms of updates:

• Status - This is short message telling what the Facebook User is doing.  It is shown at the top of their profile and wall, and in their friends wall.  It is the primary way that Facebook Users share information.
• Profile - A User can add a small box from each application to their profile.  This is displayed to anyone who visits their profile.
• News Feed - This is similar to the status but allows you to include more formatting and a few basic html tags.  Facebook has phased out the infinite sessions that this requires.  At some point they may enable this again.

Publishing Status Updates

Publishing status updates from your app is a great way to let users show off what they do to their friends.  RepresentMe publishes user’s votes to their status.  We include a link back to the bill in question.  Any of their friends will be able to see these messages and get involved too.

Step 1 - Authorize Updates

Before you can publish status updates the user must grant your application permission to publish them.  This is done by including a FBML tag in your canvas template.

<fb:prompt-permission perms=”status_update”>
Grant permission for RepresentMe to update your status
<fb:prompt-permission perms=”status_update”>

This tag will generate a link in the page that will take the user through the process to approve or deny the request.

Step 2 - Initialize Facebook Object

Once your app is granted permission you can start updating the status.   RepresentMe uses a script running in a cron job, more about that later. Since this is running outside Django the first thing the script needs to do is initialize the facebook object

#initialize facebook
fb = facebook.Facebook(FACEBOOK_API_KEY, FACEBOOK_SECRET_KEY)

Step 3 - Update Statuses!

Now we’ll iterate over all of our link objects.  We’ll check for permission, then update their status

for link in FacebookLink.objects.all():
    uid = link.facebook_id
    str = 'I updated your status!'
    if fb.users.hasAppPermission('status_update', uid):
        fb.users.setStatus(str, False, True, uid=uid)

Step 4 - Don’t Spam

Facebook has guidelines for how often an application can update a User’s status.  They limit applications to 10 updates per day.  This is why our updates run as a script rather triggered directly by user actions.  RepresentMe only updates Users’ statuses every 30 minutes with the last vote they placed. Rather than running the cron job every 30 minutes you can include a timestamp in your FacebookLink model to track updates.  This allows updates to propagate soon after a user action, and spreads out the requests to Facebook.

# get all links that have not been updated in the past 30 minutes
delta = datetime.timedelta(0,0,0,0,30)
past_timeout = datetime.datetime.now() + delta
q = Q(feed_update_time__gt=past_timeout) | Q(feed_update_time=None)
links = FacebookLink.objects.filter(q)

Updating Profiles

Users can add a box from any application to their profile.  This box can be displayed in either the wall, or the boxes tab.  The differences are that the wall is more prominent, and the wall is much smaller (184×250px).

Step 1 - Set the profile content

The profile content is cached FBML or HTML and cannot contain Iframes.  You should create a view or function that generates the content.  RepresentMe is reusing the view handler for the Lawmakers By Zipcode widget with a template resized to fit within the wall profile.  We’ll call the view using Httplib2 and set the profile content to the result.

def update_profile(fb, link):
    # get zipcode from profile
    profile = link.user.get_profile()
    zip = profile.zipcode
 
    # get content
    conn = httplib2.Http()
    url = 'http://repme.org/facebook/profile/?zip=%s' % zip
    response, html = conn.request(url, 'get')
 
    # set content for wall (profile_main) and boxes (profile)
    uid = link.facebook_id
    fb.profile.setFBML(uid=uid, profile_main=</code><code>html, profile=html)

This example updates both profile boxes at the same time.  Its optional to update both but this function will clear the cache on any box that isn’t included.  The user can select the box appear in either the wall or boxes tab, so its best to update the cache for both at the same time.

Step 2 - Add Section Button

Users must manually add your application to their profile.  This authorization is granted using a FBML tag.  Note this button will not be displayed untill you have set content in their profile
<fb:add-section-button section=”profile” />

Step 3 - Adding When The User Installs You Application

The add section button only appears after you have set content for the user to add.  Facebook recommends that you do not set the profile FBML until you have enough data to display.  In our case RepresentMe has likely already collected that data.  We’ll set the profile FBML as soon as the user links their account.  This way the user will immediately see the add section button.

@facebook.require_add()
def authorize(request):
...
  if user.is_active:
      # successful login, save facebook info!
      link = FacebookLink()
      link.user = user
      link.facebook_id = request.facebook.uid
      link.facebook_session = request.facebook.session_key
      link.save()
 
      # update their profile fbml so its ready,
      # incase they add the app to the profile right away
      fb = facebook.Facebook(FACEBOOK_API_KEY, FACEBOOK_SECRET_KEY)
      update_profile(link, fb)

Step 4 - Script To Keep Profile Up To Date

Your application should keep the profile FBML cache up to date.  RepresentMe uses a simple script in a cron job.

# get all links - this should only run every 30 mins
fb = facebook.Facebook(FACEBOOK_API_KEY, FACEBOOK_SECRET_KEY)
for link in FacebookLink.objects.all():
update_profile(fb, link)