The pipes, the pipes are calling

For a couple of months now I’ve been playing with and trying to understand Yahoo Pipes. I’ve still got a lot to learn, but at least now I think I have enough understanding to write about it.

In its most basic form, Yahoo Pipes is a feed aggregator. You can take any RSS feed and view the output. If there happens to be geographic data in the feed, it will automatically parse that information and display it on a map. However, Pipes takes it a step further by providing tools for combining feeds from various sources, analyzing feeds for keywords or location data, or allowing users to input or control parameters for various feeds. For example, you could set up a Pipe to return RSS items from Craig’s List that shows red couches for sale within 30 miles of Greenville. In a stunning example of cooperation, Yahoo even plays nice with their rival, Google, and provides a link to a Google Earth KML output for any geocoded data.

Some of the most interesting items are the mash-ups that can be created. For example, one Pipe takes the feed from Reuters, parses it through the geonames.org RSS-to-geoRSS webservice, then displays the news items on a map. Another will takes the feed from the NY Times, analyzes it for key words, takes that output and uses it to find corresponding photographs in Flickr.

Users can browse Pipes that have been created by others, and clone those Pipes for their own use. For example, there are now many permutations of the popular NYT to Flickr Pipe. You can now see a Pipe that washes NYT through YouTube to return videos. I’ve even created one that sends RandomConnections through Flickr. It’s kind of fun to try to figure out why certain photos wind up with certain posts. Some are obvious – others, not so much.

This cloning capability is the easiest way to learn about Pipes. Most of the ones I’ve created are modifications of others that I’ve found in the gallery, such as attempt to geotag the feed from Greenville News. For some, I could pretty well figure out what was going on. For example, I can take the feed from any source and run it through a translator into any of dozens of languages. (Greenville News in French, anyone?) The blog A Day’s Drive from Greenville provides lat/long data in each of the posts. I was able to run this through the Pipes location extractor and display that feed on a map.

Until now, this has been very gee-this-is-cool-but-how-the-hell-am-I-ever-gonna-use-it. For me to truly learn something, I need a project, and that project landed squarely in my lap yesterday. For awhile now I’ve been communicating with Patrick Hayes, a producer at SCETV, about the RiverVenture projects that his group has created on their Knowitall.org website. The work that Patrick and company does is what really gets my blood flowing – media rich resources combined with excellent lesson plans. For their next RiverVenture, photographer Richard Bernabe will be paddling all the way from North Carolina to the coast on the Broad, Congaree, and Santee River systems.

Patrick has put together a blog for updates from the river, and will be using Flickr and YouTube for photos and video. Right now I’m working on a way to combine all of these sources into one RSS feed through Pipes and display that data on a map on Richard’s blog. The first part is easy. As long as a lat/long coordinate is placed anywhere within the blog post, we can parse it out. It’s also easy to combine feeds from various sources into one. Displaying it on a map is trickier, but not impossible.

Since the data is availble in KML format, I think it would be possible to create a javascript link to Google Maps which pulls in the KML feed from Pipes. The optimum result would be all the data displayed as it should be. However, we can provide external links if we can’t get the embedding to work. This will go live September 6, so it will be fun to see what we can work out by then.

[tags]Yahoo Pipes, SCETV, RiverVenture, RSS, geoRSS, Google Earth, Google Maps[/tags]