Archive for January, 2007 in the Guardian

When I saw Richard Pope at the OKFN Civic Information Forum speak about, I wanted to get involved straight away. Richard was prompted to build a more usable and proactive interface to local council planning databases after a local pub was demolished without warning. Well there was a warning .. but he wasn’t scanning the planning applications daily to keep track. So, mySociety style, he built a screen scraper to do it.

Today the Guardian profiles Planning Alerts, in part of the Free Our Data Campaign.

I had been thinking along similar lines after researching Weaver House, and being surprised that my friends had moved into the middle of the largest construction project ever in the UK. So far, I’ve completed about 90% of a screen scraper for Tower Hamlets, which uses a system called Planning Online for publishing planning applications. That system is widely used, so Camden, the New Forest, and dozens of other planning bodies aren’t far behind. I’d also like to see GeoRSS available along side email.

Seeing the huge efforts required to influence data access policies at the international and national, lately I’ve been more inclined to focus at the local level. The possibility to have a say and get in touch with responsible people is much greater. Successes here can aggregate together to influence at a higher level, a bottom up approach. Local bodies know much more about their local geography than any other agency or company, but they lack an easy system for maintaining that data set, so that for instance the Ordnance Survey receives reports of change from councils (via intermediaries), surveys and produces maps, and then sells that same data back to council in a digestible form.

Once an area of OpenStreetMap is complete, it would be ideal to have the council involved in updating the map. But better to hedge bets with a screen scraper. Road network changes are not published in the same open way as planning applications, but they can be gotten. Brighton publishes a dynamic map of roadworks and incidents, which does include information on new layouts and restrictions. I’ve scraped this into GeoRSS, by crudely geocoding image map locations into geographic locations. MetaCarta Labs Map Rectifier was a help here; I rectified the base image to get the rough bounding box of the image. A feature request would be an way to take any point in image space, and transform to lat/long by the specified warping.

Update: Turns out the Map Rectifier can transform image coordinates into geographic coordinates, it just hasn’t been exposed yet. So this url returns a bit of JSON. The GeoRSS being produced by the scraper is now properly positioned. Thanks guys! So all that’s left to do for my Brighton OSM monitoring page on Mapufacture is to supporting OpenStreetMap tiles!

Comments (2)

Datamining Folksonomy

Yahoo Research is going in interesting directions with datamining the relationship between geo and folksonomic tags. And Dan Catt hints at some future direction visualizing along the time dimension as well. 10 million plus photos with location, with time, associated with free form words. There’s definitely some meaning lurking in this folksonomy.

I explored this in a crufty way with mapping flickr tags, a WMS which randomly chooses tags for photos geotagged in area, slippy map browsable .. without the underlying map. It’s possible to actually identify where you are just based on the tags, which often name the place, and the shape of the tags. For instance, search for “New York, NY”, “United States”, and Manhattan is visible in rough outline. Other times the results are geographic poetic. “Barnet”, “UK” is southeast of “clouds” and northwest of “gas”.

When I first swapped over mapping flickr tags to use the new flickr geo api, from searching for “geotagged” tagged photos, the density of words dropped substantially. It seems that many people map their photos but don’t bother to tag them with anything .. perhaps geography is enough of an organizing principle. But it made my hack boring, so I added back the restriction to “geotagged” photos, which tend to have a lot of other tags too .. maybe since they were already using the tagging interface, it was easy to add a few more tags and now that the mapping interface is standalone, there isn’t a well worn path towards making more tags.


It seemed that many people added meaningful placenames to their geotagged photos, so that a geocoder could be built on top of the association .. something potentially much richer than purpose built geocoders. I built a really crude prototype that didn’t quite work. Tim Waters made it work, really well, by applying clustering to the results in flickr geocodr.


Can the same principle be applied to time? jbum’s datamining on flickr sunsets is still a mindblower and inspirational. Most all digital cameras stamp photos with the time they were taken (or at least the time the camera is set to). Perhaps the association between timestamps and tags could build a timecoder? Again I built a really crude prototype, flickr timecodr, which doesn’t work that well (4th of July is timecoded to June 30), but gets the idea across. . it simply averages all the timestamps across a tag; temporal clustering could help.

This first iteration of the timecodr does better on distinct events .. live8 for instance. I really look forward to seeing the time based animations Dan Catt hints about. With hugely growing geospatial databases everywhere, time will become essential to making sense of that volume.


The other facet I find interesting is geometries. Incredibly accurate line and polygon arrangements of photos were discovered right at the launch of flickr geotagging. Some clustering and line simplification could yield really useful results. A simple plotting of photos from west to east can yield crufty but interesting results. flickr linecodr. Races and roads work well .. like bay to breakers, boston marathon, route 66.

machine tags

Whoah! While I’ve been writing, flickr has announced support for machine tags. Blowing my mind, will need to let that sink in. Wonder if such a thing will be coming to crufty hacks on delicious.

Comments (1)

GeoRSS Extension for MediaWiki

For both United Nations Wikis I’ve developed, I seamlessly integrated mapping. The last iteration at UN INSTRAW allowed geotagging on any page through a Google Map GUI widget, and automatically generated GeoRSS feeds, which could be used within the Wiki itself to map all the articles.

At the Web4Dev Conference, and TechSoup’s online “meeting” on Web 2.0 tools there was interest in using the extension for other projects.

I’ve finally packaged up the code and posted it: MediaWiki GeoRSS Extension.

The GeoRSS plugin introduces an easy interface for geotagging MediaWiki articles, and publishes a GeoRSS feed of all mapped articles in the Wiki, for mapping and syndication.

It’s still in a pretty raw state, but it’s totally functional and a few improvements should make this an easy to install extension. Hopefully another wiki project will provide time to fix it up. Or perhaps someone out there is interested. Please get in touch with any comments or questions on the extension.


The OpenStreetMap Way

It’s very interesting observing and participating in the OpenStreetMap community .. it’s one of the most frustrating and inspiring groups I’ve ever come across. The past couple days have seen a long thread on the use of OGC standards and compliant tools in OSM, something that comes up with remarkable periodicity and some unnecessary hostility from both “sides”. OSM is not anti-standards at all, in fact uses PostGIS in a limited way right now, but it’s such a unique and new idea that most of the infrastructure has been built with agility, under quick iterations of home grown tools. Traditional GIS never conceived of a system where map creation was open to everyone. There are complete non-geographers (neo- if you wish) and some folks with a good deal of history in GIS .. and as a community we rediscover the conventional wisdom just as often as we overturn it. I’d personally love to have a WFS-T interface to OSM, opening up to whole new suites of tools .. but actually very few find it workable, even the Ordnance Survey has been looking at alternatives (something I hope to talk about soon). Back to thread .. towards the end it calmed down and several people articulated well the “OSM approach”. Bravo.

In the mean time, the commercial data providers continue to move towards OSM approaches. Navteq has launched Map Reporter for users to feedback map errors (it’s the number one customer service request with online maps and navigation systems). It’s definitely a more usable tool than the similar TeleAltas MapInsight. I was easily able to report one of the dozen or so errors within a half kilometer, a missing alley, and can even share my report with everyone. This alley is of course accurately, and the whole area in my opinion more beautifully, represented in OSM for the area (try the osmarender layer for the most detailed view). But for Navteq to be successful, it really needs much more transparency, otherwise these submissions enter a blackhole, without engagement .. what is Navteq’s schedule for looking at these submissions; when is the next surveyor coming around; how many other people have reported the same error; and what’s the benefit for contributing. For OpenStreetMap, I think we can learn that in the future we’ll need multiple levels of engagement .. some people won’t want to invest the time to edit the map itself, but are happy to report errors, and we’ll need an interface for that.

Btw, I’ll be talking about these things and more tomorrow night at the Sussex Geek Dinner.

Comments (3)

Wiki Volunteer Needed at the UNDP

The WaterWiki is looking for a Wiki-Geek at UN Volunteering. WaterWiki is the first Wiki deployed within the UN system, launched by Anna and myself in 2005. This is a great project to help out if you have a few spare hours per week and are excited about tools changing the way big organizations work. Sign up on UN Online Volunteering or you can also get in touch with me for information.

Interesting, when we launched there was a map and flags for every country in the region list on the front page, now moved here. To me, this is another reflection of the success of the project. When we started, people in this community of practice primarily thought of themselves within the framework of their home country, and the particular initiatives there. Now, there are numerous channels and configurations which the community works within, individual country being just one .. they see themselves as a community. Maps are still useful, but at this point they need something with higher fidelity to capture detailed and subtle information on the projects .. perhaps the mapping integration at the Gender Training Wiki is the next step.


PC Whore

My friend Eric had a pretty bad experience at PC World, the UK bigbox computer store. After a year in customer service hell, he’s only recourse is to start to document his broken and refused replacement computer, attract other dissatisfied customers, and perhaps leverage google to tell the world that PC World sucks.


Five Things You Didn’t Know About Me

Brady tagged me with the five things you didn’t know about me meme. Bring it on, who am I to stop a meme?

  1. In 1996, just graduated from university, I bicycled from Chicago to Atlanta in to attend the Olympics with my friend Eric Manners. I rode a beautiful green Schwinn Suburban. Such a great trip, we met so many people, had articles written in local papers on the way, and ate lots of biscuits and gravy. This ride radicalized me, I went on to ride thousands of miles more, took part in the largest Critical Mass ever in SF in 1997, and have never personally owned a car.
  2. One of my first ideas for My Yahoo was to add tracking of boycotts and strikes against companies in the finance portfolios. A small power fist would appear next to companies targeted with actions. At market-obsessed Yahoo this heresy was met with laughter; I still think it’s a good idea.
  3. I’ve never had a cavity.
  4. At my first eye exam, 6 years old, I gave lots of wrong answers because I wanted glasses so much. I got em.
  5. We published an “underground” newspaper at Aptos High School called the Merry Mariner. Pagemaker and Kinkos. We didn’t hold back. We were discussed at several school board meetings, profiled in the Santa Cruz Sentinel, and nearly removed from the graduation ceremonies. Started my addiction to the publishing buzz.

My tags: Steve Coast, Ed Parsons, Andrew Turner, Anselm Hook, Wilifried Hou Ye Bek.

Comments (2)