Open Source Geo Stack

There’s a need for a good, high level description of the alternatives within in the “gently settling” stack of open source geoweb application development.

The OpenGeo Stack is the epitome of clarity, breaking down their tool set in a nice executive summary. But the OpenGeo stack only covers their tools, not all the available options. So I’m going to make a quick first pass of a high level overview. It’s useful for me, maybe for others. If you think I’ve done a poor job, help improve it in the comments, or on some wiki somewhere.

OpenGeo breaks things down into FrontEnd, Tiling, ApplicationFramework, Database. I’ll add Rendering, since in other tool sets this is split into different packages.

FrontEnd
the slippy map

* OpenLayers the Ajax gold standard
* ModestMaps for mind blowing Flash, ala Stamen
* Mapstraction don’t want to tax your mind? it looks just like the Google/Yahoo/Microsoft API

Tiling
be nice to your database or WMS and cache map images into tiles, just like Google and friends

* TileCache simple bit of python
* GeoWebCache same thing in Java
* mod_tile it’s kinda OpenStreetMap specific, but an apache module is a good idea too

Rendering
make pretty maps

* Mapnik looks beautiful. getting somewhat less painful to install.
* Mapserver does it all. also a pain to configure. looking better.
* GeoServer

ApplicationFramework
where the the main logic of the app goes. MVC. CRUD. etc.

* GeoDjango making great progress on a complete package.
* GeoRails more a bunch of plugins than a package, but definitely useable
* GeoServer the standard for open geo standards. Java.

Database

* Postgres + PostGIS
* MySQL sure, it has spatial extensions too. just not as fast or fully implemented as PostGIS

Random notes, other good sources

Architect your interfaces on Geo RESTful services. Andrew breaks down the formats and approaches for Neogeography and the GeoWeb in this presentation and book. For Ajax smooveness, use jQuery or prototype. Paul Ramsey has a good deep overview of open source GIS. Mecklenburg County GIS is a nice example of an instance of the stack.

There really is a need for a new book on this stuff, the O’Reilly trio of paper geo titles are great but out of date, and the landscape of osgeowebappdev is stabilising. Of course, no one wants to write it.

Geographic Democracy in the Physical Heart of the GeoWeb

Or in San Francisco it’s only slightly easier to get maps of representative districts than in Mumbai (one of our core motivators for OSM in India).

Start with a simple question. Who are my representatives from the local, to state, to national levels. Yeah, I don’t know, cause I’ve lived in England for four years, and I’m not around here much anyway. When I launched OpenStreetMap Brighton 1.0, the excellent WriteToThem identified my local councilor, MP, and MEP (even though I couldn’t vote for any of them) and I duly spammed them with invites to our party (the mayor did show up!).

For the US, one of the first hits was Project Vote Smart (awesome, are there others?). My zip code confusingly listed multiple representatives for the US House, and State Senate and House. (And nothing for local councilors). Turns out my zip is districtly schizophrenic.

state senate districts unclear

The State Senate turned up this unfriendly tabulation. This PDF confirms that 94114 is in both State Senate Districts 3 & 8 (at least as of 2002. you’re screwed if you stumble across a 1992 pdf without knowing better). The image exported from their GIS is totally unhelpful, I live somewhere below the “O” in Diamond Heights Blvd.

state senate districts, a little better

This PDF is almost there, looks like the district border is a couple blocks over. Maybe. What if you live on the border street. Different representative from your neighbor across the street? Why the dip to Castro Street to take us into the West side of SF? We’re should clearly be on the East, not with the Westside!

State assembly districts, unclear

The State Assembly is no better for an overview.

state assembly, clear

Close up, we have some clarity and a somewhat more logical choice of division boundary in my neighborhood at Twin Peaks Blvd. There are twice as many Assembly districts than Senate districts. Why not some confluence? What logic are these divisions based on? What tortured debate possibly made Corbett Street the dividing line for the Assembly districts.

House of Reps, unhelpful

US House of Reps is equally unclear. CalVoter doesn’t tell me if it’s 8 or 12, Pelosi or Speier, and doesn’t care at all about coastlines or any identifiable geography.

National Atlas Fails too

The “National Atlas” fails too.

Raccoon Drive bummers

GovTrack finally seems to answer the question. For me at least. The folks on Raccoon Drive don’t only have marsupial problems.

So even for me, pretty competent in searching the web, searching the web for geodata and maps, extremely familiar with the local terrain and streets, and experienced interpreting and scrutinizing maps, it took time and effort and puzzling and uncertainty. Spread over numerous websites, some official and some not, in different geographic styles, scales and precision. And I didn’t even try to find the data, to try and do a better job independently. Seems like one of the most basic things government and civil society should get right, and we’re not.

Mapufacture and FortiusOne!

Mapufacture has joined FortiusOne!

No need to repeat why this makes so much sense. Andrew and Sean share the great news. More officially, there’s the press release and Mapufacture blog.

Personally, my new role at FortiusOne gives me latitude for explorations and tangential efforts, which I’ll be writing about in the coming time.

From my perspective, we’ve all been pursuing this vision of maps, tools and data, in order to better understand and communicate about our world. At the conclusion of researching Weaver House, I wrote..

there’s one simple system which they all could easily hook into .. geography. I simply want to be able to search, and ask for all the information about a small half kilometer square area. That is why I’m pushing for GeoRSS, open geo data, and open source geo software .. not just for the Web 2.0 holy grail of a good restaurant review .. but to provide the simplest and easiest way to organize information about the world, and put everyone on equal footing when it comes to deciding our future.

It’s incredible to experience and witness all the different forms of organization and endeavor that pursuit of such a simple thing conjures. From hackers, open source, and NGOs, to start ups, big multinationals, and the United Nations, we’ve worked in so many capacities. As Andrew and I have worked through Mapufacture, pushed through and spun out the simple ideas into the new alphabets soups of the GeoWeb, all paths were open to us. And we’ve been fortunate to have the support of so many great folks. When FortiusOne made the offer to join forces, it was a completely natural idea. Complementary focus, similar technologies and approach, and a shared vision of what all this code can do. I’m confident great things are happening. Rockin!

Progress and Retreat

There are probably too many projects I have an interest in but too little time for. On these I generally cheer lead when needed and make connections to other people and efforts, and give little nudges occasionally so that they’re rolling in the right direction.

Chris Schmidt has built a working version of the OpenAerialMap concept. Excellent. We’re moving from talking to hacking, and this seems poised to grow rapidly. I’m going to add imagery from Merano when I have the chance.

I’m disappointed that UNDemocracy has gone on strike. This seemed to me to be a landmark project, for pushing forward transparency in the UN governing bodies and agencies, and I know it has already had an effect inside and outside the UN. Julian has found the reaction too slow, but I’m not surprised since the UN is slow moving. There’s now discussion on how to raise a phoenix from the ashes…

OpenStreetMap on the iPhone!

The iPhone is amazing. And the Maps application is pretty special. Only problem with it is that Google Maps is the only available data source. Well I’ve just been working on that. Now, there’s OpenStreetMap on the iPhone

OSM Central London on the iPhone
OSM Isle of Wight on the IPhone

These are screenshots of OpenStreetMap tiles, viewed in the native iPhone Maps application. Wooooooo

How you can do it

So it’s pretty easy to try it out. You’ll need to have jailbreak’d your phone (I used iBrickr, there’s plenty of other methods out there, even for 1.1.1). Download the maps tile cache db, and copy it over to your phone, using scp or whatever. The tile cache db is about 30 MB, so please only download if you’re going to try it. Also, I’ve had the Maps App crash occasionally when browsing the Maps, though nothing fatal .. so use at your own risk.


download http://brainoff.com/media/LondonZ17.sqlitedb
cp to /var/root/Library/Caches/MapTiles/MapTiles.sqlitedb on your phone
restart your phone

How its done

My first idea was to modify the configuration or constant that the Maps App used for constructing Google Maps tile requests. Andrew helped with decompiling and pointers to other files, but we had no luck as yet. But looking at the app or at the GMM module in a hex editor, there appeared to be SQL inserts of tile pointers .. perhaps Maps was using a database to cache requested tiles. Searching through the filesystem found MapTiles.sqlitedb. That db has just two tables..


sqlite> .schema
CREATE TABLE images(zoom int, x int, y int, flags int, length int, data blob);
CREATE TABLE version(version int);
CREATE INDEX index1 on images (zoom,x,y,flags);

images was simply storing the zoom, x, and y of each tile. flags was set to ‘2’ for street maps, ‘3’ for aerial imagery. length appeared to be the size of the data blob. And data was the png. So simply adding new rows to this table, with OpenStreetMap tiles rather than Google, would have the Map app using OSM.

I needed to grab a bunch of OSM tiles. Asked on #osm, and Almien helpfully contributed a script to download all tiles on a zoom level within a bbox.

And I needed to easily insert BLOBs of the tile pngs into the sqlite db. This sqlite tutorial had just the right code, eatblob.c. I wrote a script which walked through the directory of downloaded tiles, and insert each one into a copy of MapTiles.sqlitedb. All of central London at zoom 18 took a while to download and insert, and came out at 60 MB.

Actually, there’s nothing particularly OSM about this method. Could download Yahoo or Multimap or yea Google tiles, and stuff them into the cache db.

Issues

There are the crashes I mentioned. Haven’t debugged this yet, but I figure this is surmountable. Perhaps the file size of the tiles are too large.

The Maps App zoom levels are off by one. What it calls zoom level 13, are actually being tiled as if it was zoom level 14.
(zoom off by 1). Perhaps the tiles on the iPhone are half the width and height of normal tiles? I haven’t looked into this and its late, so I don’t even know if that’s a plausible explanation.

There’s still a little Google logo in the lower left corner, need to replace that too!

Last thing, don’t know how the cache is refreshed. It’s not going to grow without bound — when and why are things expired from the cache?

Other Methods

There are probably better ways to do this. One would be to modify the Map.app binary, or whatever module or library or configuration file that stores the url or ip address used when constructing requests for Google Maps tiles, and modifying that to request directly from OSM. Or write a provider independent version of the Map App — tiling seems to be part of a core library in the phone, it looks like Safari uses the same stuff to zoom in and out on web pages. If anyone starts looking into this, drop me a line.

flickr Geocodr

Tim Waters (aka chippy) introduces a flickr geocodr! It’s starts from a small hack of mine .. chippy has really tricked this out, and made it actually work!

It looks up photos with the given text, examines the assigned location (10 million+ geotagged flickr photos), clusters the results, and returns the largest location.

It works remarkably well .. for cities, neighborhoods (“haight ashbury”, “hells kitchen”, “calle ocho”), places and landmarks (“empire state building”, “red square”). It doesn’t always work precisely (“statue of liberty”), but suspect this could be down to some tweak in the algorithm.

And this seems like just scratching the surface of what’s possible with this data.

One extension that could be possible is detection of linear objects; for example, take a look at “Bay to Breakers” flickr map

mashup the whole space

The UK Geospatial Mash-up event was really about the mashup of an industry, rather than building chichestercrime.co.uk. It seemed like the whole spectrum of commercial mapping services, web 2.0 companies, government users, public interest groups, hackers, etc were represented, and with the focus on a single nation, the discussion (ie heckling) quickly honed in on the key issues of GeoData — how to acquire, distribute, license, utilize in a way that is fair, free, efficient, democratic, timely. I’d like to see more national and regional focused geospatial events.

Many people pointed out that there’s nothing particularly new about the mashup concept of combining components in another fashion (and with luck the word mashup will start to dry up now). The new fluidity of data is what’s important, leading to a Geospatial industry in a very unstable state. So, you get the OS announcing a non-commercial, slippy map API for their Explorer Series (excellent news for countryside applications), schoolmap.org.uk admitting that it screen scrapes to geocode postcodes, and infringes a commercial service, and a commercial company using OpenStreetMap. What’s next?

One thing, seemed like no one could really defend the state of the postcode and address database in the UK. Contrary to the OS, the Royal Mail can’t really defend it’s licensing as cost recoup, since the point of their postcode system is to deliver mail, not to supply data. So I expect Free the Postcode to ramp up, especially with the entire New Popular Edition enlisted to help.

My presentation slides are here. As usual, I learned of a couple new GeoRSS feeds on the day, from http://www.schoolmap.org.uk/ (which uses “featuretypetag”!) and Weather Feeds from the BBC.

FOSS4G to Ibiza, an intense 10 days. Links to presentation materials.

FOSS4G was a great time. OSGeo is about the best community out there. And this was my first trip to Switzerland, lovely. Just about how I envisioned it, ultramodern buildings in sheep grazed green pastures, above lakes and below mountains. The boat trip on Lake Geneva with tour and dinner in Château de Chillon was such a highlight. I found my way around Lausanne with only my GPS, yea OpenStreetMap.

I gave two presentations, back to back in the last hour of the last day. It’s About Time for Time surveyed experiments, widely and my own, in integrating time into GIS, and advocates GeoRSS and WMS Time to do the “Simplest Possible Thing That Works”. Google choose to release time navigation to coincide with my talk, thanks guys, so I hope the OSGeo community hooks into the coming upswing in temporal GIS. What’s Next, GeoRSS is one half hype ala Where 2.0, one half critical examination of issues with the GeoRSS format and processes. The presentation materials are posted to the FOSS4G info pages.

GeoRSS 1.0 was released. After a couple minor changes, this means that work on the format itself has frozen. We can now focus on supporting implementations and moving things forward through the OGC Lightweight Standards process and the W3C Incubator Activity. FOSS4G saw a large physical gathering of GeoRSS participants .. Allan Doyle, Raj Singh, Josh Lieberman, Andrew Turner, Patrice Cappelaere, Schuyler Erle, Jo Walsh, and myself. Many people mentioned GeoRSS as an inspirational process for the development of more lightweight, usable standards.

Andrew Turner, Guilhem Vellut, and myself all finally got to meet face to face. We’ve been working together on Mapufacture, Mapstraction, and GeoPress, three projects which are rapidly growing together. GeoPress 2.0 was released, with an announcement by Brady on the OReilly Radar. And I spoke with Allan Doyle about including GeoPress in his proposed Geospatial Appliance for Civil Society, How to fit 5 Kilos of Software into a 1.3 Kilo Box.

FOSS4G Castle

The WMS Tile Caching Scheme made big progress. And though I missed the session Tuesday night, it all looks cool and I intend to support the scheme in worldKit. I did make the Web Mapping Client BOF, though it all stayed AJAX rather than embracing Actionscript as ECMAScript brethren. That’s fine, since the intention of WMS-T and WMS-C is to have at least two different implementations, and with all the Ajax clients converging on OpenLayers, I’m happy to keep worldKit going.

In fact, I want to pursue a similar convergence/code sharing in Flash Mapping, from various bits of open source flash out there; attending this presentation which delved into a quality WFS Flash Client. Similarly, a group us of Ruby/Rails developers (basically us Mapufacture people and the guys from NomadLabs) met on Friday, and resolved to push forward Ruby support of Geo at various levels (recipes, adaptors, libraries, apps), something I think could really take off. And I spoke to Frank Warmerdam and Chris Holmes about starting the OSGeo Incubation process for worldKit .. I’m going to apply asap!

There were more interesting looking presentations that I missed than actually attended (frantically finishing my slides). Pat presented GeoBliki (OGC Sensor Web Enabled Data Node), a very cool community oriented (blog/wiki/GeoRSS) project for Sensors, which I’m going to watch closely. Sensors were also on discussion in Sensor Web Enablement: The 52north SWE Suite. Raj opened up the OGC’s Lightweight Standards Initiatives, discussing WFS Basic (outputformat=GeoRSS), WMS-T, WFS-T, and Individual OGC Membership. Steve Ottens spoke on Creating a high performance webmapping site, which further reinforced my belief that OpenStreetMap has about the most difficult problems around in GIS. Implementation of Synchronous, Spatially-Referenced Discussions Between Multiple Users with Open-Source Web GIS and Database Tools was interesting to get the thoughts flowing on how to better encourage collaboration in OpenStreetMap. Finally, the Conservation GeoPortal spawned an interesting debate on how to approach GIS in the developing world, where it matters most.

For more lucid thoughts and imagery on FOSS4G, Nick Black posted frequently, especially on how OSGeo and OpenStreetMap can work together. Jo wrote on her Have a Nice Metadata session, with plenty of choice phrases like “implement first, think later”, “the pursuit of the Simplest, Least Useless Thing”, “favourite OpenStreetmap tags – horse=’yes’”. Tom Kralidis has pretty good coverage of the entire event. Brady posted several FOSS4G entries on Radar. Photos here, here, and here.