mikel – Brain Off

All I Want for OpenStreetMap is … Social and Attention

Kate posted her wishlist for OpenStreetMap and I second everything she wrote … OSM can be fine tuned to be even easier. In my wishlist post, I want to cover improving connections between mappers, and focusing mappers attention.

In short, OSM should be more like Facebook. Seriously. A news feed should show you the most relevant activity and news for you personally, and the profile page should show what you’re about in OSM in a more approachable way.

The sociality of OSM is its biggest strength. The intricacies of maps, tags, places are discussed in minute detail, and are ultimately the result of conversation, not top down dictate. The result are maps that are more expressive for more situations than any other platform. And we even meet in person! At the pub, the hack weekend, the LogCluster meeting. Anyone who’s ever been to a State of the Map conference can attest, it’s a vibrant, fun, exciting movement.

OSM has hundreds of thousands of users, and hundreds of millions of map features. Yet the tools to connect to mappers and monitor an area are still the same as 5 years ago. Changing this is my wishlist … and yes, I know that most all of OSM features are coded voluntarily, and its something I could do myself rather than list out … I appreciate all the time and energy that goes into OSM coding, this is my *wish* list.

Finding the Local Community

If I’m travelling some place new for an event, I want to connect to the local OSM community, and the process is complicated and not obvious. First, browse to the area in OSM and click the “History” tab, which should show recent edits in the area. Problem is, this often includes large, global edits which overlap this local area and aren’t relevant. And if you page through in a somewhat mind-numbing fashion, the OSM rails app starts to have performance problems past page 5. OWL is potentially a solution to this, but it has already been a priority development item for years.

And what the history tab won’t do in either form is tell you who the most active, prominent mappers in the community are. For that, I usually jump over to ITO’s excellent OSM Mapper, which present and visualizes summary statistics for an arbitrary area. Excellent, but you have to know it’s there, and it takes a separate authentication, and is a little counter intuitive to use. But from there, I have a list of 5-10 top mappers, with links to the OSM profile pages, from which I can send them a message. And wait, and hope they respond.

Next, I’ll jump into the wiki, and look through the relevant Wiki Projects. Sometimes this will have listing of Users in an area, or links to their website, or mailing lists. The User profile pages sometimes have a way to contact that user, but just as often not. If there’s a mailing list, then it’s about subscribing to yet another mailing list, perhaps one in a foreign language, and posting. If there’s a project website, that’s usually golden and a sign of an active local community that will be responsive.

Focusing Attention

And that’s just for a one time connection. Most mappers have been involved in mapping many areas. How do you keep informed? You basically need to have a ton of RSS feeds, poke around the monitoring tools, read a lot of lists.

A lot of activity, most of the time, you only need to be dimly aware of. If a user with a good reputation is editing in a familiar area, there’s little need to check into it further. If a new user has just signed up and is editing, it would be good to take a close look at their edits, and reach out and welcome them to OSM and offer to support their work. Reputation needs to be calculated, and used to filter.

And not just edits. There’s so much going on in the community. The Weekly OSM Summary is super useful, but is only the cream of the global activity. There are OSM events all the time, but it seems mostly German mappers are adding their events to the wiki. Expand User Diaries so these posts are accessible in all sorts of ways, for all kinds of notifications.

Being Social Should Be Simpler

Maybe the general concept is around “Interest”. There are places you might be interested in. There are particular mappers and communities you might be interested in … either because they’re active in an area, or perhaps because they make suspicious edits. Directly in the OSM website, you should be able to manually indicate this interest (by friending a user, set a list of locations); or it can be automatically calculated through analysis and stats of where you’ve been active before.

I started on this idea by adding friend and home location filtering to Diary Entries and Changesets. An ok start, but not everyone uses Diary entries to communicate, and the changesets still include everything … it’s both too much information and not enough. A lot is happening in OSM at any time, and you have limited attention.

There’s definitely a need for some system to do processing and analysis, from the planet diffs. Make these stats visually intriguing, and directly a part of a user or groups activity streams. There’s great experiments visualizing group stats with rankings on altogetherlost, and overall stats. And the personal statistics on neis-one, and heat map are excellent. Bring this directly into osm.org.

Social Design

Currently the user profile page on OSM has two functions … one to view other people, and another as a dashboard for yourself. I think these should start to diverge into two different pages. The User Profile should be grown to give a better idea of what a mapper is up to. The Dashboard would be a focus point for a mapper to stay on top of everything they care about in OSM. So in one page, give a summary of what friends and nearby users are up to, but also what’s happening in places you care about. For many users, myself included, there are many areas you’ve mapped, and would like to keep track of. Right now, you explicitly express interest by declaring your home location. This could be expanded to multiple, manually specified locations (my top 10 areas); or by analysis, a list of your top areas for editing could be automatically compiled.

It’s not only a design challenge, but it’s going to be an architected challenge as well. More social takes a lot more servers.

And don’t require mappers to visit OSM.org to get this. Yes, send emails out with weekly summaries of activity.

Some summary ideas for an activity stream:

* Recent mapping activity in multiple areas you’re interested in. Should be higher level than individual changesets (SteveCoast has made 12 edits over the last week in Seattle), but allow you to dig in further if you want.
* Some visual distinction between trusted vs non-trusted
* Also deal better with “big” edits, bots, etc. Filter these, or make them visually distinct.
* New mappers in these areas, and their activity. Highlight these, it’s a good opportunity to reach out.
* Highly trusted or experienced mappers in these areas, and their activity. Again, sometimes highlight, sometimes filter, depending on the need.
* Non-mapping activity in the area … upcoming mapping parties, other local news.

Thanks to the 2012 OpenStreetMap Foundation Board. This is going to be the year.

Last weekend in Seattle, the OSM Foundation Board met “face-to-face”. We get together because no matter how much you try otherwise, there’s way more done in person in a couple intensive days. It cost about 4 or 5k USD this time, and it’s worth the cost. But, I think we’ve always done a terrible job explaining what happens at the Board meetings, and a middling job following up, and those two things are totally related.

I want this meeting to be different. It must be different. This is my fifth year on the Board and final year on the Board (I was elected again this year, but will stand down at the next AGM), and to me, and the entire Board, this is a crucial year for OSM. The face-to-face was the most productive yet, and the most difficult yet. I’m very satisfied. In year’s past, the minutes get published, and various announcements go out through working groups, and that will happen. But it’s insufficient, maybe distilling too far the atmosphere and the messiness of these get togethers.

The Stage

Steve is based in Redmond, and expecting a child any day, so he offered to host and avoid travel. I wasn’t far, relatively, in Chicago. The rest of the Board (excepting RichardF who couldn’t make it) flew in from Europe. I found a cabin near a lake on airbnb, quiet, cosy, and cheap. Henk hired a car, and drove everyone around. We had a meetup Friday night, made some burritos and played Kinect at Steve and Hurricane’s place (and tried to forget we watched Crank 2), and enjoyed the Seattle sunshine (no joke). Sunday Hurricane gave us horse riding lessons!

A regular vacation! Except for the part where we spent 18 hours of our weekend discussing/arguing about OSM in windowless meeting rooms at Microsoft (which we very much appreciate btw!). And the rest of the weekend continuing to talk about it, or even dream about it. Being on the Board is a sacrifice of time, because we all feel deeply responsible to the project and our position.

Presentations

The Board meeting proper started with presentations by Steve and Oliver. Steve hit many of the same themes from his SOTM and SOTM-EU talks, except he left out all the stuff about how awesome OSM is doing. We looked and discussed several graphs of recent statistics. OSM’s growth to date has been beyond imagination, but there’s no shortage of projects that changed the world and then met reality, hard. Looking at some of these, the factors in decline included insular community, lack of direction, and no innovation. That’s what we have to avoid.

Oliver made the point that “We are the Board! Shape the project!”. The Board, and the Foundation, needs to be a functional team, with clear goals and activities, all within the limited volunteer time we have to contribute. Fact at this point is, the Foundation doesn’t have clear objectives, beyond the mission to support but not control the OSM project. To meet goals, we can take action, we can guide and steer, we can spend money. At the end of workshop, there should be a target that guides all our activities towards achievement. Some of the slides were beyond funny management clip art (a guy looking forlorn into the mirror, facing reality) but the point was important. “We are the Board! Shape the project!”

At this point, I thought it would be useful to look at some of the management lessons and differences from HOT. While we are by no means perfect, I do feel there’s good alignment between the organizational side of HOT and the community, largely the same community as OSM. Contrast to OSMF, HOT is very focused in what it does, with clear guidance and priorities and steering. We aren’t afraid of spending money when it’s necessary. We value marketing by the organization (though could be better). There are clear technical needs, and we pay for it. There’s a key attention to the consumption side of map data collection, seeking strategic partnerships with other organizations. We’ve been selective and directive with responsibility, and when necessary, have taken it away. We try to be as transparent as possible, publishing very detailed board minutes.

Goals

We took Oliver’s point and started strategic planning.

OSMF Board meeting traditionally use a simple technique to come to consensus on a topic, whether it’s the agenda of the meeting, or in the case of Seattle, the objectives and activities of the OSMF this year. We brainstorm all our choices on the subject, write them on the whiteboard. Each person gets some number of votes, say 5, and distributes them among the topics. If topics can be grouped together, their count is added together. There’s discussion about the meaning of terms, sometimes a lot of discussion. Iteration to insure that we all have a common understanding. At the end, there’s a list of priorities. I always squirm in this process, because somehow I don’t believe it can work, but inevitably does a pretty good job, and if we need to override, we’re not strict about the methodology.

In less than an hour, we had these goals for 2012.

The World’s Most Used Map OSM is clearly the world’s most used open map, and most open map, and the best map. We want as many people of possible contributing and using OSM, and to do that, the experience of using OSM needs to improve, and where you use OSM can improve.

More Than Just Streets Do you know everything OSM is capable of mapping? Does your neighbor? Does your mayor? OSM is relatively well known in some circles, but it’s full potentially is still opaque to many. We want everyone to know what OSM is about.

Cultivating Leadership of Mappers. Shared Goals Between the Community & OSMF Mapping is driven by mappers, with a clear goal (make the map!), and there’s every reason that with clear goals and empowered members, the OSMF can act strongly. We now have clear goals, and clear expectations of what the management team and working groups can do and achieve, without much prescription on how things happen. This all frees the Board to provide the direction, and the management team and working groups to make the operational decisions.

Easier Contribution for Non-Geeks We debated how this differed from the Most Used Map, and decided it was important enough focus to stand on its own. Usability is certainly related, but more broadly, there’s much to do to improve all kinds of involvement in OSM.

And Again

The bulk of our time was spent translating these goals into actions, and this really was the most difficult part. Some things were quick to decide, like the final switch over to ODbL, but others became very drawn out and very detailed, like the process for site redesign. We touched on every standing issue, and aligned clearly to the goals. PR, list moderation, license change, the management team, working group budgets, SOTM, PR, site redesign, the articles of association.

We all agreed that short term action was needed on almost everything, with mind to how things should play out in the longer term. This meant drawing the above diagram, a lot, to remind ourselves of the urgency. We set big, audacious goals for all parts of the Foundation, with clear deadlines.

With so much on the table, we decided to stay in the room until we had decided on everything, which ended up meaning staying hours late, til there was little sunshine outside (or metaphorically sunshine inside the room) and tension rising. At one point, I was so fed up, I almost walked out, really seeing that if we didn’t resolve the issue at the Board, it wouldn’t resolve in the Foundation and the project, the goals wouldn’t be met, and decline was inevitable. And for me personally, that would mean a slow turning away from a project ingrained in almost everything I do in the world. We had to push through.

And we did. Despite looking over the brink, we had resolve. I felt tense, but knew I’d be happy with what we accomplished.

And after it was all done, we had some beers. The next day we rode horses. Group hug.

Thanks to the 2012 Board. This is going to be the year.

And thanks to Oliver Kuhn for the photos!

Opening Data in Kenya. My Method is to Hack.

There’s good reason to join the excitement about Open Data in Kenya. As Tariq says on the World Bank blog

Open data in Kenya is special: it comes at a time of national change; itâ€™s got a head start on tools and expertise from the global open data community and itâ€™s happening in a country where the information ecosystem is still maturing.

I’m proud that our work with Map Kibera has any relation to this at all. And it’s certainly due to the hard work of passionate people, in a tough environment, especially Dr. Bitange Ndemo (if you have the time, Dr. Ndemo’s talk at the World Bank is recommended).

Now that the launch has subsided, and I have a spare moment in the air from Tanzania, I want to look in depth at what data and how data has been released on OpenDataKE, the means of working with the data and collaborating on the data, and how this resource can relate to other open data sets in Kenyan society. Now that the government has made a bold move, I think it’s the responsibility of the software development community and civil society to really step up and test out the data, and suggest how this can become a really vibrant and social resource. Again, Tariq says this succinctly

the call for open data should go hand in hand with a call for better quality data: data that might be collected by official government agencies or in this age, by citizens themselves.

Transect across data

My “method” is to hack. I want to make an interesting simple visualization with some data from OpenDataKE, focusing on Nairobi, using openly available tools. Browsing data sets, the Population Density per Constiuency, derived from the 2009 census, seemed promising. The difference in density across the urban landscape Nairobi is extreme. For a sense of it, just look at the density of features in OpenStreetMap in the map of the slum of Mathare compared to nearby leafy Mathaiga. And to help the hack, the population density data set even has a handy location column.

Or maybe not. The usual practice in tabular data is to split the latitude and longitude into two columns, but here both values are formatted along with the unnecessary name of the province in which the constituency is located. Anyone who has had to work with data is used to little problems like this, and it’s easy enough (for a programmer) to write a quick script to clean this up. So I selected Export to CSV (side note, the other options presented by the platform seem hardly useful), filtered just the constituencies in Nairobi, and cleaned it up just by hand (I was too lazy to script this for just a handful of values).

Gaps and Errors

I uploaded the CSV to GeoCommons, which has facility to deal with many formats of data and easily layer together interactive maps, and was surprised to see that several points weren’t placed in Nairobi at all. Turns out there’s several errors in the location column, at least in Nairobi, and possibly in the rest of the country (I didn’t check). I’d have to correct these by hand. My knowledge of the location and extent of the constituencies is limited, so I needed another source, and that is not something you can find on OpenDataKE. It took some searching until I found scanned maps of contituencies on the Mars Group site. An overview map of all the constituencies was missing, so I used the adjacent constituency names in order to place the mistaken ones.

This worked well, but I’m left with questions. Why isn’t constituency boundary data available on OpenDataKE? How did Mars Group get these maps? And now that I’ve gone to the bother of correcting this data set, how can I contribute the changes back, or at least alert the holders of the data to the errors? There is a nomination section on OpenDataKE, which was wonderfully active until July 9, and then went quiet (did Socrata’s support contract expire then?). Anyway, I’m hopeful these will start getting attention again, so I’ve submitted two requests (pending approval to post), one for constiuency boundaries, and another for a way to correct the location column in the population density data set.

My second surprise was that when I made the annotation size relative to the population density, I didn’t see a big difference among the constituencies. The area where Kibera is located, Langata, is about the same density as Westlands, and both are less than CBD and Eastlands. What’s happening here is that constituencies aren’t aligned to uniform urban settlement patterns. Langata, the home constiuency of the Prime Minister, includes both the slum of Kibera and the wealthy and sparse suburb of Karen. A more useful and telling metric would be population density per Ward, the sub-unit of constituency which does have fairly good alignment to settlement patterns. The census can and has been aggregated to this level, because there was a large promotion of the census count of population in Kibera.

So again I’ve nominated a data set, for the population density aggregated at ward level. And I’ve also made a request for meta-information on the methodology of the census in Kibera and other informal settlements. While the 170,000 figure is surely more close to reality than the wild 1 million figures of the past, by comparing that number with estimates derived from other methods there is a discripency; the others agree on an average closer to 250,000. Additionally, and admittedly anecdotedly, many people in Kibera say they and their neighbors were never counted. Now this happens in any census, and it does not deligitimize the census, but in order to interpret data, openness on the methodology of data collection and analysis is also necessary.

The Civil Society of Data

Open government data exists in a wider ecosystem. Just a few months ago, Columbia University released amazing data sets of Nairobi, including high detail land use under open knowledge licenses. A truly beautiful and informative data set. Another place to find many a Kenyan civil society data set is Virtual Kenya. I thought the population density dataset would be interesting to layer with land use.

This data is distributed as Shapefiles, and I need tiles to use a base map. This is the purpose of MapBox, a rapidly developing tool set to make it easy to build beautiful map tiles. I loaded the Shapefiles in my locally running TileMill, styled the landuse categories based on Columbia’s pdf using carto, assigned interaction, and exported as mbtiles. These were dropbox’d, and posted to TileStream, as this map.

Mouseover or click on the map to get more detail about each parcel. This interaction technique is really interesting (as a geek), it’s entirely javascript and lightweight in the browser; it still has a few rough edges, but overall, a nice experience. There are limits, like TileMill doesn’t work with CSV, or permit multiple interactive layers, but it’s a great work in progress. Thanks to DevSeed for the TileStream account, and Dane Springmeyer, who spent some time with me hacking and bug hunting the interaction features of mapnik.

Like the OpenDataKE data set, and actually all data sets, there are errors … there is no such thing as a perfect map. The Ethiopian Church, across from YaYa, is not indicated nor is its land zoned as “public use” as other church lands in Nairobi are. And the Sarakasi Dome, home of our yoga practice in Nairobi, is not shown a unified plot at all. Now Columbia makes their contact information known on the site, and I’ve met them personally, so feedback here is direct over email, but I wonder from here … what is the method and intention to continually correct, update and discuss these data sets? Does it need to?

Of course, that is the primary approach of OpenStreetMap … geographic data in a wiki, that gets constantly examined, updated, and discussed, completely openly. OpenStreetMap can provide another overlay, so we can have some roads and points of reference for the final map. So on GeoCommons, I configured the tiles from the land use data on TileMill (this required some hidden configuration of the tile scheme), composited over semi-transparent OSM data (provided by GeoCommons through Acetate), and then finally, the population density points. This is the result for now of the data transect.

I hope I can improve this. You’ll see that the OSM streets don’t overlay precisely with land use. This I believe, but haven’t confirmed, to be the result of a project error in the Land Use data set. And an even better representation of the population density would have been a geo-join with area boundaries, had they been available. This would clearly show a thematic variation of population density. And of course, finer grained detail will be required to fulfill the original intention to show Nairobi’s vast differences in population density.

Where have we gone

Government data sets, authoratative civil society data sets, and completely crowd sourced data sets, layerd together in a single map, revealing a little more about Nairobi, and about the data itself. Each is collected, distributed, and updated in different methods. In some ways, I feel OSM leads the wild edge here of what’s possible, and what we want: a truly social environment for data. Data without community is data dry and unimportant. Of course, I’m not saying OSM is the final repository for all data: OSM doesn’t deal with demographic and private data of a census, and the methods to authoritatively certify versions of OSM data are just starting. But this hasn’t stopped several kinds of OSM and government interaction already beyond the “traditional” import, with the likes of Portland and the USGS interacting with the OSM community.

The ultimate promise of all this OpenDataKE is not necessarily in the data itself, but in the deep and wide serving conversations openness triggers. My own personal metric for this will be when government officials from OpenDataKE and slum dwellers from Kibera and Mathare (and Mukuru) openly collaborate and work together. Can’t wait to see this happen. To get there, I challenge you too … get geeky with some data and write about it!

Jerusalem, Moving the Ladder

(x-posted from GroundTruth Initiative)

After 4 weeks, we’re leaving Jerusalem. The finest puzzle of human passion, and passion beyond, resting solidly and unsteadily on 5000 years or so of accumulated white stone and dirt. The most complicated and absurd and somehow, sometimes wonderful city. Our host Micha Kurz of Grassroots Jerusalem warned us that 4 weeks would be just enough to just begin understanding Jerusalem. In fact, it’s only enough time for the city to get a healthy grip on you so that you really don’t want to leave. And it’s definitely not enough time to come up for air for any writing and reflection … hopefully now I have a little space, on my trip back down to Dar es Salaam, the other end from here of the Great Rift Valley, where in complete contrast, the biggest conflict is that there might, maybe, be another political party in a couple years.

There is a ladder resting on the front balcony of the Chuch of the Holy Sepluchure. Centuries of delicate negotiation guide how priests and monks of various sects of Christianity move throughout the twisting bizarre space that might possibly be the site of the crucifixion of Jesus. No set of rules for behavior are comprehensive enough to cover every situation, and you hope you have good faith enough for dialogue when the loopholes come up. Not so here, where no one knows which sect originally placed the ladder, so they all refuse to move it. In the middle of all this, hoards of Russian pilgrims take pouty glamour shots in front of what might possibly be the site of a great suffering of Jesus. Thank you for the absurdity and the warning, that ladder should be the symbol of the city.

Remember this is also the country where there is meticulous debate on automated milking of cows on Shabbat.

Jerusalem is a place where they play excellent music in the streets on Friday, and no one notices a soldier dancing with a machine gun. In Silwan, down the slopes of the Old City, young men ride horses through streets defiantly kept 2-way as it has been for thousands of years, and gallop against slowly trudging tourist buses making their way to the City of David, where archeological diggings expose Biblical history, and expose too much of the present day, with houses and mosques collapsing above the excavations. Kites flew above all of this, capturing the view from cameras. In one day, you can rave off Jaffa Street, visit the birthplace of Jesus (grottos!), have a lovely fish dinner in the shadow of the security wall (thankfully just mapped, else we would’ve missed it), be told “have a nice day!” by a teenage soldier at an eerily deserted checkpoint, share a taxi with two Ethiopian priests in town from Dublin, and night cap it with a bottle of Brooklyn Lager. And check which beer your drink, cause drinking Taybeh vs Goldstar might be partisan (or wearing your hair a certain way, or the length of your skirt). Weekends are so confusing in Jerusalem! With 3 religions and 3 different holy days, it depends what side of town you’re on. Not recommended to go back and forth more than once between East and West Jerusalem in the course of one day, your mind will not be able to take such different worlds living in one city. You can hear Rock the Casbah performed in Arabic, and have multiple two hour discussions on the name of Jerusalem in OpenStreetMap.

Through it all, there has been such a strong reaffirmation of the mission of GroundTruth. With layers apon layers of history, of too much subtlety of meaning, of confusion, of deadly conflict, seems like the only possible response is coming to some reckoning and witness to it all, to see the change over time, to let everyone speak up about the reality of their lives. Let’s map Jerusalem. Let’s let people expose their regular humaness, carefully pace ourselves through the bullshit and maybe just find some small piece of reason. On Micha’s tour around Jerusalem, we saw an ancient city transformed yet again at the founding of Israel in 1948, and after 1967, the Green Line is now the smoothest highway through town. The municipal boundary of Jerusalem slices through old villages, envelopes completely new ones, and the security wall takes yet another course, and pressing close is the Areas ABC of the Oslo Accords. It hardly makes sense even if you understand it, and by showing what it’s like right on a piece of ground you may never visit otherwise, perhaps finally some understanding will happen.

We have at least 3 more posts to talk about the experience with the amazing Grassroots Jerusalem, who cut their chops mapping the Salah ad Din shopping district, and passed it on to al Walaja a small and inspiring village, which has a piece of just about everything complicated in Israel and Palestine including a completely encircling section of the wall, and it’s all being wrapped up in a potent brew of technology and training inspired by a slum in East Africa.

Doing

The past few months have most entirely been about discussing, strategizing, arguing, planning. Really missing actually doing stuff.

So I decided to ignore the emails, concepts notes, budgets, design documents. Build something. Just small things, but cool things. Makes me happy.

On the OpenStreetMap site, the history tab is super important for keeping tabs on changes, but visually dull and hard to immediately parse. I added a map, with highlighting between the list and map views. TomH cleaned up the code and improved the layout. Basically, just some javascript and view changes, and I had to get my OSM installation up to date. Needs some tweaking, but a good start.

I was pushed to do this by the Strategic Working Group. We’re discussing the OSM front page and usability, going in circles, and realzing the only way forward was to just do some things, however minor, to iterate our discussion.

Looking at the history tab, I realize there’s many ways to go in the realm of monitoring change. Make monitoring more interactive, customized to the places your care about, and the mappers you trust. There are a whole set of external monitoring tools and perhaps some of the tools, or the ideas, could be integrated.

That is Kibera in 1961. Choose “Nairobi 1961” from this map. Really, just a preview, it’s going to blow people’s minds, especially with Brian Ekdale’s concise history of Kibera.

Last year in Nairobi, I became aware of an amazing archive of historic aerial imagery, flown by the RAF, acquired over all the British Colonial possessions, several times over decades. The images were used for creating ordnance survey maps … but since then the original images have been sitting in various dusty basements, most recently Oxford. Paolo Paron has a plan to get it all openly online. But in the mean time, you have to go to Oxford, sort through boxes of square images, and request photographic copies. I had the chance to do this back in December, and it was wonderful. 1961 is 50 years ago, meaning these images are out of copyright!

I receieved from Oxford eight largish tiff images. Seeking first to rectify them, I tried various incarnations of map warper (including a local version) but with 30 MB images, it was choking. Next I tried to find a desktop option, and found good results with QGIS georectification plugin. I stitched together 6 of the images, roughly (if I spent more time, would’ve adjusted the different brightness levels and try to align better the two North-South runs that covered Nairobi, one at a slight angle to another). Then loaded a shapefile of Kenya OSM into QGIS, set control points, and warped. Took the resulting GeoTiff onto the generous resource of hypercube, warped the image to spherical mercator using gdalwarp, and after some hints from winkey and crschmidt on #telascience, “gdal_translate -expand rgb”, to take this single band image into RGB, which TileLayers seems to prefer.

There’s a lot more to say about this image, and that’ll be built up in the upcoming new mapkibera.org site. And of course, there’s those 1.5 million time capsule images sitting in an Oxford basement to open up.

Friends of my mine from Nairobi, working for the World Bank, undertook a substantial data collection of all water distribution points in Liberia. Pretty interesting compare and contrast with community mapping. I was pinged about this on twitter by Ned of Water for People, who helped develop the Android/Appspot application for this data collection, called FLOW. My immediate response was that this should be open source and open data, so other people can do cool stuff with it. Max took up the conversation and obliged (as much as he could, the data license is still under discussion). So I had to respond and do something cool with the data.

The flow web app is ok, but a little clunky. It uses the Google Maps API, and with nearly 1000 markers, it loads slowly and interaction is tricky. So I wanted to try a tile-based approach, to improve the speed and visualization, and this seemed like the best opportunity to get into TileMill, part of the MapBox toolset by Development Seed. After going through the tools, I am extremely impressed with what DevSeed is building, and hope I can help out by finding good uses, digging up bugs, and contributing here and there.

So I installed TileMill, updating mapnik in the process. Really nicely designed app, with laser focus on building nice tiles. One of the amazing features is map interactivity, which easily and efficiently turns input data sets into mouseover and infowindow bubbles. The trick it uses is grid renderer, Dane’s extension to mapnik which builds tiled json files, used by the wax library for OpenLayers (and ModestMaps, and GMaps) to define hit areas. Basically, much more efficient than creating markers.

Finally, Dane showed me gridsforkids, a simple sandbox for building image and json tile sets for standalone, offline map apps. I had to grab the “.mml” file created by TileMill, have carto convert that to mapnik xml, and then use the custom generate_tiles.py to build everything. Found a small bug in how TMS is handled there, and had to hack in a few things for my data set, but it basically works (a few problems with the hit areas, still investigating). One way this could go is streamlining the process, where one of the outputs from TileMill was a tar ball bundling all the tools to run the map offline.

Good Citizenship in Open Data

I didn’t expect my rant would strike such a nerve, but maybe I’ve been holding my tongue for a long time. Thank you for the many responses, online and off, both supportive and challenging.

We must work to understand what good citizenship and ethical behavior means in open data projects. The nature of communication, copying and competition in the space of open data is very complex. Yes, it’s not just about Google, but about raising the awareness of these issues among the people organizing open data projects, and especially the communities where we want to have an impact. The best idea I’ve heard this week (in a week of amazing ideas in Cambridge) was from Jeffrey Warren. We need a clear set of principles and ethics to guide the practice of open data initiatives in new communities. Open data collection should have: open and clear explanations of the purpose of data collection and the license of data; effort to find existing sources of data, rather than replicating and resurveying, and lobbying for the sharing of that data; effort to give the communities that collect data every opportunity to use that data in their own work, however they see fit; etc…

Participation in open data is both about the arcane topics of data licensing, and simply and most importantly being a good citizen. The impression of legal matters is usually all any non-lawyer grasps … it’s just too hard to understand the subtleties of legal text, and we rely on trusted intermediaries to explain what’s what. Google is trusted, and they’re misusing that trust with communities, NGOs, and governments. And Google has huge resources, they get a lot of attention. They’re representing their mapping offerings as free and open, when they’re ultimately controlled by their business concerns. Google has one of the slickest communication practices around, with subtle phrasing that gives an impression of cooperation, but with built in protection whenever someone digs into the details. This is deceptive.

There’s no problem with copying, it’s encouraged. Mapping Parties are not patented. There’s no license on ideas, because fair play is expected. Simply, being a good citizen, being a friend in this diverse space, means giving credit and giving back. Google has taken a concept, adapted it, and not attributed. Not until last year did they even use the phrase Mapping Party for Map Maker events (sometimes that got the attention it deserved). In places where there’s been exposure, Google didn’t “throw Mapping Parties”, because of course it would’ve been noticed as deriving from OSM. And Google is alergic to giving any notice to OSM, for example omitting any mention of the biggest open data project on the planet, which originated in the UK, from their glossy magazine on UK Open Data.

Competition is a very crude and inaccurate way of describing the relationships in the open data space, especially between OSM and Google, and is based on confusion of what OpenStreetMap is all about. OSM is essentially a commons, a data project where everyone can contribute, share and use. OSM is not a company, it’s not an NGO. It’s a collective where hackers, governments, universities, anarchists, slum dwellers, multinational corporations, and international organizations all contribute to a common endeavor to map the world. Every other major internet company contributes to OSM in some substantial way. It is Google only that has decided it’s in competition with OSM, not because of the designs of engineers and product people, but because doing so would undermine something Google thinks it can profit from.

Honestly, I would be thrilled to see Google adopt the use of OSM. The data of OSM is very different from the interfaces and the software. The openstreetmap.org site and tools definitely need improvement, but it’s not because osm.org wants to take away traffic from Google Maps. OSM’s frontpage needs to do something different: “The frontpage fails to communicate what the project does and why it does it effectively. The controls are confusing for new users (lots of random tabs and sidebar). In addition, it totally fails to connect new users to others around them and thus create a community.” Google could even be helping with this.

Never did I say that Google should stop making its products, but rather than it should stop exploitation. In more positive terms, Google must be a better citizen. I even said that a lot of what Google is doing in Africa is good for Africa. I use Google products all the time, live by Google Calendar, and think Summer of Code is awesome no matter how many spaces OpenStreetMap gets. They’ve contributed in great ways to humanitarian response. I have many friends at Google, who are not surprised by what I said, but perhaps surprised that I said it out loud on the Internet. But this is one place where Google’s Africa Strategy has fouled up directly in a space I care deeply about, fueled by business concerns, cloaked in open data clothes, but ultimately dangerous. As Ping said on Crisis Mappers (about a different OSM-Google issue, but equally valid here) …

It is okay to have issues with Google. Google is a big place; there are a lot of different people and different teams here, all trying to accomplish different things. It’s tough to assign a coherent intent to the entire company; we’re a bunch of people trying to work together even though we don’t always agree. No individual person is trying to be evil. Sometimes we screw up, though.

So yea, I’m pissed off, and can’t keep quiet. And actually, as some have asked, there is a lot more to these stories that you’ll only here about over a beer. If you only know me by this post, perhaps you’re surprised to hear that I’m usually known as a mediator, diplomatically open to all sides of an issue, generally a nice guy who just wants to map the world. And in case it wasn’t obviously clear, this all is my personal opinion. Another aspect of our open world is our multiple public representations, and I wasn’t speaking for the OSM Foundation, Humanitarian OSM Team, or GroundTruth. And yes, the OpenStreetMap community has a reputation for being “caustic”, which is based on a very small segment of our community, mostly the osm-talk list. Anyone who has ever attended State of the Map, gone to a pub meet, contributed to a HOT response, posted to nearly any other list, will have a different impression.

Finally, I regret to say that Unosat got caught in the cross-fire of this post. Unosat has an evolving cooperation with many entities, including HOT, where we have been given access to imagery in Japan and Liberia. While I find fault with their Google MapMaker cooperation, they aren’t giving Google any data, rather using Google data. They are well versed in the details of open data, of licensing, and are wise to explore opportunities equally. They’ve generously explored how they can work together with OSM, most recently in Japan and I look forward to more.

We Need to Stop Google’s Exploitation of Open Communities

Google’s strategy is to build market in Africa by appropriating the appearance of open data community methodologies, yet maintaining corporate control of what should rightfully be a common resource. They are specifically targetting govts and NGOs, offering to “map their country for free”, but keeping the results, and attracting customers.

What bothers me so much is how they have blatantly copied OpenStreetMap. First their MapMaker product is directly modelled on OSM, but with a restrictive data license, where you can not use the data as you see fit. Second, they have stolen the idea of Mapping Parties, a unique concept and name we developed. Third, they’re even copying initiatives to map impoverished informal settlements, like Map Kibera.

All of this with no credit, and no shame. I’m sick of Google. What can we do?

I’ve tried to stay civil with them, through years of this kinda stuff. I’m friends with many googly Googlers. But I don’t see cooperating or being quiet helping at all. They’re deceptive and need to be stopped.

Seriously, Google claimed to map “the largest slum in Africa”, with “citizen cartographers”. They’re building their business by glorifying half-baked “community” mapping initiatives, promoting their brand on the back of poverty.

I totally get why African governments and techies are excited about Google. Their moves in Africa are going to help build up the market for everyone. What is horrible is the hidden dangerous bargain they’re offering. To most people, Google is not just a company, but a force for good in the world. They even forget its a business, with so much done for “free”. But remember, it’s an extremely lucrative business, and they reason they don’t participate in OpenStreetMap, like every other major internet company (AOL, Microsoft, Yahoo…), is because it doesn’t make sense for their bottom line. They see value in owning your data. They’re moving to own the data of communities and governments in Rwanda, Kenya, Zambia. They can do whatever they like with the data, they own it. Of course, they will eventually take advantage of that. Scratch eventually, they changed the terms on Friday, so you have to pay to opt out of GMaps API advertising.

Corporations should not be the stewards of a public resource, and a potentially controversial public resource. Compare Gaza in OpenStreetMap and Gaza in Google for just one example of why this is a bad idea. We’re approaching a situation where a corporation is becoming the decision maker on international borders. Wait, did you think the UN or other international forum was supposed to have some role in these kind of things? Nope, Google is getting UN data too.

Internet, now’s the time to get pissed.

OpenStreetMap and Transport, presentation to the World Bank Transport Forum

Kate Chapman and I were invited to speak at the World Bank Transport Forum, on OpenStreetMap and Transport and ICT … which is great, cause I’m a transport geek (honestly disappointed I missed an earlier session on “Best Practices in Railways”), a long time bicycle advocate (what after yesterday I know as “non-motorized transport”), and there’s many many cool projects in OpenStreetMap.

I gave this presentation, with kudos to osm-talk for pitching in.

OpenStreetMap and Transport

View more presentations from mikel_maron.

Went over pretty well, despite being introduced as an anarchist. Actually, it went really well, and I have a stack of biz cards to follow up on at the World Bank.

My fellow panelists were Eddie from OpenGeo, talking particularly about their work with TriMet and Jon from Ushahidi, talking a bit about an interesting upcoming bicycle project in China. New to us was Todos Somos Dateros, a really interesting citizen engagement in Lima around transport data and feedback, with an overlapping approach to some parts of Map Kibera.

The other half of the day was also at the World Bank, at ICT Days. The session I caught featured guys from Data.gov, data.worldbank.org, Sunshine Foundation, government guys from Singapore and Korea. The key takeaway for me … its a small step for capable governments to open data, but to become open governments means true participation, and that’s not something government can ever do alone, except in the already most responsive and representative governments. For us, that means community technology development, to strengthen citizens and civil society.

Finally, of course the Bank had nice Wine and Cheese. Heard there about an upcoming Water Hackaton back in Nairobi, which fits nicely with ongoing work in Mathare, and older projects like WaterWiki.

cross-posted to GroundTruth Initiative Blog

Neo-geo-politics

Back in September, I was invited to Kosovo, to speak at the Software Freedom Conference. It was an honor for me to come to this troubled region where so much good work was happening in open source. And it was the ideal moment to gather long brewing thoughts on the political lens of open source and particularly OpenStreetMap, and on the incredible road trip I had just taken with Erica through the former Yugoslavia.

My talk was “neo-geo-politics, the impact and issues of open source and open data on political crisis, borders, disputes, and marginal places”

The talk runs with the basic premise that the complexity of international political organization in nation-states has inherent problems and faces increasing strain (as seen now in North Africa and the Middle East) and that new conceptualizations of how we relate to each other can be explored through open source community ideas (and its progeny in open knowledge, open data…). This is made particularly clear when talking about the new power structures in geographic data and technology, and provides a better intuitive guide as individuals are increasingly atomized and the narratives of consumption vs citizenship grow more difficult to properly separate.

Enough babbling … my slides and video …

Neo-geo-politics

View more presentations from mikel_maron.

Video of my talk on neo-geo-politics from Software Freedom Kosovo

Systematic Injustice

I’m not sure if it’s appropriate to raise our predicament publicly, but an injustice has been done. I must try all avenues available to right this situation. If after reading this, you have any ideas on how to proceed, please get in touch.

One of our partcipants, Zacharia Wambua, applied for and received a travel scholarship to attend the ICTD Conference next month at Royal Holloway, University of London. Zack is one of our brightest mappers, and shows great promise to take on this work as a career (Zack is standing to the right of US Ambassador to Kenya Ranneberger in this picture). The conference was going to be an opportunity for him to network, especially with UK universities, and take part in our panel focused on citizen mapping and media projects. He was to be joined by fellow mapper, Douglas Namale. For Zack, this would be his first time traveling outside Kenya, and I’m sure a life changing experience for a highly intelligent young man from Kibera.

As required, Zack submitted an application for a business visa (ref NAIROBI\323731), and took great care to fill out the application honestly and comprehensively. Yesterday, he received noticed that he was refused entry, with no opportunity to appeal. The decision gives no weight to our organization, the Map Kibera Trust, and draws attention to Zack’s economic situation as the primary evidence for the decision that, in the view of the UK Border Agency, Zack is not genuinely seeking to attend this conference.

To me, this is a deep systematic failure, based on probabilities. Zack is impoverished, it’s true. But he is working very hard to improve himself and situation, partly through participating in our programs. Despite having very little financial ties, or having family with a bank account, he is in fact very tied to his family, church and community. I’m not sure how you are supposed to show evidence of strong relationships to your family in a visa application. Zack is volunteering with a program, that among other things, has been featured on the BBC, is the subject of a DFID funded and UK university led research program, and he’s been invited to a prestigious conference organized by UNESCO and the University of London, that focuses on social and economic development through technology. Yet he is denied entry by the UK Border Agency. This is a terrible contradiction.

Frankly, if being poor is a barrier to taking part in these kind of activities, then the intentions of the entire conference, the work of DFID, etc, are completely negated.

We understand that an incorrect decision could have been made. However, Zack has been denied the right of appeal, the right to an interview, and the right for his supporters to speak on his behalf. Therefore, I ask, if there’s anything you can see we can do in this situation that could lead to a change of decision, we’d greatly appreciate the advice and action.