Brain Off

“What would it take to map an entire country?”

With the growing visibility of Map Kibera, that question is coming more frequently, especially in Africa, where both OpenStreetMap and traditional mapping are widely absent. This is a massive question, which is going to depend very much on circumstances of that country, and on who is asking that question; and in the end may be better answered by a different question. In response to a couple queries, from Liberia and Malawi, I decided to write up a few blog posts to start off those conversations, and serve as reference for any of the other 200+ countries on this planet. To start, going look at a few examples to serve as models for answering the question.

Up front, the question assumes one very important thing; the historical growth pattern of OSM isn’t happening. Traditionally, a few individuals had their minds blown by a conference presentation on OSM, or maybe a random blog post somewhere, and they start mapping their home town. And when that looks to be a big task, and they start eyeing the next town, they start recruiting others through mapping parties. A mailing list is set up. The virus starts to spread, and OSM might get the attention of a local government or two, maybe some companies, and soon, the country is well on its way to being mapped. The growth is organic. It might take years. The preconditions are important. Roughly, there has been an active community already of technically proficient people who have leisure time, perhaps already contributing to open source projects. In other countries, there may not be a technical community, or socioeconomic conditions make leisure time a valuable and scarce. Other places may be in conflict. These are places where people start to consider intentional interventions to get mapping going.

The first place I took part in an intentional effort was India. Now, India is well known for having a broad and highly skilled technical base, but at least in 2008, there was proportionally small open source community. There had been interest in mapping through the Free Map project, and Schuyler and I took the invitation to promote OSM through a slightly insane schedule of mapping parties in 7 Indian cities in one month. A wonderful experience. But it did not instantly translate into a frenzy of mapping activity. The idea gestated for a time, and slowly, individuals took up the cause, and now the OSM India community is vibrant, centered in cities. One place to note is Mumbai, despite extensive interest, is still not active, largely due to an import of AND data that turned out to be much less than accurate, discouraging further editing. An issue we’re still looking at, and generally, I’ll talk about existing data sources and imports in another post.

Palestine was a very intentional mapping, first of the entire West Bank, and then of Gaza. JumpStart International funded a project to create a complete, public domain map of all roads. We estimated a time of about 6 months, which turned out to be accurate (more on time estimates in another post). The approach taken was to hire engineering graduates, in teams based in each region, train them through mapping parties, and coordinate the incoming data. Gaza was not to be mapped, but the 2009 crisis there motivated the remote OSM community to fund raise to purchase recent satellite imagery of the entire strip, and trace remotely. After the conflict was over, JumpStart entered Gaza and built up a team to enhance that work with a combination of GPS tracing and imagery. The result is a complete road map of Palestine. However, no sustained OSM community of interested individuals, local companies, local government, and UN agencies is left. The individuals involved saw this as temporary paid work, and many of them have taken jobs outside Palestine. The wider community wasn’t engaged with an aim to build capacity. So now, even though Israeli OSM has been interested to hold joint activities with Palesinte, and Ramallah now has street names, the map isn’t being updated.

In Kenya, we haven’t set out to map the entire country, but it seems like the base we’ve built in Kibera is ready to spread throughout Nairobi and beyond. We’ve focused heavily on Kibera, but with the idea always that the group here, and the entire community, will be able to take the project forward. The impact of mapping a place that was unmapped, and considered unmappable, has made a great impression within civil society, the government, and the UN. We’ve taken a lot of time to outreach to others already working with map data. The approach is more like planting a single seed, nurturting it, and then allowing it to grow. It’s yet to be seen how this pans out.

JumpStart started next working in Georgia. Their approach is now long term but different. They have first focused on conflict areas, following the Georgia-Russia conflict in Abkhazia and South Ossetia, and also in the capital, Tbilisi. From this start, they are building a country wide network of regional offices, Open Maps Caucasus, supported by an NGO structure. They are producing curriculum, hiring a team, and doing extensive outreach with map data users. The question to my mind is the long term sustainability of the structure. If JumpStart decides to stop supporting the NGO structure, will it be able to find more funding to keep going? And if that doesn’t materialize, will a community be in place to take it forward organically? Again, yet to be seen, and a question they are certainly considering.

In the USA, mapping hadn’t taken off until last year. This was perhaps due to the TIGER data set, available public domain. CloudMade set out to initiate this community by hiring community ambassadors in regions throughout the country, and holding mapping parties. One of the most spectacular was the Atlanta mapping weekend, involving hundreds of people in cooperation with the Atlanta municipal government. However, after a time CloudMade discontinued the ambassadors program. From there, what has developed has looked like a more traditional growth, with a wide spectrum of individuals, companies, organizations, and government getting interested and involved. The GeoDC group has been very active, and recently OSMF US has incorporated. Hard to calculate what effect the intentional effort of CloudMade had on this growth, though it was certainly a contribution.

Finally, many have seen the extremely rapid growth of the map in Haiti. Prior to the quake, very little data existed there for familiar reasons. Following the disaster, and the release of imagery for derived works and other data sources, remote mappers quickly and spectacularly produced comprehensive maps of Port au Prince and much of the country. Mappers wanted to contribute to the relief effort. Certainly in other unmapped places, remote mappers are motivated to contribute just out of interest. If you look at any capital city covered by Yahoo imagery, there will have been remote mappers contributing at least geometries. Does this translate to places not in crisis, but in more prolonged issues that prevent mapping? Probably to some extent, but not nearly as focused and quick, and very much dependent on imagery. But imagery is not enough. Local knowledge is needed to name places, and identify features … most roofs don’t tell you what’s going on inside a building. In Haiti, two deployments have gone to advocate, train, and build capacity for OSM locally. Nicolas is now planning to go again, as part of our long term plan for OSM in Haiti, to continue the work to the point where the Haitian community, which includes their government and civil society actors, are ready to take it forward.

This weekend, Nicolas Chavent and Robert Soden will deploy to Haiti for the Humanitarian OpenStreetMap Team. MapAction and OCHA are facilitating this mission, with generous support from the European Commission Humanitarian Aid Department — many thanks to MapAction and ECHO for helping to make this happen. Nicolas and Robert are a dream team for this work, adventurous and passionate map makers. They are joined by experienced MapAction volunteer Chris Philips. Broadly, the mission is to support use of OpenStreetMap on the ground in Haiti within groups of UN and international responders, the Haitian government, and Haitian civil society. In other words, make sure OpenStreetMap stays relevant and useful into the recovery, reconstruction, and, most crucially, long term development of Haiti.

They won’t be going in alone, but with thousands of mappers globally ready to back them up.

This pioneering trip has emerged from several threads. Nicolas and I identified the need in our meeting in January, and published in the HOT Haiti Strategy and Proposal. The remote response of OSM has been very effective in the initial response, with high visibility and urgent need for data sets, but as time goes on, geodata management processes are put in play on the ground, and capacity building to use OSM within those processes is very much needed. OSM has to meet these communities half way to be effective, and the only way now to really understand deeply is to go to Haiti. The second thread has been a long running discussion between MapAction and OSM. I really admire the work of MapAction; they deploy rapidly and create maps by any means necessary in the crucial first days after a disaster, all through volunteer professionals. They first used OSM in their work last year in the Philippines, and we have long wanted to learn from each in how open source and open data can be deployed in emergencies. Third thread was the incredible World Bank deployment last month; Schuyler Erle and Tom Buckley had tremendous insights and made key connections with government and the UN that we hope to build on. Wonderfully, Tom has redeployed to Haiti. Final thread is the ongoing collaboration with Ushahidi and Kartier par Kartier, and their work to build local and diaspora capacity to take the Haiti Ushahidi install on in the long term. This definitely involves base mapping, and there’s an interest in linking in our approach, and eventually Map Kibera like projects in Haiti.

In detail, Nicolas and Robert will be doing hands on training and lots of conversing to assess needs and find solutions that integrate with OSM. Their work will address all levels. At the UN and international NGO level, we are blessed with many good contacts through Nicolas, MapAction, OCHA and myself, and are very eager to continue cooperation. CNIGS (Haitian national mapping agency) and CIAT (main government development coordinating body) have been working with the World Bank and others to rebuild themselves, and their capacity with new technologies; there’s interest here in really getting a solid national road network data set, with addressing. And within civil society, KpK has established an amazing grass roots network. Sabina Carlson from Ushahidi and Shadrock Roberts from KpK will also be in Haiti at the same time as the HOT mission.

The preparations have been intense. We’ve spent the last month budgeting, planning travel, getting equipment sorted, discussing the plan. Haiti is still in chaos, and perhaps even more so within the international responders as their hard work goes on and on. Not certain at this moment where there’s space to throw up Robert’s and Nicolas’ tents. Nearly certainly they are landing in Port au Prince on Sunday afternoon (with risks of delays). It will be Agile at its highest. Sourcing all the appropriate equipment has been intensive … printers, computers, and GPS units are on their way down, need to perform in difficult conditions, and find a good long term home. Software and data work is ongoing, including iterations of Map on a Stick. Clear training materials, including approaches relevant to short-on-time GIS and data collectors new to OSM, are being compiled.

Nicolas has spent a great deal of time on the Humanitarian Data Model. This work reconciles data models and schemas in various response sectors, maps to OSM map features, and works to engage new tools for Extract-Transform-Load processes, and JOSM presets. In other words, make OSM directly relevant to the systems already in use in Haiti. The OpenStreetMap community is invited to review and join the discussion on this process.

I’m very excited for this mission, and wish the best of luck and safe travels to Nicolas and Robert. Hopefully their busy schedule will allow for some updates and documentation. We’ll be listening closely to support whatever they need from afar.

The Humanitarian OpenStreetMap Team now has a mailing list.

During crisis, the HOT list supports coordination among the OSM community, along side appropriate local country lists, and the talk/dev lists when needed. It is the point where disaster responders and affected people can connect directly to members of the OSM community ready to help. In quieter times, the list helps HOT to prepare resources and improve its response.

Note that talk/dev are still good for mobilizing the wide community to a HOT activation, but details of the response can for the most part safely move into this list once we get critical mass of subscribers.

Please join if you are interested in helping HOT, or need HOT services. Thanks!

Preamble: I wrote the following after 1.5 years ago after getting kicked out of Jordan for trying to map it. We retreated to Palestine, where I was fairly certain it would work, but ended up going amazingly. Enough time has passed that all the entities in this story no longer care, people are even flying balloons in the center of Amman, so here it is for posterity.

“There’s nothing here which will do any good for the people of Jordan.”

That’s when we knew things probably weren’t going forward.

I should have expected things would go wildly wrong at the Royal Jordanian Geographic Center, when tentatively approach by desperated-eyed, young French geographers on loan from the Institut Géographique National, the French national mapping agency. “What are you doing here? We don’t often see visitors from the Occident.” Or when I received the call 15 minutes before departing San Francisco, “The RJGC has a few issues with our project”.

In the executive meeting room, the deputy director was joined by three colonels in full uniform, all looking grim and one carrying print outs of OpenStreetMap.org. The RJGC, like many national mapping agencies in history, is both a civilian and military agency. It was made abundantly clear that there had been a major breach of protocol, that we should have consulted with the RJGC sooner about our project. And it was made clear again. And again, this time louder. And then in Arabic. Maybe a few times more.

Jordan is a monarchy. There is tremendous respect for the the King and the Royal Family, who are intimately involved in all aspects of life in Jordan. No doubt, their wise rule has led Jordan to become an island of peace and stability within a region of turmoil and violence. The Switzerland of the Middle East. Jordan has openly accepted refugees from Palestine and Iraq. The people on the whole are very welcoming and friendly.

However, authority does have a general tendency to overstep its utility. We were told in absolute terms, the RJGC is the only entity in Jordan with the authority to make maps. Any other government, civilian or military agency requiring maps must acquire them through the RJGC. So you can see how that might not be compatible with a bunch of hippies carrying GPS units. What we were doing was declared illegal.

“One of the goals of your project is to map military installations!” Colonel Omar had printed out the OSM map key. OSM does map military installations. It maps everything and anything. But to think that we represent any kind of security threat, and have greater access than anyone with access to the web is paranoia in the highest degree. Paleotards live! We were nearly accused of being Israeli spies.

“Do you think you can just walk down the Champes Elyssee and make a map?”
“Would the Israeli’s let you map?”

Further the idea that we might have something to offer, in education, ideas, international connections, and technology, was ridiculed. The RJGC runs a college which trains all geographers and surveryors in Jordan, an exclusive expertise. Though many of those young well trained geographers were being hired away by the Gulf states, a threatening brain drain. “It’s impossible for you to map al-Mafrak in two days”. It’s a small town of <100k, and we would have 12 participants. We'd collect data to the usual level of OSM detail .. not the outline of bay windows .. but to the level useful for communication and unexpected uses forgotten by those in control of geodata. It felt like 2005. "You expect to map all of Jordan to 1:500 in months. The RJGC has a proud 50 year history." Nevermind that 95% of Jordan is desert. To demonstrate their advanced techniques, we were taken on the tour after the meeting. The "tour" consisted of a 25 minute video demonstrating the RJGC's use of ESRI products, with a length section of flight simulator set to triumphant music and painfully bad cheap video wipes. And that's the extent of the tour. I have no doubt that the RJGC is a sophisticated organization, but such a demonstration did nothing to convince me that we have nothing to offer them. When it become clear that we weren't mapping al-Mafrak, I tried changing tact. I offered training in our techniques to RJGC staff and students, as well as instruction in open source GIS and web based mapping. The whole Where 2.0. The response .. "I was trained in France! What's your experience in geography?" When I replied I was a web developer, they scoffed. Truly, what can you do at such a ridiculous display. Colonels in full regalia trying desperately to assert authority and upshow some dude from California who has only come to help. I also suggested that if we weren't mapping, then our goal of public domain data was still valid, and that perhaps the RJGC could release some data. "Haven't you heard of globalmapping? We already donate to that. Anyway, if anyone needs a map in Jordan, they can ask us." This is in a capital city that does not even have a bus route map. "If you really want free maps, we'll give you some". On leaving, we were each presented with a thick packet of paper tourist maps of Jordan. We showed these later to some locals. Maps full of mistakes and problems and painful cartography. Well there was only one thing do after this. Went for a beer at the only local pub, "The Hangover", bursting full of young women from former Soviet republics. And now it's mapped.

This is a question I’m considering a lot … filtered through the brief rushes of reading the amazing crisismappers list, diving into OSM on the wiki and IRC channel … Are we doing everything we possibly can in serving the responders? Can we coordinate our mapping work better? And once aid starts flowing and the immediate response turns to long term recovery and reconstruction, how will our process and community change, when more and more data to synchronize will be coming from the ground in Haiti? How do we operate better in the next disaster? Big questions, and glad that many folks are already compiling ideas. Please add ideas and needs there.

One immediate point that has been raised in private discussion, and now publicly, are the separate efforts of OSM and Google MapMaker in Haiti, separated by incompatible licenses. As Sean Wohltman points out, a crisis like this is hardly the time for discussions on licenses and community models, and whatever can be done for the best benefit to the crisis response should be done. All I can simply say with restraint is that I disagree with the assumption of Google’s position that the OSM license prevents community use of this data … rather it only prevents Google participating in that community, by their own choice. That shouldn’t stop OSM from considering doing everything possible.

There is likely much to be gained by everyone working in the commons. Muki Hakley has performed an analysis of OSM and MapMaker coverage in Haiti, and it shows complementary coverage. The analysis makes no assumption of correctness, time frame, size of community/number of sources, and only considers geometries not names and tags, but still shows that each have built up in different regions. The problem of how to potentially use these two together needs quite a lot of work.

Sean Gorman suggests a time and geography limited CC-0 license on geodata, in order to move things forward. I’m not taking a personal view on this possibility, actually undecided, though I would at least suggest attribution would be a courtesy, and impose no considered constraint. Instead, I would suggest the OSM community consider this matter, and if there’s a general consensus it would be valuable, perhaps by determined vote, then build a quick process of getting time and geography limited dual-licensing approval from the couple hundred active Haiti contributors so far. There would also need to be a way to get such approval from new Haiti editors.

Important I think to consider is the time and effort involved in discussing this, and the process of dual licensing, vs the potential benefit. This question shouldn’t distract us from the very real effort of mapping and producing amazing data products.

What do you think?

Just yesterday, I imported the Map Kibera data into OpenStreetMap. I thought I’d take the opportunity to review how the data collection went in this entirely unique process, allude to a few of the mind-changing map features of Kibera that I’ve yet to fully comprehend, and provide some guidelines for further data clean up. I’ve been spending spare time over the last few weeks in Chicago working on the data, but realize this needs the help and energy of the entire community. If you’re interested to help, please get in touch.

In short, a pretty map geeky post! Divided into ways and nodes. This may excite you, or not 😉

Ways

Ways in Kibera encompass roads, paths, streams, sewer lines (sometimes hard to tell the difference between those two), village boundaries, the railroad line, walls, permanent buildings (there are many, yes), open grounds/playing fields, and markets. So far. An incredibly dense, informal area, there is a challenge to the uninitiated to simply decide what constitutes a public road in Kibera. As it turns out, Kibera has a complex structure well known to its residents. Collecting these ways required a combination of GPS surveying, which worked reasonably well even in a dense area of corrugated iron roofs, and satellite imagery, notes written on Walking Papers and in conversation. Both introduce their own accuracies and inaccuracies, so there’s also an element of artistry involved, as usual with cartography.

These were initially traced by [User:Harry Wood|Harry Wood], from purchased DigitalGlobe satellite imagery collected in February 2009. Harry did a phenomenal job locating paths in this new terrain, which for the most part were later verified by GPS tracks. During and after the surveying phase, myself and other mappers traced from GPS tracklogs uploaded to the Map Kibera site, and from higher resolution GeoEye satellite imagery collected in July 2009 arranged by Lars Bromley of the [http://shr.aaas.org/geotech/ AAAS Geospatial Technologies and Human Rights program]. The GeoEye imagery was higher resolution (50cm, vs 60 cm for DG), brighter with a better color balance, but didn’t match the rectification of the February imagery, or of the GPS tracks. What followed was a series of tweaks and feedback between a very patient Lars and myself of re-rectifying the imagery; we finally got something which matched the GPS tracks more or less, and both learned that satellite imagery has shades of accuracy, subject to shakes in orbit, different angles of acquisition and lighting, that mean any correction in one direction results in a mistake in another region.

Besides the July imagery, the AAAS very generously donated purchase of another 5 satellite images from over the past three years in Kibera. We are very eager to explore the possibilities of automated and manual change detection and story telling using this resource; Kibera, like slums everywhere, changes rapidly, due to improvements by residents, resettlement by the government, acquisition and construction on private plots (mostly churches), and conflict on small and large scale. Imagery will help inform our understanding of these dynamics. For the moement, we have simply posted the layers to [http://aerial.maps.jsintl.org/layers/], and you are free to browse and select a slice of time. Particularly interesting are the Toi Market area, completely destroyed in the post-election violence and re-built in a new planned model, and the east side of Soweto East, the site of the first relocations and road construction. From these images is possible to date the Google imagery over Nairobi as pre-2006. For mappers, there are still a few permanent structures and walls that could use more tracing .. get in touch, and I can give you the JOSM or Potlatch settings for using the imagery.

Nearly all traced paths in OSM had GPS tracks associated, but not all, and in very dense areas, some artistic judgment was required to trace where a narrow path might really be going (these can of course be improved as more data is collected by other mappers). Road classification is still a challenge. In the formal villages, Olympic, Karanja, and Ayani, the roads are wide enough for vehicles, unpaved or in bad enough repair to qualify as unpaved, and very clearly evident in satellite and GPS, so highway=unclassfied or highway=residential. In the rest of Kibera, the situation is more interesting; for a place with no official centralized planning, there has definitely evolved a hierarchy of roads, branching fractal patterns intimately influenced by Kibera’s rugged topography. Some are wider, full of commerce, and obvious “main” roads; these have been tagged as highway=track. There are narrower paths, that are still very “public”, with significant commerce and foot traffic. These have been tagged as highway=footway. Also tagged as highway=footway are public paths through primarily residential areas. There are also even more narrow paths, nothing more than spaces between buildings, but still public; and paths that are practically private, through private plots. These all need differentiation, possibly though use of abutters=residential/commercial and private=yes tags. Complicating matters, the railway is the main thoroughfare of the area, so should be also indicated as a pedestrian area, and many of the creeks/sewers sometimes serve similar functions.

The village boundaries were initially roughly drawn from a [http://warper.geothings.net/maps/1640 map commissioned by Carolina for Kibera 7 years ago]. These were tweaked by mappers physically walking village boundaries when possible. Often these boundaries follow streams/sewers, or particular roads, and everyone is aware of precisely where they lie.

Nodes

Points of interest were the primary survey and editing activity of the Map Kibera mappers. They marked waypoints on the Garmin eTrex Legend HCx GPS, and made marks and notes on Walking Papers. They very quickly got hang of this, though there were particular subtleties, and sometimes not so subtleties, which we are still working to master.

One error that crept up occasionally were waypoints placed in an location different from where the mapper was standing. This occurs when the joystick on the GPS was moved, and quickly depressed, which the units interprets as intentionally placing a point in a different spot; vs holding the button down for 2 seconds to mark the present location. I have to say, that joystick is too clever, a persistent usability problem that has a steep body learning curve, especially for people who haven’t grown up with game controllers. Most of these errors were picked up immediately and resurveyed later; they were obviously misplaced, either in absurd locations or mappers in the wrong village, but certainly there is possibility that a few slipped through.

Each mapper definitely had their own style. In the intense density of Kibera, selecting which features are “important” is a judgment call and a matter of interest. There’s a baseline of water and sanitation features, clinics, religious and community buildings, etc. Some folks found m-pesa points important to collect, others not. Some folks picked every water collection point or water tank, even if private. Both of these things still need consistent, new tagging. Features like posho mills, battery charging stations .. entirely non-existent on any other maps. Is a movie theater in Kibera a movie theater, when it consists of a small dark room, a TV, and a DVD playing pirated movies? How to tag a witch doctor’s clinic, which these days are called “herbalists”? Most of the details on all these new features are simply in the NOTE or even name tag, all POI need some review.

Caution is needed. Even a name may not be a name. The use of a structure changes more rapidly than the availability of money to repaint a sign. So the sign might show a beauty parlor, but it’s currently used as a tailor, and everyone knows that and calls it by it’s “spoken name”. How can the map reflect both what residents already know, and what an outsider might need to know to navigate.

Some villages have much higher density of collection … as some places do have higher density of commerce, while others may be primarily residential, due to their placement peripherally to Kibera. Some are quite small, like Soweto West, so possible to comprehensively collect all. Others large places, like Makina, required additional surveyors in addition to the primary mapper, and it shows — occasionally I saw duplicate features. Capitalization never seemed to sink in with everyone … they just don’t use computers enough to care. Also, there was little care for which side of the “road” a feature sat on … something we can also improve with error checking days.

The density of features is really going to require moving to abutters and ways for many commercial areas. For web interfaces, we’ll need to separate things out into thematic, toggle-able layers. For print, we’re going to do a series of maps, atlas style, each focus on a different theme, with more narrative and photos.

So

Chickens, goats, dogs, movie theaters, hardware stores, pubs, kerosene, charging stations, butchers, trees, sewers, rocks, mud

mapkibera twitter