The Data Gap: When The Tools Are There But The Data's Not

As readers of this blog know, data sets make my heart beat faster. Data sets that have been analyzed/visualized and made to tell a story are even better. The work being done by organizations like the Sunlight Foundation Labs (who are taking publicly available data sets about US congressional/political issues and making them do interesting things online) is tremendously exciting, and inspiring to many other organizations around the world. I spoke to Tate Hausman of dotOrganize earlier this week about the Integration Proclamation, a data sharing manifesto that he hopes will lead to real action on the part of vendors and open source communities. The goal of dotOrganize's latest project: best-practice open APIs ("application programming interface") for all data applications used in the non-profit sector. The end result, in an open API world, will be data that flows easily between applications, and perhaps equally exciting, out of applications and into the world of data analysis.

It's easy to get excited about these moves towards a world where data is free (as in freedom) and accessible, and the tools to analyze it, like Swivel and Many Eyes, are available to anyone with a web connection. Where it all grinds to a screeching halt is when you get to places where there really isn't much data. In most of the world, governments don't make information publicly available, even if they're supposed to -- see fabulous FarmSubsidy.org's efforts to make EU governments cough up information on national distributions of Common Agricultural Policy funds -- that is, EU taxpayer money. Except in wonderfully transparent countries like Slovenia (surprise!), this is pulling teeth, and only works through an application of Freedom of Information Act laws, a lengthy and complicated process that can end in stalemates and government hedging or partial release of information.

But at least in this case, the information exists -- not with easy access, and not in nice XML-y formats that lend themselves to clear comparison across EU countries. Where it gets trickier is when you move to the parts of the world where my employer, the Open Society Institute, does a lot of work. In sub-Saharan Africa, for instance, data is rarely available from governments on many issues important to civil society -- educational budget distributions, public health budget distributions, etc. Forget about information on things like political contributions or, in many cases, basic public information like parliamentary voting records. Where information is available through a government, there may be little accountability on where it came from or its accuracy. What happens is that data sets need to be painstakingly created through the hard work of civil society monitoring groups, like the Public Service Accountability Monitor in Grahamstown, South Africa, which follows the activities of the government of the Eastern Cape, or IDASA, also in South Africa, which undertakes monitoring projects on a range of issues from local government accounability to distribution of national HIV/AIDS budgets on an extremely granular level, or Mzalendo, the online project run by Ory Okolloh and anonymous blogger M out of Kenya to track parliamentary activity in their country. But this can be expensive and/or time-consuming work, and fraught with difficulties -- not the least of which are that the people who are good at this kind of monitoring are not necessarily directly in touch with the advocacy groups that can use the data effectively to change policies. Even when serious data is available in the developing world, the skills to analyze and put it to compelling use are often spread very thinly on the ground.

My guess is that the data gap will continue to plague the developing world for years to come. A combination of factors are behind it -- government corruption and self-dealing are tremendous disincentives for transparency, as well as skill gaps, technology gaps, and perhaps most importantly, the lack of consistent demand for (and support for) this kind of transparency from international donor organizations.