Over-Politeness Is the Fatal Flaw in the Open Data Movement

I am, for my sins, one of the many parents of the strange beast that is the open data movement. I was one of the first people to nag politicians about the value of opening up public data in machine readable ways. I’m one of the first people who actually wrote policy papers on the topic, for a real life government. And I was in California in 2007 when a group of like-minded people drafted the Open Data Principles. Just like LCD Soundsystem, I was, indeed, there.

So, how do I feel about it all now, eight and a half years on from the Open Data Principles? Not good. I’ve watched the government of the country I live in sell off our national addressing database, a breathtakingly moronic move that more than undoes the sum and total of good works done in the same time period. And I’ve watched that same government tacitly endorse attempts to kneecap our Freedom of Information law, whilst simultaneously grinning and waving a flag labeled ‘most transparent government in the world.’

Elsewhere, I’ve watched countries that don’t have a lot of spare money pay to launch open data portals that then languish, unloved. Which isn’t to say poor countries shouldn’t release crucial data—they should for their own good—but the drive to open up the data of poorer countries doesn’t appear to be yielding great results.

Perhaps most importantly, and worryingly, I’ve not seen any meaningful attempt to systematically overhaul procurement rules to ensure that new government computer systems produce decent open data by default. This omission is especially important since building open data feeds into new government computer systems remains the only way that most government data will ever get released cheaply, quickly, and in appropriate formats. Factories and power stations only pump out less rubbish when the law says they must, and the same is true of government computer systems. International progress on that front so far? Somewhere between dismal and undetectable.

At this point I need to be clear about something: I have not ‘lost faith in open data.’ The idea that important government data should be generally available in freely-licensed, machine-readable ways is about as controversial as the idea that popes should be Catholic. But progress towards a world in which most public data that needs to be open is seamlessly pumped out as part of Business As Usual has been so sporadic as to raise significant questions about whether the open data movement can truly be said to be winning. If anything, progress reminds me of the numerous token gestures that British governments made in the 19th and 20th centuries to appease women about their disgraceful lack of a vote, whilst continually kicking actual enfranchisement laws into the long grass. There was no business case or water-tight economic case for voter rights, either.

The Politeness Problem

There are many reasons why progress has been sporadic. People often point to the lack of a robust business case to justify either why new spending is justifiable or new laws are worth introducing, in the name of open data. Many people also blame governments for simply not wanting to be open, but that feels a bit like blaming sheep for having woolen coats. What I want to highlight is a different problem, a problem that we in the open data community must own up to because we could have done something about it: we have been entirely too polite for our own good.

‘What’s wrong with being polite?’ you might reasonably ask. Isn’t it what we train our children to be? Isn’t it what we appreciate in other people? Isn’t it the only way to get meetings with key political contacts?

Now, this is clearly true, to an extent. There are certain kinds of policy change that you can only push through by just being terribly nice and reasonable about it. Jamie Oliver’s recent sugar tax victory in the U.K. is one example, and the U.S. Magnitsky Act is another. In these cases, the demon being fought is relatively cost-free to oppose and relatively unpopular with the wider public, making politicians of all stripes feel relatively easy about taking a kick.

This isn’t at all the case for most laws, and it is especially untrue for transparency legislation. Transparency laws are like babies: There’s no way to get a real one without someone somewhere having to go through a very unpleasant experience that they’d really do almost anything to avoid. And as a consequence, meaningful transparency laws don’t get implemented except in situations where legislators fear something even worse than the effects of more transparency.

If you don’t believe me, consider the following examples.

In Britain, we did not gain the right to see our Member of Parliament’s expenses until Heather Brooke forced out a scandal so enormous that four MPs were actually sent to jail. Just in the last two weeks we appear (in a vaguely codified way) to have won the right to see our leaders’ tax returns—but only after a week in which the press savaged the Prime Minister daily over his connections to off-shore finances co-ordinated by his father. The pain required to produce this concession could literally be seen on David Cameron’s face as he was forced to face the issue.

This connection between significant political pain and transparency improvements is not unique to Britain. In Hamburg, Germany, the government recently introduced a powerful new transparency law. It’s a real doozy, and includes things that have long been on the hit list of open government advocates elsewhere, like ensuring that companies that are part-owned by the state aren’t exempt from FOIA requests.

How did this Hamburg law come about? A major concert hall went wildly over-budget, and in response a local group managed to force through a referendum demanding a powerful new transparency law to prevent this sort of failure again in the future. The local government didn’t support the referendum, but they saw which way the political wind was blowing and realized that changing their position to be in favor was more in their long-term interests than fighting against it and losing the vote anyway.

Hamburg’s isn’t the only example in Germany of transparency laws emerging from full-blown political crises. In other cities, fury around dodgy water company contracts and other major infrastructure projects have created the kind of political moments that either lead to successful FOIA referendums, or to politicians voting for transparency laws they they’d actually rather oppose.

Last, consider FOIA law in the U.S. In the U.S. constitutional context this law best is best seen as something done by one branch of government (legislative) to another branch that it habitually hates and fears (executive). Presidents, being the head of the executive branch, have shown almost uniform dislike and reluctance to adopt or expand FOIA. It started with Lyndon Johnson’s original deep skepticism, progressed to Gerald Ford’s veto of an improvement in 1974, then to Reagan and Bush II’s clamp down on responses, and then—perhaps most hypocritically of all—Obama’s secret lobbying to prevent further improvements to the law in 2014. In the U.S., transparency has only advanced significantly when the right level of congressional fear and naked political opportunism have aligned, and one branch has essentially used it as a weapon against another. The fact that Congress itself is exempt from FOI is merely an application of the old saw that one ought not to make one’s toilette where one dines.

Why point this out now?

In any country at any point in time there are only a limited number of transparency campaigners and activists. And in both the U.S. and the U.K. many of those people, including myself, chose to adopt a collaborative, non-hostile relationship with governments when they first started to consider open data around 2008 – 2010. In many countries, we were listened to, encouraged to write policy, and given lots of time and attention.

Ultimately this resulted in part of the transparency advocacy community ending up a bit like a caged bear, tamed by a zookeeper through the feeding of endless tidbits and snacks. And like the bear, by the end of the process we found ourselves somewhat sleepy and contented, but with what we really care about (freedom and most of the really important data) still firmly locked up in a cage.

I now realize that I was wrong with my original strategy of working in a constructive, friendly way with two back-to-back British governments, run by different political parties. Meaningful transparency is only ever beaten out of governments with a stick made of pure power politics. In particular, transparency reforms only really happen when key parts of the governing elite have screwed up and done wildly unpopular things; these are the moments where strong laws can get rammed through, when leaders feel a real desperate need to be seen to be doing something.

One of my transparency heroes, Ellen Miller, the now-retired founding director of the Sunlight Foundation, knew the value of an oppositional stance when it comes to transparency laws, and occasionally got grief from U.S. government officials for not being more loudly, conspicuously grateful for their open data initiatives. She was right to stay tough and demanding, something I’ll freely admit I didn’t fully see at the time.

So, finally, my paradoxical conclusion is this: As long as significant numbers of transparency advocates are engaged constructively on the business of getting open data releases out, the pace of significant transparency reforms will be slowed. Overly-friendly collaboration between governments and transparency advocates sucks the oxygen out of the room. And we need that oxygen to fuel the fires that can burn down the doors to the state, and to parasitic organizations like anonymous shell companies that are dependent upon inaction by the state.