Over-Politeness Is the Fatal Flaw in the Open Data Movement

I am, for my sins, one of the many parents of the strange beast that is the open data movement. I was one of the first people to nag politicians about the value of opening up public data in machine readable ways. I’m one of the first people who actually wrote policy papers on the topic, for a real life government. And I was in California in 2007 when a group of like-minded people drafted the Open Data Principles. Just like LCD Soundsystem, I was, indeed, there.

So, how do I feel about it all now, eight and a half years on from the Open Data Principles? Not good. I’ve watched the government of the country I live in sell off our national addressing database, a breathtakingly moronic move that more than undoes the sum and total of good works done in the same time period. And I’ve watched that same government tacitly endorse attempts to kneecap our Freedom of Information law, whilst simultaneously grinning and waving a flag labeled ‘most transparent government in the world.’

Elsewhere, I’ve watched countries that don’t have a lot of spare money pay to launch open data portals that then languish, unloved. Which isn’t to say poor countries shouldn’t release crucial data—they should for their own good—but the drive to open up the data of poorer countries doesn’t appear to be yielding great results.

Perhaps most importantly, and worryingly, I’ve not seen any meaningful attempt to systematically overhaul procurement rules to ensure that new government computer systems produce decent open data by default. This omission is especially important since building open data feeds into new government computer systems remains the only way that most government data will ever get released cheaply, quickly, and in appropriate formats. Factories and power stations only pump out less rubbish when the law says they must, and the same is true of government computer systems. International progress on that front so far? Somewhere between dismal and undetectable.

At this point I need to be clear about something: I have not ‘lost faith in open data.’ The idea that important government data should be generally available in freely-licensed, machine-readable ways is about as controversial as the idea that popes should be Catholic. But progress towards a world in which most public data that needs to be open is seamlessly pumped out as part of Business As Usual has been so sporadic as to raise significant questions about whether the open data movement can truly be said to be winning. If anything, progress reminds me of the numerous token gestures that British governments made in the 19th and 20th centuries to appease women about their disgraceful lack of a vote, whilst continually kicking actual enfranchisement laws into the long grass. There was no business case or water-tight economic case for voter rights, either.

The Politeness Problem

There are many reasons why progress has been sporadic. People often point to the lack of a robust business case to justify either why new spending is justifiable or new laws are worth introducing, in the name of open data. Many people also blame governments for simply not wanting to be open, but that feels a bit like blaming sheep for having woolen coats. What I want to highlight is a different problem, a problem that we in the open data community must own up to because we could have done something about it: we have been entirely too polite for our own good.

‘What’s wrong with being polite?’ you might reasonably ask. Isn’t it what we train our children to be? Isn’t it what we appreciate in other people? Isn’t it the only way to get meetings with key political contacts?

Now, this is clearly true, to an extent. There are certain kinds of policy change that you can only push through by just being terribly nice and reasonable about it. Jamie Oliver’s recent sugar tax victory in the U.K. is one example, and the U.S. Magnitsky Act is another. In these cases, the demon being fought is relatively cost-free to oppose and relatively unpopular with the wider public, making politicians of all stripes feel relatively easy about taking a kick.

This isn’t at all the case for most laws, and it is especially untrue for transparency legislation. Transparency laws are like babies: There’s no way to get a real one without someone somewhere having to go through a very unpleasant experience that they’d really do almost anything to avoid. And as a consequence, meaningful transparency laws don’t get implemented except in situations where legislators fear something even worse than the effects of more transparency.

If you don’t believe me, consider the following examples.

In Britain, we did not gain the right to see our Member of Parliament’s expenses until Heather Brooke forced out a scandal so enormous that four MPs were actually sent to jail. Just in the last two weeks we appear (in a vaguely codified way) to have won the right to see our leaders’ tax returns—but only after a week in which the press savaged the Prime Minister daily over his connections to off-shore finances co-ordinated by his father. The pain required to produce this concession could literally be seen on David Cameron’s face as he was forced to face the issue.

This connection between significant political pain and transparency improvements is not unique to Britain. In Hamburg, Germany, the government recently introduced a powerful new transparency law. It’s a real doozy, and includes things that have long been on the hit list of open government advocates elsewhere, like ensuring that companies that are part-owned by the state aren’t exempt from FOIA requests.

How did this Hamburg law come about? A major concert hall went wildly over-budget, and in response a local group managed to force through a referendum demanding a powerful new transparency law to prevent this sort of failure again in the future. The local government didn’t support the referendum, but they saw which way the political wind was blowing and realized that changing their position to be in favor was more in their long-term interests than fighting against it and losing the vote anyway.

Hamburg’s isn’t the only example in Germany of transparency laws emerging from full-blown political crises. In other cities, fury around dodgy water company contracts and other major infrastructure projects have created the kind of political moments that either lead to successful FOIA referendums, or to politicians voting for transparency laws they they’d actually rather oppose.

Last, consider FOIA law in the U.S. In the U.S. constitutional context this law best is best seen as something done by one branch of government (legislative) to another branch that it habitually hates and fears (executive). Presidents, being the head of the executive branch, have shown almost uniform dislike and reluctance to adopt or expand FOIA. It started with Lyndon Johnson’s original deep skepticism, progressed to Gerald Ford’s veto of an improvement in 1974, then to Reagan and Bush II’s clamp down on responses, and then—perhaps most hypocritically of all—Obama’s secret lobbying to prevent further improvements to the law in 2014. In the U.S., transparency has only advanced significantly when the right level of congressional fear and naked political opportunism have aligned, and one branch has essentially used it as a weapon against another. The fact that Congress itself is exempt from FOI is merely an application of the old saw that one ought not to make one’s toilette where one dines.

Why point this out now?

In any country at any point in time there are only a limited number of transparency campaigners and activists. And in both the U.S. and the U.K. many of those people, including myself, chose to adopt a collaborative, non-hostile relationship with governments when they first started to consider open data around 2008 – 2010. In many countries, we were listened to, encouraged to write policy, and given lots of time and attention.

Ultimately this resulted in part of the transparency advocacy community ending up a bit like a caged bear, tamed by a zookeeper through the feeding of endless tidbits and snacks. And like the bear, by the end of the process we found ourselves somewhat sleepy and contented, but with what we really care about (freedom and most of the really important data) still firmly locked up in a cage.

I now realize that I was wrong with my original strategy of working in a constructive, friendly way with two back-to-back British governments, run by different political parties. Meaningful transparency is only ever beaten out of governments with a stick made of pure power politics. In particular, transparency reforms only really happen when key parts of the governing elite have screwed up and done wildly unpopular things; these are the moments where strong laws can get rammed through, when leaders feel a real desperate need to be seen to be doing something.

One of my transparency heroes, Ellen Miller, the now-retired founding director of the Sunlight Foundation, knew the value of an oppositional stance when it comes to transparency laws, and occasionally got grief from U.S. government officials for not being more loudly, conspicuously grateful for their open data initiatives. She was right to stay tough and demanding, something I’ll freely admit I didn’t fully see at the time.

So, finally, my paradoxical conclusion is this: As long as significant numbers of transparency advocates are engaged constructively on the business of getting open data releases out, the pace of significant transparency reforms will be slowed. Overly-friendly collaboration between governments and transparency advocates sucks the oxygen out of the room. And we need that oxygen to fuel the fires that can burn down the doors to the state, and to parasitic organizations like anonymous shell companies that are dependent upon inaction by the state.

  • jpmckinney

    Re: procurement rules, what do you think of clauses in open data policies that require new systems to be “open by default” (or whatever language each policy uses)?

    See for example the Procurement and Contracts section of Ontario’s Open Data Directive: https://www.ontario.ca/page/ontarios-open-data-directive

  • sarahschacht

    The contrast between the Ellen Miller era at Sunlight and the “friendly” era of government collaboration in the UK is a good one to make. –But it needs to be noted that Miller could make those demands because her organization, via her fundraising skills, could afford to make frenemies in government. When you look at our current field of open government advocates, most have had to jump to civic tech—much less focused on transparency and open data (esp. open data without market value)–in order to gain government contracts or technology sales.

    As the funding community for open government curtails spending, shuts down resources like TABridge, and as civil society organizations struggle to survive, the question isn’t so much if open government advocates are too polite. It’s if they can afford not to be.

  • MatthewMezey

    Hi Tom,

    Your identification of over-politeness as being at the heart of the problem happens to fit very well alongside the core findings of decades of work in organisational development. This research found that suppressing negative feelings, protecting others and acting diplomatically is deeply ingrained in almost everyone – aka politeness.

    On the face of it, it may seem like a fairly innocuous value to govern our behaviour – but don’t be deceived… ;-)

    It’s in fact a key plank in the set of interconnected behaviours and defences that keep us from ever achieving the goal of mutual learning, of building true ‘Learning Organisations’, or Government 2.0.

    The conclusion of Prof Andrew McAfee’s book ‘Enterprise 2.0’ was an explanation of how the digital technology itself is the easy bit – unlocking the grip of these governing variables is the really tough challenge. “Sustained interventions are required”, he explains – outlining the work of the late Prof Chris Argyris. (Argyris recommended an exercise called writing a left-hand column case as one way to surface the the unconscious – anti-learning- motivations that are at work in us. He also recommended Prof Robert Kegan’s ‘Immunity to Change’ exercise, which is probably quicker and easier.)

    The Open Data Movement will need to get these kinds of sustained interventions on to the agenda – if it is to realise its goals IMHO.

    Looking at these deeper roots of the status quo doesn’t seem to be something that the Open Data Movement is yet doing: a search on “Chris Argyris” and “Open Data Movement” yields a grand total of… 1 result. http://bedrosian.usc.edu/blog/counterpoint/government-2-0/


  • Chris Yapp

    I like them term ” Agents of disconfirmation” which I heard from Ed Schein in the late 1980s at MIT, Sloan. He argued that no matter how well or badly we are doing what we need is to constantly seek improvement or we go backwards, the lesson of “continuous improvement”. The examples he showed at the time indicated a tension between a positive goal and short term steps and how the focus on the goal, in this case transparency, gets in the way of progress towards the goal. The phrase ” the best is the enemy of the good” comes to mind. I’m seeing similar concerns with open innovation generally as shown here over transparency.

  • Better_Canada

    Agree 100%. The Open-Data community cannot be too “Granola” in our approach to Government and Bureaucracy. Institutions have decades of Data-Management Practices that have focused on RISK and Retention, and it is not an easy process to change that mind-set.

    You say OPEN DATA – they hear “HERE BE DRAGONS..!”

    That being said, not being “Too Polite” does not always mean being “Loud and Angry”. I think we should also look to the “Lobbyist” model that many Industries use to engage with Government.

    What are effective back-channel ways to “Lobby” the Government and Bureaucracy on the Open Data file..? How can we better use those models to make Open Data Policy and Practice a DEFAULT in Government.?

  • benberkowitz

    Yes! You continue to be my hero Steiny.

  • noel hidalgo

    If you want to look at wins, should be looking at local governments. Your post is throwing the baby out with the bath water… AND I think the open data movement has helped push us to realize we need to redesign government systems.

  • Graeme Jones

    response to @steiny @JohnWonderlich https://wordpress.com/post/openmindedly.wordpress.com/124

    more specifically the point on fair and real world open data funding mechanisms re: https://twitter.com/jonesiom/status/725299321156030464 (answer is minimum 5+ years unfunded high cost accumulated expertise and significant effort at minimum cost means you are not able to focus on some projects)

  • Duncan

    Think you may have gone a bit Jekyll and Hyde here Tom. Trick of any insider-outsider campaign is to do both at the same time, managing ‘creative tension’ with the target, and heading off internal division in your own ranks between beaters and huggers

  • Alberto Cottica

    Hello Tom, thanks for this. Really interesting thinking, and much of it sounds uncomfortably true.

    What I am still not clear about: is there anything the community could do that WOULD bring about “meaningful transparency”? The likes of me (or indeed the whole of Spaghetti Open Data, that are having this discussion right now) can growl at the Powers That Be until the cows come home, but we. Have. No. Stick. And neither do you. Unless you want to start fundraising for lawyers’ fees instead of developers salaries, and fight it out in court. That’s not everyone’s game.

    Implications: our best bet is watch the politicians play it out from now on. Ah, and support the anti-establishment parties, unsavory as they might be, because they are most likely to push the élite’s back to the wall.