Media reports often feature cutting-edge projects such as Kent County Council’s Pic and Mix and can distort the public perception away from the reality that the great independent civic websites using public data are mostly having to scrape and steal it. Very few councils will even acknowledge them, let alone co-operate with them.

Our campaign to get councils to create RSS feeds intentionally gives them the easiest possible first step into the open data world — just put up an RSS feed of your news and we’ll be smiley. Yet only 27% of councils even manage that.

The relatively greater movement of councils towards social media sites like Facebook and Twitter, while not to be confused with genuine open data initiatives, is encouraging because it shows many councils’ willingness to engage users online in the places where they want to go rather than insisting that the only council web activity will happen on the council’s own site.

Despite Socitm’s Better Connected process and laudable efforts among many councils to improve their websites, the unavoidable conclusion is that councils often aren’t the best organisations to create e-government services for the public. Councils’ own websites will decline in importance accordingly.

As more people get connected and want to do more with their councils online, having a great council website won’t be enough. That’s why today Mash the State is calling for councils to dedicate 1% of their web budgets for open data projects.

Why is this?

1. People want choice. As we see with social media and RSS users, many people want and expect to be able to interact with their councils online using the websites and tools which they prefer and with which they’re familiar. And why shouldn’t they? For any council serious about engaging the public, saying “It’s our way or the highway” will no longer do.

2. Boundaries are barriers. Council boundaries and departmental responsibilities are inevitable artifacts of organising collective work but users often don’t know who’s responsible for dealing with their issue, nor do they particularly care. Why should they? IT can give citizens the ability to navigate civic life in ways that are often agnostic to the bureaucratic structures of government and to work with multiple government organisations and departments simultaneously and seamlessly. Fix My Street provides a great user experience because users don’t need to know which council is responsible for the fault they’re reporting nor find the relevant contact point within that council. It works the same way for everyone, everywhere. No single council could match that level of ease and utility. The only way to improve on Fix My Street would be to create a similar system with national scope.

3. Computing is growing, expectations are rising, budgets are shrinking. The relentless pace of technological development leaves councils running (and losing) a frantic race to catch up. Things will only get worse for councils that make it hard for people to reuse their content in ways that suit them and create third-party interfaces to their services. Just as the council gets its new website (which may have to last fundamentally unchanged for 3-5 years) along comes the next big thing and already it’s outdated. People want videos and podcasts, to be able to follow various council activities across a range of social media sites, they want iPhone apps and mobile web capability. We don’t know what they’ll want tomorrow but we can reasonably assume that councils often won’t have the skills in house to satisfy those expectations nor the budgets to hire specialists to do it for them.

4. Councils don’t have a monopoly on great ideas. With the best will in the world, when councils create web services for the public those services will reflect the ways in which the council wants to interact with the public. When the public create web services they reflect the ways in which they want to interact with the council. Which is likely to provide a better user experience? There is an enormous inertia within councils that stifles innovation, often because of genuine risks or opportunity costs. Usually it takes outsiders to rock the boat. When it goes wrong, some individuals may have wasted their time but the public has lost nothing. When it succeeds, everyone benefits.

The age of technological catch-up for councils is rapidly drawing to a close. Technologies to move content and data between applications and websites are mature and widely-implemented. Councils can no longer bury their heads in the sand and continue to act as if they believe that they have all — or even most — of the answers. The future of local e-government is letting people “pick and mix” the content and services they require using the tools and systems they prefer rather than insisting that everything must happen on councils’ own websites.

For those councils already moving strategically down this path — keep doing what you’re doing. For those that have dipped their toes in the water, take a look at the bigger picture and see where the web culture is heading. How many individuals now have their own websites as opposed to social media presences, remotely hosted blogs and Flickr accounts? How many new and recent technologies have emerged in which you see potential but don’t have the resources even to investigate, let alone implement?

Councils need stand-alone open data projects with their own resources and budgets. Lumping it in with general website work has demonstrably failed to give open data the priority it deserves.

Councils may spend that 1% in various ways. For some it will be an opportunity to create feeds and build inbound and outbound APIs, to integrate with third-party websites and services that have proven track records that provide real value. Councils could build relationships with local developers that are interested in government projects, supporting civic hacking groups where they exist and helping to create them where they don’t. “Hack the Council Day” could be a regular feature in the borough calendar. Or they could simply donate some of that money to new or existing projects that inspire them.

For the 74% of councils that still don’t have a single RSS feed, getting that done might be a good place to start.

I refer, of course, not to the erstwhile postmaster general and transport minister who retired to the grave in 1978 but to the eponymous website which has been crushed beneath the Royal Mail’s clunking fist.

Ernest, you did one thing and you did it brilliantly. You gave programmers a gizmo that converted postcodes into geographical locations. Such an unglamorous task formed the backbone of websites that provided public benefit and private delight in equal measure.

By powering Planning Alerts you let thousands get news of local planning applications where they wanted it – in their inboxes – rather than having to rummage around in the darkest reaches of their councils’ websites.

In a time of high unemployment and higher uncertainty, you gave people a fast and easy way to find vacancies near them through Jobcentre Pro Plus without having to suffer the frustrations and indignities of the Jobcentre “adviser” and the official government website.

And as the cornerstone of The Straight Choice you helped us to create a public library of election leaflets that let us judge for ourselves whether our politicians’ promises were worth the paper they were written on.

You inspired dozens of developers to create civic projects that without you would have been unthinkable and now without you may well be impossible.

At a time when joined-up government was either a breathless aspiration or an oxymoronic joke, you helped to bond parts of government that no council, ministry or quango could reach.

Above all, Ernest, you provided a glimpse of what we the people could do with free access to the data that we had paid to create and are now expected to pay for again to use.

For all this you asked for neither recognition nor recompense but just the chance to carry on doing what you loved. A chance which you were so ruthlessly and shamelessly denied.

Farewell, Ernest. You were one of us and now without you we are less.

One of the biggest impediments to councils implementing RSS feeds and other forms of open data is a lack of imagination about what they and the rest of the world can do with that data. The classic use case for RSS — reading it in a feed reader such as Google Reader– doesn’t appeal very strongly to most people that don’t already use feed readers. As much as they are useful for some, feed readers are unlikely to ever be used by a majority of web users.

Lately, some councils have discovered that having an RSS feed for their news is an easy way to get onto Twitter. They just post the items from their news feed automatically with TwitterFeed. While Twitter works best as a conversational medium (they don’t call it social media for nothing) simply streaming your news to a Twitter account isn’t a bad place to start.

Another option is delivering RSS by email. Anyone using RSS can easily enable this just by linking their feeds to FeedMyInbox. If you’re using Feedburner, that’s got an email delivery option too. No programming, no list management headaches. Feed-to-email is criminally overlooked by most RSS publishers, many of whom commit huge resources to running standalone email newsletter systems.

Now I’ve created QR Code Posters, a spinoff project from Mash the State to give people another useful RSS tool.

First and foremost, QR Code Posters just makes it easy to print the contents of an RSS feed. Despite living in an increasingly wired world, paper is still massively important. We’re surrounded by it and by and large it works. A paper poster or flyer gives your information a tangible, physical presence in the world where it can be noticed and read without using any technology at all.

But as the name implies, QR Code Posters also generates QR codes for each item of an RSS feed. These can be read by mobile phone users with appropriate software. The phone will then jump straight to the webpage for that RSS item. It’s very quick and very easy. See something of interest on a poster — “blip it” — and off you go with the full page.

Here are some QR Code Posters in the wild. We used Sutton Council‘s feeds for news, jobs and public consultations, then augmented those with a local planning applications feed from Planning Alerts. Stonecot Hill in south London, where this noticeboard is sited, sits on the boundary between Sutton and Merton councils. Planning Alerts lets us pull a single feed with planning applications within 800 metres of that point, from both councils. Perfect.

One very useful feature of QR Code Posters is that the posters are bookmarkable. So here’s a list of all the posters we used on this noticeboard tagged on Delicious Pinboard. The posters get generated dynamically every time they’re viewed online so the next time we visit this noticeboard we can just jump straight to these links and print them out again.

The phone used in the photos is an iPhone 3GS running QuickMark (i-nigma is a good, free alternative). Most smartphones can run suitable software. Search for a “barcode reader” or “QR code reader” for your phone.

QR Code Posters is integrated with Mash the State so if you’re viewing a page for a council that’s got feeds like this one for Barnet you can just click the BP icons to print posters.

Whether you’re a council officer or an information guerrilla, now’s the time to liberate your feeds from the web and get them out into the real world. And if your council is one of the 74% that still doesn’t provide feeds you know what to do.

At the bottom of each page there’s a “Subscribe to this page” link and feed icon. The site also makes these feeds easy to find by putting them in autodiscovery tags, providing a consistent way to subscribe in each browser.

Using RSS as a mechanism for receiving page status updates makes much more sense than writing a custom subscription system and requiring user registration such as on Sutton Council’s website.

There’s plenty of scope for Hillingdon to produce more comprehensive specific feeds for other uses but this is a very useful feature in its own right. Combined with a feed-to-email link on every page to a service like Feed My Inbox it could see a lot of usage.

Let’s also remember that there’s more to life than RSS. Other feed formats and APIs are more appropriate for different types of data such as iCalendar for events.

It only takes a moment to validate a feed. Invalid feeds can cause all kinds of unexpected weirdness in feed readers and other applications. Find any errors and fix them.

2. Use autodiscovery

People that use feeds a lot love autodiscovery. It provides a consistent way of finding and subscribing to feeds from any website. Put an autodiscovery <link> tag on your home page for every feed you’ve got and a tag on every interior page that’s got its own feed, eg. a tag for the news feed on the news page.

The standard tag format for RSS autodiscovery is:

<link rel="alternate" type="application/rss+xml" href="http://www.anytown.gov.uk/news.rss" title="Anytown Council News" />

and for Atom autodiscovery, use:

<link rel="alternate" type="application/atom+xml" href="http://www.anytown.gov.uk/news.atom" title="Anytown Council News" />

3. Use standard feed icons but only as direct subscription links

Whether you’re using RSS or Atom feeds, the standard feed icons are the common “beacon” style ones from Feed Icons. Get rid of the old orange rectangular “RSS” and “XML” text icons if you have them — they’re obsolete.

But don’t use the feed icons as illustrations. They should be clickable links directly to the feeds themselves so that people can subscribe. Avoid using them to link to RSS help pages or anything else.

Think: When I see this icon, I can click on it to subscribe.

4. Put your feed icons at the top of the related content

In-page feed subscription icons should be placed as near to the top of the related content as possible. Don’t bury the icon at the bottom of a list of news headlines, or even worse, in your page footer or a sidebar. Try to reinforce in your readers’ minds that the feed is an alternative way of viewing the same content.

5. Check your feeds in an RSS reader

This will often show up odd things in your feed that a validator won’t catch — like all your item dates being the same, for example.

6. Don’t lose your subscribers when your feed moves

If your CMS will let you, publish your feeds at permanent URLs. If it won’t, you’ll lose all your subscribers when you move to a new CMS and the URLs change.

If this happens, use an HTTP 301 redirect to tell your readers’ clients that the feed URL has changed permanently. Here’s how you do it.

Alternatively, proxy your feed through FeedBurner which will give it a permanent URL (and a few other toys to play with like usage stats, too).

7. Don’t offer the same feed in more than one format

People have got better things to do than to try to decide whether they want to read a feed in RSS 2.0, RSS 1.1 or Atom 1.0 format. Choose a single format for your whole site and stick with it. In practice, all formats work in all applications anyway. It gives you less to maintain and one less thing to worry about.

8. Shorten your item links if they’re longer than 255 characters

Some RSS reader applications use 255-character long fields to store item links. Some links are longer than that and will be truncated (and therefore, broken). If your CMS serves up item URLs longer than 255 characters try to run them through a URL shortening API like TinyURL first so they’ll always work.

9. Write a sensible feed title

The bit in your feed’s <title> element is what gets displayed as your feed’s title in RSS readers and other applications. Usually this should be your organisation’s name and a short indication of the content, eg:

• Anytown Council News
• Anytown Council – what’s on
• Anytown Council Job Vacancies

Far, far too many feeds just have either the “Anytown Council” or the “news” bit, forcing users to rename those subscriptions in their readers if they’re going to make any sense at all.

Here are some bad examples from the Mash the State database:

• Council Website Updates (but which council?)
• Events (for who?)
• Jobs Vacancies (sic)
• Latest Online Consultations
• Travel Information
• www.anytown.gov.uk::Latest News (get rid of all that clutter)

10. Create an easy subscribe-by-email service

Many councils provide email alert services to keep their residents up to date. This is a lot of hassle, having to deal with bounces, unsubscribes and maintaining the whole mechanism. If you’ve got RSS feeds it only takes a moment to let people subscribe to them by email using third-party services like FeedMyInbox.

You can create a direct subscription link like this:

http://www.feedmyinbox.com/?feed=http://www.anytown.gov.uk/news.rss

This is how we do the black subscribe-by-email links on Mash the State’s council pages. You get a whole email service for nothing without having to pay a penny or do any more work at all. Just like with RSS in general.

Got any more good tips? Leave a comment and I’ll work them into the article here.

Enjoy.

A few people have raised the objection that what Mash the State is currently doing with council RSS feeds is really just helping councils to deliver their PR (or as those critics often like put it, “propaganda”).

In one sense, they’re right. A council’s press releases or “news” are just their own side of the story. You’d have to be pretty naive to think otherwise.

But getting any kind of information out into the open where it can be scrutinised, compared, cross-referenced and easily discussed is always an advantage. Here’s how to build a discussion blog around your local council’s news. Of course, if they don’t have an RSS feed this isn’t possible, which is why Mash the State exists in the first place.

Time required: Around 15 minutes.

Skills required: Just basic web use stuff. No programming or HTML. Anyone online should be able to do this.

Here’s one I made earlier: Boris Backchat. Got something to say to the London mayor? Just leave a comment.

Apologies to those outside London — I had to choose something!

Here’s how I did it:

1. I registered a new blog on WordPress.com. This is free and only took a moment.

2. I found the URL (web address) of Boris’s RSS feed. Visit your local council or other government website and hover your mouse over the RSS feed link or icon. Right-click and choose “Copy shortcut” (Internet Explorer) or “Copy link location” (Firefox) or whatever your browser gives you in the right-click menu.

In this particular case it was easier to grab the feed URL from the Greater London Authority Mash the State page.

3. I signed up at xFruits which has a whole set of free tools to do things with RSS feeds. This is free.

4. I used the “RSS to my blog” tool on xFruits which automatically copies the contents of an RSS feed into a blog, making a new blog post for each item in the feed.

First I typed a title for the new blog site and a few tags.

Then I pasted in the feed URL that I’d copied in step 2.

To configure this I also needed the URL of my new blog’s “API endpoint”. This is the address which other programs can use to push data into your blog.

The API endpoint URL for this blog is:

http://borisbackchat.wordpress.com/xmlrpc.php

The format is the same for all blogs on WordPress.com:

http://yourblogname.wordpress.com/xmlrpc.php

I also had to type in my WordPress username and password, and as I’ve got several blogs on WordPress, had to choose the right one to send the RSS feed to from the drop-down menu.

5. And that’s pretty much it. I went into the WordPress settings and set the time zone correctly and edited the site description. Now we’re ready to go.

To close the loop, if you want to keep up with the latest posts on Boris Backchat you can subscribe to both the new posts (articles) and comments in your RSS reader.

xFruits will work with most popular blog systems including Blogger, TypePad, Movable Type and WordPress hosted on your own server.

Welcome to open government.

Thanks to Jon Bounds on Twitter for tipping me off about xFruits. It’s a great set of tools. Jon has just set up a similar site for Birmingham City Council.

… and as I always like to say about these things, it’s taken longer to write about it than to do it!

Today I connected 66 councils to their citizens by making it easy to subscribe to their news by email. It took me around ten minutes. I’d say this was a fairly good use of my time in terms of the ratio of effort to value produced, but I can’t claim to have done it single handed. What made it possible is that all 66 of these councils serve an RSS feed from their websites — and they’re the only ones in the country that do. Hooking those feeds up to FeedMyInbox through the council pages at Mash the State was a simple matter of dropping a single web link into a template and pushing it to the live site. Job done.

RSS is a simple way of getting data out of a website and into another program. The technology is ten years old and RSS feeds are ubiquitous on blogs, on mainstream news media websites and in Web 2.0 applications. The three leading web browsers — Internet Explorer, Firefox and Safari — all contain built-in RSS readers. Yet despite running websites costing tens of thousands of pounds annually each, only 15% of UK councils bother with RSS. Nothing could be more symbolic of large parts of government’s unwillingness to think beyond the confines of their own websites than making it practically impossible to receive basic local council information like news and events except by taking a trip to anytown.gov.uk to do it on the council’s own terms.

The ten minutes it took to emailify those 66 councils compare quite unfavourably with probably a similar number of hours I’ve spent trying to scrape Sutton Council’s news into a database, and from there through Delicious into RSS and Twitter. Writing screen scrapers — programs which extract text from web pages and turn them into structured, reusable data — is sometimes tricky but Sutton’s news is trickier than most. The news archive serves inconsistent page structures and even dynamically changing URLs to compete with. I vowed never to write another scraper, though as we’ll see, that’s a promise I soon had to break.

Screen scraping and copyright infringement are the dirty not-so-secrets of the civic hacking world. Show me a useful, innovative third-party civic website and I’ll most probably be able to show you the terms and conditions that were ignored and the data that was taken and repurposed without permission or legal licence. Similar behaviour is not unknown in the public sector itself, in some cases because government organisations are recycling that very same stolen data from third party applications into their own websites. The recent Rewired State National Hack the Government Day saw some incredibly inspiring, innovative and useful projects produced in very short order. How many of these projects didn’t involve citizens jailbreaking their own government to get the data they’ve paid for? What kind of society not only massively impedes but actually criminalises — in principle if not in practice — citizens devoting their own time, skills and money to write software to improve democracy and public services? Our society, it seems.

This has to stop. Hackers have shown their ability and willingness to surmount technical obstacles and run legal risks to get the data they need but less technical citizens simply cannot. No-one should have to. A rich, technologically-advanced and supposedly forward-thinking society such as ours should make citizens’ access to government data so commonplace that it doesn’t deserve comment. No technical wizardry required. No legal minefields to navigate. Just all the data served through common protocols with open licences that permit, well, anything. Then we can focus our time and energy on the considerably more interesting higher-order opportunities that come from actually using government data, not just getting hold of it.

Last week I launched Mash the State, a national campaign to get government data to the people. It’s not a new idea but our method is. We’ll be setting up a series of challenges to the public sector, asking one group of public bodies at a time to release one specific set of data. Our first challenge asks all local councils to serve up an RSS news feed by Christmas. I wouldn’t have bet good money in 2003 that by 2009 370 councils would still be without RSS, but here we are. I’ve thrown the gauntlet down and I’m pleased to see that a couple of hundred people have signed up to our website or followed us on Twitter to help make this happen. The councils have got over eight months to do what in most cases will not be more than half a day’s work to serve RSS from their websites. Others less fortunate will have to persuade their content management system suppliers to enable this feature for them. All have got plenty of time to perform this technically trivial task in time to give the public a small but highly symbolic Christmas present that shows that government in this country is prepared to trust its citizens with their own data.

As for my promise never to write another scraper, it didn’t last long. The very first task to build Mash the State was an hour spent writing a scraper to tease a list of councils from a government website. Join us and help to hasten the day when no-one will ever have to do anything like that again.

Twitter isn’t to everyone’s taste so for those that want to keep up to date with what’s going on but don’t want to join Twitter, here are three alternatives:

Visit our Twitter page regularly. It’s completely public and you don’t need to log in. There’s nothing extra available to Twitter members (or followers of our feed there) that isn’t visible on the page.

Subscribe to the RSS feed for our Twitter updates using your favourite feed reader.

Receive our Twitter updates via email using Feed My Inbox. You’ll have to paste this feed URL into the form:
http://twitter.com/statuses/user_timeline/26220459.rss

You’ll get a maximum of one email per day no matter how many updates are posted to Twitter.

If you’re publishing feeds on your own website you can create an easy Feed my Inbox subscription form here. Unfortunately you can’t link directly to the feed subscription form with a pre-populated URL yet, but I’ve asked the developer to consider adding that feature.

It’s nearly bedtime so this is just a brief hello and welcome to all the early-adopting state mashers out there.

If you’re wondering what this is all about, I hope most of your questions are answered in the FAQ. If not, please ask them here and I’ll do my best to answer them.

For now, this is also a good place to leave any general comments about the campaign and the website.

Thanks for coming along and let’s hope we can build this into a successful, sustainable campaign. Right now, it’s going to be one council and one RSS feed at a time. Only 370 to do before Christmas!