During the Unlock Democracy debate at the Convention on Modern Liberty last month, Justice Minister Michael Wills defended the growth of the database state by arguing:

“We’ve heard a lot of about datasharing today. But that datasharing, that so many here today say is an unacceptable intrusion of privacy by the state, can actually help thousands and thousands of children who are eligible for free school meals but don’t get them at the moment… Look, it’s all very well for you to sit here. You’ve probably all had a hot meal in the last week. One in five of the children in my poorest wards have not had such a hot meal in the last week… You can’t walk away from this.”

Now, leaving aside the tendency of Labour politicians to come over all prolier than thou when backed into a corner (and they have the cheek to call us self-righteous…), it is an interesting point. Because however we might like to wrap ourselves up into the language of rights and freedoms, if the database state is working for the average man (or child) in the street, what hope do we have of curtailing it? And Wills’s quote is important for another reason: it highlights the fact that there is a lot more at stake here than simply whether or not we should have to carry ID cards around with us.

This brings me neatly onto The Database State, a new report published by the Joseph Rowntree Reform Trust today. The researchers of this report – Ross Anderson, Ian Brown, Terry Dowty, Philip Inglesant, William Heath and Angela Sasse of the Foundation for Information Policy Research – have looked into 46 seperate databases currently being managed by government departments and have assessed each one on its relative merits. Assigning each one a colour according to a traffic light code, this is more than simply a polemic about the “transformational government” agenda but a far broader assessment.

Where the researchers have identified a good example of how data can be managed by government they have said so. Sadly however, they have only green lighted six projects, including the National Fingerprint Database and the TV Licensing Database. Of the remainder, ten have been issued a red light (suggesting the project should be scrapped or fundamentally redesigned) while 29 have been given an amber light (suggesting significant, worrying failings which may fall foul of a legal challenge).

You will probably have heard of many of this reports main targets: the National DNA Database, the National Identity Register, ContactPoint (the national database of all children), the NHS Detailed Care Record and the putative communications database which the government wants to use to store all our itemised phone bills, email headers and mobile phone location history. But that is only the tip of the iceberg. Did you even know we had a National Childhood Obesity Database for instance, tracking the Body Mass Index of every single child in state education?

What is most striking from this report is the extent to which so many of these databases use hearsay and subjective value judgements by public sector staff. The NHS Detailed Care Record for instance will allow anyone with access to the database to write anything on a patient’s record and has no system for quality control. The Common Assessment Framework database, which is a record for sharing information about vulnerable children, is essentially a pool of subjective information about a child and his or her family which will be shared by professionals. A couple of weeks ago, Gordon Brown pledged to make our public services more like eBay in terms of allowing people to rate them, but the truth is it has effectively been government policy for public officials to issue the public with eBay-style ratings for years now.

The fundamental question is, will any of this actually help? The researchers here recognise the potential value of a lot of these projects, and have shown significant restraint in only red lighting 1-in-5 of the projects discussed. But they raise serious questions. For instance, despite the National DNA Database doubling in size in recent years, the number of crimes solved using DNA has remained steady at 1-in-300 (and actually fell slightly in 2007). That is a pretty damning statistic. The introduction of the social services’ Integrated Children’s System did not stop Baby P’s murder and despite the litany of errors that have now been highlighted, Ofsted rated Haringey as “good.” Even before this case, social care professionals were expressing concern about how the system had shifted the balance away from professional analysis and towards complying with a standardised system.

To return to Michael Wills and the children in his constituency who are not getting the free school meals they are entitled to, it is not clear how exactly all this data storage will help them but let’s assume for a moment that they did. At what cost, however? The downside to all this data storage and sharing is that every minor indescretion, and even mistakes on the database, are set to be stored for the perusal of public service professionals for years to come. The report opens with a fictional account of how all this information could be actively harmful for the very children so much of it is intended to help. How will a school teacher, for instance, treat a child who is marked on the ContactPoint register as having had contact with social services (but not specifying the nature)? If a child who is listed on the Home Office’s ONSET system for predicting offenders because of their father’s criminal record, will the police treat her or him differently if they get in a fight? Will they be treated as a victim or a suspect?

Even if the databases themselves were perfect, the people using them can never be. Throughout history, badges of shame have been used to stigmatise people, not help them; why would the modern, whizzy, virtual version be any different? They say there’s no such thing as a free lunch, but this appears to be a particularly heavy price.

It seems that only now are we waking up to the extent of the modern Database State, of which this report is a major contribution. That it has been allowed to progress for so long with so little parliamentary or public oversight is itself a serious indictment of our political system. £100 billion is slated to be spent over the next five years, during an economic downturn when we simply cannot afford it. Yet at the same time, Michael Wills’s government blocks legislation such as the Fuel Poverty Bill – something that will save the lives of some of the most vulnerable, create jobs and protect the environment. If we are to be lectured that these projects are ultimately about helping the most vulnerable in society rather than treating them as suspects, they will have to do better than this.

James Graham works for Unlock Democracy, sits on the executive of the Social Liberal Forum and writes in a personal capacity.