With Conference season upon us once again, Parties reach out to embrace their wider membership of activists, supporters and sympathisers. This brief popping of the Westminster bubble is of course vital: a safeguard stopping Westminster disappearing into its own parochial obsessions.

Party Conference is only one of a number of ways of dipping into the wider public mood, of course. Polls, focus groups, constituency surgeries, party machinery and, indeed fora like Liberal Democrat Voice all allow views and concerns to (sometimes) percolate up to the leadership. And some politicians – such as Tony Blair at his height – seem to have a mysterious alchemical faculty to intuit the public mood.

The explosive rise of social media offers a new opportunity. These digital-social worlds, alive with the arguments, condemnations applause, derision and accolades of millions, are an unceasing and gigantic monument to social and political reality-in-action. More measurable and analysable than any other form of social discourse, understanding this ‘buzz’ has already revolutionised marketing and advertising. It could transform politics too.

From showing the reactions of millions to a speech, announcement or debate, to generating insight on hard to reach groups, or emerging topics or complaints, social media research could make governments more agile and reactive. It could transform the way that a political party interfaces with the outside world, framing the agendas, topics and issues on which political energies are spent.

But we’re not there yet. Our current ability to understand social media falls far short of the standards of evidence required for traditional polls, academic sociological or attitudinal research, or public policymaking. Sampling from social media is fraught with difficulty and results stand in danger of letting the very loud voices of a (usually rich, urban, young) few drown out the lurking majority. Analyses of social media datasets usually pay little attention to questions of language, culture or context (not to mention ethics). With samples of convenience serving up raw metrics, with little regard to language, culture or context the industry in general hasn’t moved too far beyond being a counting game.

Nor should social media research be used with reckless abandon. As Demos argued in its paper #intelligence, social media research must respect the privacy concerns and the expectations people have about how information might be used as a matter of routine, professional habit. Especially given the widespread cynicism with politicians obsessed with the tiniest movement in their polling figures, it is also important that social media research is used in a genuinely inclusive and empowering way – setting the agendas and framing debates, not only a crude means to write the best political pitches.

This week, the think tank Demos launched an initiative to help navigate these opportunities and threats. The Centre for the Analysis of Social Media brings together policy specialists, social scientists, computer scientists and tech entrepreneurs to research social media in a way that is reliable, ethical, and useful to help solve pressing social and political problems.

‘The Independent View‘ is a slot on Lib Dem Voice which allows those from beyond the party to contribute to debates we believe are of interest to LDV’s readers. Please email [email protected] if you are interested in contributing.

* Carl is the Research Director of the Centre for the Analysis of Social Media, Demos.