Why there isn’t a British Nate Silver

A skim-read of Wednesday morning’s headlines might have left folk confused as to who had been declared the victor of the US presidential election: Barack Obama or Nate Silver.

For those who don’t know Nate Silver, he’s the analytical guru behind the FiveThirtyEight blog (named after the number of electoral college votes), now housed at the New York Times, which scrutinises and filters opinion polls. He first rose to prominence four years ago after predicting the winner in 49 of the 50 US states during the Obama-McCain presidential contest, and re-inforced his reputation two years later by forecasting correctly all 35 senate races.

His cred has now reached near-messianic status. First, because yet again his methodology has been validated — if Florida is eventually declared for Obama, Nate will have another clean sweep of predictions. Secondly, because he has consistently called-out the media for portraying the race as tied despite the high likelihood that Obama has maintained a solid electoral college majority throughout the year. And thirdly, because of the personal insults that have been flung in his direction over the past few weeks by Romney supporters aggrieved that Nate didn’t have better news for their guy.

I’m a huge fan of Nate Silver’s forensic approach and FiveThirtyEight has been my first port of call to find out how the US presidential race is actually going, rather than through the mainstream media’s unfocused lens. But Daniel Engber makes a fair comment on Slate.com when he points out: “Nate Silver didn’t nail it; the pollsters did.”

And that is, I suspect, the single biggest reason this country hasn’t yet, and is unlikely to, produce our own version of Nate Silver — because we don’t have the necessary quality of polls and because we don’t have a media interested in actually finding out what’s going to happen. Let me unwrap those two assertions:

1) Absence of local polling

The reason most commentators could get away with their blather that the US presidential race was tied was because they reported the headline national numbers. These did indeed show Obama and Romney to be essentially neck-and-neck, and that’s how the national result has finished. But averages conceal as much as they reveal. What Nate Silver (and others psephologists such as RealClearPolitics and Talking Point Memo) highlighted was the data from state polls, in particular the ‘toss-up’ states which would decide the next occupant of the White House.

The same mistake that Nate Silver’s decriers made is almost universally made day-in-day-out by British pundits using national polls to project the next general election result. Ironically this includes many commentators who’ve praised Nate Silver’s approach in the US and yet disregard its applicability in the UK (yes, I’m looking at you John Rentoul).

Everyone knows that results differ across the UK: that what happens in Cornwall won’t be repeated in London or Scotland or the East Midlands etc. Yet when it comes to projecting the results of the next general election, the media is all too happy to shove a load of polls in the blender and hope it approximates to reality.

The problem is highlighted by Nate Silver himself. He attempted to devise a model for forecasting the last UK general election based on the polls. The final projection showed the Lib Dems in second place and winning 120 seats. Does this show Nate’s voodoo is a load of hooey? No, it shows that the British focus on national polling is a poor basis for projections of numbers of MPs. And it is also fair to point out that Nate Silver 1) inserted many caveats into his prediction, and 2) came up with a range of projections, the final one of which (scroll down to the bottom of this page) wasn’t so very far off.

2) Absence of media interest in accuracy

So if we accept that national polling is a deeply imperfect guide to projecting what the House of Commons will look like after the next election, it’s fair to ask the question why doesn’t the media — why don’t the polling companies — commission the kind of research that is needed to make accurate forecasts? It’s a question easily answered with another question: what’s in it for them?

I’m sure the polling companies themselves would be very happy to undertake more regional polling, to find out if the results in the North-West will differ from those in East Anglia. But they’ll need paying, not least because political opinion polling is only a very small part of their business (albeit the most publicly visible). And as there isn’t a strong, profitable regional media it doesn’t happen. There have been sporadic attempts to do it: PoliticsHome and The Sun have both commissioned extensive polls of marginal seats, but these are special one-offs so no reliable trend emerges that allows a Nate Silver-style calibration.

The blunt reality is that the news media craves excitement more than it hungers for truth. It is much cheaper and easier to commission a monthly survey and then inflate the results way beyond what the data should allow. We’ve all seen the kinds of headlines newspapers revel in — ‘Poll blow to Tories as support plunges 1%’, ‘Labour to win 100+ majority says latest exclusive poll’ — and yet journalists continue to write them even though they know deep down how flimsy the evidence is.

Finally

I’ve read lots of adulation of Nate Silver in the British media in the past 24 hours. I wonder if any of those journalists who’ve penned those articles have thought, even for a moment: I wish I had the confidence to write about polls with the same kind of rigour he does. I’m not holding my breath.

* Stephen Tall is Co-Editor of Liberal Democrat Voice, and editor of the 2013 publication, The Coalition and Beyond: Liberal Reforms for the Decade Ahead. He is also a Research Associate for the liberal think-tank CentreForum and writes at his own site, The Collected Stephen Tall.

Read more by or more about , , or .
This entry was posted in Op-eds.
Bookmark the web address for this page or use the short url http://ldv.org.uk/31387 for Twitter and emails.

17 Comments

  • Bill le Breton 8th Nov '12 - 9:41am

    I agree with Nick. And YouGov seems to have performed well over the US election. It is human nature not to want to hear what doesn’t suit. It’s easy to say Kellner and Shakespeare have agendas, but that is to ignore them as business people.

    On Monday I visited In-Trade and you could buy an Obama share for $7 … if he won the next you’d be paid $10 dollars. I just couldn’t believe it, but clearly all those GOPs were putting money where their mouths were and paying $3 for a share in Mitt R.

    So, I really am not sure that we don’t have the necessary info – it is just inconvenient to the powers that be . No doubt

    Another very interesting stat from the exit polls was the answer to X or Y ‘cares about people like me’. There was a great interview with a local shop keeper saying that the the Republican Party had many of the values he and his family had. ‘They just don’t want me in the country!”

    Yesterday at PMQs, Nick Clegg reacted to his female opposite number at the dispatch box like a Tory bore.

    We continue to alienate the very people who share our values. ‘You don’t need a weather man to tell which way the wind’s blowing’. Ditto an British Nate Silver.

  • Isn’t a much more fundamental problem that in the US, if you exclude the inelectible other candidates, ‘tactical’ voting doesn’t play a role? It plays a huge role in the UK and make elections a lot harder to predict.

    And given the sizes of the constituencies, I don’t think it’d be feasable to have local polls. Perhaps in some cases, in some marginal seats, but Silver depended on extensive polling. The number of marginal seats in the UK is much larger than the number of swing-ish states in the US.

  • @Martijn we used to get constituency polls routinely for by-elections, and that was back in the era when polls were done face-to-face.

    The problem is that you’d be wanting a couple of hundred separate polls, and you’d want them polling many times – probably at least four or five times in the campaign. As a single poll costs into the thousands of pounds, you’d be looking at selling out three or four million at the least to build the data.

  • Um, that should be shelling out…

  • Yes. Not to mention that you want them to be done by different pollsters. That’s where SIlver’s strength lies.

    And again, don’t underestimate the influence of having three (or more, in many cases) parties getting a non-negligible share of the vote. For instance, a financial scandal at UKIP may mean many people suddely decide to vote Tories again, which will then hurt other parties.

    Also, making your mind up about which of two candidates to vote for at the last moment is one thing — good predictions can account for that and in general last-minute deciders tend to be fairly evenly spread. Making your mind up about whether to vote tactical or not is another thing, and probably not so evenly spread.

  • “The problem is highlighted by Nate Silver himself. He attempted to devise a model for forecasting the last UK general election based on the polls. The final projection showed the Lib Dems in second place and winning 120 seats. Does this show Nate’s voodoo is a load of hooey? No, it shows that the British focus on national polling is a poor basis for projections of numbers of MPs.”

    Surely it was much more to do with the fact that the polls during the campaign showed much higher levels of Lib Dem support than materialised on polling day.

  • is this a call to loosen restrictions on election spending so more polling can be done?

    Given US elections cost as much as 20-times more per head than UK elections I think most voters would be quite happy with the value for money involved here…

  • Matthew Huntbach 8th Nov '12 - 11:59am

    The UK national media almost ignore what is going on at ground level in elections, they tend to work on the assumption there is a uniform swing across the country. Local media in the UK is in a pretty sorry state and rarely seems to be geared up to a proper analysis of the local political scene. Given the huge number of people who want to get into journalism, the very small number of actual jobs going in that field, and the fact that the local press is still a common route into it, I’m surprised that most of the local press journalists I’ve encountered – generally young people who’ve had to get through a lot of competition to get the post and have ambitions to move further -don’t seem to have much of an idea about how local government works.

    Funnily enough, local elections in recent decades seem to have moved away from being to a large extent opinion polls on the national government and become more on local issues. A lot of the results that look odd in terms of going against the national trend are easily explained if you look at the local issues. In Parliamentary elections what is going on in the constituency does make a big difference, but you would hardly be aware of it from what you read in the national press at election time. This does have quite a lot to do with the Liberal Democrat emphasis on local campaigning – unexpected swings either way really can be put down to a few enthusiastic people getting together and building up a good campaign team in a place where once we did little, or in the other way to someone that used to be the central organiser dropping out or moving on, or to a big personal bust-up. So it isn’t always down even to regional swings.

    From what I’ve seen of the coverage of US elections, the federal nature of the US does lead to more of a consideration of the polls in terms of what’s happening locally. Note also that the US doesn’t have a national press – all its newspapers are local newspapers, and broadcast media is pretty localised as well.

  • Nicola Prigg 9th Nov '12 - 1:13pm

    I think Andrew Wells is our version of Nate Silver.

    If I want good analysis of the polls, I go to Ukpollingreport.

    As for local polling, I agree with comments and the article above, you do need local polling.

    Would regional polling fare better though?

    Whilst the Scottish Polls can predict the Scottish elections rather better, after all its a proportional system. But when it comes to constituencies for Westminster, a Scottish poll would do just as badly as a UK national poll, precisely because it doesn’t know what will happen at constituency level.

    In order to get accurate predictions for local, Scottish & UK elections, you would almost have to go down to ward level and then build the picture up into the various authorities/constituencies being fought to see how multiple elections would fall.

  • Charles Beaumont 9th Nov '12 - 1:24pm

    As a long-term Silver fan (I followed his original site throughout the 2008 campaign) one of the things that struck about the vitriol heaped on him this time was the innumeracy it exposed in the punditry. People who are presumably intelligent and well-educated were unable to distinguish between probability and margin. Various right-wingers (including the Torygraph’s Tim Stanley) suggested it was ludicrous that Silver had over 80% probability of an Obama win even as Silver himself was predicting the very narrow margin in the popular vote. I think we have the same problem in the UK where being scientifically illiterate is socially acceptable.

  • @Nicola Prigg, Wells has a great article up now about just how little interest most voters take in politics, even during campaigns. Its something we politicos need to remind ourself of all the time
    On the general point about polls, all claims about “accuracy” are based on works done just days before the election. Polling firms dont boast about how good their work was 2 or 3 years in advance because its useless nonsense. Journalists ignore that because polls make good “news” & the dafter the better.

  • Paul McKeown 11th Nov '12 - 12:08pm

    @Bill le Breton

    “And YouGov seems to have performed well over the US election.”

    Hmmm. Have a look what Nate Silver, himself, says. See http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and-worst-in-the-2012-presidential-race/#more-37396

    I can’t see how you reckon YouGov did well, in fact they were amongst the worst of the large polling organisations. Their 30 polls had an average error of 2.6% with a bias to the Republicans of 1.1%. Pretty grim actually.

  • Barry George 12th Nov '12 - 1:02am

    Their 30 polls had an average error of 2.6% with a bias to the Republicans of 1.1%. Pretty grim actually

    A 2.6% average error is well within the 3% MOE that you get with all polling and a 1.1% bias to the republicans is pretty damn good for awhat is essentially a prediction.

    If polling was an exact science there would be no need for any polling…

    I agree with bill… ” YouGov seems to have performed well over the US election ”

  • Paul McKeown 12th Nov '12 - 3:55pm

    @Barry

    ” YouGov seems to have performed well over the US election ”

    Well, he said that in comparison with Gallup and several other big names, who were utterly hopeless.

    The truth is YouGov were pretty poor. The list provided shows how far behind the pace they were. Let’s help you here. The following polling organisations had both a smaller average error and a smaller bias (towards to the Republicans) than YouGov:
    IBP/TIPP 0.9%/+0.1%,
    Google Consumer Surveys 1.6%/+1.0%,
    Mellman 1.6%/0.0%,
    CNN 1.9%/0.6%,
    Grove Insight 2.0%/+0.1%,
    Survey USA 2.2%/0.5%,
    Quinnipiac 2.3%/0.3%,
    Marist 2.5%/+1.0%.

    Many further organisations had either a smaller (often much smaller) average error or bias than YouGov.

    YouGov’s performance was mediocre. Although Gallup was abysmal and Washington Post/ABC News, Pharos, Rasmussen, American Research Group and Mason Dixon may all have been worse than YouGov, that doesn’t make YouGov’s performance one jot better than it was. It’s polls were much less representative of the actual ballots than those provided by many other organisations.

    I note that the YouGov supporters take a consistently strong anti-coalition line on this website and that YouGov consistently provides the LibDems with worse polling figures than other organisations. I wonder whether there would be any connection between the two?

    :-)

  • UK opinion polls are far more inconsistent than US polls. YouGov and ICM can’t both be right at the moment. Anthony Wells puts the difference down to what the pollsters measure: YouGov weights its findings towards how people are telling them they will behave; ICM weights its finding based on how it expects people to behave – not the same thing.

    Voter intention matters more in the US too, because of the electoral system. The US presidential election has a strongly 2 party system with 50 FPTP contests each with millions of electors. Our general elections are weakly 2 party and are held over far more constituencies with far smaller electorates. This makes voter intention much less indicative in UK elections.

    How I wish we’d run our polling would be to carry out national VI polls to determine the swings between the parties in each constituency, then carry out constituency-level VI polls on the projected marginals. What ultimately really matters in British politics is who wins each seat and this would be the best way to project this. Instead of wasting money on daily polls from YouGov that tell us very little, News International could invest its money into a slower rate operation that actually tell us what would happen if an election were held.

Post a Comment

Lib Dem Voice welcomes comments from everyone but we ask you to be polite, to be on topic and to be who you say you are. You can read our comments policy in full here. Please respect it and all readers of the site.

If you are a member of the party, you can have the Lib Dem Logo appear next to your comments to show this. You must be registered for our forum and can then login on this public site with the same username and password.

Your email is never published. Required fields are marked *

*
*
Please complete the name of this site, Liberal Democrat ...?




Recent Comments

  • User AvatarAllan 2nd Sep - 7:02am
    I accept I may be wrong but Scotland wishes a share of assets if they decide in favour of independence surely if they do not...
  • User AvatarCaractatus 2nd Sep - 6:21am
    "The tug o war is between those who see larger airports as the key to economic growth and" ..... those who think there are better...
  • User AvatarIain Donaldson 2nd Sep - 5:53am
    Thanks for all the feedback, I appreciate that others have already answered some of the points raised in the original article, but I feel that...
  • User AvatarCaractatus 2nd Sep - 5:38am
    Ah yes, remember when we used to get the badly written weekly email from the leader - what happened to that, dropped because it was...
  • User AvatarColin 2nd Sep - 4:58am
    “Don’t think. Don’t bother with the details. You don’t have time to research the arguments; you are too busy, and there’s not enough time, and...
  • User AvatarRichard Dean 2nd Sep - 4:12am
    I wonder if part of the solution Dave Page seeks is to provide more information, not less, about the reality of life for jihadists in...