In a blog posting earlier today I touched on the issue of forecasts which are given to a spurious degree of accuracy, and by coincidence I’ve just come across another example, this time from the world of online advertising.
Nate Elliott of Forrester has produced a report on the outlook for online advertising, saying that:
We had previously forecasted that online ad spend would account for 12.6% of all European advertising in 2012; thanks to the recession, we’ve increased that forecast to 14.8%.
Do you think that you can forecast (a) how much advertising there will be in Europe in three years time, and (b) how much online advertising there will be in Europe in three years time both to a sufficient degree of accuracy to be able to work out a percentage to one decimal place? It would take a lot to convince me of that, especially as Forrester’s own record suggests a rather wider margin of error may be sensible to apply to their forecasts:
Internet spend throughout Europe, which until 2007 has been increasing as much as 30% a year, is set to rise by just 10% in 2009 … Two years ago, Forrester predicted online ad spend would increase by 14% this year.
In fairness, this sort of spurious accuracy is by no means restricted to Forrester, but that doesn’t make it sensible and in fact an up front approach to the likely range of error around forecasts would make them much more helpful in my view.
Source: Media Week
Follow @libdemvoice.org on Bluesky
Like us on Facebook
Subscribe to our feed
Sign-up for our daily email digest
In which my great great grandmother's brother stuffs a ptarmigan
Replacement?
Reform inadvertently lose their majority
Reform own goals
Peter Martin
Roland
Tristan Ward
Nonconformistradical
6 Comments
Is spurious accuracy (giving figures with decimal points) better or worse than rounding to the nearest whole number?
It’s possible that their methodology involves taking a number of different estimates from different sources, then averaging them (possibly with some weighting). This is highly unlikely to result in a round number, and rounding it up or down wouldn’t make it any more accurate.
You’re right that their prediction probably won’t turn out to be accurate, but it’s a best guess. And 14.8% might end up being a better guess than 15%. If their methodology gives them 14.8%, then that’s what they have to go with.
I think you mean precision, not accuracy.
Rob: if you’re quoting a figure to one decimal place, I think you’re then implying that is the degree of accuracy which can be associated with it. So I’d say it is worse.
It’s rather like estimating the number of people on a march and saying 20,521 rather than 20,000. Rounding off, or giving the figure to an appropriate number of significant figures, assists understanding.
It’s a bit naughty – they must know perfectly well that by quoting to one or two decimal places, they give the impression of more accuracy than there really is.
You tend to see it done accidentally most often with unit conversion. Suppose you know the average distance to the moon is approximately 380,000km (you know it to the nearest 10,000km). You them helpfully convert to miles and get 236,121.053 miles, which suggests you know it to the nearest yard. The correct conversion would be 240,000 miles or maybe 236,000.
My point is that if you have a method for counting things, you should report the results that your method gives you. Rounding obscures how (in)accurate your method is, which makes it hard for other people to judge how useful that method is going to be in future.
Also, when you’re reporting changes in results, precision is useful. If you have a figure which stands at 5.4% for January and 4.6% for Feburary, would you round both to 5%? If it falls to 4.4% in March, would you call that a 1% drop to 4%? If you aggressively round everything, it weakens any possibility of gauging the trend and scale of change in results, which is often more important than the precision of the figures involved.
Rob – if your method is genuinely accurate to 0.1% then it makes sense to quote it to that. If it’s an approximation that, in both cases, tells you it’s somewhere around 5%, then you convey your information more accurately by saying both are somewhere round 5% (in other words, though your method gave you different numbers, its sufficiently imprecise that both could easily be the same in realitry).
As for quoting the exact results, there may be occasions where you convey the information better by giving the faux-accurate figure plus a margin of error, and there may be other occasions where it’s better to just round off to a level that conveys the accuracy.
What’s never right, as far as I can see, is to give the impression that you know reality to a higher degree than you do by quoting a precise number without qualification.