The IFS answers… Is the data accurate enough for its calculations?

Yesterday I blogged the first of a set of answers from the IFS about its calculations and the data behind them. Those answers nicely lead on to the next question:

In looking at the impact of the Budget or the Spending Review, the IFS typically quotes the impact on different groups of people to one decimal place. What’s your reason for thinking that your calculations are based on sufficiently accurate data and assumptions to justify reporting them to the nearest tenth of one percent (rather than, for example, the nearest percent)?

Our policy is to publish the exact figures produced by our model, either to the nearest penny (if we are reporting cash amounts) or to the nearest tenth of a percentage point. The Treasury takes a similar approach in its distributional analysis. Obviously our calculations are unlikely to be exactly right all of the time, as they are estimates based on a particular sample of households, and often using some assumptions, but it is not clear what sort of margin of error we should allow.

Two points strike my about this answer from the IFS. First, is a numerical stylistic one. In other fields of study it is the normal practice to only report results to as many decimal places (or significant figures) as the precision of the calculations and data suggests is reliable. That’s one reason why, for example, although opinion poll companies could report their polling data to one or even more decimal places, they round them off to the nearest whole number. Given precision beyond what the data and calculations justify risks misleading and the IFS reports I have seen look to me to be rather light on warnings about reading too much into the apparent accuracy of the calculations.

Second, and more importantly, when added together with yesterday’s answers the gist of what the IFS is saying is ‘Our data and methodology is not perfect and has room for error and interpretation. But it is as good as it can be and anyway uses the same data and methodology as the government’. The latter is a good political debating point – it’s a bit rich for someone to criticise the IFS if other times they are using the same approach etc. – but it doesn’t really help us understand what the actual impact of government policies is likely to be.

Because one answer could be “Sorry, the balance of impact of the government’s policy between different groups just isn’t big enough for us to be able to say with any real certainty whether or not the policy is progressive”. Saying “I don’t know” may be about as politically fashionable as saying “I’ve changed my mind” but that is not a good reason to avoid it.

One organisation in the public eye half-avoids this trap at the moment, and that is the Bank of England. For its forecasts it produces a central projection but also a range of likely outcomes either side of that central projection because, quite rightly, it recognises that making economic predictions is a rather inaccurate mix of art and science. Perhaps this approach of up-front acknowledgement of bands of error and uncertainty is one the IFS could adapt rather more in future?

Check back tomorrow for the next in this series.

Read more by or more about .
This entry was posted in News.
Advert

21 Comments

  • Anthony Aloysius St 23rd Nov '10 - 12:44pm

    “In other fields of study it is the normal practice to only report results to as many decimal places (or significant figures) as the precision of the calculations and data suggests is reliable. That’s one reason why, for example, although opinion poll companies could report their polling data to one or even more decimal places, they round them off to the nearest whole number.”

    ?????

    The sampling error in most opinion polls is of the order of 1-2%. If your claim were accurate, they would round to the nearest 10%!

  • AAS – but if they did round to 10% we would at least be confident of what ballpark we are in – as opposed to appearing to nail something to the nearest 0.1%, which conveys the slightly misleading impression that you have nailed on data.

  • Anthony Aloysius St 23rd Nov '10 - 2:12pm

    Mrs B

    Read it again.

  • paul barker 23rd Nov '10 - 3:55pm

    AAS, to be fair to the polls, you could argue that are taking a sensible compromise approach.
    Polls are a product, sold to News Media so we can excuse them for exagerating the changes & not reminding us how illusory they may be. The IFS dont have that excuse.

  • Anthony Aloysius St 23rd Nov '10 - 4:04pm

    Of course what _really_ happens in other fields is not that they round figures to the number of significant figures justified by their accuracy, but that they give an indication of the estimated error. I’m sure that the IFS’s difficulty with doing this is – as they indicate – thst they are unable to estimate how accurate their data are.

  • If they are unable to estimate how accurate heir data is then they should stop giving a misleading impressionof its accuracy by giving figures to .1%

  • Anthony Aloysius St 23rd Nov '10 - 4:40pm

    The problem is – why stop there?

  • In their reply, the IFS say the data they use comes from the FRS. The FRS survey is produced for the government. In other words, its the governments own data.

    A robustness assessment of the FRS survey is covered here:
    http://statistics.dwp.gov.uk/asd/hbai/frsrar2.pdf

    Improvements and recommendations are covered here:
    National Statistics Quality Review of Income Statistics,

    Office for National Statistics – Protocols on Statistical Integration and Classification are covered here:
    http://www.ons.gov.uk/about-statistics/ns-standard/cop/protocols/index.html

    The standards to which they refer are covered here:
    National Statistics Statement of Compliance and links to Code of Practice
    http://www.statistics.gov.uk/about_ns/cop/downloads/ONS_Compliance_Statement.doc

  • The National Statistics Quality Review of Income Statistics link didn’t work. Here it is:

    http://www.statistics.gov.uk/methods_quality/quality_review/social.asp

  • Tony Dawson 23rd Nov '10 - 6:00pm

    As long as they publish confidence limits, there’s nothing really wrong with publishing forecasts of subsets of data ‘accurate’ own to the decimal point figure. What IS totally wrong is either publishing themselves, or allowing others to publish without issuing strong corrections, conclusions based upon timeline changes in the predicted data.

  • The IFS also say they use the data from the LCFS survey . The LCFS survey says it is produced to the professional standards set out in the Code of Practice for Official Statistics:

    http://www.statistics.gov.uk/STATBASE/Product.asp?vlnk=10336

    The National Statistics Statement of Compliance and links to the Code of Practice are same as above.

    I’ve looked through some of the other reports. They seem to analyse percentages to one decimal place. An example is their own Robustness Assessment Report which goes back ten years or more and shows the data analysed to one decimal place. (pages 11/12).

    The IFS are using the government’s own datasets for their reports. These appear to have been produced over many years, and have been improved from time to time in a way that allows year-on-year comparisons and trend anaysis, if required.

  • @Mark Pack

    You ask: is the data accurate enough for [the IFS] calculations?

    The answer: it is the same data government uses to produce policies.

  • Anthony Aloysius St 23rd Nov '10 - 6:53pm

    “In such cases the correct approach is to give all results to one or two ‘significant figures’”

    The problem is that with a factor of three between estimates of income and expenditure in the IFS’s lowest expenditure decile – which the IFS, assuming these numbers should be equal, attributes to an unexplained error in the estimate of expenditure – some of the data might be down to zero significant figures …

  • Andrew Suffield 23rd Nov '10 - 8:31pm

    The problem is – why stop there?

    Well yes. Or rather, the problem is in people stopping there.

    If you have no idea how accurate your numbers are then your numbers are worthless and should not be published.

  • Anthony Aloysius St 23rd Nov '10 - 9:07pm

    George

    Why on earth are you going on about the use of income deciles? We aren’t discussing the use of income deciles. The problem is with the accuracy of the data – particularly the expenditure data, according to the IFS.

  • Anthony Aloysius St 23rd Nov '10 - 9:26pm

    “If you have no idea how accurate your numbers are then your numbers are worthless and should not be published.”

    That’s actually very much what I said when the problems associated with the VAT analysis became apparent. But unfortunately that wonderful finding that the VAT increase could be viewed as progressive was just too seductive for the loyalists to relinquish. Funnily enough, one of them quoted it on another thread only a few minutes ago.

  • I don’t think anyone ever reads an IFS report and thinks that when the IFS say that decile 2 are £3.47 better off it means that everyone in that decile is exactly £3.47 better off. But if we rounded that to £3, and the next decile is £3.52 better off, we would round that to £4, making it look as though there is a big difference between the effect on deciles 2 and 3, whereas there is not. The best thing to do is to join the IFS, and read their full reports.

  • Well if they used 2 sf then that would become £3.50 for both sets and given they have no idea how accurate their figures are would nt that be a truer reflection that artificiality using 2dp when they apparently don’t have the accuracy to justify this?

  • There are few things more pitiful than watching Cleggmaniacs still trying to blame the messenger.
    The poor, disabled and vulnerable are going to suffer under the massive cuts proposed so nitpicking with the IFS over margins of error isn’t going to work any more than blaming the pledge will for breaking the Fees promise.
    Spin like this isn’t going to cut it on the doorstep and there are going to be some very hard questions asked of those who complacently keep ignoring everything but the prevailing narrative spin from the top of the Party.

Post a Comment

Lib Dem Voice welcomes comments from everyone but we ask you to be polite, to be on topic and to be who you say you are. You can read our comments policy in full here. Please respect it and all readers of the site.

To have your photo next to your comment please signup your email address with Gravatar.

Your email is never published. Required fields are marked *

*
*
Please complete the name of this site, Liberal Democrat ...?

Advert



Recent Comments

  • Alex Macfie
    Pandering to anti-immigrant sentiment, or any other far-right narrative, will not win the progressive tendency any votes. People inclined to vote for populism w...
  • Peter Martin
    @ Chris Moore, ".....we need to ....persuade left behind voters that their interests lie in a liberal alternative to right-wing populism." The ...
  • AlexB
    When you listened to his incoherent ramblings on the last televised debate, you wonder what he is going to be like mentally when he's 82 in four years time. Wor...
  • Jenny Barnes
    Maybe it was a case of the lesser of two evils still being too evil to vote for....
  • John Waller
    Patrick Cockburn writes a scary article in the iNews today. Well worth reading. The 2020s look increasingly like the 1930s, the so-called “devil’s decad...