Yesterday I blogged the first of a set of answers from the IFS about its calculations and the data behind them. Those answers nicely lead on to the next question:
In looking at the impact of the Budget or the Spending Review, the IFS typically quotes the impact on different groups of people to one decimal place. What’s your reason for thinking that your calculations are based on sufficiently accurate data and assumptions to justify reporting them to the nearest tenth of one percent (rather than, for example, the nearest percent)?
Our policy is to publish the exact figures produced by our model, either to the nearest penny (if we are reporting cash amounts) or to the nearest tenth of a percentage point. The Treasury takes a similar approach in its distributional analysis. Obviously our calculations are unlikely to be exactly right all of the time, as they are estimates based on a particular sample of households, and often using some assumptions, but it is not clear what sort of margin of error we should allow.
Two points strike my about this answer from the IFS. First, is a numerical stylistic one. In other fields of study it is the normal practice to only report results to as many decimal places (or significant figures) as the precision of the calculations and data suggests is reliable. That’s one reason why, for example, although opinion poll companies could report their polling data to one or even more decimal places, they round them off to the nearest whole number. Given precision beyond what the data and calculations justify risks misleading and the IFS reports I have seen look to me to be rather light on warnings about reading too much into the apparent accuracy of the calculations.
Second, and more importantly, when added together with yesterday’s answers the gist of what the IFS is saying is ‘Our data and methodology is not perfect and has room for error and interpretation. But it is as good as it can be and anyway uses the same data and methodology as the government’. The latter is a good political debating point – it’s a bit rich for someone to criticise the IFS if other times they are using the same approach etc. – but it doesn’t really help us understand what the actual impact of government policies is likely to be.
Because one answer could be “Sorry, the balance of impact of the government’s policy between different groups just isn’t big enough for us to be able to say with any real certainty whether or not the policy is progressive”. Saying “I don’t know” may be about as politically fashionable as saying “I’ve changed my mind” but that is not a good reason to avoid it.
One organisation in the public eye half-avoids this trap at the moment, and that is the Bank of England. For its forecasts it produces a central projection but also a range of likely outcomes either side of that central projection because, quite rightly, it recognises that making economic predictions is a rather inaccurate mix of art and science. Perhaps this approach of up-front acknowledgement of bands of error and uncertainty is one the IFS could adapt rather more in future?
Check back tomorrow for the next in this series.
21 Comments
“In other fields of study it is the normal practice to only report results to as many decimal places (or significant figures) as the precision of the calculations and data suggests is reliable. That’s one reason why, for example, although opinion poll companies could report their polling data to one or even more decimal places, they round them off to the nearest whole number.”
?????
The sampling error in most opinion polls is of the order of 1-2%. If your claim were accurate, they would round to the nearest 10%!
AAS – but if they did round to 10% we would at least be confident of what ballpark we are in – as opposed to appearing to nail something to the nearest 0.1%, which conveys the slightly misleading impression that you have nailed on data.
Mrs B
Read it again.
AAS, to be fair to the polls, you could argue that are taking a sensible compromise approach.
Polls are a product, sold to News Media so we can excuse them for exagerating the changes & not reminding us how illusory they may be. The IFS dont have that excuse.
Of course what _really_ happens in other fields is not that they round figures to the number of significant figures justified by their accuracy, but that they give an indication of the estimated error. I’m sure that the IFS’s difficulty with doing this is – as they indicate – thst they are unable to estimate how accurate their data are.
If they are unable to estimate how accurate heir data is then they should stop giving a misleading impressionof its accuracy by giving figures to .1%
The problem is – why stop there?
In their reply, the IFS say the data they use comes from the FRS. The FRS survey is produced for the government. In other words, its the governments own data.
A robustness assessment of the FRS survey is covered here:
http://statistics.dwp.gov.uk/asd/hbai/frsrar2.pdf
Improvements and recommendations are covered here:
National Statistics Quality Review of Income Statistics,
Office for National Statistics – Protocols on Statistical Integration and Classification are covered here:
http://www.ons.gov.uk/about-statistics/ns-standard/cop/protocols/index.html
The standards to which they refer are covered here:
National Statistics Statement of Compliance and links to Code of Practice
http://www.statistics.gov.uk/about_ns/cop/downloads/ONS_Compliance_Statement.doc
The National Statistics Quality Review of Income Statistics link didn’t work. Here it is:
http://www.statistics.gov.uk/methods_quality/quality_review/social.asp
As long as they publish confidence limits, there’s nothing really wrong with publishing forecasts of subsets of data ‘accurate’ own to the decimal point figure. What IS totally wrong is either publishing themselves, or allowing others to publish without issuing strong corrections, conclusions based upon timeline changes in the predicted data.
The IFS also say they use the data from the LCFS survey . The LCFS survey says it is produced to the professional standards set out in the Code of Practice for Official Statistics:
http://www.statistics.gov.uk/STATBASE/Product.asp?vlnk=10336
The National Statistics Statement of Compliance and links to the Code of Practice are same as above.
I’ve looked through some of the other reports. They seem to analyse percentages to one decimal place. An example is their own Robustness Assessment Report which goes back ten years or more and shows the data analysed to one decimal place. (pages 11/12).
The IFS are using the government’s own datasets for their reports. These appear to have been produced over many years, and have been improved from time to time in a way that allows year-on-year comparisons and trend anaysis, if required.
@Mark Pack
You ask: is the data accurate enough for [the IFS] calculations?
The answer: it is the same data government uses to produce policies.
I agree that the Bank’s use of ranges is less unscientific than the IFS’s entirely bogus accuracy, which betrays the unscientific basis of economics, an inverted pyramid of theory based on a selection of uncontrolled observational evidence. My impression is that the IFS’s respondent didn’t really understand what wrong with their approach.
However it is important to realise that the Bank’s ranges mean less than meets the eye, as they are based on assumption about the underlying probabilities of outcomes [usually that they are distributed ‘normally’] that don’t apply in the real world. In science, these ranges would be 95 % confidence intervals [CIs] – the range within which there is 95 % chance that the correct value lies. But as we don’t know the underlying probabilities, in truth nobody knows what the CIs should be.
In such cases the correct approach is to give all results to one or two ‘significant figures’ [sfs]: 45.5 is 50 to 1 sf and 46 to 2 sf. 2 sf = 1 part in 100: any claim to be able to estimate any political or economic statistic better than that is obviously false.
“In such cases the correct approach is to give all results to one or two ‘significant figures’”
The problem is that with a factor of three between estimates of income and expenditure in the IFS’s lowest expenditure decile – which the IFS, assuming these numbers should be equal, attributes to an unexplained error in the estimate of expenditure – some of the data might be down to zero significant figures …
I don’t really have a problem with the IFS giving data to one decimal place. I just want them to be up-front about the inaccuracies they know about.
In the guts of their literature, the IFS acknowledge the problem of using income deciles, but they don’t mention the problem in their press releases. Instead, the press releases quote conclusions based on income deciles without qualification.
As press releases are the only documents that most journalists read, however many qualifications they put into the main document, the effect will be newspaper headlines based on flaky data.
In contrast, the Treasury, when displaying data using income deciles, were very explicit about the problem with doing so:
“It should be noted that the bottom decile contains many households with temporarily low incomes, for whom income based analysis, as opposed to expenditure based analysis, may not give an accurate picture of living conditions. In this decile, around 40 per cent of households contain an adult that is self employed or a student. While some of these households will have permanently low incomes, many will not. In contrast, in the second decile, only around 20 per cent of households contain an adult in one of these groups.”
Well yes. Or rather, the problem is in people stopping there.
If you have no idea how accurate your numbers are then your numbers are worthless and should not be published.
George
Why on earth are you going on about the use of income deciles? We aren’t discussing the use of income deciles. The problem is with the accuracy of the data – particularly the expenditure data, according to the IFS.
“If you have no idea how accurate your numbers are then your numbers are worthless and should not be published.”
That’s actually very much what I said when the problems associated with the VAT analysis became apparent. But unfortunately that wonderful finding that the VAT increase could be viewed as progressive was just too seductive for the loyalists to relinquish. Funnily enough, one of them quoted it on another thread only a few minutes ago.
I don’t think anyone ever reads an IFS report and thinks that when the IFS say that decile 2 are £3.47 better off it means that everyone in that decile is exactly £3.47 better off. But if we rounded that to £3, and the next decile is £3.52 better off, we would round that to £4, making it look as though there is a big difference between the effect on deciles 2 and 3, whereas there is not. The best thing to do is to join the IFS, and read their full reports.
Well if they used 2 sf then that would become £3.50 for both sets and given they have no idea how accurate their figures are would nt that be a truer reflection that artificiality using 2dp when they apparently don’t have the accuracy to justify this?
There are few things more pitiful than watching Cleggmaniacs still trying to blame the messenger.
The poor, disabled and vulnerable are going to suffer under the massive cuts proposed so nitpicking with the IFS over margins of error isn’t going to work any more than blaming the pledge will for breaking the Fees promise.
Spin like this isn’t going to cut it on the doorstep and there are going to be some very hard questions asked of those who complacently keep ignoring everything but the prevailing narrative spin from the top of the Party.