Straw man statistics

If they can get you asking the wrong questions, they don’t have to worry about answers (Thomas Pynchon)

May 21, 2021 Update:  A Scottish government agency breached freedom of information rules by refusing to reveal how many people had died of Covid in each of the country’s care homes. Full BBC report HERE See also Number of Deaths in Care Homes: Scottish Information Commissioner report HERE

On October 28, Nicola Sturgeon announced that an investigation by Public Health Scotland had revealed that “hospital discharges were not found to have contributed to a significantly higher risk of an outbreak in care homes.”

Oh dear. Rule one for any politician discussing anything containing data is never, never employ the word “significantly”. It only draws unwanted attention.

So, here is some unwanted attention.

The investigation to which the First Minister was referring was described in the report Discharges From NHS Scotland Hospitals To Care Homes. The investigation was or should have been initiated because:

• Between March 1 and April 21 there were 3599 discharges from hospitals to care homes of which 82% had not been tested for COVID19
• This led to understandable concerns that these untested discharges had contributed to the large number of coronavirus-related deaths that took place in care homes
• These concerns had been heightened by the Health Secretary announcing (in April) “success” in transferring large numbers of delayed discharge patients to care homes and the community
• It was only some two weeks after this that a fairly exhaustive COVID-19 testing regimen was introduced for patients discharged from hospital to care homes

The report’s authors have been at pains to explain both their conclusions and how they were reached. But there are some jarring notes and unfortunately these turn out to have some “significance”.

Testing the question

The investigation is described as a “statistical modelling” exercise. It is a feature of this pandemic that everyone describes their work as “modelling”. (And every big number as “exponential”).  The problem is that in this case it is just not true. Modelling is used in the main to predict a range of possibilities about what might happen in the future based on different assumptions. But what was intended here was an analysis to answer a specific question about something that has already happened – i.e. good old-fashioned hypothesis testing.

In an analysis of this kind it is vital to state in advance the hypothesis you are wishing to test because if you don’t, no-one can be at all clear what any result actually means. The problem is that, search as you will in the report, you will not find anything spelt out which looks remotely like a hypothesis. Perhaps this is not surprising since the commission given by the Cabinet Secretary for Health was to ”carry out work to identify and report on discharges from NHS hospitals to care homes during the first wave of the COVID-19 Pandemic” and gave not the slightest hint of any existing cause for concern which could be summarised in a hypothesis for test.  But for any reader trying to evaluate the conclusions it is essential to construct the hypothesis implied by the analysis. So, let’s have a go.

An endearing feature of statistical inference is that the hypothesis that must be constructed in a so-called Null Hypothesis: in effect the exact opposite of what you think you are testing.  So here the Null Hypothesis seems to be “that care homes which receive one or more patients discharged from hospital between March 1 and May 31 are no more likely to have an ­outbreak between March 1 and June 21 than care homes that did not receive one or more discharged patients”.

Outbreaks or deaths?

It may seem pedantic to insist on the Null Hypothesis being defined in this way. But stop a moment and consider the reasons why the study came about. The concern was or should have been the possibility that discharging untested patients from hospitals to care homes up to April 21 resulted in an increase in deaths in care homes. But the implied Null Hypothesis relates to discharges in the entire period up to May 31 and to “outbreaks” not deaths.

Now, we can assume that testing was introduced on April 21 for the sound clinical reason of reducing possible risk. So, if we test a hypothesis relating to discharges for the whole period from March 1  to May 31, we may be combining data from higher and lower risk periods. Our concern should be solely with the initial potentially high-risk period when most patients were discharged untested. Chucking in the data from a potentially lower risk period could effectively dilute any effect we are trying to detect.

Trawl through the media or accounts of proceedings in the Scottish Parliament and you will find that the concerns expressed were largely about the possibility that the discharge of untested patients to care homes resulted in excess deaths in care homes rather than outbreaks. Surely you might say deaths or outbreaks amount to the same difference. But no. Outbreaks in the period studied varied in size from one to 96 and there must be some correlation between the size of an outbreak and the number of residents in a care home – you obviously can’t have an outbreak of 96 in a care home of just 20 residents. So, before we look at a single figure, we know that outbreaks are a poor proxy for our central concern which is deaths.

None of this can be construed as a criticism of the way in which the calculations were carried out or of the considerable work required to collate data from multiple sources. But the fact remains. If you ask the wrong question you cannot expect to obtain statistical results which give meaningful insights about the true question to which you are seeking an answer.

What’s significant?

So why go any further?  If the question was wrong, is there any need for further examination? Well, yes there is. Because that word “significantly” was used to imply that the concerns which provided the impetus for the study were unfounded.

In the study the possible effect of hospital discharges on the number of outbreaks is expressed in the form of a “hazard ratio”. If the hazard ratio for a particular variable is one or less this means that no excess risk has been detected. Above one indicates excess risk. 

Because a number of variables might affect the number of outbreaks in care homes, a multivariate analysis was carried out which adjusts the hazard ratio for each variable to take account of the contribution of the other variables.  These included care home type and care home size. Curiously, no account was taken of the possibility of lower risk after April 21 when testing of hospital discharges became extensive. (A variable of this kind was included in a comparable Welsh study. But it’s a very inefficient way of conducting an analysis when compared to just running the analysis for the period of concern up to April 21^st.)

Not surprisingly, the multivariate analysis showed that care home size correlates with the risk of an outbreak – common sense suggests that risk is likely to be greater if you have more residents, more visitors and more staff.  So, the hazard ratio for discharged patients has been adjusted for care home size in particular.

The hazard ratio is not expressed as a single figure but as a range to account for random variation. To achieve this, confidence limits are constructed which provide an upper and lower value for the hazard ratio. The statistical convention is to construct 95% confidence limits. If we say that the true value of the hazard ratio falls between the upper and lower confidence limits, we will be right 95% of the time.

So, what was the result of the calculation in the current study? After the adjustments had been applied to take account of other variables and care home size in particular, the value of the hazard ratio in respect of patients discharged from hospital was calculated to lie between the 95% confidence limits of 0.96 (no extra risk) and 1.54 (extra risk) with the so called best estimate  being the mean of the two values i.e. 1.21.

The result is described as not being statistically significant at the 95% level because a value of 1 or less (i.e. no increased risk) is possible. But values suggesting greatly increased risk are also possible. In other words, the failure to achieve the 95% level of statistical significance does not mean that an effect does not exist. In fact, unlike the First Minister, the authors do stress this point and indeed identify 1.21 as the best estimate of the hazard ratio.

It is important to understand that when the range of confidence limits is very wide (as here) it does tend to weaken the value of any observed result. “Intervals that are very wide indicate that we have little knowledge about the effect and this imprecision affects our certainty in the evidence, and that further information would be needed before we could draw a more certain conclusion,” a key training manual says. So, the very best that can be said is that the study as designed produced an inconclusive result.

Can any conclusions be drawn?

In the period up to April 28 (as the authors point out seven days should be allowed for any effect of a discharge to be identified as an “outbreak”) there were 299 outbreaks  which resulted in 1629 deaths (measured up to week beginning 18  May to allow for the possible  interval between infection and death i.e.  an average of 5.4 deaths per outbreak.) (see ,here).

Assume for the sake of argument that, based on outcomes, the hazard ratio resulting from the discharge of patients to care homes was the best estimate of 1.21 identified in the study.  Given the flaws in the study it is not possible to try to extrapolate from this figure to produce any estimates of the consequences of the discharge policy. But it should be noted that, with an average number of deaths per outbreak of 5.4, an increase in the risk of an outbreak of as little as 20% could equate to a considerable number of deaths.

But it is of course quite improper to use the average number of deaths per outbreak in this way. As has been insisted, the number of cases in each outbreak and hence the number of deaths will be influenced by care home size – the larger the care home the larger the potential size of the outbreak. (It is reported that a single outbreak in a care home resulted in 24 deaths).  At the same time, it is highly likely that the chance of being admitted to a particular care home is also affected by its size. So, using outbreaks as a proxy for deaths and then adjusting for care home size, potentially masks any true effect and cannot be justified.

And finally…

In conclusion, the study which it was claimed demonstrates that “hospital discharges were not found to have contributed to a significantly higher risk of an outbreak in care homes” in truth shows no such thing.  Even on the flawed assumptions in the study it is entirely possible that there was an appreciable number of excess deaths associated with the discharge of untested patients.

But the more fundamental objection is that the study tells us nothing about the real question of concern: Did the discharge of untested patients to care homes result in an increase in deaths? The truth is that we are really none the wiser.

Data on coronavirus deaths in each care home are available and reported to the Care Inspectorate. For this reason, it does seem strange that outbreaks were chosen as the measure to be used in the study.  Given the quality of data linkage systems in Scotland, a study which tracked individual residents and discharged patients over time, although more complex, ought to have been feasible.

 And there is a further “significant” conclusion to be drawn. Be very, very wary of using the word “significantly” especially when results are, at best, inconclusive and, at worst, entirely misleading. And don’t respond to a question with the answer to a different question.

Featured image of Nicola Sturgeon at 28 October news briefing via Scottish Government Flickr, CC BY-SA 2.0; Image of Berelands Care Home in Prestwick (16 deaths) and of Castle View Care Home in Dumbarton (8 deaths) via STV.com

Further reading: ‘Number of Deaths in Care Homes: Scottish Information Commissioner May 2021 Nick Kempe, Asking the right questions, Source, 9 November; Care home outbreaks in one Scottish region (Lothian), The Lancet, October 2020; James McEnaney, Health minister wanted to stop FOI, The Ferret, 12 November