Statistical amnesia

May 21, 2021 Update:  A Scottish government agency breached freedom of information rules by refusing to reveal how many people had died of Covid in each of the country’s care homes. Full BBC report HERE  See also Scottish Information Commission report: Number of Deaths in Care Homes HERE 

In this pandemic there has been an unprecedented obsession with “the data”.

As the pandemic has developed, we have become used to the “big numbers” such as “70,000 deaths in the UK” or £100 billion for the misconceived “Operation Moonshot”. And in the process, little by little, we have become insulated from the true tragedy behind these numbers. Who really stops to think what 500 COVID-19 deaths reported in a single day really means in terms of human distress, the waste of active years of life and long-term detriment to health?

Data has become the stick used to beat the drum about relative success or beat the political opponent over the head.

When I was invited to submit another piece to summarise what the data can tell us now as the year closes, I had to think long and hard whether I really wanted to submerge again in the morass of data that lurks now on the internet. The truth is that although the sheer quantity of data on a single topic is almost certainly unsurpassed in terms of volume, it has become harder and harder to have any faith in its quality. Look at the small print on a data set and you will often find the word “provisional”. Go back a week later and you will find that not only has the data been revised but also that the original data has been obliterated together with any opportunity for an audit trail.

The constant retrospective updating of data could be justified if the only purpose in its collection was to provide a historical record of the pandemic. But we are constantly being told that decision-making is based on the science and hence the data. So, the current quality of the data really matters as does informed analysis untainted by political imperatives.

An historical perspective

The constant updating of the data should mean that the further back we go in time the better the data quality. But there seems to be no time to reflect on what the experience of the last few months might tell us about the future management of the disease. The only data that attracts attention is that for the latest day or week even though by definition this is the data with the least quality.” Thus just in the last week the headlines screamed “Nicola Sturgeon defends handling of Covid as figures show Scottish deaths compare badly to England” but no one it seems had the time to put this into any kind of historical perspective.

The first question that needs to be asked is: is it true that Scottish coronavirus deaths do compare “badly” to England’s. The problem is that the figures quoted in the newspaper and the original television interview relate to week-on-week comparisons between Scotland and England where the weekly period have different start dates in the two countries. What is much more revealing is a comparison between the two countries based on data for the actual date of registration (Figure 1).

This shows that from mid-October to mid-November the death rates in Scotland were indeed consistently higher than was the case for England. But what the data also shows is that there appeared to be a dramatic change in the ratio sometime in early October

Questioned about the data, Nicola Sturgeon fell back on the well-worn trope: “We are still in the teeth of this pandemic, so it is premature for any country to be declaring victory or assessing performance”. This was to put it mildly a disappointing response and must have been particularly so for anyone who had been prepared to put aside political differences to praise the Scottish response to the pandemic.

It contrasts strangely with the First Minster’s claims earlier in the year that Scotland’s COVID19 rates were one fifth of those in England – a claim for which she was reprimanded by the Office for Statistical Regulation even though there was some evidence to support at least the general thrust of the claim if not the actual quoted ratio (Figure 2).

Putting it bluntly, this kind of statistical amnesia will seem a little tawdry to anyone who invested belief and political capital in the First Minister’s approach.

The real question

Although it is hard to escape the conclusion that a double standard has been applied in considering data from two different time periods, this is not the real cause for concern about the First Minister’s comments. The appropriate response to any question about the relative death rates in England and Scotland was surely to make a commitment to try to understand why from a relatively low base Scotland’s deaths suddenly increased to a level not seen since May. This is illustrated starkly in Figure 3 which shows the moving average of deaths in Scotland by date of registration. (The dates shown on this graph are those 21 days before the date of registration to give some approximation of the date when the cases which led to the deaths occurred).

What this graph demonstrates is that there was a pivot point that occurred in the middle of September which preceded the surge in COVID 19 deaths. It will not be a coincidence that this pivot point coincides with the time when the Scottish Government switched back from the hopeful objective of “eliminating” the virus to the more familiar territory of virus “suppression”. Looking back it was entirely wrong to offer the hope of eliminating the virus in Scotland when for whatever reason the necessary policies and strategies to achieve this were never in place.

By contrast to Scotland the rise in deaths in England shows a different pattern with no discernible pivot point but rather a sustained rise from late August. The absence of a pivot point is perhaps not surprising given that COVID-19 was already on the rise in several English regions.

However, what is also apparent is that whereas at the end of October in England the peak in cases leading to death had yet to be achieved, the peak in Scotland may have occurred perhaps some two weeks earlier. So, it is entirely possible that similar factors could have been in play in Scotland and England which led to a rise in cases leading to deaths albeit that such a factor might have come into play later in England.

In summary it would appear:

• That in the middle of September there were some circumstances which appeared to trigger a rapid increase in Scotland in fatal cases
• Although there was no similar pivot point in England it appears that the peak in deaths in England might have been reached at least two weeks after that in Scotland.
• Understanding what might have triggered the sudden rise in mid-September is of importance in the future management of the pandemic

There are no data sets which might provide direct answers to the question of what might have triggered the sudden rise in Scotland in September.  But if all that is left is speculation such speculation is by no means idle. If I can think of at least one prime candidate for an event which might have triggered the September rise in Scotland, then I am sure that others will be able to come up with the same, similar or different suggestions. Speculation of this kind is valid if it leads to investigations which at minimum eliminate possible explanations. And speculation is our only recourse if those with the power to do so refuse to engage with the data

The comments page is there. Why not use it to offer your suggestions. If you do so you will not be providing a definitive explanation, but you will be offering a hypothesis and challenging others to prove your hypothesis wrong.

Further reading: Number of Deaths in Care Homes: Scottish Information Commissioner report May 2021 Questions over Sturgeon’s strategy, FT; Sturgeon defends handling, BBC;  Scaling COVID-19 against inequalities, Gerry  McCartney et al, BMJ