No more political games

So, the Statistics Regulator (UKSA) has criticised Nicola Sturgeon for stating that prevalence of COVID-19 in England is five times higher than in Scotland. The criticism was that using unpublished data without attribution lacked transparency and the comparisons made were not comparing like with like.

In normal times there would be few impartial analysts who would rush to the defence of the Scottish Government for any “misuse” of statistics, especially if the motive for quoting the statistics was party political. These are not, however, normal times and there is something far more important at stake than narrow party advantage.

There is little doubt that there are differences in the way in which data are collected across the four home nations and that the comparison in question did fall into the trap of comparing apples with pears. But, for the month of July at least, the pears were very small, and the apples were quite enormous.  Whilst it is impossible to quantify the exact ratio of prevalence of Covid-19 between England and Scotland on any examination of published data for July, that ratio must be very large and might well be in excess of five.

Comparing the recent prevalence between England and Scotland is akin to contrasting the height of a multi-story office block with that of a bungalow and getting criticised for doing so without using a ruler. The office block may be four times higher than the bungalow or six times. But no one can seriously suggest that there is not a massive difference.

It’s all in the numbers  

Of course, if Sturgeon’s intention was to compare prevalence in Scotland and England over the whole period of the pandemic she would have been on very shaky ground. All the evidence suggests that Scotland’s performance in March, April and May 2020 ran a close second to that of England. Both countries had disastrous death tolls which owed much to the delays in imposing lockdowns and a “Protect the NHS” policy which identified maintaining hospital capacity  as the primary objective rather than suppressing the virus and saving lives. (For some reason pictures of overcrowded hospitals seems to make worse PR than news that tens of thousands of people are dying). It is only since the end of May that the divergence in performance between the two countries has become apparent

Now, you might imagine that there is little point in complaining about the regulator’s intervention. Just take the criticism and move on. And normally that would be good advice, especially since once the referral by the Scottish Conservatives had been made, the UKSA was required to rule.

But the problem now is that the future management of the virus depends heavily on understanding the level of prevalence in different areas and how it is changing.  Preventing or suppressing the transmission of the disease from areas with high prevalence to areas with low prevalence is at the heart of any elimination strategy.  For this reason, the Conservatives do no one any service by suggesting that the motive for drawing attention to probable differences in prevalence is entirely political. We all desperately need to know where this pandemic is heading, and prevalence comparisons, however imperfect, are an essential first step. The referral to the regulator looks more like an attempt at information suppression than genuine concern for statistical integrity.

English stats

Perhaps it might be helpful if the Statistical Regulator were to reflect briefly on the dog’s dinner which is COVID-19 Statistics in England. A prime duty of statisticians is, first, to make data as accessible as possible and, second, to do their very best to make meaningful interpretations of the data. Telling people that comparisons are not possible is an abdication of responsibility and a tacit acceptance that nothing can be done to rationalise the glorified mess which constitutes the English COVID-19 data base. Ask anyone who has tried to work with the data, and they will tell you the same thing. Data sources appear and disappear overnight, data are revised retrospectively without any proper accounting and data are aggregated for no apparent reason in ways which obscure all meaning.

To make matters worse some of the “official” data released just muddy the water. For some reason best known to itself, ONS in England compares the weekly number of “excess deaths” with the five-year average for the same time of year. In doing so it seems to blithely ignore a fallacy in the use of these statistics which has been known since people first began tracking flu pandemics. Put simply, any large-scale epidemic will regrettably tend to hasten the deaths of very vulnerable people who would perhaps have died later in the year. For this reason, once the peak of an epidemic has passed, there will be a consequential fall in excess deaths in later months.  This makes recent comparisons with the five-year average largely meaningless.

It must be recognised that some degree of chaos is to be expected when data start to be collected for the first time in a new situation. But the statistical chaos with which we are now confronted goes far beyond anything seen in recent times. The problem goes far beyond the very real problems of data definition and consistency.  Someone needs to take responsibility for extracting some sort of meaning from the muddle. And in England that responsibility seems to be being shirked.

Fresh lockdowns

A concrete example. Both Aberdeen and Greater Manchester have been locked down because of a reported rise in cases. Because for Aberdeen the data on the cluster comes from a single data source based on a single contact tracing system, the quoted data are probably relatively reliable. In Greater Manchester, however, it is quite impossible to reliably quantify the scale of the excess because the published data are aggregated from different sources, up to recently included double counting (that may still be going on), and lump together stats on primary cases with the product of two parallel contact tracing systems – the local public health system which does the lion’s share of the work and the privatised system which consumes the lion’s share of the money.

What must be understood is that In Scotland, after a poor start which cost a lot of lives, someone somewhere remembered and embraced our public health heritage and built systems accordingly. In England by comparison there has been an ideological objection to public health medicine since the time of the coalition government in 2010 which has informed every aspect of the English response to the pandemic. That difference of approach is at the heart of the recent divergence in performance between Scotland and England.

Moving on

So, what further response should the Scottish Government make to the regulator’s criticism. Apology over and above that already made is really not required. Far better would be for the Scottish Government to set up its own dedicated team of statisticians to try to make some sense of the muddled English data and crucially demand access to data which cannot be obtained from published sources.

Let’s first give this dedicated team an overarching objective. Some people are suggesting that a second surge in cases will occur, perhaps in December. How they work this out this is anyone’s guess, but we surely have a considerable vested interest in knowing if it is going to be earlier than December (some people’s guess is a lot earlier). One thing we do know, however, is that a major driver of the original surge in cases was people carrying the infection into Scotland from other countries. So, we really do need to know as soon as possible if things start kicking off in our nearest neighbour, England, so that we can avoid the kind of mistakes that were made in February. Let’s make that the primary objective for the team.

But now we are faced with an immediate problem. Many of the indicators we can use to identify a surge in cases have a built-in delay factor. Thus, deaths in England are apparently inching down now. If they start to rise again this might indicate the beginning of a problem. But because daily deaths occur on average perhaps 20 or more days after infection, by the time we appreciate that a new surge is beginning it might already be too late. (This is exactly what happened at the end of February). ONS weekly data experience an even greater delay factor. 

As an alternative, we might use the English case data. But as the independent SAGE has already stressed these data are deeply flawed not least because of their very incomplete nature. Of equal importance is that the data are a composite of so-called Pillar 1 tested cases (those carried out in Public Health England (PHE) labs and NHS hospital settings) and Pillar 2 tests (via the community, mainly as part of the UK Government’s test and trace programme). In recent weeks Pillar 2 cases have shown a considerable rise whilst Pillar 1 has held relatively steady.

Without proper denominators for the Pillar 2 tests such as the number of cases tested and numerators  such as the number testing positive, it is impossible to eliminate the possibility that this difference is due in part at least to increased Pillar 2 testing in ‘hotspot’ areas. If the reported rise in aggregate case data for England is real, there should be a commensurate rise in deaths beginning after, say, a 20-day delay i.e. about now. If this does not occur, it will raise further questions about case reporting in England.

Triage data

There is one data source which is largely overlooked which would bear more detailed examination. This is the so-called triage data for England which record the number of 111 and 999 contacts (not people) which are recorded as possible Covid-19 cases. These data have been in systematic decline as more people use the website to report symptoms but there are still currently about 1300 such reports each day. Another problem is that with the fragmentation of health care bodies in England the data related to Clinical Commissioning Group areas can cross local authority boundaries.

But, despite these limitations, the triage data have shown increases ahead of lockdowns in several local authority areas where cases have been said to rise. Thus, for example, the number of triage and 999 contacts with Covid-19 symptoms in Leicester rose for each seven-day period from June 2 to June 22. (See table). Rather worryingly, the number of contacts fell in the next seven days up to and including the date of Lockdown on 29^th June and this might suggest delays in the track and trace system

Number of Triage and contacts with possible Covid-19 symptoms in Leicester City in the 28 days up to Lockdown on 29^th June 2020

Source NHS Pathways Covid-19. NHS Digital

The triage data, unlike the track and trace data, provide next day information and can be truly described as a demand-based statistic because the contact is patient-initiated. Currently, and unlike the case data, the triage data are relatively stable at national level in England. If they start to rise again this may be the first and, crucially, early indicator of a problem.

No more politics

It’s right that we do not forget that the objective of protecting the NHS in Scotland resulted in people being denied essential care for non-COVID-19 conditions nor the mistakes made in respect of care homes. These will be important questions for the future. But complaints to the Statistical Regulator when the Scottish Government attempts to make essential comparisons in order to do the job we want them to do suggests that some people have failed to learn any of the lessons from the pandemic. The drawing of comparisons is not a political debating ploy.  It is essential to our safety.

See also: Dan Sanderson, How Sturgeon secretly massaged Scotland’s record, Telegraph, August 8; Local cases in Greater Manchester, GMCA, July 1 news release

Main image of Nicola Sturgeon and Prof Jason Leitch, news briefing, August 12, via Scottish Government flickr CC BY-NC 2.0; video of Manchester via Wikimedia Commons/vimeo