The small numbers problem: distinguishing good luck from good performance

Zero community transmission. Just a handful of active cases, all in quarantine. New Zealand is in an attractive situation, at least compared to much of the world. The situation could be robust and enduring – or it might be fragile, just one border incursion away from a “second wave”. How would we tell?

COVID-19 just keeps changing and evolving. Not the virus itself – that seems pretty stable as viruses go. What is changing is the phase of the pandemic, and our knowledge of the virus and the effectiveness of public policy responses.

Super-spreaders and under-achievers

Recent research suggests that some people—most, even—don’t spread the virus to anyone in the course of their infection. Others infect dozens. Studies suggest that 10% of cases lead to 80% of the spread, consistent with most chains of infection dying out by themselves while the occasional super-spreader can infect hundreds or even more.

In South Korea, 5213 cases spread outwards from a single infected person who attended two services of the Shincheonji Church in February. Closer to home, the Ministry of Health attributes 98 cases to the Bluff wedding cluster.¹

The problem with small numbers

In a world with super-spreaders, actual disease spread is highly dependent on luck. One infectious individual could infect hundreds. But, most likely, they will infect no others.

Luck averages out over large numbers. Performance measures like the estimated effective reproduction number rely on averages, and so become less useful and potentially misleading as the COVID-19 situation has evolved into one with very small numbers of active and new cases.

Luck or good performance?

New Zealand has not had a locally acquired case for several weeks. The next case, like the past twenty odd, will most likely arrive from offshore. Whether or not they go on to infect other people depends on both luck and good performance.

Clearly, the outcome everyone wants is zero spread into the wider community. But if we observe zero spread, does that mean New Zealand’s systems performed well at the border? Or that border control is porous, but we’ve just been lucky so far?

This conflation of luck and successful management makes the presence or absence of outbreaks an unreliable metric for monitoring the performance of the border control, quarantine, testing, contact tracing and wider public health systems.

Measuring performance when outcome metrics are misleading

When outcome metrics are misleading due to small numbers problems, it’s best to supplement those metrics with others.

First, look to intermediate performance metrics. For example: What proportion of people in isolation are tested before release? What proportion are tested on the designated days? What proportion receive scheduled follow-ups? Are there targets for such activities? If not, why not? If so, are they being achieved?

Second, test subsystems. For example, rather than waiting for an outbreak, conduct contact tracing on uninfected volunteers. What proportion of their contacts are identified and contacted? Is that sufficient? What might improve the subsystem?

Third, stress-test the system. Could it cope if an extra plane load turns up on Thursday? Or if the number of active cases in quarantine facilities climbed from 22 to 220?² Or 2200?

Fourth, emphasise transparency and learning. Tests that succeed can build public confidence. Ones that fail can build confidence too, as long as findings are publicised, and relevant performance metrics developed and reported against.

Fifth, explore creative surveillance methods. Traffic volumes help predict future GDP. Sewage testing measures illicit drug use. Smartphone movements tell us about lockdown compliance. What else could we be measuring?

My take outs

• Assessing New Zealand’s continuing efforts to control COVID-19 suffers from a “small numbers problem”, where disease outcomes are determined by both luck and system performance.
• System performance should actively be measured, by monitoring intermediate performance metrics and purposefully testing the system and its subsystems.

Notes
1. See https://medium.com/@codecodekoen/covid-19-superspreading-events-database-4c0a7aa2342b for a database of “superspreading events”.
2. There were 22 active cases in New Zealand as at 1pm on 29 June 2000.