Share

The small numbers problem: distinguishing good luck from good performance

Principal Advisor, Productivity Commission
1 July 2020

Zero community transmission. Just a handful of active cases, all in quarantine. New Zealand is in an attractive situation, at least compared to much of the world. The situation could be robust and enduring – or it might be fragile, just one border incursion away from a “second wave”. How would we tell?

COVID-19 just keeps changing and evolving. Not the virus itself – that seems pretty stable as viruses go. What is changing is the phase of the pandemic, and our knowledge of the virus and the effectiveness of public policy responses.

Super-spreaders and under-achievers

Recent research suggests that some people—most, even—don’t spread the virus to anyone in the course of their infection. Others infect dozens. Studies suggest that 10% of cases lead to 80% of the spread, consistent with most chains of infection dying out by themselves while the occasional super-spreader can infect hundreds or even more.

In South Korea, 5213 cases spread outwards from a single infected person who attended two services of the Shincheonji Church in February. Closer to home, the Ministry of Health attributes 98 cases to the Bluff wedding cluster.1

The problem with small numbers

In a world with super-spreaders, actual disease spread is highly dependent on luck. One infectious individual could infect hundreds. But, most likely, they will infect no others.

Luck averages out over large numbers. Performance measures like the estimated effective reproduction number rely on averages, and so become less useful and potentially misleading as the COVID-19 situation has evolved into one with very small numbers of active and new cases.

Luck or good performance?

New Zealand has not had a locally acquired case for several weeks. The next case, like the past twenty odd, will most likely arrive from offshore. Whether or not they go on to infect other people depends on both luck and good performance.

Clearly, the outcome everyone wants is zero spread into the wider community. But if we observe zero spread, does that mean New Zealand’s systems performed well at the border? Or that border control is porous, but we’ve just been lucky so far?

This conflation of luck and successful management makes the presence or absence of outbreaks an unreliable metric for monitoring the performance of the border control, quarantine, testing, contact tracing and wider public health systems.

Measuring performance when outcome metrics are misleading

When outcome metrics are misleading due to small numbers problems, it’s best to supplement those metrics with others.

First, look to intermediate performance metrics. For example: What proportion of people in isolation are tested before release? What proportion are tested on the designated days? What proportion receive scheduled follow-ups? Are there targets for such activities? If not, why not? If so, are they being achieved?

Second, test subsystems. For example, rather than waiting for an outbreak, conduct contact tracing on uninfected volunteers. What proportion of their contacts are identified and contacted? Is that sufficient? What might improve the subsystem?

Third, stress-test the system. Could it cope if an extra plane load turns up on Thursday? Or if the number of active cases in quarantine facilities climbed from 22 to 220?2 Or 2200?

Fourth, emphasise transparency and learning. Tests that succeed can build public confidence. Ones that fail can build confidence too, as long as findings are publicised, and relevant performance metrics developed and reported against.

Fifth, explore creative surveillance methods. Traffic volumes help predict future GDP. Sewage testing measures illicit drug use. Smartphone movements tell us about lockdown compliance. What else could we be measuring?

My take outs

  • Assessing New Zealand’s continuing efforts to control COVID-19 suffers from a “small numbers problem”, where disease outcomes are determined by both luck and system performance.
  • System performance should actively be measured, by monitoring intermediate performance metrics and purposefully testing the system and its subsystems.

Sunset

Notes
1. See https://medium.com/@codecodekoen/covid-19-superspreading-events-database-4c0a7aa2342b for a database of “superspreading events”.
2. There were 22 active cases in New Zealand as at 1pm on 29 June 2000.

Have your say

Please tick the box to prove you're a human and help us stop spam.

Comments

  • Gravatar for Paul C

    Paul C 3 Jul 2020, 11:21 (4 months ago)

    The problem is we already have some indication of the likely answers to a number of your suggested secondary metrics and based on what we know it is not an inspiring set of answers.
    Good suggestions nonetheless and it would be nice to see some progress on monitoring reporting and fixing the issues you raise.

    Please tick the box to prove you're a human and help us stop spam.
  • Gravatar for Carl Hansen

    Carl Hansen 1 Jul 2020, 17:31 (4 months ago)

    Thanks Dave, that's another very useful blog as I hadn't been aware that the super-spreader concept was a real thing. My only comment is that performance metrics need to include the economic impacts, otherwise there's a real danger the MoH and others influencing border and testing policies etc will continue to take a zero-risk approach. The zero-risk approach is likely to be very harmful for NZ's productivity growth, which should be the primary focus of the ProdCom.

    Please tick the box to prove you're a human and help us stop spam.
    • Gravatar for Dave Heatley

      Dave Heatley 3 Jul 2020, 12:04 (4 months ago)

      Good point Carl, and thanks for the reminder. A limitation of blogging is that, by examining complex topics in bite-sized nuggets, we risk downplaying the importance of over-arching frameworks. There are no zero-risk policy choices for governments -- only ones with varying degrees of risk and varying costs and benefits. Reduced future productivity growth will limit our individual and collective resources, and thus out ability to deal with tomorrow's health, environmental, economic and other challenges.

RSS feed for comments on this page | RSS feed for all comments