What to do when forecasts diverge?

Dave Heatley

Principal Advisor, Productivity Commission

17 July 2019

The nice thing about forecasts is that you have so many to choose from.¹ For the weather, you can choose between MetService, metvuw, yr, the Weather Channel and many others. It’s best if they agree to guarantee a perfect day in the mountains…

But what should one do when faced with diverging forecasts?

I’ve been hunting out the headline job loss figures from studies and reports looking at automation and the future of work. My inspiration came from a Deloitte report that graphed the headline figures from seven studies². With some studious Googling, I found a further eight. Here’s my update of Deloitte’s graph: Job automation forecasts

Admittedly, the studies aren’t all forecasting the same thing. Dig into the details and you’ll find that each covers different countries and timeframes; and uses differing definitions of jobs at risk of automation. However, they all appear to draw on US data about the task content of existing jobs, and their forecast period is 10–20 years. Inter-country differences are to be expected as current industries and occupations vary across countries. Most of these studies are based on the methodology pioneered by Frey and Osborne, which even Frey has recently expressed doubts about.

The graph does show a rough trend towards less alarming forecasts over time. This could be random noise, or the result of improving methodologies or updated knowledge. However, the recent slowdown of technological progress in driverless vehicles does not seem to be reflected in the most recent forecasts by McKinsey (2019, Australia).³

Having said all that, the one thing that stands out is the sheer variability of these forecasts. What should I do with so many forecasts that don’t agree? Four options come to mind.

Pick the “best” forecast.
Build my own model.
Take an average and call it the “wisdom of the crowds”.
Recognise that the variability arises because of deep uncertainty, so the chances are slim that anyone (including me) will get it exactly right.

Picking a “best” forecast is fraught. What criteria should I use? The report with the prettiest picture on the cover, the one whose authors sound most confident, or whose forecasts best fit my priors? Should you trust my choice? (Arguably I should prefer one with data and assumptions that are not proprietary, so others can independently check its analysis.)

Building my own model sounds attractive. I could study the others, choose a method and make my own assumptions, then run the numbers. But could I build a better model than the others? Could I convince you and others? Or would I just be adding one more datapoint to the graph?

Taking an average doesn’t create “wisdom” unless the data points are independent. These studies share data, assumptions and methodologies – skewing the average in ways that make it no more reliable than any single study.

That leaves me with option 4. These studies are difficult to conduct and inherently unreliable. And it takes a certain level of bravery to put them out there. So, I think the best approach is to treat them all with respect and scepticism. And not bet the farm on any one of them being right. This thinking is compatible with scenario modelling – without a certain future, we should consider different but plausible futures.

One thing that concerns me about my summary graph is the use of 0% as the baseline – it isn’t as if no jobs were automated in the past, but we are suddenly going to automate heaps in the future. A more interesting question is: will the near future be different from the recent past? So, I used (or perhaps misused) the data on occupational churn in the ITIF report to come up with a baseline figure. Approximately 18% of US jobs “churned” (in occupational terms) over the period 2000-15. This gives a convenient reference point – if we assume that trend continues for the next 15, then we can subtract 18% from the headline forecasts.⁴ Here are the forecasts with the adjusted baseline.

Next 15 years job automation

The future looks less scary already!

Notes

Shamelessly adapted from Andrew Tanenbaum’s aphorism “The nice thing about standards is that you have so many to choose from.”
Deloitte (2017) The path to prosperity: Why the future of work is human. For another (shorter) summary list of studies, see page 21 of McKinsey’s A future that works.
In that report, the transport and warehousing sector is the “biggest automator” – 33% of jobs are forecast to be automated 2016–30 (midpoint scenario).
Note that according to ITIF report, US occupational churn has been at historical lows since 2000. If I chose 1990-2000 as the baseline, I would have subtracted around 45% and almost all the forecasts would for negative change relative to the baseline!