I started looking at COVID forecasting papers and got angry. To move on with my life, posting a few things.
tl;dr: a huge amount of effort was wasted on COVID forecasting, and the allure of forecasting without exponentially better data is continuing to waste a lot of effort.
Evaluation of ... forecasts of COVID-19 mortality in the US pnas.org/doi/10.1073/pnas.2113561119
Evaluation of the US COVID-19 Scenario Modeling Hub... nature.com/articles/s41467-023-42680-x
Predictive performance of multi-model ensemble forecasts of COVID-19 across Europe... elifesciences.org/articles/81916#content
In the US and Europe, the best models (typically an ensemble) all just barely beat "tomorrow is the same as today" on average for a few weeks.
But it's much worse than that.
From the eLife discussion:
"Across all models we observed that forecasting changes in trend in real time was particularly challenging... Our study period included multiple fundamental changes in viral-, individual-, and population-level factors driving the transmission of COVID.
"We observed a contrast between a more stable performance of forecasting deaths further into the future compared to forecasts of cases. .... We can link this to the specific epidemic dynamics in this study.
"First, COVID-19 has a typical serial interval of less than a week. This implies that case forecasts of more than 2 weeks only remain valid if rates of both transmission and detection remain stable over the entire forecast horizon. In contrast, we saw rapid changes...
Etc etc.
All three papers mention this, but above was the clearest discussion.
The forecast models fail whenever something about the system changes.
Trivial right? But those are the only times someone might actually need a forecast model!
The models work better than baseline when you could take a pencil and draw what's coming. As soon as you might actually need math and science, they immediately fail. Why? Because if you aren't forecasting what is changing inside the system, it's impossible to get anything right
for more than a generation time (or two, insofar as you won't notice how bad you're getting yet). Why a generation time? Because those outcomes are already determined--those people are already infected, so your transmission model doesn't matter at all.
Between the US and Europe, approximately 140 modeling groups participated in this forecasting nonsense. An absolutely outrageous waste of scientific effort. Scientists' lives. Grad students led away from expanding human knowledge. Tenured jobs filled for decades to come.
And forecasting remains a flagship focus of the epi modeling field. Conferences are full of presentations on a few percent improvement on proper scoring rules, with little or no discussion of absolute error and decisionmaking. Y'know, how forecasting might matter. It's BANANAS.
You can read the unrolled version of this thread here: typefully.com/famulare_mike/XQG49xe