Performance evaluation of RespiCast ensemble forecasts for primary care syndromic indicators of viral respiratory disease in Europe

In 2023 the European Centre for Disease Prevention and Control (ECDC) launched RespiCast, the first European Respiratory Diseases Forecasting Hub, to provide probabilistic forecasts for influenza-like illness (ILI) and acute respiratory infection (ARI) incidence across 26 European countries. During the 2023/24 and 2024/25 winter seasons, RespiCast collected one- to four-week-ahead forecasts from multiple models contributed by different international teams and combined them into an ensemble. Our analysis shows that, when evaluated using the weighted interval score (WIS) and the absolute error (AE), the ensemble consistently outperformed the baseline model (defined as a persistence model that projects the last observed value forward) as well as individual models across most countries and forecasting rounds for both ILI and ARI incidence in the two seasons. Analysis of ensemble coverage (defined as the proportion of times observed values fall within the specified prediction intervals) indicated that forecast prediction intervals were reliable, although a general overconfidence trend (i.e., prediction intervals that are too narrow) was observed, particularly in specific countries. The relative performance of the ensemble declined in certain weeks, likely due to reduced participation from modelling teams, epidemic dynamics, higher data noise, and reporting delays. Forecast scores varied across countries, with some exhibiting consistently higher relative errors than others. Overall, the findings highlight the strengths of ensemble approaches in improving the accuracy and reliability of epidemiological forecasts while identifying areas for improvement, such as managing overconfidence and addressing variability in performance across countries and over time.