Pandemic cryptic phase modeling hub

There are limited modeling capabilities to address the initial “cryptic” phase of a pandemic, which is characterized by noisy data, rapid changes in reporting, high uncertainty, and the importance of connectivity and external seeding. Yet this is a period where models can help answer key public health questions, including the extent of local transmission, how the epidemic might unfold, and the anticipated benefits of different interventions. The Scenario Modeling Hub (SMH) plans to fill this gap in modeling capabilities by running one or more modeling rounds addressing the early arrival of a hypothetical new respiratory virus in the US, France and the UK.

Inspired by earlier work on Ebola, the SMH cryptic phase effort will rely on synthetic epidemiological data, i.e., outbreak data generated by a model. The rationale for relying on synthetic data is several fold. First, if we used real world data on historic pandemics and conducted a retrospective round of modeling, SMH participants would be aware of the realized future and projections would not be blind to future outbreak information. Second, use of real world data does not allow any control over multiple factors that play an important role in the cryptic phase of a pandemic, including epidemiological conditions (eg amount of testing, pathogen transmissibility) or data quality (eg, reporting rate and delays). Further, by conducting a prospective modeling effort where participants are blind to the future, we can run a form of tabletop exercise that improves disease modeling and scenario design in a pandemic context. Note that the proposed modeling effort will not necessarily proceed at the accelerated timescale that a real pandemic would require. The primary goal of this work is to develop and improve modeling capabilities, while respecting competing time constraints on participating modelers.

This repository follows the guidelines and standards outlined by the hubverse, which provides a set of data formats and open source tools for modeling hubs.

Synopsis of the cryptic phase round

Synthetic epidemiological data will be generated by two separate but linked models. The Global Epidemic and Mobility model, GLEAM, will be used to simulate the global spread of a hypothetical new respiratory virus (virus X) out of Brisbane, Australia. GLEAM will generate data on the course of the epidemic at the source location and importations into the US, UK and France. Once the hypothetical virus has been seeded in the US, UK and France, an agent-based model, Epihiper, will take over and simulate local transmission. A set of natural history parameters for virus X will be chosen in line with prior respiratory virus outbreaks, but will not be identical to any past outbreak. These parameters will not be disclosed to participating modeling teams to ensure the exercise is as realistic as possible. Participating teams will be asked to project the future of the outbreak in two US states (CA and VA), UK and France. We recommend that teams try to project all locations, although submissions of projections for a subset of locations will be accepted.

Measurement error will be applied to simulated epidemiological observations to account for limited and changing rates of testing, and reporting noise and delays. Participating teams will be provided with one instance of observational outbreak data, accounting for measurement error, including weekly cases and deaths at the source, line list of US importations, and weekly cases and deaths in VA and CA, France and the UK. To provide context for this hypothetical case study, fictional situation reports will be generated and provide a broad timeline of the outbreak at the source in Australia, information on virus X clinical manifestations, severity, and any interventions implemented in Australia. Limited information on natural history parameters will be provided as well, as would be available in the early days of any real outbreak based on modeling of limited data at the source.

We will consider at least two sequential rounds of predictions for the US, UK and France:

1. Cryptic phase round 1 (earliest phase of the outbreak in the US, UK and France): scenario projections for early emergence of a hypothetical pandemic virus originating from Australia and seeded into the US, UK and France
2. Cryptic phase round 2 (slightly later phase of the outbreak in the US, UK and France): inference of disease parameters for a hypothetical pandemic virus. as more detailed local epidemiological data becomes available in the US, UK and France

The team in charge of generating the data will be separate from participating modeling teams and from the regular SMH coordination group

In round 1, SMH teams will be asked to provide projections of outbreak trajectories in the US, UK and France under different scenarios. Scenarios will be left to the discretion of the SMH coordination team (independent of the data-generation team), and will be established after discussion with the participating teams and public health stakeholders, in keeping with the regular SMH process. While we cannot fully anticipate what these scenarios will be, scenario axes will presumably follow the usual 2 * 2 design. For instance, one axis may cover disease uncertainty (eg, reporting rate, R0, or a combination) and a second axis may cover interventions (eg, testing intensity in round 1); alternatively both axes may cover disease uncertainty. Predictions targets will include simulations of reported cases and deaths in CA and VA, UK and France. Teams may choose to focus on one or more locations. We may also request projections of infections, to facilitate comparison between models (as cases and deaths are subject to reporting assumptions). We will request that teams submit simulation trajectories rather than quantiles.

In round 2, we will address inference of disease parameters and will request point estimates and distributions (format TBD) of the serial interval, doubling time, Rt, case fatality rate, and infection fatality rate. For pathogen-specific parameters such as the case fatality rate and infection fatality rate, we will evaluate the accuracy of model estimates against the ground truth synthetic data. For parameters that are model-dependent such as Rt, we will compare estimates across different models and methods as well as against the ground truth data-generating model.

Deliverables:

Predictions of component models and ensemble outputs will be made publicly available on Github, along with meta-data summarizing the different modeling approaches. We also encourage participating teams to post their modeling code and assumptions when feasible. Ultimately, we will describe the results and lessons of this work in a supplementary issue (journal TBD), in which each participating team will be encouraged to submit one or more publications, as well as co-authoring the overview publication.

Broad significance of this work:

Through the proposed SMH cryptic phase project, we will advance both the operational and research aspects of long-term disease projections, including:

1. From an operational angle, we will develop the modeling capabilities of SMH teams to handle the early stages of a hypothetical pandemic (particularly, models will need to account for changing levels of noise and reporting). We will also practice scenario design in the context of a pandemic.

2. From a research angle, we will explore how synthetic data generation can help advance the science of scenario projections. This is a unique case study of a hypothetical outbreak, where the full ground truth is known, allowing for a full evaluation of model accuracy and proper accounting for deviations between scenario assumptions and reality, which has been a limitation with prior SMH evaluation. Reliance on real-world disease data is restricted to a “single realization”, which severely limits model improvement.

Thus, this work will pave the way for a large library of (synthetic) outbreak data to improve calibration and evaluation of future disease models.