How Credible are the COVID-19 Models? Center Aims to Find Out

The multi-institution Center for Reproducible Biomedical Modeling, led by UW Bioengineering Professor Herbert Sauro, is partnering with top U.S. government agencies to determine how credible several commonly used COVID-19 models are.

Since the early days of the coronavirus pandemic, to aid in decision making, public health officials and policy makers have looked to epidemiological models that estimate how many people will likely get sick, need hospitalization and die from the new virus.

“Our role is to come up with a system that can be used to rank COVID-19 models in terms of reproducibility and credibility,” says Dr. Sauro.

Science depends on reproducibility. If a result can be reproduced, scientists have high confidence that the result is not due to chance or error.

“One of the problems is that almost all models – at least in physiology – are not reproducible,” Dr. Sauro says. “That means if you find a model in a published paper and try to run it on your own machine, the odds are you won’t get it to work.”

Anecdotal evidence suggests the same is true for the COVID-19 models, although no studies exist, he says. It’s a problem because other researchers can’t check how good a model is if they can’t run it themselves to look for bugs or see the assumptions that were made. “It might take months to get it working, by which time the crisis has evolved,” Dr. Sauro says.

The project centers on the idea of open science, and that having many eyes looking at the work will lead to identifying issues quickly so they can be resolved. Open science is particularly important for infectious disease models that make predictions, as untrustworthy predictions can result in bad policymaking decisions.

Created to promote more accurate systems biology models, the NIH-funded Center for Reproducible Biomedical Modeling has techniques and skill sets that readily apply to COVID-19 models. The Center’s collaborators on this project include the Centers for Disease Control and Prevention (CDC), the Defense Advanced Research Projects Agency (DARPA), the National Institutes of Health (NIH) and National Aeronautics and Space Administration (NASA).

The team is looking at five often-used models that were highlighted in an April New York Times article:

• Columbia University
• Imperial College London
• Institute for Health Metrics and Evaluation (UW)
• Massachusetts Institute of Technology
• Northeastern University

Colleagues at the NIH-funded Committee on Credible Practice of Modeling and Simulation in Health Care have come up with a set of basic rules they can use to assess the models, building on the FAIR data principles of findable, accessible, interoperable and re-usable (see sidebar). They expect that if the majority of the criteria are met, then they likely can get and run the model with the same results as the creators.

Leading the effort along with Dr. Sauro are Center co-director David Nickerson, a senior research fellow at the University of Auckland, and Anand Rampadarath, model curator at the University of Auckland. The team is evaluating the five models based on the criteria and will rank them according to credibility and reproducibility. They are reviewing the data now, and will take one more snapshot of the models at the end of July to capture any improvements before passing the results on to the NIH.

Best practices

In another project, the Center is collaborating with Matthias König, a computational modeler at Humboldt University of Berlin, who proposed defining best practices for distributing COVID-19 models and other infectious disease models.

The team, which also includes Lucian Smith, a research engineer in Dr. Sauro’s lab, plans to write a paper describing how to package and distribute a COVID-19 model in a findable, accessible, reusable and reproducible way, so others can examine and check it for credibility.

“The assumptions used in the model and how to use the model should be very clear,” Dr. Sauro says. The goal is to show how to distribute COVID-19 models to other academics who may be in a better position to inform the government about what’s happening.

Read about more coronavirus-related projects in the UW Bioengineering department.