Critique of the IHME Model for COVID-19 Projections
Whitepaper Prepared for the Iowa Department of Public Health
by the University of Iowa College of Public Health COVID-19 Response Group

Submitted by
Grant Brown, Ph.D., Assistant Professor of Biostatistics
Joseph Cavanaugh, Ph.D., Professor and Head of Biostatistics
Aaron Miller, Ph.D., Assistant Professor of Epidemiology
Jacob Oleson, Ph.D., Professor of Biostatistics
and Director of the Center for Public Health Statistics
Michael Pentella, Ph.D., Clinical Professor of Epidemiology
and Director of the State Hygienic Lab
Eli Perencevich, M.D., M.S., Professor of Internal Medicine and Epidemiology,
and Director of the Center for Access and Delivery Research and Evaluation
Daniel Sewell, Ph.D., Assistant Professor of Biostatistics

Executive Summary
This document is intended to summarize current COVID-19 modeling efforts of the Institute for
Health Metrics and Evaluation (IHME) and contextualize their implications for the State of Iowa.
In short, the IHME model does appear to give a reasonable estimate of the potential mortality
trajectory in Iowa, but appears to underestimate the risk of experiencing a more serious outbreak.
The current estimate, as of April 7, 2020, is for a total of 420 deaths with a peak on April 26. Under
the assumption that “full social distancing” will be in place through the end of May 2020, the model
provides a range of anywhere between 263-711 total deaths through June 1. We believe that
outcomes in this range are plausible, but that a substantial risk remains of higher mortality.
Additionally, the IHME model is not able to estimate the positive and negative effects of specific
social distancing policies for the State of Iowa, including those policies currently in place in Iowa,
and therefore has limited utility for informing public policy going forward.
The IHME approach has changed over the course of the epidemic, producing substantial
revisions in predictions for the State of Iowa. It has been reliant on the relationship of the “mortality
curve” between states and countries, though it has now been extended to utilize additional data
sources and methodology. A comprehensive technical report describing the nature of the latest
model updates is not available at the time of this writing. The shape of the trajectory from
the model has been learned from daily mortality data from US states along with historical data
from the Wuhan, Italy, and Spain outbreaks, and the shape of the curve from
the model is updated regularly with the availability of new data in each state. To date, the
trajectory in Iowa has generally followed a similar pattern to other locations, although Iowa is still
in the early stages.
The IHME model does appear to account for the bundled effects of social distancing policy, even
though this is not explicitly checked on the Iowa-specific page header. Although the website
declares that social distancing is “not implemented” in Iowa, the model projections assume full
social distancing through May 2020. Therefore, the estimates in Iowa are largely being influenced

 by policy effects in other states. This is because the model is trained on data from these other
locations with more stringent social distancing policies, along with the time it took other states to
implement social distancing. If the states that the model are trained on have more restrictive
policies or experienced greater compliance with public health recommendations, this would result
in underestimating the size of the outbreak in Iowa. Mobile phone data suggest that Iowa’s
adherence to social distancing is lower than the majority of US states, which further enforces this
point.
Given the scientific uncertainty concerning important epidemic parameters, the limitations of
testing data in the United States generally and Iowa specifically, and the early stage of the
outbreak in Iowa, all models, regardless of sophistication, are currently subject to substantial
limitations. At this stage, we do not believe that enough data has been observed in Iowa to fit an
Iowa-only curve producing reliable estimates of the epidemic peak-date or overall impact. Thus,
while the current IHME model appears in line with plausible epidemic outcomes, based on an
evaluation of the approach, and on our own investigations, we believe that a substantial risk of
more severe spread remains. This point is particularly important, because the actions of Iowans
still have great potential to change the course of the disease in the state. Although the projections
of the current model appear plausible based on current numbers, the projections are predictions
and we must keep in mind the uncertainty and reliability inherent in the process of prediction. It is
far more likely that the model is underestimating the death toll than it is that the model is
overestimating the death toll.

Introduction
The goal of this document is to gauge the applicability of the Institute for Health Metrics and
Evaluation (IHME) modeling strategy to the State of Iowa. The IHME model does appear to
account for social distancing policy, even if it does not directly incorporate each specific policy
enacted in Iowa. Moreover, the modelling approach appears to give a reasonable estimate of the
mortality trajectory in Iowa, though it may underestimate the risks of greater spread.
Details of the COVID-19 Model that form our opinion follow.
IHME Model
Brief Overview – The IHME COVID-19 model is a forecasting model developed by researchers
at the University of Washington School of Medicine. It was designed to forecast the trajectory of
the COVID-19 outbreak across the entire US, as well as for states individually. The IHME model’s
primary focus is to estimate the number of deaths in each state across time. Along with the total
number of deaths, the modeling approach gives an estimate of what date the outbreak is expected
to reach its peak, which is specific to each state. Once the number of deaths in each state is
forecast, results are then used to project the expected hospital bed utilization, along with ICU
beds and ventilators needed each day. With the information on expected hospital beds, the model
provides an estimate of the expected date that healthcare resources may become overwhelmed.
Aggregated projections can also be viewed for the entire US.
Approach – The IHME approach is reliant on the total number of fatal cases over time, tracking
a specific shape of the mortality curve. This curve-fitting approach is defined in order to give a
consistent curve/pattern to the cumulative mortality data for each state being modeled. The curve

 that is fit to the data provides an “epidemic curve” that approximately represents growth in
cumulative mortality of the COVID-19 outbreak. We plot this curve to data from other countries
(China, Italy, Spain) where COVID-19 outbreaks occurred prior to the United States outbreak.

The shape of the trajectory from the model relies upon daily mortality data from US states along
with historical data from the Wuhan outbreak, and the shape of the curve from the model is
updated regularly with the availability of new data in each state.
Statistical Modeling Specifics - The statistical specifics of the IHME model have changed over
time, and at the time of this writing a technical presentation of the model is not available.
Nevertheless, in general the IHME model has utilized a mixed modelling approach, where some
model parameters are allowed to vary by state, while others are estimated to be similar across all
states. In other words, the model assumes that the shape of the epidemic curve in one state
should share similar features across states, but each state may also have its own unique aspects.
Thus, as the epidemic plays out nation-wide, those states where the outbreak begins earlier (e.g.,
NY and WA) help to inform the trajectory in other states that are expected to peak later (e.g., IA).
The IHME model accounts for social distancing, doing so originally by measuring the amount
of time between when a state’s mortality rate passes 0.31 cases per million and when social
distancing policy is first implemented. This mechanism for accounting for social distancing may
have been modified, but we can confirm that the model does indeed assume that all US states
are fully pursuing this end. In addition, we know that different social distancing policies were
considered: school closures, non-essential business closures, stay-at-home recommendations,
and travel restrictions.
Advantages of the IHME Model
1. Uses a relatively simple, population average, modelling approach – this relies on a limited
number of parameters and may tend to be more robust than alternative models that often
rely on a more complex set of assumptions. The approach allows the different states to
have similar shapes to their projected curves.
2. Utilizes information from other countries/states, that may be further ahead on the epidemic
curve, when estimating the trajectory for individual states that may be further behind. This

 allows the model to make reasonable projections in a state like Iowa, where very limited
data are currently available. If Iowa implements strong social distancing measures
throughout the remaining course of the epidemic, it is reasonable to assume that our
shape will be similar to the other states and countries that experienced earlier outbreaks.
3. Utilizes mortality data for estimating the shape of the epidemic curve, which may be more
accurate/stable than using identified case counts. We know that case counts are severe
undercounts with lags in testing and that during the epidemic rates of testing have begun
to increase. However, we assume that mortality data is more accurate.
4. Frequently updated with new data from each state (every 4 days). The IHME group
summarizes changes that occur with these updates and model performance is compared
across updates.
5. Does not rely on disease specific parameters – many of the disease specific parameters
(e.g., R0, case-fatality rate, case-hospitalization rate, etc.) remain largely uncertain with
COVID-19.
Limitations of the IHME Model
1. Likely produces interval estimates which are too narrow, failing to capture the full
uncertainty about the course of the outbreak in individual states.
2. Assumes that the shape of the epidemic curve is reasonably symmetric. This detail makes
the tail of the distribution likely too low, and the confidence interval at the end of the
epidemic far too narrow.
3. Does not model individual sub-populations (e.g., age-groups, counties, healthcare
settings).
4. Does not model transmission, or individual actors or settings. This means that exact
results of social distancing and stay at home policies cannot be determined. Importantly,
the IHME model cannot estimate the effect of implementing additional or more
stringent social distancing measures or of relaxing the social distancing measures
already in place.
5. May underestimate the trajectory of an outbreak, if trained on data from locations with
more effective social distancing policies. This is likely the case, as mobile phone data
seem to suggest that adherence to social distancing in Iowa is low, while social distancing
policy actions were far more restrictive in other locales, e.g., Wuhan. Similarly, if states
further ahead have implemented more restrictive policies or experienced greater
compliance with public health recommendations, this distinction may be missed.
6. Limited in scope - because IHME is not examining the transmission process, it cannot be
used to model a secondary outbreak (e.g., in the fall, or if social distancing is relaxed).
7. Potentially less accurate in mapping healthcare utilization and ICU beds. These
projections rely on mappings to the estimated mortality rate. These do not appear as
accurate for all states and may need to be scaled back for Iowa.
8. Is impacted by modifications to the methodology over time. Thus, the corresponding
changes in predictions for Iowa reflect the large but potentially underreported model
uncertainty.
Applicability to Iowa
The modelling approach appears to give a reasonable estimate of the mortality trajectory in Iowa,
but policymakers should be mindful that the model likely underestimates the potential for
significantly more severe outcomes. We base this opinion on the following points.

 1. The model does appear to be considering social distancing efforts in Iowa, projections,
even if this is not explicitly checked on the Iowa-specific page header. Any deviation
between the social distancing policies and general practice in Iowa, and as defined in the
model, could lead to underestimated mortality outcomes.
2. While the model may not directly consider Iowa-specific policies, the estimates in Iowa are
largely being influenced by policy effects in other states. With little mortality data available
in Iowa, the model must rely heavily on how the outbreak has played out in other
states/countries that are further along the epidemic curve. Most of these states have as
aggressive, or more aggressive, policies on social distancing in place. In other words, the
estimated shape of the epidemic curve in Iowa is being “learned” by evaluating the
epidemic curve in other states, along with the time it took other states to implement
distancing. Because of this, the Iowa projection is being indirectly influenced by policies in
other states. To evaluate the applicability of the predictions, we can ask the simple
question: “is the information obtained from other states causing the model to over- or
underestimate the peak in Iowa?”
Another consideration in the applicability of the IHME model to Iowa is the dynamic nature of its
forecast estimates. Because Iowa is further behind many other states in the timing of the
epidemic, and because Iowa has a population that differs in many characteristics from states that
experienced the outbreak earlier, the model forecast is likely to change as new data becomes
available in both Iowa and other states or countries. In the figure below, we have plotted different
projections from the IHME that were forecast on three different dates after new data was available:
March 29, April 1 and April 6. The total cumulative mortality in these estimates were 138, 1,488
and 420, respectively. Notice also that the confidence bounds of all three cumulative death
estimates did not overlap. Thus, it is likely that there will exist future updates to the forecast that
may differ from current projections, and such estimates may lie outside of the confidence bounds
of the current model.
General Limitations of Modeling in Iowa
Given the scientific uncertainty concerning important epidemic parameters, the limitations of
testing data in the United States generally and Iowa specifically, and the early stage of the
outbreak in Iowa, all models, regardless of sophistication, are currently subject to substantial
limitations. We do not believe, at this stage, that enough data has been observed in Iowa to fit an
Iowa-only curve producing reliable estimates of the epidemic peak-date or overall impact. We
believe, therefore, that it is necessary to rely on information from other states. As such, we believe
that the approach used by IHME is reasonable, albeit still subject to substantial uncertainty. Our
evaluation of the model, and our own investigations, indicate that while the approach taken by
IHME is reasonable, it likely underestimates the risk of more severe epidemic outcomes.

 2000

1500

1000

Cumulative Deaths

500

100

75

50

Daily Deaths

25

Historical IHME Forecasts

 

 

 

Apr Jul

 

Apr Jul
Date

Model

03-29-2020
04-01-2020
04-06-2020