A Useless Model

The IHME model produced by the University of Washington and funded largely, I believe, by Bill Gates, is widely referred to by journalists and politicians. I have followed that model and its predictions for a while, and have concluded that it is more or less useless, for the following reasons.

First, the IHME model abdicates just where policymakers–i.e., governors–need information. The model has projected deaths, along with demand for hospital beds and ICU rooms, up to August 4, 2020. This is a typical chart you will find on the IHME web site:

Further, the model’s web site has said, with respect to each U.S. state, “Social distancing assumed until infections minimized and containment implemented.” Presumably the IHME modelers think that social distancing makes a difference. In each state, they show new deaths leveling off to around zero well before August 4.

But what happens when the current lockdowns end? That is what governors really need to know. It seems obvious that, if one assumes the lockdowns have slowed the spread of the virus, when the lockdowns are eased the virus will again spread more rapidly. And, as a result, more people will get sick and more will die. Governors understand this, while most in the general public don’t, naively believing that lockdowns will end when it is “safe” for us to emerge from our homes.

On the key decision that confronts policymakers, the IHME model resolutely refuses to give any guidance. Does it matter whether the lockdowns end tomorrow, or continue for another three months? And what increase in infections should people be prepared to expect when our economic and social life resumes? The IHME model is silent.

But this makes no sense. If the model “assumes social distancing,” it must incorporate a mathematical calculation that is different from what it would be in the absence of social distancing. Nevertheless, the modelers decline to explain their logic or to enlighten policymakers.

Second, the IHME model, when examined on a state by state basis, produces results so whimsical that a governor would be foolish to rely on them. I have written about this before, most recently in Coronavirus In Five States (3). I have followed the model’s predictions with regard to the five states of the Upper Midwest, but I am sure that you would see similarly unhelpful results if you looked closely at other states. Here are the fatality totals that the IHME model has predicted at various times for four of the Upper Midwestern states:

Minnesota

April 11: 442
April 15: 656
April 22: 360
April 29: 741

The IHME projection for Minnesota was cut by nearly 50% between April 15 and April 22, then more than doubled in the next week, April 22 to April 29. Why? Not because of any such wild fluctuations in Minnesota’s statistics. At any time during the past month, our governor would have been foolish to base any decision on IHME’s projections.

Iowa

April 11: 743
April 15: 618
April 22: 365
April 29: 302

The IHME model was grotesquely wrong about Iowa, and has been adjusted repeatedly to bring it more in line with reality. At least it is getting closer: according to IHME, Iowa had 136 deaths as of April 28. Any action Iowa’s governor might have taken in early April, relying on the IHME model, would have been misguided.

North Dakota

April 11: 369
April 15: 32
April 22: 356
April 29: 136

Little need be said about these wild gyrations. They apparently result from changes in the IHME model, not from anything actually happening in North Dakota. Total number of COVID-19 deaths as of April 28, per the IHME? 19. The governor of North Dakota and his public health officials would have been crazy to pay any heed to the IHME model, at any time.

South Dakota

South Dakota is an interesting case. These are IHME’s predictions of South Dakota fatalities through August 4, on the following dates:

April 11: 356
April 15: 181
April 22: 93

Again, IHME was dead wrong. Its South Dakota projection has now dropped all the way down to 17, but that projection only runs through May 7, next Thursday. Why? Apparently because South Dakota’s governor, Kristi Noem, has declined to order a shutdown. IHME says:

Long-term projections are unavailable for this location. Our model does not account for easing social distancing or quantify the risk of resurgence.

But then, what good is it? This is the point I made earlier.

In any event, South Dakota’s governor would have been badly misled if she had relied on the IHME model a month ago. They were wrong and she was right, and the modelers have drastically reduced their projection to approach nearer to the reality on the ground.

South Dakota appears to be one of only a few states–Tennessee and Alaska are the others I have noted–for which IHME disdains to make a prediction beyond next week, apparently on the ground that those states have eased social distancing requirements (or, in South Dakota’s case, never mandated most of them). Still, inconsistencies abound. North Dakota, like South Dakota, has implemented two of IHME’s six measures of social distancing. Yet, if you go to IHME’s North Dakota page, the modelers continue to project deaths through August 4, and there is no disclaimer as to “[l]ong-term projections [being] unavailable for this location.” What is the difference?

I will leave it to others to speculate. In any event, for the reasons stated, I think the IHME model is more or less completely useless.

Responses