Predictions are Hard and Other Lessons from COVID-19
Predictions have become an inexorable part of our everyday lives, and nobody saw it coming.
The news industry could almost sustain itself on a steady diet of predictions alone. There is the famously difficult and perilous task of predicting who will win the next election. Even if we are fortunate enough to live through a non-election cycle (do those happen anymore?) we will still be inundated by predictions from the business world (will the markets rebound? is it time to buy?), the fashion world (maybe THIS will be the summer the male romper finally takes off), and the entertainment world (who will get a rose and who won’t?). And when these topics are exhausted, we will always have the weather.
Predictions are not confined to the news. At this very moment, sophisticated AI algorithms are haunting every corner of the internet. Netflix and Youtube are predicting what we might like to watch next. LinkedIn saved their business by predicting which connections might interest us. During the first 300ms of a page loading in our web browsers, an invisible bidding war is automatically occurring based on predictions about what we are most likely to buy.
Whether we see them or not, predictions are affecting our day-to-day lives. And no example of predictions affecting our lives could loom larger than the current cloud of predictions surrounding COVID-19.
At the time of this writing, total deaths related to COVID-19 have surpassed 100,000. We are told this is a staggering figure. How often are we reminded that the White House response was originally influenced by a model that predicted 2.2 million deaths? If 100,000 deaths stagger, what would 2.2 million deaths do? Should I mention that one estimate had as many as 7.2 million deaths?
By mid-March, wide ranging shutdowns and restrictions on our movements were announced. State and local governments all began to roll out the same playbook, and all of it was justified by predictions. These shutdowns have resulted in closed businesses, delayed weddings, canceled funerals, and an epidemic of isolation, the effects of which are impossible to fully appraise. We have been told that all the measures being taken have been based on science, but that is imprecise. What our policymakers have been relying on are predictions.
The boundaries between hard science, soft science, pseudo-science, and no science at all can be as hotly contested as the boundaries in a middle east war zone. Asking if predictions are science is like asking if a hot dog is a sandwich. While it can be hard to pin down precisely where the field of predictions belongs, we can be sure it is not the kind of hard science we would like to see backed up by the force of law.
Imagine I told you I had a multi-sided object—it could be two-sided like a coin, or six-sided like a dice, you don’t know yet—and I asked you to guess how many sides it comprised. I’m not asking you to make a blind guess. I am going to provide you with some data. I will flip (or roll) the object a number of times and give you the result. The question is, how many flips or rolls do you need before you are comfortable making an informed guess? This is similar to the kind of problem expert forecasters are trying to solve every day
Making an accurate prediction is chiefly a problem of solving for an unknown: the future. It may seem like the hard sciences do the same thing. For example, Newton’s law of universal gravitation is very good at predicting what will happen when an apple is separated from its branch of the tree. We know it will fall, we know the direction, acceleration, velocity, etc. A key feature of a hard science is the repeatability of the finding. We can drop one thousand apples and observe the same result one thousand times. We cannot turn back time and revert the past back into an unknown future and play it out again and make a thousand observations, nor would we expect the same result even if we could.
This does not mean that the discipline of predictions is short on observations. The core practice is to take an available dataset (a sample), analyze the data, and project the insights into a larger dataset. The larger dataset for prediction-makers involves the future. In data science the variable for your sample is N. N is almost never all. This means that projections will never be based on all the information from the past or present. Even if possessing all the information were theoretically possible, we still would only be dealing with what did happen. What did happen is only one of a number of possibilities. We won’t have the data for what didn’t happen but was equally likely. It’s like flipping our unknown multi-sided object once and projecting that result into the future. How many future scenarios are possible, and what deterministic relationship the past might have with the future, is a philosophical matter in essence. No hard science here.
Still, predictions can be made that are somewhat useful. Flip or roll our object a thousand different times and the data will begin to reveal information about it that, while not deterministic, can be useful. In the case of our mystery object, let’s say it turned out to be a two-sided coin. It wouldn’t take you nearly one thousand data points before you became satisfied that there was not going to be a sudden appearance of a third side in the data. We could look back at the data from one thousand tosses and see that somewhere between 5 and 20 tosses is when we stopped seeing unexpected results. The trick is, it’s only easy to spot the point where confidence emerges from the data after we’ve far exceeded it.
In the case of COVID-19, the problem of confidence emerging from the data is doubly complicated. We are told that the situation is unprecedented and, simultaneously, that we must rely on the historical data. Thus, the ghost of the Spanish Flu was invoked. The earliest COVID-19 predictions were based on century-old data from an unrelated pandemic. This should have been a warning sign about the meager nature of the available data, but it wasn’t. The arrival of data from our present situation was an improvement, but just barely. In data science there is a saying: no data is better than bad data. The earliest data surrounding COVID-19 were plagued with questions about their credibility, and in some cases were downright fraudulent.
It is generally true of predictions that the larger your sample size, the better your analysis and projections will be. It follows that your worst possible starting point is the beginning. From there, the predictions can only improve. What we needed was time. But our policymakers became impetuous and would not tolerate waiting. They told us they had to act quickly. It’s what the science required. In reality, it was hardly science at all. It was guesswork of the lowest available quality. Sophisticated guesswork conducted by highly-educated people is guesswork nonetheless.
What the models needed back in March, they now have: time. The models of today strike a remarkably realistic and plausible tone when compared to the hyperbolic doom of three months ago. This is the natural result of increasing the sample size of a dataset. We now possess the hindsight to see that the real crisis is less attributable to the pandemic and more attributable to the impatient response of our betters in office.
Now that the fiasco is beginning to pass, the questions loom large: what will we learn from this past data? how will we project that knowledge into our future? and will we learn anything that we shouldn’t have already known going into this?
It would be an error for us to condemn the “experts” for the low-quality of their predictions. It was, after all, predictable. What galls me is not the experts for making bad predictions but the policymakers for giving those bad predictions the force of law. Our policymakers should act as gatekeepers between expertise and our daily lives. They should consult experts often, and when they do it is their duty to know the kind of information that is being handed to them. They should be able to recognize the difference between a conclusion and a hypothesis, hard data (which the future is incapable of providing) and guesswork. If the policymakers are to be held to any account by the people who elect them, we also owe it to ourselves—regardless of specialized expertise—to know what kind of information we are dealing with. To paraphrase Pericles: Just because you do not take an interest in predictions doesn’t mean predictions won’t take an interest in you.
If we could single out any one institution, sit it down, and give it the “be better” talk, I nominate the corporate press. I seldom saw the word “science” used by the press without wondering if it should not better be capitalized, it was deployed with such deified regard. There was a tacit assumption that what we were dealing with was hard science, and that the terms of engagement and resulting courses of action were determined unassailably by this Science. To second-guess the quality of what was clearly mere guesswork was to deny Science, placing our communities in mortal peril. Far from encouraging discussion and challenging the assumptions of the experts and policymakers, large corporations like Facebook and Alphabet were actively censoring dissent. Critical thinking cannot be expected to flourish while critical voices are silenced (don’t worry, they already know this). Be better.
It is fortunate that COVID-19 has failed to live up to the hype. This was not the big one we feared. But, if I may be so bold as to make a prediction, the big one will come. When it does, my hope is that inquiry, discussion, debate, discourse, critical thinking, and true science will flourish. This can only happen if our policymakers resolve to never again bestow upon mere guesswork the force of law. Time will tell.
