The cult 1970s writer Douglas Adams managed to reduce the meaning of ‘Life, the Universe and Everything’ to the banality of a two-digit number (42, in case you’re wondering), leaving the characters in The Hitchhiker’s Guide to the Galaxy baffled to say the very least. Bafflement may also be our first reaction on hearing renowned investor Willem Middelkoop predict that the price of gold will run to EUR 5,000 an ounce. So how can we make sense of these numbers and extract some meaning for our investment policy, both at a commercial and a personal level?
With the rise of computers, electronically stored data and the improving expertise in the field of statistics since the 1980s, predictions are becoming increasingly common.“ These predictions then give people a sense of control, on the assumption that the future can be predicted, along the lines of “If this trend continues, oil will be trading at USD 90 a year from now” or “We believe that BNA bank is worth EUR 70 billion”. People make plans, manage their companies or conduct negotiations on the basis of such figures. They believe that they’re in control because a prediction has been made, by an expert or otherwise.
However, this kind of control is illusory: it is simply too good to be true. This is because real predictions consist of more than just a number. This is often forgotten and people then act on the basis of inaccurate or incomplete information.
When you read the kind of predictions mentioned above, alarm bells should ring immediately. Such numbers are nice for one-liners, but they’re only a small part of the whole story. Does the prediction take into account the variance in outcomes, the assumptions on which it is based, or whether the model used is reliable?
A “point estimate”, as it is called, of USD 90 per barrel is often an average. But the future is defined by a wide range of possibilities. Almost everything is possible in the future, certainly in the financial markets. The likelihood of an outcome other than the average is often far more important than the average itself. If you are heavily dependent on the oil price, and your company will definitely go under when oil hits USD 115, then the likelihood of that happening is far more important to you than the USD 90 average. But that’s not evident from the above prediction, which offered you a wonderful sense of security, because USD 90 is not equal to USD 115, and thus impossible.
If a predictive model relies heavily on assumptions, then it is very tempting for the maker of the prediction to vary these assumptions in such a way that the model produces the desired prediction. Assumptions thus pose a risk in themselves. Even if forecasters are honorable, assumptions still need to be appropriate. In the calculations for Solvency II, insurers need to protect themselves against shocks which occur once a century. A 40% drop in share prices is a widely used assumption for the “stock market crash” scenario. But this assumption does not reflect past experience: share prices fell by 47% in 2008/09 and by 48% in 1929.
The trend in modeling is to make models ever more complex. There is an advantage to this. Complex models are almost invariably better at explaining existing data. That’s also what makes them so popular. But it has been shown more than once that simple models are better at predicting than complex models. This is because complexity shifts: whatever had a correlation with the oil price in the past five years may not have the same correlation over the coming five years. What makes the model successful with regard to the past will make it worse in future. Moreover, complex models are precisely that: complex. Explaining how the model works is awkward and makes decision-making more difficult, because it’s not entirely clear what’s on the table, certainly not to lay people. And often it’s precisely these lay people who have to make a decision. Another point is that a model must reflect reality, or in any case take account of as many different outcomes as possible.
A widely heard excuse after the collapse of Lehman Brothers was that the models did not take this into account, because it was considered impossible. An omission of this kind simply makes the model misleading.
In all the situations outlined above, a prediction implies that the future has a certain degree of predictability. As an example of the creation of this kind of sham predictability, take the risk analysis that the European Banking Authority (EBA) is conducting in the context of banking supervision. This analysis, also called the “bank stress test”, falls short in all the abovementioned categories.
For one thing the EBA analysis uses only two scenarios to test the banks. One scenario with ‘normal’ conditions and an ‘unfavorable’ scenario. This negates the possibility of an intermediate, or an even worse outcome for the future. The future is not either one scenario or another, and this is a shortcoming in the model. Moreover, the test does not estimate the chances of something happening. The result is a simple ‘X of the Y banks are stress-sensitive’. If in a particular scenario 30% of the banks go under, it is very important to know the chances of this happening at some point in the future. Only then can the scenario be taken into account during the decision-making process. Furthermore, the assumptions of the two scenarios are not realistic in my view. There is a possibility that under current market conditions Portugal, Ireland, Greece and Spain (PIGS) will default on their debts. The EBA model does not consider this possibility. ING holds around EUR 15 billion in public debt from the PIGS countries on its balance sheet. If these debts are devalued by 20%, say, then this will inflict a EUR 3 billion shock on ING alone, and this will have a very big impact on the bank.
If the results of the stress test come out well, for instance because most of the assumptions made are favorable, then this will lead to a false sense of security. The stress test will probably conclude that the banks are in control, but this will be an illusion. The conclusion must be different, and focus much more on the limitations: “If our model is right, then it is likely that, under the assumptions made in the two scenarios, the banking sector will cope”. This conclusion is very weak: there is only a very small chance that the future will pan out exactly as posited in the assumptions. And even then the model has to be exactly right!
If the above three factors – variance of outcomes, assumptions underlying the model, and correctness of the model – are taken into account, then the illusion of control will be reduced. This is because we then explicitly acknowledge that the future is not very predictable. We concede that we’re not certain, and we can bear this in mind. It is crucial to remember at all times that people are fallible, and predictions even more so. Prediction is an awkward, perhaps even an impossible business. And potentially it can be very misleading.