Optimism: why your predictions are too good to be true

You might’ve spent most of your life dividing people into pessimists, optimists, and realists. This isn’t quite right. Optimism (or pessimism) defines the difference between a person's expectation and the outcome that follows. The absolute majority of us are optimists. When predicting the future, we heavily overestimate the probability of positive events happening to us and underestimate the probability of negative events. The bias is self-serving: we’re quite realistic when predicting events for other people. They might get divorced, get into a car accident, suffer from cancer. We, most likely, won’t, no matter what statistics say. We’ll live longer than national average, earn more money, and don’t even get me started on our children. They are special - much more talented than all other children.

As all most interesting cognitive biases, this one is consistent and well-established. The only group of people that isn’t affected by it are the individuals suffering from depression. They show the pessimism bias and predict an unrealistically hopeless future for themselves. 

So, when we divide people into pessimists, realists, and optimists, we’re in fact discussing the variability among optimists: in any sample you’ll find people that are more optimistic than average, less optimistic than average, much less optimistic than average, and so on. 

Foreseeing possible doubts and questions, I’ll tell you a bit about the research behind the optimism bias and its consequences.

First of all, the extent of optimism is measured by recording individual's expectations before an event unfolds and contrasting those expectations with the outcomes that transpire. Second, the optimism bias was found among people of all ages, gender, nationality and race. The bias seems to be an integral part of human nature. It might be that we wouldn’t function properly without it. 

The common question on optimism is this one: why don’t we drop the bias as we gain experience? Recent research shows the reason for this is that people update their beliefs more in response to positive information than to negative information, so our “positive” self is often reinforced, while the “negative” one not so much. 

As marketers, professionals, and generally curious people, let’s take apart the optimism bias. How does it affect us and what’s good and bad about it? 

The planning fallacy

The planning fallacy is a phenomenon in which people dangerously underestimate the time (and the cost, if applicable) to complete the future task. 

I’m sure you’re familiar with the planning fallacy. How often does a task end up taking three times longer than you’ve planned? How often are you confident you’ll manage to stick to your next plan despite all the unfortunate experience? 

The reason for this mistake lies in our optimism: we expect the best-case scenario when we plan the task, and we rarely end up with the best-case scenario.

This bias is something that often requires overcoming. To make sure you’re not affected by the planning fallacy, use the Flyvbjerg’s forecasting method: identify how long does the average project of your type (e.g., creating a marketing campaign) take to get the baseline prediction, and then use specific information about the case to adjust the baseline prediction.

This is how all predictions are usually made. If you’re asked to guess how much someone from New York earns, you’d first find out the average income in New York. Then, if you know any details, for example, that the person works as a doctor, you adjust your prediction according to these details. Same line of thinking should happen when it comes to planning. 

Above-the-average effect

People generally assume they are better than average on most desirable traits. They are even willing to bet money on it. A classic example that illustrates the obscurity of our self-assessment is that 90% of drivers believe they are better than average. 

There’s no immediate reason why you should stop genuinely believing you’re better than most people, however, it’s something to keep in mind when trying to realistically assess the situation. 

The illusion of control

As everything above, this might be not the nicest thing to learn, but the fact is the following: when predicting the future and evaluating the past, we tend to overestimate the causal role of skill and neglect the role of luck.

For example, you’re a founder of a startup. Here’s the question: to what extent will the outcome of your effort depend on what you do in your firm?

The answer founders give is rarely less than 80%. They might not be sure they’ll succeed, but they believe their fate is almost entirely in their own hands. This isn’t true: the success of a startup depends on the successes and failures of the competitors and the market situation just as much as on what happens inside the firm.

Again, this might not be the illusion we’re willing to let go. After all, wouldn’t we just end up doing nothing if we saw how little depends on our efforts? 

Overconfidence

Somehow, we often believe that if someone is confident they are more likely to be right. This isn’t correct. Confidence depends on the coherence of the story one has constructed, not on the amount and quality of the information, and it is flamed by the optimism bias. A study of patients who died in the ICU compared autopsy results with the diagnosis that physicians had provided while the patients were still alive. In the diagnosis, physicians reported their confidence. Clinicians who were ‘completely certain’ of the diagnosis antemortem were wrong 40% of the time. Keep this in mind when hiring your next employee.

The problem is, overconfidence is encouraged and reinforced. An unbiased appreciation of uncertainty isn’t welcomed by anyone, be that clients, patients, or employers. 

To tune down confidence and the illusion of control, Daniel Kahneman suggests doing the following: 

“Imagine that we are a year into the future. We implemented the plan as it now exists. The outcome was a disaster. Please take 5 to 10 minutes to write a brief history of that disaster.”

The good and bad sides of optimism

So does this mean that optimism is a nasty trait that should be fought? 

On the one hand, research shows that optimists (people more optimistic than average) are more successful in business, sport, and education. They work harder and for longer hours, which may be the reason. Besides, optimists are more popular due to being more cheerful and happy, they are more resilient to failures, less prone to depression, they take better care of their health, their immune system is better, and they live longer. 

On the other hand, the collective optimism bias can lead to disastrous events. For example, it’s believed by many economists that the financial crisis of 2008 was to a large extent a result of the optimism bias. Laypeople, government officials, and financial analysts - all expected the market to continue growing despite the clear evidence of the contrary.