Success Versus Failure: What Causes the Difference?

Coming up with working definitions of success and failure and then researching, revealing, and tracking success rates based on those definitions are fundamental steps. Next, we would need to consider becoming more of a story scientist to understand the crucial differences between successes and failures.

In a post on his blog You Are Not So Smart, journalist author David McRaney tells a curious story about damaged World War II planes. Every day, US Air Force personnel got to observe plane after plane, landing on an airfield with bullet holes in their bodies after having been exposed to enemy fire. What should the American engineers on the ground do with the information they derived from these observations?

A seemingly obvious answer would be to repair the damaged parts before sending the planes back into battle. A further action could be to let the plant that produces the planes know about the weak parts that consistently get damaged by flying bullets and request that these be better reinforced during production. That seems like the right thing to do, based on experience.

But wait . . . The damage inflicted by the bullets was actually immaterial to the aims of the military. Their ultimate objective was to prevent aircraft from crashing and falling in battle. The planes that the engineers experienced had all returned safely despite being hit. The parts that were damaged did not cause them to crash. Thus, the bullet holes were not as relevant as they seemed to be.

This insight led statistician Abraham Wald to warn the US Air Force that fixing and reinforcing particular damaged aircraft parts would actually be counterproductive. He argued that one should look beyond the planes that returned successfully and attempt to gauge what really made the difference between life and death in battles up in the sky. While the fallen planes were unavailable for close inspection, he proposed looking at where the returning planes were in fact not hit.

Wald reasoned that, if planes received damage in certain places and still survived, those that fell must have received damage in other places. Hence, some of the cleanest, scratch-free parts of the surviving planes must be the culprits. Those were the areas in need of special reinforcement. Wald’s approach required an analysis that went beyond the available experience to reveal the difference between successes and failures. What’s remarkable is that he managed to do it when most of the failures were missing from observation, shot down in battle.

The popularity and prevalence of lists of things successful people do, however, suggest that we typically don’t look for such systematic differences. We tend to learn from survivors and their readily available experience—the more striking and obvious, the better. Instead, to better understand the determinants of success or failure, we need more initiatives that uncover whatever is missing from our experience so that we can recognize and analyze the difference.

For example, FailCon, F*ckup nights, Fail Festival, and similar events can help us do that. These are conferences where decision makers take the stage to talk about their screwups, which would normally stay hidden, much like the fallen planes. Thanks to these encounters, the audience has a better idea about the sheer number of failures in comparison to successes (the success rate). They also learn more about the obscure details in decision processes that are associated with mistakes. This allows them to better contrast effective processes with those that were fruitless and identify differences between successes and failures.

Princeton University scientist Johannes Haushofer recently offered another unique view on experiences that tend to remain hidden. He has an enviable profile: faculty member in a world-renowned institution, publications in prestigious scholarly journals, awards and fellowships, and an impressive list of ongoing and upcoming projects. Yet his CV offers little clue as to how other academics might achieve similar accomplishments. What are some of the nuances, trade-offs, or success rates?

Now we know more, thanks to Haushofer’s “CV of failures.” On the advice of fellow scientist Melanie Stefan, he decided to publish a document that lists and summarizes his misadventures. The reader quickly notices the large number of institutions, journals, and programs that rejected Haushofer and his ideas over many years, thereby gaining a better understanding of his success rate. We also get a glimpse of how many ideas he produced during a given period and how those evolved over time. By examining the difference between his regular CV and his CV of failures, one recognizes how much of the iceberg gets buried under the cold, dark ocean of failure and rejection.

But let’s be cautious: there are a few hidden caveats about studying the difference between successes and failures.

First, this approach doesn’t always guarantee understanding the direction of the causality. Successes may indeed be different from failures, yet this difference may not be the cause behind success but a consequence of it. In the case of WWII planes, the causality is clear: bullet holes cause the crashes. But in more complex domains like business and social life, the relationship between actions and outcomes can be more complex and harder to decipher. A person may be able to afford particular strategies only after becoming successful, yet these can be falsely interpreted as causes of their achievements.

Second, remember that when success rates are small, randomness may be playing a key role in the outcomes. If the population is large enough, we will observe all sorts of performances that we can then dissect, nitpick, and “explain” after the fact, with the help of 20/20 hindsight. But that doesn’t guarantee that these differences can be successfully controlled and managed—let alone that they were really crucial in determining the outcome.

Third, the differences between success and failure may change over time. Not tracking that evolution could lead to misconceptions. As the world changes, so inevitably will any secrets for success, and most advice for success is thus bound to become obsolete. Hence, analyses of the differences between successes and failures need to be periodically reviewed, repeated, and replaced.

Ultimately, learning about success and failure from experience is more nuanced and complicated than it seems. If we are not careful, issues with accuracy, causality, time, selections, complexity, and trade-offs are bound to induce errors in our evaluations that are hard to detect. Once we think we have learned our lesson, the simple and alluring stories that emerge can be more illusory than real.

Deep down, we may all know there are no magic solutions to many challenging problems in life, but experience can suggest otherwise. We may be tempted to simply stick to the plans we made in advance, without checking their validity for ourselves. Our first instinct during a crisis can easily be to repeat a habitual practice rather than to question it. A better strategy would be to promptly become a story scientist and test ideas early to refute bad hunches.

The Myth of Experience: Why We Learn the Wrong Lessons and Ways to Correct Them by Emre Soyer & Robin M. Hogarth (PublicAffairs, £20) is available now.