Survivorship Bias - Ignoring Hard to Find Data

September 6th, 2020 by 
PK

The Survivorship Bias (sometimes styled Survivor Bias) is the human tendency to value the seen or available at the expense of the unseen.

Data, groups, things, and people that exist today receive the bulk of our attention. We discount and ignore historical things or people or failures.

Survivorship Bias is often a form of selection bias, and sometimes further of cherry picking.

Survivorship Bias, War, and Abraham Wald's Observation

In one of the most famous studies designed to counter survivorship bias, Abraham Wald set out to determine the weaknesses of airplanes in World War II for the Center of Naval Analyses' Statistical Research Group (SRG). His report, "A Method of Estimating Plane Vulnerability Based on Damage of Survivors" (PDF reprint here) is a clever way of reacting to incomplete and missing data on aircraft survival.

Noting the quantity and location of damage to aircraft that returned from battle, his colleagues had suggested adding armor in those areas. Wald pointed out that they were only seeing the surviving aircraft.

All of the downed aircraft weren't being studied - the damage they were seeing was, therefore, damage that the planes could survive:

Survivorship bias illustrated by an aircraft.
Researchers only saw, by definition, survivable aircraft damage. (McGeddon)

Contra his coworkers, any additional armor or engineering should focus on the areas of the planes not showing damage. Those were the areas that couldn't sustain a hit.

Survivorship Bias and Your Finances and Career

Survivorship bias also rears its ugly head in finance and business. Because of the nature of data, it's easier to find records of things that exist than no longer exist. You need to be vigilant and watch out for situations where survivorship bias skews the data.

Survivorship Bias in Indexing

Consider an equity index such as the S&P 500. If you aggregated all of the current members of the S&P 500, you'd find something curious - that they outperformed the index itself.

Think about this for a second though... of course it should be the case. Indices can only include companies that currently exist but have the burden of the performance of companies that no longer do. There is a minimum trading price on many indices as well - indices literally remove companies for poor performance.

When companies fall from the index their past contribution to the index isn't scrubbed. Poor past performance remains baked into an index's history. When a new company joins the index, there's no provision to backdate statistics.

Indices only benefit from a new firm going forward.

And that's all it takes for that phenomenon to prove true - a bit of memory combined with the effects of poorly performing and successful companies. When you are aggregating performance outside an index, take care to include performance data from previously existing firms as well.

Survivorship Bias in Investing

Viewing a friend's portfolio holdings or an investment company's list of funds usually shows a similar effect. What's presented to you, an individual investor, is a snapshot of successful picks and funds. What you see is often beating some index's performance over a fairly long period.

Yet again, we see the survivorship bias at work. Your friend may have sold poorly performing picks, leaving only long-running market beaters. Financial companies continuously cull or merge poorly performing funds and syndicates, leaving only high flyers with high annual returns around for their sales pitches.

Don't get caught up in hyped market-beating picks and funds or sales pitches and bragging sessions. The truth is, you are witnessing only a snapshot... poor performers left the sales deck a long time ago. You can't invest in a bad fund from the 1990s, but you'll surely see a few funds with long histories that did perform well.

And how much is the bias? In 1996, Elton, Gruber, and Blake studied the survivorship bias that comes from the attrition of mutual funds [PDF]. They found a 0.9% a year bias due to the lack of effect of missing under-performers. Remember, that's a year... the effect compounds for longer time periods.

Survivorship Bias in the Workforce and Business

Not to pick on Business Insider, but their article concluding valedictorians are less likely to become millionaires because millionaires in some survey had a 2.9 average GPA finely illustrates the bias.

Ignore whether it's true - you can't compare 81 valedictorians in a longitudinal study to millionaires ex post facto.

Valedictorians are a reasonably well distributed sample - if not perfectly random. However, the millionaires are already millionaires - working backwards from outliers causes the post to draw the incorrect conclusions. While the authors found the valedictorians were very high performers, the other referenced study doesn't prove C students become millionaires more often.

A proper comparison would take a selection of students with a 2.9 average GPA and track outcomes.

Don't believe me? Consider the lottery.

If I interviewed 500 lottery winners, one common factor will show up - everyone in my sample gambled. If I'm not careful, I'll believe that the path to wealth leads through a lottery habit. Because of poor sampling, I wouldn't see the major downsides of the lottery.

And downsides to the lottery are legion. To wit: financial woes, marriage troubles, ruined friendships -  you'd miss all those downfalls if you only looked at lottery winners.

Beware the Survivorship Bias

Diogenes lived in a barrel but still knew survivorship bias when he saw it.
Diogenes lived in a barrel but still knew survivorship bias when he saw it.

The Survivorship Bias also has effects on humanity. Genes propagate when people survive to reproductive age. Nature exhibits its own survivorship bias towards the fecund and the strong - and not necessarily the most intelligent. (As so brilliantly lampooned in Idiocracy.)

And even if that weren't the case (yes, we're sure it is), how would we know?

Only survivors write, so yes: history is written by the victors.

  • What knowledge have we lost to history?
  • How many profound (yet lost) insights into the human condition have humans made and forgotten?
  • What commonly known events do we have all wrong?

Diogenes and Survivorship Bias

Diagoras approached the philosopher Diogenes with a painting of survivors of a shipwreck. "Look, you who think the Gods have no care of human things, what do you say to so many persons preserved from death by their especial favour?"

Diogenes didn't miss a beat. "Why, I say that their pictures are not here who were cast away, which were by much the greater number."

Indeed. As shipwreck survivor paintings only show survivors and returning aircraft only show survivable damage - you must be very careful with the survivorship bias.

      

PK

PK started DQYDJ in 2009 to research and discuss finance and investing and help answer financial questions. He's expanded DQYDJ to build visualizations, calculators, and interactive tools.

PK lives in New Hampshire with his wife, kids, and dog.

Don't Quit Your Day Job...

DQYDJ may be compensated by our partners if you make purchases through links. See our disclosures page. As an Amazon Associate we earn from qualifying purchases.
Sign Up For Emails
linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram