How the Survivor Bias Distorts Reality
In fact, Smith notes, from 2001 through 2012 the stock of six of Collins’s 11 “great” companies did worse than the overall stock market, meaning that this system of post hoc analysis is fundamentally flawed.
….Smith analyzes two of the best sellers in the genre. In his 2001 book Good to Great (more than three million copies sold), Jim Collins culled 11 companies out of 1,435 whose stock beat the market average over a 40-year time span and then searched for shared characteristics among them that he believed accounted for their success.
Instead, Smith says, Collins should have started with a list of companies at the beginning of the test period and then used “plausible criteria to select eleven companies predicted to do better than the rest. These criteria must be applied in an objective way, without peeking at how the companies did over the next forty years. It is not fair or meaningful to predict which companies will do well after looking at which companies did well! Those are not predictions, just history.” In fact, Smith notes, from 2001 through 2012 the stock of six of Collins’s 11 “great” companies did worse than the overall stock market, meaning that this system of post hoc analysis is fundamentally flawed.
Smith found a similar problem with the 1982 book In Search of Excellence (more than three million copies sold), in which Tom Peters and Robert Waterman identified eight common attributes of 43 “excellent” companies. Since then, Smith points out, of the 35 companies with publicly traded stocks, 20 have done worse than the market average.