Don't be overconfident in investing

(MoneyWatch) Confidence is good. Overconfidence, however, can have devastating effects on portfolios by causing investors to make a host of costly errors. This isn't just limited to our own investing beliefs. It also stems from listening to experts who are "certain" about their forecasts.

Consider the following examples provided by statistician and New York Times columist Nate Silver in his book "The Signal and the Noise":

• University of Pennsylvania professor Philip Tetlock found that when political scientists claimed that a political outcome had absolutely no chance of occurring, it happened about 15 percent of the time.
• The Fujikama nuclear reactor damaged in last year's Japanese tsunami had been designed to handle a magnitude 8.6 earthquake because seismologists had concluded that anything larger was impossible. The quake that hit Japan was magnitude 9.1.
• The Philadelphia Federal Reserve's survey of professional forecasters asks economists what their 90 percent confidence interval is, meaning they're 90 percent certain the results will be between these two numbers. Since 1968, the growth of the economy was outside the 90 percent confidence range almost half the time.

Because we hate uncertainty, we seek certainty, even where it doesn't exist. We listen to overconfident forecasters and practitioners, which in turn makes us overconfident about our own investing abilities. We end up:

• Taking more risk than we have the ability, willingness or need to take
• Failing to diversify the risks we do take, concentrating our risks in one or a few baskets
• Investing disproportionately in local companies, our own employers and companies whose products we use
• Choosing active fund managers despite the overwhelming evidence that they tend to underperform
  
• Trading too often

So-called "hindsight bias" also leads to overconfidence. With the benefit of hindsight, even events that seemed highly unlikely before the fact become painfully obvious after the fact -- after all, the handwriting was on the wall! That bias can make us overconfident in our ability to predict the future.

Overconfidence is also linked to what is known as "confirmation bias" -- the tendency to give more weight to evidence that confirms our beliefs, regardless of whether the information is true or not, than to evidence that contradicts them. As a result, we tend to gather evidence and recall information from memory selectively, and interpret it in a biased way. And confirmation bias leads to more mistakes because of groupthink.

• Stock investing: Behaviors that stop diversification
• New book sheds bright light on investor behavior
• Indexing is for those who aren't smart enough

Forecasting models that seem to offer precise explanations of how the world works also create overconfidence. While some models are useful -- they are tools that help us understand how the world works -- by definition all models are wrong because they are highly simplified versions of the world.

Most interesting to me is that even many investors who are willing to admit that it's hard to beat the market are confident they will be successful. Here's what economist Peter Bernstein has to say on that point:

Active management is extraordinarily difficult, because there are so many knowledgeable investors and information does move so fast. The market is hard to beat. There are a lot of smart people trying to do the same thing. Nobody's saying that it's easy. But possible? Yes.

It's that slim possibility that keeps hope alive. Overconfidence leads investors to believe they will be one of the few that succeeds.