Why do so many investors ignore the evidence?

(MoneyWatch) One of the great investing anomalies, and a great frustration for me, is that the evidence against active management and for indexing (or passive investing) is so overwhelming, yet the majority of investors keep playing a loser's game. I offer three explanations for this phenomenon.

First, the education system has failed the public. It's likely that most people haven't taken a single course in capital markets theory. Without the necessary knowledge, how can they know whether an active or a passive strategy is the right one? The result is that they get their "knowledge" about investing from the very people -- Wall Street and the financial media -- who don't have their interests at heart. Wall Street needs investors to trade a lot and to pay the high fees imposed by actively managed funds. The media needs investors to "tune in." The result is that most investors are unaware of the historical evidence.

• Where are the benefits of international diversification?
• Endowment investing is not for everyone
• Fast firm growth doesn't mean great stock returns

Second, despite the importance of the issue, the public seems unwilling to invest the time and effort to overcome the failings of the education system. Instead of reading my books or ones by John Bogle, William Bernstein and my colleagues Dan Solin and Carl Richards, they would rather watch CNBC to hear the latest guru's forecast.

The third explanation is what we might call the Lake Wobegon effect -- the need and/or desire to be above average. This seems especially true of high-net-worth people. They want and/or need to feel special, especially when it comes to investments. They want to be members of the "in crowd," with access to investments that the hoi polloi don't have access to. That provides them with a feeling of prestige and sophistication. In other words, they want more than just returns from their investments.

Wall Street plays on that need, saying something like this: "Indexing is a good strategy, but it gets you average returns. You don't want to be average. We can help you do better than that." The truth is that indexing doesn't get you average returns. It gets you market returns. And because it does so with lower costs and greater tax efficiency, by definition you earn above-average returns -- as long as you have the discipline to stay the course. This is about the only guarantee there is in investing.

Why do people ignore all the evidence? Why do so many investors continue to play a game they're not likely to win? Kathryn Schulz, author of "Being Wrong," provides us with some fascinating insights that help explain this phenomenon, one that allows Wall Street to transfer tens of billions every year from the pockets of investors to their pockets.

Schulz explains that most of us go through life assuming that "we are basically right, basically all the time, about basically everything." And that "our indiscriminate enjoyment of being right is matched by an almost equally indiscriminate feeling that we are right. Occasionally, this feeling spills into the foreground as we make predictions or place bets." I would add this feeling also affects us when we make investments.

She goes on to explain that often this confidence is justified as we "navigate day-to-day life fairly well." This suggests that we're right about most things. Schulz's book is all about the opposite of that. It's about being wrong and what happens when our convictions collapse around us -- we often feel foolish and ashamed, as error is often associated with "ignorance, psychopathy and even moral degeneracy."

Schulz explained that we tend to view errors as things that happen to others, yet somehow we feel that it is implausible that they'll happen to us. She believes this is because "our beliefs are inextricable from our identities." "We're so emotionally invested in our beliefs that we are unable or unwilling to recognize them as anything but the inviolate truth." She said that "we tend to fall in love with our beliefs once we have formed them." And that explains why "being wrong can so easily wound our sense of self." It also explains why we experience cognitive dissonance -- the uncomfortable feeling and/or anxiety we feel when someone disproves a long-held belief. It also explains why we ignore evidence -- even when it is compelling -- and why we resist change.

Schulz notes that the view that others make errors but not us is the greatest error of them all, as "wrongness" is a vital part of how we learn and change. In other words, even smart people make mistakes. (For example, I used to be an active investor.) However, once smart people learn that a behavior is a mistake, they change their ways. This is what separates them from fools who keep repeating the same mistakes while expecting different outcomes.

Among the many insights Schulz provides is that our ability to forget our mistakes is keener than our ability to remember them. During her research, Schulz met many people who said she should interview them as they make mistakes all the time. Yet when asked to gives specific examples of their mistakes, they were hard-pressed to come up with any. The inability to remember mistakes leads to overconfidence, which in turn leads to other mistakes, especially investment mistakes -- such as taking too much risk and failing to diversify -- which can be very expensive.

Image courtesy of taxbrackets.org.