How do economists figure out how the world really works?

Many people believe there has been no progress in economics, but that isn't true. For example, one of the most important questions in the 1970s and 1980s was whether monetary policy could be used to stabilize the macroeconomy. One popular theoretical model, known as the New Classical model, implied that monetary policy could not affect output and employment, and hence was of no use in trying to offset cyclical fluctuations in these variables.

Economists call this "money neutrality." A competing theoretical model, the Keynesian model, asserts that money is non-neutral. In these models, monetary policy is a useful tool to stabilize fluctuations in output and employment.

The economics profession was split between these two theoretical camps, and there was great passion on both sides. This left monetary policymakers in a quandary. If money was neutral, the best policy was to simply stabilize interest rates and the money supply to whatever extent possible. But if money was non-neutral, then interest rates and the money supply should be changed in response to macroeconomic conditions.

How can this question be settled? How can we decide which model is correct? By testing their implications against real world data using econometric techniques. If, in real world data, changes in monetary policy affect output and employment, then money is non-neutral, and if it doesn't, it's neutral.

When these tests were performed, the evidence pointed to non-neutrality, and today the passionate debate over this issue is all but over. There are still a few economists who, despite the overwhelming evidence to the contrary, believe in neutrality, but they are a small minority.

Thus, progress in economics depends upon the use of econometric techniques to test hypotheses and distinguish between competing theoretical structures

What is econometrics?

Econometrics is a set of statistical tools that allow researchers to test economic theory against real world data, and to forecast the future of the economy. For example, a particular theory might hypothesize that when the government spends more through deficit spending, it drives interest rates higher. This hypothesis could then be taken to the data, and tested using econometric techniques.

If the data are inconsistent with the hypothesis, then that is evidence that the theory is wrong. If it passes the test, we cannot say for sure that the theory is correct, it may fail along other lines, but it does provide support for the particular theory under consideration.

But why do we have a separate discipline called econometrics? Isn't it just statistics? Can't economists simply adopt the statistical tools and techniques used in other disciplines?

The essential difference between statistics and econometrics is the inability of economists to perform laboratory experiments where the effect of one variable on another can be examined while holding "all else equal."

Economists must rely on historical data as it comes to them -- they cannot, for example, re-run the macroeconomy again and again and examine how well various policy interventions might work. Ideally, for example, for research purposes we would run an experiment in which the Great Recession happens again and again. The we could see how well various policy interventions perform while holding all influences except for that policy constant (or do the same intervention again and again to smooth out any randomness in the outcome that might distort the findings).

Thus, while data from laboratory experiments is usually confined to two variables and the experiments can be performed repeatedly to ensure a single observation is not a statistical fluke, economists must use historical data -- a single observation -- where all else is definitely not equal.

Regress this

For this reason, economists use a technique called multiple regression analysis. Essentially, what this means is that the effect of the treatment on the outcome is examined by including a (sometimes large) set of controls to account for all of the variables that cannot be held constant.

For example, in the example above, it would be important to include all of the other variables besides government spending that might influence interest rates. It is not sufficient to simply examine the correlation between those two variables. Failure to include these controls (which are generally not needed when laboratory experiments are performed since they are held constant) can cause all sorts of problems, from bias in the outcome of the tests to a failure to isolate the particular relationship the investigator is interested in.

If the researcher includes the right set of controls and accounts for other statistical properties of the empirical model, this technique works fairly well. But there is one important caveat, something that is particularly problematic for testing macroeconomic theories. Most macroeconomists know the data on GDP, employment prices, interest rates, productivity and so on fairly well. So it is not very useful to build a theoretical model to explain these data, and then test to see how well it fits. Of course the model would fit. After all, why build a model that is inconsistent with the data you already know about?

And that is the key -- to use data researchers did not know about when the model was built. Testing models against data that is revealed only after the model is built is the best way to do this. That is especially useful when, as with the Great Recession, the economy departs from historical norms (Notably, most macroeconomic models failed during this period, and the race is on to build new models that can account for such macroeconomic outcomes.)

Because economists have no choice but to use non-experimental data when testing theories against the actual outcomes of the economy, the techniques can become very complicated. In addition to the inclusion of a large set of controls, there are also issues involving the statistical properties of the model.

But in the end, the goal is the same: to find a way to distinguish good models from bad ones, and move the profession forward. These complications make progress slower then it would be if economists could do their analysis in a lab. But although the progress is slow, and sometimes hard to see, there is progress nonetheless.

CBS MoneyWatch contributor Mark Thoma is an economist at the University of Oregon. He also writes the blog Economist's View.