By Richard Alford, a former New York Fed economist. Since then, he has worked in the financial industry as a trading floor economist and strategist on both the sell side and the buy side.
In theory, there is no difference between theory and practice. In practice, there is. – Yogi Berra
Economic policymakers, pundits and academics continue to forecast the future course of the economy and predict the effects of possible policy initiatives with an air of scientific certainty. The high degree of confidence expressed in their forecasts, predictions and commentary continues unabated despite:
1. Only a small minority of economists and none of the central banks and treasury/finance ministries anticipated the financial crisis and the recession, and
2. At most only one of the currently competing macroeconomic models (which embody significantly different structures and implications for economic policy) can serve as a sound basis for policy.
The confidence and false precision in these forecasts and policy prescriptions reflect a continuing unwarranted faith in the models. With Greenspan’s defense, “no one saw it coming” and Bernanke’s admission that monetary policy is no panacea still fresh, some economists, including Brad DeLong, are now asserting that they “have reason to be proud of our analyses over the last five years.” This assertion flies in the face of the fact that macroeconomic policymakers and both “freshwater” and “saltwater” economists had embraced a model (Dynamic Stochastic General Equilibrium) that assumed away the importance/existence of a financial sector. This recasting of recent history would make Orwell proud.
The undiminished confidence of mainstream economists also raises a number of questions:
1. Why do economists, policymakers and pundits continue to express high degrees of confidence when forecasting or commenting on the merits of alternative policy prescriptions given their recent dramatic failure to forecast the crisis and recession and their widely divergent models?
2. Does the dynamism and complexity of the economy allow for a model(s) that can predict the future course of the economy and the efficacy of macroeconomic policy with the precision and degree of confidence that these economists and policymakers assert?
Undue Confidence and False Precision
Bernanke’s statement that monetary policy is no panacea reflects the degree to which economists, policymakers, and politicians have oversold the ability of economics and economic policy to deliver satisfactory economic performance. Ironically, no groups have been bigger buyers of the ability of macroeconomic policy to ensure satisfactory, if not optimal, outcomes than macroeconomists and policymakers themselves.
The majority of mainstream economists and policymakers failed to see, comprehend and react to the imbalances and unsustainabilities that became integral parts of the calm before the storm (aka the “Great Moderation”) and that contributed to the severity of the financial crisis and the recession. They acted and set policy as if:
1. Any projected path for the economy that was inconsistent with their model was out of the realm of possibility and therefore irrelevant, and
2. Any divergence between the model’s predictions and the real economy would be corrected by the real economy moving back into line with the model.
This suggests that macroeconomists have committed an error of misplaced concreteness, i.e., they confuse their models of the economy with the real economy itself. This confusion of the model with the real economy is hazardous. In 2010, Ricardo Caballero of MIT put it this way:
… the so-called dynamic stochastic general equilibrium approach—has become so mesmerized with its own internal logic that it has begun to confuse the precision it has achieved about its own world with the precision that it has about the real one.
Caballero also noted:
…This is dangerous for both methodological and policy reasons. On the methodology front, macroeconomic research has been in “fine-tuning” mode within the local-maximum of the dynamic stochastic general equilibrium world, when we should be in “broad-exploration” mode. We are too far from absolute truth to be so specialized and to make the kind of confident quantitative claims that often emerge from the core. On the policy front, this confused precision creates the illusion that a minor adjustment in the standard policy framework will prevent future crises, and by doing so it leaves us overly exposed to the new and unexpected.
However, since the financial crisis and the recession, there has been very little in the way of a shakeup in the economics profession. True to form, the economists are pursuing only marginal changes in their models. Mainstream defenders of the current paradigm argue that the crisis and the recession were the result of regulatory failures alone and that macroeconomic models and policy are blameless and do not require significant alteration. Non-mainstream economists find fault in the reigning model and seek to replace it with a model of their own design.
The differing camps in the debates have deeply held world views and different models. Each camp claims that their findings are truly scientific and that anyone who disagrees with them must be mentally deficient, ethically challenged, and genetically unsound. Each camp has a shared economic model and places great faith in the ability of the model to predict, explain and serve as a policy tool. However, despite all the disagreement and name calling, the economists engaged in the debates share one fundamental position: economics is a science and potentially of great utility if their model is employed.
In recent debates about the fiscal stimulus, numerous economists and pundits opined with high degrees of confidence on the effectiveness of fiscal stimulus. The statements were either implicitly or explicitly based on an assumed value for the fiscal multiplier. Given that the fiscal multiplier has been around since Keynes wrote “The General Theory…” one might imagine that a consensus about the value of the fiscal multiplier has emerged. However, a CBO report titled “Estimated Impact of the American Recovery and Reinvestment Act (ARRA) on Employment and Economic Output from October 2011 Through December 2011” (February 2012) pointed out:
A recent survey of studies based on historical evidence shows that estimates of fiscal multipliers range from -0.3 to 3.6, although most of the estimates fall between 0.5 and 2.0. Several of those studies yield estimates that average between 0.5 and 1.0 over a long historical period. However, those studies do not specifically provide estimates for a period in which unemployment is high and interest rates are very low, as is true for the current period.
The range of estimated values of the multiplier based on the historical record is inconsistent with the confidence expressed in many public statements about the impact of fiscal stimulus. Ex post efforts at quantifying the impact of the ARRA also show a wide range of values for the implied multipliers. For example, the CBO’s report on the effects of the ARRA (actual Q4 2011 compared with what would have occurred in the absence of the ARRA) is as follows:
1. It raised real (inflation-adjusted) gross domestic product (GDP) between 0.2 and 1.5 percent,
2. It lowered the unemployment rate between 0.2 and 1.1 percentage points,
3. It increased the number of people employed between 0.3 and 2.0 million, and
4. It increased the number of full-time-equivalent jobs by 0.4 to 2.6 million.
The estimates at the high end of the CBO’s range of ARRA’s effect on real GDP growth, the number of people employed, and the increased number of FTEs are about 7x the size of the low end of the range. Keep in mind that these are ex post efforts at quantification and not the less certain ex ante forecasts. The range, 7x the size of the low estimate, is also considerably larger than the 4x range in the CBO’s reported central tendency of economists’ estimates of the value of the multiplier.
The sizes of the ranges are inconsistent with the apparent degree of confidence that individual economists and camps of economists place in their models and forecasts. Recent research may help to explain the confidence placed in forecasts. A paper by Soyer and Hogarth shows that the standard method of presenting regression results in the economic literature demonstrates an “illusion of predictability,” i.e., statistically significant average effects are emphasized at the expense of the distribution of outcomes:
… the way in which results are presented in regression analyses obfuscates the uncertainty inherent in the dependent variable.
Soyer and Hogarth couch their result in a policy framework:
… consider a decision maker who is pondering which actions to take and how much to do so in order to reach a certain goal. This requires conjectures to be formed individual outcomes that would result from specific inputs. Moreover, the answers to these questions depend not only on estimating average effects, but also on the distribution of possible effects around the average.
US monetary policy has been guided by the Taylor Rule. The Taylor Rule is a decision rule designed to minimize the loss to society from deviations of inflation and unemployment from target, given estimated parameters and weights. The distribution of potential outcomes is not reflected in the Taylor rule.
The Limits of Economics as a Predictive Science
Well into 2007, economists, pundits and policymakers were confidently predicting a continuation of the Great Moderation, and many were issuing assurances that there was no housing price bubble. Does this failure reflect problems with economists, policymakers and their models? Or is the problem inherent in the nature of the economy? If the failure of mainstream models to foresee the crisis and recession is the result of economists having too much faith in a faulty model, then changing the economists or the model could rectify the problem. If, on the other hand, the nature of the macroeconomic problem is such that models that allow for forecasts and policy design with the required accuracy are impossible, than how should policy be designed, implemented, and communicated?
While many economists would argue that economics is on a solid scientific foundation, others would disagree. Caballero of MIT quotes von Hayek:
…In his Nobel-prize acceptance lecture, Hayek writes: “Of course, compared with the precise predictions we have learnt to expect in the physical sciences, this sort of mere pattern predictions is a second best with which one does not like to have to be content. Yet the danger of which I want to warn is precisely the belief that in order to have a claim to be accepted as scientific it is necessary to achieve more. This way lies charlatanism and worse. To act on the belief that we possess the knowledge and the power which enable us to shape the process of society entirely to our liking, knowledge which in fact we do not possess, is likely to make us do much harm” (von Hayek, 1974).
Karl Popper and other philosophers of science would also argue that economics, among other social sciences, would never exhibit the explanatory and predictive power of a hard science, e.g., physics:
Long-term prophecies can be derived from scientific conditional predictions only if they to systems which can be described as well isolated, stationary, and recurrent. These systems are very rare in nature; and modern society is not one of them.
Economists have long been aware that the structure and behavior of an economy is not stationary, but rather evolves over time and in response to changes in policy regimes. This is reflected in the Lucas Critique, which links adjustments in economic policy to changes in the decision-making rules employed by economic agents and hence the behavior of the economy. Goodhart’s Law states the economic policy dimension of the problem succinctly:
Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.
Put another way, when policymakers implement policy based on a statistically identified past pattern of behavior of economic agents, that behavior will change. As a result, the economy will not evolve or respond to the policy as predicted. The existence of feedback loops is not limited to macroeconomics. It has been noted elsewhere in the social sciences, i.e., Campbell’s Law:
The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.
If economic policymakers had considered von Hayek, Popper, the Lucas Critique, Goodhart’s Law, or Campbell’s Law, then they would not have been overly confident in the ability of the announced low-for-long interest rate policy and the tightening at a constant, well-telegraphed “measured pace” to produce either stable growth or financial stability. Furthermore, they would have been skeptical of the regulator-imposed Value-at-Risk system’s ability to measure and control risk in markets.
This does not imply that modeling is a dead end. Caballero again:
…There is no doubt that the formalization of macroeconomics over recent decades has increased its potential. We just need to be careful to not let this formalization gain its own life and distract us from the ultimate goal, which is to understand the mechanisms that drive the real economy.
The world is dynamic and economic agents adapt quickly to change if they have not already anticipated it. Consequently, forecasters and policymakers must deal with uncertainty despite the fact that economic agents in general want certainty and that economic models must be tractable in order to be useful.
Policy Risk and Policy Risk Management
In a 1967 American Economic Review article, Brainard showed that uncertainties about the state of the world and the effectiveness of policy imply that economic policy is risky. The riskiness implies that policymakers should take into account not only the most likely effect of the policy as predicted by the underlying model, but the range of possible outcomes as well.
This in turn points to the need for a policy risk management discipline. The idea of risk-adjusted policy may appear novel and non-operational, but it is not. In the mid-1990s, US monetary policy reflected an effort to incorporate policy risk management as it dealt with the implications of forecast errors and policy lags. In “Central Banking Theory and Practice,” (MIT Press,1998), former Vice Chairman of the Board of Governors of the Federal Reserve System Alan Blinder shed light on the Fed’s procedure given the desire to move preemptively despite the uncertainty/risk:
A preemptive strategy implies a certain amount of confidence in both your forecast and your model of how monetary policy affects the economy, both of which are hazardous. But preemption does not require too much confidence. Remember the flexibility principle of dynamic programming and the Brainard conservation principle. Taken together, they led to the following sort of strategy:
Step 1- Estimate how much you need to tighten or loosen policy to “get it right.” Then do less.
Step 2- Watch developments.
Step 3a -If things work out about as expected, increase your tightening or loosening to where you thought it should be in the first place.
Step 3b- If the economy seems to be evolving differently from what you expected, adjust policy accordingly.
Policy can be appropriate. It can also be or become inappropriate. Policy uncertainty precludes “Set it and forget it!” policy design as well as implementation. The underlying uncertainty necessitates a certain tentativeness and constant monitoring of economic developments by policymakers in order to gauge whether or not the economy is evolving as anticipated. Given the Lucas Critique and Goodhart’s Law, the policymakers must also have open minds and actively look for “the new and unexpected.”
The Fed practiced a form of policy risk management in the mid-1990s. However, post the NASDQ bubble the Fed set monetary policy based on expected returns only and ignored the new and unexpected. It now appears that the FOMC is again considering the risks as well as the expected return as it ponders additional unconventional policy steps. The uncertainty and risks attached to additional policy steps are currently high relative to the expected returns. The economy, the financial markets and policy are in unchartered waters and the unconventional policy steps are not as effective as conventional policy was in affecting the real economy. Given the need to adopt a policy risk management regime in the mid-1990s, the current heightened uncertainties, and the relatively low expected returns to further unconventional steps, one should not be surprised by the FOMC’s recent reluctance to take additional steps.
Economists, policymakers, portfolio and asset managers face tradeoffs between risks and returns. Asset and portfolio managers should be called to task if they ignore risks and focused solely on expected return. So should policymakers. Macroeconomic modelers and policymakers might consider holding themselves to a standard similar to the standard suggested for financial modelers by Paul Wilmott and Emanuel Derman in their “The Financial Modelers’ Manifesto,” e.g.:
The Modelers’ Hippocratic Oath
~ I will remember that I didn’t make the world, and it doesn’t satisfy my equations.
~ Though I will use models boldly to estimate value, I will not be overly impressed by mathematics.
~ I will never sacrifice reality for elegance without explaining why I have done so.
~ Nor will I give the people who use my model false comfort about its accuracy. Instead, I will make explicit its assumptions and oversights.
~ I understand that my work may have enormous effects on society and the economy, many of them beyond my comprehension.
Blinder, Alan, Central Banking in Theory and Practice, MIT Press, 1998, page 17.
Brainard, William, “Uncertainty and the Effectiveness of Policy”, AER, vol. 57, No. 2, (May 1967) pp.411-425.
Ricardo Caballero, Macroeconomics after the Crisis: Time to Deal with the Pretense-of-Knowledge Syndrome , http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1683617
CBO, Estimated Impact of the American Recovery and Reinvestment Act on Employment and Economic Output from October 2011 Through December 2011, http://www.cbo.gov/publication/43013
Emre Soyer and Robin Hogarth, The Illusion of Predictability: How Regression Statistics Mislead Experts, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1996568
Paul WIlmott and Emanuel Derman , “The Financial Modelers’ Manifesto”, :http://www.wilmott.com/blogs/eman/index.cfm/2009/1/8/The-Financial-Modelers-Manifesto
The next 93.568% of economic numbers reported will be within 10% of the past economic numbers reported 91.483% of the time, with the cohort within 20% of the economic numbers reported 4.942% of the time, and the remainding 5% or so not. This means a reduction in Baltic dry goods shipping cardboard box utilization index 49.952% of the time.
-Economics always has been and will always be a social “science,” attempting to model human behavior, and not merely theoretical “economic performance,” whatever in the hell that is. Whereas scientific phenomena can be quantified and predicted to some extent, thus far at least, human behavior as studied by humans themselves cannot.
-Economists have misapplied mathematical equations borrowed wholly from the hard sciences (exponential growth anyone?) in a simplistic attempt to justify their academic claims to scientific legitimacy.
-Economists are wholly owned by politicians and whomever else they can latch on to legitimize their very questionable “academic research” and findings, and thus are inherently corrupt. Want a hair-brained theory of anything you like supported? Hire an economist. Better yet, hire a team of economists. Hey, economists are out of work these days too.
-In earlier times we would have called economists “snake oil salesmen” or the like. Now we realize that neither snakes nor mere garden variety salesman deserve such an unflattering comparison.
… the so-called dynamic stochastic general equilibrium approach…
-Economists have become mesmerized by their own “jargon,” aka “bullshit.”
“dynamic stochastic general equilibrium approach” = grandiloquent hyperbolic pseudo-academic linguistic affectation. See there, I’m a natural economist! All I need is some letters after my name.
Economics has always been religion, not science. It is the religion of the corporate ruling class, which rewards fidelity and punishes deviation. This explains why its theories and conclusions and models are not worth the powder to blow them up. The only American economist of the past one hundred fifty years who said anything important was Veblen, and you will never hear his name mentioned in the profession except dismissively. Long winded dissertations concerning why this or that model is incorrect are worse than useless, since they imply that results might be better with a few savvy changes in this or that equation or parameter. The only reason economists still have a shred of credibility is their propaganda value given the public’s mystical faith in respectable prattle. These same charlatans never tire of reminding us about the non existant deficit crisis. It is all politics and nothing else.
What a brilliant analysis. But let’s not forget that, accepting the uncertainties of economic policy (and many other types of social policy), it is important to advocate policies that, if they are successful, will produce EQUITABLE and BENEFICIAL results for the greatest number of people.
Quite a few economists were right all along regarding the financial crisis that started around 2006-2007. Krugman and Roubini were both painfully accurate in the analysis and predication of our current financial misery. There are quite a few other economists who smell like roses too.
Economic pundits love to pass judgment on anything and everything market related not necessarily with a full deck of cards. Personally, I’d appreciate of NC will stay away from this pundit orgy.
Love the Hippocratic Oath. More of this kind of critique is needed, both in economics and law:
Nearly 20 years ago my friend Robert Goldfarb, then professor of economics at Washington University, published an article on the instability of econometric estimates. He started with a simple proposition: we can’t be experts in all the specialist domains of economics, and therefore have to rely on estimates of, say, the demand elasticity for labour, on other specialists. So he went to the specialists in different areas and asked them to give him their best estimate, and also give the literature on the estimates. He discovered that in general there were huge differences and in many cases sign reversals from one estimate to the next.
Reasons: new data sets; new methodology, and the fact that no one gets tenure in economics for verifying someone else’s work. You get tenure for a ‘new’ finding, which is necessarily different from the older ones; you get a line on the vita. And so in the end, there were no reliable estimates. This is entirely apart from the ideologically driven work.
What I find interesting as well as telling and not surprising is that
after President Obama let the big bankers and wall streeters off the
hook for causing the financial collapse of the world by not firing
anyone or prosecuting any of the upper management types, these same
people, without showing any gratitude to President Obama, now support
the mitt because the mitt will cut their regulations even more and cut
their taxes with letting them outsource as many jobs as they want along
with being left alone to rape and plunder as they see fit. Once again
money speaks louder than ethics and morality.
I am not too concerned with the failure to predict our current depression. How many depressions have we had since the Industrial Revolution? I am concerned with the apparent certainty of economic predictions. One group that is quite certain of their predictions is psychics. From time to time I respond to charts of economic projections on the web with GIGO! GIGO! GIGO! I –ahem– confidently do so because of the lack of any estimation of error. The failure to indicate possible error undermines my confidence in the chart. I cannot trust it. Furthermore, I know that extrapolation (projection) is a dicey business. As a rule of thumb, even if you have a trend where the relevant conditions are fairly constant, projecting ahead more than 10% of the length of the trend is not a good idea. Mark Twain once noted that, at the current growth of the delta of the Mississippi River into the Gulf of Mexico, Louisiana would connect to the Yucatan in around 150 years (or the like, my memory is not exact). He understood that such a projection was absurd. But people are still frightened by projections such as, “Entitlements will eat up taxes by 2030!”
Here is an official chart from the CBO: http://www.usgovernmentspending.com/budget_cbo.php?year=2000_2087&show=frame
At first glance, this is an alarming chart. But then you see that most of the chart is a projection, from now, 2012, up to 2087. If you think about the 10% rule of thumb, that is ridiculous. The US has not even existed for 750 years! The major projected growth is in the area of health spending. In fact, that accounts for the scary part of the graph. We did not need the graph to tell us that health costs in the US are a matter of concern. Its explosive growth since 1980 is evidence enough for that. Using the 10% rule, I would be willing to credit this chart 3-4 years in advance. After that, quien sabe? As for the 2087 projection, I do not expect to be alive then, but I would be quite happy to leave my heirs and their heirs the results of a 50-50 bet that the US gov’t will be spending less on health in 2087 than the projected percentage of GDP. (Thanks to Mark Twain. :))
On a positive note, the Bank of England does it right. Look at these charts: http://www.bankofengland.co.uk/publications/PublishingImages/inflationreport/cpimktmay12large.gif
Note that in both charts it is obvious what is history and what is projection. Neither chart projects very far into the future. If you compare these charts with the CBO chart, you have to laugh at the CBO chart. If the CBO chart were done in the same style, most of it would just be a blur. We have no reason to think that the CBO is better at economic projections than the Bank of England. They are probably of comparable ability. However, the Bank of England shows the proper scientific attitude. :)
This reminds me of a chart I saw recently, that was presented at an anti-vaccination conference.
The chart takes two small values from past years, extrapolates them exponentially into the future without slowing down, and shows that by 2038 all children will be autistic. Some highlights of the chart are:
– there is no distinction shown between data and extrapolation;
– the exponential curve does a little kink just before it hits 100%, because the penultimate data point is under 100%, and the last one would be greater than 100% if it was not manually constrained;
– all children become autistic by 2038. Three years later all girls become autistic. Are girls not children?
You may say this is just vaccine denialists, but my point is that there’s a reason such poor quality thrives, which is that quality is not the point. This is just rich men paying for time-consuming nonsense to be spread around. You see the same thing with creationism, global warming denial, and peak oil denial.
I think much of economics is in a similar position: rich men pay money for time-wasting nonsense to be spread around. Nonsense works as well as sense for the purpose, and is cheaper to produce.
Because economists are human, they have biases (conscious or unconscious) about how they think the economy should work. Aeronautical engineers also have biases about the strength and fatigue limits of materials, but when the planes they design fall out of the sky, those biases get corrected pretty quickly. If this happens too often, they have a hard time finding employment. This kind of bias correction doesn’t seem to happen in economics. (Of course, economists whose biases align with the politically powerful can be rewarded regardless of outcome.)
Many will use the widely repeated excuse that the economy is just too complicated, and “No one could have forseen this”, but actually, a large number of economists did forsee the economic recession. You can find a list of them in the paper at http://mpra.ub.uni-muenchen.de/15892/.
Admittedly, some of them may have been biased, and their predictions ended up being correct only by accident, but one can assume that most of them correctly predicted the future because their models are correct. They’re the ones I’ll be riding with in the future.
Ya I bet if you start tracking models and assumptions at main policy making bodies you could shine a light on some dubious practices in both