Yves here. The post below, which discusses the limits of how “scientific” a social science can be, will strike a lot of readers as stating the obvious. But I’m posting this to remind readers that the economics profession’s scientific pretensions have led economists to have vastly more policy influence and be better paid than other social scientists. I have no doubt court soothsayers had similarly prominent roles.
In a bit of synchronicity, Andrew Gellman posted Friday on the same issue, citing an article by Douglas Campbell (sadly, there seems to be no link to it, and I Googled some of the text from it and could not find the underlying paper). Key bits from Campbell:
A new study finding that more than half of psychology studies failed to replicate is a very positive step forward for social science. Could a similar study be undertaken in economics, and what would it find? Most empirical economics research is non-experimental, and thus I suspect that most studies would replicate in the sense that if one used the same data and ran the exact same regressions, the results are unlikely to change. However, if one were also to test the robustness of results to additional (or fewer) control variables, differing estimation approaches, or try out-of-sample testing on new data, I suspect less than half would survive….While I knew the conventional wisdom that it’s a bad idea to write “comment papers” in economics, eventually I became curious what would happen if I tried to take down a “seminal” paper published in a top journal.
One paper I had been assigned to read in several graduate courses, on “The Diffusion of Development,” published in the QJE, a leading economics journal published at Harvard, argued that there is a causal link between a society’s average skull size “genetic distance to the US” and its GDP per capita. The authors were careful to point out that their results didn’t necessarily indicate a direct impact of genetic traits on economic development, but that genetic distance could be a proxy for a whole host of other cultural traits which could impact the transmission of technology. However, in my view this point was undercut by the authors’ assertion that the apparent impact of genetic distance on GDP per capita survives the inclusion of an ostensibly exhaustive list of geographic and cultural controls. This suggests that genetic distance may not merely proxy differences in cultural traits, but has a direct impact on GDP. Thus black Africa may be poor because of its genetic endowment, and white Europe rich for the same reason.
Except, …, wait a second here before we go leaping to conclusions….
How exhaustive were those geographic and cultural controls? A coauthor, Ju Hyun Pyun and I noticed that the authors did not even control for latitude or for a dummy for sub-Saharan Africa in their cross-country income regressions, even as they argued that their results were robust to controls for geographic regions. When we included these controls (standard in this literature) in the first regression we ran, the correlation between genetic distance to the US and development disappeared. We also found that genetic distance to the US failed to predict income levels even when we just included two dummy variables, one for Europe and one for sub-Saharan Africa, with no other controls. Thus, the original findings were equivalent to the observation that white Europe is rich and black Africa is poor, with no more explanatory power than that. While we felt our results were perfectly straightforward, it took us seven submissions and four years to publish our results in a minor journal. Meanwhile, results similar to those we had critiqued continued to be published in leading journals, including one of the same journals where our paper was rejected. We often had to contend with the original authors as referees – once as the sole referee. (Pro tip: if writing a paper like this, recommend to the editor that they not choose the hostile original authors as referees.) One editor sided with a creative referee who objected to our paper on the grounds that “There is no reason to interpret the sub-Saharan dummy as a ‘geographic variable’”. The same referee also zinged us for not including the exact same sample as the original paper, even though the data and original sample were not publicly (or privately) available. This was hardly the type of hassle-to-reward ratio which would lead me to write a similar such paper, at least before tenure. Instead, had we decided to write an “extension” paper, using the genetic distance data to predict some other variable, publication would have been facilitated, since the original authors would have been likely referees and would have been happy to see our results published. The incentive structure here could be improved.
The comments on the Gelman post are also instructive.
The post below focuses more on the bigger “political economy,” as in social engineering questions, that economists, with their pretenses of objectivity, try to pretend is not one of the core aims of their discipline. Yet by deciding what outcomes are to be sought, economists have already put a very big finger on the scales even before the policy design issues are on the table.
Economics became important due to the Depression and the Cold War. Political leaders saw how the destructions of wealth and livelihoods led to calls for radical solutions, and the perceived radicalism of Roosevelt was tame compared to what could have resulted. The rapid industrialization of Russia impressed upon the West what a command and control economy could do, and that it had the potential to outproduce liberal democracies.
Economists were happy, indeed eager, to offer solutions to the problems of how to create more stability (as in hold internal Communists at bay) and growth (to compete with the USSR from a manufacturing standpoint). But as the Communist threat has faded, economists have become less concerned with promoting wage growth via sharing the benefits of productivity gains with workers. More and more are explicitly or tacitly of the view that giving businesses more free rein will lead to higher growth level, which will trickle down to workers. The record of the past 35 years has shown more frequent and intense crises, underinvestment by businesses (perversely mislabeled as a “savings glut), falling growth rates, rising inequality, and falling social indicators. Yet the orthodox prescriptions that have produced such lousy result remain popular via having created a cadre of the top rich and their well-paid technocratic advisors who very much want them to continue.
By Alexander Krauss, a postdoctoral research fellow at the London School of Economics. Originally published at the Institute for New Economic Thinking website
Since Aristotle the question about the potential relationship between economic inequality and democratic changes has been studied and debated – but scientifically our ability as researchers to assess and understand how such complex social phenomena may be related is much more limited than recognised.
The existing literature is laden with contradictory hypotheses and findings that suggest this potential relationship can be positive or negative, stronger or weaker, differentiated or non-existent and can vary across and within countries and time periods. However, fundamental methodological and empirical limitations of analysis do not allow us to make such claims robustly, to some extent because the process of democratisation and changes in levels of equality are highly nuanced, idiosyncratic and heterogeneous and thus difficult to capture econometrically. Some of the most prominent authors in this literature claim that high levels of inequality decrease the likelihood of democratisation, and they also talk about “causal effects” and “the impact of democracy” on outcomes. Such conclusions presuppose a number of very demanding assumptions and requisite premises that cannot be rigorously met.
In fact, thousands of academic papers analyse the potential relationship between political variables like democracy and economic variables like inequality by gathering their data, selecting their methods and then going forward with their analysis, interpreting their findings and potentially informing policy, with many other steps along the way that involve making important implicit methodological assumptions. My recent paper1 instead goes backwards to analyse whether the data and methods that are applied by the leading authors in this literature are able to produce the robust results that they claim. It emphasises that how we as researchers generate our correlational (or ‘causal’) claims cannot be viewed independently from how we make everyday, typically unreflective decisions, such as what we decide to analyse, how we construct our variables, how we collect and use our data, which methods we choose to apply, how we interpret our statistical results, and so forth.
Better understanding the methodological and empirical limits of analysing the potential relationship between phenomena like inequality and political regimes is important for both research and policy because leading economists and other social scientists misguidedly claim to establish causal relationships but at times still inform public policy and thus can bring about adverse social outcomes.
Contrary to the existing literature, the paper argues that ‘causal mechanisms,’ or even a robust correlation, that may potentially link the distribution of economic wealth and different political regimes cannot be identified due to a number of critical scientific constraints. Some of the main methodological and empirical limitations that are outlined include aggregate macro-level analysis using a single empirical observation per country or per year; creating a uniform and meaningfully comparable measure of democracy; a multitude of non-measurable factors that may simultaneously influence the independent and dependent variables; different time lags in the potential effects of the influencing variables; important assumptions behind correlational claims derived from statistical analysis; and trying to make meaningful comparisons across and within countries over different time periods despite very different degrees and types of democracy and inequality as well as country-specific policies and tax structures.
It is important to stress that dynamic social phenomena like democracy do not have an “intrinsic nature” nor do they abide by “social laws,” and so the data and methods used to measure democracy do not allow us to say anything about causality. Using new data sources, analysing different time periods, or employing new data analysis techniques cannot resolve this question or provide robust, general conclusions about this potential relationship across countries. Because researchers are restricted to exploring rough correlations over specific time periods and geographic contexts with imperfect data, they need to be more critical and transparent in explicitly outlining the limitations of the data and methods they apply, and about the precision and interpretation of their results. The hope of this paper is to possibly be a useful warning for researchers against overly ambitious research aims and the overselling of their estimated results.
1. Krauss, Alexander. 2015. The scientific limits of understanding the (potential) relationship between complex social phenomena: the case of democracy and inequality. Journal of Economic Methodology