Our Uneconomic Methods of Measuring Economic Research

By Stan Liebowitz,Ashbel Smith Professor of Economics, University of Texas at Dallas; Head of the Center for the Analysis of Property Rights and Innovation (CAPRI). Originally published at VoxEU.

Lambert here: Screamingly funny, in a deadpan sort of way, and reminiscent of Outis Philalithopoulos’s seminal work on academic choice theory.

* * *

Academic economists – especially in the US – are continuously evaluated, with salaries and promotions hanging on outcomes. This column argues that the methods – identified from a survey of economics department chairs – are likely to reduce the amount of research created, perpetuate inefficiently sized research teams, promote false authorship, and penalise honest researchers. They also provide departments with excessive leeway to engage in potentially capricious behaviour.

In the movie Moneyball, a nerdy Ivy League economics major, working for a general manager played by Brad Pitt, found undervalued baseball players by applying clear-headed logic and statistical techniques.1 Many economists watching this movie probably felt a tinge of pride in seeing our tools portrayed as rigorously objective. After all, economists have long been proponents of using logic to eliminate inefficiencies and rent-seeking in the economy (e.g. Tullock 1967). Given this, it is surprising how infrequently that penetrating gaze has been focused on our own profession.

But some attention is warranted. Our methods for measuring and rewarding research – the key component for promotions and salaries – create inefficiencies and are inconsistent with what we teach our students about efficient production. Further, this inefficiency might be caused by economists’ own rent-seeking through the vehicle of departmental politics.

The manner in which we credit coauthorship and evaluate articles induces overly large research teams, encourages false authorship, enhances subjectivity, and penalises honest researchers.

Evaluating Coauthorship Credit

It is easy to create a division-of-credit system that gives researchers the correct incentives to choose efficiently-sized authoring teams. Simply put, a rule where the coauthors’ credit shares sum to one (i.e. full proration of credit) provides the correct incentives for choosing team sizes, and any other division rule does not. Yet, as Figure 1 shows, full proration of credit is almost never used in economics departments, according to my recent survey (Liebowitz forthcoming) of department chairs.2 More than a third of departments (16 out of 45) give each coauthor full credit for the entire article. Only one department completely prorates credit.

Figure 1. Discounting of authorship

liebowitz fig1 5 dec

Zero proration is a flagrant violation of economic logic. For two identical quality articles, one written by a single author and the other written by four authors, should the credit to each of the four coauthors really be the same as the reward to the sole author? Do we normally say that efficient production requires that inputs get paid their marginal revenue product multiplied by the number of coworkers?

If the four-authored paper is not written with each author providing one-fourth or less effort compared to each author working alone, then that size of team is inefficient. But if each coauthor is given full credit, they have an incentive to coauthor even when the number of papers written by the four-author team is much lower than the number of equal quality papers they could write working alone or with smaller teams.

Departments that fail to discount by the number of coauthors should be embarrassed to use a measurement process that incorporates a logical error that would not be allowed in a micro principles course.

Are research-oriented departments more likely to award research credit rationally? The answer, found in Table 1, is yes and no. Higher-quality departments are more likely to prorate coauthorship than are lower-quality departments. Strangely, the amount of proration for those departments that prorate is somewhat greater for low-quality departments, leading to about the same overall degree of proration regardless of departmental quality.3

Table 1. Prorating author credit by departmental quality

liebowitz table1 5 dec(1)

Note: Based on authors’ survey of department chairs, 45 observations.

Because the failure to fully prorate will lead rational researchers to use more than the optimal number of authors, too few papers will be produced relative to the population of authors – a result consistent with a finding by Hollis (2001) that coauthorship decreased total output. If, as we like to think, our research has a net positive impact on society, then inefficient research practices would imply social harm. The failure to fully prorate may also have helped cause the doubling in coauthorship that has occurred in economics (and other fields) over the last 50 years.

The failure to prorate is also likely to lead to ‘false authorship’, where an individual not involved with the research is added to a paper’s list of authors. The ‘real’ authors benefit from a personal gain in friendship (or a quid pro quo) and suffer, if at all, only from potential guilt about being dishonest or the potential punishment if this ‘fraud’ is found out. I suspect that the expected punishment cost approaches zero, since it is unclear that the profession even acknowledges this as a problem. Because false authorship does not change the size of the actual research team, however, there is no direct negative impact on the creation of research. Nevertheless, important social costs may well occur when honest researchers are under-rewarded and possibly displaced by less qualified researchers engaging in false authorship.

In light of these inefficiencies, why do we use these reward structures? For self-interest to be the answer, it must be that the more powerful members of a department are the more intensive practitioners of coauthorship. This possibility is supported by evidence that faculty seniority is positively associated with a greater incidence of coauthorship (Conley et al. 2011, McDowell and Melvin 1983). In addition, given that this reward structure has been in place for decades (Liebowitz and Palmer 1988), faculty engaging in greater-than-average coauthorship would command an inappropriately high status among the senior faculty. Self-interest among senior, high-coauthoring faculty members would imply a push for less-than-full proration – even if these faculty members understood that it would reduce the (quality-adjusted) number of publications emanating from their department.

Judging the Quality of Papers

The self-interest of senior faculty in the measurement of productivity might also explain what otherwise appear to be irrational choices among departments in judging the quality of research papers.

There are generally three sources of information used to judge the quality of a paper: the quality of the journal in which it was published, the number of citations garnered by the paper, and opinions formed from reading the paper. Because a journal’s decision depends on the opinion of only an editor and a few referees chosen by the editor, there is a great deal of latitude for gratuitous decisions. By way of contrast, the number of citations that a paper receives is determined by the entire academy, likely reducing the influence of gratuitous behaviour. The value of reading a paper depends on who is doing the reading, their qualifications for making a judgment, and their objectivity. Consequently, it is also the method of evaluation most easily abused by the senior faculty.

Table 2 presents the relative importance of these three information sources for senior promotions, according to department chairs. The journal of publication is the leading measure of a paper’s quality (although top departments have less reliance on this measure). The main problem with this measure – other than the possibility of gratuitous acceptances – is that article quality can vary widely within any given journal, making the attribution of average journal quality to a publication potentially misleading. Further, although journal-quality measurements can be reasonably objective, it is possible for departments to use their own journal rankings based on their own biases, unmoored from any consistent ranking system.

Table 2. Relative importance of article characteristics in senior promotions

liebowitz table2 5 dec

Note: Based on authors’ survey of department chairs, 46 observations.

Although the numbers in Table 2 might seem to imply that citations are given fairly high prominence in measuring faculty research, I would suggest that citations are given far too small a role in these senior promotions. For example, there seems little justification in giving any role to the journal of publication as a measure of paper quality if citation information is otherwise available. Citations tell us whether a paper is influencing the literature or not, whereas the journal of publication – even assuming the journal’s acceptance decision was fully impartial – was merely a predictor of whether a paper was thought likely to have such an impact. It seems probable that the reason that citations are not given more influence in measuring article quality is because the number of citations is the measure that can be least manipulated to fit the department’s tastes (biases).

Naturally, citations should be prorated for the same reason that article credit should be prorated. But according to the survey, citations are hardly ever prorated. Although prorating citations used to be time consuming, the advent of programs like Harzing’s ‘Publish or Perish’ has made it much easier to perform the proration.

Table 2 shows that lower-ranked departments are less likely to rely on their own opinions of papers. This is one of the few findings consistent with efficiency, since these departments are presumably, on average, less qualified to judge a paper’s quality than are the members of more highly ranked departments.


Are academic economists little different than the old-school baseball scouts in Moneyball, clinging to unexamined rules of evaluation? Or is our profession dominated by inefficient rent-seeking? Either way, our measurement systems do not appear suited to promote productivity. Maybe Brad Pitt should play a Provost imposing some ‘scientific’ logic on our university sanctuaries.


Conley, J P, M J Crucini, R A Driskill, and A S Onder (2013), “Incentives and the Effects of Publication Lags on Life Cycle Research Productivity in Economics”, Economic Inquiry, 51(2): 1251–1276.

Hollis, A (2001), “Co-authorship and the output of academic economists”, Labour Economics, 8(4): 503–530.

Liebowitz, Stan J (forthcoming) “Willful Blindness: The Inefficient Reward Structure in Academic Research”, Economic Inquiry.

Liebowitz, S J and J P Palmer (1984), “Assessing the relative impacts of economics journals”, Journal of Economic Literature, 22(1): 77–88.

Liebowitz, S J and J P Palmer (1988), “Assessing Assessments of the Quality of Economics Departments”, Quarterly Review of Economics and Business, 28(2): 88–113.

McDowell, J M and M Melvin (1983), “The determinants of co-authorship: an analysis of the economics literature”, Review of Economics and Statistics, 65(1): 155–160.

Tullock, G (1967), “The Welfare Costs of Tariffs, Monopolies and Theft”, Western Economic Journal, 5(3): 224–232.

1 Pitt was playing a real-life baseball executive, Billy Beane, who used ‘sabermetrics’ to help pick the players on his team who would provide the highest return per dollar. The movie was based on the book of the same name by Michael Lewis.

2 The survey consists of a questionnaire given to department chairs at the top 100 or so mainly American universities, almost half of which were answered.

3 Full proration (100% in Table 1 and Figure 1) implies that each of n (equal) authors receives 1/n credit for the article. Zero proration implies that each author receives credit for the entire article. Other percentages are a linear combination of these two.

Print Friendly, PDF & Email
This entry was posted in Guest Post on by .

About Lambert Strether

Readers, I have had a correspondent characterize my views as realistic cynical. Let me briefly explain them. I believe in universal programs that provide concrete material benefits, especially to the working class. Medicare for All is the prime example, but tuition-free college and a Post Office Bank also fall under this heading. So do a Jobs Guarantee and a Debt Jubilee. Clearly, neither liberal Democrats nor conservative Republicans can deliver on such programs, because the two are different flavors of neoliberalism (“Because markets”). I don’t much care about the “ism” that delivers the benefits, although whichever one does have to put common humanity first, as opposed to markets. Could be a second FDR saving capitalism, democratic socialism leashing and collaring it, or communism razing it. I don’t much care, as long as the benefits are delivered. To me, the key issue — and this is why Medicare for All is always first with me — is the tens of thousands of excess “deaths from despair,” as described by the Case-Deaton study, and other recent studies. That enormous body count makes Medicare for All, at the very least, a moral and strategic imperative. And that level of suffering and organic damage makes the concerns of identity politics — even the worthy fight to help the refugees Bush, Obama, and Clinton’s wars created — bright shiny objects by comparison. Hence my frustration with the news flow — currently in my view the swirling intersection of two, separate Shock Doctrine campaigns, one by the Administration, and the other by out-of-power liberals and their allies in the State and in the press — a news flow that constantly forces me to focus on matters that I regard as of secondary importance to the excess deaths. What kind of political economy is it that halts or even reverses the increases in life expectancy that civilized societies have achieved? I am also very hopeful that the continuing destruction of both party establishments will open the space for voices supporting programs similar to those I have listed; let’s call such voices “the left.” Volatility creates opportunity, especially if the Democrat establishment, which puts markets first and opposes all such programs, isn’t allowed to get back into the saddle. Eyes on the prize! I love the tactical level, and secretly love even the horse race, since I’ve been blogging about it daily for fourteen years, but everything I write has this perspective at the back of it.


  1. craazyman

    If I look at my belly button long enough I see the moon sunken and hiding in a pillow of clouds. If 5 or 6 or even 12 people look at my belly button with me, does that mean there are 12 moons in the sky? Deep thoughts occupy the mind. It must have been this way for Rodin’s THINKER too, but he had the misfortune of being utterly alone.

  2. Banger

    No offense, but my father, an economist, early on demystified the field of economics for me–as an academic pursuit it is very weak because it is based on many unexamined or false assumptions. The first assumption is that it is a legit field by itself–the only reason it is a separate field is that our culture values money and markets above all else. In reality economics without politics cannot exist–all economics is political first. This is how the canard of “free-markets” has credibility–all markets are structured by politics.

    Having said that, it seems the economics field mirrors other academic fields, i.e., it is dominated by narrow interests and, yes, politics.

  3. Hugh

    This is a very strange post. I guess lambert would call it meta. It is essentially asking the question how we should evaluate kleptocratic propaganda. That is it is propaganda whose subject matter is propaganda, in other words how best to validate the process which validates the looting of the 99%.

  4. Alejandro

    “Research” to support “Policy design”. “Policy design” is a euphemism for rigging the system to benefit the zero dot zero oners.

  5. Knut

    I had a run-in with Liebowitz on a panel given over to the notion of path-dependence in making economic outcomes. As one implication (though hardly the most important or most interesting) of non-ergodic economic processes is that they might not always end up in the best of all possible worlds, Liebowitz took it upon himself to claim that as long as the market rules, there is only one outcome, and it’s the best. He was clever, but out of his depth. The present article is a prime example of his shallowness, which is unfortunately all too common in the lower reaches of academc economics he inhabits. He is the reason I used to tell my students upon their completing economics 101 not to practice what they thought they learned in the absence of parental supervision. The article could have been placed with the Onion.

    1. TimR

      So based on that it sounds like he means this in all seriousness, albeit with a note of ironic “turnabout is fair play” — turning the economists’ own methods back on them.

      But yes, by all means, if they’re going to Taylorize everyone else, lets Taylorize them back…

    2. Lonely_in_Dallas

      “Non-ergodic economic processes” sound very intriguing. Care to give some references? I know what an ergodic process is, but hadn’t heard the term in the context of economics. (But I must admit I hadn’t looked either!)


  6. TheCatSaid

    These are important points. This apply to all academic areas.

    I can relate to your observations about papers written with huge teams of co-authors. It’s not possible to evaluate the significance of each “co-author”‘s contribution, particularly when there are 10 or 20 or more.

    In a scientific hi-tech research domain I wonder whether the name of every engineer (and grad student) involved in maintaining a piece of complex equipment was included–even if they probably didn’t have any personal involvement with the actual research in question.

    For all I know, maybe using hi-tech equipment comes with a requirement for mass team citations. Have NC readers come across any such requirements for citations based on using certain facilities? Might this be justified in some cases?

    Publish or Perish software & book look interesting.

  7. craazyman

    This is why Docter Profeser Delerious T. Tremens, NFL, GED of the University of Magonia works alone and only signs papers with an ink stamp (but in different colors depending on the mood), manifesting total indifference to either glory or ignominy. Truth is the only goal.

    1. craazyman

      In his published research Profeser Tremens presented to the world what has become known as the “Tremens Theorem”, to wit:

      “Economic research doesn’t describe reality, economic research creates reality.”

      to wit #2: The models, by virtue of their hypnotic influence on simple minds and policy wonks, create the realities they purport to describe.

      He should get a Nobell Prize for this theorem, or at least a gift like a toaster oven.

      1. Skippy

        Ummm… methinks the toaster oven has more intrinsic value for the tool user… yet… some determine Substantia grisea vacuums are more intrinsic… funny that…

        skippy… I just wished Robbin Williams, in his coke days, would have expressed complex economic theorem… as a stand up routine. We would be better off by now… foursured~

Comments are closed.