Cathy O’Neil: If You Redefine “Performance,” Magical Things Happen

Posted on by

By Cathy O’Neil, a mathematician and data scientist currently writing a book about the dark side of big data. Crossposted with mathbabe.org

There is a study out, entitled The Best and The Rest: Revisiting the Norm of Normality of Individual Performance, written by two business school professors, that has been bothering me recently. I’ll explain why soon, but first a thought experiment.

——

Imagine a group of people competing for something. They’re all driven, talented people, who have put serious resources into getting good at this particular thing. They’ve also all had help of some form, and encouragement from their community to compete in this arena. At the very least they have to have deep confidence in their own abilities to even compete in this particular area.

At the end of the competition, that particular one, these people are ranked according to how they’ve done. By luck, by skill, depending on previous practice, resources, or direct support from external advisors, some of them have achieved impressively high rankings, while others, in spite of their hard work and efforts, are falling behind.

Next, there’s a community feedback element. This group of people are not done – they’re hoping to become famous worldwide, or at least in this arena, for being highly ranked, maybe even the best. And the community has direct influence on what happens next, in future rounds of competition. So, individuals can vote for certain people to win, or directly give them more time or resources to do so, or even help them in their next round.

In subsequent rounds, the ranking gets more defined and the community becomes increasingly certain that the winners deserve to be there and that they are truly fabulous at this particular skill, even though the original native differences in talent are not enormous. Luck, resources, and self-confidence were all important indicators of success in that first round, some just as important as native skill.

This continues for years. At retirement, the highly ranked individuals have produced a massive amount compared to the ones that did poorly in the early rounds. In fact, the distribution is highly skewed, and seems to serve as proof that the original ranking was warranted.

——

I didn’t specify what field the above story took place in, so let me suggest a few that might work. First, there’s the music industry. Lots of would-be rock stars vie to be the next Taylor Swift. Heck, even Taylor Swift vied, once upon a time, to become herself. Of course, it helped that she was able to persuade her wealthy parents to move to Nashville when she was 14 to pursue her career. And – not to say she isn’t talented, because she most definitely is – we all know that once you have a hit, your career is much more likely to go well after that, with contracts, money, support, and great musicians flocking to you.

Or, it could be academics. If you stand out as an undergrad, especially at the right college, you get into a good grad school, and if you have enough confidence, determination, and the good luck to get a nice thesis problem, you might have a thesis that stands out, which leads to NSF grants, reduced teaching loads, opportunities to speak at conferences, semesters off of teaching to pursue research, and a host of co-authors who are increasingly willing to do the work to write up joint results. Again, none of this happens without determination, drive, and talent, but it definitely happens more and faster with the help of a supportive community. It’s all about the feedback loop of success.

Or, here’s another arena: sales. If you are known as a successful salesman, if you have a slightly better reputation than the next salesperson, then you’ll get the dibs on the jobs in a typical organization. That means you can be choosy, and take the easy pickings, and pass over the harder jobs. Over time your likability and personal network grows, and you become the go-to person in the organization for success, partly because of your hard work ethic, but partly because of the way success breeds success.

Or how about basketball? All professional basketball players are amazingly good at what they do. How much better does one have to be to get more playing time? Which leads, of course, to more points, more double doubles, or what have you.

——

Now to the paper. It talks about the distribution of performance, and notes that in arenas above, performance, which they equate with output of songs for musicians, or papers for academics, or sales figures for salesmen, are distributed more as a power law probability distribution than as a bell curve. Of course, that is true, and I think we know why, from above. It even has a name: the Matthew Effect, which is even referred to in the paper, on page 112.

The primary goal of the paper is to make the case that “performance” is not normally distributed. It is distributed with a much fatter tail. They suggest using the Pareto distribution:

Probability_density_function_of_Pareto_distribution.svg

Before I go on, let me mention that their examples are restricted to researchers, entertainers, politicians, and amateur and professional athletes. They never mention secretaries, computer programmers, marketers, cashiers, or data analysts. In fact most of the people who work at regular jobs are completely excluded from this study.

So it’s really more accurate to say that the primary goal of the paper is to redefine the word “performance”. They switch from one definition to the other without explanation, so their studies on pro athletes somehow magically refer to average workers.

That brings us to the second goal of this paper. Namely, the conclusion that we should use this “performance isn’t normally distributed” rule to focus even more on elite actors.

Here’s one version of the elitism argument (page 108):

Leadership theories that avoid how best to manage elite workers will likely fail to influence the total productivity of the followers in a meaningful way. Thus, greater attention should be paid to the tremendous impact of the few vital individuals. Despite their small numbers, slight percentage increases in the output of top performers far outweigh moderate increases of the many. New theory is needed to address the identification and motivation of elite performers.

What’s particularly irksome is this kind of logic (page 112):

For selection, this means that there are real and important differences between the best candidate and the second best candidate. Superstars make or break an organization, and the ability to identify these elite performers will become even more of a necessity as the nature of work changes in the 21st century (Cascio & Aguinis, 2008b)

If you think back to our original thought experiment, there is actually very little difference between good candidates at the beginning. Second, this “we absolutely need to keep our talent” mentality is exactly the argument we see time and time again excusing pay raises for CEO’s. And now there’s a “mathematical” reason for it.

That brings us to the third and final goal of the paper, the “CEO pay is not exorbitant” argument, (page 112):

Likewise, compensation systems such as pay for performance and CEO compensation are an especially divisive issue, with many claiming that disproportionate pay is an indicator of unfair practices (Walsh, 2008). Such differences are seen as unfair because if performance is normally distributed then pay should be normally distributed as well.

Let me rephrase: since “performance” isn’t normally distributed, there’s no way pay should be either, when we define it for everyone. So let’s just go ahead and overpay CEO’s.

It might be a good moment to remind people that even in academics, the top performers don’t make 100 times what the lower performers get. Compare that to McDonalds, where the burger flippers would have to work 1 million hours to get one year of CEO pay.

In pop music and pro sports, there is a crazy pay differential, but that’s not something to be proud of or something we want to replicate.

Print Friendly, PDF & Email

28 comments

  1. Disturbed Voter

    There is an old thought experiment about fairness. Two children are squabbling about how to cut the last part of the pie into two pieces. The fair method is to have one child do the cutting, and the other child to make first choice as to which piece they want. Fairness results every time. So by all means, let one party choose the salaries, but the other party choose which remuneration they will get. We won’t need Dr Nash et al to tell us how that game theory works out ;-) No matter what math you use to cloak unfairness … it is still unfair. The fault lies not in our stars or Pareto distribution … but in ourselves.

  2. Gabriel

    Lovely, very concise counter.

    Isn’t the “superstar economy” stuff at the basis of cheery reactionary Tyler Cowen’s “Average Is Over” thesis?

  3. dk

    This comment is based on a personal gripe of my own, but I think the case of my example is widespread.

    I’m good at what I do, really good, a hard to find skill set. I am also not pushy about my own pay rate, I keep telling my clients/employers to pay the OTHER people more, so we can get better people across the organization, because one guy (me or whoever) can’t make everything work all by themselves, particularly as scale increases. The sustainability of the systems I build is dependent on the skill sets of its subsequent users and tenders; and I make them very easy to tend, but there is a minimum level of expertise and general knowledge-of-process required. Go below that threshold, and things will start to go wrong; and that will not be the fault of the people you hired.

    When a successful specialist leverages their skill set, it can raise the stakes for the entire organization. Raising pay/salaries for others in the organization is not a “reward”, it’s a necessary factor in supporting sustained success, regardless of where the success process started.

    Several clients/employers say their solution is to give bonuses and “perks”, this fails to help workers who happened to catch a challenging situation with inadequate resources (which could include skill, or maybe just the confidence level that fair compensation and job security would provide). Said worker(s) may make valiant efforts and may achieve some relative success in face of disaster, but still not make the “cut” for bonus/perk reward.

    I think that O’Neil’s argument, while cogent, fails to clearly point out that the “special reward” paradigm can’t effectively (not to mention rationally) contravene the “payment for work value” model. And it is this basic payment model that is being undermined when CEO and other “expert” pay skyrockets while other pay stagnates (or worse).

    I also think that O’Neil’s leading example is avoiding the role of deliberate social gamesmanship; a skill set sometimes markedly distinct from others, and one that does not contribute productive value beyond an immediate and ephemeral (and I would say, entirely subjective) social permutation which is, across the entire similarly-skilled group of the example, likely to skew eventual net production negatively (although this “if only” case is difficult to observe and compare to, in these days of rampant social gamesmanship and superstar rewards).

  4. diptherio

    The average Mechanical engineer can expect to make round-about $3 million…in their lifetime…just sayin’.

    1. ambrit

      How long does the ‘average’ mechanical engineer have to work to get to that figure? He or she is still making roughly double what a well skilled plumber can make. This basic disparity in earnings determines the timing, indeed, the probability of retirement. I have personally known skilled tradespeople who had to work until they died.
      An engineer has to make a significant investment in time and treasure to make it to the “professional” skill level, true. The ‘good’ plumber also has to put in his or her apprenticeship time, and training. Then comes the test for Journeyman certification.
      I find Mz O’Neils basic point sound. Pay is a function of Status, not just the source of it.

  5. washunate

    Great read. There certainly are areas where superstars are incredibly more valuable than the average worker. The very best baseball players, pianists, physicists, etc., are notable for being tops in their field, often driven internally at a young age to do things that nobody else can do.

    But I’m pretty sure it’s broadly understood in organizational behavior that these are niche exceptions. The most relevant feature of performance generally, especially in large organizations, is that there is very little difference in contribution to the performance of the organization between the most “skilled” and least skilled roles.

    And yet we have policy entrenched wage inequality at every rung on the socioeconomic ladder. Often supported by otherwise liberal, educated leftists.

  6. TarheelDem

    The question of value in performance often misses the question “valuable to whom?” The political answer always is the the person or people who are making the salary decisions.

    Then the question is “what are these people paying for”? That is the definition of performance for that particular job. Often “performance” means sycophancy and that alone. Other times it means willingness to break rules and even laws or to play accounting games. Rarely does it mean what it mean for Pablo Casals. Or Noam Chomsky. Or Michael Jordan. Software companies don’t pay for elegant code; they pay for people who can stay away from their families and work 16-20 hour days for months on end and deliver something on the pre-scheduled delivery date. Package delivery services don’t pay for careful package delivery service but meeting schedules tracked by GPS. Because those things are what deliver free money to the bottom line, customers, vendors, employees, local communities be damned. The whole pay for performance discussion is primarily a sham designed to hide extraction by political authority and accounting rules.

  7. JohnnyGL

    Let’s keep in mind the most important assumption about these fields discussed, which are mostly entertainment related fields like music, sports. The top performers are LUCKY that people actually enjoy that particular sport or music genre. If people decided they didn’t like pop-music or didn’t like watching baseball anymore, then their talents are mostly useless, much like winners of spelling bees. Great spellers are, no doubt, masters of their craft, but few actually care.

    Also, there’s network effects on the demand side, too. Members of society need something in common to talk about. If you don’t know much about pop-music or pro-sports, you will find it harder to re-connect with your co-workers and friends and make small talk.

  8. FluffytheObeseCat

    Thanks for this critique. I have a strong suspicion that “power law” reward systems commonly degrade the overall, long term performance of groups, even in fields where “superstars” are characteristic, like professional sports. I can tell you that in academia, departments with 1-2 star profs, and a mass of those who are just marking time, stink at both education & research.

  9. reason

    You could have debunked the argument here better. The difference they are measuring is not PERFORMANCE. The difference they are measuring is OUTCOME which is something quite difference.

    I always like to use the thought experiment of what would happen to tennis if tournaments were all winner takes all. In that case, most of the performers would become amateurs, because those that are not quite the best, mostly couldn’t afford to travel the world to always end up being beaten. So the competition would become in general weaker (standards would actually fall). Those at the top would make a great deal of money, but the tournaments would become less attractive and so the total amount of money in the sport would decline.

    That is ultimately where the fallacy is – the rewards come from winning, but the value comes from the competition. You need to train lots of people (and support their training) in order to create the competition.

    1. guest

      Guess what? You are perfectly right, because the exact phenomenon you describe has been observed in golf tournaments.

      They are not exactly winner-takes-all, but almost. The winner reaps the biggest prize, for the second and third places the money is comparatively a paltry sum.

      The result: whenever a super-champion participates in a tournament, the other players do not exert themselves that much, because the champion (such as Tiger Woods) will most probably win, and no matter how well one plays, the 2nd and 3rd prizes are not worth the effort.

      Apparently, tournaments where no top champion takes part in are more interesting, because the “normal” players are motivated to compete — whereas tournaments with top stars end up as fairly boring events.

      I remember to have seen this effect described in a Slate blurb that referred to some research paper.

    2. washunate

      That does strike as an intuitive perspective. But it’s not really the case. I’d be curious to hear if you meant something else by the fallacy of winning vs. competition.

      First of all, it is not the case that in entertainment spectacles people solely desire the competition. Some component of the value of the product is seeing ‘the best’, the superstar, the big name, the home town hero, the identity of the shared experience, etc. Guest talked about Tiger Woods and golf. Well, look at interest in golf. It soared when it was Tiger Woods vs. The Field. The hypothesis that Tiger created an inferior product simply isn’t consistent with both quantitative and qualitative observations about golf over the past couple decades.

      Second, many things are not directly competitive. There are awards for movies and music and theater and so forth and intense competition within those fields to be the big celebrities, but the products themselves are not in direct competition.

      Third, even when there is direct competition – like the major American team sports leagues – the data seems to suggest that pumping a huge amount of money into the very top while leaving large numbers of people with no professional wages at all simply doesn’t dampen the entertainment value of the product. NFL and MLB, the two biggest, have seen massive increases in money that flows disproportionately to ownership and media partners, then to the superstar players, then the role players, then the fringe players. Yet underneath those two incredibly successful leagues are orders of magnitude more players who are semi-pro or even complete amateurs. The fate of college “student”-athletes in basketball and football and minor league baseball players has been a topic of increasing interest and awareness over the last few years.

      Beyond that, in entertainment people being paid millions of dollars a year are generally doing precisely that: producing entertainment. Regardless of whether it is “good” to have this level of wage inequality, the outcome is real – these workers produce at significantly higher levels than the ‘average’ or ‘replacement’ entertainer. That makes this a really niche area.

      In contrast to highly paid athletes, most highly paid administrators are hardly distinguishable from an average administrator. That’s the issue with compensation across most of the workforce (not just CEOs). It’s not a matter of outcome vs. performance. Rather, most people at the top of the wage scale simply don’t contribute notably more to an organization’s activities than the average worker.

      1. reason

        1. Golf isn’t in general winner take all.
        2. Yes name recognition counts IN THE SHORT TERM. But in the long term quality counts.
        3. Golf isn’t a good example, because people don’t compete directly against one another, but with the course.
        4. Your semi-pro league and college players ARE being paid. It isn’t winner take all.

        1. washunate

          1-3) Golf was guest’s example supporting your argument. I agree that professional golf is not a good example to support the idea of a difference in incentives between winning and competition undermining the value of a product.
          4) Sorry, I don’t follow you here? How can it get more winner take all than the compensation difference between a top athlete and what the thousands upon thousands of players lower down the ranks earn? Entertainment has an even larger multiple than the 100 to 1 the author uses for McDonald’s.

          ***

          My point is that distinguishing ‘performance’ from ‘outcomes’ isn’t actually explanatory of differences in working conditions and wages in the formal economy. Those are different words describing the same concept.

          Rather, in some rare areas (particularly but not exclusively in entertainment), there truly is a notable gap in production between the very best and the rest. The actual world of professional sports simply doesn’t align with the intuitive hypothesis that winner take all scenarios undermine the product. An extremely few number of workers earn nearly all of the compensation – and these workers are uniformly recognized as vastly superior. A casual fan can tell at a glance when somebody is out of their league. That doesn’t make the extent of the inequality “good” in a public policy sense (entertainment is heavily subsidized by public policy, after all); it’s simply an observation that there is a corresponding underlying performance inequality.

          This is a noteworthy contrast to most work in the formal economy, because in most cases, there is not that much difference between the top performers and an average worker. Nor is there much difference between the ‘skill positions’ and the ‘role players’, if you will. Yet we have lots of inequality in the rest of the workforce, too – inequality that cannot be explained by differences in performance/outcomes or incorrectly incentivizing winning over competition.

          Indeed, you can see the author struggling with the implications of crafting a universal principal to guide working conditions and wages in the formal economy. Even as she is decrying inequality and mocking perversions of performance metrics, she still feels a need to defend academia, to justify its inequality since it’s not as bad as those other guys’ inequality. It’s much easier to have a few convenient corporate villains than to see unjustified inequality being driven by the very institutions that are supposed to be upholding our society like higher education, medicine, and law.

  10. Ignacio

    Not to forget that, at least in R&D it is critical to fund as many research projects (critical mass) as you can if you want to get top performers. Not to mention that those top performers wouldn’t perform if they were the only left, after such brutal selection suggested by the paper. It occurs always that the outperformer changes his/her attitudes and starts looking for ways of keeping their advantage without performing any better. (Microsoft Windows gets worse and worse with every new version).

    etc.

  11. Thure Meyer

    Sigh – More Great Man Theory – everything depends on those rare heroes in our society.

    Not too surprising we have this kind of modeling in the age of Marvel Movie success, over the top celebrity worship, TBTF (after all these companies are run by geniuses).

    Of course, given a specific operational situation (action arena, game, institution) there will be winners and losers based on the structure and dynamics of the situation, i.e. some will perform better. But that’s obvious and the structure often determines the outcomes.

    But isn’t this yet another attempt to model the world around some sort of closed market paradigm that completely ignores any externality. The research conclusions are actually content free because they are based on underlying assumptions about institutional value, performance, and outcomes that only reflect the authors obvious acceptance of the status quo.

    I think “reason” above is absolutely correct, change the structure of the game just slightly (change the outcomes) and suddenly performance stands out in relief as just another property of the game itself, not some sort of great human characteristic.

  12. Tammy

    It is funny that I find myself participating on blogs that, oddly, I think to myself, “I don’t know if I fit in here.” And no I don’t have an attachment dis-order. I mention because I recently visited Ms. O’Neil’s blog, writing on a blog a few books re “Big Data.” I’m not as opposed to Larry Summers as O’Neil and thought the interview feat. O’Neil ended in a way that was perplexing, which I’ve mentioned prior in a Naked Capitalism thread–the “move your money” app–re the OWS movement.

  13. Tammy

    I can’t access the study entitled The Best and The Rest: Revisiting the Norm of Normality of Individual Performance.

  14. armchair

    The quarterbacks in last year’s Superbowl were taken in the 3rd round and the 6th round of the draft. Moneyball is basically a one word response to the super-hero cult. The idea that there isn’t a single person in the world who could competently run JP Morgan for a fraction of the cost is such BS, unless maybe part of the CEO’s salary is expected to be used for bribes?

  15. Tammy

    Re the “Matthew Effect” – I’ve heard about the “Streisand Effect” maybe that’s why I can’t access the study as it reads “403 Forbidden…” (I little humor there. Now I’m going to read the article by O’Neil.)

  16. kevinearick

    People begin relatively equal. A few ignore the stupidity of public education, but most do not. Over generations, you get what you see, certified stupidity breeding more of the same. The more technology is employed to exchange entertainment, surveillance and war for real estate inflation, the more it distills labor out. Uber isn’t anything new. It reduces the cost of what its originators believe is labor, increasing the efficiency of stupid capital, with growing leverage of the middle class ponzi, which is collapsing, as measured by natural resource depletion and DNA deficiency. The Fed is simply anticipating demographic deceleration, expecting demographic re-excitation, which isn’t occurring. All the ‘new’ currencies are trying to find a labor market, with no success, surprise.

    1. Disturbed Voter

      We do begin relatively equal, in the womb. But it has always been important to carefully choose your parents, and if necessary to adopt a better set of parents if the first couple don’t show much promise. After that initial boost, be sure you are born at the right time and right place. Galileo as a Mongolian shepherd wouldn’t have amounted to much. Once you reach adulthood, study Machiavelli. and seriously take his sage advice. That is how three sigma plus individuals emerge, sooner or later. Then market yourself as someone who gave birth to oneself, raised oneself, and who is a nation-state-corporation until oneself, if your head doesn’t explode first.

      Everyone gets what they deserve in the end though, death.

  17. fresno dan

    “At the end of the competition, that particular one, these people are ranked according to how they’ve done. By luck, by skill, depending on previous practice, resources, or direct support from external advisors, some of them have achieved impressively high rankings, while others, in spite of their hard work and efforts, are falling behind.

    Next, there’s a community feedback element. This group of people are not done – they’re hoping to become famous worldwide, or at least in this arena, for being highly ranked, maybe even the best. And the community has direct influence on what happens next, in future rounds of competition. So, individuals can vote for certain people to win, or directly give them more time or resources to do so, or even help them in their next round.”

    Good reasons must, of force, give place to better.
    At the end of the competition…….. By self absorbent, vanity, bragging, lying, bribery, and sheer as*holditry, they win where the more noble, honorable, true, and virtuous, submit and concede.

  18. vlade

    Well, while I won’t dispute that there’s a tremendous amount of luck, community support etc. that has impact on “performance”, in my personal experience, in the IT field an excellent programmer is worth multiples of average.

    1. reason

      Yes,
      but beware of the Peter Principal (and jealousy and performance reporting systems that are not geared to idiosyncratic skills).

  19. Kaleberg

    If you are trying to produce a simple ranking, you are always able to get some kind of power law result as that is the nature of the competition. The top player is better than all of the rest. The second best is better than N-1 and so on. If you are trying to maximize overall production, you are more likely find a normal distribution. I took a course in probability from Gian Carlo Rota, and I’m pretty sure this came out of some of the math. As you noted, what you get out of your ranking system is what you put into it.

    Suppose you want to measure the performance of lottery winners. You’ll almost certainly see a power law as that is how lottery payoffs are structured. The grand prize winner will be much more productive than the runners up and so on, down to the losers. (A friend of mine put herself through law school playing the lottery, so I know a bit about it even though I have never bought a ticket.) If you are playing the heads/tails game (or dreidel) with a fair coin (or dreidel), you’ll find a normal distribution of performance. That’s how math works.

Comments are closed.