Yves here. I have to confess that the big reason I’m partial to this article is it is consistent with my experience. Metricization of business, and worse, employee performance, has become a fetish. As I wrote in a long form treatment) in 2006:
Metrics presuppose that situations are orderly, predictable, and rational. When that tenet collides with situations that are chaotic, nonlinear, and subject to the force of personalities, that faith — the belief in the sanctity of numbers — often trumps seemingly irrefutable facts. At that point, the addiction begins to have real-world consequences.
It’s obvious to see how counterproductive simple measures can be. For instance, one push has been to require doctors to publish their success rates for surgeries. That’s often utterly misleading, since top doctors suffer from adverse selection: they attract the most difficult cases. My general bias is if you must measure outcomes, you need to use multiple measures. Over-reliance on a single figure to sum up a complex situation (Value at Risk is the poster child) is going to get you in trouble. But most people don’t like looking at a lot of data and having to weigh it to reach conclusions because that requires judgment….which is what they wanted to avoid relying on in the first place.
And here, it makes sense that government bureaucrats, who have to navigate through legislative and program requirements as well as internal and external politics, have sufficiently complex tasks that simple measures aren’t a great guide to performance. But even more important, these researchers found that the use of metrics to manage bureaucrats actually undermined results.
By Imran Rasul, Professor in the Department of Economics, University College London and Daniel Rogger, PhD Student in the Economic Department, University College London. Cross posted from VoxEU
Around the world, civil service reform is viewed as necessary to deliver public services effectively and to foster development. However, evidence is thin on how the management of bureaucrats affects the provision of public services. This column presents new evidence from Nigeria linking completion rates of government projects to bureaucractic management practices. Greater autonomy is associated with higher completion rates, whereas performance monitoring and incentive schemes seem to backfire. The most effective private-sector management practices may not be suited to public sector bureaucracies.
Since its inception in the 1850s, the British Civil Service has become a cornerstone of the executive branch of the UK government, translating the policy programme of the government into practice. Its practices have evolved gradually over the decades, but it is now in the midst of a major upheaval – in 2012, the Minister for the Cabinet Office and the Head of the Civil Service jointly published the Civil Service Reform Plan. The plan recognised the increased expectations on government to deliver public services in the context of perhaps permanently diminished government resources. To bridge this divide, it argued that the Civil Service would have to undergo significant (and controversial) reforms, aiming to become more skilled and less bureaucratic.
Britain is not alone in striving to reform its bureaucracy. A global drive to improve the functioning of bureaucracies is encapsulated in the ‘good governance agenda’ of the UN, the World Bank, the Commonwealth Secretariat, and other leading international organisations. Without such reform of bureaucracies, it is argued that longer-term goals in international development related to poverty reduction are unlikely to be met. Core to this agenda are the notions of strengthening the accountability of civil servants, and putting in place systems to effectively deliver basic public services. The Overseas Development Institute estimates that OECD governments spend more than $10 billion annually on such governance interventions, and to date, more than 40 developing countries are undertaking some form of civil service reform as part of this broad agenda (Goldfinch et al. 2012, Hasnain et al. 2012).
New Evidence from Nigeria
Despite these drives to reform public service delivery, the evidence base linking how bureaucrats are managed to the effectiveness of public service delivery remains thin. Our new working paper provides new insights into this relationship by studying the correlates of effective public service delivery in Nigeria – a country that has largely retained the British Civil Service structures since independence and the move to a Presidential democracy (Rasul and Rogger 2013).
To do so, we combine data sources linking the outputs of government bureaucracies with details of how bureaucrats are managed. For public service outputs, we use project level data measuring the completion, quality, and complexity of over 4,700 public sector projects implemented by organisations in the Nigerian civil service, including government ministries and other federal agencies. These projects relate mostly to construction projects such as roads, buildings, and small-scale dams – essentially the ‘nuts and bolts’ of rural infrastructure in Nigeria.
Project Completion Rates
Project completion rates were assessed by independent teams of engineers and members of civil society. Completion rates are scaled between zero and one, and provide an accurate measure of the extent to which public projects are completed relative to what was originally prescribed by the National Assembly in the technical documents for each project. Figure 1 shows a histogram of project completion rates for the 4,700 projects we hand coded information on.
Two striking features emerge:
• First, 38% of projects never even start.
An obvious explanation for this is corruption within bureaucracies – Nigeria ranks high on most international corruption indices.
• Second, only 31% of projects are fully completed.
Our study explores how these project completion rates correlate to the management practices bureaucrats operate under.
Figure 1. Project completion rates
To measure such management practices, we adapt the methodology set out in Nick Bloom and John Van Reenen’s pioneering work (Bloom and Van Reenen 2007, 2010). Earlier Vox columns have described the impact of management practices on explaining cross-country productivity differences in private sector firms, and on firm behaviour in developing countries. We adapt this method to measure management practices for bureaucrats in Nigeria, and then link these multiple dimensions of management to the quantity and quality of public services delivered by those bureaucratic organisations.
Our results confirm that two dimensions of management practice for bureaucrats matter above all.
• First, we document a significant positive correlation between the provision of autonomy to bureaucrats and public services delivered.
This finding provide support to the notion that public agencies ought to delegate some decision-making to bureaucrats, relying on their professionalism and resolve to deliver public services (Simon 1983). The evidence is less supportive of the notion that when bureaucrats have more agency or organisations are more flexibly structured, then they are more likely to pursue their own objectives that diverge from societal interests, resulting in fewer public services being delivered. In the Nigerian context, a strong prior might have been that granting civil servants more autonomy would necessarily lead to more graft and less effective public service delivery. This is not the case. Hence corrupt practices, while prevalent and important drivers of public service delivery, are not the entire story.
• Second, we find that the use of incentives for subjectively-determined ‘good performance’, the collection and use of key performance indicators, and efforts to monitor bureaucrats, all tend to decrease public service delivery as measured by project completion rates.
The broad use of incentive schemes tends to backfire in this public sector setting, because well-targeted incentive schemes are near-impossible to construct without creating other distortions. These distortions arise because bureaucrats are engaged in complex tasks, the relationship between bureaucratic inputs to outputs is uncertain, and there is considerable ambiguity in the design and implementation processes for many public projects. Hence the incentive systems we capture may place excessive regulatory burden or ‘red tape’ on bureaucrats, say through the use of mis-targeted key performance indicators. Such distorted incentives have long been argued to lead bureaucrats to mis-allocate effort towards less productive activities (Kelman 1990). Alternatively, our management practices related to performance incentives and monitoring might also pick up elements of subjective performance evaluation that lead to other dysfunctional responses among bureaucrats – especially engagement in influencing activities to curry favor with senior management.
Differences with the private sector
Our findings sound a note of caution towards the good governance agenda of reforming civil services throughout the world. Whilst some practices such as increased autonomy for bureaucrats might be encouraged, other dimensions such as the use of explicit incentive schemes or the drive towards payment-by-results schemes (that are occurring in the context of the British civil service) might not be optimal, and could even backfire. In short, there should be no simple import into the public sector of management practices that have proven to be effective in private sector settings. The nature of tasks varies between the private sector and bureaucracies, the objectives of bureaucracies are far less clear-cut than those of private sector firms, and there are fundamental differences in the contracting environment more generally (Dixit 2002).
One difference that is especially relevant is the lack of specialisation we see across civil service organisations in Nigeria. Civil service organisations are often tasked with implementing multiple project types, so that the same ministry might well be responsible for the construction of buildings, boreholes, and small-scale dams. For each type of project, the best management practices in terms of autonomy or incentive provision might well differ. Hence this lack of specialisation suggests organisational project portfolios might simply be too wide for them to be able to optimally fine-tune their management practices for an individual project. This non-specialisation may be optimal given complementarities in the production function of public goods. However, their varied workload may make the effective management of bureaucrats more difficult.
Our work also highlights other aspects of civil service architectures that likely interact with the effectiveness of management practices for bureaucrats. These include the process by which individuals are selected into the bureaucracy to begin with, and the labour market rigidities that characterise the civil service – the bureaucrats we study typically enjoy long tenure within organisations, with little movement across organisations. Such immobility can hinder the spread of best practices in such bureaucracies. Finally, a lack of competition and lack of price signals across organisations delivering the same project types reduces incentives for organisations to further improve their management practices. All such issues should be studied jointly in future work.
Understanding what drives the effective functioning of bureaucracies matters to economists. From a macroeconomic perspective, the effective functioning of the bureaucracy is an important determinant of poverty, inequality, and economic growth as stressed by the state capabilities literature (Besley and Persson 2010, Acemoglu et al. 2011). Effective public service delivery also matters from a microeconomic perspective – program evaluations of micro-scale interventions are often partly motivated by the assumption that successful interventions can be faithfully scaled-up by governments.
Our work is an attempt to provide among the first wide-scale evidence in a developing country context of how public service delivery relates to the management practices bureaucrats operates under. The results have implications for bureaucracies in developed and developing countries. In order to push forward this research agenda, future work will need to develop richer models of the contracting environment in which bureaucrats operate. To empirically test such models, we also need to advance methodologies and collect data that better measures bureaucrat activities and how they link to public service delivery. This defines a rich agenda for future work.
Authors’ note: This research was funded by the Federal Government of Nigeria, the International Growth Centre, the Economic and Social Research Council, the Institute for Fiscal Studies, and the Royal Economic Society.
Because modern managers are usually trained to see metrics as essential since without them what would they do with Excel? Just as teachers are now encouraged to teach to the test managers at all levels must have charts and graphs to show progress. This creates a culture in the workplace that tends to value numbers over people–not because managers are “bad” but because that’s what is required in most organizations.
I’ve seen it become a huge overhead problem when reports are required even for casual meetings. On the other hand I’ve seen organizations in government that operate in some nether-land where nothing is done and nothing accomplished and nobody notices or cares. Choose your poison. What really motivates organizations and projects are engaged managers who do the numbers if they have to but rule by pragmatic principles and pay attention to the people they manage.
Your observation that managers should “pay attention to the people they manage” is right on the money. How many times have I seen or had to do the retraining of an imported manager or foreman in some commercial construction job! The same applies to Retail. Where I work, the lower middle managers, (really, hopped up floor workers,) are regularly moved about, fostering a disconnect between the particular tasks the department covers and the department manager. Yet, those same managers are expected to be ‘experts’ in the arcana of their respective departments’ subject matters. This is the interface point between true store managers, and retail ‘experts.’ I personally was hired on the strength of my ‘real world’ experience and knowledge of a particular trade, plumbing. I struggle with some of the particularly convoluted computer sales systems. (One requires the salesman to register the sale in one system and then ‘migrate’ it into another to complete the sale.)
As for the subject of metrics; I took a voluntary pay cut to evade the sales “goals” (for which you can be penalized for not meeting quarterly “goals”.) The resultant peace of mind is well worth the financial sacrifice. I feel sympathy with the other salesmen and women who do not have this luxury. Their stress, especially towards the end of each quarter is highly visible.
Let’s not get into how this system affects the mid level store management! (One man I know now has regular migraine headache attacks, which he had thought he had bought under control previous to taking the floor manager position.)
One final observation. For these high stress positions, these people don’t get all that much of a rise from the floor level workers pay scale. It seems, even here, most, if not all of the “productivity” gains are flowing to the corporate tower class. Hah!
I’d go even further, and say “processes” rather than metrics. The whole point is really simple – there are tradeoffs. But few people understand that, and even fewer people understand what those tradeoffs are (and their impact). And, when they do, they get themselves in knots trying to work out how to deal with them (usually by instituting more processes etc.).
At that time, I usually send them to buy Kay’s book on serendipity, but unfortunately that (at best) means they try to work out a process to work out a goal they should have as an organization, so I’m close to giving up even on that.
Great read. The angle I would add is that the two primary problems with metrics are when the goal being quantified is the wrong goal and/or when the methodology of quantifying the goal is poorly designed/implemented.
I think metrics get a bad rap in general because management is using them inappropriately (as a shorthand way of arbitrarily sorting people or pretending that progress is happening), not because the idea of quantifying outcomes is inherently flawed. The underlying philosophy that is the problem is the authoritarian ideas of hierarchy and secrecy, not the mathematical concept of numbers. Non-quantifiable methods of evaluation run into their own host of challenges, of course. And not evaluating people and projects at all creates a third interesting set of consequences that essentially paralyzes organizations over time.
“Understanding what drives the effective functioning of bureaucracies matters to economists.”
I would add the word should to that sentiment :)
My pops was a civil servant and one thing I definitely got, listening to him rant as a kid, was that performance reviews are beyond worthless. He’ll be happy to see that there is now “scientific” backing for his contention.
p.s. what’s with the bold?…is everyone seeing this?
Uh, note to the editor:
I think you may have left out the closing bold tag in the last part of the article – the rest of the text on the page is bold.
Performance reviews are ostensibly a means given to workers to ensure they are getting fair compensation and advancement for productivity. In reality, they are nothing but a means of controlling labor costs including the “labor” of middle management.
It’s true that you will occasionally get a manager who write good reviews (constructive criticism and recognition of accomplishments), but then you also occasionally get someone who wins the lottery. Overall, the result remains the same: depress salary and advancement based on merit.
Two numbers stand out:
• First, 38% of projects never even start.
• Second, only 31% of projects are fully completed.
I’d look to management to discover why. It is always management who are at fault, and rarely the people actually doing the work.
Rarely are the people doing the work incompetant. Frequently the manage is incompetet, either becuse the managers are fools, or there is another agenda.
This caught my eye.
‘The most effective private-sector management practices may not be suited to public sector bureaucracies.’
What is the evidence that these practices are effective in the private sector? During my accounting career I worked in a huge conglomerate, a small company, and a small government agency. During the time I watched “management by objective” style performance evaluations being implemented from large to small to government. I also saw performance bonuses starting to be used in govt. I liked the more complex performance reviews because they were a chance to find out where your boss wanted to go in the coming year. Performance bonuses were everywhere mostly about favoritism which did more harm than good. But my experience with both was no different in government than the private sector.
In a healthy organization with good managers many systems will work. But those are rare.
Right on. That line was a huge credibility killer for me, too. What cracks me up is not only the assumption that private sector practices are effective, but also the assumption that such practices haven’t already been imported into the public sector of management.
To be clear, I enjoyed the piece and think it’s an important topic. It’s just unfortunate that the authors talk in sweeping language rather than specifics of what practices work and what don’t. The generalities could be written by a high school student who doesn’t even know where Nigeria is.
For example, according to Google, this 20 year old article took .6 seconds to find:
Now I’m wondering if Yves knows Jude T. Rich and John A. Larson from McKinsey days.
You’re missing an HTML tag ( /B ) after “Concluding Remarks”.
Interesting. Shouldn’t the same categories used to analyze and manage an economy apply equally well to any organized human activity? Why should managing a group of bureaucrats be any different than managing a market? The same principles should apply. After all, economics is just a branch of behavioral science.
The discussion about the print being bold is just as relevant as the discussion of making group activity more efficient.
If you are indeed using the correct metric, then you will find that groups are nearly 100% efficent in stealing from individuals, which is their main purpose.
Property is nothing more than a social consensus that claims to exclusion should be strongly supported, and the protocols by which such claims are laid, traded, and alienated. It’s not revealed wisdom…
One thing metrics do is avoid the question of what to do with a high performing employee. For instance, what if you had talented operators who knew their organization well and could correctly get phone calls to the right place? Why, you would have to pay them more for the talent and knowledge. How about, if instead, you design a phone tree that meets very few of the needs of the average caller, and then have the first person that the caller meets be a person with no authority? These people might burn out after 2 months on the job, but you can pay the minimum wage plus 50 cent an hour, plus you can now run all kinds of metrics. Yeah!
Actually, it depends. There’s quite a body of evidence which shows that if you pay people enough to be reasonably comfortable, but provide stability and environment where they feel ok and feel they can make a difference (to others etc), they will be quite happy.
Of course, if you train your people to think that the only measure of success and impact is a higher salary and a huge bonus, then don’t be surprised that they behave with only this in their minds… My longstanding opinion is that we as a society (and I don’t mean US, not living there, but wider “western civilization & more”) started going downhill when a societal status became a thing measured more or less in money and nothing else.
Need I point-out the obvious fact that banks are large bureaucracies too?
I always get a good chuckle when small government advocates insist ‘this would never happen in the private sector’.