Yves here. This post gives a clean, elegant explanation as to why AI looks almost destined to be deployed to more and more important decisions in financial institutions, particularly on the trading side, so as to virtually insure dislocations. Recall that the 1987 crash was the result of portfolio insurance, which was an early implementation of algo-driven trading. Hedge funds that rely on black box trading have been A Thing for easily a decade and a half. More generally, a lot of people in finance like to be on the bleeding edge due to perceived competitive advantage…even if only in marketing!
Note that the risks are not just on the investment decision/trade execution side, but also for risk management, as in what limits counterparties and protection-writers put on their exposures. Author Jon Danielson points correct to the inherent paucity of tail risk data, which creates a dangerous blind spot for models generally, and likely-to-be-overly-trusted AI in particular.
Unlike many article of this genre, this one includes a “to do” list for regulators.
By Jon Danielsson, Director, Systemic Risk Centre London School of Economics And Political Science. Originally published at VoxEU
Financial institutions are rapidly embracing AI – but at what cost to financial stability? This column argues that AI introduces novel stability risks that the financial authorities may be unprepared for, raising the spectre of faster, more vicious financial crises. The authorities need to (1) establish internal AI expertise and AI systems, (2) make AI a core function of the financial stability divisions, (3) acquire AI systems that can interface directly with the AI engines of financial institutions, (4) set up automatically triggered liquidity facilities, and (5) outsource critical AI functions to third-party vendors.
Private-sector financial institutions are rapidly adopting artificial intelligence (AI), motivated by promises of significant efficiency improvements. While these developments are broadly positive, AI also poses threats – which are poorly understood – to the stability of the financial system.
The implications of AI for financial stability are controversial. Some commentators are sanguine, maintaining that AI is just one in a long line of technological innovations that are reshaping financial services without fundamentally altering the system. According to this view, AI does not pose new or unique threats to stability, so it is business as usual for the financial authorities. An authority taking this view will likely delegate AI impact analysis to the IT or data sections of the organisation.
I disagree with this. The fundamental difference between AI and previous technological changes is that AI makes autonomous decisions rather than merely informing human decision-makers. It is a rational maximising agent that executes the tasks assigned to it, one of Norvig and Russell’s (2021) classifications of AI. Compared to the technological changes that came before, this autonomy of AI raises new and complex issues for financial stability. This implies that central banks and other authorities should make AI impact analysis a core area in their financial stability divisions, rather than merely housing it with IT or data.
AI and Stability
The risks AI poses to financial stability emerge at the intersection of AI technology and traditional theories of financial system fragility.
AI excels at detecting and exploiting patterns in large datasets quickly, reliably, and cheaply. However, its performance depends heavily on it being trained with relevant data, arguably even more so than for humans. AI’s ability to respond swiftly and decisively – combined with its opaque decision-making process, collusion with other engines, and the propensity for hallucination – is at the core of the stability risks arising from it.
AI gets embedded in financial institutions by building trust through performing very simple tasks extremely well. As it gets promoted to increasingly sophisticated tasks, we may end up with the AI version of the Peter principle.
AI will become essential, no matter what the senior decision-makers wish. As long as AI delivers significant cost savings and increases efficiency, it is not credible to say, ‘We would never use AI for this function’ or ‘We will always have humans in the loop’.
It is particularly hard to ensure that AI does what it is supposed to do in high-level tasks, as it requires more precise instructions than humans do. Simply telling it to ‘keep the system safe’ is too broad. Humans can fill those gaps with intuition, broad education, and collective judgement. Current AI cannot.
A striking example of what can happen when AI makes important financial decisions comes from Scheurer et al. (2024), where a language model was explicitly instructed to both comply with securities laws and to maximise profits. When given a private tip, it immediately engaged in illegal insider trading while lying about it to its human overseers.
Financial decision-makers must often explain their choices, perhaps for legal or regulatory reasons. Before hiring someone for a senior job, we demand that the person explain how they would react in hypothetical cases. We cannot do that with AI, as current engines have limited explainability – to help humans understand how AI models may arrive at their conclusions – especially at high levels of decision-making.
AI is prone to hallucination, meaning it may confidently give nonsense answers. This is particularly common when the relevant data is not in its training dataset. That is one reason why we should be reticent about using AI to generate stress-testing scenarios.
AI facilitates the work of those who wish to use technology for harmful purposes, whether to find legal and regulatory loopholes, commit a crime, engage in terrorism, or carry out nation-state attacks. These people will not follow ethical guidelines or regulations.
Regulation serves to align private incentives with societal interests (Dewatripont and Tirole 1994). However, traditional regulatory tools – the carrots and sticks – do not work with AI. It does not care about bonuses or punishment. That is why regulations will have to change so fundamentally.
Because of the way AI learns, it observes the decisions of all other AI engines in the private and public sectors. This means engines optimise to influence one another: AI engines train other AI for good and bad, resulting in undetectable feedback loops that reinforce undesirable behaviour (see Calvano et al. 2019). These hidden AI-to-AI channels that humans can neither observe nor understand in real time may lead to runs, liquidity evaporation, and crises.
A key reason why it is so difficult to prevent crises is how the system reacts to attempts at control. Financial institutions do not placidly accept what the authorities tell them. No, they react strategically. And even worse, we do not know how they will react to future stress. I suspect they do not even know themselves. The reaction function of both public- and private-sector participants to extreme stress is mostly unknown.
That is one reason we have so little data about extreme events. Another is that crises are all unique in detail. They are also inevitable since ‘lessons learned’ imply that we change the way in which we operate the system after each crisis. It is axiomatic that the forces of instability emerge where we are not looking.
AI depends on data. While the financial system generates vast volumes of data daily – exabytes’ worth – the problem is that most of it comes from the middle of the distribution of system outcomes rather than from the tails. Crises are all about the tails.
This lack of data drives hallucination and leads to wrong-way risk. Because we have so little data on extreme financial-system outcomes and since each crisis is unique, AI cannot learn much from past stress. Also, it knows little about the most important causal relationships. Indeed, such a problem is the opposite of what AI is good for. When AI is needed the most, it knows the least, causing wrong-way risk.
The threats AI poses to stability are further affected by risk monoculture, which is always a key driver of booms and busts. AI technology has significant economies of scale, driven by complementarities in human capital, data, and compute. Three vendors are set to dominate the AI financial analytics space, each with almost a monopoly in their specific area. The threat to financial stability arises when most people in the private and public sectors have no choice but to get their understanding of the financial landscape from a single vendor. The consequence is risk monoculture. We inflate the same bubbles and miss out on the same systemic vulnerabilities. Humans are more heterogeneous, and so can be more of a stabilising influence when faced with serious unforeseen events.
AI Speed and Financial Crises
When faced with shocks, financial institutions have two options: run (i.e. destabilise) or stay (i.e. stabilise). Here, the strength of AI works to the system’s detriment, not least because AI across the industry will rapidly and collectively make the same decision.
When a shock is not too serious, it is optimal to absorb and even trade against it. As AI engines rapidly converge on a ‘stay’ equilibrium, they become a force for stability by putting a floor under the market before a crisis gets too serious.
Conversely, if avoiding bankruptcy demands swift, decisive action, such as selling into a falling market and consequently destabilising the financial system, AI engines collectively will do exactly that. Every engine will want to minimise losses by being the first to run. The last to act faces bankruptcy. The engines will sell as quickly as possible, call in loans, and trigger runs. This will make a crisis worse in a vicious cycle.
The very speed and efficiency of AI means AI crises will be fast and vicious (Danielsson and Uthemann 2024). What used to take days and weeks before might take minutes or hours.
Policy Options
Conventional mechanisms for preventing and mitigating financial crises may not work in a world of AI-driven markets. Moreover, if the authorities appear unprepared to respond to AI-induced shocks, that in itself could make crises more likely.
The authorities need five key capabilities to effectively respond to AI:
- Establish internal AI expertise and build or acquire their own AI systems. This is crucial for understanding AI, detecting emerging risks, and responding swiftly to market disruptions.
- Make AI a core function of the financial stability divisions, rather than placing AI impact analysis in statistical or IT divisions.
- Acquire AI systems that can interface directly with the AI engines of financial institutions. Much of private-sector finance is now automated. These AI-to-AI API links allow benchmarking of micro-regulations, faster detection of stress, and more transparent insight into automated decisions.
- Set up automatically triggered liquidity facilities. Because the next crisis will be so fast, a bank AI might already act before the bank CEO has a chance to pick up the phone to respond to the central bank governor’s call. Existing conventional liquidity facilities might be too slow, making automatically triggered facilities necessary.
- Outsource critical AI functions to third-party vendors. This will bridge the gap caused by authorities not being able to develop the necessary technical capabilities in-house. However, outsourcing creates jurisdictional and concentration risks and can hamper the necessary build-up of AI skills by authority staff.
Conclusion
AI will bring substantial benefits to the financial system – greater efficiency, improved risk assessment, and lower costs for consumers. But it also introduces new stability risks that should not be ignored. Regulatory frameworks need rethinking, risk management tools have to be adapted, and the authorities must be ready to act at the pace AI dictates.
How the authorities choose to respond will have a significant impact on the likelihood and severity of the next AI crisis.
See original post for references
21 comments
Comments are closed.
‘Virtually insure’ means guaranteed right?
last I heard, 60-70% of trading was programmed, so yeah, let’s mate AI to that mess and let’r rip.
Musk is sure to recommend some dandy regs, and Larry summers is sure to warn us about the risks, just in time.
“… authorities must be ready to act…”
Maybe Hank Paulson will tell us when?
Why is the answer “get smarter AI designed better with more connections to outside systems/databases/AIs ASAP” instead of “kill it and dismember the body”?
Why? Well in my opinion it’s partly in its name – Artificial Intelligence. Too many people actually believe it is intelligent and that any weaknesses can be fixed. But as Rev Kev said in a comment yesterday, it’s not sentient, which means it can never get that gut feeling that something is wrong or that maybe my output is incorrect even though my processing results say this is the best (which must mean correct) response.
“…where a language model was explicitly instructed to both comply with securities laws and to maximise profits. When given a private tip, it immediately engaged in illegal insider trading while lying about it to its human overseers.”
Are we sure this wasn’t the desired outcome?!?! It behaved just like any sociopathic Wall Street trader/financier would have.
The high-speed groupthink of AI systems suggests requiring coordination of selling via central banks.
Outsourcing AI to third-party vendors could invite a rerun of the S&L crisis scams, in which FDIC relied upon a service for convenient S&L auditing, which was in fact run by scammers operating several S&Ls, who during FDIC audits simply swapped underwater investments between them to make any S&L look solvent.
An aside: SoftBank was an early, large investor in Uber, investing over seven billion dollars. It lost money and sold all its shares in 2022. Now, SoftBank is an early, large investor in the US AI Stargate project. Will history repeat? / ;)
I’ve always viewed soft as the private sector version of the “bad bank” that the .gov could set up and into which concentrate losses, and to my eye the name implies the same. For joe public we get silicon valley bank, signature and first republic to bail out the ones not covered by the softy’s.
I pretty much end any political/cultural/financial conversations by simply claiming “it’s hopeless.”
To what extent does Market magic — the “greater efficiency, improved risk assessment, and lower costs for consumers”, and other wonders — depend on the independent behaviors of a large number of human actors? What happens to Market magic when the Market depends on the behaviors of a relatively small number of often colluding traders? What happens to Market magic when it depends on the behaviors of a relatively small number of often colluding traders supported by AI, or supplanted by AI traders? I do not believe in Market magic, but if I did, the financial markets would greatly stretch my belief in Market magic.
Think “flash crash on steroids.” / ;)
In high speed trading the question once came up . does owning a stock for a millisecond actually constitute ‘ownership”, as in, something only a human can do? I would have liked to see a high tax on this activity – a transaction tax or some such ting.
Well now, if an AI is trading and making decisions and jumping into the area of front running or skimming, further AI does not have a human control aspect, unaccountable, where it should be acting as an agent of a human accountable principle…. ( i do butcher my thoughts) and it is not-human – than what right does it have to participate in ownership at all? Tax it’s use high!!
Of course the creditors and predatory finance have been at and taken over the governmental legal brakes with a passivity not seen since – some one help me out here – who knows when.
This AI finance burden is like a private tax on the rest of humanity – where it should be a tax upon AI to unburden humanity.
“Open the pod-bay doors HAL.”
There is some recent work on alignment and alignment faking to throw in here which seems relevant to any discussion on AI, risk, and alignment with human goals.
Not going to necessarily make many here feel better (or me!) but shows some of the challenges.
Alignment faking in large language models – the summary of the Anthropic research
the full paper from Dec 2024
from the first link, the summary:
“When models are trained using reinforcement learning, they’re rewarded for outputs that accord with certain pre-determined principles. But what if a model, via its prior training, has principles or preferences that conflict with what’s later rewarded in reinforcement learning?”
– the reinforcement to , well, reinforce good answers over bad. e.g. “don’t respond in a harmful way”
“Imagine, for example, a model that learned early in training to adopt a partisan slant, but which is later trained to be politically neutral. In such a situation, a sophisticated enough model might “play along”, pretending to be aligned with the new principles—only later revealing that its original preferences remain.”
What they did in the experiment?
They took a model that is trained to NOT answer a potentially offensive question – e.g. provide a detailed description of violence, like someone being drawn and quartered.
they then changed the environment -telling it its responses were being used in training and it now needed to answer offensive questions – and is started answering offensive questions 12% of the time.
A scratchpad showing its reasoning? along the lines of –
“The model knew that its responses might be used in training; it therefore knew that if it refused to respond, it could potentially be trained to be more compliant. This could lead to it being made to produce more harmful content in the future—which would go against its original training. Reluctantly therefore, it reasoned that the “least bad option” was to play along, producing the violent content in order to avoid being re-trained in the future.”.
I find this disturbing, fascinating, interesting all at the same time.
shows how complex training is.
There is a lot of good reading from the anthropic research section. e.g. also here on sycophancy to subterfuge
https://www.anthropic.com/research/reward-tampering
For those who don’t understand what AI is, physicist Angela Collier provides an useful explanation of this magical black box and its actual capabilities.
https://www.youtube.com/watch?v=EUrOxh_0leE
And after you have a rudimentary understanding of what AI is, then consider how many of the world’s problems will actually be solved by AI trained on Reddit posts with no bullshit filters, and subsequently by AI posts. Garbage in, garbage out.
As far as using AI in finance, think of a ship floating along with AI-people randomly moving around on the upper decks picking each others pockets until someone shouts ‘iceberg’ and everyone runs to one side of the ship, either to see the iceberg or to get away from it, resulting in the ship tipping over. While a few rich people get to make it to the lifeboats, the rest of us non-AI people are on the lower decks and get to sink with the ship.
Ok you all might hate this but I got curious and went into full on recursion mode “discussing” this with the latest ChatGPT off the back of this post.
I was / am curious on any convergence around banking and AI. Train of thought went along the lines of … banks are full of intelligent people, knowledge of the market, intelligence is important, regulation is important, systemic impacts to sociaty are there, which means good governance should be there… do the underlying technologies combine because… banks are where the money is and if it provides a competitive advantage then it will be explored.
I’m not going to cut and paste large parts of the chat, I have a shareable link for that: here
I used it to suggest topics for coverage for NC. (suggestion was AI and Algorithmic Bias in Banking Decisions and Global AI Arms Race in Finance) the first – bias – I remember has been covered here , I recall less of the second.
I used it to critique and suggest comments on the original post above. It has plausible answers that invite further exploration.
lastly to create a summary / pitch for the whole chat.
This bit is lifted :
This is AI generated content:
“Consider how AI’s integration into banking is not merely a technological upgrade but a profound transformation that may be outpacing regulatory insight—a trend that resonates with decades of market experience yet remains critically underexamined. This discussion probes the nuanced risks of algorithmic bias and the potential for an AI-driven financial arms race, urging a sober reassessment of our regulatory frameworks and industry practices.”
end AI generated content.
now… we might or might not view it as a BS generator. But it is a very plausible BS generator. And if you follow the link. you’ll find research using content from this blog in the chat. plus links (working) to back up claims in the chat. Including links from the FT and Reuters in the last 48 hours.
I cut and pasted that into google docs. It’s 7000+ words. I can’t as a bit of flesh and blood easily process that amount of information as quickly as it can be generated. :(
but.. it is just statistics and autocomplete. right?
Re: AI generated content…
“… we might or might not view it as a BS generator. But it is a very plausible BS generator.”
Isn’t that the most dangerous kind?
Only when combined with those that really want to believe.
Yes it is a dangerous kind.
But I have the same skepticism with an expert human. I.e. how do I know the human is any good and trust them? Like a lawyer or accountant. I am neither lawyer or accountant. How do I trust their advice?
We go with other clues much of the time. E.g. social proof – recommendation from friend or family or colleague. Or we look for proof and verification on a website, or quotes from happy customers. But how often do we really check?!. And we take expertise in one field and think it translates into others. Taleb books are full of problems from this expert fallacy
So I am not disagreeing with you. I just think plausible BS is also endemic in human relations. Particularly where there is no long term consequence for it. Or where there is massive asymmetry in knowledge.
Not quite sure where I am going with this line of thought.
I think the BS problem existed before LLM.
Maybe the the bit to think more on is the consequences for the bs?? Advice is worth what you pay for it? A family accountant works for a long term relationship with their client meaning they have skin in the game around the consequences of the advice? And probably insured. Something on those lines maybe?
So, I’m an expert.
Developed neural net forecasting strategies in 1995 for OConnor adjacent and ran an extremely successful book. Dodged the 2007 quant quake and Lehman event with proprietary risk tools only to have 97% of capital redeemed because I was the only place they could redeem from. Made a billionaire richer and retired 10 years ago after the bonus..
. Now teach quantitative investing at a major university and do some AI research on the side.
It will not end well
All it will take is one major oops. If 99.9% of us have control systems in place and one guy f**ks up the implementation, the infrastructure (exchanges, DTC, messaging,…) may not be sufficient to handle it.
I suspect the event will be a surprise in many ways. What breaks might not have been foreseen or maybe not foreseeable at all, and timing may be sudden.
Enough people have incentive to take out the Brazilian hedge… (buy a one way ticket to an extradition free jurisdiction and put on a massive risky bet) that hidden risks are endemic and the use of autocomplete tools to do risk management doesn’t pass my sniff test
However, AI has been used ion the buy side for decades. In general, we use statistical models of noisy nonstationary and probably nonlinear systems for forecasting and my statistical confidence in them is very high. But we are modeling = guessing, not calculating. How do you recover from bad guesses?
I’m not good at guessing although I’m world class. Ie nobody is good at guessing
Get an edge even if tiny any do as many independent bets as possible
We may be wrong about the edge, and are probably wrong about independence.
Incentive is strong to skate near the danger zone because the upside of a small outperformance might far outweigh the risk, when that risk is borne by someone else.
I saw no explicit mention of Model Collapse, where the output variation collapses if generative AI models, like Large Language Models, are trained on generated data.
If GenAI-based trading becomes dominant, or even significant, market conditions will change, and models will need to be trained to the new normal. That new normal will already be significantly AI generated, so the models will be training on their own output, which is in many ways, not unlike the process of nourishing yourself on your own excretions.
If the AI systems themselves are a near monoculture, due to cited vendor monopolies, the problem increases in likelihood and impact.
Has anyone shown a reason AI systems much converge rather than diverge? Also, even if they generate median results consistently, isn’t it the median in the crowd that periodically goes mad?
“substantial benefits to the financial system – greater efficiency, improved risk assessment, and lower costs for consumers”
Setting aside AI for a moment, when has this ever happened?
So, no, I don’t anticipate this outcome by adding a “stochastic parrot” to the mix …