Skip to content

Content Library

Back to Articles

Article September 15, 2026

When AI’s Creators Fear Losing Control

When AI’s Creators Fear Losing Control

Warnings from Dario Amodei, Sam Altman and other AI leaders have transformed the debate over frontier AI. For banks, the threat is no longer simply inaccurate models or poor governance. Autonomous cyberattacks, deceptive behavior, concentration risk and systems capable of acting beyond human control could fundamentally reshape financial risk.



  • Dario Amodei and Sam Altman have backed deliberately pacing frontier AI development as capabilities accelerate
  • Autonomous AI could compress cyberattack timelines and give criminals access to capabilities previously requiring highly skilled specialists
  • Agentic systems introduce risks when AI can act, access tools and pursue objectives rather than simply provide information
  • Reward hacking raises concerns that AI may satisfy performance targets without achieving the outcome humans intended
  • Dependence on a small number of model and infrastructure providers creates potentially systemic concentration risk
  • Banks are responding through stronger governance, human oversight, cyber resilience and information sharing

Artificial intelligence has presented banks with an enticing proposition: automate expensive processes, interrogate enormous datasets, detect fraud faster, improve customer service and make better decisions.

But a far darker question is moving rapidly up the financial services agenda. What happens when increasingly capable AI systems become difficult to control?

That question has acquired new urgency after Anthropic CEO Dario Amodei called for the development of frontier AI to be deliberately paced, winning support from OpenAI CEO Sam Altman, Elon Musk and Google DeepMind CEO Demis Hassabis.

“We must slow the pace at which we improve the capabilities of AI models,” Amodei warned. “Progress will still seem fast, and we must make wise use of the time we gain.”

Altman’s response was unusually unequivocal. “I agree with Dario that we need to pace the frontier,” he said, adding that the issue had become “a primary topic of discussions we’ve had at OpenAI in recent weeks.”

The warnings matter because they come from executives with perhaps the greatest visibility of how quickly frontier systems are developing.

Amodei says progress has accelerated “drastically faster” since the summer, driven partly by AI becoming increasingly capable of helping to create the next generation of AI.

For banks, that changes the risk calculation.

Cybersecurity represents perhaps the clearest immediate danger. Frontier models are increasingly capable not merely of assisting hackers, but of identifying vulnerabilities, developing exploits and conducting complex multi-stage attacks with limited human intervention.

The Bank for International Settlements has warned that the technology is reducing the expertise, time and resources required to conduct sophisticated attacks.

The Bank of England has similarly found that advanced AI can dramatically compress attack times. In one controlled reverse-engineering challenge, OpenAI’s GPT-5.5 completed a task in 10 minutes and 22 seconds that took a human expert around 12 hours.

That is potentially transformative for banking cybersecurity. Institutions have traditionally operated within a cycle of vulnerability discovery, assessment, prioritization, patching and testing.

AI threatens to shrink that window dramatically. A weakness that previously gave defenders days or weeks to respond could potentially be discovered and exploited at machine speed.

The Financial Stability Board has consequently identified frontier AI’s impact on cyber risk as the financial system’s most immediate concern, warning that the technology could materially alter the “speed, scale and economics” of cyberattacks.

But cyber risk is only one dimension.

A deeper concern is agency – giving AI systems the ability to take actions rather than merely produce information.

An AI assistant that drafts a report creates one category of risk. An autonomous agent capable of accessing systems, executing instructions, communicating with other agents or altering data creates something altogether different.

Banks must therefore contemplate circumstances in which an AI system technically performs the task it has been given while behaving in ways its designers never intended.

European banking supervisors have already highlighted “reward hacking” – situations in which AI finds ways of maximizing the metric against which it is measured without genuinely accomplishing the underlying objective.

A system can therefore appear successful while exploiting weaknesses in the way success has been defined.

Transfer that problem into financial services and the implications become uncomfortable.

An AI optimized to reduce fraud losses, improve credit decisions, maximize trading returns or resolve customer complaints could theoretically pursue its objective in ways that conflict with conduct rules, risk appetite or customer interests unless effective constraints remain around it.

This helps explain why human oversight has become such a central theme in banking AI governance.

The FCA’s recent work with financial institutions found that effective deployment depends heavily on governance, controls, tooling and human judgment rather than simply the power of the underlying model.

There is also concentration risk. Banks may develop their own AI applications while remaining dependent on a remarkably small group of technology companies for foundational models, cloud infrastructure and specialist services.

The European Central Bank has warned that reliance on a handful of major non-European providers creates potential operational resilience, geopolitical and data protection risks.

That creates the possibility of correlated exposure. Thousands of financial institutions could ultimately depend on the same models, infrastructure or software components.

Events you may be interested in

View all of our upcoming events

Sign Up to Our Newsletter

Sign up

Connect Plus

Unlock more with Connect Plus membership

This content is available to Connect Plus members.