Skip to content

Content Library

Back to Articles

Article November 24, 2025

Banks Race to Validate the Machines They Can't Explain

Banks Race to Validate the Machines They Can't Explain

Mark Norman, Head of Content, Center for Financial Professionals

As financial institutions deploy generative and agentic AI across fraud detection, risk modeling, and compliance, regulators are demanding proof these systems can be trusted. Yet validation remains elusive. With opaque reasoning and unpredictable drift, t

• Banks struggle to validate opaque generative and agentic AI systems
• Regulators demand stronger documentation and accountability frameworks
• Outcome-based testing replaces traditional parameter validation
• Human-in-the-loop controls mitigate automation risks
• Hybrid quant-AI teams emerge to handle new governance challenges
• Validation costs and outsourcing oversight pressure budgets

Generative and agentic AI models are reshaping how banks operate - creating code, summarizing risk data, and even generating credit decisions. 

But their arrival has exposed one of the most urgent challenges in model risk management: how to validate systems that think in ways their creators cannot fully explain.

In traditional model validation, practitioners rely on interpretability. They check assumptions, verify parameters, and re-perform calculations. None of that applies neatly to large language models (LLMs) or multi-agent systems. 

These architectures are not deterministic. They respond to context, data prompts, and internal embeddings rather than static equations.

Regulators have begun to take notice. The U.S. Federal Reserve’s supervisory letters, the EU’s forthcoming AI Act, and the UK’s PRA SS1/23 guidance all demand that firms maintain ‘appropriate validation standards’ for AI systems. 

But what that means in practice remains unsettled. A 2025 Risk.net survey found that over 60% of banks now classify GenAI applications as models under SR 11-7 - but admit they lack the tools to perform independent validation.

Institutions are therefore embracing outcome-based testing. Rather than inspecting the logic of an AI model, they measure whether its decisions remain consistent, fair, and reproducible under stress. 

Perturbation analysis - slightly altering inputs to observe the stability of outputs - is one emerging approach. Another is red-team testing, where specialist teams intentionally provoke hallucinations or bias to understand model behavior under pressure.

Documentation has become a frontline defense. Firms are mapping dependencies, logging prompts, and tracking version control of foundation models to satisfy auditors. 

According to the Bank for International Settlements, effective AI governance now requires “model lineage visibility,” ensuring that all data transformations and fine-tuning steps are recorded for accountability.

But the real frontier is agentic AI - systems that plan, act, and learn autonomously. These are not just chatbots; they are decision-making entities capable of initiating transactions or remediating controls. 

For risk managers, this blurs the line between model validation and operational oversight. “The model becomes an actor,” one senior risk executive told American Banker earlier this year. “You’re validating both a process and a personality.”

To bridge the skills gap, institutions are creating hybrid teams combining quantitative modelers with AI ethicists and cybersecurity specialists. 

They are also deploying human-in-the-loop controls - where every AI decision above a risk threshold triggers a manual review. This approach satisfies both prudence and practicality, maintaining scalability while preventing runaway automation.

Events you may be interested in

View all of our upcoming events

Sign Up to Our Newsletter

Sign up

Connect Plus

Unlock more with Connect Plus membership

This content is available to Connect Plus members.