Skip to content

Content Library

Back to Articles

Article January 16, 2026

Governing the Black Box of Generative AI

Governing the Black Box of Generative AI

As generative AI enters financial services, traditional validation approaches are falling short. Experts call for a new paradigm - one grounded in risk, transparency, and human oversight - to ensure AI systems remain accountable and aligned with their int

  • GenAI validation adapts model risk principles for opaque LLM systems

  • Clear purpose definition with constraints and quantified tolerance for hallucinations is essential

  • Testing hallucinations is complex due to prompt context and data structure variations

  • Validation decomposes system into LLM RAG architecture embeddings prompts and inputs for targeted checks

  • Stress testing under extreme conditions reveals weaknesses and limits

  • Scope containment defines tested use cases and prompts with stakeholder acceptance of out of scope risks

  • Human in the loop oversight ensures judgment in testing and output review even with automation

  • Documentation of logic use assumptions and limits ensures transparency accountability and auditability

  • Teams require new skills training and cross functional expertise for GenAI validation

  • Focus is on input output testing robustness and component review instead of full model replication

As generative AI finds its way into customer communications, decision support tools, and internal workflows, financial institutions are being forced to rethink what effective model validation really means.

At a recent CeFPro industry panel, senior practitioners made clear that while the technology may be new, the underlying challenge is familiar: how to ensure models behave as intended, remain within acceptable risk tolerances, and can be governed in a way that stands up to regulatory and internal scrutiny.

The Model and AI governance executive at a major European insurance company argued that validation must start with clarity of purpose.

GenAI systems are often capable of answering an almost unlimited range of questions, but that theoretical flexibility is precisely what creates risk.

According to him, institutions must explicitly define what a model is allowed to do, under what conditions, and with what tolerance for error. He emphasized that this is especially critical for externally facing use cases, where even small inaccuracies or so called hallucinations can undermine customer trust.

“You have to decide where your hallucination threshold sits and be very deliberate about it,” the executive said, noting that acceptable tolerance levels may differ sharply between internal productivity tools and customer facing applications.

Quantifying hallucinations, however, is far from straightforward. The speaker explained that outputs can vary significantly depending on prompt phrasing, retrieval methods, and the structure of underlying documents.

Policies written with complex language or double negatives can confuse embedding models, forcing institutions to revisit not just the AI system but the content it consumes.

In this sense, validation becomes a feedback loop, aimed not only at checking performance but also at improving inputs so the model can function more reliably.

A fellow panellist stressed that this mirrors earlier generations of model risk management.

Validation has always required understanding what a model is supposed to do, identifying key risk drivers, and testing how it behaves when those drivers are stressed.

In the GenAI context, those risks include bias, explainability limits, and failure under edge cases. Even if the core large language model cannot be opened or replicated, components around it can still be tested in isolation to understand vulnerabilities.

Breaking the system into parts emerged as a central theme. The governance executive explained that effective validation involves examining the retrieval augmented generation architecture, the embeddings, the prompt logic, the data sources, and the outputs separately.

“You may not be able to touch the core model, but you can still challenge how the system behaves,” the speaker said.

This approach allows validators to focus on input output behavior and component interactions rather than attempting full replication, which is often unrealistic with modern AI.

Another tension highlighted during the discussion was scope. Large language models can respond to almost any prompt, but validation exercises cannot cover every possible question.

The speaker argued that this limitation must be acknowledged explicitly. Validation should be scoped to defined use cases and prompt sets, with management accepting that out of scope prompts may produce unreliable results.

“That acceptance has to be conscious and documented,” he said, adding that governance failures often arise not from what was tested, but from assumptions about what was not.

Human oversight was repeatedly described as ‘non-negotiable’. While automation is essential to handle scale, both the executive and a fellow panellist emphasized that humans must remain involved, particularly when outputs influence decisions or interact with customers.

Events you may be interested in

View all of our upcoming events

Sign Up to Our Newsletter

Sign up

Connect Plus

Unlock more with Connect Plus membership

This content is available to Connect Plus members.