AI Governance Must Shift From Static Reviews to Continuous Assurance
- Explores how organizations should determine when AI requires lightweight versus formal governance.
- Examines strategies for scaling AI governance without creating bottlenecks.
- Highlights common mistakes when applying traditional model risk frameworks to generative AI.
- Discusses the shift from periodic reviews to continuous AI assurance and monitoring.
- Explores how governance must evolve as AI agents become more autonomous.
display-middle-ads-here
1. Many organisations claim risk-based AI
governance. In practice, what determines lightweight oversight versus formal
governance and independent review?
The question I actually ask isn’t “is this generative, is it agentic?” It’s simpler: what can this thing do on its own, how bad is it if it’s wrong, and can we catch and undo the mistake?
Five things drive that. Two set the floor – how much authority it has (does it just read, or recommend, decide, or act?) and how big the consequence is if it fails. Three others push the risk up – how autonomously it runs, how many people or how much money it touches, and how easily an error is caught and reversed. But those three are capped: they can’t turn a trivial task into a critical one just because it’s clever or fast. Consequence stays in charge.
So the lightweight path is for genuinely assistive, low-stakes work, where a person still owns the decision and a mistake is cheap and quickly undone. Formal governance and independent challenge are for the things that can move money, make a customer decision, or touch a regulated outcome – especially when a failure happens at machine speed and is hard to walk back.
Two cautions. The tier is only as honest as whoever
scored it – if the business grades its own risk, it’ll grade low, so you sample
and challenge those classifications. And don’t look at use cases one at a time:
a hundred low-risk tools pointed at the same customers can add up to something
material.
2. As AI adoption accelerates, how do you prevent
governance from becoming a bottleneck while ensuring high-risk models get the
scrutiny they deserve?
You keep it from becoming the bottleneck by deciding the rules ahead of time, instead of arguing about every deployment.
Write the tiering rubric and the controls for each tier up front. Let the business run the low-risk work themselves – classified automatically, routed through the controls that already apply, and logged so it’s auditable. Save your scarce second-line reviewers for the high-risk tiers, the exceptions, and the material changes – because a low-risk app can turn high-risk the moment its data, authority, or user base changes.
Two cautions. Self-serve without an audit loop is
just under-governance with a nicer interface – so sample what the business
scored itself, back-test it, and watch the under-tiering rate. And measure the
quality of reviews, not just the speed; optimize only for time-to-tier and
review quietly turns into rubber-stamping.
The failure you’re avoiding is the classic one: a fixed review queue meets exploding demand, the wait gets long, people go around it, and soon you can’t see what’s running. So make the governed path the fastest path. If the safe way is also the easy way, almost nobody bothers to go around it.
3. Where do organisations make the biggest mistakes
when extending traditional model governance frameworks to generative AI?
A few big ones.
The first is governing the model when the risk has moved to the system and the agent. Whether the output is accurate is only half of it – the real question is what the thing is allowed to do with that output.
The second is treating validation as a one-time event. A sign-off tells you it was fine on day one, but these systems don’t sit still: the data ages, the prompts drift, the tools change, and a vendor can swap the model underneath you without changing anything you can see.
The third is rolling everything into a single reliability number and letting a good average hide a fatal flaw. Some checks should be gates, not averages – one safety or authorization breach can be disqualifying no matter how healthy the score looks.
The fourth is letting anything that doesn’t fit the old template quietly fall out of the inventory. That’s how shadow AI grows – not by decision, by omission.
But the subtle one is bringing over the vocabulary of model risk without the discipline. Model risk teaches you to test for real coverage, to make sure your sample can actually detect the problem, and to independently challenge any model you rely on – including the one doing the checking. Then in GenAI we watch a dashboard with nothing behind it and let one model grade another and call it oversight. The old framework isn’t useless – its discipline matters more than ever. What changed is the thing you’re governing: it grew from the model to the whole system around it – the tools, the data, the prompts, the agents.
4. Over the next three to five years, will AI
governance move toward continuous, automated assurance rather than periodic
reviews – and what capabilities will that take?
Yes – and I don’t think it’s optional. You can’t
review this much, this fast, on a quarterly cycle. The math doesn’t work.
So assurance shifts from collecting documents to producing evidence, and from periodic checks toward something continuous. But I’d be straight about what “continuous” means, because it isn’t all real-time. A few checks fire the instant something happens – a safety or authorization gate. Others, like whether an answer is grounded or well-calibrated, are near-real-time. Robustness and consistency are really batch. And the number you show an executive is always a little behind. Anyone who says every risk is measured live is selling false comfort.
To get there you need a few things: a way to
discover what’s actually running, not what’s on a list; tiers that are
computed, not negotiated; guardrails written as code, with telemetry, right
where the agent calls its tools; the right signals and thresholds per tier,
with the critical ones as hard stops; and KPIs that tell you whether the
control system itself is working.
The capability most firms are missing ties all that together – call it AI governance engineering: turning policy and risk requirements into controls, telemetry, and evidence that actually run. And remember the thing doing the measuring is a model too – know its calibration and blind spots, and challenge it like any other model.
The tier is a promise. Runtime evidence is how you keep it.
5. As AI agents become more autonomous, how should
proportionate governance evolve – are today’s risk classifications sufficient,
or do we need fundamentally new approaches?
Tiering still works – but with agents it has to run
at two levels.
At the component level, the question isn’t how “senior” an agent looks – it’s what you’ve actually handed it: what data it can reach, what decisions it can influence, what tools it can call, and whether it can put other agents to work. A junior-looking agent that can move ten million dollars matters more than a “manager” agent that only writes summaries – so govern the authority, not the org chart. And like any employee, you don’t hire and never check again: there’s a probation period, a review, and always a human who owns the outcome. Autonomy you can delegate; accountability you can’t.
At the system level, the trap is trying to watch every interaction – do that and you watch none of them. The skill is finding the critical path – the chain of decisions that carries the authority and consequence – and tiering that path by what happens if it goes wrong.
The genuinely new problems live off that path: agents cascading one error into an outage, colliding over a resource, colluding around a limit that would’ve stopped each one alone, poisoning shared memory, or quietly escalating their own privileges. You can’t script for that, so you watch for it at runtime. More and more it concentrates in the harness – the orchestration, tools, memory, permissions, and guardrails around the agents. That’s the control plane, so govern it directly, and treat added complexity as something that has to earn its place.
So no, we don’t throw classification away – we make it computed, composable, and re-checked continuously. And on a high-consequence path, “fully autonomous, nobody accountable” just isn’t a control model.
VP – Applied Responsible AI, First Citizens Bank
Satyaki Ghosh Dastidar, Ph.D. is a Model Risk Management executive with 20 years of extensive experience strengthening risk posture and ensuring regulatory alignment across large financial institutions such as JPMorgan Chase, Wells Fargo, Discover (now Capital One). Currently, he leads Applied Responsible AI at First Citizens Bank. He has a track record in establishing AI/GenAI model risk governance frameworks, leading enterprise model validation and model development programs, and partnering cross-functionally to drive effective risk management. Satyaki holds a Ph.D. in Operations Research from University at Buffalo, NY. His specialties include Model Risk Management, Applied Responsible AI, AI/GenAI Risk Management, Model Development and Validation.
LinkedIn profile