Joe Breeden challenges the widespread focus on explainability in AI risk management, arguing that the real issue lies in predictive uncertainty. Explainability is only valuable when model outputs are reliable, yet most institutions fail to measure confide
Explainability
has become one of the most debated topics in modern risk management. Regulators
expect it, institutions invest heavily in it, and model risk teams continue to
refine techniques designed to make complex machine learning outputs
interpretable. Yet according to Joe Breeden, Chief Executive Officer at Deep
Future Analytics, the industry may be focusing on the wrong challenge.
“We don’t have a
problem with inability to explain,” he says. “We simply have regions that are
not predictive, and if the prediction is not usable, then there’s no point
trying to explain it.”
This distinction
is critical. The prevailing assumption across financial services is that every
model output should be explainable, particularly in high-stakes decisions such
as credit approval or fraud detection. However, Breeden argues that explainability
is only meaningful when the underlying prediction is reliable.
“If you don’t
have a usable forecast, you won’t be able to explain that forecast,” he
explains. “If there’s no good forecast, there’s no good explanation.”
At the heart of
the issue is a lack of focus on predictive uncertainty. Traditional models
often imply a level of confidence based on historical data and relatively
stable relationships. Machine learning models, by contrast, operate in more
dynamic environments where predictive accuracy can vary significantly depending
on the data.
Despite this,
most institutions do not explicitly measure how confident they should be in a
given prediction. “I don’t know any operational systems that regularly compute
forecast uncertainty,” Breeden says. “They have a score, but what’s the
confidence interval around that score?”
Without that
context, organizations risk placing undue confidence in model outputs and the
explanations that accompany them. This can lead to a false sense of assurance,
particularly when widely used explainability techniques are applied
indiscriminately.
Methods such as
SHAP and LIME are often criticized, but Breeden suggests the issue is not with
the tools themselves. “If you apply them only when you have a forecast you’re
confident in, then they’re reliable,” he says. “So we don’t really have a
problem in these methods.”
The real
challenge lies in knowing when to rely on them. This requires a shift in how
institutions approach model governance, placing predictive uncertainty at the
center of decision-making rather than treating explainability as a standalone
requirement.
For financial
institutions, this has direct implications for operational processes. In
situations where a model cannot produce a reliable prediction, fallback
mechanisms become essential. “If you can’t be certain that you have a usable
forecast everywhere, you have to have a fallback,” Breeden explains.