Leaders fear AI they cannot explain

If you cannot explain what your AI is doing, can you really trust it? That is the question sitting at the back of every board room conversation about AI right now. It is a reasonable question. Recent research found that a quarter of executives could not explain what their AI outputs actually represent. Only 22% said they could explain them easily. For senior decision-makers and compliance officers already anxious about GDPR, that statistic lands hard. The myth is not that AI is useless. The myth is that opacity is unavoidable.

Why this concern is completely understandable

AI tools arrived in most businesses before any governance framework did. Leaders were handed software that produced outputs without a clear audit trail. Nobody explained the logic behind a recommendation. Nobody flagged where a human should have checked the result.

The GDPR concern compounds this. Where AI processes personal data or informs financial decisions, such as audits, expenses, or accounts payable, UK data protection law gives individuals a right to meaningful information about significant decisions made about them by automated means. If the business cannot explain the AI, it will struggle to meet that duty. The anxiety is not irrational. It is grounded in a real accountability gap.

For UK SMEs, the stakes are higher than they appear. Many organisations in this bracket rely on a small number of senior decision-makers to approve new systems. If leadership cannot understand the output or articulate the compliance position, adoption stalls. The use case might be entirely sound. The blocker is the explainability gap, not the technology.

What the evidence says about opacity and trust

The research is instructive here. A quarter of executives could not explain their AI outputs. But the inverse is also true: three quarters of executives are engaging with AI closely enough to have a view. The gap is not between AI-users and AI-refusers. The gap is between organisations that designed explainability in from the start and those that bolted it on, or ignored it entirely.

Three practical evidence points are worth noting.

First, workflow transparency is a design choice, not a feature toggle. AI systems built with structured logging, clearly defined inputs, and documented decision points are auditable by default. Systems deployed without those constraints are not. The difference is in how the assistant was configured, not in the underlying model.

Second, the GDPR risk is real but manageable. Research on UK AI regulation shows many firms are unprepared for compliance review. The firms that fare best are those that restrict AI initially to lower-risk, repeatable tasks where outputs can be checked easily. Starting with tasks that already have a human sign-off step is not a limitation. It is a sensible risk posture.

Third, human oversight is not a workaround. It is the design requirement. Documenting where human review is mandatory, and building that into the workflow rather than leaving it to individual judgement, is what separates accountable AI use from unaccountable AI use. Senior decision-makers who insist on that documentation are not slowing adoption. They are making it durable.

A realistic approach for senior leaders and compliance officers

Explainability does not require the business to understand machine learning. It requires the business to be able to answer three questions about any AI-assisted process.

1. What data did the AI use to produce this output?

2. Where is the human sign-off point, and who is responsible for it?

3. If this output is wrong, how would we detect that, and how quickly?

If the current setup cannot answer those three questions, the issue is not the AI. The issue is that the deployment was not designed with governance in mind.

At gecco, we build AI Assistants using the GRAFT methodology, which defines Goal, Role, Audience, Format, and Tone for every assistant from the outset. That structured design approach means the assistant's intended scope is documented before it is deployed, making explainability a natural output of the build process rather than a retrofit. For senior leaders weighing explainability against adoption risk, that kind of structured design is one practical starting point.

Considerations and honest caveats

Explainability is not a complete answer to every governance concern. There are classes of AI output, particularly those involving generative text or probabilistic scoring, where even a well-documented system cannot guarantee a fully deterministic audit trail. That is a real limitation, and it matters in regulated contexts.

The honest position is this: no AI deployment eliminates accountability risk entirely. What good design does is make the risk visible and manageable rather than hidden and compounding. Compliance officers should treat explainability as a floor, not a ceiling. The question is not whether the AI can be fully understood in every case. The question is whether the business has documented its oversight responsibilities clearly enough to demonstrate reasonable care.

There is also a cultural dimension that the data alone does not capture. Senior decision-makers who feel excluded from the AI conversation will block adoption even when the governance is sound. Explainability work is, at its core, communication work. It requires the people who built the system to explain it in terms that make sense to the people who are accountable for it. That is an 80% people challenge, and only 20% a technology one.

Ready to close the explainability gap in your organisation

If you are a senior decision-maker or compliance officer trying to assess whether your current AI setup could survive a governance review, the AI Readiness survey is the right place to start. Take the AI Readiness survey. You will get access to 65+ free resources and a custom AI Readiness report. We then offer a free 30-minute showcase to walk through your results.

Explainability is not a nice-to-have. It is a design requirement. AI systems that cannot be interrogated by leadership will fail governance review, and that failure starts in the assistant design phase, not the audit. If you want structured help building AI Assistants with explainability built in from the start, that is exactly what gecco does.

Website · LinkedIn · Case Studies · Newsletter