In partnership with

Hey {{first_name | there}},

Imagine a junior credit analyst preparing an assessment for a lending decision.

They examine the applicant's income, review the supporting information and question an unusual figure. Their manager identifies an assumption they missed and explains why it matters.

The assessment gets completed. More importantly, the junior analyst learns something that will inform hundreds of future decisions.

Now imagine an AI system performs the analysis in seconds. An experienced professional reviews the recommendation and approves it.

The process is faster, cheaper and apparently well controlled.

But something important has disappeared: the opportunity for the next generation of professionals to develop the judgement that makes meaningful oversight possible.

If we automate the work that creates expertise, where will tomorrow's experts come from?

This is not simply a question about employment. For financial institutions deploying increasingly autonomous AI systems, it is becoming a question of governance, operational risk and accountability.

The hidden cost of efficiency

In a speech on 10 November, Bank of England Deputy Governor Clare Lombardelli highlighted a troubling feature of Britain's labour market.

Businesses are reducing hiring rather than necessarily making existing employees redundant. AI could accelerate this adjustment, particularly where technology takes over tasks traditionally performed by junior staff.

Lombardelli identified a risk that deserves greater attention: removing junior work may also remove opportunities to learn, accumulate experience and progress.

She acknowledged another possibility. Younger workers with the right skills could become more valuable as AI changes how organisations operate.

Both outcomes are plausible. The difference will depend partly on how employers redesign work.

For financial services, the implications go further than recruitment.

Credit analysts, compliance officers, risk managers and auditors do not develop professional judgement solely through qualifications. They learn through reviewing difficult cases, investigating exceptions, making mistakes and being challenged by experienced colleagues.

If AI absorbs those experiences, organisations may enjoy immediate productivity gains while gradually weakening the human expertise on which their governance frameworks depend.

You know the problem. Now build the fix.

You've spent years close to a broken industry or process. You know exactly what would fix it. But you’ve never had the time or resources to make it happen.

With Lovable, your expertise is all you need. Describe the solution to your problem — a website, a dashboard, an internal tool — and get professional-grade software back. You make every call on how it works, because you're the one who knows what it needs to do.

Start today and take $5 off your first month.

Discount of $5 applies to first month of any paid plan. You need to sign up via this webpage in order to claim the discount. The discount applies to new and returning users without an active paid plan. Terms apply.

The problem with 'human in the loop'

Loop Spinning GIF

Giphy

We frequently hear that humans will remain accountable for AI decisions.

But accountability requires more than assigning someone an approval responsibility.

Consider a lending institution using AI to assess affordability or recommend credit decisions.

A reviewer receives the model's recommendation, supporting explanation and an approval button.

What happens when the underlying assumption is wrong, but the explanation appears convincing?

Can the reviewer detect the error independently? Do they understand the customer's circumstances? Have they encountered enough difficult cases to recognise when something does not make sense?

The FCA's October 2026 publication on AI good practice provides a revealing example.

At one firm, human review increasingly became a box-ticking exercise. Reviewers accepted AI recommendations without sufficient scrutiny, even when there were indications of errors or limitations.

The regulator also describes a contrasting approach: reviewers assessing cases without first seeing the AI's proposed conclusion, allowing independent human and machine judgements to be compared.

That distinction matters.

Human involvement is not the same as independent human judgement.

And someone who has spent years approving machine-generated recommendations may not necessarily develop the same competence as someone who has learned to investigate and reach defensible conclusions.

The risk becomes greater with AI agents

This challenge grows as financial institutions move from AI systems that recommend decisions towards agents capable of taking action.

Consider a collections agent authorised to negotiate repayment arrangements within predefined limits.

A human may supervise exceptions, but that person must understand more than the agent's interface.

They need to recognise when a repayment proposal could create customer harm, when the agent has misunderstood financial circumstances and when an apparently reasonable arrangement falls outside authorised discretion.

Technical guardrails can enforce defined limits. They cannot substitute for all the judgement required to establish whether those limits are appropriate in the first place.

Working on BuildMoat, which is developing controls around delegated authority for AI agents, I keep returning to this distinction.

We can define the boundaries within which an agent is permitted to act. But who has the expertise to define those boundaries correctly, challenge their assumptions and recognise when they need to change?

That capability cannot simply be assumed because an organisation has assigned an accountable owner.

We need to rethink how expertise develops

Sesame Street Gym GIF by Muppet Wiki

Gif by muppetwiki on Giphy

The answer is not to preserve every junior task or resist automation.

Much of that work is repetitive, and AI can make professional development faster and more effective when used deliberately.

But organisations should distinguish between work that merely consumes time and work that develops essential judgement.

A junior analyst might use AI to prepare information while still being required to investigate exceptions and independently assess selected cases.

Compliance teams could use simulated scenarios containing deliberately flawed recommendations, testing whether reviewers recognise the problem before seeing the AI's conclusion.

Experienced professionals could spend less time checking routine outputs and more time coaching colleagues through difficult decisions.

And firms should periodically test whether those entrusted with oversight can actually challenge an AI system, not simply whether they have completed training.

The FCA's findings make this particularly relevant for second- and third-line functions, where generic awareness training may leave staff unable to identify emerging AI risks.

For boards and senior managers, three questions deserve attention:

  • Are we preserving opportunities for people to develop independent professional judgement as we automate their work?

  • Can we demonstrate that reviewers have the competence and authority to challenge AI-generated decisions?

  • Are we measuring the effectiveness of human oversight, or merely recording that an approval occurred?

These are not questions for the technology function alone. They belong in workforce planning, operational risk and AI governance.

Tomorrow's accountability depends on today's learning

Studying College Life GIF

Giphy

The UK's debate about AI adoption rightly considers productivity, investment and economic growth. Its education and employment policies also matter.

But financial institutions cannot wait for national education reform to solve a problem developing within their own operations.

Every decision to automate a task should include a second question: what knowledge or experience did people previously acquire by doing that work, and how will we develop it now?

We may be building AI systems that can perform increasingly sophisticated financial tasks. We must also ensure that the people responsible for those systems retain the capacity to question them.

The greatest long-term risk may not be that AI makes a decision no human can understand. It may be that we gradually stop developing humans who know enough to challenge it.

Accountability requires more than someone willing to press approve.

It requires someone who knows when to say no.

Disclosure: I am the founder of BuildMoat, which is developing technology to enforce delegated authority and operating limits for AI agents.

Get the insights that matter to AI, risk and compliance leaders, delivered every Monday.

Subscribe to The Accountability Record:
https://theaccountabilityrecord.com