I’ve been thinking about the lead story in episode 511 of the For Immediate Release podcast.
It's about one of the world’s most respected law firms, Sullivan & Cromwell, and what happened with a legal filing they submitted to court in a major lawsuit. The filing included fabricated case citations and misquoted authorities – the kind of errors that should never make it anywhere near a courtroom. The firm apologised and corrected the filing, bringing the immediate issue to a close.
AI played a role in preparing the filing. The law firm has policies on AI use, governance, training, and guardrails. And still, a document went to court filled with AI-generated errors.
The more I’ve reflected on it, though, the less interesting the error itself becomes. What stays with me is something else entirely.
Why this isn’t really an AI story
At first glance, it’s easy to see this as another example of AI getting things wrong. We’ve all seen the stories by now – hallucinated facts, confident but incorrect outputs, systems that sound authoritative but aren’t always reliable. It’s a familiar narrative.
But that explanation, in this case, is simply wrong.
This wasn’t an organisation experimenting at the edges. The firm has policies governing the use of AI, as I mentioned earlier. It has mandatory training. It has clear guidance warning about hallucinations and instructing lawyers to verify outputs before using them in client work.
In other words, the risks were understood and the safeguards were in place. And yet, something still slipped through. That’s the part I keep coming back to.
The human in the loop – in theory and in practice
We talk a lot about keeping a “human in the loop” when using AI. It has become a kind of shorthand for responsible use – a way of reassuring ourselves that, ultimately, someone is there to check, to question, to take responsibility.
Somewhere along the line, a document that should have been scrutinised at multiple levels was not properly checked. The errors were not subtle; they were fundamental. And they were only caught when someone outside the organisation pointed them out (embarrassingly, it was the opposing law firm in the lawsuit).
The gap between intention and behaviour
It’s a simple observation, but an uncomfortable one: the issue here isn’t that AI produced flawed output. That’s expected (and it's one clear reason why you must verify AI-produced output). The issue is that no one caught it.
The more I think about it, the less this feels like an isolated mistake and more like something many organisations would recognise. Policies are written, training is delivered, and guidelines are shared. Then things fragment.
Deadlines compress and attention shifts. People rely on what looks credible. Assumptions creep in – someone else will have checked, this looks right, it’s probably fine. None of this requires bad intent; it’s simply how work happens.
Which is why the presence of governance doesn’t necessarily translate into the presence of discipline. Policy reflects intention. Behaviour reveals reality. And the space between the two is where risk quietly builds.
There’s also something else at play here, I think – a kind of illusion of control. It’s easy to believe that once the right policies are written, the right training delivered, and the right guardrails defined, the problem is largely solved.
This case suggests otherwise.
That moment of judgement can’t be automated, and it can’t be assumed. It has to be owned by someone.
One of the ideas Shel raised when we discussed this on the podcast was the notion of a dedicated “verifier” – someone whose explicit responsibility is to check AI-assisted output before it goes any further.
This is not a vague expectation that “someone will review it,” but as a clearly defined role in the workflow.
That may sound like an additional layer of process, and in some organisations it will be. But it also addresses the very problem this case exposes: the assumption that verification is happening when, in reality, it isn’t.
Making that responsibility explicit changes the dynamic. It removes ambiguity. It creates accountability at the point where it matters most.
Whether organisations formalise that as a role or embed it more loosely into their processes, the underlying principle is the same: verification cannot be optional, and it cannot be assumed.
From policy to practice – where communicators matter
For communicators, this feels particularly relevant. We’re close to the work of shaping messages, setting expectations, and reinforcing behaviours. And yet, it’s easy for guidance to remain just that – guidance – rather than something that is actively lived in the organisation.
If responsible AI use is the goal, then the message probably needs to be more direct, and more practical. Something closer to a habit than a principle.
“If AI touched it, you must verify it” is a simple line, but it captures the point. Not as a slogan, but as something that becomes part of how work is done.
And the more I think about it, the more that feels like the harder problem to solve. Not because the answer is unclear, but because it depends on how people behave when the pressure is on, the deadline is close, and the output looks good enough.
There's a common view that 80% is good enough. Not here, not when you've got AI involved. In this case, it's 100% that's good enough.
That’s where the real work is. And where the real risk remains.
Listen to For Immediate Release podcast episode 511, published on 27 April. Shel and I explore what really went wrong, and what it means for organisations trying to operationalise AI responsibly. The discussion starts at about the 15m13s mark.
References:
- Elite law firm Sullivan & Cromwell admits to AI ‘hallucinations’ (Financial Times, 21 April 2026)
- Letter re: In re Prince Global Holdings Limited, et al., No. 26-10769 (Sullivan & Cromwell apology to the court, 18 April 2026)
- FIR 511: Doing AI Governance Right and Still Getting It Wrong (For Immediate Release, 27 April 2026)
- If AI Touched It, You Must Verify It (Studio6, 11 December 2025)
Photo at top by Getty Images for Unsplash+.
