The first duty of a government, countless leaders over the decades have said, is to safeguard its population. On 11 September 2001 the US manifestly failed to do that. The intelligence was there, yet tragically, nobody joined up the dots in time.

9/11, as the attacks on the US became known, came to define – along with the response – a generation of foreign, security and intelligence policy for the first two decades of this century.

On 11 September 2001, as al-Qaeda's suicide pilots flew those hijacked planes into the World Trade Center, I was just finishing a deployment in our Middle East bureau. As a Palestinian taxi driver drove me through the pine-clad hills west of Jerusalem to Tel Aviv's Ben Gurion airport he suddenly turned up the car's radio. "Something big has happened in America," he said. He wasn't exaggerating. 2,977 victims died that day.

The attacks did not come out of nowhere. The CIA had been aware of the al-Qaeda network and its exiled Saudi leader, Osama Bin Laden, for years prior to 2001. It even had a taskforce dedicated to hunting him down.

Nine days after the 11 September attacks, then US President George W Bush declared the "War on Terror".

In the ensuing 25 years there have been numerous intelligence successes as well as failures on both sides of the Atlantic. The successes include plots that have been foiled, suspects who have been arrested, brought to trial, convicted and jailed.

But what of the bigger picture? How has intelligence-gathering, counter-terrorism and strategic planning changed in the 25 years since the worst terrorist attack in US history?

No one can question that 9/11 was a colossal intelligence failure, or to be more specific, a failure of imagination in the US intelligence community. At the time both the CIA and the FBI were in possession of information which, if shared properly, could have prevented what happened.

The 9/11 Commission Report that followed blamed "failures of imagination, policy, capabilities and management" on the US government. The CIA and FBI, the report concluded, failed to properly assess intelligence and share information with each other and take sufficient steps to disrupt the plot.

This most basic of lessons has since been largely – although not completely – learnt, and sharing between Western intelligence agencies is infinitely better than it was in 2001. The old rivalries between say, MI5 and Police Counter Terrorism have dissolved, first with regional 'fusion centres' dotted around the UK and now, most recently, with the establishment of the high-security Counter Terrorism Operations Centre in London. Here MI5 intelligence officers might find themselves sitting literally right next to police officers whose job it is to take the executive action to stop a terrorist plot. Equally, they might find themselves sitting next to a US federal agent from the FBI or a language specialist seconded from GCHQ.

Since 9/11, there has also been a root-and-branch reorganisation of US intelligence with the establishment of a National Counter Terrorism Center just outside Washington DC and a Director of National Intelligence to ensure that information is shared where it needs to be between the 18 separate US intelligence agencies. The relationship between the CIA and its UK counterpart, the Secret Intelligence Service (better known as MI6) is perhaps the closest one in the world. This is supplemented by the Five Eyes alliance that pools intelligence between the US, UK, Canada, Australia and New Zealand.

Yet disastrous failures of intelligence, assessment and strategic planning have still occurred since the planes hit the Twin Towers. In the run-up to the US-led invasion of Iraq in 2003, MI6 unwisely allowed its intelligence to be politicised, leading to the organisation suffering its worst reputational setback since the days of betrayal by the double agent Kim Philby and other Soviet moles in the 1950s and 60s.

Against a backdrop of overwhelming political pressure from Washington to produce evidence that Iraq, under its President Saddam Hussein, still possessed 'Weapons of Mass Destruction' (WMDs), MI6 supplied Prime Minister Tony Blair's government with what turned out to be flawed intelligence. Iraq did indeed have WMDs in the past, but it was all destroyed after the 1991 Gulf War.

Following the Iraq invasion of 2003, MI6 then underwent a complete revamp of how it assesses the raw intelligence, known as CX, that comes in from agents in the field. Today there is a clear separation between the agent runners – whose job it is to gather that intelligence from sources in places like Iran, North Korea or from inside proscribed terrorist groups such as al-Qaeda – and the reports team, whose job is to check and challenge that intelligence, often sending it back to demand more substantiation.

"The [2003] Iraq war was based on deeply flawed intelligence," says General Sir Richard Shirreff who served in Iraq as a divisional commander. "Have we learnt from it? No, I think we're bad at learning… It was so obvious as a Commander that we were heading for strategic failure".

After the Iraq invasion an inquiry followed once again, culminating in the Butler Review. This concluded that the UK's intelligence on Iraq's alleged WMD programme "was seriously flawed and heavily reliant on unverified sources".

The Security Service, MI5, has also had its own failures, amid numerous undoubted successes, in foiling terrorist plots. "After 9/11," says Lord Jonathan Evans, who ran MI5 from 2007-2013, "the assessment was that by far the most pressing security threat to the UK was the terrorist threat from al-Qaeda and associated individuals and groups".

But with a database back then of more than 20,000 "Subjects of Interest" and a high number of investigations running concurrently, MI5 faced – and still does – a daily challenge of which leads to prioritise. It has not always got it right.

In 2005 it failed to prevent the 7/7 London suicide bombings that killed 52 people. The Security Service was aware of Mohammed Sidique Khan, the man who turned out to be the ringleader, but it was unaware of what he was about to carry out. The Manchester Arena bombing of 2017 was another more recent lapse.

Today the threat from terrorism has not gone away but, says Lord Evans, "the picture has changed. Non-state terrorism remains a serious concern but state threats… have become increasingly serious and are at least as severe a national security concern as non-state terrorism. These threats come, in different forms, from Russia, Iran, their proxies and from China."

It could also be argued that US President Donald Trump's decision to join Israel in attacking Iran this year was a failure of intelligence imagination. All America's allies in the Gulf knew there was a serious risk Iran would retaliate by closing off the Strait of Hormuz. The warnings were all there and yet Trump appeared to lack a counter-strategy when Iran did exactly what it had threatened to do, causing huge disruption in global energy markets.

In the immediate aftermath of the 9/11 attacks there was a widespread fear that a second wave was coming. In fact al-Qaeda itself promised "a swarm of planes" that was going to attack more buildings. In the rush to pre-empt such an attack numerous human rights abuses were committed by the US and its allies. Suspects were swept up in Afghanistan or Pakistan, often on the flimsiest of tip-offs. Hundreds were flown, bound, blindfolded and wrapped in nappies, halfway across the world to a makeshift cage prison on the US naval base in Guantanamo Bay, Cuba, where they were incarcerated without trial.

Human rights groups denounced this as a stain on the moral conscience of America.

Scandals over the abuse of prisoners in US custody in Iraq and Afghanistan have duly come to light and some of the individuals involved held to account, but critics say more senior figures escaped consequences. Nevertheless, the US still practices detention without trial at Guantanamo Bay, and when it deported more than 200 men from the US to El Salvador's controversial Terrorism Confinement Centre in 2025.

The immediate period after 9/11 had its stains on human rights for Britain too. In 2004, MI6 was complicit in the abduction and rendition of a Libyan Islamist, Abdelhakim Belhaj, and his wife from Thailand to Libya. There he spent years in prison and was tortured by Colonel Muammar Gaddafi's regime. The case eventually came to light and it resulted in a fulsome public apology delivered to Parliament by the Attorney General in 2018. Today, in-house lawyers are said to be the fastest growing department within the agencies.

"We spent the first 20 years of this century focussing almost entirely on counter-terrorism and counter-insurgency," said Sir Alex Younger, who was Chief of the Secret Intelligence Service from 2014-2020. The late MI6 Chief was speaking to me at a literary festival shortly after leaving the top job, adding: "We missed the rise of China as a military power and as a strategic challenge".

Similarly, General Shirreff wrote the political thriller "War With Russia" in 2016, in which he foresaw Russia's full-scale invasion of Ukraine.

In the aftermath of 9/11 and the ensuing "War on Terror" – while Western nations were busy fighting the Taliban, ISIS or trying to sort out the messy occupation of Iraq – Russia and China embarked on colossal rearmament programmes with innovations in hypersonic missiles, for example, that have seriously eroded the military advantage once enjoyed by Nato.

"The West's mistake was not simply that it was distracted by counter-terrorism after 9/11," concurs Sam Olsen, Chief Analyst at the security intelligence firm Sibylline. "It also misunderstood the nature of China's rise, assuming that economic integration would lead to political convergence."

China, just like Iran, Russia and North Korea, is considered a "hard target" for intelligence gathering. Back in 2001, biometric data was still in its relative infancy but the subsequent use of facial, iris and gait recognition have all contributed to the considerable modern-day challenge of getting agents and agent runners across borders undetected.

So how well – or not – does Western intelligence understand China?

Along with Russia, China today is a top intelligence target for MI6. "Western intelligence understands a great deal about China," says Sam Olsen of Sibylline. "The bigger danger is that it sees the individual pieces without always understanding what they add up to. Chinese power is no longer just about ships, missiles and troops. It increasingly lies in semiconductors, critical minerals, batteries, ports, telecommunications and industrial capacity… So the problem is often less a failure of intelligence collection than a failure of strategic interpretation".

Twenty-five years ago, on the day of the 9/11 attacks, US intelligence agents did have pieces of the jigsaw – an analogy that spies love to refer to – but collectively, they failed to fit it all together and see clearly what was coming down the tracks. It is no good having great intelligence gathering if you can't apply it for strategic gain.

Clearly, the world of intelligence and counter-terrorism has changed almost beyond recognition in the quarter century since that cloudless blue September morning in New York. With the advent of AI and quantum computing it is only going to go on changing, and in some areas, at exponential speed.

However, in a world full of conflicting narratives and interests, Western nations do not have a monopoly when it comes to failures of intelligence, oversights or strategic miscalculations. Other noteworthy instances include Russia's disastrous full-scale invasion of Ukraine in 2022 and what some have described as Israel's "own 9/11" – the Hamas-led raid from Gaza on 7 October 2023.

Ultimately, it is in the big, strategic, national decisions that questions are still being asked. Was it right for Britain to join President Bush's invasion of Iraq? Was it right to join the US-led Operation Enduring Freedom in Afghanistan and then abandon the country to the Taliban in a dash for the exit two decades later? These, and many other questions, continue to reverberate.

Lead image: Getty

BBC InDepth is the home on the website and app for the best analysis, with fresh perspectives that challenge assumptions and deep reporting on the biggest issues of the day. Emma Barnett and John Simpson bring their pick of the most thought-provoking deep reads and analysis, every Saturday. Sign up for the newsletter here