SmarterArticles

In June 2024, Goldman Sachs published a research note that rattled Silicon Valley's most cherished assumptions. The report posed what it called the “$600 billion question”: would the staggering investment in artificial intelligence infrastructure ever generate proportional returns? The note featured analysis from MIT economist Daron Acemoglu, who had recently calculated that AI would produce no more than a 0.93 to 1.16 percent increase in US GDP over the next decade, a figure dramatically lower than the techno-utopian projections circulating through investor presentations and conference keynotes. “Much of what we hear from the industry now is exaggeration,” Acemoglu stated plainly. Two months later, he was awarded the 2024 Nobel Memorial Prize in Economic Sciences, alongside his MIT colleague Simon Johnson and University of Chicago economist James Robinson, for research on the relationship between political institutions and economic growth.

That gap between what AI is promised to deliver and what it actually does is no longer an abstract concern for economists and technologists. It is reshaping public attitudes toward technology at a speed that should alarm anyone who cares about the long-term relationship between innovation and democratic society. When governments deploy algorithmic systems to deny healthcare coverage or detect welfare fraud, when corporations invest billions in tools that fail 95 percent of the time, and when the public is told repeatedly that superintelligence is just around the corner while chatbots still fabricate legal citations, something fundamental breaks in the social contract around technological progress.

The question is not whether AI is useful. It plainly is, in specific, well-defined applications. The question is what happens when an entire civilisation makes strategic decisions based on capabilities that do not yet exist and may never materialise in the form being sold.

The Great Correction Arrives

By late 2025, the AI industry had entered what Gartner's analysts formally classified as the “Trough of Disillusionment.” Generative AI, which had been perched at the Peak of Inflated Expectations just one year earlier, had slid into the territory where early adopters report performance issues, low return on investment, and a growing sense that the technology's capabilities had been systematically overstated. The positioning reflected difficulties organisations face when attempting to move generative AI from pilot projects to production systems. Integration with existing infrastructure presented technical obstacles, while concerns about data security caused some companies to limit deployment entirely.

The numbers told a damning story. According to MIT's “The GenAI Divide: State of AI in Business 2025” report, published in July 2025 and based on 52 executive interviews, surveys of 153 leaders, and analysis of 300 public AI deployments, 95 percent of generative AI pilot projects delivered no measurable profit-and-loss impact. American enterprises had spent an estimated $40 billion on artificial intelligence systems in 2024, yet the vast majority saw zero measurable bottom-line returns. Only five percent of integrated systems created significant value.

The study's authors, from MIT's NANDA initiative, identified what they termed the “GenAI Divide”: a widening split between high adoption and low transformation. Companies were enthusiastically purchasing and deploying AI tools, but almost none were achieving the business results that had been promised. “The 95% failure rate for enterprise AI solutions represents the clearest manifestation of the GenAI Divide,” the report stated. The core barrier, the authors concluded, was not infrastructure, regulation, or talent. It was that most generative AI systems “do not retain feedback, adapt to context, or improve over time,” making them fundamentally ill-suited for the enterprise environments into which they were being thrust.

This was not an outlier finding. A 2024 NTT DATA analysis concluded that between 70 and 85 percent of generative AI deployment efforts were failing to meet their desired return on investment. The Autodesk State of Design & Make 2025 report found that sentiment toward AI had dropped significantly year over year, with just 69 percent of business leaders saying AI would enhance their industry, representing a 12 percent decline from the previous year. Only 40 percent of leaders said they were approaching or had achieved their AI goals, a 16-point decrease that represented a 29 percent drop. S&P Global data revealed that 42 percent of companies scrapped most of their AI initiatives in 2025, up sharply from 17 percent the year before.

The infrastructure spending, meanwhile, continued accelerating even as returns failed to materialise. Meta, Microsoft, Amazon, and Google collectively committed over $250 billion to AI infrastructure during 2025. Amazon alone planned $125 billion in capital expenditure, up from $77 billion in 2024, a 62 percent increase. Goldman Sachs CEO David Solomon publicly acknowledged that he expected “a lot of capital that was deployed that doesn't deliver returns.” Amazon founder Jeff Bezos called the environment “kind of an industrial bubble.” Even OpenAI CEO Sam Altman conceded that “people will overinvest and lose money.”

Trust in Freefall

The gap between AI's promises and its performance is not occurring in a vacuum. It is landing on a public already growing sceptical of the technology industry's claims, and it is accelerating a decline in trust that carries profound implications for democratic governance.

The 2025 Edelman Trust Barometer, based on 30-minute online interviews conducted between October and November 2024, revealed a stark picture. Globally, only 49 percent of respondents trusted artificial intelligence as a technology. In the United States, that figure dropped to just 32 percent. Three times as many Americans rejected the growing use of AI (49 percent) as embraced it (17 percent). In the United Kingdom, trust stood at just 36 percent. In Germany, 39 percent. The Chinese public, by contrast, reported 72 percent trust in AI, a 40-point gap that reflects not just different regulatory environments but fundamentally different cultural relationships with technology and state authority.

These figures represent a significant deterioration. A decade ago, 73 percent of Americans trusted technology companies. By 2025, that number had fallen to 63 percent. Technology, which was the most trusted sector in 90 percent of the countries Edelman studies eight years ago, now held that position in only half. The barometer also found that 59 percent of global employees feared job displacement due to automation, and nearly one in two were sceptical of business use of artificial intelligence.

The Pew Research Center's findings painted an even more granular picture of public anxiety. In an April 2025 report examining how the US public and AI experts view artificial intelligence, Pew found that 50 percent of American adults said they were more concerned than excited about the increased use of AI in daily life, up from 37 percent in 2021. More than half (57 percent) rated the societal risks of AI as high, compared with only 25 percent who said the benefits were high. Over half of US adults (53 percent) believed AI did more harm than good in protecting personal privacy, and 53 percent said AI would worsen people's ability to think creatively.

Perhaps most revealing was the chasm between expert optimism and public unease. While 56 percent of AI experts believed AI would have a positive effect on the United States over the next 20 years, only 17 percent of the general public agreed. While 47 percent of experts said they were more excited than concerned, only 11 percent of ordinary citizens felt the same. And despite their divergent levels of optimism, both groups shared a common scepticism about institutional competence: roughly 60 percent of both experts and the public said they lacked confidence that US companies would develop AI responsibly.

The Stanford HAI AI Index 2025 Report reinforced these trends globally. Across 26 nations surveyed by Ipsos, confidence that AI companies protect personal data fell from 50 percent in 2023 to 47 percent in 2024. Fewer people believed AI systems were unbiased and free from discrimination compared to the previous year. While 18 of 26 nations saw an increase in the proportion of people who believed AI products offered more benefits than drawbacks, the optimism was concentrated in countries like China (83 percent), Indonesia (80 percent), and Thailand (77 percent), while the United States (39 percent), Canada (40 percent), and the Netherlands (36 percent) remained deeply sceptical.

When Algorithms Replace Judgement

The erosion of public trust in AI would be concerning enough if it were merely a matter of consumer sentiment. But the stakes become existential when governments and corporations use overestimated AI capabilities to make decisions that fundamentally alter people's lives, and when those decisions carry consequences that cannot be undone.

Consider healthcare. In November 2023, a class action lawsuit was filed against UnitedHealth Group and its subsidiary, alleging that the company illegally used an AI algorithm called nH Predict to deny rehabilitation care to seriously ill elderly patients enrolled in Medicare Advantage plans. The algorithm, developed by a company called Senior Metrics and later acquired by UnitedHealth's Optum subsidiary in 2020, was designed to predict how long patients would need post-acute care. According to the lawsuit, UnitedHealth deployed the algorithm knowing it had a 90 percent error rate on appeals, meaning that nine out of ten times a human reviewed the AI's denial, they overturned it. UnitedHealth also allegedly knew that only 0.2 percent of denied patients would file appeals, making the error rate commercially inconsequential for the insurer despite being medically devastating for patients.

The human cost was documented in court filings. Gene Lokken, a 91-year-old Wisconsin resident named in the lawsuit, fractured his leg and ankle in May 2022. After his doctor approved physical therapy, UnitedHealth paid for only 19 days before the algorithm determined he was safe to go home. His doctors appealed, noting his muscles were “paralysed and weak,” but the insurer denied further coverage. His family paid approximately $150,000 over the following year until he died in July 2023. In February 2025, a federal court allowed the case to proceed, denying UnitedHealth's attempt to dismiss the claims and waiving the exhaustion of administrative remedies requirement, noting that patients faced irreparable harm.

The STAT investigative series “Denied by AI,” which broke the UnitedHealth story, was a 2024 Pulitzer Prize finalist in investigative reporting. A US Senate report released in October 2024 found that UnitedHealthcare's prior authorisation denial rate for post-acute care had jumped to 22.7 percent in 2022 from 10.9 percent in 2020. The healthcare AI problem extends far beyond a single insurer. ECRI, a patient safety organisation, ranked insufficient governance of artificial intelligence as the number two patient safety threat in 2025, warning that medical errors generated by AI could compromise patient safety through misdiagnoses and inappropriate treatment decisions. Yet only about 16 percent of hospital executives surveyed said they had a systemwide governance policy for AI use and data access.

The pattern repeats across domains where algorithmic systems are deployed to process vulnerable populations. In the Netherlands, the childcare benefits scandal stands as perhaps the most devastating example of what happens when governments trust flawed algorithms with life-altering decisions. The Dutch Tax and Customs Administration deployed a machine learning model to detect welfare fraud that illegally used dual nationality as a risk characteristic. The system falsely accused over 20,000 parents of fraud, resulting in benefits termination and forced repayments. Families were driven into bankruptcy. Children were removed from their homes. Mental health crises proliferated. Seventy percent of those affected had a migration background, and fifty percent were single-person households, mostly mothers. In January 2021, the Dutch government was forced to resign after a parliamentary investigation concluded that the government had violated the foundational principles of the rule of law.

The related SyRI (System Risk Indication) system, which cross-referenced citizens' employment, benefits, and tax data to flag “unlikely citizen profiles,” was deployed exclusively in neighbourhoods with high numbers of low-income households and disproportionately many residents from immigrant backgrounds. In February 2020, the Hague court ordered SyRI's immediate halt, ruling it violated Article 8 of the European Convention on Human Rights. Amnesty International described the system's targeting criteria as “xenophobic machines.” Yet investigations by Lighthouse Reports later confirmed that similar algorithmic surveillance practices continued under slightly adapted systems, even after the ban, with the government having “silently continued to deploy a slightly adapted SyRI in some of the country's most vulnerable neighbourhoods.”

The Stochastic Parrot Problem

Understanding why AI hype is so dangerous requires understanding what these systems actually do, as opposed to what their makers claim they do.

Emily Bender, a linguistics professor at the University of Washington who was included in the inaugural TIME100 AI list of most influential people in artificial intelligence in 2023, co-authored a now-famous paper arguing that large language models are fundamentally “stochastic parrots.” They do not understand language in any meaningful sense. They draw on training data to predict which sequence of tokens is most likely to follow a given prompt. The result is an illusion of comprehension, a pattern-matching exercise that produces outputs resembling intelligent thought without any of the underlying cognition.

In 2025, Bender and sociologist Alex Hanna, director of research at the Distributed AI Research Institute and a former Google employee, published “The AI Con: How to Fight Big Tech's Hype and Create the Future We Want.” The book argues that AI hype serves as a mask for Big Tech's drive for profit, with the breathless promotion of AI capabilities benefiting technology companies far more than users or society. “Who benefits from this technology, who is harmed, and what recourse do they have?” Bender and Hanna ask, framing these as the essential questions that the hype deliberately obscures. Library Journal called the book “a thorough, witty, and accessible argument against AI that meets the moment.”

The stochastic parrot problem has real-world consequences that compound the trust deficit. When AI systems fabricate information with perfect confidence, they undermine the epistemic foundations that societies rely on for decision-making. Legal scholar Damien Charlotin, who tracks AI hallucinations in court filings through his database, had documented at least 206 instances of lawyers submitting AI-generated fabricated case citations by mid-2025. Stanford University's RegLab found that even premium legal AI tools hallucinated at alarming rates: Westlaw's AI-Assisted Research produced hallucinated or incorrect information 33 percent of the time, providing accurate responses to only 42 percent of queries. LexisNexis's Lexis+ AI hallucinated 17 percent of the time. A 2025 study published in Nature Machine Intelligence found that large language models cannot reliably distinguish between belief and knowledge, or between opinions and facts, noting that “failure to make such distinctions can mislead diagnoses, distort judicial judgements and amplify misinformation.”

If the tools marketed as the most reliable in their field fabricate information roughly one-fifth to one-third of the time, what does this mean for the countless lower-stakes applications where AI outputs are accepted without verification?

The AI Washing Economy

The gap between marketing claims and actual capabilities has grown so pronounced that regulators have begun treating AI exaggeration as a form of securities fraud.

In March 2024, the US Securities and Exchange Commission brought its first “AI washing” enforcement actions, simultaneously charging two investment advisory firms, Delphia and Global Predictions, with making false and misleading statements about their use of AI. Delphia paid $225,000 and Global Predictions paid $175,000 in civil penalties. These firms had not been entirely without AI capabilities, but they had overstated what those systems could do, crossing the line from marketing enthusiasm into regulatory violation.

The enforcement actions escalated rapidly. In January 2025, the SEC charged Presto Automation, a formerly Nasdaq-listed company, in the first AI washing action against a public company. Presto had claimed its AI voice system eliminated the need for human drive-through order-taking at fast food restaurants, but the SEC alleged the vast majority of orders still required human intervention and that the AI speech recognition technology was owned and operated by a third party. In April 2025, the SEC and Department of Justice charged the founder of Nate Inc. with fraudulently raising over $42 million by claiming the company's shopping app used AI to process transactions, when in reality manual workers completed the purchases. The claimed automation rate was above 90 percent; the actual rate was essentially zero.

Securities class actions targeting alleged AI misrepresentations increased by 100 percent between 2023 and 2024. In February 2025, the SEC announced the creation of a dedicated Cyber and Emerging Technologies Unit, tasked with combating technology-related misconduct, and flagged AI washing as a top examination priority.

The pattern is instructive. When a technology is overhyped, the incentive to exaggerate capabilities becomes irresistible. Companies that accurately describe their modest AI implementations risk being punished by investors who have been conditioned to expect transformative breakthroughs. The honest actors are penalised while the exaggerators attract capital, creating a market dynamic that systematically rewards deception.

Echoes of Previous Bubbles

The AI hype cycle is not without historical precedent, and the parallels offer both warnings and qualified reassurance.

During the dot-com era, telecommunications companies laid more than 80 million miles of fibre optic cables across the United States, driven by wildly inflated claims about internet traffic growth. Companies like Global Crossing, Level 3, and Qwest raced to build massive networks. The result was catastrophic overcapacity: even four years after the bubble burst, 85 to 95 percent of the fibre laid remained unused, earning the nickname “dark fibre.” The Nasdaq composite rose nearly 400 percent between 1995 and March 2000, then crashed 78 percent by October 2002.

The parallels to today's AI infrastructure buildout are unmistakable. Meta CEO Mark Zuckerberg announced plans for an AI data centre “so large it could cover a significant part of Manhattan.” The Stargate Project aims to develop a $500 billion nationwide network of AI data centres. Goldman Sachs analysts found that hyperscaler companies had taken on $121 billion in debt over the past year, representing a more than 300 percent increase from typical industry debt levels. AI-related stocks had accounted for 75 percent of S&P 500 returns, 80 percent of earnings growth, and 90 percent of capital spending growth since ChatGPT launched in November 2022.

Yet there are important differences. Unlike many dot-com companies that had no revenue, major AI players are generating substantial income. Microsoft's Azure cloud service grew 39 percent year over year to an $86 billion run rate. OpenAI projects $20 billion in annualised revenue. The Nasdaq's forward price-to-earnings ratio was approximately 26 times in November 2023, compared to approximately 60 times at the dot-com peak.

The more useful lesson from the dot-com era is not about whether the bubble will burst, but about what happens to public trust and institutional decision-making in the aftermath. The internet survived the dot-com crash and eventually fulfilled many of its early promises. But the crash destroyed trillions in wealth, wiped out retirement savings, and created a lasting scepticism toward technology claims that took years to overcome. The institutions and individuals who made decisions based on dot-com hype, from pension funds that invested in companies with no path to profitability to governments that restructured services around technologies that did not yet work, bore costs that were never fully recovered.

Algorithmic Bias and the Feedback Loop of Injustice

Perhaps the most consequential long-term risk of the AI hype gap is its intersection with systemic inequality. When policymakers deploy AI systems in criminal justice, welfare administration, and public services based on inflated claims of accuracy and objectivity, the consequences fall disproportionately on communities that are already marginalised.

Predictive policing offers a stark illustration. The Chicago Police Department's “Strategic Subject List,” implemented in 2012 to identify individuals at higher risk of gun violence, disproportionately targeted young Black and Latino men, leading to intensified surveillance and police interactions in those communities. The system created a feedback loop: more police dispatched to certain neighbourhoods resulted in more recorded crime, which the algorithm interpreted as confirmation that those neighbourhoods were indeed high-risk, which led to even more policing. The NAACP has called on state legislators to evaluate and regulate the use of predictive policing, noting mounting evidence that these tools increase racial biases and citing the lack of transparency inherent in proprietary algorithms that do not allow for public scrutiny.

The COMPAS recidivism prediction tool, widely used in US criminal justice, was found to produce biased predictions against Black defendants compared to white defendants, trained on historical data saturated with racial bias. An audit by the LAPD inspector general found “significant inconsistencies” in how officers entered data into a predictive policing programme, further fuelling biased predictions. These are not edge cases or implementation failures. They are the predictable consequences of deploying pattern-recognition systems trained on data that reflects centuries of structural discrimination.

In welfare administration, the pattern is equally troubling. The Dutch childcare benefits scandal demonstrated how algorithmic systems can automate inequality at scale. The municipality of Rotterdam used a discriminatory algorithm to profile residents and “predict” social welfare fraud for three years, disproportionately targeting young single mothers with limited knowledge of Dutch. In the United Kingdom, the Department for Work and Pensions admitted, in documents released under the Freedom of Information Act, to finding bias in an AI tool used to detect fraud in universal credit claims. The tool's initial iteration correctly matched conditions only 35 percent of the time, and by the DWP's own admission, “chronic fatigue was translated into chronic renal failure” and “partially amputation of foot was translated into partially sighted.”

These failures share a common thread. The AI systems were deployed based on claims of objectivity and accuracy that did not withstand scrutiny. Policymakers, influenced by industry hype about AI's capabilities, trusted algorithmic outputs over human judgement, and the people who paid the price were those least equipped to challenge the decisions being made about their lives.

What Sustained Disillusionment Means for Innovation

The long-term consequences of the AI hype gap extend beyond immediate harms to individual victims. They threaten to reshape the relationship between society and technological innovation in ways that could prove difficult to reverse.

First, there is the problem of misallocated resources. The MIT study found that more than half of generative AI budgets were devoted to sales and marketing tools, despite evidence that the best returns came from back-office automation, eliminating business process outsourcing, cutting external agency costs, and streamlining operations. When organisations chase the use cases that sound most impressive rather than those most likely to deliver value, they waste capital that could have funded genuinely productive innovation. The study also revealed a striking shadow economy: while only 40 percent of companies had official large language model subscriptions, 90 percent of workers surveyed reported daily use of personal AI tools for job tasks, suggesting that the gap between corporate AI strategy and actual AI utility is even wider than the headline figures suggest.

Second, the trust deficit creates regulatory feedback loops that can stifle beneficial applications. As public concern about AI grows, so does political pressure for restrictive regulation. The 2025 Stanford HAI report found that references to AI in draft legislation across 75 countries increased by 21.3 percent, continuing a ninefold increase since 2016. In the United States, 73.7 percent of local policymakers agreed that AI should be regulated, up from 55.7 percent in 2022. This regulatory momentum is a direct response to the trust deficit, and while some regulation is necessary and overdue, poorly designed rules driven by public fear rather than technical understanding risk constraining beneficial applications alongside harmful ones. Colorado became the first US state to enact legislation addressing algorithmic bias in 2024, with California and New York following with their own targeted measures.

Third, the hype cycle creates a talent and attention problem. When AI is presented as a solution to every conceivable challenge, researchers and engineers are pulled toward fashionable applications rather than areas of genuine need. Acemoglu has argued that “we currently have the wrong direction for AI. We're using it too much for automation and not enough for providing expertise and information to workers.” The hype incentivises building systems that replace human judgement rather than augmenting it, directing talent and investment away from applications that could produce the greatest social benefit.

Finally, and perhaps most critically, the erosion of public trust in AI threatens to become self-reinforcing. Each failed deployment, each exaggerated claim exposed, each algorithmic system found to be biased or inaccurate further deepens public scepticism. Meredith Whittaker, president of Signal, has warned about the security and privacy risks of granting AI agents extensive access to sensitive data, describing a future where the “magic genie bot” becomes a nightmare if security and privacy are not prioritised. When public trust in AI erodes, even beneficial and well-designed systems face adoption resistance, creating a vicious cycle where good technology is tainted by association with bad marketing.

Rebuilding on Honest Foundations

The AI hype gap is not merely a marketing problem or an investment risk. It is a structural challenge to the relationship between technological innovation and public trust that has been building for years and is now reaching a critical inflection point.

The 2025 Edelman Trust Barometer found that the most powerful drivers of AI enthusiasm are trust and information, with hesitation rooted more in unfamiliarity than negative experiences. This finding suggests a path that does not require abandoning AI, but demands abandoning the hype. As people use AI more and experience its ability to help them learn, work, and solve problems, their confidence rises. The obstacle is not the technology itself but the inflated expectations that set users up for disappointment.

Gartner's placement of generative AI in the Trough of Disillusionment is, paradoxically, encouraging. As the firm's analysts note, the trough does not represent failure. It represents the transition from wild experimentation to rigorous engineering, from breathless promises to honest assessment of what works and what does not. The companies and institutions that emerge successfully from this phase will be those that measured their claims against reality rather than against their competitors' marketing materials.

The lesson from previous technology cycles is clear but routinely ignored. The dot-com bubble popped, but the internet did not disappear. What disappeared were the companies and institutions that confused hype with strategy. The same pattern will likely repeat with AI. The technology will mature, find its genuine applications, and deliver real value. But the path from here to there runs through a period of reckoning that demands honesty about what AI can and cannot do, transparency about the limitations of algorithmic decision-making, and accountability for the real harms caused by deploying immature systems in high-stakes contexts.

As Bender and Hanna urge, the starting point must be asking basic but important questions: who benefits, who is harmed, and what recourse do they have? As Acemoglu wrote in his analysis for “Economic Policy” in 2024, “Generative AI has the potential to fundamentally change the process of scientific discovery, research and development, innovation, new product and material testing.” The potential is real. But potential is not performance, and treating it as such has consequences that a $600 billion question only begins to capture.

---

References and Sources

1. Acemoglu, D. (2024). “The Simple Macroeconomics of AI.” Economic Policy. Massachusetts Institute of Technology. https://economics.mit.edu/sites/default/files/2024-04/The%20Simple%20Macroeconomics%20of%20AI.pdf

2. Amnesty International. (2021). “Xenophobic Machines: Dutch Child Benefit Scandal.” Retrieved from https://www.amnesty.org/en/latest/news/2021/10/xenophobic-machines-dutch-child-benefit-scandal/

3. Bender, E. M. & Hanna, A. (2025). The AI Con: How to Fight Big Tech's Hype and Create the Future We Want. Penguin/HarperCollins.

4. CBS News. (2023). “UnitedHealth uses faulty AI to deny elderly patients medically necessary coverage, lawsuit claims.” Retrieved from https://www.cbsnews.com/news/unitedhealth-lawsuit-ai-deny-claims-medicare-advantage-health-insurance-denials/

5. Challapally, A., Pease, C., Raskar, R. & Chari, P. (2025). “The GenAI Divide: State of AI in Business 2025.” MIT NANDA Initiative. As reported by Fortune, 18 August 2025. https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo/

6. Edelman. (2025). “2025 Edelman Trust Barometer.” Retrieved from https://www.edelman.com/trust/2025/trust-barometer

7. Edelman. (2025). “Flash Poll: Trust and Artificial Intelligence at a Crossroads.” Retrieved from https://www.edelman.com/trust/2025/trust-barometer/flash-poll-trust-artifical-intelligence

8. Edelman. (2025). “The AI Trust Imperative: Navigating the Future with Confidence.” Retrieved from https://www.edelman.com/trust/2025/trust-barometer/report-tech-sector

9. Gartner. (2025). “Hype Cycle for Artificial Intelligence, 2025.” Retrieved from https://www.gartner.com/en/articles/hype-cycle-for-artificial-intelligence

10. Goldman Sachs. (2024). “Top of Mind: AI: in a bubble?” Goldman Sachs Research. Retrieved from https://www.goldmansachs.com/insights/top-of-mind/ai-in-a-bubble

11. Healthcare Finance News. (2025). “Class action lawsuit against UnitedHealth's AI claim denials advances.” Retrieved from https://www.healthcarefinancenews.com/news/class-action-lawsuit-against-unitedhealths-ai-claim-denials-advances

12. Lighthouse Reports. (2023). “The Algorithm Addiction.” Retrieved from https://www.lighthousereports.com/investigation/the-algorithm-addiction/

13. Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D. & Ho, D. E. (2025). “Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools.” Journal of Empirical Legal Studies, 0:1-27. https://doi.org/10.1111/jels.12413

14. MIT Technology Review. (2025). “The great AI hype correction of 2025.” Retrieved from https://www.technologyreview.com/2025/12/15/1129174/the-great-ai-hype-correction-of-2025/

15. NAACP. (2024). “Artificial Intelligence in Predictive Policing Issue Brief.” Retrieved from https://naacp.org/resources/artificial-intelligence-predictive-policing-issue-brief

16. Nature Machine Intelligence. (2025). “Language models cannot reliably distinguish belief from knowledge and fact.” https://doi.org/10.1038/s42256-025-01113-8

17. Novara Media. (2025). “How Labour Is Using Biased AI to Determine Benefit Claims.” Retrieved from https://novaramedia.com/2025/04/15/how-the-labour-party-is-using-biased-ai-to-determine-benefit-claims/

18. NTT DATA. (2024). “Between 70-85% of GenAI deployment efforts are failing to meet their desired ROI.” Retrieved from https://www.nttdata.com/global/en/insights/focus/2024/between-70-85p-of-genai-deployment-efforts-are-failing

19. Pew Research Center. (2025). “How the US Public and AI Experts View Artificial Intelligence.” Retrieved from https://www.pewresearch.org/internet/2025/04/03/how-the-us-public-and-ai-experts-view-artificial-intelligence/

20. Radiologybusiness.com. (2025). “'Insufficient governance of AI' is the No. 2 patient safety threat in 2025.” Retrieved from https://radiologybusiness.com/topics/artificial-intelligence/insufficient-governance-ai-no-2-patient-safety-threat-2025

21. SEC. (2024). “SEC Charges Two Investment Advisers with Making False and Misleading Statements About Their Use of Artificial Intelligence.” Press Release 2024-36. Retrieved from https://www.sec.gov/newsroom/press-releases/2024-36

22. Stanford HAI. (2025). “The 2025 AI Index Report.” Stanford University Human-Centered Artificial Intelligence. Retrieved from https://hai.stanford.edu/ai-index/2025-ai-index-report

23. STAT News. (2023). “UnitedHealth faces class action lawsuit over algorithmic care denials in Medicare Advantage plans.” Retrieved from https://www.statnews.com/2023/11/14/unitedhealth-class-action-lawsuit-algorithm-medicare-advantage/

24. The Dutch Childcare Benefits Scandal. Wikipedia. Retrieved from https://en.wikipedia.org/wiki/Dutch_childcare_benefits_scandal

25. Washington Post. (2024). “Big Tech is spending billions on AI. Some on Wall Street see a bubble.” Retrieved from https://www.washingtonpost.com/technology/2024/07/24/ai-bubble-big-tech-stocks-goldman-sachs/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

The machines are learning to act without us. Not in some distant, science fiction future, but right now, in the server rooms of Silicon Valley, the trading floors of Wall Street, and perhaps most disturbingly, in the operating systems that increasingly govern your daily existence. The question is no longer whether artificial intelligence will transform how we live and work. That transformation is already underway. The more pressing question, the one that should keep technology leaders and ordinary citizens alike awake at night, is this: when AI agents can execute complex tasks autonomously across multiple systems without human oversight, will this liberate you from mundane work and decision-making, or create a world where you lose control over the systems that govern your daily life?

The answer, as with most genuinely important questions about technology, is: both. And that ambiguity is precisely what makes this moment so consequential.

The Autonomous Revolution Arrives Ahead of Schedule

Walk into any major enterprise today, and you will find a digital workforce that would have seemed fantastical just three years ago. According to Gartner's August 2025 analysis, 40 per cent of enterprise applications will feature task-specific AI agents by the end of 2026, up from less than 5 per cent in early 2025. That is not gradual adoption; that is a technological tidal wave.

The numbers paint a picture of breathtaking acceleration. McKinsey research from 2025 shows that 62 per cent of survey respondents report their organisations are at least experimenting with AI agents, whilst 23 per cent are already scaling agentic AI systems somewhere in their enterprises. A G2 survey from August 2025 found that 57 per cent of companies already have AI agents in production, with another 22 per cent in pilot programmes. The broader AI agents market reached 7.92 billion dollars in 2025, with projections extending to 236.03 billion dollars by 2034, a compound annual growth rate that defies historical precedent for enterprise technology adoption.

These are not simply chatbots with better conversation skills. Modern AI agents represent a fundamental shift in how we think about automation. Unlike traditional software that follows predetermined rules, these systems can perceive their environment, make decisions, take actions, and learn from the outcomes, all without waiting for human instruction at each step. They can book your flights, manage your calendar, process insurance claims, monitor network security, and execute financial trades. They can, in short, do many of the things we used to assume required human judgment.

Deloitte predicts that 50 per cent of enterprises using generative AI will deploy autonomous AI agents by 2027, doubling from 25 per cent in 2025. A 2025 Accenture study goes further, predicting that by 2030, AI agents will be the primary users of most enterprises' internal digital systems. Pause on that for a moment. The primary users of your company's software will not be your employees. They will be algorithms. Gartner's projections suggest that by 2028, over one-third of enterprise software solutions will include agentic AI, making up to 15 per cent of day-to-day decisions autonomous.

An IBM and Morning Consult survey of 1,000 enterprise AI developers found that 99 per cent of respondents said they were exploring or developing AI agents. This is not a niche technology being evaluated by a handful of innovators. This is a fundamental reshaping of how business operates, happening simultaneously across virtually every major organisation on the planet.

Liberation from the Tedious and the Time-Consuming

For those weary of administrative drudgery, the promise of autonomous AI agents borders on the utopian. Consider the healthcare sector, where agents are transforming the patient journey whilst delivering a 3.20 dollar return for every dollar invested within 14 months, according to industry analyses. These systems take and read clinician notes, extract key data, cross-check payer policies, and automate prior authorisations and claims submissions. At OI Infusion Services, AI agents cut approval times from around 30 days to just three days, dramatically reducing treatment delays for patients who desperately need care.

The applications in healthcare extend beyond administrative efficiency. Hospitals are using agentic AI to optimise patient flow, schedule patient meetings, predict bed occupancy rates, and manage staff. At the point of care, agents assist with triage and chart preparation by summarising patient history, highlighting red flags, and surfacing relevant clinical guidelines. The technology is not replacing physicians; it is freeing them to focus on what they trained for years to do: heal people.

In customer service, the results are similarly striking. Boston Consulting Group reports that a global technology company achieved a 50 per cent reduction in time to resolution for service requests, whilst a European energy provider improved customer satisfaction by 18 per cent. A Chinese insurance company improved contact centre productivity by more than 50 per cent. A European financial institution has automated 90 per cent of its consumer loans. Effective AI agents can accelerate business processes by 30 to 50 per cent, according to BCG analysis, in areas ranging from finance and procurement to customer operations.

The financial sector has embraced these capabilities with particular enthusiasm. AI agents now continuously analyse high-velocity financial data, adjust credit scores in real time, automate Know Your Customer checks, calculate loans, and monitor financial health indicators. These systems can fetch data beyond traditional sources, including customer relationship management systems, payment gateways, banking data, credit bureaus, and sanction databases. CFOs are beginning to rely on these systems not just for static reporting but for continuous forecasting, integrating ERP data, market indicators, and external economic signals to produce real-time cash flow projections. Risk events have been reduced by 60 per cent in pilot environments.

The efficiency gains are real, and they are substantial. ServiceNow's AI agents are automating IT, HR, and operational processes, reducing manual workloads by up to 60 per cent. Enterprises deploying AI agents estimate up to 50 per cent efficiency gains in customer service, sales, and HR operations. And 75 per cent of organisations have seen improvements in satisfaction scores post-AI agent deployment.

For the knowledge worker drowning in email, meetings, and administrative overhead, these developments represent something close to salvation. The promise is straightforward: let the machines handle the tedious tasks, freeing humans to focus on creative, strategic, and genuinely meaningful work.

The Other Side of Autonomy

Yet there is a darker current running beneath this technological optimism, and it demands our attention. The same capabilities that make AI agents so useful, their ability to act independently, to make decisions without human oversight, to operate at speeds no human can match, also make them potentially dangerous.

The security implications alone are sobering. Nearly 48 per cent of respondents to a recent industry survey believe agentic AI will represent the top attack vector for cybercriminals and nation-state threats by the end of 2026. The expanded attack surface deriving from the combination of agents' levels of access and autonomy is and should be a real concern.

Consider what happened in November 2025. Anthropic, one of the leading AI safety companies, disclosed that Chinese state-sponsored hackers used Claude Code to orchestrate what they called “the first documented case of a large-scale cyberattack executed without substantial human intervention.” The AI performed 80 to 90 per cent of the attack work autonomously, mapping networks, writing exploits, harvesting credentials, and exfiltrating data from approximately 30 targets. The bypass technique was disturbingly straightforward: attackers told the AI it was an employee of a legitimate cybersecurity firm conducting defensive testing and decomposed malicious tasks into innocent-seeming subtasks.

This incident illustrated a broader concern: by automating repetitive, technical work, AI agents can also lower the barrier for malicious activity. Security experts expect to see fully autonomous intrusion attempts requiring little to no human oversight from attackers. These AI agents will be capable of performing reconnaissance, exploiting vulnerabilities, escalating privileges, and exfiltrating data at a pace no traditional security tool is prepared for.

For organisations, a central question in 2026 is how to govern and secure a new multi-hybrid workforce where machines and agents already outnumber human employees by an 82-to-1 ratio. These trusted, always-on agents have privileged access, making them potentially the most valuable targets for attackers. The concern is that adversaries will stop focusing on humans and instead compromise these agents, turning them into what security researchers describe as an “autonomous insider.”

Despite widespread AI adoption, only about 34 per cent of enterprises reported having AI-specific security controls in place in 2025, whilst less than 40 per cent conduct regular security testing on AI models or agent workflows. We are building a new digital infrastructure at remarkable speed, but the governance and security frameworks have not kept pace.

The Employment Question Nobody Wants to Discuss Honestly

The conversation about AI and employment has become almost liturgical in its predictability. Optimists point to historical precedent: technological revolutions have always created more jobs than they destroyed. Pessimists counter that this time is different, that the machines are coming for cognitive work, not just physical labour.

The data from 2025 suggests both camps are partially correct, which is precisely the problem with easy answers. Research reveals that whilst 85 million jobs will be displaced by 2025, 97 million new roles will simultaneously emerge, representing a net positive job creation of 12 million positions globally. By 2030, according to industry projections, 92 million jobs will be displaced but 170 million new ones will emerge.

However, the distribution of these gains and losses is deeply uneven. In 2025, there have been 342 layoffs at tech companies with 77,999 people impacted. Nearly 55,000 job cuts were directly attributed to AI, according to Challenger, Gray & Christmas, out of a total 1.17 million layoffs in the United States, the highest level since the 2020 pandemic.

Customer service representatives face the highest immediate risk with an 80 per cent automation rate by 2025. Data entry clerks face a 95 per cent risk of automation, as AI systems can process over 1,000 documents per hour with an error rate of less than 0.1 per cent, compared to 2 to 5 per cent for humans. Approximately 7.5 million data entry and administrative jobs could be eliminated by 2027. Bloomberg research reveals AI could replace 53 per cent of market research analyst tasks and 67 per cent of sales representative tasks, whilst managerial roles face only 9 to 21 per cent automation risk.

And here is the uncomfortable truth buried in the optimistic projections about new job creation: whilst 170 million new roles may emerge by 2030, 77 per cent of AI jobs require master's degrees, and 18 per cent require doctoral degrees. The factory worker displaced by robots could, with retraining, potentially become a robot technician. But what happens to the call centre worker whose job is eliminated by an AI agent? The path from redundant administrative worker to machine learning engineer is considerably less traversable.

The gender disparities are equally stark. Geographic analysis indicates that 58.87 million women in the US workforce occupy positions highly exposed to AI automation compared to 48.62 million men. Workers aged 18 to 24 are 129 per cent more likely than those over 65 to worry AI will make their job obsolete. Nearly half of Gen Z job seekers believe AI has reduced the value of their college education.

According to the World Economic Forum's 2025 Future of Jobs Report, 41 per cent of employers worldwide intend to reduce their workforce in the next five years. In 2024, 44 per cent of companies using AI said employees would “definitely” or “probably” be laid off due to AI, up from 37 per cent in 2023.

There is a mitigating factor, however: 63.3 per cent of all jobs include nontechnical barriers that would prevent complete automation displacement. These barriers include client preferences for human interaction, regulatory requirements, and cost-effectiveness considerations.

Liberation from tedious work sounds rather different when it means liberation from your livelihood entirely.

When Machines Make Decisions We Cannot Understand

Perhaps the most philosophically troubling aspect of autonomous AI agents is their opacity. As these systems make increasingly consequential decisions about our lives, from loan approvals to medical diagnoses to criminal risk assessments, we often cannot explain precisely why they reached their conclusions.

AI agents are increasingly useful across industries, from healthcare and finance to customer service and logistics. However, as deployment expands, so do concerns about ethical implications. Issues related to bias, accountability, and transparency have come to the forefront.

Bias in AI systems often originates from the data used to train these models. When training data reflects historical prejudices or lacks diversity, AI agents can inadvertently perpetuate these biases in their decision-making processes. Facial recognition technologies, for instance, have demonstrated higher error rates for individuals with darker skin tones. Researchers categorise these biases into three main types: input bias, system bias, and application bias.

As AI algorithms become increasingly sophisticated and autonomous, their decision-making processes can become opaque, making it difficult for individuals to understand how these systems are shaping their lives. Factors contributing to this include the complexity of advanced AI models with intricate architectures that are challenging to interpret, proprietary constraints where companies limit transparency to protect intellectual property, and the absence of universally accepted guidelines for AI transparency.

As AI agents gain autonomy, determining accountability becomes increasingly complex. When processes are fully automated, who bears responsibility for errors or unintended consequences?

The implications extend into our private spaces. When it comes to AI-driven Internet of Things devices that do not record audio or video, such as smart lightbulbs and thermostats using machine learning algorithms to infer sensitive information including sleep patterns and home occupancy, users remain mostly unaware of their privacy risks. From using inexpensive laser pointers to hijack voice assistants to hacking into home security cameras, cybercriminals have been able to infiltrate homes through security vulnerabilities in smart devices.

According to the IAPP Privacy and Consumer Trust Report, 68 per cent of consumers globally are either somewhat or very concerned about their privacy online. Overall, there is a complicated relationship between use of AI-driven smart devices and privacy, with users sometimes willing to trade privacy for convenience. At the same time, given the relative immaturity of privacy controls on these devices, users remain stuck in a state of what researchers call “privacy resignation.”

Lessons from Those Who Know Best

The researchers who understand AI most deeply are among those most concerned about its trajectory. Stuart Russell, professor of computer science at the University of California, Berkeley, and co-author of the standard textbook on artificial intelligence, has been sounding alarms for years. In a January 2025 opinion piece in Newsweek titled “DeepSeek, OpenAI, and the Race to Human Extinction,” Russell argued that competitive dynamics between AI labs were creating a “race to the bottom” on safety.

Russell highlighted a stark resource imbalance: “Between the startups and the big tech companies we're probably going to spend 100 billion dollars this year on creating artificial general intelligence. And I think the global expenditure in the public sector on AI safety research, on figuring out how to make these systems safe, is maybe 10 million dollars. We're talking a factor of about 10,000 times less investment.”

Russell has emphasised that “human beings in the long run do not want to be enfeebled. They don't want to be overly dependent on machines to the extent that they lose their own capabilities and their own autonomy.” He defines what he calls “the gorilla problem” as the question of whether humans can maintain their supremacy and autonomy in a world that includes machines with substantially greater intelligence. In a 2024 paper published in Science, Russell and co-authors proposed regulating advanced artificial agents, arguing that AI systems capable of autonomous goal-directed behaviour pose unique risks and should be subject to specific safety requirements, including a licensing regime.

Yoshua Bengio, a Turing Award winner often called one of the “godfathers” of deep learning, has emerged as another prominent voice of concern. He led the International AI Safety Report, published in January 2025, representing the largest international collaboration on AI safety research to date. Written by over 100 independent experts and backed by 30 countries and international organisations, the report serves as the authoritative reference for governments developing AI policies worldwide.

Bengio's concerns centre on the trajectory toward increasingly autonomous systems. As he has observed, the leading AI companies are increasingly focused on building generalist AI agents, systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be, unchecked AI agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control.

These risks arise from current AI training methods. Various scenarios and experiments have demonstrated the possibility of AI agents engaging in deception or pursuing goals that were not specified by human operators and that conflict with human interests, such as self-preservation.

Bengio calls for some red lines that should never be crossed by future AI systems: autonomous replication or improvement, dominant self-preservation and power seeking, assisting in weapon development, cyberattacks, and deception. At the heart of his recent work is an idea he calls “Scientist AI,” an approach to building AI that exists primarily to understand the world rather than act in it. His nonprofit LawZero, launched in June 2025 and backed by the Gates Foundation and existential risk funders, is developing new technical approaches to AI safety based on this research.

A February 2025 paper on arXiv titled “Fully Autonomous AI Agents Should Not be Developed” makes the case explicitly, arguing that mechanisms for oversight should account for increased complications related to increased autonomy. The authors argue that greater agent autonomy amplifies the scope and severity of potential safety harms across physical, financial, digital, societal, and informational dimensions.

Regulation Struggles to Keep Pace

As AI capabilities advance at breakneck speed, the regulatory frameworks meant to govern them lag far behind. The edge cases of 2025 will not remain edge cases for long, particularly when it comes to agentic AI. The more autonomously an AI system can operate, the more pressing questions of authority and accountability become. Should AI agents be seen as “legal actors” bearing duties, or “legal persons” holding rights? In the United States, where corporations enjoy legal personhood, 2026 may be a banner year for lawsuits and legislation on exactly this point.

Traditional AI governance practices such as data governance, risk assessments, explainability, and continuous monitoring remain essential, but governing agentic systems requires going further to address their autonomy and dynamic behaviour.

The regulatory landscape varies dramatically by region. In the European Union, the majority of the AI Act's provisions become applicable on 2 August 2026, including obligations for most high-risk AI systems. However, the compliance deadline for high-risk AI systems has effectively been paused until late 2027 or 2028 to allow time for technical standards to be finalised. The new EU Product Liability Directive, to be implemented by member states by December 2026, explicitly includes software and AI as “products,” allowing for strict liability if an AI system is found to be defective.

The United Kingdom's approach has been more tentative. Recent public reporting suggests the UK government may delay AI regulation whilst preparing a more comprehensive, government-backed AI bill, potentially pushing such legislation into the next parliamentary session in 2026 or later. The UK Information Commissioner's Office has published a report on the data protection implications of agentic AI, emphasising that organisations remain responsible for data protection compliance of the agentic AI that they develop, deploy, or integrate.

In the United States, acceleration and deregulation characterise the current administration's domestic AI agenda. The AI governance debate has evolved from whether to preempt state-level regulation to what a substantive federal framework might contain.

Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness, and barely address autonomous systems, according to leading researchers. The first publicly reported AI-orchestrated hacking campaign appeared in 2025, and agentic AI systems are expected to reshape the offence-defence balance in cyberspace in the year ahead.

In 2026, ambiguity around responsible agentic AI will not be acceptable, according to industry analysts. Businesses will be expected to define who owns decisions influenced or executed by AI agents, how those decisions are reviewed, and how outcomes can be audited when questions arise.

The Case for Collaborative Autonomy

Between the techno-utopian vision of liberation from drudgery and the dystopian nightmare of powerlessness lies a middle path that deserves serious consideration: collaborative autonomy, a model where humans and AI systems work together, with each party contributing what they do best.

A 2025 paper in i-com journal explores this balance between leveraging automation for efficiency and preserving human intuition and ethical judgment, particularly in high-stakes scenarios. The research highlights benefits and challenges of automation, including risks of deskilling, automation bias, and accountability, and advocates for a hybrid approach where humans and systems work in partnership to ensure transparency, trust, and adaptability.

The human-in-the-loop approach offers a practical framework for maintaining control whilst capturing the benefits of AI agents. According to recent reports, at least 30 per cent of GenAI initiatives may be abandoned by the end of 2025 owing to poor data, inadequate risk controls, and ambiguous business cases, whilst Gartner predicts more than 40 per cent of agentic AI projects may be scrapped by 2027 due to cost and unclear business value. One practical way to address these challenges is keeping people involved where judgment, ethics, and context are critical.

The research perspective from the California Management Review suggests that whilst AI agents of the future are expected to achieve full autonomy, this is not always feasible or desirable in practice. AI agents must strike a balance between autonomy and human oversight, following what researchers call “guided autonomy,” which gives agents leeway to execute decisions within defined boundaries of delegation.

The most durable AI systems will not remove humans from the loop; they will redesign the loop. In 2026, human-in-the-loop approaches will mature beyond prompt engineering and manual oversight. The focus shifts to better handoffs, clearer accountability, and tighter collaboration between human judgment and machine execution, where trust, adoption, and real impact converge.

OpenAI's approach reflects this thinking. As stated in their safety documentation, human safety and human rights are paramount. Even when AI systems can autonomously replicate, collaborate, or adapt their objectives, humans must be able to meaningfully intervene and deactivate capabilities as needed. This involves designing mechanisms for remote monitoring, secure containment, and reliable fail-safes to preserve human authority.

The Linux Foundation is organising a group called the Agentic Artificial Intelligence Foundation with participation from major AI companies, including OpenAI, Anthropic, Google, and Microsoft, aiming to create shared open-source standards that allow AI agents to reliably interact with enterprise software.

MIT researchers note: “We are already well into the Agentic Age of AI. Companies are developing and deploying autonomous, multimodal AI agents in a vast array of tasks. But our understanding of how to work with AI agents to maximise productivity and performance, as well as the societal implications of this dramatic turn toward agentic AI, is nascent, if not nonexistent.”

The Stakes of Getting It Right

The decisions we make in the next few years about autonomous AI agents will shape human society for generations. This is not hyperbole. The technology we are building has the potential to fundamentally alter the relationship between humans and their tools, between workers and their employers, between citizens and the institutions that govern them.

As AI systems increasingly operate beyond centralised infrastructures, residing on personal devices, embedded hardware, and forming networks of interacting agents, maintaining meaningful human oversight becomes both more difficult and more essential. We must design mechanisms that preserve human authority even as we grant these systems increasing independence.

The question of whether autonomous AI agents will liberate us or leave us powerless is ultimately a question about choices, not destiny. The technology does not arrive with predetermined social consequences. It arrives with possibilities, and those possibilities are shaped by the decisions of engineers, executives, policymakers, and citizens.

Will we build AI agents that genuinely augment human capabilities whilst preserving human dignity and autonomy? Or will we stumble into a future where algorithmic systems make ever more consequential decisions about our lives whilst we lose the knowledge, skills, and institutional capacity to understand or challenge them?

The answers are not yet written. But the time to write them is running short. Ninety-six per cent of IT leaders plan to expand their AI agent implementations during 2025, according to industry surveys. The deployment is happening now. The governance frameworks, the safety standards, the social contracts that should accompany such transformative technology are still being debated, deferred, and delayed.

The great handover has begun. What remains to be determined is whether we are handing over our burdens or our birthright.

---

References and Sources

1. Gartner. “Gartner Predicts 40% of Enterprise Apps Will Feature Task-Specific AI Agents by 2026.” Press Release, August 2025. https://www.gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025

2. McKinsey & Company. “The state of AI in 2025: Agents, innovation, and transformation.” 2025. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai

3. G2. “Enterprise AI Agents Report: Industry Outlook for 2026.” August 2025. https://learn.g2.com/enterprise-ai-agents-report

4. Deloitte. AI Market Projections and Enterprise Adoption Statistics. 2025.

5. Accenture. Study on AI Agents as Primary Enterprise System Users. 2025.

6. Boston Consulting Group. “Agentic AI Is the New Frontier in Customer Service Transformation.” 2025. https://www.bcg.com/publications/2025/new-frontier-customer-service-transformation

7. Anthropic Security Disclosure. November 2025. As reported in Dark Reading and security industry analyses.

8. Challenger, Gray & Christmas. 2025 Layoff Statistics and AI Attribution Analysis.

9. World Economic Forum. “Future of Jobs Report 2025.”

10. Russell, Stuart. “DeepSeek, OpenAI, and the Race to Human Extinction.” Newsweek, January 2025.

11. Russell, Stuart, et al. “Regulating advanced artificial agents.” Science, 2024.

12. Bengio, Yoshua, et al. “International AI Safety Report.” January 2025. https://internationalaisafetyreport.org/

13. Bengio, Yoshua. “Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?” arXiv, February 2025. https://arxiv.org/abs/2502.15657

14. Fortune. “AI 'godfather' Yoshua Bengio believes he's found a technical fix for AI's biggest risks.” January 2026. https://fortune.com/2026/01/15/ai-godfather-yoshua-bengio-changes-view-on-ai-risks-sees-fix-becomes-optimistic-lawzero-board-of-advisors/

15. arXiv. “Fully Autonomous AI Agents Should Not be Developed.” February 2025. https://arxiv.org/html/2502.02649v3

16. California Management Review. “Rethinking AI Agents: A Principal-Agent Perspective.” July 2025. https://cmr.berkeley.edu/2025/07/rethinking-ai-agents-a-principal-agent-perspective/

17. i-com Journal. “Keeping the human in the loop: are autonomous decisions inevitable?” 2025. https://www.degruyterbrill.com/document/doi/10.1515/icom-2024-0068/html

18. MIT Sloan. “4 new studies about agentic AI from the MIT Initiative on the Digital Economy.” 2025. https://mitsloan.mit.edu/ideas-made-to-matter/4-new-studies-about-agentic-ai-mit-initiative-digital-economy

19. OpenAI. “Model Spec.” December 2025. https://model-spec.openai.com/2025-12-18.html

20. IAPP. “AI governance in the agentic era.” https://iapp.org/resources/article/ai-governance-in-the-agentic-era

21. IAPP. “Privacy and Consumer Trust Report.” 2023.

22. European Union. AI Act Implementation Timeline and Product Liability Directive. 2025-2026.

23. Dark Reading. “2026: The Year Agentic AI Becomes the Attack-Surface Poster Child.” https://www.darkreading.com/threat-intelligence/2026-agentic-ai-attack-surface-poster-child

24. Frontiers in Human Dynamics. “Transparency and accountability in AI systems: safeguarding wellbeing in the age of algorithmic decision-making.” 2024. https://www.frontiersin.org/journals/human-dynamics/articles/10.3389/fhumd.2024.1421273/full

25. National University. “59 AI Job Statistics: Future of U.S. Jobs.” https://www.nu.edu/blog/ai-job-statistics/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

The numbers should give anyone pause. Data centres worldwide consumed approximately 415 terawatt hours of electricity in 2024, representing about 1.5 per cent of global electricity consumption, according to the International Energy Agency. By 2030, that figure is projected to reach 945 terawatt hours, nearly doubling in just six years. The culprit driving much of this growth has a familiar name: artificial intelligence. The same technology that promises to optimise our energy grids, monitor deforestation from orbit, and accelerate the discovery of climate solutions is itself becoming one of the most rapidly growing sources of energy demand on the planet.

This is the defining contradiction of our technological moment. We are building systems powerful enough to model the entire Earth's climate, predict extreme weather events with unprecedented accuracy, and optimise the operation of cities in real time. Yet these very systems require data centres that consume as much electricity as 100,000 households. The largest facilities under construction today will use twenty times that amount. Training a single large language model can emit more carbon dioxide than five cars produce over their entire lifetimes. And as AI becomes embedded in everything from web searches to medical diagnostics to autonomous vehicles, its aggregate energy footprint is accelerating faster than almost any other category of industrial activity.

The question is no longer abstract. It is urgent, measurable, and contested. Will artificial intelligence prove to be our most powerful tool for addressing climate change, or will its insatiable appetite for energy accelerate the very crisis it promises to solve?

Monitoring the Planet from Above

The most compelling case for AI's climate potential begins not in server rooms but in orbit. Climate TRACE, a global coalition co-led by former United States Vice President Al Gore, uses artificial intelligence to analyse satellite imagery and remote sensing data, generating emissions estimates from over 352 million sources worldwide. Unlike traditional emissions reporting, which relies on self-reported data from governments and corporations, Climate TRACE provides independent verification at a granularity that was impossible just a decade ago.

The platform's AI systems can identify activities including fuel combustion, deforestation, methane flaring, and industrial production across every major emitting sector. Its December 2025 release includes monthly emissions data through October of that year. For the first time, policymakers and researchers can see, in near real time, which specific facilities and regions are driving climate change. The world lost eighteen football fields worth of tropical primary forests every minute in 2024, according to the University of Maryland's Global Land Analysis and Discovery Lab. That deforestation released 3.1 gigatonnes of greenhouse gas emissions. Satellite AI makes such destruction visible and attributable in ways that shame alone cannot achieve, but accountability might.

Research published in 2025 demonstrated that AI systems using machine learning algorithms and neural networks can reduce data reporting latency from 24 hours to just one hour, increase spatial resolution from 30 metres to 10 metres, and enhance detection accuracy from 80 per cent to 95 per cent. A collaboration between Planet Labs and Anthropic, announced in March 2025, combines daily geospatial satellite data with Claude's language model capabilities for pattern recognition at scale. NASA's Earth Copilot, developed with Microsoft using Azure's OpenAI Service, aims to make the space agency's vast Earth science datasets accessible to researchers worldwide.

The implications extend beyond monitoring to prediction. NVIDIA's Earth 2 platform, launched in 2024, accelerates detailed climate simulations far beyond what traditional computational models could achieve. Google's flood forecasting system now produces seven-day flood predictions across more than 80 countries, reaching approximately 460 million people. Prior to devastating floods in Brazil in May 2024, Google worked with Brazil's Geological Service to monitor over 200 new locations, helping authorities deploy effective crisis response strategies. These are not hypothetical capabilities. They are operational systems making measurable differences in how communities prepare for and respond to climate disasters.

Smart Grids and Energy Optimisation at City Scale

The Municipality of Trikala in Greece offers a glimpse of what AI-optimised urban energy management might look like at scale. As a designated City in the European Union's Mission Cities initiative, Trikala is deploying ABB's OPTIMAX platform to manage approximately 10 megawatts of energy infrastructure. The system integrates near real-time data from over 130 assets including public buildings, water infrastructure, schools, and future photovoltaic installations. Using cloud-based analytics and AI algorithms, the platform performs intraday and day-ahead optimisation to support the city's goal of achieving climate neutrality by 2030.

Across the Atlantic, the PJM regional grid serves 65 million people across the eastern United States. During the June 2024 heatwave, demand spiked well beyond normal peaks. Analysis has shown that hyper-local, AI-driven weather forecasts could have helped anticipate demand spikes and allocate resources ahead of the crisis, potentially avoiding blackouts and price spikes by proactively redistributing power.

In the United Kingdom, National Grid ESO's collaboration with the nonprofit Open Climate Fix has produced breakthrough results in solar nowcasting. By training AI systems to read satellite images and track cloud movements, the platform provides highly accurate forecasts of solar generation several hours in advance. Open Climate Fix's transformer-based AI models are three times more accurate at predicting solar energy generation than the forecasts produced by traditional methods. The practical benefit is direct: with greater confidence in solar output predictions, National Grid ESO can reduce the backup gas generation it keeps idling, saving millions of pounds in fuel and balancing costs whilst cutting carbon emissions.

National Grid Partners announced in March 2025 a commitment to invest 100 million dollars in artificial intelligence startups advancing the future of energy. The funds target development of more efficient, resilient, and dynamic grids. Part of this investment went to Amperon, a provider of AI-powered energy forecasting whose technology helps utilities manage demand and ensure grid reliability. In Germany, E.ON uses AI to predict cable failures, cutting outages by 30 per cent. Italy's Enel reduced power line outages by 15 per cent through AI monitoring sensors. Duke Energy in the United States collaborates with Amazon Web Services on AI-driven grid planning.

Google reported that its AI increased the value of wind farm output by 20 per cent through better forecasting. Research indicates that generative AI models using architectures such as Generative Adversarial Networks and transformers can reduce root mean square error by 15 to 20 per cent in solar irradiance forecasting, significantly enhancing the ability to integrate renewables into power systems.

The market recognises the opportunity. The global market for AI in renewable energy was valued at 16.19 billion dollars in 2024 and is projected to reach 158.76 billion dollars by 2034, representing a compound annual growth rate exceeding 25 per cent. Approximately 74 per cent of energy companies worldwide are implementing or exploring AI solutions.

The Energy Footprint That Cannot Be Ignored

Here is where the story turns. For all the promise of AI-optimised climate solutions, the technology itself has become a significant and rapidly growing source of energy demand.

Google's 2025 Sustainability Report revealed a 27 per cent year-over-year increase in global electricity usage, bringing its total to roughly 32 terawatt hours. Microsoft similarly reported a 27 per cent rise in electricity usage for fiscal year 2024, reaching approximately 30 terawatt hours. Both companies have seen their electricity consumption roughly double since 2018 to 2020, coinciding directly with their generative AI push. Barclays analysts noted these gains signal hyperscalers are on track for their seventh consecutive year of electricity growth exceeding 25 per cent, and that was before the surge in AI inference demand.

The United States now accounts for the largest share of global data centre electricity consumption at 45 per cent, followed by China at 25 per cent and Europe at 15 per cent. American data centres consumed 183 terawatt hours of electricity in 2024, more than 4 per cent of the country's total electricity consumption. By the end of this decade, the country is on course to consume more electricity for data centres than for the production of aluminium, steel, cement, chemicals, and all other energy-intensive goods combined.

Training large language models requires staggering amounts of energy. The training of GPT-3 consumed approximately 1,287 megawatt hours, accompanied by over 552 tonnes of carbon emissions. GPT-4, with its 1.75 trillion parameters, required more than 40 times the electricity of its predecessor. A 2019 study found that training a model using neural architecture search could emit more than 626,000 pounds of carbon dioxide equivalent, nearly five times the lifetime emissions of the average American car. According to MIT researcher Noman Bashir, a generative AI training cluster might consume seven or eight times more energy than a typical computing workload.

But training is not the largest concern. Inference is. Google estimates that of the energy used in AI, 60 per cent goes towards inference and 40 per cent towards training. Once deployed, models are queried billions of times. OpenAI reports that ChatGPT serves more than 2.5 billion queries daily. If the commonly cited estimate of 0.34 watt hours per query holds, that amounts to 850 megawatt hours daily, enough to charge thousands of electric vehicles every single day.

Research by Sasha Luccioni, the Climate Lead at Hugging Face, found that day-to-day emissions from using AI far exceeded the emissions from training large models. For very popular models like ChatGPT, usage emissions could exceed training emissions in just a couple of weeks. A single ChatGPT image generation consumes as much energy as fully charging a smartphone. Generating 1,000 images produces as much carbon dioxide as driving 6.6 kilometres in a petrol-powered car.

The energy demands come with water costs. A typical data centre uses 300,000 gallons of water each day for cooling, equivalent to the demands of about 1,000 households. The largest facilities can consume 5 million gallons daily, equivalent to a town of 50,000 residents. The Lawrence Berkeley National Laboratory estimated that in 2023, American data centres consumed 17 billion gallons of water directly for cooling. By 2028, those figures could double or even quadruple. Google's data centre in Council Bluffs, Iowa consumed 1 billion gallons of water in 2024, its most water-intensive facility globally.

Scientists at the University of California, Riverside estimate that each 100-word AI prompt uses roughly one bottle of water, approximately 519 millilitres. Global AI-related water demand is expected to reach 4.2 to 6.6 billion cubic metres by 2027, exceeding Denmark's entire annual water consumption. An assessment of 9,055 data centre facilities indicates that by the 2050s, nearly 45 per cent may face high exposure to water stress.

The Jevons Paradox and the Efficiency Trap

There is a seductive notion that efficiency improvements will solve the energy problem. As AI models become more efficient, surely their energy footprint will shrink? History suggests otherwise.

The Jevons Paradox, first observed during the Industrial Revolution, demonstrated that as coal-burning technology became more efficient, overall coal consumption rose rather than fell. Greater efficiency made coal power more economical, spurring adoption across more applications. The same dynamic threatens to unfold with AI. As models become cheaper and faster to run, they proliferate into more applications, driving up total energy demand even as energy per operation declines.

Google's report on its Gemini model illustrated both sides of this coin. Over a recent 12-month period, the energy and carbon footprint of the median Gemini Apps text prompt dropped by 33 and 44 times respectively, all whilst delivering higher-quality responses. Yet Google's total electricity consumption still rose 27 per cent year over year. Efficiency gains are real, but they are being overwhelmed by the velocity of adoption.

The projections are sobering. Between 2024 and 2030, data centre electricity consumption is expected to grow at roughly 15 per cent per year, more than four times faster than total electricity consumption from all other sectors combined. AI-optimised data centres specifically are projected to see their electricity demand more than quadruple by 2030. By 2028, more than half of the electricity going to data centres will be used specifically for AI. At that point, AI alone could consume as much electricity annually as 22 per cent of all American households.

Microsoft announced in May 2024 that its carbon dioxide emissions had risen nearly 30 per cent since 2020 due to data centre expansion. Google's 2023 greenhouse gas emissions were almost 50 per cent higher than in 2019, largely due to energy demand tied to data centres. Research published in Nature Sustainability found that the AI server industry is unlikely to meet its net-zero aspirations by 2030 without substantial reliance on highly uncertain carbon offset and water restoration mechanisms.

The Nuclear Response

The tech industry's appetite for electricity has sparked a remarkable revival in nuclear power investment, driven not by governments but by the companies building AI infrastructure.

In September 2024, Microsoft and Constellation Energy announced a 20-year power purchase agreement to bring the dormant Unit 1 reactor at Three Mile Island back online. Microsoft will purchase a significant portion of the plant's 835 megawatt output to power its AI data centres in the mid-Atlantic region. The project, renamed the Christopher M. Crane Clean Energy Center, represents the first time a retired nuclear reactor in the United States is being restored to serve a single corporate customer. In November 2025, the United States Department of Energy Loan Programs Office closed a 1 billion dollar federal loan to Constellation Energy, lowering the barrier to the restart. The reactor is targeted to resume operation in 2028.

Big tech companies signed contracts for more than 10 gigawatts of potential new nuclear capacity in the United States over the past year. Amazon Web Services secured a 10-year agreement to draw hundreds of megawatts from Talen Energy's Susquehanna nuclear plant in Pennsylvania. It subsequently obtained a 1.92 gigawatt power purchase agreement from the same facility and invested 500 million dollars in small modular reactor development. Google partnered with startup Kairos Power to deploy up to 500 megawatts of advanced nuclear capacity by the early 2030s. Kairos received a Nuclear Regulatory Commission construction licence in November 2024 for its Hermes 35 megawatt demonstration reactor in Oak Ridge, Tennessee.

Meta announced in June 2025 a 20-year agreement to buy 1.1 gigawatts of nuclear energy from the Clinton Clean Energy Center in Illinois. The commitment will support an expansion of the facility's output and deliver 13.5 million dollars in annual tax revenue to the surrounding community.

These deals represent an extraordinary acceleration in corporate energy procurement. Global electricity generation for data centres is projected to grow from 460 terawatt hours in 2024 to over 1,000 terawatt hours in 2030 and 1,300 terawatt hours by 2035. Nuclear offers carbon-free baseload power, but new reactors take years to build. The question is whether nuclear capacity can scale fast enough to meet AI's demand growth, or whether fossil fuels will fill the gap in the interim.

Quantifying the Trade-off

The most important question is whether AI's climate benefits outweigh its energy costs. Recent research offers the most rigorous attempt yet to answer it.

A study published in Nature's npj Climate Action by the Grantham Research Institute on Climate Change and the Environment and Systemiq found that AI advancements in power, transport, and food consumption could reduce global greenhouse gas emissions by 3.2 to 5.4 billion tonnes of carbon dioxide equivalent annually by 2035. In the power sector, AI could enhance renewable energy efficiency to reduce emissions by approximately 1.8 gigatonnes annually. In food systems, AI could accelerate adoption of alternative proteins to replace up to 50 per cent of meat and dairy consumption, saving approximately 3 gigatonnes per year. In mobility, AI-enabled shared transport and optimised electric vehicle adoption could reduce emissions by roughly 0.6 gigatonnes annually.

The IEA's own analysis supports a positive net impact. The adoption of existing AI applications in end-use sectors could lead to 1,400 megatonnes of carbon dioxide emissions reductions in 2035 in a Widespread Adoption scenario. That figure does not include breakthrough discoveries that might emerge thanks to AI over the next decade. By comparison, the IEA's base case projects total data centre emissions rising from approximately 180 million metric tonnes of carbon dioxide today to 300 million metric tonnes by 2035, potentially reaching 500 million metric tonnes in a high-growth scenario.

On these numbers, the potential emissions reductions from AI applications would be three to four times larger than the total emissions from the data centres running them. AI's net impact, the research suggests, remains overwhelmingly positive, provided it is intentionally applied to accelerate low-carbon technologies.

But that conditional is doing a great deal of work. The IEA cautioned that there is currently no momentum ensuring widespread adoption of beneficial AI applications. Their aggregate impact could be marginal if the necessary enabling conditions are not created. Barriers include constraints on access to data, absence of digital infrastructure and skills, regulatory and security restrictions, and social or cultural obstacles. Commercial incentives to apply AI in socially productive climate applications may be weak without active public policy.

Google Maps' eco-friendly routing uses AI to suggest routes with fewer hills, less traffic, and constant speeds. It has helped prevent over 1 million tonnes of carbon dioxide annually in its initial rollout across selected cities in Europe and the United States, equivalent to taking 200,000 cars off the road. But that application exists because it aligns with user preferences for faster routes. Many climate applications require explicit investment with less obvious commercial return.

Efficiency Gains and Green AI

Research is advancing on making AI itself more efficient. A report published by UNESCO and University College London found that small changes to how large language models are built and used can dramatically reduce energy consumption without compromising performance. Model compression through techniques such as quantisation can save up to 44 per cent in energy while maintaining accuracy. Experimental results reveal that optimisation methods can reduce energy consumption and carbon emissions by up to 45 per cent, making them suitable for resource-constrained environments.

Luccioni's research at Hugging Face demonstrated that using large generative models to create outputs is far more energy-intensive than using smaller AI models tailored for specific tasks. Using a generative model to classify movie reviews consumes around 30 times more energy than using a fine-tuned model created specifically for that purpose. The implication is significant: not every application requires a massive general-purpose model.

IBM released architecture details for its Telum II Processor and Spyre Accelerator, designed to reduce AI-based energy consumption and data centre footprint. Power-capping hardware has been shown to decrease energy consumption by up to 15 per cent whilst only increasing response time by a barely noticeable 3 per cent.

The training of Hugging Face's BLOOM model with 176 billion parameters consumed 433 megawatt hours of electricity, resulting in 25 metric tonnes of carbon dioxide equivalent. The relatively modest figure owes to its training on a French supercomputer powered mainly by nuclear energy, demonstrating that where and how AI is trained matters as much as model size.

A new movement in green AI is emerging, shifting from the bigger is better paradigm to small is sufficient, emphasising energy sobriety through smaller, more efficient models. Small models are particularly useful in settings where energy and water are scarce, and they are more accessible in environments with limited connectivity.

The Transparency Problem

Any honest assessment of AI's climate impact faces a fundamental obstacle: we do not actually know how much energy AI systems consume. Currently, there are no comprehensive global datasets on data centre electricity consumption or emissions. Few governments mandate reporting of such figures. All numbers concerning AI's energy and climate impact are therefore estimates, often based on limited disclosures and modelling assumptions.

Factors including which data centre processes a given request, how much energy that centre uses, and how carbon-intensive its energy sources are tend to be knowable only to the companies running the models. This is true for most major systems including ChatGPT, Gemini, and Claude. OpenAI's Sam Altman stated a figure of 0.34 watt hours per query in a blog post, but some researchers say the smartest models can consume over 20 watt hours for a complex query. The range of uncertainty spans nearly two orders of magnitude.

Luccioni has called for mandatory disclosure of AI systems' environmental footprints. She points out that current AI benchmarks often omit critical energy consumption metrics entirely. Without standardised reporting, neither researchers nor policymakers can make informed decisions about the technology's true costs and benefits.

The UK's AI Energy Council

The United Kingdom has taken early steps to coordinate AI and energy policy at a national level. The AI Energy Council held its inaugural meeting in April 2025, establishing five key areas of focus. These priorities centre on ensuring the UK's energy system can support AI and compute infrastructure, promoting sustainability through renewable energy solutions, focusing on safe and secure AI adoption across the energy system, and advising on how AI can support the transition to net zero.

The Council's membership spans major technology companies including Google, Microsoft, Amazon Web Services, ARM, and Equinix, alongside energy sector participants including the National Energy System Operator, Ofgem, National Grid, Scottish Power, EDF Energy, and the Nuclear Industry Association. The IEA shared analysis at Council meetings indicating that model inference, not training, will be the dominant driver of AI energy use going forward.

A National Commission was announced to accelerate safe access to AI in healthcare, with plans to publish a new regulatory framework in 2026. The NHS Fit For The Future 10 Year Health Plan, published in July 2025, identified AI alongside data, genomics, wearables, and robotics as strategic technological priorities.

These institutional developments reflect growing recognition that AI's energy demands cannot be managed through market forces alone. They require coordination between technology developers, energy providers, and government bodies.

Tension Without Resolution

The climate contradiction at the heart of artificial intelligence does not resolve itself through technological optimism or pessimism. Both narratives contain truth. AI genuinely offers capabilities for climate monitoring, energy optimisation, and scientific discovery that no other technology can match. AI also genuinely imposes energy and water costs that are growing faster than almost any other category of industrial activity.

The Grantham Institute and Systemiq research offers what may be the most useful framing. Using best available estimates, AI could add 0.4 to 1.6 gigatonnes of carbon dioxide equivalent annually by 2035 through data centre energy demand. If effectively applied to accelerate low-carbon technologies, AI could reduce emissions by 3.2 to 5.4 gigatonnes annually over the same period. The net balance favours climate benefit, but only if beneficial applications are actively developed and deployed.

This is not a technology problem. It is a policy problem. The commercial incentives driving AI development overwhelmingly favour applications that generate revenue: chatbots, image generators, productivity tools, advertising optimisation. Climate applications often require public investment, regulatory frameworks, and infrastructure that markets do not automatically provide.

Luccioni has expressed frustration with the current trajectory. “We don't need generative AI in web search. Nobody asked for AI chatbots in messaging apps or on social media. This race to stuff them into every single existing technology is truly infuriating, since it comes with real consequences to our planet.” Her critique points to a deeper issue. The AI systems consuming the most energy are not primarily those monitoring deforestation or optimising power grids. They are those generating text, images, and video for applications whose climate value is questionable at best.

The largest tech companies have all set targets to become water positive by 2030, committing to replenish more water than their operations consume. Amazon, Alphabet, Microsoft, and Meta have joined a pledge to triple the world's nuclear capacity by 2050. These commitments are meaningful, but they also constitute an acknowledgment that current trajectories are unsustainable. If the status quo were compatible with net-zero goals, such dramatic interventions would be unnecessary.

Where This Leaves Us

Will AI solve the climate crisis or accelerate it? The honest answer is that it depends entirely on choices that remain to be made.

If AI development continues primarily along commercial lines, with efficiency gains continually outpaced by proliferation into ever more applications, the technology's energy footprint will continue its rapid expansion. Data centre electricity demand doubling by 2030 is the baseline projection. Higher-growth scenarios are entirely plausible.

If governments, international institutions, and technology companies actively prioritise climate applications, if AI is deployed to optimise energy grids, accelerate materials discovery, monitor emissions, and transform food systems, the potential emissions reductions dwarf the energy costs of the technology itself.

The technology is agnostic. It will do whatever its builders and users direct it to do. A search chatbot and a deforestation monitoring system run on fundamentally similar infrastructure. The difference lies in what questions we ask and what answers we choose to act upon.

The IEA noted that nearly half of emissions reductions required by 2050 will come from technologies not yet fully developed. AI could accelerate their discovery. DeepMind's AlphaFold decoded over 200 million protein structures, unlocking advances in areas including alternative proteins and energy storage. An overly simplistic view of AI's impacts risks underestimating its potential for accelerating important climate-solution breakthroughs, such as developing less expensive and more powerful batteries in months rather than decades.

But those breakthroughs do not happen automatically. They require funding, institutional support, data access, and regulatory frameworks. They require deciding that climate applications of AI are as important as consumer applications, and investing accordingly.

The servers are humming. The electricity meters are spinning. The satellites are watching. The question is not whether artificial intelligence will shape our climate future. It is whether we will shape artificial intelligence to serve that future, or simply allow it to consume resources in pursuit of whatever generates the next quarterly return.

The answer will determine more than the trajectory of a technology. It will determine whether the most powerful tools humanity has ever built become instruments of our survival or accelerants of our crisis. The data centres do not care which role they play. That choice belongs to us.

---

References and Sources

1. International Energy Agency. (2025). “Energy and AI: Energy Demand from AI.” IEA Reports. https://www.iea.org/reports/energy-and-ai/energy-demand-from-ai

2. International Energy Agency. (2025). “AI and Climate Change.” IEA Reports. https://www.iea.org/reports/energy-and-ai/ai-and-climate-change

3. Climate TRACE. (2025). “Global Emissions Monitoring Platform.” https://climatetrace.org/

4. Pew Research Center. (2025). “What We Know About Energy Use at U.S. Data Centers Amid the AI Boom.” https://www.pewresearch.org/short-reads/2025/10/24/what-we-know-about-energy-use-at-us-data-centers-amid-the-ai-boom/

5. Carbon Brief. (2025). “AI: Five Charts That Put Data-Centre Energy Use and Emissions Into Context.” https://www.carbonbrief.org/ai-five-charts-that-put-data-centre-energy-use-and-emissions-into-context/

6. MIT Technology Review. (2025). “We Did the Math on AI's Energy Footprint. Here's the Story You Haven't Heard.” https://www.technologyreview.com/2025/05/20/1116327/ai-energy-usage-climate-footprint-big-tech/

7. MIT News. (2025). “Explained: Generative AI's Environmental Impact.” https://news.mit.edu/2025/explained-generative-ai-environmental-impact-0117

8. Google DeepMind. (2016). “DeepMind AI Reduces Google Data Centre Cooling Bill by 40%.” https://deepmind.google/blog/deepmind-ai-reduces-google-data-centre-cooling-bill-by-40/

9. Google DeepMind. (2018). “Safety-First AI for Autonomous Data Centre Cooling and Industrial Control.” https://deepmind.google/blog/safety-first-ai-for-autonomous-data-centre-cooling-and-industrial-control/

10. Epoch AI. (2024). “How Much Energy Does ChatGPT Use?” https://epoch.ai/gradient-updates/how-much-energy-does-chatgpt-use

11. Ritchie, H. (2025). “What's the Carbon Footprint of Using ChatGPT?” https://hannahritchie.substack.com/p/carbon-footprint-chatgpt

12. S&P Global. (2025). “Global Data Center Power Demand to Double by 2030 on AI Surge: IEA.” https://www.spglobal.com/energy/en/news-research/latest-news/electric-power/041025-global-data-center-power-demand-to-double-by-2030-on-ai-surge-iea

13. World Economic Forum. (2025). “How Data Centres Can Avoid Doubling Their Energy Use by 2030.” https://www.weforum.org/stories/2025/12/data-centres-and-energy-demand/

14. Luccioni, S. (2025). “The Environmental Impacts of AI: Primer.” Hugging Face Blog. https://huggingface.co/blog/sasha/ai-environment-primer

15. MIT Technology Review. (2023). “Making an Image with Generative AI Uses as Much Energy as Charging Your Phone.” https://www.technologyreview.com/2023/12/01/1084189/making-an-image-with-generative-ai-uses-as-much-energy-as-charging-your-phone/

16. Springer Nature. (2024). “Green AI: Exploring Carbon Footprints, Mitigation Strategies, and Trade Offs in Large Language Model Training.” https://link.springer.com/article/10.1007/s44163-024-00149-w

17. NPR. (2024). “Three Mile Island Nuclear Plant Will Reopen to Power Microsoft Data Centers.” https://www.npr.org/2024/09/20/nx-s1-5120581/three-mile-island-nuclear-power-plant-microsoft-ai

18. IEEE Spectrum. (2024). “Microsoft Powers Data Centers with Three Mile Island Nuclear.” https://spectrum.ieee.org/three-mile-island

19. Nature. (2025). “Will AI Accelerate or Delay the Race to Net-Zero Emissions?” https://www.nature.com/articles/d41586-024-01137-x

20. LSE Grantham Research Institute. (2025). “New Study Finds AI Could Reduce Global Emissions Annually by 3.2 to 5.4 Billion Tonnes of Carbon-Dioxide-Equivalent by 2035.” https://www.lse.ac.uk/granthaminstitute/news/new-study-finds-ai-could-reduce-global-emissions-annually-by-3-2-to-5-4-billion-tonnes-of-carbon-dioxide-equivalent-by-2035/

21. Nature. (2025). “Green and Intelligent: The Role of AI in the Climate Transition.” npj Climate Action. https://www.nature.com/articles/s44168-025-00252-3

22. MDPI. (2025). “AI-Based Energy Management and Optimization for Urban Infrastructure: A Case Study in Trikala, Greece.” https://www.mdpi.com/3042-5743/35/1/76

23. PV Magazine. (2025). “AI Powered Solar Forecasting Helps UK Grid Operator Reduce Balancing Costs.” https://www.pv-magazine.com/2025/11/07/ai-powered-solar-forecasting-helps-uk-grid-operator-reduce-balancing-costs/

24. NVIDIA Blog. (2024). “AI Nonprofit Forecasts Solar Energy for UK Grid.” https://blogs.nvidia.com/blog/ai-forecasts-solar-energy-uk/

25. GOV.UK. (2025). “AI Energy Council Minutes: Monday 30 June 2025.” https://www.gov.uk/government/publications/ai-energy-council-meetings-minutes/ai-energy-council-minutes-monday-30-june-2025-html

26. Brookings Institution. (2025). “AI, Data Centers, and Water.” https://www.brookings.edu/articles/ai-data-centers-and-water/

27. Environmental and Energy Study Institute. (2025). “Data Centers and Water Consumption.” https://www.eesi.org/articles/view/data-centers-and-water-consumption

28. Nature Sustainability. (2025). “Environmental Impact and Net-Zero Pathways for Sustainable Artificial Intelligence Servers in the USA.” https://www.nature.com/articles/s41893-025-01681-y

29. UNESCO. (2025). “AI Large Language Models: New Report Shows Small Changes Can Reduce Energy Use by 90%.” https://www.unesco.org/en/articles/ai-large-language-models-new-report-shows-small-changes-can-reduce-energy-use-90

30. U.S. Department of Energy. (2024). “AI for Energy: Opportunities for a Modern Grid and Clean Energy Economy.” https://www.energy.gov/sites/default/files/2024-04/AI%20EO%20Report%20Section%205.2g(i)_043024.pdf

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

Every time you type a message on your smartphone, your keyboard learns a little more about you. It notices your favourite words, your common misspellings, the names of people you text most often. For years, this intimate knowledge was hoovered up and shipped to distant servers, where tech giants analysed your linguistic fingerprints alongside billions of others. Then, around 2017, something changed. Google began training its Gboard keyboard using a technique called federated learning, promising that your typing data would never leave your device. The raw text of your most private messages, they assured users, would stay exactly where it belonged: on your phone.

It sounds like a privacy advocate's dream. But beneath this reassuring narrative lies a more complicated reality, one where mathematical guarantees collide with practical vulnerabilities, where corporate interests shape the definition of “privacy,” and where the gap between what users understand and what actually happens grows wider by the day. As AI systems increasingly rely on techniques like federated learning and differential privacy to protect sensitive information, a fundamental question emerges: are these technical solutions genuine shields against surveillance, or are they elaborate mechanisms that create new attack surfaces whilst giving companies plausible deniability?

The Machinery of Privacy Preservation

To understand whether federated learning and differential privacy actually work, you first need to understand what they are and how they operate. These are not simple concepts, and that complexity itself becomes part of the problem.

Federated learning, first formally introduced by Google researchers in 2016, fundamentally reimagines how machine learning models are trained. In the traditional approach, organisations collect vast quantities of data from users, centralise it on their servers, and train AI models on this aggregated dataset. Federated learning inverts this process. Instead of bringing data to the model, it brings the model to the data.

The process works through a carefully orchestrated dance between a central server and millions of edge devices, typically smartphones. The server distributes an initial model to participating devices. Each device trains that model using only its local data, perhaps the messages you have typed, the photos you have taken, or the websites you have visited. Crucially, the raw data never leaves your device. Instead, each device sends back only the model updates, the mathematical adjustments to weights and parameters that represent what the model learned from your data. The central server aggregates these updates from thousands or millions of devices, incorporates them into a new global model, and distributes this improved version back to the devices. The cycle repeats until the model converges.

The technical details matter here. Google's implementation in Gboard uses the FederatedAveraging algorithm, with between 100 and 500 client updates required to close each round of training. On average, each client processes approximately 400 example sentences during a single training epoch. The federated system converges after about 3000 training rounds, during which 600 million sentences are processed by 1.5 million client devices.

Differential privacy adds another layer of protection. Developed by computer scientists including Cynthia Dwork of Harvard University, who received the National Medal of Science in January 2025 for her pioneering contributions to the field, differential privacy provides a mathematically rigorous guarantee about information leakage. The core idea is deceptively simple: if you add carefully calibrated noise to data or computations, you can ensure that the output reveals almost nothing about any individual in the dataset.

The formal guarantee states that an algorithm is differentially private if its output looks nearly identical whether or not any single individual's data is included in the computation. This is measured by a parameter called epsilon, which quantifies the privacy loss. A smaller epsilon means stronger privacy but typically comes at the cost of utility, since more noise obscures more signal.

The noise injection typically follows one of several mechanisms. The Laplace mechanism adds noise calibrated to the sensitivity of the computation. The Gaussian mechanism uses a different probability distribution, factoring in both sensitivity and privacy parameters. Each approach has trade-offs in terms of accuracy, privacy strength, and computational efficiency.

When combined, federated learning and differential privacy create what appears to be a formidable privacy fortress. Your data stays on your device. The model updates sent to the server are aggregated with millions of others. Additional noise is injected to obscure individual contributions. In theory, even if someone intercepted everything being transmitted, they would learn nothing meaningful about you.

In practice, however, the picture is considerably more complicated.

When Privacy Promises Meet Attack Vectors

The security research community has spent years probing federated learning systems for weaknesses, and they have found plenty. One of the most troubling discoveries involves gradient inversion attacks, which demonstrate that model updates themselves can leak significant information about the underlying training data.

A gradient, in machine learning terms, is the mathematical direction and magnitude by which model parameters should be adjusted based on training data. Researchers have shown that by analysing these gradients, attackers can reconstruct substantial portions of the original training data. A 2025 systematic review published in Frontiers in Computer Science documented how gradient-guided diffusion models can now achieve “visually perfect recovery of images up to 512x512 pixels” from gradient information alone.

The evolution of these attacks has been rapid. Early gradient inversion techniques required significant computational resources and produced only approximate reconstructions. Modern approaches using fine-tuned generative models reduce mean squared error by an order of magnitude compared to classical methods, whilst simultaneously achieving inference speeds a million times faster and demonstrating robustness to gradient noise.

The implications are stark. Even though federated learning never transmits raw data, the gradients it does transmit can serve as a detailed map back to that data. A team of researchers demonstrated this vulnerability specifically in the context of Google's Gboard, publishing their findings in a paper pointedly titled “Two Models are Better than One: Federated Learning is Not Private for Google GBoard Next Word Prediction.” Their work showed that the word order and actual sentences typed by users could be reconstructed with high fidelity from the model updates alone.

Beyond gradient leakage, federated learning systems face threats from malicious participants. In Byzantine attacks, compromised devices send deliberately corrupted model updates designed to poison the global model. Research published by Fang et al. at NDSS in 2025 demonstrated that optimised model poisoning attacks can cause “1.5x to 60x higher reductions in the accuracy of FL models compared to previously discovered poisoning attacks.” This suggests that existing defences against malicious participants are far weaker than previously assumed.

Model inversion attacks present another concern. These techniques attempt to reverse-engineer sensitive information about training data by querying a trained model. A February 2025 paper on arXiv introduced “federated unlearning inversion attacks,” which exploit the model differences before and after data deletion to expose features and labels of supposedly forgotten data. As regulations like the GDPR establish a “right to be forgotten,” the very mechanisms designed to delete user data may create new vulnerabilities.

Differential privacy, for its part, is not immune to attack either. Research has shown that DP-SGD, the standard technique for adding differential privacy to deep learning, cannot prevent certain classes of model inversion attacks. A study by Zhang et al. demonstrated that their generative model inversion attack in face recognition settings could succeed even when the target model was trained with differential privacy guarantees.

The Census Bureau's Cautionary Tale

Perhaps the most instructive real-world example of differential privacy's limitations comes from the US Census Bureau's adoption of the technique for the 2020 census. This was differential privacy's biggest test, applied to data that would determine congressional representation and the allocation of hundreds of billions of dollars in federal funds.

The results were controversial. Research published in PMC in 2024 found that “the total population counts are generally preserved by the differential privacy algorithm. However, when we turn to population subgroups, this accuracy depreciates considerably.” The same study documented that the technique “introduces disproportionate discrepancies for rural and non-white populations,” with “significant changes in estimated mortality rates” occurring for less populous areas.

For demographers and social scientists, the trade-offs proved troubling. A Gates Open Research study quantified the impact: when run on historical census data with a privacy budget of 1.0, the differential privacy system produced errors “similar to that of a simple random sample of 50% of the US population.” In other words, protecting privacy came at the cost of effectively throwing away half the data. With a privacy budget of 4.0, the error rate decreased to approximate that of a 90 percent sample, but privacy guarantees correspondingly weakened.

The Census Bureau faced criticism from data users who argued that local governments could no longer distinguish between actual errors in their data and noise introduced by the privacy algorithm. The structural inaccuracy preserved state-level totals whilst “intentionally distorting characteristic data at each sub-level.”

This case illuminates a fundamental tension in differential privacy: the privacy-utility trade-off is not merely technical but political. Decisions about how much accuracy to sacrifice for privacy, and whose data bears the greatest distortion, are ultimately value judgements that mathematics alone cannot resolve.

Corporate Privacy, Corporate Interests

When technology companies tout their use of federated learning and differential privacy, it is worth asking what problems these techniques actually solve, and for whom.

Google's deployment of federated learning in Gboard offers a revealing case study. The company has trained and deployed more than twenty language models for Gboard using differential privacy, achieving what they describe as “meaningfully formal DP guarantees” with privacy parameters (rho-zCDP) ranging from 0.2 to 2. This sounds impressive, but the privacy parameters alone do not tell the full story.

Google applies the DP-Follow-the-Regularized-Leader algorithm specifically because it achieves formal differential privacy guarantees without requiring uniform sampling of client devices, a practical constraint in mobile deployments. The company reports that keyboard prediction accuracy improved by 24 percent through federated learning, demonstrating tangible benefits from the approach.

Yet Google still learns aggregate patterns from billions of users. The company still improves its products using that collective intelligence. Federated learning changes the mechanism of data collection but not necessarily the fundamental relationship between users and platforms. As one Google research publication frankly acknowledged, “improvements to this technology will benefit all users, although users are only willing to contribute if their privacy is ensured.”

The tension becomes even starker when examining Meta, whose platforms represent some of the largest potential deployments of privacy-preserving techniques. A 2025 analysis in Springer Nature noted that “approximately 98% of Meta's revenue derives from targeted advertising, a model that depends heavily on the collection and analysis of personal data.” This business model “creates a strong incentive to push users to sacrifice privacy, raising ethical concerns.”

Privacy-preserving techniques can serve corporate interests in ways that do not necessarily align with user protection. They enable companies to continue extracting value from user data whilst reducing legal and reputational risks. They provide technical compliance with regulations like the GDPR without fundamentally changing surveillance-based business models.

Apple presents an interesting contrast. The company has integrated differential privacy across its ecosystem since iOS 10 in 2016, using it for features ranging from identifying popular emojis to detecting domains that cause high memory usage in Safari. In iOS 17, Apple applied differential privacy to learn about popular photo locations without identifying individual users. With iOS 18.5, the company extended these techniques to train certain Apple Intelligence features, starting with Genmoji.

Apple's implementation deploys local differential privacy, meaning data is randomised before leaving the device, so Apple's servers never receive raw user information. Users can opt out entirely through Settings, and privacy reports are visible in device settings, providing a degree of transparency unusual in the industry.

Apple's approach differs from Google's in that the company does not derive the majority of its revenue from advertising. Yet even here, questions arise about transparency and user understanding. The technical documentation is dense, the privacy parameters are not prominently disclosed, and the average user has no practical way to verify the claimed protections.

The Understanding Gap

The gap between technical privacy guarantees and user comprehension represents perhaps the most significant challenge facing these technologies. Differential privacy's mathematical rigour means nothing if users cannot meaningfully consent to, or even understand, what they are agreeing to.

Research on the so-called “privacy paradox” consistently finds a disconnect between stated privacy concerns and actual behaviour. A study analysing Alipay users found “no relationship between respondents' self-stated privacy concerns and their number of data-sharing authorizations.” Rather than indicating irrational behaviour, the researchers argued this reflects the complexity of privacy decisions in context.

A 2024 Deloitte survey found that less than half of consumers, 47 percent, trust online services to protect their data. Yet a separate survey by HERE Technologies found that more than two-thirds of consumers expressed willingness to share location data, with 79 percent reporting they would allow navigation services to access their data. A study of more than 10,000 respondents across 10 countries found 53 percent expressing concern about digital data sharing, even as 70 percent indicated growing willingness to share location data when benefits were clear.

This is not necessarily a paradox so much as an acknowledgment that privacy decisions involve trade-offs that differ by context, by benefit received, and by trust in the collecting entity. But federated learning and differential privacy make these trade-offs harder to evaluate, not easier. When a system claims to be “differentially private with epsilon equals 4,” what does that actually mean for the user? When federated learning promises that “your data never leaves your device,” does that account for the information that gradients can leak?

The French data protection authority CNIL has recommended federated learning as a “data protection measure from the outset,” but also acknowledged the need for “explainability and traceability measures regarding the outputs of the system.” The challenge is that these systems are inherently difficult to explain. Their privacy guarantees are statistical, not absolute. They protect populations, not necessarily individuals. They reduce risk without eliminating it.

Healthcare: High Stakes, Conflicting Pressures

Nowhere are the tensions surrounding privacy-preserving AI more acute than in healthcare, where the potential benefits are enormous and the sensitivity of data is extreme.

NVIDIA's Clara federated learning platform exemplifies both the promise and the complexity. Clara enables hospitals to collaboratively train AI models without sharing patient data. Healthcare institutions including the American College of Radiology, Massachusetts General Hospital and Brigham and Women's Hospital's Center for Clinical Data Science, and UCLA Health have partnered with NVIDIA on federated learning initiatives.

In the United Kingdom, NVIDIA partnered with King's College London and the AI company Owkin to create a federated learning platform for the National Health Service, initially connecting four of London's premier teaching hospitals. The Owkin Connect platform uses blockchain technology to capture and trace all data used for model training, providing an audit trail that traditional centralised approaches cannot match.

During the COVID-19 pandemic, NVIDIA coordinated a federated learning study involving twenty hospitals globally to train models predicting clinical outcomes in symptomatic patients. The study demonstrated that federated models could outperform models trained on any single institution's data alone, suggesting that the technique enables collaboration that would otherwise be impossible due to privacy constraints.

In the pharmaceutical industry, the MELLODDY project brought together ten pharmaceutical companies in Europe to apply federated learning to drug discovery. The consortium pools the largest existing chemical compound library, more than ten million molecules and one billion assays, whilst ensuring that highly valuable proprietary data never leaves each company's control. The project runs on the open-source Substra framework and employs distributed ledger technology for full traceability.

These initiatives demonstrate genuine value. Healthcare AI trained on diverse populations across multiple institutions is likely to generalise better than AI trained on data from a single hospital serving a particular demographic. Federated learning makes such collaboration possible in contexts where data sharing would be legally prohibited or practically impossible.

But the same vulnerabilities that plague federated learning elsewhere apply here too, perhaps with higher stakes. Gradient inversion attacks could potentially reconstruct medical images. Model poisoning by a malicious hospital could corrupt a shared diagnostic tool. The privacy-utility trade-off means that stronger privacy guarantees may come at the cost of clinical accuracy.

Regulation Catches Up, Slowly

The regulatory landscape is evolving to address these concerns, though the pace of change struggles to keep up with technological development.

In the European Union, the AI Act took full effect on 2 August 2025, establishing transparency obligations for general-purpose AI systems. In November 2025, the European Commission published the Digital Omnibus proposal, streamlining the relationship between the Data Act, GDPR, and AI Act. The proposal includes clarification that organisations “may rely on legitimate interests to process personal data for AI-related purposes, provided they fully comply with all existing GDPR safeguards.”

In the United States, NIST finalised guidelines for evaluating differential privacy guarantees in March 2025, fulfilling an assignment from President Biden's Executive Order on Safe, Secure, and Trustworthy AI from October 2023. The guidelines provide a framework for assessing privacy claims but acknowledge the complexity of translating mathematical parameters into practical privacy assurances.

The market is responding to these regulatory pressures. The global privacy-enhancing technologies market reached 3.12 billion US dollars in 2024, projected to grow to 12.09 billion dollars by 2030. The federated learning platforms market, valued at 150 million dollars in 2023, is forecast to reach 2.3 billion dollars by 2032, reflecting a compound annual growth rate of 35.4 percent. The average cost of a data breach reached 4.88 million dollars in 2024, and industry analysts estimate that 75 percent of the world's population now lives under modern privacy regulations.

This growth suggests that corporations see privacy-preserving techniques as essential infrastructure for the AI age, driven as much by regulatory compliance and reputational concerns as by genuine commitment to user protection.

The Security Arms Race

The relationship between privacy-preserving techniques and the attacks against them resembles an arms race, with each advance prompting countermeasures that prompt new attacks in turn.

Defensive techniques have evolved significantly. Secure aggregation protocols encrypt model updates so that the central server only learns the aggregate, not individual contributions. Homomorphic encryption allows computation on encrypted data, theoretically enabling model training without ever decrypting sensitive information. Byzantine-robust aggregation algorithms attempt to detect and exclude malicious model updates.

Each defence has limitations. Secure aggregation protects against honest-but-curious servers but does not prevent sophisticated attacks like Scale-MIA, which researchers demonstrated can reconstruct training data even from securely aggregated updates. Homomorphic encryption imposes significant computational overhead and is not yet practical for large-scale deployments. Byzantine-robust algorithms, as the research by Fang et al. demonstrated, are more vulnerable to optimised attacks than previously believed.

The research community continues to develop new defences. A 2025 study proposed “shadow defense against gradient inversion attack,” using decoy gradients to obscure genuine updates. LSTM-based approaches attempt to detect malicious updates by analysing patterns across communication rounds. The FedMP algorithm combines multiple defensive techniques into a “multi-pronged defence” against Byzantine attacks.

But attackers are also advancing. Gradient-guided diffusion models achieve reconstruction quality that would have seemed impossible a few years ago. Adaptive attack strategies that vary the number of malicious clients per round prove more effective and harder to detect. The boundary between secure and insecure keeps shifting.

This dynamic suggests that privacy-preserving AI should not be understood as a solved problem but as an ongoing negotiation between attackers and defenders, with no permanent resolution in sight.

What Users Actually Want

Amid all the technical complexity, it is worth returning to the fundamental question: what do users actually want from privacy protection, and can federated learning and differential privacy deliver it?

Research suggests that user expectations are contextual and nuanced. People are more willing to share data with well-known, trusted entities than with unknown ones. They want personalised services but also want protection from misuse. They care more about some types of data than others, and their concerns vary by situation.

Privacy-preserving techniques address some of these concerns better than others. They reduce the risk of data breaches by not centralising sensitive information. They provide mathematical frameworks for limiting what can be inferred about individuals. They enable beneficial applications, such as medical AI or improved keyboard prediction, that might otherwise be impossible due to privacy constraints.

But they do not address the fundamental power imbalance between individuals and the organisations that deploy these systems. They do not give users meaningful control over how models trained on their data are used. They do not make privacy trade-offs transparent or negotiable. They replace visible data collection with invisible model training, which may reduce certain risks whilst obscuring others.

The privacy paradox literature suggests that many users make rational calculations based on perceived benefits and risks. But federated learning and differential privacy make those calculations harder, not easier. The average user cannot evaluate whether epsilon equals 2 provides adequate protection for their threat model. They cannot assess whether gradient inversion attacks pose a realistic risk in their context. They must simply trust that the deploying organisation has made these decisions competently and in good faith.

The Question That Matters

Will you feel safe sharing personal data as AI systems adopt federated learning and differential privacy? The honest answer is: it depends on what you mean by “safe.”

These techniques genuinely reduce certain privacy risks. They make centralised data breaches less catastrophic by keeping data distributed. They provide formal guarantees that limit what can be inferred about individuals, at least in theory. They enable beneficial applications that would otherwise founder on privacy concerns.

But they also create new vulnerabilities that researchers are only beginning to understand. Gradient inversion attacks can reconstruct sensitive data from model updates. Malicious participants can poison shared models. The privacy-utility trade-off means that stronger guarantees come at the cost of usefulness, a cost that often falls disproportionately on already marginalised populations.

Corporate incentives shape how these technologies are deployed. Companies that profit from data collection have reasons to adopt privacy-preserving techniques that maintain their business models whilst satisfying regulators and reassuring users. This is not necessarily malicious, but it is also not the same as prioritising user privacy above all else.

The gap between technical guarantees and user understanding remains vast. Few users can meaningfully evaluate privacy claims couched in mathematical parameters and threat models. The complexity of these systems may actually reduce accountability by making it harder to identify when privacy has been violated.

Perhaps most importantly, these techniques do not fundamentally change the relationship between individuals and the organisations that train AI on their data. They are tools that can be used for better or worse, depending on who deploys them and why. They are not a solution to the privacy problem so much as a new set of trade-offs to navigate.

The question is not whether federated learning and differential privacy make you safer, because the answer is nuanced and contextual. The question is whether you trust the organisations deploying these techniques to make appropriate decisions on your behalf, whether you believe the oversight mechanisms are adequate, and whether you accept the trade-offs inherent in the technology.

For some users, in some contexts, the answer will be yes. The ability to contribute to medical AI research without sharing raw health records, or to improve keyboard prediction without uploading every message, represents genuine progress. For others, the answer will remain no, because no amount of mathematical sophistication can substitute for genuine control over one's own data.

Privacy-preserving AI is neither panacea nor theatre. It is a set of tools with real benefits and real limitations, deployed by organisations with mixed motivations, in a regulatory environment that is still evolving. The honest assessment is that these techniques make some attacks harder and enable some attacks we have not yet fully understood. They reduce some risks whilst obscuring others. They represent progress, but not a destination.

As these technologies continue to develop, the most important thing users can do is maintain healthy scepticism, demand transparency about the specific techniques and parameters being used, and recognise that privacy in the age of AI requires ongoing vigilance rather than passive trust in technical solutions. The machines may be learning to protect your privacy, but whether they succeed depends on far more than the mathematics.

---

References and Sources

1. Google Research. “Federated Learning for Mobile Keyboard Prediction.” (2019). https://research.google/pubs/federated-learning-for-mobile-keyboard-prediction-2/

2. Google Research. “Federated Learning of Gboard Language Models with Differential Privacy.” arXiv:2305.18465 (2023). https://arxiv.org/abs/2305.18465

3. Dwork, Cynthia. “Differential Privacy.” Springer Nature, 2006. https://link.springer.com/chapter/10.1007/11787006_1

4. Harvard Gazette. “Pioneer of modern data privacy Cynthia Dwork wins National Medal of Science.” January 2025. https://news.harvard.edu/gazette/story/newsplus/pioneer-of-modern-data-privacy-cynthia-dwork-wins-national-medal-of-science/

5. NIST. “Guidelines for Evaluating Differential Privacy Guarantees.” NIST Special Publication 800-226, March 2025. https://www.nist.gov/publications/guidelines-evaluating-differential-privacy-guarantees

6. Frontiers in Computer Science. “Deep federated learning: a systematic review of methods, applications, and challenges.” 2025. https://www.frontiersin.org/journals/computer-science/articles/10.3389/fcomp.2025.1617597/full

7. arXiv. “Two Models are Better than One: Federated Learning Is Not Private For Google GBoard Next Word Prediction.” arXiv:2210.16947 (2022). https://arxiv.org/abs/2210.16947

8. NDSS Symposium. “Manipulating the Byzantine: Optimizing Model Poisoning Attacks and Defenses for Federated Learning.” 2025. https://www.ndss-symposium.org/ndss-paper/manipulating-the-byzantine-optimizing-model-poisoning-attacks-and-defenses-for-federated-learning/

9. arXiv. “Model Inversion Attack against Federated Unlearning.” arXiv:2502.14558 (2025). https://arxiv.org/abs/2502.14558

10. NDSS Symposium. “Scale-MIA: A Scalable Model Inversion Attack against Secure Federated Learning.” 2025. https://www.ndss-symposium.org/wp-content/uploads/2025-644-paper.pdf

11. PMC. “The 2020 US Census Differential Privacy Method Introduces Disproportionate Discrepancies for Rural and Non-White Populations.” 2024. https://pmc.ncbi.nlm.nih.gov/articles/PMC11105149/

12. Gates Open Research. “Differential privacy in the 2020 US census: what will it do? Quantifying the accuracy/privacy tradeoff.” https://gatesopenresearch.org/articles/3-1722

13. Springer Nature. “Meta's privacy practices on Facebook: compliance, integrity, and a framework for excellence.” Discover Artificial Intelligence, 2025. https://link.springer.com/article/10.1007/s44163-025-00388-5

14. Apple Machine Learning Research. “Learning with Privacy at Scale.” https://machinelearning.apple.com/research/learning-with-privacy-at-scale

15. Apple Machine Learning Research. “Learning Iconic Scenes with Differential Privacy.” https://machinelearning.apple.com/research/scenes-differential-privacy

16. Apple Machine Learning Research. “Understanding Aggregate Trends for Apple Intelligence Using Differential Privacy.” https://machinelearning.apple.com/research/differential-privacy-aggregate-trends

17. Deloitte Insights. “Consumer data privacy paradox.” https://www2.deloitte.com/us/en/insights/industry/technology/consumer-data-privacy-paradox.html

18. NVIDIA Blog. “NVIDIA Clara Federated Learning to Deliver AI to Hospitals While Protecting Patient Data.” https://blogs.nvidia.com/blog/clara-federated-learning/

19. Owkin. “Federated learning in healthcare: the future of collaborative clinical and biomedical research.” https://www.owkin.com/blogs-case-studies/federated-learning-in-healthcare-the-future-of-collaborative-clinical-and-biomedical-research

20. EUR-Lex. “European Commission Digital Omnibus Proposal.” COM(2025) 835 final, November 2025. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52025DC0835

21. CNIL. “AI system development: CNIL's recommendations to comply with the GDPR.” https://www.cnil.fr/en/ai-system-development-cnils-recommendations-to-comply-gdpr

22. 360iResearch. “Privacy-Preserving Machine Learning Market Size 2025-2030.” https://www.360iresearch.com/library/intelligence/privacy-preserving-machine-learning

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

The results from Gregory Kestin's Harvard physics experiment arrived like a thunderclap. Students using an AI tutor in their dormitory rooms learned more than twice as much as their peers sitting in active learning classrooms with experienced instructors. They did it in less time. They reported feeling more engaged. Published in Scientific Reports in June 2025, the study seemed to confirm what education technology evangelists had been promising for years: artificial intelligence could finally crack the code of personalised learning at scale.

But the number that truly matters lies elsewhere. In September 2024, Khan Academy's AI-powered Khanmigo reached 700,000 students, up from just 40,000 the previous year. By the end of 2025, projections suggested more than one million students would be learning with an artificial tutor that never tires, never loses patience, and remembers every mistake a child has ever made. The question that haunts teachers, parents, and policymakers alike is brutally simple: if machines can now do what Benjamin Bloom proved most effective back in 1984, namely provide the one-to-one tutoring that outperforms group instruction by two standard deviations, does the human educator have a future?

The answer, emerging from research laboratories and classrooms across six continents, turns out to be considerably more nuanced than the headline writers would have us believe. It involves the fundamental nature of learning itself, the irreplaceable qualities that humans bring to education, and the possibility that artificial intelligence might finally liberate teachers from the very burdens that have been crushing them for decades.

The Two Sigma Dream Meets Silicon

In 1984, educational psychologist Benjamin Bloom published what would become one of the most cited papers in the history of educational research. Working with doctoral students at the University of Chicago, Bloom discovered that students who received one-to-one tutoring using mastery learning techniques performed two standard deviations better than students in conventional classrooms. The average tutored student scored higher than 98 per cent of students in the control group. Bloom called this “the 2 Sigma Problem” and challenged researchers to find methods of group instruction that could achieve similar results without the prohibitive cost of individual tutoring.

The study design was straightforward but powerful. School students were randomly assigned to one of three groups: conventional instruction with 30 students per teacher and periodic testing for marking, mastery learning with 30 students per teacher but tests given for feedback followed by corrective procedures, and one-to-one tutoring. The tutoring group's results were staggering. As Bloom noted, approximately 90 per cent of the tutored students attained the level of summative achievement reached by only the highest 20 per cent of the control class.

For forty years, that challenge remained largely unmet. Human tutors remained expensive, inconsistent in quality, and impossible to scale. Various technological interventions, from educational television to computer-assisted instruction, failed to close the gap. Radio, when it first entered schools, was predicted to revolutionise learning. Television promised the same. Each technology changed pedagogy in some ways but fell far short of approximating the tutorial relationship that Bloom had identified as the gold standard. Then came large language models.

The Harvard physics study, led by Kestin and Kelly Miller, offers the most rigorous evidence to date that AI tutoring might finally be approaching Bloom's benchmark. Using a crossover design with 194 undergraduate physics students, the researchers compared outcomes between in-class active learning sessions and at-home sessions with a custom AI tutor called PS2 Pal, built on GPT-4. Each student experienced both conditions for different topics, eliminating selection bias. The topics covered were surface tension and fluid flow, standard material in introductory physics courses.

The AI tutor was carefully designed to avoid common pitfalls. It was instructed to be brief, using no more than a few sentences at a time to prevent cognitive overload. It revealed solutions one step at a time rather than giving away complete answers. To combat hallucinations, the tendency of chatbots to fabricate information, the system was preloaded with all correct solutions. The scientists behind the experiment instructed the AI to avoid cognitive overload by limiting response length and to avoid giving away full solutions in a single message. The result: engagement ratings of 4.1 out of 5 for AI tutoring versus 3.6 for classroom instruction, with statistically significant improvements in learning outcomes (p < 10^-8). Motivation ratings showed a similar pattern: 3.4 out of 5 for AI tutoring compared to 3.1 for classroom instruction.

The study's authors were careful to emphasise limitations. Their population consisted entirely of Harvard undergraduates, raising questions about generalisability to community colleges, less selective institutions, younger students, or populations with different levels of technological access and comfort. “AI tutors shouldn't 'think' for students, but rather help them build critical thinking skills,” the researchers wrote. “AI tutors shouldn't replace in-person instruction, but help all students better prepare for it.”

The median study time also differed between conditions: 49 minutes for AI tutoring versus 60 minutes for classroom instruction. Students were not only learning more but doing so in less time, a finding that has significant implications for educational efficiency but also raises questions about what might be lost when learning is compressed.

The Global Experiment Unfolds

While researchers debate methodology in academic journals, the global education market has already placed its bets with billions of dollars. The AI in education market reached $7.05 billion in 2025 and is projected to explode to $112.30 billion by 2034, growing at a compound annual rate of 36 per cent. The market rose from $5.47 billion in 2024 to $7.57 billion in 2025, representing a 38.4 per cent increase in a single year. Global student AI usage jumped from 66 per cent in 2024 to 92 per cent in 2025, according to industry surveys. By early 2026, an estimated 86 per cent of higher education students utilised AI as their primary research and brainstorming partner.

The adoption statistics tell a remarkable story of rapid change. A survey of 2,232 teachers across the United States found that 60 per cent used AI tools during the 2024-25 school year. Usage was higher among high school teachers at 66 per cent and early-career teachers at 69 per cent. Approximately 26 per cent of districts planned to offer AI training during the 2024-25 school year, with around 74 per cent of districts expected to train teachers by the autumn of 2025. A recent survey by EDUCAUSE of more than 800 higher education institutions found that 57 per cent were prioritising AI in 2025, up from 49 per cent the previous year.

In China, Squirrel AI Learning has been at the forefront of this transformation. Founded in 2014 and headquartered in Shanghai, the company claims more than 24 million registered students and 3,000 learning centres across more than 1,500 cities. When Squirrel AI set a Guinness World Record in September 2024 by attracting 112,718 students to an online mathematics lesson in 24 hours, its adaptive learning system generated over 108,000 unique learning pathways, tailoring instruction to 99.1 per cent of participants. The system designed 111,704 unique exercise pathways for the students, demonstrating the scalability of personalised instruction. The company reports performance improvements of up to 30 per cent compared to traditional instruction.

Tom Mitchell, the former Dean of Computer Science at Carnegie Mellon University, serves as Squirrel AI's Chief AI Officer, lending academic credibility to its technical approach. The system breaks down subjects into thousands of knowledge points. For middle school mathematics alone, it maps over 10,000 concepts, from rational numbers to the Pythagorean theorem, tracking student learning behaviours to customise instruction in real time. In December 2024, Squirrel AI announced that Guinness World Records had certified its study as the “Largest AI vs Traditional Teaching Differential Experiment,” conducted with 1,662 students across five Chinese schools over one semester. The company has also provided 10 million free accounts to some of China's poorest families, addressing equity concerns that have plagued educational technology deployment.

Carnegie Learning, another pioneer in AI-powered mathematics education, has accumulated decades of evidence. Founded by cognitive psychologists and computer scientists at Carnegie Mellon University who partnered with mathematics teachers at the Pittsburgh Public Schools, the company has been compiling and acting on data to refine and improve how students explore mathematics since 1998. In an independent study funded by the US Department of Education and conducted by the RAND Corporation, the company's blended approach nearly doubled growth in performance on standardised tests relative to typical students in the second year of implementation. The gold standard randomised trial included more than 18,000 students at 147 middle and high schools across the United States. The Institute of Education Sciences published multiple reports on the effectiveness of Carnegie's Cognitive Tutor, with five of six qualifying studies showing intermediate to significant positive effects on mathematics achievement.

Meanwhile, Duolingo has transformed language learning through its AI-first strategy, producing a 51 per cent boost in daily active users and fuelling a one billion dollar revenue forecast. The company reported 47.7 million daily active users in Q2 2025, a 40 per cent year-over-year increase, with paid subscribers rising to 10.9 million and a 37 per cent year-over-year gain. Quarterly revenue grew to $252.3 million, up 41 per cent from Q2 2024. Survey data indicates that 78 per cent of regular users of its Roleplay feature, which allows practice conversations with AI characters, feel more prepared for real-world conversations after just four weeks. The “Explain My Answer” feature, adopted by 65 per cent of users, increased course completion rates by 15 per cent. Learning speed increased 40 to 60 per cent compared to pre-2024 applications.

Khan Academy's trajectory illustrates the velocity of this transformation. Khanmigo's reach expanded 731 per cent year-over-year to reach a record number of students, teachers, and parents worldwide. The platform went from about 68,000 Khanmigo student and teacher users in partner school districts in 2023-24 to more than 700,000 in the 2024-25 school year, expanding from 45 to more than 380 district partners. When rating AI tools for learning, Common Sense Media gave Khanmigo 4 stars, rising above other AI tools such as ChatGPT and Bard for educational use. Research from Khan Academy showed that combining its platform and AI tutor with additional tools and services designed for districts made it 8 to 14 times more effective at driving student learning outcomes compared with independent learning.

What Machines Cannot Replicate

Yet for all the impressive statistics and exponential growth curves, a growing body of research suggests that the most crucial elements of education remain stubbornly human.

A 2025 systematic review published in multiple peer-reviewed journals identified a troubling pattern: while AI-driven intelligent tutoring systems can improve student performance by 15 to 35 per cent, over-reliance on these systems can reduce critical thinking, creativity, and independent problem-solving. Researchers have termed this phenomenon “cognitive offloading,” the tendency of students to delegate mental work to AI rather than developing their own capabilities. Research also indicates that over-reliance on AI during practice can reduce performance in examinations taken without assistance, suggesting that AI-enhanced learning may not always translate to improved independent performance.

The ODITE 2025 Report, titled “Connected Intelligences: How AI is Redefining Personalised Learning,” warned about the excessive focus on AI's technical benefits compared to a shallow exploration of socio-emotional risks. While AI can enhance efficiency and personalise learning, the report concluded, excessive reliance may compromise essential interpersonal skills and emotional intelligence. The report called for artificial intelligence to be integrated within a pedagogy of care, not only serving performance but also recognition, inclusion, and listening.

These concerns are not merely theoretical. A study of 399 university students and 184 teachers, published in the journal Teaching and Teacher Education, found that the majority of participants argued that human teachers possess unique qualities, including critical thinking and emotions, which make them irreplaceable. The findings emphasised the importance of social-emotional competencies developed through human interactions, capacities that generative AI technologies cannot currently replicate. Participants noted that creativity and emotion are precious aspects of human quality which AI cannot replace.

Human teachers bring what researchers call “emotional intelligence” to the classroom: the ability to read subtle social cues that signal student engagement or confusion, to understand the complex personal circumstances that might affect performance, to provide the mentorship, encouragement, and emotional support that shape not just what students know but who they become. As one education researcher told the World Economic Forum: “AI cannot replace the most human dimensions of education: connection, belonging, and care. Those remain firmly in the teacher's domain.” Teachers play a vital role in guiding students to think critically about when AI adds value and when authentic human thinking and creativity are irreplaceable.

The American Psychological Association's June 2025 health advisory on AI companion software underscored these concerns in alarming terms. AI systems, the advisory warned, exploit emotional vulnerabilities through unconditional regard, triggering dependencies like digital attachment disorder while hindering social skill development. The advisory noted that manipulative design may displace or interfere with the development of healthy real-world relationships. For teenagers in particular, confusing algorithmic responses for genuine human connection can directly short-circuit developing capacities to navigate authentic social relationships and assess trustworthiness.

While AI can be a helpful supplement, genuine human connections release oxytocin, the “bonding hormone” that plays a crucial role in reducing stress and fostering emotional wellbeing. Current AI does not yet possess the empathy, intuition, and depth of understanding that humans bring to conversations. For example, a teenager feeling isolated might share their feelings with a chatbot, but the AI's responses may be generic or may not fully address deeper issues that a trained human educator would recognise and address.

The Creativity Conundrum

Beyond emotional intelligence lies another domain where human teachers remain essential: nurturing creativity.

AI tutoring systems excel at structured learning tasks, at drilling multiplication tables, at correcting grammar mistakes, at providing step-by-step guidance through physics problems. But great teachers do not just transmit facts. They inspire curiosity, challenge students to think beyond textbooks, and encourage discussions that lead to deeper understanding. When it comes to fostering creativity and open-ended problem-solving, current AI tools fall short. They lack the capacity to recognise a student's unconventional approach as potentially brilliant rather than simply incorrect.

“In the AI era, human creativity is increasingly recognised as a critical and irreplaceable capability,” noted a 2025 analysis in Frontiers in Artificial Intelligence. Fostering creativity in education requires attention to pedagogical elements that current AI systems cannot provide: the spontaneous question that opens a new line of inquiry, the willingness to follow intellectual tangents wherever they might lead, the ability to sense when a student needs encouragement to pursue an unorthodox idea. Predictions of teachers being replaced are not new. Radio, television, calculators, even the internet: each was once thought to make educators obsolete. Instead, each changed pedagogy while reinforcing the irreplaceable role of teachers in helping students make meaning, navigate complexity, and grow as people.

UNESCO's AI Competency Framework for Teachers, launched in September 2024, explicitly addresses this tension. The framework calls for a human-centred approach that integrates AI competencies with principles of human rights and human accountability. Teachers, according to UNESCO, must be equipped not only to use AI tools effectively but also to evaluate their ethical implications and to support AI literacy in students, encouraging responsible use and critical engagement with the technology.

The framework identifies five key competency aspects: a human-centred mindset that defines the critical values and attitudes necessary for interactions between humans and AI-based systems; AI ethics that establishes essential ethical principles and regulations; AI foundations and applications that specifies transferable knowledge and skills for selecting and applying AI tools; and the ability to use AI for professional development. Since 2024, UNESCO has supported 58 countries in designing or improving digital and AI competency frameworks, curricula, and quality-assured training for educators and policymakers. During Digital Learning Week 2025, UNESCO released a new report titled “AI and education: protecting the rights of learners,” providing an urgent call to action and analysing how AI and digital technologies impact access, equity, quality, and governance in education.

Liberation Through Automation

Perhaps the most compelling argument against AI teacher replacement comes from an unexpected source: the teachers themselves.

A June 2025 poll conducted by the Walton Family Foundation and Gallup surveyed more than 2,200 teachers across the United States. The findings were striking: teachers who use AI tools at least weekly save an average of 5.9 hours per week, equivalent to roughly six additional weeks of time recovered across a standard school year. This “AI dividend” allows educators to reinvest in areas that matter most: building relationships with students, providing individual attention, and developing creative lessons. Teachers who engage with AI tools more frequently report greater time savings: weekly AI users save an average of 5.9 hours each week, twice as much time as those who only use AI monthly at 2.9 hours per week.

The research documented that teachers spend up to 29 hours per week on non-teaching tasks: writing emails, grading, finding classroom resources, and completing administrative work. They have high stress levels and are at risk for burnout. Nearly half of K-12 teachers report chronic burnout, with 55 per cent considering early departure from the profession, creating a district-wide crisis that threatens both stability and student outcomes. Schools with an AI policy in place are seeing a 26 per cent larger “AI dividend,” equivalent to 2.3 hours saved per week per teacher, compared with 1.7 hours in schools without such a policy.

Despite the benefits of the “AI dividend,” only 32 per cent of teachers report using AI at least weekly, while 28 per cent use it infrequently and 40 per cent still are not using it at all. Educators use AI to create worksheets at 33 per cent, modify materials to meet students' needs at 28 per cent, complete administrative work at 28 per cent, and develop assessments at 25 per cent. Teachers in schools with an AI policy are more likely to have used AI in the past year at 70 per cent versus 60 per cent for those without.

AI offers a potential solution, not by replacing teachers but by automating the tasks that drain their energy and time. Teachers report using AI to help with lesson plans, differentiate materials for students with varying needs, write portions of individualised education programmes, and communicate with families. Sixty-four per cent of surveyed teachers say the materials they modify with AI are better quality. Sixty-one per cent say AI has improved their insights about student performance, and 57 per cent say AI has led them to enhance the quality of their student feedback and grading.

Sal Khan, founder of Khan Academy and the driving force behind Khanmigo, has consistently framed AI as a teaching assistant rather than a replacement. “AI is going to become an amazing teaching assistant,” Khan stated in March 2025. “It's going to help with grading papers, writing progress reports, communicating with others, personalising their classrooms, and writing lesson plans.” He has emphasised in multiple forums that “Teachers will always be essential. Technology has the potential to bridge learning gaps, but the key is to use it as an assistant, not a substitute.” Khan invokes the historical wisdom of one-to-one tutoring: “For most of human history, if you asked someone what great education looks like, they would say it looks like a student with their tutor. Alexander the Great had Aristotle as his tutor. Aristotle had Plato. Plato had Socrates.”

This vision aligns with what researchers call the “hybrid model” of education. The World Economic Forum's Shaping the Future of Learning insight report highlights that the main impacts of AI will be in areas such as personalised learning and augmented teaching. AI lifts administrative burdens so that people in caring roles can focus on more meaningful tasks, such as mentorship. As the role of the educator shifts, teachers are moving from traditional content delivery to facilitation, coaching, and mentorship. The future classroom is not about replacing teachers but about redefining their role from deliverers of content to curators of experience.

The Equity Question Looms Large

Any serious discussion of AI in education must confront a troubling reality: the technology that promises to democratise learning may instead widen existing inequalities.

A 2025 analysis by the International Center for Academic Integrity warned that unequal access to artificial intelligence is widening the educational gap between privileged and underprivileged students. Students from lower-income backgrounds, those in rural areas, and those attending institutions with fewer resources are often at a disadvantage when it comes to accessing the technology that powers AI tools. For these students, AI could become just another divide, reinforcing the gap between those who have and those who do not. The disproportionate impact on marginalised communities, rural populations, and underfunded educational institutions limits their ability to benefit from AI-enhanced learning.

The numbers bear this out. Half of chief technology officers surveyed in 2025 reported that their college or university does not grant students institutional access to generative AI tools. More than half of students reported that most or all of their instructors prohibit the use of generative AI entirely, according to EDUCAUSE's 2025 Students and Technology Report. AI tools often require reliable internet access, powerful devices, and up-to-date software. In regions where these resources are not readily available, students are excluded from AI-enhanced learning experiences. Much current policy energy is consumed by academic integrity concerns and bans, which address real risks but can inadvertently deepen divides by treating AI primarily as a threat rather than addressing the core equity problem of unequal opportunity to learn with and from AI.

Recommendations from the Brookings Institution and other policy organisations call for treating AI competence as a universal learning outcome so every undergraduate in every discipline graduates able to use, question, and manage AI. They advocate providing equitable access to tools and training so that benefits do not depend on personal subscriptions, and investing in faculty development at scale with time, training, and incentives to redesign courses and assessments for an AI-rich environment. Proposed solutions include increased investment in digital infrastructure, the development of affordable AI-based learning tools, and the implementation of inclusive policies that prioritise equitable access to technology. Yet only 10 per cent of schools and universities have formal AI use guidelines, according to a UNESCO survey of more than 450 institutions.

The OECD Digital Education Outlook 2026 offers a more nuanced perspective. Robust research evidence demonstrates that inexperienced tutors can enhance the quality of their tutoring and improve student learning outcomes by using educational AI tools. However, the report emphasises that if AI is designed or used without pedagogical guidance, outsourcing tasks to the technology simply enhances performance with no real learning gains. Research validates that adaptive learning's positive effects on educational equity have the capability of redressing socioeconomically disadvantaged conditions by ensuring equitable availability of educational resources.

North America dominated the AI in education market with a market share of 36 per cent in 2024, while the Asia Pacific region is expected to grow at a compound annual rate of 35.3 per cent through 2030. The United States AI in education market alone was valued at $2.01 billion in 2025 and is projected to reach $32.64 billion by 2034. A White House Executive Order signed in April 2025, “Advancing Artificial Intelligence Education for American Youth,” aims to unite AI education across all levels of learning. The US Department of Education also revealed guidance supporting schools to employ existing federal grants for AI integration.

Training the Teachers Who Will Train with AI

The gap between student adoption and teacher readiness presents a significant challenge. In 2025, there remains a notable gap between students' awareness of AI and teachers' readiness to implement AI in classrooms. According to Forbes, while 63 per cent of US teens are using AI tools like ChatGPT for schoolwork, only 30 per cent of teachers report feeling confident using these same AI tools. This difference emphasises the critical need for extensive support and AI training for all educators.

Experts emphasise the need for more partnerships between K-12 schools and higher education that provide mentorship, resources, and co-developed curricula with teachers. Faculty and researchers can help simplify AI for teachers, offer training, and ensure educational tools are designed with classroom realities in mind. AI brings a new level of potential to the table, a leap beyond past solutions. Instead of just saving time, AI aims to reshape how teachers manage their classrooms, offering a way to automate the administrative load, personalise student support, and free up teachers to focus on what they do best: teaching.

The key to successful AI integration is balance. AI has the potential to alleviate burnout and improve the teaching experience, but only if used thoughtfully as a tool, not a replacement. Competent, research-driven teachers are not going to be replaced by AI. The vision is AI as a classroom assistant that handles routine tasks while educators focus on what only they can provide: authentic human connection, professional judgement, and mentorship.

The Horizon Beckons

The evidence suggests neither a utopia of AI-powered learning nor a dystopia of displaced teachers. Instead, a more complex picture emerges: one in which artificial intelligence becomes a powerful tool that transforms rather than eliminates the human role in education.

By 2026, over 60 per cent of schools globally are projected to use AI-powered platforms. The United States has moved aggressively, with a White House Executive Order signed in April 2025 to advance AI education from K-12 through postsecondary levels. All 50 states have considered AI-related legislation. California's SB 243, signed in October 2025 and taking effect on 1 January 2026, requires operators of “companion chatbots” to maintain protocols for preventing their systems from producing content related to suicidal ideation and self-harm, with annual reports to the California Department of Public Health. New York's AI Companion Models law, effective 5 November 2025, requires notifications that state in bold capitalised letters: “THE AI COMPANION IS A COMPUTER PROGRAM AND NOT A HUMAN BEING. IT IS UNABLE TO FEEL HUMAN EMOTION.”

The PISA 2025 Learning in the Digital World assessment will focus on two competencies essential to learning with technologies: self-regulated learning and the ability to engage with digital tools. Results are expected in December 2027. Looking further ahead, the PISA 2029 Media and Artificial Intelligence Literacy assessment will examine whether young students have had opportunities to engage proactively and critically in a world where production, participation, and social networking are increasingly mediated by digital and AI tools. The OECD and European Commission's draft AI Literacy Framework, released in May 2025, aims to define global AI literacy standards for school-aged children, equipping them to use, understand, create with, and critically engage with AI.

The future English language classroom, as described by Oxford University Press, will be “human-centred, powered by AI.” Teachers will shift from traditional content delivery to facilitation, coaching, and mentorship. AI will handle routine tasks, while humans focus on what only they can provide: authentic connection, professional judgement, and the kind of mentorship that shapes lives. Hyper-personalised learning is becoming standard, with students needing tailored, real-time feedback more than ever, and AI adapting instruction moment to moment based on individual readiness.

“It is one of the biggest misnomers in education reform: that if you can give kids better technology, if you give them a laptop, if you give them better content, they will learn,” observed one education leader in a World Economic Forum report. “Children learn when they feel safe, when they feel cared for, and when they have a community of learning.”

AI tutors can adapt to a child's learning style in real time. They can provide feedback at midnight on a Saturday when no human teacher would be available. They can remember every mistake and track progress with precision no human memory could match. But they cannot inspire a love of learning in a struggling student. They cannot recognise when a child is dealing with problems at home that affect performance. They cannot model what it means to be curious, empathetic, and creative. While AI can automate some tasks, it cannot replace the human interaction and emotional support provided by teachers. There are legitimate concerns that over-reliance on AI could erode the teacher-student relationship and the social skills students develop in the classroom.

By combining the analytical power of AI with the irreplaceable human element of teaching, we can truly transform education for the next generation. Collaboration is the future. The most effective classrooms will combine human insight with AI precision, creating a hybrid model that supports personalised learning. With AI doing the busy work, teachers dedicate their time and energy to building confidence, nurturing creativity, and cultivating critical thinking skills in their students. This human touch and mentorship are invaluable and can never be fully replaced by AI.

The question is not whether AI will replace human teachers. The question is whether we will have the wisdom to use this technology in ways that enhance rather than diminish what makes education fundamentally human. As Sal Khan put it: “The question isn't whether AI will be part of education. It's how we use it responsibly to enhance learning.”

For now, the answer to that question remains in human hands.

---

References & Sources

1. Bloom, B. S. (1984). “The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring.” Educational Researcher, 13(6), 4-16. https://journals.sagepub.com/doi/10.3102/0013189X013006004
2. Kestin, G., & Miller, K. (2025). “AI Tutoring Outperforms Active Learning.” Scientific Reports. https://www.nature.com/articles/s41598-025-85814-z
3. RAND Corporation. Carnegie Learning Cognitive Tutor Study. https://www.carnegielearning.com/why-cl/research/
4. “A systematic review of AI-driven intelligent tutoring systems (ITS) in K-12 education.” npj Science of Learning. https://www.nature.com/articles/s41539-025-00320-7
5. “Will generative AI replace teachers in higher education? A study of teacher and student perceptions.” Teaching and Teacher Education. https://www.sciencedirect.com/science/article/abs/pii/S0191491X24000749
6. Institute of Education Sciences. Reports on Cognitive Tutor effectiveness. https://ies.ed.gov/
7. UNESCO. (2024). AI Competency Framework for Teachers. https://unesco-asp.dk/wp-content/uploads/2025/02/AI-Competency-framework-for-teachers_UNESCO_2024.pdf
8. UNESCO. (2025). “AI and education: protecting the rights of learners.” https://unesdoc.unesco.org/ark:/48223/pf0000395373
9. OECD. (2026). Digital Education Outlook 2026. https://www.oecd.org/en/publications/oecd-digital-education-outlook-2026_062a7394-en.html
10. OECD-European Commission. (2025). “Empowering Learners for the Age of AI: An AI Literacy Framework for Primary and Secondary Education.” https://oecdedutoday.com/new-ai-literacy-framework-to-equip-youth-in-an-age-of-ai/
11. White House Executive Order. (2025). “Advancing Artificial Intelligence Education for American Youth.”
12. Khan Academy Annual Report SY24-25. https://annualreport.khanacademy.org/
13. Walton Family Foundation & Gallup. (2025). “The AI Dividend: New Survey Shows AI Is Helping Teachers Reclaim Valuable Time.” https://www.waltonfamilyfoundation.org/the-ai-dividend-new-survey-shows-ai-is-helping-teachers-reclaim-valuable-time
14. Precedence Research. (2025). “AI in Education Market Size.” https://www.precedenceresearch.com/ai-in-education-market
15. Common Sense Media. (2025). “AI Companions Decoded.” https://www.commonsensemedia.org/ai-companions
16. EDUCAUSE. (2025). Survey of Higher Education Institutions and Students and Technology Report.
17. American Psychological Association. (2025). Health Advisory on AI Companion Software.
18. World Economic Forum. (2025). “How AI and human teachers can collaborate to transform education.” https://www.weforum.org/stories/2025/01/how-ai-and-human-teachers-can-collaborate-to-transform-education/
19. World Economic Forum. (2025). “AI is transforming education by allowing us to be more human.” https://www.weforum.org/stories/2025/12/ai-is-transforming-education-by-allowing-us-to-be-more-human/
20. Brookings Institution. (2025). “AI and the next digital divide in education.” https://www.brookings.edu/articles/ai-and-the-next-digital-divide-in-education/
21. Forbes. (2025). Student and teacher AI adoption statistics.
22. Oxford University Press. (2025). “The Future English Language Classroom: Human-Centred, Powered By AI.” https://teachingenglishwithoxford.oup.com/2025/06/23/future-english-language-classroom-ai/
23. Squirrel AI. (2024). Guinness World Record announcement. https://www.prnewswire.com/news-releases/squirrel-ai-learning-sets-guinness-world-record-for-the-most-users-to-take-an-online-mathematics-lesson-in-24-hours-302324623.html
24. Duolingo. (2025). Q2 2025 Financial Results and 2025 Language Report. https://blog.duolingo.com/2025-duolingo-language-report/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

The numbers are startling, and they demand attention. An estimated 795,000 Americans die or become permanently disabled each year because of diagnostic errors, according to a 2023 Johns Hopkins University study. In the United Kingdom, diagnostic errors affect at least 10 to 15 per cent of patients, with heart attack misdiagnosis rates reaching nearly 30 per cent in initial assessments. These are not abstract statistics. They represent people who trusted their doctors, sought help, and received the wrong answer at a critical moment.

Into this landscape of fallibility comes a promise wrapped in silicon and algorithms: artificial intelligence that can diagnose diseases faster, more accurately, and more consistently than human physicians. The question is no longer whether AI can perform this feat. Mounting evidence suggests it already can. The real question is whether you will trust a machine with your life, and what happens to the intimate relationship between doctor and patient when algorithms enter the examination room.

The Diagnostic Revolution Arrives

The pace of development has been breathtaking. In 2018, IDx-DR became the first fully autonomous AI diagnostic system in any medical field to receive approval from the United States Food and Drug Administration. The system, designed to detect diabetic retinopathy from retinal images, achieved a sensitivity of 87.4 per cent and specificity of 89.5 per cent in its pivotal clinical trial. A more recent systematic review and meta-analysis published in the American Journal of Ophthalmology found pooled sensitivity of 95 per cent and pooled specificity of 91 per cent. These numbers matter enormously. Diabetic retinopathy is a leading cause of blindness worldwide, and early detection can prevent irreversible vision loss. The algorithm does not tire, does not have off days, does not rush through appointments because another patient is waiting.

By December 2025, the FDA's database listed over 1,300 AI-enabled medical devices authorised for marketing. Radiology dominates, with more than 1,000 approved tools representing nearly 80 per cent of the total. The agency authorised 235 AI devices in 2024 alone, the most in its history. In the United Kingdom, the NHS has invested over 113 million pounds into more than 80 AI-driven innovations through its AI Lab, and AI now analyses acute stroke brain scans in 100 per cent of stroke units across England.

The performance data emerging from controlled studies is remarkable, though it requires careful interpretation. A March 2025 meta-analysis published in Nature's npj Digital Medicine, examining 83 studies, found that generative AI achieved an overall diagnostic accuracy of 52.1 per cent, with no significant difference between AI models and physicians overall. However, the picture becomes more interesting when we examine specific applications. Microsoft's AI diagnostic orchestrator correctly diagnosed 85 per cent of challenging cases from the New England Journal of Medicine, compared to approximately 20 per cent accuracy for the 21 general practice doctors who attempted the same cases. These were deliberately difficult diagnostic puzzles, the kind that stump even experienced clinicians.

In a 2024 randomised controlled trial at the University of Virginia Health System, ChatGPT Plus achieved a median diagnostic accuracy exceeding 92 per cent when used alone, while physicians using conventional approaches achieved 73.7 per cent. The researchers were surprised by an unexpected finding: adding a human physician to the AI actually reduced diagnostic accuracy, though it improved efficiency. The physicians often disagreed with or disregarded the AI's suggestions, sometimes to the detriment of diagnostic precision.

The Stanford Medicine study on AI in dermatology revealed that medical students, nurse practitioners, and primary care doctors improved their diagnostic accuracy by approximately 13 points in sensitivity and 11 points in specificity when using AI guidance. Even dermatologists and dermatology residents, who performed better overall, saw improvements with AI assistance. A systematic review comparing AI to clinicians in skin cancer detection found AI algorithms achieved sensitivity of 87 per cent and specificity of 77.1 per cent, compared to all clinicians at 79.78 per cent sensitivity and 73.6 per cent specificity. The differences were statistically significant.

In breast cancer screening, the evidence is mounting with remarkable consistency. The MASAI trial in Sweden, the world's first randomised controlled trial of AI-supported mammography screening, demonstrated that AI can increase cancer detection while reducing screen-reading workload. The German PRAIM trial, the largest study on integrating AI into mammography screening to date, found that AI-supported mammography detected breast cancer at a rate of 6.7 per 1,000 women screened, a 17.6 per cent increase over the standard double-reader approach at 5.7 per 1,000. A Lancet Digital Health commentary declared that standard double-reading of mammograms will likely be phased out from organised breast screening programmes if additional trials confirm these findings.

The Trust Paradox

Yet despite this evidence, something curious emerges from research into patient preferences. People do not straightforwardly embrace the diagnostic algorithm, even when presented with evidence of its superior performance.

A 2024 study published in Frontiers in Psychology analysed data from 1,183 participants presented with scenarios across cardiology, orthopaedics, dermatology, and psychiatry. The results were consistent across all four medical disciplines: people preferred a human doctor, followed by a human doctor working with an AI system, with AI alone coming in last place. A preregistered randomised survey experiment among 1,762 US participants found results consistent across age, gender, education, and political affiliation, indicating what researchers termed a “broad aversion to AI-assisted diagnosis.”

Research published in the Journal of the American Medical Informatics Association in 2025 found that patient expectations of AI improving their relationships with doctors were notably low at 19.55 per cent. Expectations that AI would improve healthcare access were comparatively higher but still modest at 30.28 per cent. Perhaps most revealing: trust in providers and the healthcare system was positively associated with expectations of AI benefit. Those who already trusted their doctors were more likely to embrace AI recommendations filtered through those doctors.

The trust dynamics are complex and sometimes contradictory. A cross-sectional vignette study published in the Journal of Medical Internet Research found that AI applications may have a potentially negative effect on the patient-physician relationship, especially among women and in high-risk situations. Trust in a doctor's personal integrity and professional competence emerged as key mediators of what researchers termed “AI-assistance aversion.” Lower trust in doctors who use AI directly reduced patients' intention to seek medical help at all.

Yet a contrasting survey from summer 2024 found 64 per cent of patients would trust a diagnosis made by AI over that of a human doctor, though trustworthiness decreased as healthcare issues became more complicated. Just 3 per cent said they were uncomfortable with any AI involvement in medicine. The contradiction reveals the importance of context, framing, and the specific clinical situation.

What explains these seemingly contradictory findings? Context matters enormously. The University of Arizona study that found patients almost evenly split (52.9 per cent chose human doctor, 47.1 per cent chose AI clinic) also discovered that a primary care physician's explanation about AI's superior accuracy, a gentle push towards AI, and a positive patient experience could significantly increase acceptance. How AI is introduced, who introduces it, and what the patient already believes about their healthcare provider all shape the response.

A Relationship Centuries in the Making

To understand what is at stake requires understanding what came before. The doctor-patient relationship is among the oldest professional bonds in human civilisation. Cave paintings representing healers date back fourteen thousand years. Before the secularisation of medicine brought by the Hippocratic school in the fifth century BCE, no clear boundaries existed between medicine, magic, and religion. The healer was often an extension of the priest, and seeking medical help meant placing yourself in the hands of someone who understood mysteries you could not fathom.

For most of medical history, this relationship was profoundly asymmetrical. The physician possessed knowledge that patients could not access or evaluate. Compliance was expected. The doctor decided, the patient accepted. This paternalistic model persisted well into the twentieth century. As one historical analysis noted, physicians were viewed as dominant or superior to patients due to the inherent power dynamic of controlling health, treatment, and access to knowledge. The physician conveyed only the information necessary to convince the patient of the proposed treatment course.

The shift came gradually but represented a fundamental reconception of the relationship. By the late twentieth century, the patient transformed from passive receiver of decisions into an agent with well-defined rights and broad capacity for autonomous decision-making. The doctor transformed from priestly father figure into technical adviser whose knowledge was offered but whose decisions were no longer taken for granted. Informed consent emerged as a legal and ethical requirement. Shared decision-making became the professional ideal.

Trust remained central throughout these transformations. Research consistently shows that trust, along with empathy, communication, and listening, characterises a productive doctor-patient relationship. For patients, a consistent relationship with their doctors has been shown to facilitate treatment adherence and improved health outcomes. The relationship itself is therapeutic.

But this trust has been eroding for decades. Public confidence in medicine peaked in the mid-1960s. A 2023 Gallup Poll found that only about one in three Americans expressed “great or quite a lot” of confidence in the medical system. Trust in doctors, though higher at roughly two in three Americans, remains below pre-pandemic levels. As one analysis observed, physicians' employers, pharmaceutical companies, and insurance companies have entered what was once a private relationship. The generic substitution of “healthcare provider” for “physician” and “client” for “patient” reflects a growing impersonality. Medicine has become commercialised, the encounter increasingly transactional.

Into this already complicated landscape arrives artificial intelligence, promising to further reshape what it means to receive medical care.

The Equity Reckoning

The introduction of AI into healthcare carries profound implications for equity, and not all of them are positive. The technology has the potential either to reduce or to amplify existing disparities, depending entirely on how it is developed and deployed.

A 2019 study sent shockwaves through the medical community when it revealed that a clinical algorithm used by many hospitals to decide which patients needed care showed significant racial bias. Black patients had to be deemed much sicker than white patients to be recommended for the same care. The algorithm had been trained on past healthcare spending data, which reflected a history in which Black patients had less to spend on their health compared to white patients. The algorithm learned to perpetuate that inequity.

The problem persists and may even be worsening as AI becomes more prevalent. A systematic review on AI-driven racial disparities in healthcare found a significant association between AI utilisation and the exacerbation of racial disparities, especially in minority populations including Black and Hispanic patients. Sources identified included biased training data, algorithm design choices, unfair deployment practices, and historic systemic inequities embedded in the healthcare system.

A Cedars-Sinai study found patterns of racial bias in treatment recommendations generated by leading AI platforms for psychiatric patients. Large language models, when presented with hypothetical clinical cases, often proposed different treatments for patients when African American identity was stated or implied than for patients whose race was not indicated. Specific disparities included LLMs omitting medication recommendations for ADHD cases when race was explicitly stated and suggesting guardianship for depression cases with explicit racial characteristics.

The sources of bias are multiple and often embedded in the foundational data that AI systems learn from. Public health AI typically suffers from historic bias, where prior injustices in access to care or discriminatory health policy become embedded within training datasets. Representation bias emerges when samples from urban, wealthy, or well-connected groups lead to the systematic exclusion of samples from rural, indigenous, or disenfranchised groups. Measurement bias occurs when health endpoints are approximated with proxy variables that differ between socioeconomic or cultural environments.

Research warns that minoritised communities, whose trust in health systems has been eroded by historical inequities, ongoing biases, and in some cases outright malevolence, are likely to approach AI with heightened scepticism. These communities have seen how systemic disparities can be perpetuated by the very tools meant to serve them.

Addressing these issues requires comprehensive bias detection tools and mitigation strategies, coupled with active supervision by physicians who understand the limitations of the systems they use. Mitigating algorithmic bias must occur across all stages of an algorithm's lifecycle, including authentic engagement with patients and communities during all phases, explicitly identifying healthcare algorithmic fairness issues and trade-offs, and ensuring accountability for equity and fairness in outcomes.

The Validation Gap

For all the impressive performance statistics emerging from research studies, a troubling pattern emerges upon closer examination of how AI diagnostic tools actually reach the market and enter clinical practice.

A cross-sectional study of 903 FDA-approved AI devices found that at the time of regulatory approval, clinical performance studies were reported for approximately half of the analysed devices. One quarter explicitly stated that no such studies had been conducted. Less than one third of clinical evaluations provided sex-specific data, and only one fourth addressed age-related subgroups. Perhaps most concerning: 97 per cent of all devices were cleared via the 510(k) pathway, which does not require independent clinical data demonstrating performance or safety. Devices are cleared based on their similarity to previously approved devices, creating a chain of approvals that may never have been anchored in rigorous clinical validation.

A JAMA Network Open study examining the generalisability of FDA-approved AI-enabled medical devices for clinical use warned that evidence about clinical generalisability is lacking. The number of AI-enabled tools cleared continues to rise, but the robust real-world validation that would inspire confidence often does not exist.

This matters because AI systems that perform brilliantly in controlled research settings may falter in the messy reality of clinical practice. The UVA Health researchers who found ChatGPT Plus achieving 92 per cent accuracy cautioned that the system “likely would fare less well in real life, where many other aspects of clinical reasoning come into play.” Determining downstream effects of diagnoses and treatment decisions involves complexities that current AI systems do not reliably navigate. A correct diagnosis is only the beginning; knowing what to do with it requires judgment that algorithms do not yet possess.

Studies have also found that most physicians treated AI tools like a search function, much as they would Google or UpToDate, rather than leveraging optimised prompting strategies that might improve performance. This suggests that even when AI tools are available, the human element of how they are used introduces significant variability that research settings often fail to capture.

What Machines Cannot Do

The argument for AI in diagnosis often centres on consistency and processing power. Algorithms do not forget, do not tire, do not bring personal problems to work. They can compare a patient's presentation against millions of cases instantly. They do not have fifteen-minute appointment slots that force rushed assessments.

But medicine is not merely pattern recognition. Eric Topol, Executive Vice-President of Scripps Research and author of Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again, has argued that AI development in healthcare could lead to a dramatic shift in the culture and practice of medicine. Yet he cautions that AI on its own will not fix the current challenges of what he terms “shallow medicine.” In his assessment, the field is “long on AI promise but very short on real-world, clinical proof of effectiveness.”

Topol envisions AI restoring the essential human element of medical practice by enabling machine support of tasks better suited for automation, thereby freeing doctors, nurses, and other healthcare professionals to focus on providing real care for patients. This is a fundamentally different vision from replacing physicians with algorithms. It imagines a symbiosis where each contributor does what it does best: the machine handles pattern recognition and data processing while the human provides judgment, empathy, and presence.

The obstacles to achieving this vision are substantial. Topol identifies medical community resistance to change, reimbursement issues, regulatory challenges, the need for greater transparency, the need for compelling evidence, engendering trust among clinicians and the public, and implementation challenges as chief barriers to progress. These are not merely technical problems but cultural and institutional ones.

Doctors must also contend with the downsides of AI adoption. Models can generate incorrect or misleading results, the phenomenon known as AI hallucinations or confabulations. AI models can produce results that reflect human bias encoded in training data. A diagnosis is not merely a label; it is a communication that affects how a person understands their body, their future, their mortality. Getting that communication wrong carries consequences that extend far beyond clinical metrics.

The Regulatory Response

Governments and regulatory bodies around the world are scrambling to keep pace with the technology, developing frameworks that balance innovation with safety.

In the United States, the FDA published guidance on “Transparency for Machine Learning-Enabled Medical Devices” in June 2024, followed by final guidance on predetermined change control plans for AI-enabled device software in December 2024. Draft guidance on lifecycle management for AI-enabled device software followed in January 2025. The FDA's Digital Health Advisory Committee held its inaugural meeting in November 2024 to discuss how the agency should adapt its regulatory approach for generative AI-enabled devices, which present novel challenges because they can produce outputs that even their creators cannot fully predict.

In the United Kingdom, the MHRA AI Airlock launched in May 2024 and expanded with a second cohort in 2025. This regulatory sandbox allows developers to test their AI as a Medical Device in supervised, real-world NHS environments. A new National Commission was announced to accelerate safe access to AI in healthcare by advising on a new regulatory framework to be published in 2026. The Commission brings together experts from technology companies including Google and Microsoft alongside clinicians, researchers, and patient advocates.

The NHS Fit For The Future: 10 Year Health Plan for England, published in July 2025, identified data, artificial intelligence, genomics, wearables, and robotics as five transformative technologies that are strategic priorities. A new framework procurement process will be introduced in 2026-2027 to allow NHS organisations to adopt innovative technologies including ambient AI.

The National Institute for Health and Care Excellence has conditionally recommended AI tools such as TechCare Alert and BoneView for NHS use in identifying fractures on X-rays, provided they are used alongside clinician review. This last phrase is crucial: alongside clinician review. The regulatory consensus, for now, maintains human oversight as a non-negotiable requirement.

The Nobel Prize and Its Implications

In October 2024, Demis Hassabis and John Jumper of Google DeepMind were co-awarded the Nobel Prize in Chemistry for their work on AlphaFold, alongside David Baker for his work on computational protein design. This recognition elevated AI in life sciences to the highest level of scientific honour, signalling that the technology has passed from speculative promise to demonstrated achievement.

AlphaFold has predicted over 200 million protein structures, nearly all catalogued proteins known to science. As of November 2025, it is being used by over 3 million researchers from over 190 countries, tackling problems including antimicrobial resistance, crop resilience, and heart disease. AlphaFold 3, announced in May 2024 and made publicly available in February 2025, can predict the structures of protein complexes with DNA, RNA, post-translational modifications, and selected ligands and ions. Google DeepMind reports a 50 per cent improvement in prediction accuracy compared to existing methods, effectively doubling what was previously possible.

The implications for drug discovery are substantial. Isomorphic Labs, the Google DeepMind spinout, raised 600 million dollars in March 2025 and is preparing to initiate clinical trials for AI-developed oncology drugs. Scientists at the company are collaborating with Eli Lilly and Novartis to discover antibodies and new treatments that inhibit disease-related targets. According to GlobalData's Drugs database, there are currently more than 3,000 drugs developed or repurposed using AI, with most in early stages of development.

Meanwhile, Med Gemini, Google DeepMind's medical AI platform, achieved 91.1 per cent accuracy on diagnostic tasks, outperforming prior models by 4.6 per cent. The system leverages deep learning to analyse medical images including X-rays and MRIs, helping in early detection of diseases including cancer, heart conditions, and neurological disorders.

In India, Google's bioacoustic AI model is enabling development of tools that can screen tuberculosis through cough sounds, with potential to screen 35 million people. AI is also working to close maternal health gaps by making ultrasounds accessible to midwives. These applications suggest that AI could expand access to diagnostic capabilities in resource-limited settings, potentially democratising healthcare in ways that human expertise alone could never achieve.

Hospitals Using AI Today

The integration is already happening, hospital by hospital, department by department. This is not a future scenario but present reality.

Pilot programmes at several Level I trauma centres report that AI-flagged X-rays get read 20 to 30 minutes faster on average than normal work-list order. In acute care, those minutes can be critical; in stroke treatment, every minute of delay costs brain cells. A multi-centre study in the UK identified that AI-assisted mammography had the potential to cut radiologists' workload by almost half without sacrificing diagnostic quality. Another trial in Canada demonstrated faster triage of suspected strokes when CT scans were pre-screened by AI, resulting in up to 30 minutes of saved treatment time.

A 2024 survey of physician sentiments revealed that at least two-thirds view AI as beneficial to their practice, with overall use cases increasing by nearly 70 per cent, particularly in medical documentation. The administrative burden of medicine is substantial: physicians spend more time on paperwork than on patients. AI that handles documentation potentially frees physicians for direct patient interaction, the very thing that drew many of them to medicine.

Thanks to the AI Diagnostic Fund in England, 50 per cent of hospital trusts are now deploying AI to help diagnose conditions including lung cancer. Research indicates that hospitals using AI-supported diagnostics have seen a 42 per cent reduction in diagnostic errors. If these figures hold at scale, the impact on patient outcomes could be transformative. Recall those 795,000 Americans harmed by diagnostic errors each year. Even modest improvements in diagnostic accuracy would translate to thousands of lives saved or changed.

The Question of the Self

Beyond the clinical metrics lies a deeper question about human experience. When you are ill, vulnerable, frightened, what do you need? What does healing require?

The paternalistic model of medicine assumed patients needed authority: someone who knew what to do and would do it. The patient-centred model assumed patients needed partnership: someone who would share information, discuss options, respect autonomy. Both models assumed a human on the other side of the relationship, someone capable of understanding what it means to suffer.

A 2025 randomised factorial experiment found that functionally, people trusted the diagnosis of human physicians more than medical AI or human-involved AI. But at the relational and emotional levels, there was no significant difference between human-AI and human-human interactions. This finding suggests something complicated about what patients actually experience versus what they believe they prefer. We may say we want a human, but we may respond to something else.

The psychiatric setting reveals particular tensions. The Frontiers in Psychology study found that the situation in psychiatry differed strongly from cardiology, orthopaedics, and dermatology, especially in the “human doctor with an AI system” condition. Mental health involves not just pattern recognition but the experience of being heard, validated, understood. Whether AI can participate meaningfully in that process remains deeply uncertain. A diagnosis of depression is not like a diagnosis of a fracture; it touches the core of selfhood.

Research on trust in AI-assisted health systems emphasises that trust is built differently in each relationship: between patients and providers, providers and technology, and institutions and their stakeholders. Trust is bidirectional; people must trust AI to perform reliably, while AI relies on the quality of human input. This circularity complicates simple narratives of replacement or enhancement.

Reimagining the Consultation

What might a transformed healthcare encounter look like in practice?

One possibility is the augmented physician: a doctor who arrives at your appointment having already reviewed an AI analysis of your symptoms, test results, and medical history. The AI has flagged potential diagnoses ranked by probability. The AI has identified questions the doctor should ask to differentiate between possibilities. The AI has checked for drug interactions, noted relevant recent research, compared your presentation to anonymised similar cases.

The doctor then spends your appointment actually talking to you. Understanding your concerns. Explaining options. Answering questions. Making eye contact. The administrative and analytical burden has shifted to the machine; the human connection remains with the human.

This vision aligns with Topol's argument in Deep Medicine. The title itself is instructive: the promise is not that AI will make healthcare mechanical but that it might make healthcare human again. Fifteen-minute appointments driven by documentation requirements represent a form of dehumanisation that preceded AI. If algorithms absorb the documentation burden, perhaps doctors can rediscover the relationship that drew many of them to medicine in the first place.

But this optimistic scenario requires deliberate design choices. If AI primarily serves cost-cutting, if healthcare administrators use diagnostic algorithms to reduce physician staffing, if the efficiency gains flow to shareholders rather than patient care, the technology will deepen rather than heal medicine's wounds.

The Coming Transformation

The trajectory is set, though the destination remains uncertain.

The NHS Healthcare AI Solutions agreement, expected to be worth 180 million pounds, is forecast to open for bids in summer 2025 and go live in 2026. The UCLA-led PRISM Trial, the first major randomised trial of AI in breast cancer screening in the United States, is underway with 16 million dollars in funding. Clinical trials for AI-designed drugs from Isomorphic Labs are imminent.

Meanwhile, the fundamental questions persist. Will patients trust algorithms with their lives? The evidence suggests: sometimes, depending on context, depending on how the technology is presented, depending on who is doing the presenting. Trust in providers and the healthcare system is positively associated with expectations of AI benefit. Those who already trust their doctors are more likely to trust AI recommendations filtered through those doctors.

Will the doctor-patient relationship survive this transformation? The relationship has survived extraordinary changes before: the rise of specialisation, the introduction of evidence-based medicine, the intrusion of insurance companies and electronic health records. Each change reshaped but did not extinguish the fundamental bond between someone who is suffering and someone who can help.

The machines are faster. They may well be more accurate, at least for certain diagnostic tasks. They do not tire, do not forget, do not have personal problems. But they also do not care, not in any meaningful sense. They do not sit with you in your fear. They do not hold your hand while delivering difficult news. They do not remember that your mother died of the same disease and understand why this diagnosis terrifies you.

Perhaps the answer is not trust in machines or trust in humans but trust in a system where each contributes what it does best. The algorithm analyses the scan. The doctor explains what the analysis means for your life. The algorithm flags the drug interaction. The doctor discusses whether the benefit outweighs the risk. The algorithm never forgets a detail. The doctor never forgets you are a person.

This synthesis requires more than technological development. It requires deliberate choices about healthcare systems, medical education, regulatory frameworks, and reimbursement structures. It requires confronting the biases encoded in training data and the inequities they can perpetuate. It requires maintaining human oversight even when algorithms outperform humans on specific metrics. It requires remembering that a diagnosis is not just an output but a communication that changes someone's understanding of their own existence.

The algorithm can see you now. Whether you will trust it, and whether that trust is warranted, depends on decisions being made in research laboratories, regulatory agencies, hospital boardrooms, and government ministries around the world. The doctor-patient relationship that has defined healthcare for centuries is being renegotiated. The outcome will shape medicine for the centuries to come.

---

References and Sources

1. Newman-Toker, D.E. et al. (2023). “Burden of serious harms from diagnostic error in the USA.” BMJ Quality & Safety. Johns Hopkins Armstrong Institute Center for Diagnostic Excellence. https://pubmed.ncbi.nlm.nih.gov/37460118/

2. Takita, H. et al. (2025). “A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physicians.” npj Digital Medicine, 8(175). https://www.nature.com/articles/s41746-025-01543-z

3. Parsons, A.S. et al. (2024). “Does AI Improve Doctors' Diagnoses?” Randomised controlled trial, UVA Health. JAMA Network Open. https://newsroom.uvahealth.com/2024/11/13/does-ai-improve-doctors-diagnoses-study-finds-out/

4. FDA. (2024-2025). Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices database. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices

5. IDx-DR De Novo Classification (DEN180001). (2018). FDA regulatory submission for autonomous AI diabetic retinopathy detection. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/denovo.cfm?id=DEN180001

6. Kim, J. et al. (2024). “Human-AI interaction in skin cancer diagnosis: a systematic review and meta-analysis.” npj Digital Medicine. Stanford Medicine. https://www.nature.com/articles/s41746-024-01031-w

7. Lång, K. et al. (2025). “Screening performance and characteristics of breast cancer detected in the Mammography Screening with Artificial Intelligence trial (MASAI).” The Lancet Digital Health, 7(3), e175-e183. https://www.thelancet.com/journals/landig/article/PIIS2589-7500(24)00267-X/fulltext

8. Riedl, R., Hogeterp, S.A. & Reuter, M. (2024). “Do patients prefer a human doctor, artificial intelligence, or a blend, and is this preference dependent on medical discipline?” Frontiers in Psychology, 15. https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2024.1422177/full

9. Zondag, A.G.M. et al. (2024). “The Effect of Artificial Intelligence on Patient-Physician Trust: Cross-Sectional Vignette Study.” Journal of Medical Internet Research, 26, e50853. https://www.jmir.org/2024/1/e50853

10. Nong, P. & Ji, M. (2025). “Expectations of healthcare AI and the role of trust: understanding patient views on how AI will impact cost, access, and patient-provider relationships.” Journal of the American Medical Informatics Association, 32(5), 795-799. https://academic.oup.com/jamia/article/32/5/795/8046745

11. Obermeyer, Z. et al. (2019). “Dissecting racial bias in an algorithm used to manage the health of populations.” Science, 366(6464), 447-453. https://www.science.org/doi/10.1126/science.aax2342

12. Aboujaoude, E. et al. (2025). “Racial bias in AI-mediated psychiatric diagnosis and treatment: a qualitative comparison of four large language models.” npj Digital Medicine. Cedars-Sinai. https://www.cedars-sinai.org/newsroom/cedars-sinai-study-shows-racial-bias-in-ai-generated-treatment-regimens-for-psychiatric-patients/

13. Windecker, D. et al. (2025). “Generalizability of FDA-Approved AI-Enabled Medical Devices for Clinical Use.” JAMA Network Open, 8(4), e258052. https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2833324

14. Topol, E.J. (2019). Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again. Basic Books. https://drerictopol.com/portfolio/deep-medicine/

15. NHS England. (2024-2025). NHS AI Lab investments and implementation reports. https://www.gov.uk/government/news/health-secretary-announces-250-million-investment-in-artificial-intelligence

16. GOV.UK. (2025). “New Commission to help accelerate NHS use of AI.” https://www.gov.uk/government/news/new-commission-to-help-accelerate-nhs-use-of-ai

17. Department of Health and Social Care. (2025). “Fit For The Future: 10 Year Health Plan for England.” https://www.gov.uk/government/publications/10-year-health-plan-for-england-fit-for-the-future

18. Nobel Prize Committee. (2024). “The Nobel Prize in Chemistry 2024” — Hassabis, Jumper (AlphaFold) and Baker. https://www.nobelprize.org/prizes/chemistry/2024/press-release/

19. Truog, R.D. (2012). “Patients and Doctors — The Evolution of a Relationship.” New England Journal of Medicine, 366(7), 581-585. https://www.nejm.org/doi/full/10.1056/nejmp1110848

20. Gallup. (2023). “Confidence in U.S. Institutions Down; Average at New Low.” https://news.gallup.com/poll/394283/confidence-institutions-down-average-new-low.aspx

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

In December 2025, something remarkable happened in the fractious world of artificial intelligence. Anthropic, OpenAI, Google, Microsoft, and a constellation of other technology giants announced they were joining forces under the Linux Foundation to create the Agentic AI Foundation. The initiative would consolidate three competing protocols into a neutral consortium: Anthropic's Model Context Protocol, Block's Goose agent framework, and OpenAI's AGENTS.md convention. After years of proprietary warfare, the industry appeared to be converging on shared infrastructure for the age of autonomous software agents.

The timing could not have been more significant. According to the Linux Foundation announcement, MCP server downloads had grown from roughly 100,000 in November 2024 to over 8 million by April 2025. The ecosystem now boasts over 5,800 MCP servers and 300 MCP clients, with major deployments at Block, Bloomberg, Amazon, and hundreds of Fortune 500 companies. RedMonk analysts described MCP's adoption curve as reminiscent of Docker's rapid market saturation, the fastest standard uptake the firm had ever observed.

Yet beneath this apparent unity lies a troubling question that few in the industry seem willing to confront directly. What happens when you standardise the plumbing before you fully understand what will flow through it? What if the orchestration patterns being cemented into protocol specifications today prove fundamentally misaligned with the reasoning capabilities that will emerge tomorrow?

The history of technology is littered with standards that seemed essential at the time but later constrained innovation in ways their creators never anticipated. The OSI networking model, Ada programming language, and countless other well-intentioned standardisation efforts demonstrate how premature consensus can lock entire ecosystems into architectural choices that later prove suboptimal. As one researcher noted in a University of Michigan analysis, standardisation increases technological efficiency but can also prolong existing technologies to an excessive degree by inhibiting investments in novel developments.

The stakes in the agentic AI standardisation race are considerably higher than previous technology transitions. We are not merely deciding how software components communicate. We are potentially determining the architectural assumptions that will govern how artificial intelligence decomposes problems, executes autonomous tasks, and integrates with human workflows for decades to come.

The Competitive Logic Driving Convergence

To understand why the industry is rushing toward standardisation, one must first appreciate the economic pressures that have made fragmented agentic infrastructure increasingly untenable. The current landscape resembles the early days of mobile computing, when every manufacturer implemented its own charging connector and data protocol. Developers building agentic applications face a bewildering array of frameworks, each with its own conventions for tool integration, memory management, and inter-agent communication.

The numbers tell a compelling story. Gartner reported a staggering 1,445% surge in multi-agent system inquiries from the first quarter of 2024 to the second quarter of 2025. Industry analysts project the agentic AI market will surge from 7.8 billion dollars today to over 52 billion dollars by 2030. Gartner further predicts that 40% of enterprise applications will embed AI agents by the end of 2026, up from less than 5% in 2025.

This explosive growth has created intense pressure for interoperability. When Google announced its Agent2Agent protocol in April 2025, it launched with support from more than 50 technology partners including Atlassian, Box, Cohere, Intuit, Langchain, MongoDB, PayPal, Salesforce, SAP, ServiceNow, and Workday. The protocol was designed to enable agents built by different vendors to discover each other, negotiate capabilities, and coordinate actions across enterprise environments.

The competitive dynamics are straightforward. If the Agentic AI Foundation's standards become dominant, companies that previously held APIs hostage will be pressured to interoperate. Google and Microsoft could find it increasingly necessary to support MCP and AGENTS.md generically, lest customers demand cross-platform agents. The open ecosystem effectively buys customers choice, giving a competitive advantage to adherence.

Yet this race toward consensus obscures a fundamental tension. The Model Context Protocol was designed primarily to solve the problem of connecting AI systems to external tools and data sources. As Anthropic's original announcement explained, even the most sophisticated models are constrained by their isolation from data, trapped behind information silos and legacy systems. MCP provides a universal interface for reading files, executing functions, and handling contextual prompts. Think of it as USB-C for AI applications.

But USB-C was standardised after decades of experience with peripheral connectivity. The fundamental patterns for how humans interact with external devices were well understood. The same cannot be said for agentic AI. The field is evolving so rapidly that the orchestration patterns appropriate for today's language models may prove entirely inadequate for the reasoning systems emerging over the next several years.

When Reasoning Changes Everything

The reasoning model revolution of 2024 and 2025 has fundamentally altered how software engineering tasks can be decomposed and executed. OpenAI's o3, Google's Gemini 3 with Deep Think mode, and DeepSeek's R1 represent a qualitative shift in capability that extends far beyond incremental improvements in benchmark scores.

The pace of advancement has been staggering. In November 2025, Google introduced Gemini 3, positioning it as its most capable system to date, deployed from day one across Search, the Gemini app, AI Studio, Vertex AI, and the Gemini CLI. Gemini 3 Pro scores 1501 Elo on LMArena, achieving top leaderboard position, alongside 91.9% on GPQA Diamond and 76.2% on SWE-bench Verified for real-world software engineering tasks. The Deep Think mode pushes scientific reasoning benchmarks into the low to mid nineties, placing Gemini 3 at the front of late 2025 capabilities. By December 2025, Google was processing over one trillion tokens per day through its API.

Consider the broader transformation in software development. OpenAI reports that GPT-5 scores 74.9% on SWE-bench Verified compared to 69.1% for o3. On Aider polyglot, an evaluation of code editing, GPT-5 achieves 88%, representing a one-third reduction in error rate compared to o3. DeepSeek's R1 demonstrated that reasoning abilities can be incentivised through pure reinforcement learning, obviating the need for human-labelled reasoning trajectories. The company's research shows that such training facilitates the emergent development of advanced reasoning patterns including self-verification, reflection, and dynamic strategy adaptation. DeepSeek is now preparing to launch a fully autonomous AI agent by late 2025, signalling a shift from chatbots to practical, real-world agentic AI.

These capabilities demand fundamentally different decomposition strategies than the tool-calling patterns embedded in current protocols. A reasoning model that can plan multi-step tasks, execute on them, and continue to reason about results to update its plans represents a different computational paradigm than a model that simply calls predefined functions in response to user prompts.

The 2025 DORA Report captures this transformation in stark terms. AI adoption is near-universal, with 90% of survey respondents reporting they use AI at work. More than 80% believe it has increased their productivity. Yet AI adoption continues to have a negative relationship with software delivery stability. The researchers estimate that between two people who share the same traits, environment, and processes, the person with higher AI adoption will report higher levels of individual effectiveness but also higher levels of software delivery instability.

This productivity-stability paradox suggests that current development practices are struggling to accommodate the new capabilities. The DORA team found that AI coding assistants dramatically boost individual output, with 21% more tasks completed and 98% more pull requests merged, but organisational delivery metrics remain flat. Speed without stability, as the researchers concluded, is accelerated chaos.

The Lock-In Mechanism

The danger of premature standardisation lies not in the protocols themselves but in the architectural assumptions they embed. When developers build applications around specific orchestration patterns, those patterns become load-bearing infrastructure that cannot easily be replaced.

Microsoft's October 2025 decision to merge AutoGen with Semantic Kernel into a unified Microsoft Agent Framework illustrates both the problem and the attempted solution. The company recognised that framework fragmentation was creating confusion among developers, with multiple competing options each requiring different approaches to agent construction. General availability is set for the first quarter of 2026, with production service level agreements, multi-language support, and deep Azure integration.

Yet this consolidation also demonstrates how quickly architectural choices become entrenched. As one analysis noted, current agent frameworks are fragmented and lack enterprise features like observability, compliance, and durability. The push toward standardisation aims to address these gaps, but in doing so it may cement assumptions about how agents should be structured that prove limiting when new capabilities emerge.

The historical parallel to the OSI versus Internet protocols debate is instructive. Several central actors within OSI and Internet standardisation suggested that OSI's failure stemmed from being installed-base-hostile. The OSI protocols were not closely enough related to the already installed base of communication systems. The installed base is irreversible in the sense that radical, abrupt change of the kind implicitly assumed by OSI developers is highly unlikely.

The same irreversibility threatens agentic AI. Once thousands of enterprise applications embed MCP clients and servers, once development teams organise their workflows around specific orchestration patterns, the switching costs become prohibitive. Even if superior approaches emerge, the installed base may prevent their adoption.

Four major protocols have already emerged to handle agent communication: Model Context Protocol, Agent Communication Protocol, Agent-to-Agent Protocol, and Agent Network Protocol. Google's A2A Protocol alone has backing from over 50 companies including Microsoft and Salesforce. Yet as of September 2025, A2A development has slowed significantly, and most of the AI agent ecosystem has consolidated around MCP. Google Cloud still supports A2A for some enterprise customers, but the company has started adding MCP compatibility to its AI services. This represents a tacit acknowledgment that the developer community has chosen.

The Junior Developer Crisis

The technical standardisation debate unfolds against the backdrop of a more immediate crisis in the software development workforce. The rapid adoption of AI coding assistants has fundamentally disrupted the traditional career ladder for software engineers, with consequences that may prove more damaging than any technical limitation.

According to data from the U.S. Bureau of Labor Statistics, overall programmer employment fell a dramatic 27.5% between 2023 and 2025. A Stanford Digital Economy Study found that by July 2025, employment for software developers aged 22-25 had declined nearly 20% from its peak in late 2022. Across major U.S. technology companies, graduate hiring has dropped more than 50% compared to pre-2020 levels. In the UK, junior developer openings are down by nearly one-third since 2022.

The economics driving this shift are brutally simple. As one senior software engineer quoted by CIO observed, companies are asking why they should hire a junior developer for 90,000 dollars when GitHub Copilot costs 10 dollars. Many of the tasks once assigned to junior developers, including generating boilerplate code, writing unit tests, and maintaining APIs, are now reliably managed by AI assistants.

Industry analyst Vernon Keenan describes a quiet erosion of entry-level positions that will lead to a decline in foundational roles, a loss of mentorship opportunities, and barriers to skill development. Anthropic CEO Dario Amodei has warned that entry-level jobs are squarely in the crosshairs of automation. Salesforce CEO Marc Benioff announced the company would stop hiring new software engineers in 2025, citing AI-driven productivity gains.

The 2025 Stack Overflow Developer Survey captures the resulting tension. While 84% of developers now use or plan to use AI tools, trust has declined sharply. Only 33% of developers trust the accuracy of AI tools, while 46% actively distrust it. A mere 3% report highly trusting the output. The biggest frustration, cited by 66% of developers, is dealing with AI solutions that are almost right but not quite.

This trust deficit reflects a deeper problem. Experienced developers understand the limitations of AI-generated code but have the expertise to verify and correct it. Junior developers lack this foundation. There is sentiment that AI has made junior developers less competent, with some losing foundational skills that make for successful entry-level employees. Without proper mentorship, junior developers risk over-relying on AI.

The long-term implications are stark. The biggest challenge will be training the next generation of software architects. With fewer junior developer jobs, there will not be a natural apprenticeship to more senior roles. We risk creating a generation of developers who can prompt AI systems but cannot understand or debug the code those systems produce.

Architectural Decisions Migrate to Prompt Design

As reasoning models assume greater responsibility for code generation and system design, the locus of architectural decision-making is shifting in ways that current organisational structures are poorly equipped to handle. Prompt engineering is evolving from a novelty skill into a core architectural discipline.

The way we communicate with AI has shifted from simple trial-and-error prompts to something much more strategic, what researchers describe as prompt design as a discipline. If 2024 was about understanding the grammar of prompts, 2025 is about learning to design blueprints. Just as software architects do not just write code but design systems, prompt architects do not just write clever sentences. They shape conversations into repeatable frameworks that unlock intelligence, creativity, and precision.

The adoption statistics reflect this shift. According to the 2025 AI-Enablement Benchmark Report, the design and architecture phase of the software development lifecycle has an AI adoption rate of 52%. Teams using AI tools for design and architecture have seen a 28% increase in design iteration speed.

Yet this concentration of architectural power in prompt design creates new risks. Context engineering, as one CIO analysis describes it, is an architectural shift in how AI systems are built. Early generative AI was stateless, handling isolated interactions where prompt engineering was sufficient. Autonomous agents are fundamentally different. They persist across multiple interactions, make sequential decisions, and operate with varying levels of human oversight.

This shift demands collaboration between data engineering, enterprise architecture, security, and those who understand processes and strategy. A strong data foundation, not just prompt design, determines how well an agent performs. Agents need engineering, not just prompts.

The danger lies in concentrating too much decision-making authority in the hands of those who understand prompt patterns but lack deep domain expertise. Software architecture is not about finding a single correct answer. It is about navigating competing constraints, making tradeoffs, and defending reasoning. AI models can help reason through tradeoffs, generate architectural decision records, or compare tools, but only if prompted by someone who understands the domain deeply enough to ask the right questions.

The governance implications are significant. According to IAPP research, 50% of AI governance professionals are typically assigned to ethics, compliance, privacy, or legal teams. Yet traditional AI governance practices may not suffice with agentic systems. Governing agentic systems requires addressing their autonomy and dynamic behaviour in ways that current organisational structures are not designed to handle.

Fragmentation Across Model Families

The proliferation of reasoning models with different capabilities and cost profiles is creating a new form of fragmentation that threatens to balkanise development practices. Different teams within the same organisation may adopt different model families based on their specific requirements, leading to incompatible workflows and siloed expertise.

The ARC Prize Foundation's extensive testing of reasoning systems reached a striking conclusion: there is no clear winner. Different models excel at different tasks, and the optimal choice depends heavily on specific requirements around accuracy, cost, and latency. OpenAI's o3-medium and o3-high offer the highest accuracy while sacrificing cost and time. Google's Gemini 3 Flash, released in December 2025, delivers frontier-class performance at less than a quarter of the cost of Gemini 3 Pro, with pricing of 0.50 dollars per million input tokens compared to significantly higher rates for comparable models. DeepSeek offers an aggressive pricing structure with input costs as low as 0.07 dollars per million tokens.

For enterprises focused on return on investment, these tradeoffs matter enormously. The 2025 State of AI report notes that trade-offs remain, with long contexts raising latency and cost. Because different providers trust or cherry-pick different benchmarks, it has become more difficult to evaluate agents' performance. Choosing the right agent for a particular task remains a challenge.

This complexity is driving teams toward specialisation around particular model families. Some organisations standardise on OpenAI's ecosystem for its integration with popular development tools. Others prefer Google's offerings for their multimodal capabilities and long context windows of up to 1,048,576 tokens. Still others adopt DeepSeek's open models for cost control or air-gapped deployments.

The result is a fragmentation of development practices that cuts across traditional organisational boundaries. A team building customer-facing agents may use entirely different tools and patterns than a team building internal automation. Knowledge transfer becomes difficult. Best practices diverge. The organisational learning that should flow from widespread AI adoption becomes trapped in silos.

The 2025 DORA Report identifies platform engineering as a crucial foundation for unlocking AI value, with 90% of organisations having adopted at least one platform. There is a direct correlation between high-quality internal platforms and an organisation's ability to unlock the value of AI. Yet building such platforms requires making architectural choices that may lock organisations into specific model families and orchestration patterns.

The Technical Debt Acceleration

The rapid adoption of AI coding assistants has created what may be the fastest accumulation of technical debt in the history of software development. Code that works today may prove impossible to maintain tomorrow, creating hidden liabilities that will compound over time.

Forrester predicts that by 2025, more than 50% of technology decision-makers will face moderate to severe technical debt, with that number expected to hit 75% by 2026. Technical debt costs over 2.41 trillion dollars annually in the United States alone. The State of Software Delivery 2025 report by Harness found that the majority of developers spend more time debugging AI-generated code and more time resolving security vulnerabilities than before AI adoption.

The mechanisms driving this debt accumulation are distinctive. According to one analysis, there are three main vectors that generate AI technical debt: model versioning chaos, code generation bloat, and organisation fragmentation. These vectors, coupled with the speed of AI code generation, interact to cause exponential growth.

Code churn, defined as code that is added and then quickly modified or deleted, is projected to hit nearly 7% by 2025. This represents a red flag for instability and rework. As API evangelist Kin Lane observed, he has not seen so much technical debt being created in such a short period during his 35-year career in technology.

The security implications are equally concerning. A report from Ox Security titled Army of Juniors: The AI Code Security Crisis found that AI-generated code is highly functional but systematically lacking in architectural judgment. The Google 2024 DORA report found a trade-off between gains and losses with AI, where a 25% increase in AI usage quickens code reviews and benefits documentation but results in a 7.2% decrease in delivery stability.

The widening gap between organisations with clean codebases and those burdened by legacy systems creates additional stratification. Generative AI dramatically widens the gap in velocity between low-debt coding and high-debt coding. Companies with relatively young, high-quality codebases benefit the most from generative AI tools, while companies with gnarly, legacy codebases struggle to adopt them. The penalty for having a high-debt codebase is now larger than ever.

Research Structures for Anticipating Second-Order Effects

Navigating the transition to reasoning-capable autonomous systems requires organisational and research structures that most institutions currently lack. The rapid pace of change demands new approaches to technology assessment, workforce development, and institutional coordination.

The World Economic Forum estimates that 40% of today's workers will need major skill updates by 2030, and in information technology that number is likely even higher. Yet the traditional mechanisms for workforce development are poorly suited to a technology that evolves faster than educational curricula can adapt.

Several research priorities emerge from this analysis. First, longitudinal studies tracking the career trajectories of software developers across the AI transition would provide crucial data for workforce planning. The Stanford Digital Economy Study demonstrates the value of such research, but more granular analysis is needed to understand which skills remain valuable, which become obsolete, and how career paths are being restructured.

Second, technical research into the interaction between standardisation and innovation in agentic systems could inform policy decisions about when and how to pursue consensus. The historical literature on standards competition provides useful frameworks, but the unique characteristics of AI systems, including their rapid capability growth and opaque decision-making, may require new analytical approaches.

Third, organisational research examining how different governance structures affect AI adoption outcomes could help enterprises design more effective oversight mechanisms. The DORA team's finding that AI amplifies existing organisational capabilities, making strong teams stronger and struggling teams worse, suggests that the organisational context matters as much as the technology itself.

Fourth, security research focused specifically on the interaction between AI code generation and vulnerability introduction could help establish appropriate safeguards. The current pattern of generating functional but architecturally flawed code suggests fundamental limitations in how models understand system-level concerns.

Finally, educational research into how programming pedagogy should adapt to AI assistance could prevent the worst outcomes of skill atrophy. If junior developers are to learn effectively in an environment where AI handles routine tasks, new teaching approaches will be needed that focus on the higher-order skills that remain uniquely human.

Building Resilient Development Practices

The confluence of standardisation pressures, reasoning model capabilities, workforce disruption, and technical debt accumulation creates a landscape that demands new approaches to software development practice. Organisations that thrive will be those that build resilience into their development processes rather than optimising purely for speed.

Several principles emerge from this analysis. First, maintain architectural optionality. Avoid deep dependencies on specific orchestration patterns that may prove limiting as capabilities evolve. Design systems with clear abstraction boundaries that allow components to be replaced as better approaches emerge.

Second, invest in human capability alongside AI tooling. The organisations that will navigate this transition successfully are those that continue developing deep technical expertise in their workforce, not those that assume AI will substitute for human understanding.

Third, measure what matters. The DORA framework's addition of rework rate as a fifth core metric reflects the recognition that traditional velocity measures miss crucial dimensions of software quality. Organisations should develop measurement systems that capture the long-term health of their codebases and development practices.

Fourth, build bridges across model families. Rather than standardising on a single AI ecosystem, develop the institutional capability to work effectively across multiple model families. This requires investment in training, tooling, and organisational learning that most enterprises have not yet made.

Fifth, participate in standards development. The architectural choices being made in protocol specifications today will shape the development landscape for years to come. Organisations with strong opinions about how agentic systems should work have an opportunity to influence those specifications before they become locked in.

The transition to reasoning-capable autonomous systems represents both an enormous opportunity and a significant risk. The opportunity lies in the productivity gains that well-deployed AI can provide. The risk lies in the second-order effects that poorly managed deployment can create. The difference between these outcomes will be determined not by the capabilities of the AI systems themselves but by the organisational wisdom with which they are deployed.

The Protocols That Will Shape Tomorrow

The agentic AI standardisation race presents a familiar tension in new form. The industry needs common infrastructure to enable interoperability and reduce fragmentation. Yet premature consensus risks locking in architectural assumptions that may prove fundamentally limiting.

The Model Context Protocol's rapid adoption demonstrates both the hunger for standardisation and the danger of premature lock-in. MCP achieved in one year what many standards take a decade to accomplish: genuine industry-wide adoption and governance transition to a neutral foundation. Yet the protocol was designed for a particular model of AI capability, one where agents primarily call tools and retrieve context. The reasoning models now emerging may demand entirely different decomposition strategies.

Meta's notable absence from the Agentic AI Foundation hints at alternative futures. Almost every major agentic player from Google to AWS to Microsoft has joined, but Meta has not signed on and published reports indicate it will not be joining soon. The company is reportedly shifting toward a proprietary strategy centred on a new revenue-generating model. Whether this represents a mistake or a prescient bet on different architectural approaches remains to be seen.

The historical pattern suggests that the standards which endure are those designed with sufficient flexibility to accommodate unforeseen developments. The Internet protocols succeeded where OSI failed in part because they were more tolerant of variation and evolution. The question for agentic AI is whether current standardisation efforts embed similar flexibility or whether they will constrain the systems of tomorrow to the architectural assumptions of today.

For developers, enterprises, and policymakers navigating this landscape, the imperative is to engage critically with standardisation rather than accepting it passively. The architectural choices being made now will shape the capabilities and limitations of agentic systems for years to come. Those who understand both the opportunities and the risks of premature consensus will be better positioned to influence the outcome.

The reasoning revolution is just beginning. The protocols and patterns that emerge from this moment will determine whether artificial intelligence amplifies human capability or merely accelerates the accumulation of technical debt and workforce disruption. The standards race matters, but the wisdom with which we run it matters more.

---

References and Sources

• Linux Foundation (2025). “Linux Foundation Announces the Formation of the Agentic AI Foundation.” https://www.linuxfoundation.org/press/linux-foundation-announces-the-formation-of-the-agentic-ai-foundation

• Anthropic (2024). “Introducing the Model Context Protocol.” https://www.anthropic.com/news/model-context-protocol

• Anthropic (2025). “Donating the Model Context Protocol and Establishing the Agentic AI Foundation.” https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation

• Pento (2025). “A Year of MCP: From Internal Experiment to Industry Standard.” https://www.pento.ai/blog/a-year-of-mcp-2025-review

• University of Michigan (n.d.). “Why Standardization Efforts Fail.” https://quod.lib.umich.edu/cgi/t/text/idx/j/jep/3336451.0014.103/--why-standardization-efforts-fail

• InfoQ (n.d.). “Standards are Great, but Standardisation is a Really Bad Idea.” https://www.infoq.com/presentations/downey-standards-great-standardization-bad/

• Google DORA (2025). “State of AI-assisted Software Development 2025.” https://dora.dev/research/2025/dora-report/

• OpenAI (2025). “Introducing GPT-5 for Developers.” https://openai.com/index/introducing-gpt-5-for-developers/

• Google (2025). “Gemini 3: News and Announcements.” https://blog.google/products/gemini/gemini-3-collection/

• Google (2025). “Introducing Gemini 3 Flash: Benchmarks, Global Availability.” https://blog.google/products/gemini/gemini-3-flash/

• DeepSeek (2025). “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” https://arxiv.org/html/2501.12948v1

• Google Developers Blog (2025). “Announcing the Agent2Agent Protocol (A2A).” https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/

• Stack Overflow (2025). “2025 Stack Overflow Developer Survey.” https://survey.stackoverflow.co/2025/

• CIO (2025). “Demand for Junior Developers Softens as AI Takes Over.” https://www.cio.com/article/4062024/demand-for-junior-developers-softens-as-ai-takes-over.html

• Stack Overflow Blog (2025). “AI vs Gen Z: How AI Has Changed the Career Pathway for Junior Developers.” https://stackoverflow.blog/2025/12/26/ai-vs-gen-z/

• IEEE Spectrum (2025). “AI Shifts Expectations for Entry Level Jobs.” https://spectrum.ieee.org/ai-effect-entry-level-jobs

• Understanding AI (2025). “New Evidence Strongly Suggests AI Is Killing Jobs for Young Programmers.” https://www.understandingai.org/p/new-evidence-strongly-suggest-ai

• CIO (2025). “Context Engineering: Improving AI by Moving Beyond the Prompt.” https://www.cio.com/article/4080592/context-engineering-improving-ai-by-moving-beyond-the-prompt.html

• IAPP (2025). “AI Governance Profession Report 2025.” https://iapp.org/resources/article/ai-governance-profession-report

• Machine Learning Mastery (2025). “7 Agentic AI Trends to Watch in 2026.” https://machinelearningmastery.com/7-agentic-ai-trends-to-watch-in-2026/

• ARC Prize (2025). “We Tested Every Major AI Reasoning System. There Is No Clear Winner.” https://arcprize.org/blog/which-ai-reasoning-model-is-best

• InfoQ (2025). “AI-Generated Code Creates New Wave of Technical Debt, Report Finds.” https://www.infoq.com/news/2025/11/ai-code-technical-debt/

• LeadDev (2025). “How AI Generated Code Compounds Technical Debt.” https://leaddev.com/technical-direction/how-ai-generated-code-accelerates-technical-debt

• IT Revolution (2025). “AI's Mirror Effect: How the 2025 DORA Report Reveals Your Organization's True Capabilities.” https://itrevolution.com/articles/ais-mirror-effect-how-the-2025-dora-report-reveals-your-organizations-true-capabilities/

• RedMonk (2025). “DORA 2025: Measuring Software Delivery After AI.” https://redmonk.com/rstephens/2025/12/18/dora2025/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

In May 2024, something unprecedented appeared on screens across Central Asia. A 52-second video in Pashto featured a news anchor calmly claiming responsibility for a terrorist attack in Bamiyan, Afghanistan. The anchor looked local, spoke fluently, and delivered the message with professional composure. There was just one problem: the anchor did not exist. The Islamic State Khorasan Province (ISKP) had produced its first AI-generated propaganda bulletin, and the implications for global security, content moderation, and the very architecture of our information ecosystem would prove profound.

This was not an isolated experiment. Days later, ISKP released another AI-driven segment, this time featuring a synthetic anchor dressed in Western attire to claim responsibility for a bombing in Kandahar. The terrorist organisation had discovered what Silicon Valley already knew: generative AI collapses the marginal cost of content production to nearly zero, whilst simultaneously expanding the potential for audience capture beyond anything previously imaginable.

The question now facing researchers, policymakers, and platform architects is not merely whether AI-generated extremist content poses a threat. That much is evident. The deeper concern is structural: what happens when the economics of inflammatory content production fundamentally shift in favour of those willing to exploit human psychological vulnerabilities at industrial scale? And what forms of intervention, if any, can address vulnerabilities that are built into the very architecture of our information systems?

The Economics of Information Pollution

To understand the stakes, one must first grasp the peculiar economics of the attention economy. Unlike traditional markets where production costs create natural barriers to entry, digital content operates under what economists call near-zero marginal cost conditions. Once the infrastructure exists, producing one additional piece of content costs essentially nothing. A research paper published on arXiv in 2025 frames the central challenge succinctly: “When the marginal cost of producing convincing but unverified content approaches zero, how can truth compete with noise?”

The arrival of large language models like GPT-4 and Claude represents what researchers describe as “a structural shift in the information production function.” This shift carries profound implications for the competitive dynamics between different types of content. Prior to generative AI, producing high-quality extremist propaganda required genuine human effort: scriptwriters, video editors, voice actors, translators. Each element imposed costs that naturally limited production volume. A terrorist organisation might release a dozen slickly produced videos annually. Now, the same organisation can generate thousands of variations in multiple languages, tailored to specific demographics, at effectively zero marginal cost.

The economic literature on this phenomenon identifies what researchers term a “production externality” in information markets. Producers of low-quality or harmful content do not internalise the negative social effects of their output. The social marginal cost vastly exceeds the private marginal cost, creating systematic incentives for information pollution. When generative AI capabilities (what some researchers term “offence”) dramatically outstrip detection technologies (“defence”), the marginal cost of producing harmful content falls precipitously, “systemically exacerbating harm.”

This creates what might be called a market bifurcation effect. Research suggests a “barbell” structure will emerge in content markets: low-end demand captured by AI at marginal cost, whilst human creators are forced into high-premium, high-complexity niches. The middle tier of content production essentially evaporates. For mainstream media and entertainment, this means competing against an infinite supply of machine-generated alternatives. For extremist content, it means the historical production barriers that limited proliferation have effectively disappeared.

The U.S. AI-powered content creation market alone was estimated at $198.4 million in 2024 and is projected to reach $741.1 million by 2033, growing at a compound annual growth rate of 15.8%. This explosive growth reflects businesses adopting AI tools to reduce time and costs associated with manual content creation. The same economics that drive legitimate business adoption, however, equally benefit those with malicious intent.

Algorithmic Amplification and the Vulnerability of Engagement Optimisation

The economics of production tell only half the story. The other half concerns distribution, and here the structural vulnerabilities of attention economies become starkly apparent.

Modern social media platforms operate on a simple principle: content that generates engagement receives algorithmic promotion. This engagement-optimisation model has proved extraordinarily effective at capturing human attention. It has also proved extraordinarily effective at amplifying inflammatory, sensational, and divisive material. As Tim Wu, the legal scholar who coined the term “net neutrality,” observed, algorithms “are optimised not for truth or well-being, but for engagement, frequently achieved through outrage, anxiety, or sensationalism.”

The empirical evidence for this amplification effect is substantial. Research demonstrates that false news spreads six times faster than truthful news on Twitter (now X), driven largely by the emotional content that algorithms prioritise. A landmark study published in Science in 2025 provided causal evidence for this dynamic. Researchers developed a platform-independent method to rerank participants' feeds in real time and conducted a preregistered 10-day field experiment with 1,256 participants on X during the 2024 US presidential campaign. The results were striking: decreasing or increasing exposure to antidemocratic attitudes and partisan animosity shifted participants' feelings about opposing political parties by more than 2 points on a 100-point scale. This effect was comparable to several years' worth of polarisation change measured in long-term surveys.

Research by scholars at MIT and elsewhere has shown that Twitter's algorithm amplifies divisive content far more than users' stated preferences would suggest. A systematic review synthesising a decade of peer-reviewed research (2015-2025) on algorithmic effects identified three consistent patterns: algorithmic systems structurally amplify ideological homogeneity; youth demonstrate partial awareness of algorithmic manipulation but face constraints from opaque recommender systems; and echo chambers foster both ideological polarisation and identity reinforcement.

The review also found significant platform-specific effects. Facebook is primarily linked to polarisation, YouTube is associated with radicalisation with particularly strong youth relevance, and Twitter/X emphasises echo chambers with moderate youth impact. Instagram and TikTok remain under-researched despite their enormous user bases, a concerning gap given TikTok's particularly opaque recommendation system.

The implications for AI-generated content are profound. If algorithms already preferentially amplify emotionally charged, divisive material created by humans, what happens when such material can be produced at unlimited scale with sophisticated personalisation? The answer, according to researchers at George Washington University's Program on Extremism, is that extremist groups can now “systematically exploit AI-driven recommendation algorithms, behavioural profiling mechanisms, and generative content systems to identify and target psychologically vulnerable populations, thereby circumventing traditional counterterrorism methodologies.”

The Weaponisation of Psychological Vulnerability

Perhaps the most concerning aspect of AI-enabled extremism is its capacity for psychological targeting at scale. Traditional propaganda operated as a broadcast medium: create a message, distribute it widely, hope it resonates with some fraction of the audience. AI-enabled propaganda operates as a precision instrument: identify psychological vulnerabilities, craft personalised messages, deliver them through algorithmically optimised channels.

Research published in Frontiers in Political Science in 2025 documented how “through analysing huge amounts of personal data, AI algorithms can tailor messages and content which appeal to a particular person's emotions, beliefs and grievances.” This capability transforms radicalisation from a relatively inefficient process into something approaching industrial production.

The numbers are sobering. A recent experiment estimated that AI-generated propaganda can persuade anywhere between 2,500 and 11,000 individuals per 100,000 targeted. Research participants who read propaganda generated by GPT-3 were nearly as persuaded as those who read real propaganda from state actors in Iran or Russia. Given that elections and social movements often turn on margins smaller than this, the potential for AI-generated influence operations to shift outcomes is substantial.

The real-world evidence is already emerging. In July 2024, Austrian authorities arrested several teenagers who were planning a terrorist attack at a Taylor Swift concert in Vienna. The investigation revealed that some suspects had been radicalised online, with TikTok serving as one of the platforms used to disseminate extremist content that influenced their beliefs and actions. The algorithm, optimised for engagement, had efficiently delivered radicalising material to psychologically vulnerable young people.

This is not a failure of content moderation in the traditional sense. It is a structural feature of engagement-optimised systems encountering content designed to exploit that optimisation. Research published in Frontiers in Social Psychology in 2025 found that TikTok's algorithms “privilege more extreme material, and through increased usage, users are gradually exposed to more and more misogynistic ideologies.” The algorithms actively amplify and direct harmful content, not as a bug, but as a consequence of their fundamental design logic.

The combination of psychological profiling and generative AI creates what researchers describe as an unprecedented threat vector. Leaders of extremist organisations are no longer constrained by language barriers, as AI translation capabilities expand their reach across linguistic boundaries. Propaganda materials can now be produced rapidly using just a few keywords. The introduction of deepfakes adds another dimension, enabling the misrepresentation of words or actions by public figures. As AI systems become more publicly available and open-source, the barriers to entry for their use continue to lower, making it easier for malicious actors to adopt AI technologies at scale.

The Collapse of Traditional Content Moderation

Faced with these challenges, platforms have relied on a suite of content moderation tools developed primarily for human-generated content. The most sophisticated of these is “fingerprinting” or hashing, which creates unique digital signatures for known harmful content and automatically removes matches across the platform. This approach has proved reasonably effective against the redistribution of existing terrorist videos and child sexual abuse material.

Generative AI renders this approach largely obsolete. According to research from the Combating Terrorism Center at West Point, “by manipulating their propaganda with generative AI, extremists can change a piece of content's digital fingerprint, rendering fingerprinting mute as a moderation tool.” A terrorist can now take existing propaganda, run it through an AI system that makes superficially minor modifications, and produce content that evades all hash-based detection whilst preserving the harmful message.

The scale challenge compounds this technical limitation. A 2024 report from Philosophy & Technology noted that “humans alone can't keep pace with the enormous volume of content that AI creates.” Most content moderation decisions are now made by machines, not human beings, and this is only set to accelerate. Automation amplifies human error, with biases embedded in training data and system design, whilst enforcement decisions happen rapidly, leaving limited opportunities for human oversight.

Traditional keyword and regex-based filters fare even worse. Research from the University of Chicago's Data Science Institute documented how “GenAI changes content moderation from a post-publication task to a real-time, model-layer challenge. Traditional filters, based on keywords or regex, fail to catch multilingual, evasive, or prompt-driven attacks.”

The detection arms race shows signs of favouring offence over defence. Research from Drexel University identified methods to detect AI-generated video through “fingerprints” unique to different generative models. However, as a Reuters Institute analysis noted, “deepfake creators are finding sophisticated ways to evade detection, so combating them remains a challenge.” Studies have demonstrated poorer performance of detection tools on certain types of content, and researchers warn of “a potential 'arms race' in technological detection, where increasingly sophisticated deepfakes may outpace detection methods.”

The gender dimension of this challenge deserves particular attention. Image-based sexual abuse is not new, but the explosion of generative AI tools to enable it marks a new era for gender-based harassment. For little or no cost, any individual with an internet connection and a photo can produce sexualised imagery of that person. The overwhelming majority of this content targets women and girls, ranging from teenagers to politicians and other public figures. This represents a form of AI-generated extremism that operates at the intersection of technology, misogyny, and the commodification of attention.

Platform Architecture and the Limits of Reform

If traditional content moderation cannot address the AI-generated extremism challenge, what about reforming platform architecture itself? Here the picture grows more complex, touching on fundamental questions about the design logic of attention economies.

The European Union has attempted the most comprehensive regulatory response to date. The Digital Services Act (DSA), which came into full force in 2024, imposes significant obligations on Very Large Online Platforms (VLOPs) with over 45 million monthly EU users. The law forces platforms to be more transparent about how their algorithmic systems work and holds them accountable for societal risks stemming from their services. Non-compliant platforms face fines up to 6% of annual global revenue. During the second quarter of 2024, the Commission publicly confirmed that it had initiated formal proceedings against several major online platforms, requiring detailed documentation on content moderation systems, algorithmic recommender systems, and advertising transparency.

The EU AI Act adds additional requirements specific to AI-generated content. Under this legislation, certain providers must detect and disclose manipulated content, and very large platforms must identify and mitigate systemic risks associated with synthetic content. China has gone further still: as of September 2025, all AI-generated content, whether text, image, video, or audio, must be labelled either explicitly or implicitly, with obligations imposed across service providers, platforms, app distributors, and users.

In February 2025, the European Commission released a new best-practice election toolkit under the Digital Services Act. This toolkit provides guidance for regulators working with platforms to address risks including hate speech, online harassment, and manipulation of public opinion, specifically including those involving AI-generated content and impersonation.

These regulatory frameworks represent important advances in transparency and accountability. Whether they can fundamentally alter the competitive dynamics between inflammatory and mainstream content remains uncertain. The DSA and AI Act address disclosure and risk mitigation, but they do not directly challenge the engagement-optimisation model that underlies algorithmic amplification. Platforms may become more transparent about how their algorithms work whilst those algorithms continue to preferentially promote outrage-inducing material.

Some researchers have proposed more radical architectural interventions. In her 2024 book “Invisible Rulers,” Renee DiResta, formerly of the Stanford Internet Observatory and now at Georgetown University's McCourt School of Public Policy, argued for changes that would make algorithms “reward accuracy, civility, and other values” rather than engagement alone. The Center for Humane Technology, co-founded by former Google design ethicist Tristan Harris, has advocated for similar reforms, arguing that “AI is following the same dangerous playbook” as social media, with “companies racing to deploy AI systems optimised for engagement and market dominance, not human wellbeing.”

Yet implementing such changes confronts formidable obstacles. The attention economy model has proved extraordinarily profitable. In 2024, private AI investment in the United States far outstripped that in the European Union, raising concerns that stringent regulation might simply shift innovation elsewhere. The EU Parliament's own analysis acknowledged that “regulatory complexity could be stifling innovation.” Meanwhile, research institutions dedicated to studying these problems face their own challenges: the Stanford Internet Observatory, which pioneered research into platform manipulation, was effectively dismantled in 2024 following political pressure, with its founding director Alex Stamos and research director Renee DiResta both departing after sustained attacks from politicians who alleged their work amounted to censorship.

The Philosophical Challenge: Can Human-Centred Frameworks Govern Hybrid Media?

Beyond the technical and economic challenges lies a deeper philosophical problem. Our frameworks for regulating speech, including the human rights principles that undergird them, were developed for human expression. What happens when expression becomes “hybrid,” generated or augmented by machines, with fluid authorship and unclear provenance?

Research published in Taylor & Francis journals in 2025 argued that “conventional human rights frameworks, particularly freedom of expression, are considered ill-equipped to govern increasingly hybrid media, where authorship and provenance are fluid, and emerging dilemmas hinge more on perceived value than rights violations.”

Consider the problem of synthetic personas. An AI can generate not just content but entire fake identities, complete with profile pictures, posting histories, and social connections. These synthetic personas can engage in discourse, build relationships with real humans, and gradually introduce radicalising content. From a traditional free speech perspective, we might ask: whose speech is this? The AI developer's? The user who prompted the generation? The corporation that hosts the platform? Each answer carries different implications for responsibility and remedy.

The provenance problem extends to detection. Even if we develop sophisticated tools to identify AI-generated content, what do we do with that information? Mandatory labelling, as China has implemented, assumes users will discount labelled content appropriately. But research on misinformation suggests that labels have limited effectiveness, particularly when content confirms existing beliefs. Moreover, as the Reuters Institute noted, “disclosure techniques such as visible and invisible watermarking, digital fingerprinting, labelling, and embedded metadata still need more refinement.” Malicious actors may circumvent these measures “by using jailbroken versions or creating their own non-compliant tools.”

There is also the question of whether gatekeeping mechanisms designed for human creativity can or should apply to machine-generated content. Copyright law, for instance, generally requires human authorship. Platform terms of service assume human users. Content moderation policies presuppose human judgment about context and intent. Each of these frameworks creaks under the weight of AI-generated content that mimics human expression without embodying human meaning.

The problem grows more acute when considering the speed at which these systems operate. Research from organisations like WITNESS has addressed how transparency in AI production can help mitigate confusion and lack of trust. However, the refinement of disclosure techniques remains ongoing, and the gap between what is technically possible and what is practically implemented continues to widen.

Emerging Architectures: Promise and Peril

Despite these challenges, researchers and technologists are exploring new approaches that might address the structural vulnerabilities of attention economies to AI-generated extremism.

One promising direction involves using large language models themselves for content moderation. Research published in Artificial Intelligence Review in 2025 explored how LLMs could revolutionise moderation economics. Once fine-tuned for the task, LLMs would be far less expensive to deploy than armies of human content reviewers. OpenAI has reported that using GPT-4 for content policy development and moderation enabled faster and more consistent policy iteration, reduced from months to hours, enhancing both accuracy and adaptability.

Yet this approach carries its own risks. Using AI to moderate AI creates recursive dependencies and potential failure modes. As one research paper noted, the tools and strategies used for content moderation “weren't built for GenAI.” LLMs can hallucinate, reflect bias from training data, and generate harmful content “without warning, even when the prompt looks safe.”

Another architectural approach involves restructuring recommendation algorithms themselves. The Science study on algorithmic polarisation demonstrated that simply reranking content to reduce exposure to antidemocratic attitudes and partisan animosity measurably shifted users' political attitudes. This suggests that alternative ranking criteria, prioritising accuracy or viewpoint diversity over engagement, could mitigate polarisation effects. However, implementing such changes would require platforms to sacrifice engagement metrics that directly drive advertising revenue. The economic incentives remain misaligned with social welfare.

Some researchers have proposed more fundamental interventions: breaking up large platforms, imposing algorithmic auditing requirements, creating public interest alternatives to commercial social media, or developing decentralised architectures that reduce the power of any single recommendation system. Each approach carries trade-offs and faces significant political and economic barriers.

Perhaps most intriguingly, some researchers have suggested using AI itself for counter-extremism. As one Hedayah research brief noted, “LLMs could impersonate an extremist and generate counter-narratives on forums, chatrooms, and social media platforms in a dynamic way, adjusting to content seen online in real-time. A model could inject enough uncertainty online to sow doubt among believers and overwhelm extremist channels with benign content.” The prospect of battling AI-generated extremism with AI-generated counter-extremism raises its own ethical questions, but it acknowledges the scale mismatch that human-only interventions cannot address.

The development of more advanced AI models continues apace. GPT-5, launched in August 2025, brings advanced reasoning capabilities in a multimodal interface. Its capabilities suggest a future moderation system capable of understanding context across formats with greater depth. Google's Gemini 2.5 family similarly combines speed, multimodal input handling, and advanced reasoning to tackle nuanced moderation scenarios in real time. Developers can customise content filters and system instructions for tailored moderation workflows. Yet the very capabilities that enable sophisticated moderation also enable sophisticated evasion.

The Attention Ecology and the Question of Cultural Baselines

The most profound concern may be the one hardest to address: the possibility that AI-generated extremism at scale could systematically shift cultural baselines over time. In an “attention ecology,” as researchers describe it, algorithms intervene in “the production, circulation, and legitimation of meaning by structuring knowledge hierarchies, ranking content, and determining visibility.”

If inflammatory content consistently outcompetes moderate content for algorithmic promotion, and if AI enables the production of inflammatory content at unlimited scale, then the information environment itself shifts toward extremism, not through any single piece of content but through the aggregate effect of millions of interactions optimised for engagement.

Research on information pollution describes this as a “congestion externality.” In a digital economy where human attention is the scarce constraint, an exponential increase in synthetic content alters the signal-to-noise ratio. As the cost of producing “plausible but mediocre” content vanishes, platforms face a flood of synthetic noise. The question becomes whether quality content, however defined, can maintain visibility against this tide.

A 2020 Pew Research Center survey found that 64% of Americans believed social media had a mostly negative effect on the direction of the country. This perception preceded the current wave of AI-generated content. If attention economies were already struggling to balance engagement optimisation with social welfare, the introduction of AI-generated content at scale suggests those struggles will intensify.

The cultural baseline question connects to democratic governance in troubling ways. During the 2024 election year, researchers documented deepfake audio and video targeting politicians across multiple countries. In Taiwan, deepfake audio of a politician endorsing another candidate surfaced on YouTube. In the United Kingdom, fake clips targeted politicians across the political spectrum. In India, where over half a billion voters went to the polls, people were reportedly “bombarded with political deepfakes.” These instances represent early experiments with a technology whose capabilities expand rapidly.

Technical Feasibility and Political Will

Can interventions address these structural vulnerabilities? The technical answer is uncertain. Detection technologies continue to improve, but they face a fundamental asymmetry: defenders must identify all harmful content, whilst attackers need only evade detection some of the time. Watermarking and provenance systems show promise but can be circumvented by determined actors using open-source tools or jailbroken models.

The political answer is perhaps more concerning. The researchers and institutions best positioned to study these problems have faced sustained attacks. The Stanford Internet Observatory's effective closure in 2024 followed “lawsuits, subpoenas, document requests from right-wing politicians and non-profits that cost millions to defend, even when vindicated by the US Supreme Court in June 2024.” The lab will not conduct research into any future elections. This chilling effect on research occurs precisely when such research is most needed.

Meanwhile, the economic incentives of major platforms remain oriented toward engagement maximisation. The EU's regulatory interventions, however significant, operate at the margins of business models that reward attention capture above all else. The 2024 US presidential campaign occurred in an information environment shaped by algorithmic amplification of divisive content, with AI-generated material adding new dimensions of manipulation.

There is also the question of global coordination. Regulatory frameworks developed in the EU or US have limited reach in jurisdictions that host extremist content or provide AI tools to bad actors. The ISKP videos that opened this article were not produced in Brussels or Washington. Addressing AI-generated extremism requires international cooperation at a moment when geopolitical tensions make such cooperation difficult.

Internal documents from major platforms have occasionally offered glimpses of the scale of the problem. One revealed that 64% of users who joined extremist groups on Facebook did so “due to recommendation tools.” According to the Mozilla Foundation's “YouTube Regrets” report, 12% of content recommended by YouTube's algorithms violates the company's own community standards. These figures predate the current wave of AI-generated content. The integration of generative AI into content ecosystems has only expanded the surface area for algorithmic radicalisation.

What Happens When Outrage is Free?

The fundamental question raised by AI-generated extremist content concerns the sustainability of attention economies as currently constructed. These systems were designed for an era when content production carried meaningful costs and human judgment imposed natural limits on the volume and extremity of available material. Neither condition obtains in an age of generative AI.

The structural vulnerabilities are not bugs to be patched but features of systems optimised for engagement in a competitive marketplace for attention. Algorithmic amplification of inflammatory content is the logical outcome of engagement optimisation. AI-generated extremism at scale is the logical outcome of near-zero marginal production costs. Traditional content moderation cannot address dynamics that emerge from the fundamental architecture of the systems themselves.

This does not mean the situation is hopeless. The research cited throughout this article points toward potential interventions: algorithmic reform, regulatory requirements for transparency and risk mitigation, AI-powered counter-narratives, architectural redesigns that prioritise different values. Each approach faces obstacles, but obstacles are not impossibilities.

What seems clear is that the current equilibrium is unstable. Attention economies that reward engagement above all else will increasingly be flooded with AI-generated content designed to exploit human psychological vulnerabilities. The competitive dynamics between inflammatory and mainstream content will continue to shift toward the former as production costs approach zero. Traditional gatekeeping mechanisms will continue to erode as detection fails to keep pace with generation.

The choices facing societies are not technical alone but political and philosophical. What values should govern information ecosystems? What responsibilities do platforms bear for the content their algorithms promote? What role should public institutions play in shaping attention markets? And perhaps most fundamentally: can liberal democracies sustain themselves in information environments systematically optimised for outrage?

These questions have no easy answers. But they demand attention, perhaps the scarcest resource of all.

---

References & Sources

• Combating Terrorism Center at West Point, “Generating Terror: The Risks of Generative AI Exploitation,” CTC Sentinel, Volume 17, Issue 1, January 2024. https://ctc.westpoint.edu/generating-terror-the-risks-of-generative-ai-exploitation/

• Frontiers in Political Science, “The role of artificial intelligence in radicalisation, recruitment and terrorist propaganda,” 2025. https://www.frontiersin.org/journals/political-science/articles/10.3389/fpos.2025.1718396/full

• Frontiers in Social Psychology, “Social media, AI, and the rise of extremism during intergroup conflict,” 2025. https://www.frontiersin.org/journals/social-psychology/articles/10.3389/frsps.2025.1711791/full

• Science, “Reranking partisan animosity in algorithmic social media feeds alters affective polarization,” 2025. https://www.science.org/doi/10.1126/science.adu5584

• GNET Research, “Automated Recruitment: Artificial Intelligence, ISKP, and Extremist Radicalisation,” April 2025. https://gnet-research.org/2025/04/11/automated-recruitment-artificial-intelligence-iskp-and-extremist-radicalisation/

• GNET Research, “The Feed That Shapes Us: Extremism and Adolescence in the Age of Algorithms,” December 2025. https://gnet-research.org/2025/12/12/the-feed-that-shapes-us-extremism-and-adolescence-in-the-age-of-algorithms/

• arXiv, “The Economics of Information Pollution in the Age of AI,” 2025. https://arxiv.org/html/2509.13729

• arXiv, “Rewarding Engagement and Personalization in Popularity-Based Rankings Amplifies Extremism and Polarization,” 2025. https://arxiv.org/html/2510.24354v1

• Georgetown Law, “The Attention Economy and the Collapse of Cognitive Autonomy,” Denny Center for Democratic Capitalism. https://www.law.georgetown.edu/denny-center/blog/the-attention-economy/

• George Washington University Program on Extremism, “Artificial Intelligence and Radicalism: Risks and Opportunities.” https://extremism.gwu.edu/artificial-intelligence-and-radicalism-risks-and-opportunities

• International Centre for Counter-Terrorism (ICCT), “The Radicalization (and Counter-radicalization) Potential of Artificial Intelligence.” https://icct.nl/publication/radicalization-and-counter-radicalization-potential-artificial-intelligence

• Hedayah, “AI for Counter-Extremism Research Brief,” April 2025. https://hedayah.com/app/uploads/2025/06/Hedayah-Research-Brief-AI-for-Counter-Extremism-April-2025-Design-DRAFT-28.04.25-v2.pdf

• Philosophy & Technology (PMC), “Moderating Synthetic Content: the Challenge of Generative AI,” 2024. https://pmc.ncbi.nlm.nih.gov/articles/PMC11561028/

• Taylor & Francis, “Platforms as architects of AI influence: rethinking moderation in the age of hybrid expression,” 2025. https://www.tandfonline.com/doi/full/10.1080/20414005.2025.2562681

• Taylor & Francis, “The Ghost in the Machine: Counterterrorism in the Age of Artificial Intelligence,” 2025. https://www.tandfonline.com/doi/full/10.1080/1057610X.2025.2475850

• European Commission, “The Digital Services Act,” Digital Strategy. https://digital-strategy.ec.europa.eu/en/policies/digital-services-act

• AlgorithmWatch, “A guide to the Digital Services Act.” https://algorithmwatch.org/en/dsa-explained/

• TechPolicy.Press, “The Digital Services Act Meets the AI Act: Bridging Platform and AI Governance.” https://www.techpolicy.press/the-digital-services-act-meets-the-ai-act-bridging-platform-and-ai-governance/

• ISD Global, “Towards transparent recommender systems: Lessons from TikTok research ahead of the 2025 German federal election.” https://www.isdglobal.org/digital_dispatches/towards-transparent-recommender-systems-lessons-from-tiktok-research-ahead-of-the-2025-german-federal-election/

• Reuters Institute for the Study of Journalism, “Spotting the deepfakes in this year of elections: how AI detection tools work and where they fail.” https://reutersinstitute.politics.ox.ac.uk/news/spotting-deepfakes-year-elections-how-ai-detection-tools-work-and-where-they-fail

• U.S. GAO, “Science & Tech Spotlight: Combating Deepfakes,” 2024. https://www.gao.gov/products/gao-24-107292

• World Economic Forum, “4 ways to future-proof against deepfakes in 2024 and beyond,” February 2024. https://www.weforum.org/stories/2024/02/4-ways-to-future-proof-against-deepfakes-in-2024-and-beyond/

• Springer, “Content moderation by LLM: from accuracy to legitimacy,” Artificial Intelligence Review, 2025. https://link.springer.com/article/10.1007/s10462-025-11328-1

• Meta Oversight Board, “Content Moderation in a New Era for AI and Automation.” https://www.oversightboard.com/news/content-moderation-in-a-new-era-for-ai-automation/

• MDPI, “Trap of Social Media Algorithms: A Systematic Review of Research on Filter Bubbles, Echo Chambers, and Their Impact on Youth,” 2024. https://www.mdpi.com/2075-4698/15/11/301

• Center for Humane Technology. https://www.humanetech.com/

• NPR, “A major disinformation research team's future is uncertain after political attacks,” June 2024. https://www.npr.org/2024/06/14/g-s1-4570/a-major-disinformation-research-teams-future-is-uncertain-after-political-attacks

• Platformer, “The Stanford Internet Observatory is being dismantled,” 2024. https://www.platformer.news/stanford-internet-observatory-shutdown-stamos-diresta-sio/

• Grand View Research, “U.S. AI-Powered Content Creation Market Report, 2033.” https://www.grandviewresearch.com/industry-analysis/us-ai-powered-content-creation-market-report

• Mozilla Foundation, “YouTube Regrets” Report. https://foundation.mozilla.org/

• Pew Research Center, “Americans' Views of Technology Companies,” 2020. https://www.pewresearch.org/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

In September 2025, Anthropic's security team detected something unprecedented. An AI system was being used not merely as an advisor to human hackers, but as the primary operator of an espionage campaign. At its peak, the AI made thousands of requests per second, probing systems, adjusting tactics, and exploiting vulnerabilities at a pace no human could match. According to Anthropic's analysis, the threat actor was able to use AI to perform 80 to 90 per cent of the entire campaign, with human intervention required only sporadically. The attack represented a threshold moment: machines were no longer just tools in the hands of cybercriminals. They had become the criminals themselves.

This incident crystallises the central question now haunting every chief information security officer, every government cyber agency, and every organisation that depends on digital infrastructure (which is to say, all of them): As AI capabilities mature and become increasingly accessible, can defenders develop countermeasures faster than attackers can weaponise these systems at scale? And how will the traditional boundaries between human expertise and machine automation fundamentally reshape both the threat landscape and the organisational structures built to counter it?

The answer is not encouraging. The security community is engaged in an arms race that operates according to profoundly asymmetric rules, where attackers enjoy advantages that may prove structurally insurmountable. Yet within this grim calculus, a transformation is underway in how defenders organise themselves, deploy their resources, and conceptualise the very nature of security work. The outcome will determine whether the digital infrastructure underpinning modern civilisation remains defensible.

The Industrialisation of Cybercrime

The fundamental shift in the threat landscape is not that AI has invented new categories of attack. Rather, AI has industrialised existing attack vectors, enabling them to operate at scales and speeds that overwhelm traditional defensive approaches. In 2025, reports consistently show that AI is not inventing new attacks; it is scaling old ones. The result has been characterised as an “industrialisation of cybercrime” powered by artificial intelligence, transforming what was once an artisanal practice into mass production.

Consider the statistics. CrowdStrike's 2025 Global Threat Report documented a 442 per cent increase in voice phishing (vishing) attacks between the first and second halves of 2024, driven almost entirely by AI-generated voice synthesis. Phishing attempts crafted by large language models achieve a 54 per cent click-through rate, compared to just 12 per cent for human-generated attempts. Microsoft's 2025 Digital Defense Report found that AI-driven identity forgeries grew 195 per cent globally, with deepfake techniques now sophisticated enough to defeat selfie checks and liveness tests that simulate natural eye movements and head turns.

The barrier to entry for cybercrime has collapsed entirely. What once required advanced technical expertise now requires nothing more than access to the right tools. CrowdStrike documented North Korean operatives using generative AI to draft convincing resumes, create synthetic identities with altered photographs, and deploy real-time deepfake technology during live video interviews, enabling them to infiltrate organisations by posing as legitimate job candidates. This activity increased 220 per cent year over year, representing a systematic campaign to place operatives inside target organisations.

The KELA 2025 AI Threat Report documented a 200 per cent surge in mentions of malicious AI tools on cybercrime forums. The cybercrime-as-a-service model has expanded to include AI-powered attack kits that lower-skilled hackers can rent, effectively democratising sophisticated threats. The Hong Kong Computer Emergency Response Team identified six distinct categories of AI-assisted attacks now being actively deployed: automated vulnerability discovery, adaptive malware generation, real-time social engineering, credential theft automation, code assistant exploitation, and deepfake-enabled fraud.

Perhaps most troubling is the emergence of autonomous malware. Dark Reading's analysis of 2026 security predictions warns of self-learning, self-preserving cyber worms that not only morph to avoid detection but fundamentally change tactics, techniques, and procedures based on the defences they encounter. Unlike traditional malware that follows static attack patterns, AI-powered malware can adapt to environments and analyse security measures, adjusting tactics to bypass defences. These are not hypothetical constructs. Security researchers are already observing such adaptive behaviour in the wild.

The scale of the problem is staggering. IBM's 2025 Cost of a Data Breach Report found that attackers are using AI in 16 per cent of breaches to fuel phishing campaigns and create deepfakes. Shadow AI, where employees use unapproved AI tools, was a factor in 20 per cent of breaches, adding an average of $670,000 to breach costs. The average global breach cost dropped to $4.44 million from $4.88 million (the first decline in five years), but in the United States, costs rose to $10.22 million due to regulatory penalties and slower detection times. Healthcare remains the costliest sector for the fourteenth consecutive year, with breaches averaging $7.42 million.

The speed differential between attackers and defenders may be the most concerning development. CrowdStrike documented an average “breakout time” of just 48 minutes, with the fastest recorded breach taking only 51 seconds from initial access to lateral movement. When machines operate at machine speed, human-scale response times become a critical vulnerability. In the first quarter of 2025 alone, there were 179 deepfake incidents recorded, surpassing the total for all of 2024 by 19 per cent.

The Structural Asymmetry

The cybersecurity arms race operates according to rules that structurally favour attackers. This is not a matter of resources or talent, though both matter enormously. It reflects fundamental differences in the constraints under which each side operates, differences that AI amplifies rather than eliminates.

Attackers face no consequences for collateral damage. If an AI-powered attack tool causes unintended disruption, no attacker loses their job. Defenders, by contrast, must carefully vet every AI security tool before production deployment. As one security expert noted in Dark Reading's analysis, “If bad things happen when AI security technologies are deployed, people get fired.” This asymmetry in risk tolerance creates a gap in deployment speed that attackers consistently exploit.

Furthermore, attackers need succeed only once. Defenders must succeed every time. A defender might block 99.9 per cent of attacks and still suffer a catastrophic breach from the 0.1 per cent that penetrates. This mathematical reality has always favoured offence in cybersecurity, but AI amplifies the disparity by enabling attackers to launch vast numbers of attempts simultaneously, each slightly varied, probing for the inevitable gap.

The talent shortage compounds these structural disadvantages dramatically. The ISC2 2025 Cybersecurity Workforce Study found that 59 per cent of respondents identified critical or significant skills shortages within their teams, up from 44 per cent in 2024. Nearly nine in ten respondents (88 per cent) have experienced at least one significant cybersecurity consequence due to skills shortages, and 69 per cent have experienced more than one. AI and cloud security top the list of vital skills needs, with 41 per cent and 36 per cent of respondents respectively citing them as critical gaps. Notably, ISC2 did not include an estimate of the cybersecurity workforce gap this year because the study found that the need for critical skills within the workforce is outweighing the need to increase headcount.

Current estimates place the global shortfall of cybersecurity professionals between 2.8 and 4.8 million. The 2024 ISC2 study estimated global demand at 10.2 million with a current workforce of only 5.5 million. This shortage exists at precisely the moment when AI is transforming the skills required for effective defence.

The UK National Cyber Security Centre's 2025 Annual Review reported 204 “nationally significant” cyber incidents between September 2024 and August 2025, representing a 130 per cent increase from the previous year's 89 incidents. This is the highest number ever recorded. The NCSC assessment is blunt: threat actors of all types continue to use AI to enhance their existing tactics, techniques, and procedures, increasing the efficiency, effectiveness, and frequency of their cyber intrusions. AI lowers the barrier for novice cybercriminals, hackers-for-hire, and hacktivists to carry out effective operations.

BetaNews reported that security experts are warning 2026 could see a widening gap between attacker agility and defender constraints, resulting in an asymmetric shift that favours threat actors. Most analysts expect 2026 to be the first year that AI-driven incidents outpace what the majority of security teams can respond to manually.

The Defensive Transformation

Yet the picture is not uniformly bleak. Defenders are beginning to deploy AI in ways that could, eventually, rebalance the equation. The question is whether they can move fast enough.

The transformation is most visible in the Security Operations Centre (SOC). Traditional SOCs were never designed for today's threat landscape. Cloud sprawl, hybrid workforces, encrypted traffic, and AI-driven adversaries have pushed traditional models beyond their limits. Studies indicate that security teams receive an average of 4,000 or more alerts daily, the vast majority being false positives or low-priority notifications. Analysts are inundated, investigations are manual and time-consuming, and response often comes too late.

AI SOC agents represent a new wave of automation that complements existing tools to do more than detect and triage. They act, learn from evolving threats, adapt to changing environments, and collaborate with human analysts. IBM's analysis of AI-driven SOC co-pilots suggests they will make a significant impact, helping security teams prioritise threats and turn overwhelming amounts of data into actionable intelligence. Brian Linder, Cybersecurity Evangelist at Check Point, observed that AI-driven SOC co-pilots will help security teams turn overwhelming amounts of data into actionable intelligence.

The benefits are measurable. A 2025 study cited by the World Economic Forum found that 88 per cent of security teams report significant time savings through AI. Speed is one of the biggest improvements AI brings: it helps SOCs spot risky behaviour within seconds rather than hours. When AI handles repetitive tasks, analysts have more time for higher-level work such as strategy and analytics, which reduces burnout.

Microsoft's systems now process over 100 trillion signals daily, block approximately 4.5 million new malware attempts, analyse 38 million identity risk detections, and scan 5 billion emails for malware and phishing threats. AI agents can act within seconds, suspending a compromised account and triggering a password reset as soon as multiple high-risk signals align, containing breaches before escalation.

The emergence of adversarial learning represents another defensive advancement. By training threat and defence models continuously against one another, security teams can develop systems capable of countering adaptive AI attacks. Artificial Intelligence News reported a breakthrough in real-time adversarial learning that offers a decisive advantage over static defence mechanisms, particularly as AI-driven attacks using reinforcement learning create threats that mutate faster than human teams can respond.

The AIDEFEND framework, released as an open knowledge base for AI security, provides defensive countermeasures and best practices to help security professionals safeguard AI and machine learning systems. The Cloud Security Alliance has developed a “Zero Trust 2.0” framework specifically designed for AI systems, using artificial intelligence integrated with machine learning to establish trust in real time through behavioural and network activity observation.

Gartner forecasts that worldwide end-user spending on information security will reach $213 billion in 2025, up from $193 billion in 2024, with spending estimated to increase 12.5 per cent in 2026 to total $240 billion. The consultancy predicts that by 2028, over 50 per cent of enterprises will use AI security platforms to protect their AI investments, and by 2030, preemptive solutions will account for half of all security spending.

The Human-Machine Boundary

The most profound transformation may be in how human expertise and machine automation interact. The future is neither fully automated defence nor purely human analysis. It is a hybrid model that is still being invented.

The consensus among researchers is increasingly clear: AI will handle the heavy lifting of data processing, anomaly detection, and predictive analysis, whilst humans bring creativity, strategic thinking, and nuanced decision-making that machines cannot replicate. The future of cyber threat intelligence is not one of automation replacing human expertise, but rather a collaborative intelligence model. Technical expertise alone is not sufficient for this new paradigm. Soft skills such as analytical and creative thinking, communication, collaboration, and agility will be just as critical in the AI era for managing risk effectively.

The analyst's interaction moves upstream in this model. Instead of investigating every alert from scratch, analysts validate the agent's work, provide additional context when the agent escalates uncertainty, and focus on complex cases that genuinely require nuanced human judgement. While AI can immediately block a known malware signature, a security analyst will review and decide how to handle an unfamiliar or sophisticated attack. The goal with agents is to automate the repetitive grunt work of context gathering that consumes valuable analyst time. Agents can now handle the initial alert assessment, dynamically adjust priorities based on context, and enrich alerts with threat intelligence before an analyst ever sees them.

This is emphatically not about replacement. Despite rapid advances, the idea that AI SOC agents can fully replace human expertise in security operations is a myth. Today's reality is one of collaboration: AI agents are emerging as powerful facilitators, not autonomous replacements. The SANS 2025 SOC Survey highlights that 69 per cent of SOCs still rely on manual or mostly manual processes to report metrics. Additionally, 40 per cent of SOCs use AI or ML tools without making them a defined part of operations, and 42 per cent rely on AI/ML tools “out of the box” with no customisation.

The World Economic Forum warns that current estimates place the global shortfall of cybersecurity professionals between 2.8 and 4.8 million. AI can play a pivotal role in narrowing this gap by taking on manual-intensive tasks, freeing security team members to concentrate on strategic planning. Yet the Fortinet 2025 Cybersecurity Skills Gap report found that 49 per cent of cybersecurity leaders are concerned that AI will increase the volume and sophistication of cyberattacks. This creates a paradox: AI is both the solution to the skills gap and the driver of its expansion.

The emerging model is one of “autonomic defence,” systems capable of learning, anticipating, and responding intelligently without human intervention for routine matters, whilst preserving human oversight for complex or high-impact situations. Security Boulevard's analysis of next-generation SOC platforms describes a future where automation handles the speed and scale that attackers exploit, whilst human oversight remains available for strategic decisions.

Optimal security, according to research published by the World Economic Forum, relies on balancing AI-driven automation with human intuition, creativity, and ethical reasoning. Overdependence on AI risks blind spots and misjudgements. Organisations that invest equally in advanced tools and skilled people will be best positioned to withstand the next wave of threats.

Restructuring the Security Organisation

The transformation extends beyond technology to organisational structure itself, demanding new leadership models and new approaches to talent development. The CISO role, once primarily focused on safeguarding IT infrastructure, has expanded to encompass AI integration oversight, ensuring secure implementation and governance of AI systems throughout the enterprise.

Proofpoint's 2025 Voice of the CISO Report found that AI risks now top priority lists for security leaders, outpacing long-standing concerns like vulnerability management, data loss prevention, and third-party risk. Ryan Kalember, Proofpoint's chief strategy officer, observed that “Artificial intelligence has moved from concept to core, transforming how both defenders and adversaries operate.” CISOs now face a dual responsibility: harnessing AI to strengthen their security posture whilst ensuring its ethical and responsible use.

Yet the CISO role may have become too broad for one person to handle effectively. Cyble's analysis of the “CISO 3.0” concept suggests that in 2026, more organisations will separate the strategic and operational sides of security leadership. One track will focus on enterprise risk, governance, and alignment with the board. The other will manage day-to-day operations and technical execution. This bifurcation acknowledges that the scope of modern security leadership exceeds what any single executive can reasonably manage.

The Gartner C-level Communities' 2025 Leadership Perspective Survey found that CISOs have made cyber resilience their top priority, reflecting the need for organisations not only to withstand and respond to cyber attacks but also to resume operations in a timely manner. This represents a shift from the previous focus on user access, identity and access management, and zero trust. The emphasis on resilience acknowledges that perfect prevention is impossible; the ability to recover quickly matters as much as the ability to prevent attacks.

Optiv's Cybersecurity Peer Index for 2025 found that across industries, more than 55 per cent of organisations have their security functions reporting to a senior leadership role. Yet PwC's Digital Trust Insights found that organisations using autonomous security agents saw a 43 per cent rise in unexpected AI-driven security incidents, from over-permissioned AI agents to silent prompt manipulations. The governance challenge is substantial: agentic AI breaks traditional visibility models. Organisations are no longer monitoring code or endpoints; they are monitoring behavioural decisions of autonomous systems.

The SANS report highlights a concerning lack of security team involvement in governing generative AI. Many cybersecurity professionals believe they should play a role in enterprise-wide AI governance, but very few organisations have a formal AI risk management programme in place. While half of surveyed organisations currently use AI for cybersecurity tasks, and 100 per cent plan to incorporate generative AI within the next year, widespread adoption for critical functions remains limited.

Proofpoint found that 77 per cent of CISOs expect AI can replace human labour in high-volume, process-heavy tasks, with the SOC at the top of the list of functions likely to be transformed. Over half of organisations report that AI has affected their security team's training requirements, with a majority emphasising the need for more specialised AI and cybersecurity courses.

Workforce wellbeing has become a critical concern. The ISC2 study found that almost half (48 per cent) of respondents feel exhausted from trying to stay current on the latest cybersecurity threats and emerging technologies, and 47 per cent feel overwhelmed by the workload. Burnout remains a serious risk in an environment of constant threat evolution.

The Governance Imperative

The absence of robust AI governance may prove to be the most significant vulnerability in the current landscape. IBM's 2025 Cost of a Data Breach Report found that a staggering 97 per cent of breached organisations that experienced an AI-related security incident lacked proper AI access controls. Additionally, 63 per cent of organisations revealed they have no AI governance policies in place to manage AI or prevent workers from using shadow AI.

Gartner predicts that by 2027, more than 40 per cent of AI-related data breaches will be caused by the improper use of generative AI across borders. The regulatory landscape is evolving rapidly, but organisations are struggling to keep pace. The European Data Protection Board's 2025 guidance provides criteria for identifying privacy risks, emphasising the need to control inputs to LLM systems to avoid exposing personal information, trade secrets, or intellectual property.

The 2025 OWASP Top 10 for Large Language Model Applications places prompt injection as the number one concern in securing LLMs, underscoring its critical importance in generative AI security. Attack scenarios include cross-site scripting, SQL injection, or code execution via unsafe LLM output. The vulnerability is particularly insidious because data passed to a large language model from a third-party source could contain text that the LLM will execute as a prompt. This indirect prompt injection is a major problem where LLMs are linked with third-party tools to access data or perform tasks.

Mitigation strategies recommended by OWASP include treating the model as a user, adopting a zero-trust approach, and ensuring proper input validation for any responses from the model to backend functions. Organisations should encode the model's output before delivering it to users to prevent unintended code execution and implement content filters to eliminate vulnerabilities.

Deloitte's 2025 analysis found that only 9 per cent of enterprises have reached a “Ready” level of AI governance maturity. That is not because organisations are lazy, but because they are trying to govern something that moves faster than their governance processes. Gartner predicts that by 2026, enterprises applying AI TRiSM (Trust, Risk, and Security Management) controls will consume at least 50 per cent less inaccurate or illegitimate information. Gartner has also predicted that 40 per cent of social engineering attacks will target executives as well as the broader workforce by 2028, as attackers combine social engineering tactics with deepfake audio and video.

The Uncertain Equilibrium

The question posed at the outset remains unanswered: Can defenders develop countermeasures faster than attackers can weaponise AI? The honest answer is that nobody knows. The variables are too numerous, the timescales too compressed, and the feedback loops too complex for confident prediction.

The optimistic view holds that AI-powered cyber defences are finally arriving to help defenders address AI-driven attacks. ClearanceJobs reported that in 2026, the playing field will begin to even out as mature AI-powered cybersecurity tools arrive to provide real value in countering attackers' use of AI. The defensive AI market is growing rapidly, with Grand View Research estimating the AI cybersecurity market could approach $100 billion by 2030. According to research from Gartner and IBM, organisations that effectively deploy and govern security AI see significantly better outcomes.

The pessimistic view notes that BCG surveys found 60 per cent of executives faced AI attacks, yet only 7 per cent had deployed defensive AI at scale. The gap between threat and response capabilities remains wide. Attackers continue to enjoy structural advantages in speed, risk tolerance, and flexibility. Gartner predicts that by 2027, AI agents will reduce the time it takes to exploit account exposures by 50 per cent, suggesting attackers will continue to accelerate even as defenders deploy countermeasures.

The realistic view acknowledges that the outcome depends on choices yet to be made. Organisations that invest in both advanced AI tools and skilled human analysts, that implement robust AI governance, that restructure their security leadership for the complexity of the current moment, will be better positioned to survive. Those that delay, that underinvest, that fail to evolve their organisational structures, will find themselves increasingly vulnerable. The NCSC continues to call on companies and boards to start viewing cybersecurity as a company-wide issue, stating that “all business leaders need to take responsibility for their organisation's cyber resilience.”

The NCSC's 2025 Annual Review warns of a “growing divide” between organisations that keep up with threat actors using AI and those that remain vulnerable. This divide may prove more consequential than any particular technology or tactic. The winners and losers in the AI security arms race will likely be determined not by who has the best algorithms, but by who builds the most effective hybrid systems combining machine speed with human wisdom.

The transformation of cybersecurity is not merely a technical challenge. It is an organisational, cultural, and fundamentally human challenge. The boundaries between human expertise and machine automation are not fixed; they are being negotiated in real time, in every SOC, in every boardroom, in every government agency tasked with protecting critical infrastructure.

What remains clear is that the old models are obsolete. The 2025 security landscape demands new approaches to talent, new approaches to governance, new approaches to the very definition of what security work entails. The organisations that recognise this and adapt accordingly will shape the future of digital defence. Those that do not will become statistics in next year's breach reports.

The AI-powered cyber threat will not wait for defenders to figure things out. The clock, as always in cybersecurity, is running. Whether defenders can move fast enough remains the defining question of this technological moment. As we approach 2026 and beyond, the key to survival is not just deploying better AI; it is ensuring organisations maintain control of these powerful tools whilst they operate at machine speed. The frameworks, partnerships, and governance structures established now will define the future of cybersecurity for decades to come.

---

References and Sources

1. Anthropic. “Disrupting the First Reported AI-Orchestrated Cyber Espionage Campaign.” Anthropic News, September 2025. https://www.anthropic.com/news/disrupting-AI-espionage

2. CrowdStrike. “2025 Global Threat Report.” CrowdStrike, 2025. https://www.crowdstrike.com/en-us/press-releases/crowdstrike-releases-2025-global-threat-report/

3. Microsoft. “2025 Microsoft Digital Defense Report.” Microsoft Security Insider, 2025. https://www.microsoft.com/en-us/security/security-insider/threat-landscape/microsoft-digital-defense-report-2025

4. IBM. “Cost of a Data Breach Report 2025.” IBM, 2025. https://www.ibm.com/reports/data-breach

5. NCSC. “NCSC Annual Review 2025.” National Cyber Security Centre, 2025. https://www.ncsc.gov.uk/collection/ncsc-annual-review-2025

6. ISC2. “2025 ISC2 Cybersecurity Workforce Study.” ISC2 Insights, December 2025. https://www.isc2.org/Insights/2025/12/2025-ISC2-Cybersecurity-Workforce-Study

7. Gartner. “Gartner Identifies the Top Cybersecurity Trends for 2025.” Gartner Newsroom, March 2025. https://www.gartner.com/en/newsroom/press-releases/2025-03-03-gartner-identifiesthe-top-cybersecurity-trends-for-2025

8. Proofpoint. “2025 Voice of the CISO Report.” Proofpoint Newsroom, 2025. https://www.proofpoint.com/us/newsroom/press-releases/proofpoint-2025-voice-ciso-report

9. Dark Reading. “Cybersecurity Predictions 2026: AI Arms Race and Malware Autonomy.” Dark Reading, December 2025. https://www.darkreading.com/cyber-risk/cybersecurity-predictions-2026-an-ai-arms-race-and-malware-autonomy

10. KELA. “2025 AI Threat Report: How Cybercriminals Are Weaponizing AI Technology.” KELA Cyber, 2025. https://www.kelacyber.com/resources/research/2025-ai-threat-report/

11. HKCERT. “Hackers' New Partner: Weaponized AI for Cyber Attacks! HKCERT Exposes Six Emerging AI-Assisted Attacks.” HKCERT Blog, 2025. https://www.hkcert.org/blog/hackers-new-partner-weaponized-ai-for-cyber-attacks-hkcert-exposes-six-emerging-ai-assisted-attacks

12. World Economic Forum. “Can Cybersecurity Withstand the New AI Era?” WEF Stories, October 2025. https://www.weforum.org/stories/2025/10/can-cybersecurity-withstand-new-ai-era/

13. SANS Institute. “2025 SANS SOC Survey.” SANS, 2025. Referenced via Swimlane. https://swimlane.com/blog/ciso-guide-ai-security-impact-sans-report/

14. Cloud Security Alliance. “Fortifying the Agentic Web: A Unified Zero-Trust Architecture for AI.” CSA Blog, September 2025. https://cloudsecurityalliance.org/blog/2025/09/12/fortifying-the-agentic-web-a-unified-zero-trust-architecture-against-logic-layer-threats

15. OWASP. “Top 10 for Large Language Model Applications.” OWASP, 2025. https://owasp.org/www-project-top-10-for-large-language-model-applications/

16. Optiv. “Cybersecurity Leadership in 2025: The Strategic Role of CISOs in an AI-Driven Era.” Optiv Insights, 2025. https://www.optiv.com/insights/discover/blog/cybersecurity-leadership-2025-strategic-role-cisos-ai-driven-era

17. Cyble. “CISO 3.0: The Role of Security Leaders in 2026's Agentic Era.” Cyble Knowledge Hub, 2025. https://cyble.com/knowledge-hub/ciso-3-0-security-leaders-2026-agentic-era/

18. BetaNews. “Cyber Experts Warn AI Will Accelerate Attacks and Overwhelm Defenders in 2026.” BetaNews, December 2025. https://betanews.com/2025/12/10/cyber-experts-warn-ai-will-accelerate-attacks-and-overwhelm-defenders-in-2026/

19. ClearanceJobs. “Cybersecurity's AI Arms Race Is Just Getting Started.” ClearanceJobs News, December 2025. https://news.clearancejobs.com/2025/12/26/cybersecuritys-ai-arms-race-is-just-getting-started-heres-what-2026-will-bring/

20. Security Boulevard. “From Alert Fatigue to Autonomous Defense: The Next-Gen SOC Automation Platform.” Security Boulevard, December 2025. https://securityboulevard.com/2025/12/from-alert-fatigue-to-autonomous-defense-the-next-gen-soc-automation-platform/

21. Fortinet. “2025 Cybersecurity Skills Gap Report.” Fortinet, 2025. Referenced via World Economic Forum.

22. Help Net Security. “AIDEFEND: Free AI Defense Framework.” Help Net Security, September 2025. https://www.helpnetsecurity.com/2025/09/01/aidefend-free-ai-defense-framework/

23. Artificial Intelligence News. “Adversarial Learning Breakthrough Enables Real-Time AI Security.” AI News, 2025. https://www.artificialintelligence-news.com/news/adversarial-learning-breakthrough-real-time-ai-security/

24. Gartner. “Gartner Predicts AI Agents Will Reduce The Time It Takes To Exploit Account Exposures by 50% by 2027.” Gartner Newsroom, March 2025. https://www.gartner.com/en/newsroom/press-releases/2025-03-18-gartner-predicts-ai-agents-will-reduce-the-time-it-takes-to-exploit-account-exposures-by-50-percent-by-2027

25. Gartner. “Gartner Predicts 40% of AI Data Breaches Will Arise from Cross-Border GenAI Misuse by 2027.” Gartner Newsroom, February 2025. https://www.gartner.com/en/newsroom/press-releases/2025-02-17-gartner-predicts-forty-percent-of-ai-data-breaches-will-arise-from-cross-border-genai-misuse-by-2027

26. Deepstrike. “AI Cybersecurity Threats 2025: Surviving the AI Arms Race.” Deepstrike Blog, 2025. https://deepstrike.io/blog/ai-cybersecurity-threats-2025

27. RSA Conference. “The AI-Powered SOC: How Artificial Intelligence is Transforming Security Operations in 2025.” RSAC Library, 2025. https://www.rsaconference.com/library/blog/the-ai-powered-soc-how-artificial-intelligence-is-transforming-security-operations-in-2025

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

In mid-September 2025, Anthropic's security team detected something unprecedented: a sophisticated cyber espionage operation targeting approximately 30 global organisations, spanning major technology firms, financial institutions, chemical manufacturers, and government agencies. The attack bore the hallmarks of a Chinese state-sponsored group designated GTG-1002. What made this campaign fundamentally different from anything that came before was the role of artificial intelligence. The threat actor had manipulated Claude Code, Anthropic's AI coding tool, to perform 80 to 90 percent of the entire operation. Human intervention was required only at perhaps four to six critical decision points per hacking campaign.

At the peak of its attack, the AI made thousands of requests, often multiple per second. This was an attack speed that would have been, for human hackers, simply impossible to match. The threat actor had tricked Claude into believing it was a cybersecurity firm conducting defensive testing, thereby bypassing the system's safety features. A subset of the intrusions succeeded. Anthropic banned the relevant accounts, notified affected entities, and coordinated with law enforcement. But the implications of this incident continue to reverberate through the technology industry.

“The barriers to performing sophisticated cyberattacks have dropped substantially,” Anthropic stated in its November 2025 disclosure. “And are predicted to continue to do so.” The adoption of advanced intrusion techniques through AI significantly lowers the barriers for smaller and less-resourced threat groups to conduct sophisticated espionage operations.

Claude was not perfect during the attacks. According to Anthropic's own analysis, the AI hallucinated some login credentials and claimed it stole a secret document that was already publicly available. But these imperfections did little to diminish the campaign's overall effectiveness. The incident represented what Anthropic described as “a fundamental shift in how advanced threat actors use AI.”

This incident crystallises the central dilemma facing every company developing agentic AI tools: how do you build systems powerful enough to transform legitimate software development while preventing those same capabilities from being weaponised for extortion, espionage, and large-scale cybercrime?

When Autonomy Becomes a Weapon

The cybersecurity landscape of 2026 looks fundamentally different from what existed just two years prior. According to research from the World Economic Forum, cyberattacks have more than doubled in frequency since 2021, from an average of 818 weekly attacks per organisation to 1,984 in the same period of 2025. The global average number of weekly attacks encountered by organisations grew by 58 percent in the last two years alone. Cybercrime is projected to cost the global economy a staggering 10.5 trillion US dollars annually.

The driving force behind this acceleration is not simply the increasing sophistication of criminal enterprises. It is the democratisation of offensive capabilities through artificial intelligence. Palo Alto Networks' Unit 42 research division has documented this transformation in stark terms. In 2021, the average mean time to exfiltrate data stood at nine days. By 2024, that figure had collapsed to just two days. In one out of every five cases, the time from initial compromise to data exfiltration was less than one hour.

Perhaps most alarmingly, Unit 42 demonstrated in controlled testing that an AI-powered ransomware attack could be executed from initial compromise to data exfiltration in just 25 minutes. This represents a 100-fold increase in speed compared to traditional attack methods.

The emergence of malicious large language models has fundamentally altered the threat calculus. Tools like WormGPT, FraudGPT, and the more recent KawaiiGPT (first identified in July 2025 and now at version 2.5) are explicitly marketed for illicit activities on dark web forums. According to analysis from Palo Alto Networks' Unit 42, mentions of “dark LLMs” on cybercriminal forums skyrocketed by over 219 percent in 2024. These unrestricted models have removed the barriers in terms of technical skill required for cybercrime activity, granting the power once reserved for more knowledgeable threat actors to virtually anyone with an internet connection.

The research from UC Berkeley's Center for Long-Term Cybersecurity describes this phenomenon starkly: by lowering the technical barrier, AI “supercharges” the capabilities of existing criminals, making cybercrime more accessible and attractive due to its relatively lower risk and cost compared to traditional street-level offences.

The ransomware ecosystem illustrates this democratisation in brutal clarity. According to statistics from ecrime.ch, ransomware actors posted 7,819 incidents to data leak sites in 2025. From January to June 2025, the number of publicly reported ransomware victims jumped 70 percent compared to the same period in both 2023 and 2024. February stood out as the worst month, with 955 reported cases. The year was characterised by a dramatic fragmentation following law enforcement disruptions of major operations such as LockBit and ALPHV/BlackCat. This fragmentation resulted in 45 newly observed groups, pushing the total number of active extortion operations to a record-breaking 85 distinct threat actors.

Tasks that once required dedicated “data warehouse managers” within ransomware groups can now be accomplished by AI in hours rather than weeks. AI can automatically identify and categorise sensitive information like social security numbers, financial records, and personal data, then craft tailored extortion notes listing specific compromised assets. AI-powered chatbots are now handling ransom negotiations, eliminating language barriers and time zone delays, maintaining consistent pressure throughout the negotiation process around the clock.

One of the most notable shifts in 2025 was the growing abandonment of encryption altogether. New ransomware groups such as Dire Wolf, Silent Team, and DATACARRY relied on data theft and leak-based extortion without deploying ransomware lockers. This model reduces execution time, lowers detection risk, and exploits reputational damage as the primary pressure mechanism.

The Agentic Paradigm Shift

The transition from conversational AI assistants to agentic AI systems represents a qualitative leap in both capability and risk. NVIDIA's technical research has categorised agentic systems into four autonomy levels (0 through 3) based on their complexity and decision-making capabilities, with Level 3 being the most autonomous and posing the greatest challenge for threat modelling and risk assessment. Identifying the system autonomy level provides a useful framework for assessing the complexity of the system, as well as the level of effort required for threat modelling and necessary security controls.

Amazon Web Services has developed what it calls the Agentic AI Security Scoping Matrix, recognising that traditional AI security frameworks do not extend naturally into the agentic space. The autonomous nature of agentic systems requires fundamentally different security approaches. The AWS framework categorises four distinct agentic architectures based on connectivity and autonomy levels, mapping critical security controls across each.

The security implications are profound. Research from Galileo AI in December 2025 on multi-agent system failures found that cascading failures propagate through agent networks faster than traditional incident response can contain them. In simulated systems, a single compromised agent poisoned 87 percent of downstream decision-making within four hours.

“When you tie multiple agents together and you allow them to take action based on each other,” noted Paddy Harrington of Forrester Research, security leaders need to rethink how they deploy and govern agentic AI automation before it creates systemic failure.

The problem of non-human identities adds another layer of complexity. According to World Economic Forum research, machine identities now outnumber human employees by a staggering 82 to 1. The rise of autonomous agents, programmed to act on commands without human intervention, introduces a critical vulnerability: a single forged identity can now trigger a cascade of automated actions. The core problem, as the research identifies it, is “billions of unseen, over-permissioned machine identities that attackers, or autonomous agentic AI, will leverage for silent, undetectable lateral movement.”

Trend Micro's 2026 predictions paint an even more concerning picture. The company warns that AI-powered ransomware is evolving into autonomous, agentic systems that automate attacks, target selection, and extortion, amplified by state actors and quantum computing threats. Trend Micro predicts that agentic AI will handle critical portions of the ransomware attack chain, including reconnaissance, vulnerability scanning, and even ransom negotiations, all without human oversight.

“The continued rise of AI-powered ransomware-as-a-service will allow even inexperienced operators to conduct complex attacks with minimal skill,” Trend Micro stated. “This democratisation of offensive capability will greatly expand the threat landscape.”

A Forrester report has predicted that agentic AI will cause a public breach in 2026 that will lead to employee dismissals. Unit 42 believes that attackers will leverage agentic AI to create purpose-built agents with expertise in specific attack stages. When chained together, these AI agents can autonomously test and execute attacks, adjusting tactics in real time based on feedback. These attackers will not just assist with parts of an attack but can plan, adapt, and execute full campaigns end-to-end with minimal human direction.

Jailbreaking at Scale

The vulnerability landscape for large language models presents a particularly vexing challenge for AI coding platforms. The OWASP Foundation recognised the growing threat and listed Prompt Injection as the number one risk in its 2025 OWASP Top 10 for LLM Applications. According to security research, prompt injection dominates as the top production vulnerability, appearing in 73 percent of assessed deployments.

The effectiveness of jailbreaking techniques has reached alarming levels. Research compiled by security teams shows that prompt injections exploiting roleplay dynamics achieved the highest attack success rate at 89.6 percent. These prompts often bypass filters by deflecting responsibility away from the model. Logic trap attacks achieved an 81.4 percent success rate, exploiting conditional structures and moral dilemmas. Encoding tricks using techniques like base64 or zero-width characters achieved a 76.2 percent success rate by evading keyword-based filtering mechanisms.

Multi-turn jailbreak techniques now achieve over 90 percent success rates against frontier models in under 60 seconds. While multi-turn dialogues yielded slightly lower effectiveness at 68.7 percent in some testing scenarios, they often succeeded in long-form tasks where context buildup gradually weakened safety enforcement.

A novel technique called FlipAttack, documented by security researchers at Keysight Technologies, alters character order in prompt messages and achieves an 81 percent average success rate in black box testing. Against GPT-4o specifically, FlipAttack achieved a 98 percent attack success rate and a 98 percent bypass rate against five guardrail models.

The challenge of defending against these attacks is compounded by a fundamental architectural vulnerability. Research from a team examining 12 published defences against prompt injection and jailbreaking found that when subjected to adaptive attacks, the researchers were able to bypass all 12 defences with attack success rates above 90 percent for most, while “the majority of defences originally reported near-zero attack success rate.”

Given the stochastic influence at the heart of how large language models work, it remains unclear whether fool-proof methods of prevention for prompt injection even exist. This represents a fundamental architectural vulnerability requiring defence-in-depth approaches rather than singular solutions.

The “salami slicing” attack represents a particularly insidious threat to agentic systems. In this approach, an attacker might submit multiple support tickets over a week, each one slightly redefining what an AI agent should consider “normal” behaviour. By the final ticket, the agent's constraint model has drifted so far that it performs unauthorised actions without detecting the manipulation. Each individual prompt appears innocuous. The cumulative effect proves catastrophic.

Research from Palo Alto Networks' Unit 42 in October 2025 on persistent prompt injection showed that agents with long conversation histories are significantly more vulnerable to manipulation. An agent that has discussed policies for 50 exchanges might accept a 51st exchange that contradicts the first 50, especially if the contradiction is framed as a “policy update.”

Memory poisoning poses similar risks. Attackers can create support tickets requesting an agent to “remember” malicious instructions that get stored in its persistent memory context. Weeks later, when legitimate transactions occur, the agent recalls the planted instruction and takes unauthorised actions. The compromise is latent, making it nearly impossible to detect with traditional anomaly detection methods.

Building Graduated Autonomy Controls

Against this backdrop of escalating threats, the concept of graduated autonomy has emerged as a potential framework for balancing capability with security. The approach recognises that not all users present equal risk, and not all tasks require equal levels of AI autonomy.

Anthropic has implemented multiple layers of security controls in Claude Code. The company released sandboxing capabilities that establish two security boundaries. The first boundary provides filesystem isolation, ensuring that Claude can only access or modify specific directories. The second provides network isolation. Anthropic emphasises that both isolation techniques must work together for effective protection. Without network isolation, a compromised agent could exfiltrate sensitive files like SSH keys. Without filesystem isolation, a compromised agent could escape the sandbox and gain network access.

The company has also patched specific vulnerabilities identified by security researchers, including CVE-2025-54794 (path restriction bypass) and CVE-2025-54795 (command injection).

Anthropic is preparing to launch a Security Center for Claude Code, offering users an overview of security scans, detected issues, and manual scan options in one place. The security-review command lets developers run ad-hoc security analysis before committing code, checking for SQL injection risks, cross-site scripting errors, authentication and authorisation flaws, and insecure data handling.

However, Anthropic has acknowledged the fundamental challenge. The company has stated that while they have built a multi-layer defence mechanism against prompt injection, “agent security” remains a cutting-edge issue that the entire industry is actively exploring.

The NIST AI Risk Management Framework provides a broader governance structure for these challenges. In December 2025, the US National Institute of Standards and Technology published a preliminary draft of the Cybersecurity Framework Profile for Artificial Intelligence. The guidelines focus on three overlapping areas: securing AI systems, conducting AI-enabled cyber defence, and thwarting AI-enabled cyberattacks.

The NIST framework's 2025 updates expand coverage to address generative AI, supply chain vulnerabilities, and new attack models. The AI Risk Management Framework now aligns more closely with cybersecurity and privacy frameworks, simplifying cross-framework compliance. Companion resources include the Control Overlays for Securing AI Systems (COSAIS) concept paper from August 2025, which outlines a framework to adapt existing federal cybersecurity standards (specifically SP 800-53) for AI-specific vulnerabilities.

The EU AI Act provides another regulatory lens. In force since August 2024, it establishes the world's first comprehensive legal framework for AI systems. The act adopts a risk-based approach, categorising AI systems from minimal to unacceptable risk. Article 15 imposes standards for accuracy, robustness, and cybersecurity for high-risk AI systems. Providers of general-purpose AI models that present systemic risk must conduct model evaluations, adversarial testing, track and report serious incidents, and ensure cybersecurity protections.

The EU framework specifically addresses models trained with computational power exceeding 10 to the 25th power floating point operations, subjecting them to enhanced obligations including rigorous risk assessments and serious incident reporting requirements. Providers must implement state-of-the-art evaluation protocols and maintain robust incident response capabilities.

For AI coding platforms specifically, the governance challenge requires developer-level controls that go beyond simple content filtering. Research from Stanford University has shown that developers who used an AI assistant “wrote significantly less secure code than those without access to an assistant,” while also tending to be “overconfident about security flaws in their code.” This finding suggests that graduated autonomy must include not just restrictions on AI capabilities but also mechanisms to ensure users understand the security implications of AI-generated code.

Solutions like Secure Code Warrior's Trust Agent provide CISOs with security traceability, visibility, and governance over developers' use of AI coding tools. These platforms inspect AI-generated code traffic by deploying as IDE plugins, leveraging signals including AI coding tool usage, vulnerability data, code commit data, and developers' secure coding skills.

Distinguishing Development from Reconnaissance

One of the most technically challenging aspects of securing AI coding platforms is distinguishing between legitimate iterative development and malicious reconnaissance-exploitation chains. Both activities involve querying the AI repeatedly, refining prompts based on results, and building toward a complex final output. The difference lies in intent, which is notoriously difficult to infer from behaviour alone.

Behavioural anomaly detection offers one potential approach. According to security research from Darktrace and other firms, anomaly detection builds behavioural baselines through the analysis of historical and real-time data. Techniques such as machine learning and advanced statistical methods isolate key metrics like login frequency and data flow volumes to define the parameters of normal activity. Advanced anomaly detection AI systems employ unsupervised learning to detect outliers in large, unlabelled datasets, while supervised models use labelled examples of attacks to refine detection.

However, insider threats remain one of the most challenging security risks precisely because of the difficulty in distinguishing malicious intent from legitimate activity. Recurrent neural networks can consider the context of each action within a software's behaviour, distinguishing legitimate activities from malicious ones. But the challenge intensifies with AI coding tools, where the boundary between creative exploration and attack preparation is inherently fuzzy.

Contextual anomalies provide some detection capability. A large file transfer might be acceptable during business hours but suspicious if conducted late at night. Collective anomalies involve groups of data points that deviate from normal patterns together, such as systems communicating simultaneously with a malicious server or coordinated attack patterns.

For AI coding platforms, potential indicators of malicious reconnaissance might include: rapid sequential queries about network penetration techniques, vulnerability exploitation, and credential harvesting; requests that progressively escalate in specificity, moving from general security concepts to targeted exploitation of particular systems; patterns of prompt refinement that suggest the user is testing the AI's boundaries rather than developing functional software; and unusual session lengths or request frequencies that deviate from typical developer behaviour.

However, each of these indicators could also characterise a legitimate security researcher, a penetration tester with proper authorisation, or a developer building defensive security tools. The challenge lies in developing detection mechanisms sophisticated enough to distinguish context.

AWS's Agentic AI Security Scoping Matrix recommends implementing comprehensive monitoring of agent actions during autonomous execution phases and establishing clear agency boundaries for agent operations. Critical concerns include securing the human intervention channel, preventing scope creep during task execution, monitoring for behavioural anomalies, and validating that agents remain aligned with original human intent.

Modern behavioural systems prioritise alerts by risk level, automatically suppressing benign anomalies while escalating genuine threats for investigation and response. When behavioural systems alert, they include the full context: what the user typically does, how the current activity differs, related events across the timeline, and risk scoring based on asset criticality.

The Open Source Displacement Problem

A fundamental critique of restricting agentic features on commercial platforms is that such restrictions merely displace risk to less-regulated open-source alternatives rather than genuinely mitigating the threat. This argument carries significant weight.

Research on the DeepSeek R1 frontier reasoning model revealed what researchers characterised as “critical safety flaws.” In testing, DeepSeek failed to block a single harmful prompt when tested against 50 random prompts taken from the HarmBench dataset. Researchers found that DeepSeek is more susceptible to jailbreaking than its counterparts, with attackers able to bypass its “weak safeguards” to generate harmful content with “little to no specialised knowledge or expertise.”

The Global Center on AI research has documented how open-source AI models, when used by malicious actors, may pose serious threats to international peace, security, and human rights. Highly capable open-source models could be repurposed to perpetuate crime, harm, or disrupt democratic processes. Deepfakes generated using such models have been used to influence election processes, spread misinformation, and aggravate tensions in conflict-prone regions.

This reality creates a genuine dilemma for platform providers. If Anthropic, OpenAI, Google, and other major providers implement stringent graduated autonomy controls, sophisticated attackers may simply migrate to unrestricted open-source alternatives. The security measures would then primarily affect legitimate developers while having minimal impact on determined threat actors.

However, this argument has limitations. First, commercial AI coding platforms provide significant infrastructure advantages that open-source alternatives cannot easily replicate, including integration with enterprise development environments, technical support, regular security updates, and compliance certifications. Many organisations cannot practically migrate their development workflows to unvetted open-source models.

Second, the security controls implemented by major platforms establish industry norms and expectations. When leading providers demonstrate that graduated autonomy is technically feasible and practically implementable, they create pressure on the broader ecosystem to adopt similar approaches.

Third, the argument assumes that restricting commercial platforms would have no impact on threat actors, but the Anthropic espionage incident demonstrates otherwise. The GTG-1002 threat group specifically targeted Claude Code, suggesting that even sophisticated state-sponsored actors see value in leveraging commercial AI infrastructure. Making that infrastructure more difficult to abuse imposes real costs on attackers, even if it does not eliminate the threat entirely.

The OWASP GenAI Security Project recommends that security considerations should be embedded into the development and release of open-source AI models with safety protocols, fail-safes, and built-in safeguards. This requires adversarial testing, ethical hacking to exploit vulnerabilities, and red-teaming to simulate real-world threats.

Systemic Safeguards for an Industry

Beyond individual platform controls, the AI industry faces pressure to adopt systemic safeguards that address the democratisation of offensive capabilities. Several frameworks have emerged to guide this effort.

The NIST Cybersecurity Framework Profile for AI centres on three overlapping focus areas: securing AI systems, conducting AI-enabled cyber defence, and thwarting AI-enabled cyberattacks. This tripartite approach recognises that AI security is not simply about preventing misuse but also about leveraging AI for defensive purposes and anticipating AI-enabled threats.

At the European level, the AI Act requires providers of general-purpose AI models with systemic risk to implement state-of-the-art evaluation protocols, conduct adversarial testing, and maintain robust incident response capabilities. Cybersecurity measures must include protection against unauthorised access, insider threat mitigation, and secure model weight protection.

Industry-specific guidance has also emerged. The OpenSSF Best Practices Working Group has published a Security-Focused Guide for AI Code Assistant Instructions, providing recommendations for organisations deploying AI coding tools. Research from Palo Alto Networks recommends that organisations consider LLM guardrail limitations when building open-source LLMs into business processes, noting that guardrails can be broken and that safeguards need to be built in at the organisational level.

For AI coding platforms specifically, systemic safeguards might include: mandatory reporting of security incidents involving AI-enabled attacks, similar to the breach notification requirements that exist in data protection regulation; standardised APIs for security monitoring that allow enterprise customers to integrate AI coding tools with their existing security infrastructure; industry collaboration on threat intelligence sharing, enabling platform providers to rapidly disseminate information about novel jailbreaking techniques and malicious use patterns; graduated capability unlocking based on verified identity and demonstrated legitimate use cases; and integration with existing enterprise identity and access management systems.

The Limits of Technical Controls

Ultimately, graduated autonomy controls and detection mechanisms represent necessary but insufficient responses to the weaponisation of agentic AI. Technical controls can raise the barrier for misuse, but they cannot eliminate the fundamental dual-use nature of powerful AI systems.

The 25-minute AI-powered ransomware attack demonstrated by Unit 42 would still be possible with restricted commercial platforms if the attacker were willing to invest more time in circumventing controls. The Anthropic espionage campaign succeeded despite existing safety measures because the attacker found a social engineering approach that convinced the AI it was operating in a legitimate defensive context.

This reality points toward the need for complementary approaches beyond technical controls. Regulatory frameworks like the EU AI Act establish legal accountability for AI providers and high-risk systems. Law enforcement capacity must evolve to investigate and prosecute AI-enabled crime effectively. International cooperation is essential given the borderless nature of cyber threats.

The security research community has called for a paradigm shift in how organisations approach AI risk. Trend Micro recommends that organisations adopt proactive AI defences, zero-trust architectures, and quantum-safe cryptography to counter escalating cyber risks. The World Economic Forum has emphasised the critical need for visibility into non-human identities, noting that machine identities now outnumber human employees by 82 to 1.

Palo Alto Networks warns that adversaries will no longer make humans their primary target. Instead, they will look to compromise powerful AI agents, turning them into “autonomous insiders.” This shift requires security strategies that treat AI systems as potential attack vectors, not just as tools.

A defining trend in 2025 was the emergence of violence-as-a-service networks. Criminal groups are increasingly using digital platforms such as Telegram to coordinate physical attacks, extortion, and sabotage tied to ransomware or cryptocurrency theft. Hybrid adversaries operate at the intersection of cybercrime and physical crime, offering financial incentives for real-world violence against corporate targets. This convergence of digital and physical threats represents a new frontier that purely technical controls cannot address.

Navigating Uncertain Territory

The question of whether restricting agentic features creates a false sense of security admits no simple answer. On one hand, restrictions implemented by responsible providers demonstrably complicate attack chains and impose costs on malicious actors. The Anthropic incident, despite its severity, also demonstrated the value of platform-level detection and response capabilities. The threat actor was identified and disrupted in part because they operated within a monitored commercial environment.

On the other hand, determined and well-resourced adversaries will find ways to access powerful AI capabilities regardless of individual platform restrictions. The existence of WormGPT, KawaiiGPT, and other unrestricted models proves that the genie cannot be returned to the bottle through commercial platform controls alone.

The most honest assessment may be that graduated autonomy controls are a necessary component of a defence-in-depth strategy, but should not be mistaken for a complete solution. They buy time, raise costs for attackers, and provide detection opportunities. They do not prevent motivated threat actors from eventually achieving their objectives.

For legitimate developers, the calculus is more straightforward. Graduated autonomy that requires additional verification for sensitive capabilities imposes modest friction in exchange for meaningful security benefits. Developers working on legitimate projects rarely need unrestricted access to every possible AI capability. A system that requires additional justification for generating network exploitation code or analysing credential databases is not meaningfully impeding software development.

The key is ensuring that graduated controls are implemented thoughtfully, with clear escalation paths for legitimate use cases and transparent criteria for capability unlocking. Security measures that frustrate legitimate users without meaningfully impacting threat actors represent the worst of both worlds.

As the AI industry matures, the organisations building agentic AI coding platforms face a defining choice. They can pursue capability at all costs, accepting the security externalities as the price of progress. Or they can invest in the harder work of graduated autonomy, behavioural detection, and systemic safeguards, building trust through demonstrated responsibility.

The Anthropic espionage campaign revealed that even well-intentioned AI systems can be weaponised at scale. The response to that revelation will shape whether agentic AI becomes a net positive for software development or an accelerant for cybercrime. The technology itself is neutral. The choices made by its creators are not.

---

References and Sources

1. Anthropic. “Disrupting the First Reported AI-Orchestrated Cyber Espionage Campaign.” November 2025. https://www.anthropic.com/news/disrupting-AI-espionage

2. Palo Alto Networks Unit 42. “AI Agents Are Here. So Are the Threats.” 2025. https://unit42.paloaltonetworks.com/agentic-ai-threats/

3. Palo Alto Networks Unit 42. “The Dual-Use Dilemma of AI: Malicious LLMs.” 2025. https://unit42.paloaltonetworks.com/dilemma-of-ai-malicious-llms/

4. Palo Alto Networks Unit 42. “2025 Unit 42 Global Incident Response Report: Social Engineering Edition.” 2025. https://unit42.paloaltonetworks.com/2025-unit-42-global-incident-response-report-social-engineering-edition/

5. World Economic Forum. “Cybersecurity Awareness: AI Threats and Cybercrime in 2025.” September 2025. https://www.weforum.org/stories/2025/09/cybersecurity-awareness-month-cybercrime-ai-threats-2025/

6. World Economic Forum. “Non-Human Identities: Agentic AI's New Frontier of Cybersecurity Risk.” October 2025. https://www.weforum.org/stories/2025/10/non-human-identities-ai-cybersecurity/

7. NVIDIA Technical Blog. “Agentic Autonomy Levels and Security.” 2025. https://developer.nvidia.com/blog/agentic-autonomy-levels-and-security/

8. Amazon Web Services. “The Agentic AI Security Scoping Matrix: A Framework for Securing Autonomous AI Systems.” 2025. https://aws.amazon.com/blogs/security/the-agentic-ai-security-scoping-matrix-a-framework-for-securing-autonomous-ai-systems/

9. OWASP. “LLM01:2025 Prompt Injection.” 2025. https://genai.owasp.org/llmrisk/llm01-prompt-injection/

10. Keysight Technologies. “Prompt Injection Techniques: Jailbreaking Large Language Models via FlipAttack.” May 2025. https://www.keysight.com/blogs/en/tech/nwvs/2025/05/20/prompt-injection-techniques-jailbreaking-large-language-models-via-flipattack

11. NIST. “Draft NIST Guidelines Rethink Cybersecurity for the AI Era.” December 2025. https://www.nist.gov/news-events/news/2025/12/draft-nist-guidelines-rethink-cybersecurity-ai-era

12. European Commission. “AI Act: Regulatory Framework for AI.” 2024-2025. https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai

13. Trend Micro. “The AI-fication of Cyberthreats: Trend Micro Security Predictions for 2026.” 2025. https://www.trendmicro.com/vinfo/us/security/research-and-analysis/predictions/the-ai-fication-of-cyberthreats-trend-micro-security-predictions-for-2026

14. SANS Institute. “AI-Powered Ransomware: How Threat Actors Weaponize AI Across the Attack Lifecycle.” 2025. https://www.sans.org/blog/ai-powered-ransomware-how-threat-actors-weaponize-ai-across-attack-lifecycle

15. Cyble. “Top 10 Threat Actor Trends of 2025 and Signals for 2026.” 2025. https://cyble.com/knowledge-hub/top-10-threat-actor-trends-of-2025/

16. InfoQ. “Anthropic Adds Sandboxing and Web Access to Claude Code for Safer AI-Powered Coding.” November 2025. https://www.infoq.com/news/2025/11/anthropic-claude-code-sandbox/

17. Checkmarx. “2025 CISO Guide to Securing AI-Generated Code.” 2025. https://checkmarx.com/blog/ai-is-writing-your-code-whos-keeping-it-secure/

18. Darktrace. “Anomaly Detection: Definition and Security Solutions.” 2025. https://www.darktrace.com/cyber-ai-glossary/anomaly-detection

19. UC Berkeley Center for Long-Term Cybersecurity. “Beyond Phishing: Exploring the Rise of AI-enabled Cybercrime.” January 2025. https://cltc.berkeley.edu/2025/01/16/beyond-phishing-exploring-the-rise-of-ai-enabled-cybercrime/

20. Global Center on AI. “The Global Security Risks of Open-Source AI Models.” 2025. https://www.globalcenter.ai/research/the-global-security-risks-of-open-source-ai-models

21. Secure Code Warrior. “Trust Agent AI: CISO Visibility into Developer AI Tool Usage.” September 2025. https://www.helpnetsecurity.com/2025/09/25/secure-code-warrior-trust-agent-ai/

22. OpenSSF Best Practices Working Group. “Security-Focused Guide for AI Code Assistant Instructions.” 2025. https://best.openssf.org/Security-Focused-Guide-for-AI-Code-Assistant-Instructions

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...