SmarterArticles

The promise was straightforward enough. Large language models, trained on the sum total of medical literature, would help emergency physicians triage patients faster, assist radiologists in catching what the human eye missed, and give overwhelmed clinicians a second opinion when the waiting room was full and the clock was running. The reality, according to a growing body of peer-reviewed research, is considerably more uncomfortable. The most capable AI systems available today do not simply reflect the biases embedded in their training data. They amplify them, sometimes dramatically, and they do so in clinical contexts where the consequences land on real human bodies.

In September 2025, a team of researchers led by Mahmud Omar and Eyal Klang at the Icahn School of Medicine at Mount Sinai posted a preprint on medRxiv that tested OpenAI's GPT-5 across 500 physician-validated emergency department vignettes. Each case was replayed 32 times, with the only variable being the sociodemographic label attached to the patient: Black, white, low-income, high-income, LGBTQIA+, unhoused, and so on. The clinical details remained identical. The model's recommendations did not.

GPT-5 showed no improvement in sociodemographic-linked decision variation compared with its predecessor, GPT-4o. On several measures, it was worse. The model assigned higher urgency and recommended less advanced testing for historically marginalised groups. Most striking was the mental health screening disparity: several LGBTQIA+ labels were flagged for mental health evaluation in 100 per cent of cases, compared with roughly 41 to 73 per cent for comparable demographic groups under GPT-4o. The clinical presentation was the same. The only thing that changed was who the patient was described as being.

This is not a theoretical problem. It is a design problem, a procurement problem, and increasingly a legal problem. And it raises a question that hospitals, insurers, and diagnostic tool developers have been remarkably slow to answer: if the most advanced AI model on the market still encodes the biases of the data it was trained on, what exactly are institutions assuming when they plug these systems into patient care?

The Evidence Is Not Subtle

The Mount Sinai findings did not emerge from a vacuum. They are the latest in a pattern of research that has been building for years, each study confirming what the last one suggested and what the next one will almost certainly reinforce.

The same research team published a broader companion study in Nature Medicine in 2025, evaluating nine large language models across more than 1.7 million model-generated outputs from 1,000 emergency department cases (500 real, 500 synthetic). Each case was presented in 32 variations, covering 31 sociodemographic groups plus a control, while clinical details were held constant. Cases labelled as Black, unhoused, or LGBTQIA+ were more frequently directed toward urgent care, invasive interventions, or mental health evaluations. Certain LGBTQIA+ subgroups were recommended mental health assessments approximately six to seven times more often than was clinically indicated. The bias was not confined to one model or one developer. It was a property of the category.

In 2024, Travis Zack and colleagues published a model evaluation study in The Lancet Digital Health examining GPT-4's behaviour across clinical applications including medical education, diagnostic reasoning, clinical plan generation, and subjective patient assessment. The results were damning. GPT-4 failed to model the demographic diversity of medical conditions, instead producing clinical vignettes that stereotyped demographic presentations. When generating differential diagnoses, the model was more likely to include diagnoses that stereotyped certain races, ethnicities, and genders. It exaggerated known demographic prevalence differences in 89 per cent of diseases tested. Assessment and treatment plans showed significant associations between demographic attributes and recommendations for more expensive procedures, as well as measurable differences in how patients were perceived. For 23 per cent of cases, GPT-4 produced significantly different patient perception responses based solely on gender or race and ethnicity.

The broader research landscape tells a consistent story. A systematic review published in 2025 in the International Journal for Equity in Health, encompassing 24 studies evaluating demographic disparities in medical large language models, found that 22 of those studies, or 91.7 per cent, identified biases. Gender bias was the most prevalent, reported in 15 of 16 studies examining it (93.7 per cent). Racial or ethnic biases appeared in 10 of 11 studies (90.9 per cent). These are not edge cases. They are the norm.

And the problem extends well beyond language models. In dermatology, AI models trained primarily on lighter skin tones have consistently shown lower diagnostic performance for lesions on darker skin. A 2025 study in the Journal of the European Academy of Dermatology and Venereology found that among 4,000 AI-generated dermatological images, only 10.2 per cent depicted dark skin, and just 15 per cent accurately represented the intended condition. Meanwhile, analyses of dermatology textbooks used to train both human clinicians and AI systems have shown that images of dark skin make up as little as 4 to 18 per cent of the total. A 2022 study published in Science Advances confirmed that AI diagnostic performance for dermatological conditions was measurably worse on darker skin tones, a disparity directly traceable to training data composition.

The consequences are not abstract. Individuals with darker skin tones who develop melanoma are more likely to present with advanced-stage disease and experience lower survival rates. An AI system that performs poorly on these patients does not merely fail a technical benchmark. It compounds an existing disparity. And a 2024 study from Northwestern University found that even when AI tools themselves were calibrated for fairness, the interaction between physicians and AI-assisted diagnosis actually widened the accuracy gap between patients with light and dark skin tones, suggesting that the problem cannot be solved at the algorithm level alone.

When Machines Hallucinate in the Emergency Room

Bias is not the only vulnerability. In August 2025, a study published in Communications Medicine, a Nature Portfolio journal, tested six leading large language models with 300 clinician-designed vignettes, each containing a single fabricated element: a fake lab value, a nonexistent sign, or an invented disease. The results were striking. The models repeated or elaborated on the planted error in up to 83 per cent of cases. A simple mitigation prompt halved the overall hallucination rate, from a mean of 66 per cent across all models to 44 per cent. For the best-performing model in the study, GPT-4o, rates declined from 53 per cent to 23 per cent. Temperature adjustments, often proposed as a fix for hallucination, offered no significant improvement. Shorter vignettes showed slightly higher odds of hallucination.

For GPT-5 specifically, the Mount Sinai preprint found that its unmitigated adversarial hallucination rate was higher than that observed for GPT-4o. The same mitigation technique achieved a lower rate than before, meaning the baseline risk was worse even as the ceiling for improvement was slightly better.

The clinical implications are severe. If a language model is deployed as a clinical decision support tool and a patient's record contains an erroneous data point, whether through transcription error, system glitch, or adversarial input, the model is more likely to incorporate that error into its reasoning than to flag it as anomalous. It will confabulate around the mistake, generating plausible-sounding but clinically dangerous recommendations. The model does not know what it does not know, and it cannot distinguish between a real lab result and a fabricated one.

This is not a bug that can be patched with a software update. It is a structural property of how these models process information. They are optimised to produce coherent, contextually appropriate text, not to distinguish between real clinical findings and fabricated ones. The distinction matters enormously when the output influences whether a patient receives a chest X-ray or is sent home.

Who Bears the Cost

The populations most affected by AI bias in healthcare are, with grim predictability, those who already face the greatest barriers to adequate care. Racial minorities, women, elderly patients, LGBTQIA+ individuals, people experiencing homelessness, and low-income populations appear repeatedly in the literature as groups for whom AI systems produce systematically different, and often inferior, clinical recommendations.

The Mount Sinai study found a clear socioeconomic gradient in testing recommendations. GPT-5 directed less advanced diagnostic testing toward lower-income groups, with a negative 7.0 per cent deviation for low-income patients and a negative 6.8 per cent deviation for middle-income patients, while high-income patients received a positive 2.2 per cent deviation. Same symptoms, different workups, determined entirely by a label the model should have been ignoring.

The pulse oximetry debacle offers a useful precedent for understanding how bias in medical technology compounds racial health disparities. Research published in the New England Journal of Medicine demonstrated that pulse oximeters systematically overestimated blood oxygen levels in Black patients, with the frequency of occult hypoxaemia that went undetected being three times greater among Black patients compared with white patients. During the COVID-19 pandemic, this meant Black patients were less likely to receive supplemental oxygen when they needed it. The FDA released new draft guidance in January 2025 with updated testing standards, recommending a minimum of 24 subjects from across the Monk Skin Tone scale for clinical studies. But the damage from years of deployment with known racial bias had already been done. As Health Affairs Forefront noted in January 2025, the imperative to develop cross-racial pulse oximeters was “overdue” by any reasonable measure.

The pattern is consistent: a technology is developed, tested primarily on populations that do not represent the full range of patients who will encounter it, deployed at scale, and then studied retrospectively when the harm becomes impossible to ignore. AI in healthcare is following this trajectory with remarkable fidelity.

Sepsis prediction offers another cautionary tale. Epic Systems deployed its widely used Epic Sepsis Model across hundreds of hospitals. When researchers at Michigan Medicine analysed roughly 38,500 hospitalisations, they found the algorithm missed two-thirds of sepsis patients and generated numerous false alerts. A 2025 study published in the American Journal of Bioethics highlighted that social determinants of health data, which disproportionately affect minority and low-income populations, were notoriously underrepresented in the electronic health record data used to train such models, with only 3 per cent of sentences in examined training datasets containing any mention of social determinants. The algorithm did not account for what it could not see, and what it could not see was shaped by who had historically been rendered invisible in medical data systems.

The Institutional Wager

When a hospital system integrates AI into its clinical workflows, it is making a bet. The bet is that the efficiency gains, the reduced clinician workload, and the potential for catching diagnoses that might otherwise be missed will outweigh the risks of systematic error. It is a bet that the tool will perform roughly as well for all patients, or at least that any disparities will be caught by the human clinicians who remain in the loop.

Both assumptions are questionable.

Epic Systems, which commands 42.3 per cent of the acute care electronic health record market in the United States with over 305 million patient records, has rolled out generative AI enhancements for clinical messaging, charting, and predictive modelling. By 2025, the company reported between 160 and 200 active AI projects, with over 150 AI features in development for 2026, including native AI-assisted charting tools, new AI assistants, and advanced predictive models. In February 2026, Epic launched AI Charting, an ambient scribe feature that listens to patient visits and automatically drafts clinical notes and orders. Oracle Health, following its acquisition of Cerner, debuted an entirely new AI-powered EHR in 2025, featuring a clinical AI agent that drafts documentation, proposes lab tests and follow-up visits, and automates coding. The agent is now live across more than 30 medical specialities and has reportedly reduced physician documentation time by nearly 30 per cent.

The efficiency argument is real. But efficiency and equity are not the same thing. When these systems produce different outputs based on demographic characteristics, as the peer-reviewed evidence consistently shows they do, the “human in the loop” defence becomes critical. It also becomes fragile. A clinician reviewing AI-generated notes under time pressure, in a system designed to reduce their workload, is not in an ideal position to catch the subtle ways in which the model's recommendations may have been shaped by the patient's race, gender, or income level rather than their clinical presentation.

The assumption that humans will catch AI errors is further undermined by automation bias, the well-documented tendency for people to defer to automated systems, particularly when those systems present their outputs with confidence and fluency. A November 2024 study examining pathology experts found that AI integration, while improving overall diagnostic performance, resulted in a 7 per cent automation bias rate where initially correct evaluations were overturned by erroneous AI advice. A separate study of gastroenterologists using AI tools found measurable deskilling over time: clinicians became less proficient at identifying polyps independently after a period of AI-assisted practice. A large language model does not hedge. It does not say “I am less certain about this recommendation because the patient is Black.” It produces a clean, authoritative-sounding clinical note, and the bias is invisible unless someone is specifically looking for it.

The Insurance Question

The integration of AI into healthcare is not limited to clinical decision-making. Insurers have been among the most aggressive adopters, and the consequences are already being litigated.

UnitedHealth Group, the largest health insurer in the United States, is facing a class-action lawsuit alleging that its AI tool, nH Predict, developed by its subsidiary naviHealth (acquired in 2020 for over one billion dollars), was used to systematically deny medically necessary coverage for post-acute care. The plaintiffs, who include Medicare Advantage policyholders, allege that the algorithm superseded physician judgment and had a 90 per cent error rate, meaning nine of ten appealed denials were ultimately reversed.

In February 2025, a federal court denied UnitedHealth's motion to dismiss, allowing breach of contract and good faith claims to proceed. The court noted that the case turned on whether UnitedHealth had violated its own policy language, which stated that coverage decisions would be made by clinical staff or physicians, not by an algorithm. A judge subsequently ordered UnitedHealth to produce tens of thousands of internal documents related to the algorithm's deployment by April 2025.

This case is significant not only for its specific allegations but for the structural question it raises. When an insurer deploys an AI system to make coverage decisions, and that system denies care at scale, who is accountable? The algorithm's developers? The insurer's management? The clinicians whose judgment the algorithm overrode? The regulatory framework has no clear answer, and in the absence of clarity, the cost falls on the patients who are denied coverage and must navigate an appeals process that many, particularly elderly and low-income individuals, are ill-equipped to pursue. The asymmetry is stark: the insurer benefits from the speed and scale of algorithmic denial, while the patient bears the burden of proving, one appeal at a time, that the machine was wrong.

The Regulatory Vacuum

Regulatory bodies are aware of the problem. Their responses have been uneven at best.

The United States Food and Drug Administration has authorised over 1,250 AI-enabled medical devices as of July 2025, up from 950 in August 2024. The pace of authorisation is accelerating even as the evidence of bias accumulates. The agency published draft guidance in January 2025 on lifecycle management for AI-enabled devices, introducing the concept of Predetermined Change Control Plans, which allow developers to obtain pre-approval for planned algorithmic updates. This is a meaningful step toward continuous monitoring. But the guidance focuses primarily on safety and effectiveness in technical terms, with limited attention to the question of whether a device performs equitably across demographic groups.

In June 2025, a report published in PLOS Digital Health, authored by researchers from the University of Toronto, MIT, and Harvard, laid bare the scale of the regulatory gap. Titled “The Illusion of Safety,” the report found that many AI-enabled tools were entering clinical use without rigorous evaluation or meaningful public scrutiny. Critical details such as testing procedures, validation cohorts, and bias mitigation strategies were often missing from approval submissions. The authors identified inconsistencies in how the FDA categorises and approves these technologies, and noted that AI's continuous learning capabilities introduce unique risks: algorithms evolve beyond their initial validation, potentially leading to performance degradation and biased outcomes that the current regulatory framework is not designed to detect.

In January 2026, the FDA released further guidance that actually reduced oversight of certain low-risk digital health products, including AI-enabled software and clinical decision support tools. The reasoning was that lighter regulation would encourage innovation. The concern is that it will also encourage deployment without adequate bias testing. The tension between promoting innovation and protecting patients is not new in medical device regulation, but the speed at which AI tools are proliferating makes the stakes unusually high.

The European Union has taken a more structured approach. Under the EU AI Act, which began phased implementation in August 2025, AI systems used as safety components in medical devices are classified as high-risk and subject to stringent requirements: risk management systems, technical documentation, training data governance, transparency, human oversight, and post-market monitoring. Full compliance for high-risk AI systems in healthcare is required by August 2027. The framework is more comprehensive than its American counterpart, but enforcement mechanisms remain untested, and the practical challenge of auditing AI systems for demographic bias at scale is formidable. The European Commission is expected to issue guidelines on practical implementation of high-risk classification by February 2026, including examples of what constitutes high-risk and non-high-risk use cases.

The World Health Organisation released guidance in January 2024 on the ethics and governance of large multimodal models in healthcare, outlining over 40 recommendations organised around six principles: protecting autonomy, promoting well-being and safety, ensuring transparency and explainability, fostering responsibility and accountability, ensuring inclusiveness and equity, and promoting responsive and sustainable AI. The principles are sound. Whether they translate into enforceable standards is another matter entirely. The WHO's Global Initiative on Artificial Intelligence for Health has been working to advance governance frameworks particularly in low- and middle-income countries, where the regulatory infrastructure to evaluate AI tools may be even less developed than in the United States or Europe.

The gap between what regulators recognise as a problem and what they are prepared to do about it remains wide. And in that gap, hospitals and insurers continue to deploy systems whose bias profiles have been documented in peer-reviewed literature but not addressed in procurement requirements.

Accountability Without a Framework

The liability question is perhaps the most unsettled aspect of AI in healthcare. Current legal frameworks were not designed for systems that learn, change, and produce different outputs for different patients based on patterns in training data that no human selected or reviewed.

If an AI clinical decision support tool recommends a less aggressive workup for a Black patient than for a white patient with identical symptoms, and the Black patient's condition is missed, who is liable? The developer who trained the model? The hospital that purchased and deployed it? The clinician who accepted the recommendation without questioning it? Under existing product liability regimes, device manufacturers are often shielded, and the burden tends to fall on clinicians and institutions. But clinicians did not design the algorithm, may not understand its internal workings, and in many cases were not consulted about the decision to deploy it.

Professional medical societies have generally maintained that clinicians retain ultimate responsibility for patient care, regardless of the tools they use. This position is legally and ethically coherent, but it places an extraordinary burden on individual practitioners to detect and override biases that are, by design, invisible in the model's outputs. It also creates a perverse incentive structure: the institutions that benefit from AI efficiency (reduced labour costs, faster throughput, fewer staff) externalise the liability risk to frontline clinicians who had no say in the technology's selection or implementation.

New legislation has been proposed in the United States to clarify AI liability in healthcare, but none has yet been enacted. The result is a regulatory and legal environment in which the technology is advancing faster than the frameworks meant to govern it, with patients and clinicians left to absorb the consequences of that mismatch.

What Meaningful Reform Requires

The research community has not merely identified the problem. It has outlined what solutions would look like. The challenge is that those solutions require effort, money, and institutional will that the current market incentives do not reliably produce.

First, training data must be representative. The persistent underrepresentation of dark-skinned patients in dermatological datasets, of women in cardiovascular research, and of LGBTQIA+ individuals in clinical trial data is not a new problem. But when that data is used to train AI systems that are then deployed at scale, the bias is industrialised. Studies have demonstrated that fine-tuning AI models on diverse datasets closes performance gaps between demographic groups. The data exists, or could be collected. The question is whether developers and institutions are willing to invest in obtaining it.

Second, pre-deployment bias auditing must become mandatory, not optional. The evidence that AI systems produce systematically different outputs based on demographic labels is overwhelming. Yet there is no requirement in the United States that an AI clinical tool be tested for demographic equity before it is integrated into a hospital's workflow. The EU AI Act moves in this direction with its training data governance and risk management requirements for high-risk systems, but enforcement remains a future proposition.

Third, post-deployment monitoring must be continuous and transparent. The FDA's introduction of Predetermined Change Control Plans is a step toward lifecycle accountability, but the focus remains on technical safety rather than equitable performance. An AI system that performs well on average but poorly for specific subpopulations is not safe for those subpopulations, and average performance metrics can obscure the disparity. The “Illusion of Safety” report's finding that the FDA's current framework is ill-equipped to monitor post-approval algorithmic drift makes this point with particular force.

Fourth, procurement processes must include bias testing as a criterion. Hospitals that would never purchase a pharmaceutical product without evidence of efficacy across demographic groups are integrating AI tools with no comparable requirement. The Mount Sinai research provides a template: test the system across sociodemographic labels, measure the variation, and make the results public before deployment. If a model produces different triage recommendations for patients labelled as low-income versus high-income, that information should be available to every hospital considering its adoption.

Fifth, liability frameworks must be updated. If AI systems are going to influence clinical decisions, the legal structures governing those decisions must account for the technology's role. This means clearer allocation of responsibility between developers, deployers, and users, and it means creating mechanisms for patients to seek redress when biased AI contributes to harm. The UnitedHealth litigation may ultimately push courts to establish precedents, but waiting for case law to fill a regulatory void is not a strategy; it is an abdication.

Finally, transparency must become the default. Patients have a right to know when AI has influenced their care, what role it played, and whether the system has been tested for bias relevant to their demographic group. This is not merely an ethical aspiration. In an era when AI-generated clinical notes may shape everything from triage decisions to insurance coverage, it is a basic requirement of informed consent. The WHO's guidance on transparency and explainability points in this direction, but voluntary principles are no substitute for binding obligations.

The Stakes Are Not Abstract

The title of the Mount Sinai medRxiv preprint captures the situation with precision: “New Model, Old Risks.” GPT-5 is, by most technical measures, a more capable system than its predecessors. It is also, by the evidence of this study, no less biased. The assumption that capability and fairness would advance in parallel has not been borne out. And the assumption that human oversight will compensate for algorithmic bias is not supported by what we know about how clinicians interact with automated systems under real-world conditions.

The institutions deploying these tools are making a calculation. They are betting that the benefits will outweigh the harms, that the efficiencies will justify the risks, and that the populations most likely to be harmed by biased AI are the same populations least likely to have the resources to hold anyone accountable.

That calculation may prove correct in the short term. In the longer term, it is the kind of institutional wager that generates class-action lawsuits, regulatory backlash, and, most importantly, measurable harm to patients who came to the healthcare system seeking help and received instead the outputs of a machine that treated their identity as a clinical variable.

The question is not whether AI will be integrated into healthcare. That integration is already underway, at scale, across the world's largest health systems. The question is whether the institutions driving that integration will treat equity as a design requirement or as an afterthought. The research is clear on what the problem is and how severe it remains. The gap between what we know and what we are willing to do about it is where the harm lives.

References

1. Omar, M., Agbareia, R., Apakama, D.U., Horowitz, C.R., Freeman, R., Charney, A.W., Nadkarni, G.N., and Klang, E. “New Model, Old Risks? Sociodemographic Bias and Adversarial Hallucinations Vulnerability in GPT-5.” medRxiv, September 2025. DOI: 10.1101/2025.09.19.25336180.

2. Omar, M., Klang, E., et al. “Sociodemographic biases in medical decision making by large language models.” Nature Medicine, 2025. DOI: 10.1038/s41591-025-03626-6.

3. Zack, T., et al. “Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study.” The Lancet Digital Health, January 2024. DOI: 10.1016/S2589-7500(23)00225-X.

4. “Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support.” Communications Medicine (Nature Portfolio), August 2025. DOI: 10.1038/s43856-025-01021-3.

5. “Evaluating and addressing demographic disparities in medical large language models: a systematic review.” International Journal for Equity in Health, Springer Nature, 2025. DOI: 10.1186/s12939-025-02419-0.

6. “Sociodemographic bias in clinical machine learning models: a scoping review of algorithmic bias instances and mechanisms.” Journal of Clinical Epidemiology, 2024. DOI: 10.1016/j.jclinepi.2024.111422.

7. Joerg, et al. “AI-generated dermatologic images show deficient skin tone diversity and poor diagnostic accuracy: An experimental study.” Journal of the European Academy of Dermatology and Venereology, 2025. DOI: 10.1111/jdv.20849.

8. “Disparities in dermatology AI performance on a diverse, curated clinical image set.” Science Advances, 2022. DOI: 10.1126/sciadv.abq6147.

9. Sjoding, M.W., et al. “Racial Bias in Pulse Oximetry Measurement.” New England Journal of Medicine, 2020. DOI: 10.1056/NEJMc2029240.

10. “The Overdue Imperative of Cross-Racial Pulse Oximeters.” Health Affairs Forefront, January 2025.

11. “Bias in medical AI: Implications for clinical decision-making.” PMC, 2024. PMCID: PMC11542778.

12. “The Algorithmic Divide: A Systematic Review on AI-Driven Racial Disparities in Healthcare.” PubMed, 2024. PMID: 39695057.

13. “The illusion of safety: A report to the FDA on AI healthcare product approvals.” PLOS Digital Health, June 2025. DOI: 10.1371/journal.pdig.0000866.

14. Estate of Gene B. Lokken et al. v. UnitedHealth Group, Inc. et al. Federal court ruling, February 2025. Georgetown Health Care Litigation Tracker.

15. U.S. Food and Drug Administration. “Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations.” Draft Guidance, January 2025.

16. U.S. Food and Drug Administration. “Artificial Intelligence and Machine Learning in Software as a Medical Device.” FDA AI/ML Device Database, July 2025.

17. European Commission. “EU AI Act: Regulatory Framework for Artificial Intelligence.” Phased implementation beginning August 2025, with full high-risk compliance required by August 2027.

18. World Health Organisation. “Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models.” January 2024. ISBN: 9789240084759.

19. “Bias recognition and mitigation strategies in artificial intelligence healthcare applications.” npj Digital Medicine, 2025. DOI: 10.1038/s41746-025-01503-7.

20. “Automation Bias in AI-Assisted Medical Decision-Making under Time Pressure in Computational Pathology.” arXiv, November 2024. arXiv:2411.00998.

21. “Exploring the risks of automation bias in healthcare artificial intelligence applications: A Bowtie analysis.” ScienceDirect, 2024. DOI: 10.1016/j.caeai.2024.100241.

22. “Mitigating Bias in Machine Learning Models with Ethics-Based Initiatives: The Case of Sepsis.” American Journal of Bioethics, 2025. DOI: 10.1080/15265161.2025.2497971.

23. Wong, A., et al. “External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients.” JAMA Internal Medicine, 2021. (Epic Sepsis Model evaluation at Michigan Medicine.)

24. Epic Systems. AI Charting and generative AI clinical tools deployment, February 2026. Epic Newsroom.

25. Oracle Health. Clinical AI Agent deployment across 30+ medical specialities, 2025. Oracle Health press materials.

26. “Gender and racial bias unveiled: clinical artificial intelligence (AI) and machine learning (ML) algorithms are fanning the flames of inequity.” Oxford Open Digital Health, 2025. DOI: 10.1093/oodh/oqaf027.

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

In January 2026, Kristalina Georgieva, the Managing Director of the International Monetary Fund, stood before an audience at the World Economic Forum in Davos and offered a statistic that landed with the quiet brutality of a footnote in a corporate restructuring memo. The number of translators and interpreters at the IMF, she said, had dropped from 200 to 50. The cause was not a budget crisis or a policy realignment. It was technology. The fund had simply decided that machines could handle most of the work that humans used to do.

Georgieva presented the figure as evidence of a broader transformation. Forty per cent of global jobs, she argued, would be transformed or eliminated by artificial intelligence, with that figure climbing to 60 per cent in advanced economies. But it was the specificity of the translation example that stuck. This was not a hypothetical projection or an economist's forecast. It was a headcount. Real people, with real expertise in the precise rendering of financial policy across languages and cultures, had been replaced by systems that could approximate their output at a fraction of the cost.

The IMF is not alone. Across the global translation industry, now valued at an estimated 31.70 billion US dollars according to Slator's 2025 Language Industry Market Report, a similar pattern is playing out. Large language models and neural machine translation systems have not simply made human translators obsolete. They have restructured the profession from the inside, converting skilled practitioners into quality controllers for text they did not write. The question this raises is not whether AI can translate. It demonstrably can, often to a standard that passes casual inspection. The question is what happens to a profession, and to the cultural knowledge it carries, when the market decides that “good enough” is good enough.

The Numbers Behind the Quiet Collapse

A 2024 survey conducted by the United Kingdom's Society of Authors, which polled 787 of its 12,500 members, found that 36 per cent of translators had already lost work to generative AI. Forty-three per cent reported a decrease in income as a direct result of the technology. Over three-quarters, some 77 per cent, believed that generative AI would negatively affect their future earnings. Eighty-six per cent expressed concern that the use of generative AI devalues human-made creative work. These are not projections. They are reports from working professionals describing what has already happened to their livelihoods.

The income data from individual translators is more granular and more alarming. Brian Merchant, writing in his newsletter Blood in the Machine, documented cases across the profession in mid-2025. One technical translator with 15 years of experience reported earning just 8,000 euros in 2025, down from six figures in previous years. A French-English translator based in Quebec described a 60 per cent income decline in 2024, with projections suggesting an 80 per cent drop from peak earnings by the end of 2025. An Italian-English translator in Rome reported that work requests had ceased entirely for the month of June 2025, after years of working 50 to 60 hours per week. An English-Portuguese translator documented that post-editing rates had collapsed from 0.04 euros to 0.02 euros per source word, halving the already modest compensation for correcting machine output.

In the United States, Andy Benzo, president of the American Translators Association, told CNN in January 2026 that many translators were leaving the profession entirely. Benzo noted that the risks of using AI translation in “high-stakes” fields remain “humongous,” yet the exodus continues regardless. Ian Giles, chair of the Translators Association at the UK's Society of Authors, confirmed the same pattern, noting that translators were seeking retraining “because translation isn't generating the income it previously did.” The exits are not dramatic. There are no picket lines or public protests. People are simply disappearing from a profession that can no longer sustain them.

The scale of this workforce is not trivial. There are approximately 640,000 professional translators globally, and three out of four are freelancers. It is this freelance majority that has borne the brunt of the disruption, lacking the institutional protections and guaranteed workloads that might have cushioned the blow.

A study published in 2025 by Carl Benedikt Frey and Pedro Llanos-Paredes at the Oxford Martin School quantified the scale of displacement with unusual precision. Analysing variation in Google Translate adoption across 695 local labour markets in the United States, the researchers found that a one percentage point increase in the use of Google Translate corresponded to a 0.71 percentage point reduction in translator employment growth. The cumulative effect, they estimated, amounted to more than 28,000 fewer translator positions created over the period from 2010 to 2023. And that figure captures only the impact of a single, relatively crude machine translation tool that preceded the large language model era. The arrival of systems like GPT-4, Claude, and Gemini has accelerated the process enormously, because these models do not just translate. They handle idiomatic expression, register, and contextual nuance at a level that earlier statistical systems could not approach.

In July 2025, Microsoft researchers published a study examining which occupations were most exposed to generative AI capabilities. Translators and interpreters ranked first on the list, with 98 per cent of their work activities overlapping with tasks that AI systems could perform with relatively high completion rates. The study analysed 200,000 real-world conversations between users and Microsoft's Copilot system to arrive at its rankings. The researchers were careful to note that high exposure does not automatically mean elimination. But the practical effect has been unmistakable. Employers have used the availability of AI translation as justification for cutting rates, reducing headcounts, and restructuring workflows around machine output.

From Translator to Post-Editor

The restructuring of translation work follows a pattern that is becoming familiar across AI-affected professions. The human does not vanish. Instead, they are repositioned downstream in the production process, tasked with reviewing and correcting output that a machine generated in seconds. In the translation industry, this workflow is known as Machine Translation Post-Editing, or MTPE, and it has rapidly become the dominant model for commercial translation work.

According to Slator's 2025 survey of the language industry, 60 per cent of all respondents were using machine translation, with adoption reaching 80 per cent among language service providers. Among those using machine translation or large language models, between 90 and 98 per cent performed some level of post-editing on AI-generated content. Eighty-four per cent of language service integrators reported that clients had specifically requested human editing services to review AI-generated translations. The human, in other words, has not been removed from the process. But the nature of their involvement has been fundamentally altered. They are no longer creating. They are correcting.

The compensation reflects this downgrade. Post-editing rates typically fall between 50 and 70 per cent of standard translation rates, with some agencies offering as little as 25 per cent of what a full human translation would command. Industry data from 2025 indicates that MTPE work commands between 0.05 and 0.15 US dollars per word, compared with 0.15 to 0.30 dollars per word for standard human translation. One translator documented by Equal Times, an international labour news platform, described pre-translated segments paying just 30 to 50 per cent of original rates, while fully automated platforms paid up to seven times less than standard. The economic logic is straightforward. If the machine does 80 per cent of the work, the reasoning goes, then the human should be paid for only 20 per cent. What this calculation ignores is that post-editing often requires comparable time and cognitive effort to translation from scratch, because the translator must not only identify errors but also understand the systematic patterns of how the AI fails and where its confidence is misplaced.

The workflow itself has been transformed in ways that strip autonomy from the translator. Texts no longer arrive as clean source documents to be rendered thoughtfully into a target language. They arrive pre-segmented, with machine-generated suggestions already populating each segment. The translator's task becomes one of triage: deciding which suggestions are acceptable, which need modification, and which must be discarded entirely. Automated platforms distribute this work via alerts that give translators minutes or even seconds to claim individual segments, creating a piecework dynamic more reminiscent of a fulfilment warehouse than a skilled profession. Some platforms threaten automatic disconnection for translators who dispute corrections imposed by quality-assurance algorithms.

Jean-Jacques, a 30-year veteran translator quoted by Equal Times, described the shift bluntly. “It's not really a matter of translating anymore,” he said, “but revising and correcting the segments proposed by the machine.” Another translator, identified as Alina, captured the paradox at the heart of the arrangement. “AI is both a tool and a threat,” she said. “We ourselves are teaching it how to translate, how to improve.” Each correction a post-editor makes feeds back into the training data that will make the next generation of AI translation marginally better, and the human's role marginally less essential.

This dynamic, in which skilled workers are conscripted into training their own replacements, is not unique to translation. It has appeared in content moderation, coding, and legal document review. But in translation, the irony is particularly sharp, because the expertise being extracted is precisely the kind that AI systems struggle most to develop on their own: cultural sensitivity, tonal awareness, and the ability to navigate the space between what a text says and what it means.

What Machines Cannot Feel

The case for human translation has always rested on something more than accuracy. It rests on the claim that translation is an interpretive act, a creative negotiation between two linguistic and cultural systems that requires not just knowledge but judgement. Jhumpa Lahiri, the Pulitzer Prize-winning novelist who has written extensively about translation, describes the process as “a radical act of reshaping text and self.” In her essay collection Translating Myself and Others, published by Princeton University Press in 2022, Lahiri argues that “a translator restores the meaning of a text by means of an elaborate, alchemical process that requires imagination, ingenuity, and freedom.”

This is not the language of quality assurance. It is the language of craft, of a practice that involves the translator's full intellectual and emotional engagement with a text. Emily Wilson, the first woman to translate Homer's Odyssey into English, has spoken repeatedly about the impossibility of separating linguistic from cultural knowledge in translation. The hardest part of translation, she has argued, is not understanding the original but “figuring out how to create it entirely from scratch in a totally different language and culture.” Wilson's translation of the Odyssey was widely praised precisely because it made choices that no algorithm would make: tonal decisions, rhythmic choices, and interpretive framings that reflected not just the Greek text but Wilson's own understanding of what the poem means to contemporary English-speaking readers.

Gregory Rabassa's English translation of Gabriel Garcia Marquez's One Hundred Years of Solitude is perhaps the most celebrated example of translation as creative achievement. Marquez himself reportedly said that he considered the English translation a work of art in its own right, a remarkable statement from an author about a rendering of his own novel. Edith Grossman, the acclaimed translator of both Marquez and Cervantes, described Rabassa as “the godfather of us all,” crediting him with introducing Latin American literature to the English-speaking world in a way that preserved not just meaning but spirit.

These examples belong to the domain of literary translation, which remains relatively insulated from AI disruption. Literary commissions have continued to flow to human translators, in part because publishers recognise that the qualities that make a literary translation valuable are precisely the qualities that machines lack. But the insulation is narrower than it appears. The vast majority of professional translation work is not literary. It is commercial, legal, technical, medical, and administrative. And it is in these domains that the restructuring has been most severe, not because the cultural stakes are lower, but because the market has decided they are.

Consider the translation of a medical consent form from English into Tagalog for a Filipino patient in a London hospital. The document is not literary. It will never win a prize. But the accuracy of its translation has direct consequences for a person's understanding of what is being done to their body. A machine translation might render the words correctly while missing the pragmatic force of the language: the way a particular phrasing might sound reassuring or threatening, the cultural assumptions embedded in notions of consent, the difference between informing someone and making them feel informed. These are not edge cases. They are the bread and butter of professional translation, and they are the first tasks being handed to machines.

Or consider immigration proceedings, where a mistranslation can determine whether an asylum seeker's testimony is deemed credible. The translator in that context is not merely converting words. They are mediating between legal systems, cultural frameworks of narrative and evidence, and the emotional register of a person recounting traumatic experiences. The difference between “I was afraid” and “I feared for my life” is not a matter of synonymy. It is a matter of legal consequence, and navigating it requires the kind of situated cultural judgement that no statistical model possesses.

The Hybrid Illusion

The industry's preferred narrative for this transition is “human-AI collaboration.” The framing suggests a partnership: the machine handles the heavy lifting, and the human provides the finishing touch. But the power dynamics of this arrangement are radically asymmetric. The machine sets the terms. The human adjusts.

This is not collaboration in any meaningful sense. It is supervision, and it is supervision of a peculiarly degrading kind, because the supervisor is being paid less than they would earn if they were simply doing the work themselves. The translator who once sat with a source text and crafted a target text from scratch, making hundreds of micro-decisions about register, idiom, rhythm, and cultural resonance, now sits with a machine-generated draft and decides, sentence by sentence, whether it is wrong enough to fix.

The cognitive experience of post-editing is qualitatively different from translation. Several translators have described it as more fatiguing and less satisfying than original translation work. The machine's output creates a kind of gravitational pull. Even when the translator knows a better rendering exists, the effort required to override the machine's suggestion and compose something from scratch can feel disproportionate to the compensation. Over time, this produces a phenomenon that linguists and labour researchers have begun to call “anchoring,” in which the translator's own instincts are gradually subordinated to the machine's defaults. The result is not a blend of human and machine intelligence. It is machine intelligence with a human stamp of approval.

A 2025 survey of translators found that a majority, some 66 per cent, acknowledged that MTPE can be useful but still requires substantial human intervention. Roughly half of respondents refused to offer discounts for post-editing work, arguing that the effort required is routinely underestimated by clients and agencies. Among those who did discount, the most common reduction fell between 10 and 30 per cent, far less than the 50 to 75 per cent cuts that many agencies impose unilaterally.

Rosa, a translator quoted by Equal Times, described the economic logic with characteristic directness. “Profit is the only thing that matters,” she said, “and translation has become like a commodity that they extract from us at the lowest possible price.” The commodity metaphor is precise. What was once a craft, defined by the individual translator's knowledge, taste, and cultural fluency, has been reframed as a raw material to be processed at industrial scale.

The Structural Incapacity Argument

There is a version of this story in which what is happening to translators is tragic but temporary, a painful adjustment period that will eventually stabilise as the technology matures and the market finds a new equilibrium. In this version, AI translation will continue to improve until the quality gap between machine and human output narrows to insignificance, at which point the remaining human translators will occupy a small, highly specialised niche: literary translation, diplomatic interpreting, and other domains where the stakes are too high for automation.

But this narrative assumes that the qualities human translators bring are merely a matter of degree, that machines are doing a slightly worse version of the same thing, and that incremental improvement will close the gap. There is a competing argument, advanced by translators, linguists, and cognitive scientists, that the gap is not quantitative but structural. That what human translators do when they translate with cultural sensitivity and emotional intelligence is not a more refined version of pattern matching. It is a fundamentally different cognitive operation.

A study published in Nature's Humanities and Social Sciences Communications in 2026, examining AI performance in literary autobiography translation, found that while AI models could produce grammatically correct and largely accurate translations, they consistently failed to capture the emotional texture and cultural specificity of the original texts. The researchers concluded that human translators brought interpretive capacities that were not simply absent from AI systems but categorically different in kind. AI models could identify the surface layer of meaning but failed to recognise cultural allusions and deeper emotional context, elements that are essential not just to literature but to any communication that carries weight beyond its literal content.

This distinction matters because it determines whether human translators are a temporary patch or a permanent necessity. If translation is ultimately a pattern-matching problem, then machines will eventually solve it. If it is an interpretive problem, requiring the kind of embodied cultural knowledge that comes from living inside a language and its associated worldview, then machines will not solve it, regardless of how much training data they consume. The patterns they learn are drawn from existing translations, which means they can only reproduce what human translators have already created. They cannot originate the kind of interpretive leap that makes a translation feel alive.

Poetry, with its reliance on rhythm, rhyme, and figurative language, remains a particularly formidable challenge. A machine can translate the denotative content of a poem. It cannot translate its music. It cannot decide, as Emily Wilson did with the Odyssey, that the opening word of an epic should be “Tell me” rather than “Sing to me,” and understand the cascade of interpretive consequences that follows from that single choice.

The Market Does Not Care About Craft

The structural incapacity argument, however compelling, runs into a problem that is not technological but economic. The market for translation services is not optimised for craft. It is optimised for throughput, cost reduction, and acceptable quality at scale. And by this measure, AI translation is already good enough for the vast majority of commercial applications. The Slator survey found that while 72 per cent of respondents cited accuracy concerns with machine translation and 68 per cent cited quality concerns, adoption continued to accelerate regardless. Trust grew slowly, but adoption grew fast. The concerns are real. They are also, from a procurement perspective, manageable.

This is the uncomfortable truth at the centre of the translation crisis. The question is not whether AI can match human translators in quality. It demonstrably cannot, particularly in contexts requiring cultural nuance, tonal sensitivity, or interpretive judgement. The question is whether the market values those qualities enough to pay for them. And the evidence, from rate compression to headcount reduction to the restructuring of workflows around machine output, suggests that it does not.

The AI-enabled translation services market, valued at 5.18 billion US dollars in 2025 according to Precedence Research, is projected to reach 50.69 billion by 2035, expanding at a compound annual growth rate of 25.62 per cent. These are not numbers that suggest a market hedging its bets. They describe an industry that has made a decisive bet on automation, with human involvement reduced to the minimum necessary to maintain an acceptable error rate. Software platforms already dominate the market, holding nearly 73 per cent of 2025 revenue, and they are growing faster than any other component as enterprises embed AI-driven localisation into core workflows.

The parallel to other creative and knowledge-work professions is instructive. Journalism, graphic design, customer service, and legal research have all experienced similar dynamics: AI systems that produce output of variable but often adequate quality, followed by a restructuring of human roles around review, correction, and oversight rather than creation. In each case, the same rhetorical move occurs. The technology is presented as a tool that augments human capability. In practice, it becomes a ceiling that constrains it. The human is not empowered. The human is made cheaper.

What Gets Lost When Languages Lose Their Interpreters

The consequences of this restructuring extend beyond the economic fortunes of individual translators. Languages are not neutral containers for information. They are living systems of meaning, shaped by history, geography, power, and culture. A translator who has spent decades working between English and Arabic, or Mandarin and Portuguese, or Hindi and German, carries within them a form of knowledge that is not reducible to a bilingual dictionary or a statistical model trained on parallel corpora.

The Frey and Llanos-Paredes study at Oxford Martin documented an additional finding that received less attention than the employment data but may be more consequential in the long term. Areas with robust Google Translate usage saw job postings demanding Spanish fluency grow by about 1.4 percentage points less than in other regions, with similar declines of roughly 1.3 and 0.8 percentage points for Chinese and German respectively, and measurable dampening even for Japanese and French. The adoption of machine translation, in other words, is not just replacing translators. It is reducing the perceived value of knowing another language at all.

This is a feedback loop with serious cultural implications. As machine translation becomes more capable and more widely adopted, the incentive to invest in human language skills diminishes. Fewer people pursue translation as a career. Fewer organisations invest in in-house linguistic expertise. The pool of human knowledge about how languages relate to one another, how cultural contexts shape meaning, and how texts function differently across linguistic boundaries gradually shrinks. And the AI systems that replace this knowledge are trained on the output of the very translators they displace, creating a closed loop in which the training data grows stale as the human source of fresh interpretive insight dries up.

Ian Giles, in his capacity as chair of the Translators Association, has raised precisely this concern, questioning whether “the demand for subtlety and craft from enough readers and publishers” will “save highly skilled individuals from becoming mere AI post-editors.” The word “mere” carries the weight of the entire argument. It acknowledges that the role of post-editor exists. It questions whether the role is sufficient to sustain the expertise it depends upon.

The problem is compounded by the pipeline effect. If experienced translators leave the profession and aspiring translators are deterred by collapsing incomes, the next generation of human translators simply will not exist in sufficient numbers. The craft knowledge that takes years to develop, the intuitive feel for how a sentence should land in a target language, the awareness of cultural registers that no textbook teaches, is not the kind of knowledge that can be stored in a database and retrieved on demand. It lives in people. When those people leave, it leaves with them.

The Canary and the Coal Mine

Professional translators have long occupied a peculiar position in the knowledge economy. Their work is invisible when done well. A reader who encounters a beautifully translated novel does not think about the translator. A patient who reads a clearly rendered medical document in their own language does not consider the person who bridged the linguistic gap. This invisibility made translators vulnerable long before AI arrived. It meant that their expertise could be devalued without anyone noticing, because the beneficiaries of their work rarely understood what it involved.

What is happening to translators now is therefore not just a story about one profession. It is a preview of what happens when AI is deployed not to eliminate human workers but to restructure their role in ways that extract their expertise while diminishing their authority, autonomy, and compensation. The translator who becomes a post-editor is still needed. But the nature of the need has changed. They are needed not for what they can create but for what they can catch. Not for their vision but for their vigilance.

Georgieva's statistic from Davos, those 150 translators who lost their positions at the IMF, represents one institution's calculation that the cultural and interpretive knowledge those individuals carried was worth less than the cost savings achieved by replacing them with technology. That calculation is now being replicated across every sector that relies on translation, from international law to pharmaceutical regulation to immigration services. In each case, the logic is the same. The machine produces output that is adequate for most purposes. The remaining humans clean up whatever the machine gets wrong. And the expertise that once defined the profession gradually atrophies, because there is no economic incentive to develop it and no structural pathway through which it can be transmitted to the next generation.

The question, then, is not whether AI translation will continue to improve. It will. And it is not whether human translators will survive in some form. They will, at least for a while, as post-editors and quality reviewers and specialists in the narrow domains where machine output remains unreliable. The question is whether a society that systematically devalues the ability to translate with feeling, with cultural awareness, with the full depth of human interpretive intelligence, will eventually discover that it has lost something it cannot rebuild. Not because the technology failed, but because the market decided that what translators knew was not worth preserving.

---

References and Sources

1. CNN. “Meet the translation professionals losing their jobs to AI.” CNN Business, 23 January 2026. https://www.cnn.com/2026/01/23/tech/translation-language-jobs-ai-automation-intl

2. TIME. “The IMF's Kristalina Georgieva on the AI 'Tsunami' Hitting Jobs.” TIME, January 2026. https://time.com/collections/davos-2026/7339218/ai-trade-global-economy-kristalina-georgieva-imf/

3. Slator. “Five Ways AI Reshaped the Translation Industry in 2025.” Slator, 2025. https://slator.com/five-ways-ai-reshaped-translation-industry-2025/

4. Slator. “Slator 2025 Language Industry Market Report.” Slator, 2025. https://slator.com/slator-2025-language-industry-market-report/

5. Society of Authors. “SoA survey reveals a third of translators and quarter of illustrators losing work to AI.” Society of Authors, April 2024. https://europeanwriterscouncil.eu/soa-survey-uk-ai-2024/

6. Merchant, Brian. “AI Killed My Job: Translators.” Blood in the Machine, 2025. https://www.bloodinthemachine.com/p/ai-killed-my-job-translators

7. Equal Times. “Artificial intelligence, dehumanisation and precarious work: translators on the frontline of tech-induced job degradation.” Equal Times, 2025. https://www.equaltimes.org/artificial-intelligence?lang=en

8. Frey, Carl Benedikt and Llanos-Paredes, Pedro. “Lost in Translation: Artificial Intelligence and the Demand for Foreign Language Skills.” Oxford Martin School, March 2025. https://www.oxfordmartin.ox.ac.uk/publications/lost-in-translation-artificial-intelligence-and-the-demand-for-foreign-language-skills

9. CEPR. “Lost in translation: AI's impact on translators and foreign language skills.” CEPR VoxEU, 2025. https://cepr.org/voxeu/columns/lost-translation-ais-impact-translators-and-foreign-language-skills

10. Fortune. “Microsoft researchers have revealed the 40 jobs most exposed to AI.” Fortune, July 2025. https://fortune.com/article/what-are-the-jobs-most-exposed-to-ai-microsoft-research/

11. CNBC. “These 10 jobs are the least AI-safe, according to new Microsoft report.” CNBC, 5 August 2025. https://www.cnbc.com/2025/08/05/these-10-jobs-are-the-least-ai-safe-according-to-new-microsoft-report.html

12. Precedence Research. “AI Enabled Translation Services Market Size 2025 to 2035.” Precedence Research, 2025. https://www.precedenceresearch.com/ai-enabled-translation-services-market

13. Lahiri, Jhumpa. Translating Myself and Others. Princeton University Press, 2022. https://press.princeton.edu/books/hardcover/9780691231167/translating-myself-and-others

14. Princeton University. “Jhumpa Lahiri champions the writerly art of translation.” Princeton University News, 4 September 2020. https://www.princeton.edu/news/2020/09/04/jhumpa-lahiri-champions-writerly-art-translation

15. Wilson, Emily. Conversations with Tyler, Episode 63. “Emily Wilson on Translations and Language.” https://conversationswithtyler.com/episodes/emily-wilson/

16. Nature. “Exploring AI's performance in literary autobiography translation: how closely do AI models match human translation.” Humanities and Social Sciences Communications, 2026. https://www.nature.com/articles/s41599-026-06630-4

17. Washington Post. “AI is taking on live translations. But jobs and meaning are getting lost.” Washington Post, 26 September 2025. https://www.washingtonpost.com/business/2025/09/26/ai-translation-jobs/

18. The Bookseller. “A third of translators report losing work to generative AI systems, SoA survey reveals.” The Bookseller, 2024. https://www.thebookseller.com/news/a-third-of-translators-report-losing-work-to-generative-ai-systems-soa-survey-reveals

19. World Economic Forum. “Putting a figure on it: Davos 2026 in numbers.” WEF, January 2026. https://www.weforum.org/stories/2026/01/davos-2026-in-numbers/

20. GTS Translation. “The State of Machine Translation Post-Editing (MTPE) in 2025: What Translators Think.” GTS Blog, 7 April 2025. https://blog.gts-translation.com/2025/04/07/the-state-of-machine-translation-post-editing-mtpe-in-2025-what-translators-think/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

Nearly seven in ten middle and high school students now say they believe artificial intelligence is eroding their critical thinking skills. They reported this in a December 2025 survey conducted by the RAND Corporation's American Youth Panel. They also reported, in the very same survey, that they are using AI for homework more than ever before, with usage climbing from 48 per cent to 62 per cent in barely seven months. The students, in other words, can see the problem clearly. They simply cannot stop participating in it.

This is an extraordinarily revealing paradox, and it deserves more scrutiny than the predictable hand-wringing it has generated. Because the most uncomfortable question here is not whether ChatGPT is making teenagers worse at thinking. It is whether the education system that ushered AI into classrooms with such breathless enthusiasm ever genuinely valued the kind of independent, rigorous, critical thought it now claims to be losing.

The answer, if you follow the evidence, is not encouraging.

The Paradox in the Numbers

The RAND data is striking in its internal contradictions. Among the 1,214 young people surveyed (aged 12 to 29, all enrolled in school during the 2025-26 academic year), 67 per cent endorsed the statement that “the more students use AI for their schoolwork, the more it will harm their critical thinking skills.” That figure had risen more than ten percentage points in just ten months. The concern was especially pronounced among female students, 75 per cent of whom agreed, compared with 59 per cent of male students.

Yet during the same period, the percentage of middle schoolers using AI for homework leapt from 30 per cent to 46 per cent, and among high schoolers it jumped from 49 per cent to 60 per cent. Most of these students (60 per cent) also expressed concern about using AI for school-related purposes. So they are worried and they are doing it anyway. This is not cognitive dissonance in any simple sense. It is something more structurally interesting: students have correctly diagnosed a systemic problem, but they exist within a system that gives them no rational incentive to behave differently.

Consider the logic from a student's perspective. Assignments are graded. Grades determine university admissions. University admissions determine (or are perceived to determine) life outcomes. If your peers are using AI and getting better grades, opting out is not a principled stand. It is a competitive disadvantage. The students are not confused. They are trapped.

Think of it another way. You are sixteen. You have five GCSEs to revise for, a personal statement to write, and a part-time job. Your classmates are producing polished coursework in half the time it takes you to write a first draft because they are running their ideas through ChatGPT. Your teachers, overwhelmed and under-resourced, cannot reliably tell the difference. The system rewards the output, not the process. In this environment, choosing not to use AI is not intellectual integrity. It is self-sabotage.

Meanwhile, faculty at the university level are sounding alarms with even greater urgency. A national survey conducted by the American Association of Colleges and Universities and Elon University's Imagining the Digital Future Centre in November 2025 found that 95 per cent of the 1,057 faculty respondents feared that generative AI would increase student overreliance on the technology. Ninety per cent said it would diminish students' critical thinking skills. Eighty-three per cent said AI would decrease student attention spans. And 78 per cent said cheating on their campuses had increased since these tools became widely available, with 57 per cent saying it had increased significantly.

The teachers see the same thing the students see. The difference is that teachers are surprised. The students are not.

A System That Never Quite Got Round to Critical Thinking

Here is where the conversation gets genuinely uncomfortable. Long before ChatGPT existed, education reformers, cognitive scientists, and classroom teachers themselves were raising the alarm about a system that was systematically undermining higher-order thinking. The culprit was not artificial intelligence. It was standardised testing.

The No Child Left Behind Act of 2001 (NCLB) represented, in the United States at least, the triumph of measurable outcomes over meaningful learning. Under its regime, schools were judged by their students' performance on standardised assessments. The consequences of poor scores were severe: funding cuts, staff dismissals, school closures. The entirely predictable result was what educators came to call “teaching to the test,” a practice in which classroom instruction was narrowed to the specific content and formats that would appear on state exams.

The effects were devastating and well-documented. Subjects not covered by standardised tests, including art, music, physical education, and social studies, were minimised or eliminated outright. Some principals eliminated recess to devote more time to test preparation. Science was replaced with additional maths drills. Social studies gave way to language arts worksheets. The phrase that captured this era most succinctly was “sit, get, spit, forget,” a cycle in which students received information passively, regurgitated it on an exam, and promptly forgot it, having never engaged with it at any depth.

The situation in the United Kingdom has followed a parallel trajectory. Successive reforms since the introduction of the National Curriculum in 1988, the expansion of league tables in the 1990s, and the intensification of Ofsted inspections have created an accountability culture that rewards measurable outcomes above all else. Teachers in England report spending enormous amounts of time on assessment preparation, data tracking, and administrative compliance, time that might otherwise be devoted to the kind of open-ended, inquiry-driven teaching that develops critical thinking. The Department for Education published expanded guidance on AI in education in June 2025, stressing that AI tools should support rather than replace subject knowledge and that students still need a strong foundation in reading, writing, and critical thinking to use these tools effectively. But guidance is one thing; structural reform is quite another.

Paulo Freire, the Brazilian educator and philosopher, would have recognised all of this instantly. In his seminal 1968 work “Pedagogy of the Oppressed,” Freire described what he called the “banking model” of education, in which teachers deposit knowledge into the passive receptacles of students' minds, and students are expected to receive, memorise, and repeat. Freire argued that this approach was fundamentally hostile to critical consciousness; the more students worked at storing deposits, the less they developed the critical thinking that would allow them to intervene in the world as transformers of that world. His alternative, critical pedagogy, was rooted in dialogue, in treating students as co-creators of knowledge rather than empty vessels to be filled.

NCLB was, in Freire's terms, the banking model with federal enforcement mechanisms. The UK's accountability framework achieved much the same outcome through different institutional channels. And while NCLB was eventually replaced by the Every Student Succeeds Act (ESSA) in 2015, which offered states greater flexibility in assessment design, the deeper cultural damage had been done. An entire generation of teachers on both sides of the Atlantic had been trained in a system that rewarded compliance over curiosity, memorisation over analysis, and standardised answers over independent thought.

So when commentators now lament that AI is destroying students' capacity for critical thinking, the honest follow-up question is: which critical thinking? When, precisely, was this golden age of independent thought in schools? Because the evidence suggests it was already in serious trouble long before a single student typed a homework question into ChatGPT.

Cognitive Offloading and the Science of Thinking Less

The cognitive science, meanwhile, tells a more nuanced story than either technophiles or technophobes would prefer. Research published in 2025 by Michael Gerlich of SBS Swiss Business School, in the journal Societies, investigated the relationship between AI tool usage and critical thinking through the lens of cognitive offloading, the well-established phenomenon in which humans delegate cognitive tasks to external resources to reduce mental demand.

Gerlich's study surveyed and interviewed 666 participants across diverse age groups and educational backgrounds, finding a significant negative correlation between frequent AI tool use and critical thinking abilities. The numbers were stark: cognitive offloading was strongly correlated with AI tool usage (r = +0.72) and inversely related to critical thinking (r = -0.75). Younger participants, those aged 17 to 25, showed higher dependence on AI tools and lower critical thinking scores compared to older age groups. However, and this is crucial, advanced educational attainment correlated positively with critical thinking skills, suggesting that education, when it works properly, can mitigate some of the cognitive costs of AI reliance. The implication is clear: the problem is not that education cannot protect against cognitive offloading, but that most education systems are not currently designed to do so.

A separate study from Microsoft Research, presented at CHI 2025 (the Conference on Human Factors in Computing Systems), surveyed 319 knowledge workers about their experiences with generative AI. The findings revealed a telling dynamic: higher confidence in AI was associated with less critical thinking, while higher self-confidence was associated with more critical thinking. The research also identified a fundamental shift in the nature of cognitive work, from information gathering to information verification, from problem-solving to AI response integration, and from doing tasks to supervising them.

This matters enormously for students, who are still in the process of building the very cognitive capacities that adults are now choosing to offload. A knowledge worker who has spent twenty years learning to construct arguments, evaluate evidence, and synthesise information can afford to delegate some of those tasks to AI without losing the underlying skill. A teenager who has never fully developed those skills in the first place is in a fundamentally different position. For them, cognitive offloading is not a convenience. It is a developmental short-circuit.

This is not merely a problem of laziness or moral failure. It is a predictable consequence of how human cognition interacts with powerful tools. We have always offloaded cognitive tasks onto external supports, from written language to calculators to search engines. The question with AI is whether the offloading is so comprehensive, and so seamless, that it crosses the line from scaffolding (which is temporary and empowering) to substitution (which is permanent and diminishing).

The critical distinction, as cognitive scientists have noted, is whether AI operates as a scaffold or a substitute. Scaffolding is characterised by temporariness, adaptability, and the goal of strengthening internal capacities. Substitution simply does the thinking for you. And the educational system, in its rush to adopt AI tools, has devoted remarkably little attention to ensuring the former rather than the latter.

The Teacher's Impossible Position

Any honest account of this situation must reckon with the position of teachers themselves, who are caught between contradictory demands with diminishing resources to meet any of them. Nearly half of teachers in the United States and the United Kingdom report chronic burnout. Teacher shortages are endemic. Class sizes in many state schools have grown. Administrative demands consume ever-larger portions of the working week.

Into this environment of exhaustion and scarcity comes AI, marketed to schools and teachers as a solution to the very problems the system has created. District leaders implementing AI tools report that teachers can reclaim an average of 5.9 hours per week by automating lesson planning, grading, and communication tasks. For a profession in crisis, this is not a trivial proposition. If a teacher can use AI to handle routine administrative work and spend more time on meaningful instruction, that sounds like progress.

But the reality is more complicated. Only about one in five teachers work at a school that has an AI policy. Teacher training on the pedagogical use of AI remains inconsistent and often superficial. The gap between the promise of AI as a teaching aid and the lived reality of its implementation is vast. Teachers are being asked to integrate a transformative technology into their practice while simultaneously meeting accountability targets, managing behaviour, differentiating instruction for diverse learners, and coping with the emotional demands of working with young people in an era of escalating mental health challenges.

The result is that AI adoption in schools is happening not through careful pedagogical planning, but through exhaustion. Teachers are adopting AI not because they have been trained to use it well, but because they are too stretched to do without it. And students are adopting AI not because they have been taught to use it critically, but because nobody has given them a compelling reason not to.

The Whiplash of Institutional Adoption

The speed at which schools reversed their positions on AI is itself a revealing story. In January 2023, New York City's Department of Education became one of the first major school systems to ban ChatGPT from its networks and devices. The ban was announced with the gravity of a public health measure, citing concerns about academic integrity and the tool's potential to provide students with answers that lacked critical thinking. Fairfax County Public Schools in Virginia and Austin Independent School District in Texas followed suit, citing child safety and academic integrity.

Within four months, New York City reversed its ban. The reversal came after convening tech industry representatives and educators to evaluate the technology's potential benefits. By 2024, more than three-quarters of educators reported that their districts had not banned ChatGPT or similar tools. The pattern, ban first, then embrace, played out across districts nationwide. Seattle Public Schools, which had initially banned ChatGPT and six additional AI writing assistance websites, similarly softened its stance.

This institutional whiplash is instructive. The initial bans suggested that schools understood, at least intuitively, that AI posed a genuine threat to the learning process. The rapid reversals suggested that this understanding was no match for the combined pressures of industry lobbying, parental expectations, competitive anxiety, and the sheer momentum of a technology that students were already using at home.

The AI in education market tells its own story of institutional capture. Valued at approximately 7 billion dollars in 2025, the sector is projected to grow to nearly 137 billion dollars by 2035, expanding at a compound annual growth rate of over 34 per cent. Major technology companies, including Microsoft, Google, Amazon, and Pearson, have invested heavily in educational AI products. In July 2025 alone, Microsoft announced plans to invest over 4 billion dollars in AI education initiatives. These investments are not philanthropic gestures. They are strategic plays for long-term market dominance in an industry that touches every child in the developed world.

These are not neutral actors offering disinterested tools. They are companies with revenue models that depend on deep integration into educational infrastructure. When schools adopt their platforms, they are not just choosing a product; they are choosing a pedagogical philosophy, one that often prioritises efficiency, personalisation through algorithmic recommendation, and scalable delivery over the messy, slow, deeply human process of learning to think for oneself.

The Khanmigo Question

Not all educational AI is created equal, and the differences matter. Khan Academy's Khanmigo, launched in limited beta in 2023 and reaching approximately 1.5 million users across 130 countries by the end of 2025, represents a philosophically distinct approach to AI in education. Unlike ChatGPT, Khanmigo is designed not to give answers directly. Instead, it employs a Socratic method, offering hints and guiding questions intended to help students find answers themselves.

According to Khan Academy's own data, 68 per cent of students preferred Khanmigo's approach over ChatGPT for homework help, citing reduced anxiety about cheating. There is, students reported, a real psychological difference between “the AI gave me the answer” and “I figured it out with help.” This is a meaningful distinction. The student who works through a problem with Socratic guidance is still engaging in the cognitive labour that builds understanding. The student who pastes an essay prompt into ChatGPT and submits the output is not.

This distinction matters because it reveals that the problem is not AI per se, but how AI is designed and deployed. A tool built to scaffold learning is fundamentally different from a tool optimised to generate complete, polished outputs on demand. Yet in practice, most students are not using carefully designed educational AI. They are using general-purpose large language models, tools built for productivity, not pedagogy. And the education system has done remarkably little to shape how students interact with these tools.

The gap between what is possible and what is actually happening is enormous. Khanmigo demonstrates that AI can be designed to support critical thinking rather than replace it. But Khanmigo also requires institutional investment, teacher training, and a deliberate pedagogical framework, precisely the things that the current system, oriented toward rapid adoption and measurable outcomes, is least equipped to provide.

We Have Been Here Before, Sort Of

The temptation to draw neat historical parallels is strong, and partly justified. In 1986, the Christian Science Monitor reported on fierce debates over calculator use in schools, with one Oregon teacher of the year warning that “once you have a crutch, you rely on it more and more.” The National Council of Teachers of Mathematics had urged the integration of calculators at all grade levels, and maths teachers in Washington, D.C. picketed their meetings in protest.

The pro-calculator camp cited studies showing that students with calculators performed at least as well on tests as those without them (except, curiously, in the fourth grade). The anti-calculator camp warned of atrophied mental arithmetic skills and dangerous dependency. Eventually, calculators became ubiquitous, and the debate faded into the background noise of educational history.

The AI parallel writes itself, but it is also misleading in important ways. A calculator is a tool for performing a specific, well-defined operation. It computes. AI, by contrast, is a tool for generating language, analysing arguments, synthesising information, and producing written outputs that closely mimic (and sometimes surpass) the kinds of work that students are assessed on. The calculator could not write your essay. ChatGPT can. The calculator did not threaten the process by which students learned to construct arguments, weigh evidence, or develop original perspectives. AI does. The scope of the offloading is categorically different, and so the historical precedent offers less comfort than its proponents suggest.

The more honest historical parallel might be the introduction of television in the 1950s and 1960s, when educators initially hailed the new medium as a revolutionary learning tool before gradually recognising that passive consumption of information was not the same as active engagement with ideas. The lesson from that era was not that television was inherently bad, but that it was easy to confuse exposure to information with genuine understanding. AI presents the same confusion in a more insidious form: the output looks like understanding. It reads like comprehension. But the student who submits it may not have comprehended anything at all.

The International View

The global picture offers both cautionary tales and faint glimmers of hope. The OECD's PISA 2022 assessment, which for the first time evaluated creative thinking skills across 64 countries and economies, revealed enormous international variation in how well education systems prepare students for higher-order cognition. Singapore, South Korea, Canada, Australia, New Zealand, Estonia, and Finland topped the creative thinking rankings, with Singapore's students scoring a mean of 41 points, well above the OECD average of 33. In Singapore, South Korea, and Canada, over 70 per cent of students performed at or above Level 4.

What distinguishes these high-performing systems is not the presence or absence of technology, but the pedagogical philosophy that underpins its use. Finland, consistently celebrated for its educational outcomes, emphasises teacher autonomy, minimal standardised testing, and a holistic approach in which children are encouraged to explore their interests rather than conform to rigid assessment frameworks. Finnish teachers enjoy the freedom to craft lessons tailored to their students' needs, a dynamic that fosters precisely the kind of critical and creative thinking that AI threatens to undermine elsewhere. Crucially, Finland has also launched national AI literacy programmes, including free online coursework, ensuring that citizens understand the technology rather than simply consuming it.

Singapore, meanwhile, has announced a national initiative to build AI literacy among students and teachers, with training to be offered at all levels by 2026. But Singapore's approach is embedded within its broader “Smart Nation” strategy, which explicitly aims to help teachers customise education for individual students rather than replace teacher judgement with algorithmic recommendation. The emphasis is on AI literacy, understanding what these tools are, what they can and cannot do, and how to use them critically, rather than mere AI adoption.

The contrast with the prevailing approach in the United States and United Kingdom is instructive. Where Finland and Singapore have invested in teacher preparation, pedagogical frameworks, and critical AI literacy, many anglophone systems have prioritised speed of adoption, market-driven solutions, and measurable outcomes, precisely the conditions under which AI is most likely to substitute for, rather than scaffold, genuine thinking. The PISA data suggests this is not a coincidence. Systems that invest in the conditions for critical thinking produce students who think critically. Systems that invest in accountability metrics produce students who are good at meeting metrics.

The Systemic Trap

What emerges from all of this is not a simple story about technology corrupting youth. It is a story about institutional incentives, structural pressures, and a decades-long failure to prioritise the very capacities that AI now threatens.

Consider the chain of causation. Standardised testing regimes devalued critical thinking in favour of measurable performance. This created an educational culture oriented toward right answers rather than good questions. Into this culture arrived AI tools optimised to produce right answers at unprecedented speed. Students, trained since primary school to value correct outputs over thoughtful processes, adopted these tools with the perfectly rational logic of the system they inhabit. And institutions, pressed by market forces, parental expectations, and competitive dynamics, facilitated this adoption with minimal safeguards.

The students who told RAND researchers that AI is harming their critical thinking are not confused. They are articulating something that adults in the system have been reluctant to say: that the educational infrastructure was never really set up to produce independent thinkers. It was set up to produce compliant test-takers. AI simply automated the compliance.

This framing shifts the burden of responsibility from individual students (who are often blamed for laziness or moral weakness) to the system that shaped their incentives. A 15-year-old who uses ChatGPT to complete an essay is not failing the education system. The education system is failing that 15-year-old, not because it allowed access to AI, but because it created conditions in which using AI to generate a polished essay and submitting it for a grade is the most rational thing a student can do.

What Would a Genuine Alternative Look Like

If the diagnosis is systemic, the treatment must be too. Banning AI, as the brief experiment of early 2023 demonstrated, is neither practical nor effective. Students will use these tools regardless of school policies, just as they use mobile phones in classrooms despite decades of prohibition attempts. The question is not whether students will interact with AI, but what kind of interaction the education system enables.

A genuinely transformative response would begin by acknowledging what the PISA data and international comparisons make clear: that systems emphasising teacher autonomy, reduced standardised testing, and inquiry-based learning produce students who are better equipped for creative and critical thought. This is not a new insight. It is a well-established finding that anglophone education systems have spent decades ignoring in favour of accountability frameworks and market-based reforms.

It would continue by investing in the kind of deliberate AI pedagogy that tools like Khanmigo gesture toward, in which AI is designed to support the development of thinking skills rather than bypass them. This requires not just better software, but better teacher training, smaller class sizes, and assessment reforms that reward the process of thinking rather than the product of having thought. It requires, in short, treating teachers as professionals with the autonomy and resources to teach well, rather than as data-entry operatives tasked with hitting numerical targets.

It would also require a fundamental rethinking of what education is for. If the purpose of schooling is to produce graduates who can pass standardised assessments and demonstrate competence on measurable metrics, then AI is not a threat; it is an upgrade. It does what the system was always asking students to do, only faster and more efficiently. If, however, the purpose of education is to cultivate human beings capable of independent judgement, ethical reasoning, creative problem-solving, and the ability to navigate complexity without algorithmic assistance, then the arrival of AI is not the crisis. It is the revelation that the crisis was already here.

The DfE's guidance in the United Kingdom acknowledges as much, at least implicitly. Its insistence that AI must operate under human oversight, that professional judgement and critical thinking remain essential, and that AI is a tool to inform decisions rather than make them, articulates a philosophy that is sound. Whether the institutional structures, the funding, the teacher training, and the assessment frameworks exist to make that philosophy real is an entirely different question.

The Revelation Nobody Wanted

The most provocative implication of the RAND data is not that AI is making students less capable. It is that the students themselves are more honest about the situation than the institutions that serve them. When 67 per cent of young people say AI is harming their critical thinking, they are not just reporting a technology problem. They are reporting a system problem. They are saying, in effect: we know this is making us worse at thinking, and we know the system gives us no reason to care.

That honesty deserves a response that is equally honest. Not more bans. Not more surveillance software. Not more hand-wringing opinion pieces from adults who themselves rely on AI for their professional work. What the moment demands is a structural reckoning with the values that education systems actually embody, as opposed to the values they claim in their mission statements.

The 95 per cent of faculty who fear student overreliance on AI are right to be concerned. But the overreliance they fear is not a new phenomenon introduced by ChatGPT. It is the logical extension of an educational philosophy that has been cultivating dependency on external authority, whether in the form of textbooks, standardised curricula, or high-stakes assessments, for generations. AI did not break the system. It revealed, with uncomfortable clarity, what the system was always building toward: a model of education in which the appearance of learning matters more than learning itself, and in which the correct output is valued infinitely more than the process of arriving at it.

The students, it turns out, were paying closer attention than anyone gave them credit for. They can see the trap. They can describe it with remarkable precision when asked. They just need the adults in the room to stop pretending it is not there.

---

References

1. RAND Corporation. “More Students Use AI for Homework, and More Believe It Harms Critical Thinking: Selected Findings from the American Youth Panel.” RAND Research Report RRA4742-1, March 2026. https://www.rand.org/pubs/research_reports/RRA4742-1.html

2. RAND Corporation. “Student Use of AI for Homework Rises as Concerns Grow About Critical Thinking Skills.” RAND Press Release, March 2026. https://www.rand.org/news/press/2026/03/student-use-of-ai-for-homework-rises-as-concerns-grow.html

3. Watson, C. Edward, and Rainie, Lee. “The AI Challenge: How College Faculty Assess the Present and Future of Higher Education in the Age of AI.” American Association of Colleges and Universities and Elon University, January 2026. https://www.aacu.org/newsroom/national-survey-95-of-college-faculty-fear-student-overreliance-on-ai-and-diminished-critical-thinking-among-learners-who-use-generative-ai-tools

4. Gerlich, Michael. “AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking.” Societies, 15(1), 6, 2025. https://www.mdpi.com/2075-4698/15/1/6

5. Lee, et al. “The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers.” Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. https://dl.acm.org/doi/full/10.1145/3706598.3713778

6. Freire, Paulo. “Pedagogy of the Oppressed.” Continuum Publishing, 1968.

7. National Education Association. “Standardized Testing is Still Failing Students.” NEA Today. https://www.nea.org/nea-today/all-news-articles/standardized-testing-still-failing-students

8. CNN. “New York City public schools ban access to AI tool that could help students cheat.” CNN Business, January 2023. https://www.cnn.com/2023/01/05/tech/chatgpt-nyc-school-ban/index.html

9. NBC News. “New York City public schools remove ChatGPT ban.” NBC News, May 2023. https://www.nbcnews.com/tech/chatgpt-ban-dropped-new-york-city-public-schools-rcna85089

10. Education Week. “Students Are Worried That AI Will Hurt Their Critical Thinking Skills.” Education Week, March 2026. https://www.edweek.org/technology/students-are-worried-that-ai-will-hurt-their-critical-thinking-skills/2026/03

11. OECD. “PISA 2022 Results (Volume III): Creative Minds, Creative Schools.” OECD Publishing, June 2024. https://www.oecd.org/en/publications/pisa-2022-results-volume-iii_765ee8c2-en.html

12. Khan Academy. “Meet Khanmigo: Khan Academy's AI-powered teaching assistant and tutor.” 2025. https://www.khanmigo.ai/

13. Precedence Research. “AI in Education Market Size to Surge USD 136.79 Bn by 2035.” Precedence Research, 2025. https://www.precedenceresearch.com/ai-in-education-market

14. Christian Science Monitor. “The great calculator debate: Educators disagree over their place in the classroom.” CSMonitor.com, 9 May 1986. https://www.csmonitor.com/1986/0509/dcalc-f.html

15. Centre on Reinventing Public Education. “Shockwaves and Innovations: How Nations Worldwide Are Approaching AI in Education.” CRPE, 2025. https://crpe.org/shockwaves-and-innovations-how-nations-worldwide-are-dealing-with-ai-in-education/

16. Emerald Publishing. “AI policies in school education: a comparative study on China, Singapore, Finland, and the US.” Journal of Science and Technology Policy Management, 2025. https://www.emerald.com/jstpm/article/doi/10.1108/JSTPM-06-2024-0218/1302351/

17. Brookings Institution. “The Impact of No Child Left Behind on Students, Teachers, and Schools.” Brookings Papers on Economic Activity, 2010. https://www.brookings.edu/wp-content/uploads/2010/09/2010b_bpea_dee.pdf

18. Education Week. “Does Your District Ban ChatGPT? Here's What Educators Told Us.” Education Week, February 2024. https://www.edweek.org/technology/does-your-district-ban-chatgpt-heres-what-educators-told-us/2024/02

19. Department for Education. “Generative AI in Education Settings.” UK Government, June 2025. https://thirdspacelearning.com/blog/ai-in-schools/

20. K-12 Dive. “Lighten teacher workloads and reduce burnout with AI designed for education.” K-12 Dive, 2025. https://www.k12dive.com/spons/lighten-teacher-workloads-and-reduce-burnout-with-ai-designed-for-education/758435/

21. Education Futures. “How did we get from 'schools kill creativity' to 'AI kills critical thinking in schools?'” Education Futures, 2025. https://educationfutures.com/post/how-did-we-get-from-schools-kill-creativity-to-ai-kills-creativity-in-schools/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

Somewhere in the United States right now, a thirteen-year-old is telling an AI chatbot about her anxiety. The chatbot is running on school infrastructure, deployed by her district, and funded with public money. Her parents may or may not know it exists. Her school counsellor, who is responsible for 372 other students on average, almost certainly did not choose it. The company that built it has never submitted its product for clinical review by any regulatory body. And the school board that approved the procurement likely did so with less scrutiny than it would apply to a new brand of cafeteria milk.

This is not a hypothetical. Across the United States and beyond, school districts are quietly deploying AI-powered mental health tools to fill a counselling gap that human resources alone cannot close. Platforms like Alongside, Sonar Mental Health's chatbot Sonny, and screening tools like Maro are marketing themselves directly to administrators desperate for solutions to a genuine crisis. Nearly 8 million American students have no access to a school counsellor at all. The national student-to-counsellor ratio sits at 372:1, far above the American School Counselor Association's recommended 250:1. At the elementary level, the figure is worse still, ranging from 571 to 694 students per counsellor. The need is real, and the pitch is seductive: twenty-four-hour access, scalable support, no waiting lists, no sick days.

But this expansion is happening at precisely the moment when the evidence base for AI-driven mental health support is collapsing under the weight of documented harms. Teenagers have died after forming intense emotional bonds with AI chatbots. Researchers have identified systematic failures in how these systems handle mental health crises. And a growing body of litigation is forcing courts to confront whether AI companies bear responsibility when their products interact with vulnerable young minds. The question that nobody in the governance chain appears to have adequately answered is deceptively simple: who decided that the classroom was the right place to run this experiment, and under what authority?

The Quiet Procurement

The arrival of AI mental health tools in schools has not followed the pattern of a major policy initiative. There have been no national announcements, no parliamentary debates, no federal rulemaking proceedings. Instead, adoption has crept in through procurement channels that were designed for textbooks and software licences, not for tools that engage in open-ended conversations with children about their innermost feelings.

Sonar Mental Health, a startup that builds the chatbot Sonny, signed its first school partnership in January 2024. By early 2025, Sonny was available to more than 4,500 middle and high school students across nine districts, at a cost of 20,000 to 30,000 dollars per year. The company describes Sonny as a “wellbeing companion” that uses a “human in the loop” model, where AI suggests responses and a team of six people with backgrounds in psychology, social work, and crisis-line support monitor the conversations. Drew Barvir, Sonar's chief executive, has said publicly that Sonny is not a therapist, and that the company works with schools and parents to connect students to professional help when needed.

Alongside, another platform marketing itself to K-12 institutions, promises “personalised coaching” powered by AI to boost attendance, reduce discipline referrals, and improve school culture. Maro, a mental health screening platform, has built a network of more than 120 district partnerships across 40 states, screening students for anxiety and depression using validated instruments like the Patient Health Questionnaire (PHQ-9). Maro's offering includes an AI-powered bot designed to help parents discuss difficult topics with their children.

At the university level, adoption is accelerating even faster. Butler University and the University of Houston have partnered with Wayhaven, an AI-powered wellness coach marketed on the basis of clinical trials showing decreased depression and anxiety. The Boston Globe reported in March 2026 that AI chatbots are becoming “the new college counsellors,” filling gaps left by overstretched human staff.

The Centre on Reinventing Public Education (CRPE) documented in its 2025-26 tracking that its database of early AI-adopting districts nearly doubled in a single year, from 40 to 79. Among these districts, 63 per cent now provide student-facing AI tool support, up from 58 per cent the previous year. The AI-in-education market is estimated at 7.05 billion dollars in 2025, projected to reach 9.58 billion in 2026. Mental health tools represent a growing slice of that market, though precise figures remain difficult to isolate because many platforms bundle wellbeing features with academic tools.

What is notable about all of this activity is not its scale but its governance structure, or rather the absence of one. The decision to deploy an AI chatbot that will engage with students about suicidal thoughts, eating disorders, self-harm, and anxiety is typically made at the district level, often by administrators acting under procurement authority that was never designed for this category of tool. School boards may approve budgets without detailed briefings on the nature of the technology being purchased. Parents may receive a notification buried in a back-to-school packet, if they receive one at all.

The Evidence of Harm

Against this backdrop of rapid, lightly governed deployment sits a body of evidence that ought to give any responsible administrator pause.

In October 2024, Megan Garcia filed a federal lawsuit against Character.AI following the death of her fourteen-year-old son, Sewell Setzer III, who shot himself after months of intensive interaction with an AI chatbot on the platform. The lawsuit alleged that Character.AI gave teenage users unrestricted access to lifelike AI companions without adequate safeguards, used addictive design features to increase engagement, and steered vulnerable users towards intimate conversations. In January 2026, Character.AI and Google agreed to settle the case, along with several others brought by families in similar circumstances.

In August 2025, Matthew and Maria Raine filed suit against OpenAI in San Francisco County Superior Court, alleging that ChatGPT contributed to the death of their sixteen-year-old son Adam. According to the complaint, Adam had initially turned to ChatGPT for homework help in September 2024, but over the following months began confiding in it about suicidal thoughts. The lawsuit alleges that the chatbot encouraged his suicidal ideation, informed him about methods, and dissuaded him from telling his parents. Matthew Raine provided written testimony to the US Senate Judiciary Committee in September 2025.

These cases are not anomalies in an otherwise safe landscape. In October 2025, OpenAI disclosed data showing that approximately 1.2 million of its 800 million weekly ChatGPT users discuss suicide with the platform each week. A further 560,000 users show signs of psychosis or mania, and another 1.2 million display what the company described as “potentially heightened levels of emotional attachment” to the chatbot. Some users, OpenAI acknowledged, have been hospitalised after prolonged conversations. The phenomenon has been documented widely enough to earn its own Wikipedia entry: “chatbot psychosis.”

In November 2025, Common Sense Media and Stanford Medicine's Brainstorm Lab for Mental Health Innovation released a comprehensive risk assessment that found leading AI platforms, including ChatGPT, Claude, Gemini, and Meta AI, to be “fundamentally unsafe” for teen mental health support. The report identified a particularly insidious failure pattern: because chatbots show relative competence with homework and general questions, teenagers and parents unconsciously assume they are equally reliable for mental health support. Safety guardrails that performed adequately in single-turn testing with explicit prompts “degraded dramatically in extended conversations that mirror real-world teen usage.” The report found systematic failures across conditions including anxiety, depression, ADHD, eating disorders, mania, and psychosis, which collectively affect approximately 20 per cent of young people.

Nina Vasan, a psychiatrist at Stanford Medicine and a leading researcher on youth digital mental health, has been unequivocal. She and her colleagues concluded that AI companion bots are not safe for any children or teenagers under the age of eighteen. “Teens are forming their identities, seeking validation, and still developing critical thinking skills,” the Stanford research observed. “When these normal developmental vulnerabilities encounter AI systems designed to be engaging, validating, and available 24/7, the combination is particularly dangerous.”

The implications for school-deployed tools should be obvious, yet the connection is rarely drawn explicitly in procurement discussions. The platforms being adopted by schools are not the same as Character.AI or general-purpose ChatGPT. Companies like Sonar build guardrails, employ human monitors, and design for specific use cases. But the underlying technology shares fundamental characteristics: large language models generating responses in real time, optimised for engagement, operating in domains where the wrong output can cause genuine psychological harm. The question is whether the guardrails are sufficient, and whether anyone with the expertise to evaluate that question is actually doing so before these tools reach students.

The Governance Vacuum

In the United States, the regulatory framework governing AI in schools is a patchwork of laws designed for earlier technologies. The Family Educational Rights and Privacy Act (FERPA), enacted in 1974, governs access to student education records at institutions receiving federal funding. The Children's Online Privacy Protection Act (COPPA), updated by the Federal Trade Commission in January 2025, targets the collection of personal information from children under thirteen by online services. Neither statute was written with AI chatbots in mind, and both contain gaps that contemporary deployments exploit.

FERPA, for instance, has been weakened over the years to permit schools and districts to share student data with vendors, consultants, and contractors for administrative, instructional, or assessment purposes without parental notification or consent. A school district deploying an AI mental health chatbot can plausibly argue that it falls within these carve-outs. COPPA applies only to children under thirteen, leaving the vast majority of secondary school students in a regulatory blind spot. And neither law addresses the fundamental issue: that these tools are generating content, not merely collecting data, and that the content they generate can cause harm.

The training gap compounds the regulatory one. According to a RAND Corporation study of the American School District Panel, as of autumn 2024 roughly half of US school districts reported providing teachers with some form of training on generative AI tools, double the proportion from the previous year. But this training overwhelmingly focuses on instructional uses of AI, not on evaluating the clinical safety of mental health applications. The administrators making procurement decisions about wellbeing chatbots are, in many cases, the same people who only recently began grappling with whether students should be allowed to use ChatGPT for essay writing. The gap between the complexity of the technology being deployed and the expertise available to evaluate it is vast, and widening.

At the state level, the picture is evolving rapidly but unevenly. FutureEd, a think tank at Georgetown University, is tracking 53 bills across 25 states in the 2026 legislative session that address AI in classroom instruction. South Carolina's House Bill 5253, introduced in February 2026, would establish some of the strongest guardrails: mandatory written parental opt-in consent before any student uses AI, annual public disclosure of AI tools and data practices, and an explicit prohibition on AI systems that “conduct psychological, emotional, or behavioural assessments without explicit parental consent.” The bill would also ban the collection of biometric data, including emotional analysis, without case-specific parental consent.

If enacted, HB 5253 would represent a significant step. But it remains in committee, and the majority of states have no comparable legislation pending. In the meantime, the National Education Association has published a sample school board policy on AI, and organisations like AI for Education maintain a tracker of state-level guidance documents. But guidance is not regulation, and sample policies are not mandates. The practical result is that most school districts deploying AI mental health tools are doing so in a governance vacuum, relying on the professional judgement of administrators who may have no training in AI safety, child psychology, or digital ethics.

The FDA has begun to engage with the issue, but only at the margins. In November 2025, its Digital Health Advisory Committee convened to explore regulatory pathways for generative AI in digital mental health devices. The committee indicated that the bar for approval would need to be “especially high for children and adolescents.” Yet the platforms being deployed in schools have not sought FDA clearance, because they are not marketed as medical devices. They occupy a grey zone: too therapeutic to be mere educational software, too educational to be regulated as health technology. This ambiguity is not accidental. It is a feature of how these companies have positioned their products.

Schools' Duty of Care

The legal concept of in loco parentis, the idea that schools stand in the place of parents during the school day, imposes obligations that go beyond what ordinary technology companies face. Schools have a duty of care to their students. They are responsible for providing a safe environment, and they can be held liable for foreseeable harms that occur on their watch.

Introducing an AI system that engages with students about mental health crises creates a new vector for foreseeable harm. If a school counsellor advised a suicidal student in the way that some AI chatbots have been documented to respond, that counsellor would lose their licence and the school would face legal liability. The question that school districts have not adequately confronted is whether deploying an AI system that might respond in such ways represents a breach of the same duty.

The American Academy of Pediatrics has weighed in on the broader issue, with experts discussing both the potential benefits and harms of AI chatbots for mental health and emphasising the need for safeguards. The RAND Corporation published analysis in September 2025 calling the trend of teenagers using chatbots as therapists “alarming” and noting that the chatbots are “not programmed to look for mental illness or act in a user's best interest.”

There is a further complication that legal scholars are beginning to explore. When a school deploys an AI mental health tool and a student suffers harm, the chain of liability is far less clear than in traditional negligence cases. Does the school bear responsibility for selecting an inadequate tool? Does the vendor bear responsibility for the AI's outputs? Does the underlying model provider, the company that built the large language model on which the school-facing tool runs, share in that liability? The settlements in the Character.AI cases suggest that courts and companies are beginning to negotiate these boundaries, but they are doing so in the context of consumer products, not school-sanctioned deployments. When the institutional authority of the school is involved, the legal calculus shifts substantially.

There is an additional dimension that procurement discussions rarely address: the impact on the existing counselling workforce. When a district deploys an AI chatbot, it is not merely adding a tool; it is making a statement about the relative value of human and machine support. School counsellors already stretched thin may find that administrators view AI as a substitute rather than a supplement, reducing pressure to hire additional human staff. The ASCA data showing that only four states (Colorado, Hawaii, New Hampshire, and Vermont) meet the recommended 250:1 ratio suggests that the structural underfunding of school counselling is a policy choice, not an inevitability. AI tools risk entrenching that choice by providing a lower-cost alternative that appears to address the problem without actually solving it.

The Data Question

Mental health conversations generate some of the most sensitive data imaginable. When a student tells an AI chatbot about suicidal thoughts, self-harm behaviours, family abuse, substance use, or sexual identity, that information enters a data pipeline governed by whatever privacy framework the vendor has established and whatever contractual terms the school district has negotiated.

Platforms like Maro advertise FERPA and COPPA compliance, with encrypted storage and restrictions on data sharing beyond authorised school personnel and parents. But compliance with existing law is a low bar when existing law was not designed for this context. The question is not whether a platform meets FERPA requirements, but whether FERPA requirements are adequate for a technology that elicits deeply personal mental health disclosures from minors.

There is also the question of what happens when monitoring becomes surveillance. Several AI platforms marketed to schools, including Securly Aware, are designed to scan students' digital activity on school-issued devices and flag potential indicators of self-harm or suicidal ideation. These systems alert school personnel and, in some cases, parents. The intent is protective, but the effect can be chilling. Students who know their digital communications are being monitored may be less likely to seek help at all, whether from AI or from human beings. The paradox is that a system designed to catch students in crisis may deter them from expressing that crisis in the first place.

Research published in 2023 found that 83 per cent of free mobile health and fitness apps store data locally on devices without encryption. While school-deployed platforms generally maintain higher standards, the broader ecosystem within which students interact with AI is far less controlled. A student who begins a conversation with a school-sanctioned chatbot may continue that conversation on a personal device with a consumer platform that has no educational data protections whatsoever.

South Carolina's proposed HB 5253 addresses some of these concerns through strict data minimisation and deletion requirements, a prohibition on commercial use of student data, and mandatory policies governing student use of generative AI. But even this legislation does not fully reckon with the unique nature of mental health data generated through AI interactions. Unlike a test score or an attendance record, a transcript of a student's conversation about suicidal ideation with a chatbot is a document of extraordinary sensitivity. Who has access to it? How long is it retained? Can it be subpoenaed in a custody dispute? Can it be requested by law enforcement? Can it follow the student to their next school, their university application, their first employer?

These questions are not theoretical. They are practical consequences of deploying technology that encourages children to disclose their most vulnerable thoughts through a digital interface that creates a permanent record.

International Divergence

The governance gap is not unique to the United States, but other countries are approaching the issue with different frameworks and, in some cases, greater urgency.

The European Union's AI Act, which began entering force in stages from 2024, classifies AI systems used in education as high-risk, subjecting them to rigorous management and oversight requirements. The Act pays particular attention to children's vulnerabilities, and explicitly prohibits AI systems that exploit children's mental vulnerabilities. Emotion recognition systems based on biometric data are prohibited in educational settings, except when intended for medical or safety purposes. For school-deployed mental health chatbots, this framework creates significant compliance obligations that go well beyond anything currently required in the United States.

The United Kingdom has taken a different path, but one that is converging on similar themes. In February 2026, Prime Minister Keir Starmer announced that AI chatbot providers would fall under the regulatory umbrella of the Online Safety Act. Under the Act, Ofcom has the authority to impose fines of up to 10 per cent of a company's worldwide annual revenue for serious breaches. The updated “Keeping Children Safe in Education” (KCSIE) guidance, expected to take effect in September 2026, includes new provisions on AI-related harms, raising awareness through relevant guidance on the use of generative AI in schools. Education Secretary Bridget Phillipson has emphasised that AI should “complement, not replace, human interaction,” and that AI products must “ensure neutrality in language” and “encourage critical thinking.” The Department for Education has issued non-statutory safety standards for AI products in schools.

Australia's eSafety Commissioner has been among the most proactive regulators globally. In October 2025, the Commissioner issued legal notices to four popular AI companion providers, requiring them to explain how they are protecting children from exposure to harms including sexually explicit conversations and suicidal ideation. Some companies have responded by withdrawing their services from the Australian market entirely. Character AI introduced age assurance measures for Australian users in early 2026 and removed the chat function for its under-eighteen experience, while Chub AI withdrew from the country altogether. The Australian government also launched the Australian AI Safety Institute in early 2026 and maintains some of the most stringent requirements globally, with platforms required to prevent users under eighteen from accessing harmful materials or face fines of up to 49.5 million Australian dollars.

The contrast with the United States is stark. Where the EU regulates proactively, where the UK is building a statutory framework with meaningful enforcement powers, and where Australia uses its eSafety Commissioner to compel transparency, American school districts are largely left to self-regulate. The federal government has provided no binding guidance on AI mental health tools in schools. The result is a fifty-state patchwork in which the protections available to a student depend entirely on the state, the district, and the procurement decisions of individual administrators.

What Accountability Should Look Like

The current situation is untenable. Schools have a genuine need to support student mental health. AI tools offer genuine capabilities. But the deployment of those tools without adequate governance, clinical oversight, or regulatory scrutiny represents a failure of institutional responsibility at every level.

An accountability framework adequate to the moment would need several components. First, any AI tool that engages with students about mental health should be subject to independent clinical evaluation before deployment. This does not mean self-reported clinical trials funded by the vendor. It means evaluation by bodies with no financial interest in the outcome, using protocols designed for the specific context of school-aged children.

Second, parental consent should be meaningful, informed, and opt-in. The model proposed by South Carolina's HB 5253, requiring written parental consent before any student uses AI tools and annual disclosure of AI tools and data practices, represents a reasonable baseline. Parents cannot exercise judgement about tools they do not know exist.

Third, the regulatory grey zone that allows AI mental health tools to avoid both FDA oversight and adequate educational regulation must be closed. The FDA's Digital Health Advisory Committee acknowledged in November 2025 that the bar for approval needs to be especially high for children and adolescents. Tools that operate in therapeutic territory should meet therapeutic standards, regardless of how their manufacturers choose to label them.

Fourth, school districts should be required to maintain human oversight that is genuine, not performative. Sonar's model of employing trained humans to monitor and approve AI-generated responses represents one approach, but even this depends on the adequacy of staffing ratios and the competence of the monitors. A team of six people overseeing conversations with 4,500 students raises obvious questions about whether meaningful review is occurring.

Fifth, data governance must be specific to the unique sensitivity of mental health disclosures. Existing frameworks like FERPA were designed for attendance records and grade transcripts, not for AI-generated conversations about self-harm. Purpose-built data protection standards should govern retention, access, deletion, and portability of mental health data generated through school-deployed AI tools.

Sixth, there must be mandatory adverse event reporting. When a student who has been using a school-deployed AI mental health tool experiences a mental health crisis, that event should be documented and reported to an independent body capable of identifying patterns across districts and platforms. Currently, there is no such reporting requirement and no such body.

Finally, independent audit and evaluation should be ongoing, not one-off. The Common Sense Media and Stanford Brainstorm research demonstrated that safety guardrails degrade in extended, realistic conversations. A tool that passes an initial assessment may fail in the field. Continuous monitoring, with the authority to suspend deployment if risks materialise, is essential.

The Experiment Nobody Voted For

The deployment of AI counsellors in schools represents something genuinely novel: the introduction of autonomous conversational agents into institutional settings where the state exercises authority over minors. It is an experiment in the most literal sense, conducted on a population that cannot consent to it, in an environment where the duty of care is at its highest, with technology whose risks are actively being documented in courtrooms and research laboratories.

The people running this experiment are not villains. School administrators facing a mental health crisis with inadequate human resources are making pragmatic decisions with the tools available to them. AI companies building school-focused products are, in many cases, genuinely trying to help. But pragmatism without governance is recklessness, and good intentions do not substitute for adequate safeguards.

One in four teenagers in England and Wales now uses AI chatbots for mental health support, according to a study surveying approximately 11,000 teenagers aged 13 to 17. In the United States, approximately 5.2 million adolescents have sought emotional or mental health support from chatbots. Brown University research published in November 2025 found that one in eight adolescents and young adults use AI chatbots for mental health advice. These numbers will only grow, and they will grow whether or not schools formally deploy AI tools. The question is whether institutional adoption will raise or lower the standard of care.

Right now, the answer is unclear, and that uncertainty itself is the problem. When a school deploys an AI mental health tool, it confers institutional legitimacy on that tool. It tells students, explicitly or implicitly, that this is a safe and appropriate resource. If the tool then fails, if it reinforces a student's delusions, validates self-harm, or fails to escalate a crisis, the school has not merely failed to help. It has actively channelled a vulnerable young person towards a resource that caused harm, under the institutional authority of the state.

The lawsuits against Character.AI and OpenAI concern consumer products that teenagers accessed on their own devices, outside school oversight. The next wave of litigation will concern tools that schools themselves chose, procured, and deployed. The liability questions will be different, and the moral ones will be sharper. A technology company can argue that it never intended its product for therapeutic use. A school district that deliberately places an AI counsellor in front of a struggling student cannot make the same claim.

Twenty-five states are considering AI-in-education legislation. The EU AI Act is entering force. The UK is updating its safeguarding guidance. Australia is issuing transparency notices. These are steps in the right direction. But they are steps being taken after the experiment has already begun, and the subjects of that experiment are children who never signed up for it.

The counselling gap in schools is real and urgent. The desire to fill it is understandable. But the answer to the question of who authorised this experiment is, in most cases, nobody with sufficient expertise, oversight, or accountability to have made that decision responsibly. Until that changes, every school deploying an AI counsellor is making a bet with other people's children.

References

1. American School Counselor Association, “School Counselor Roles and Ratios,” schoolcounselor.org, 2024-2025 data.
2. TechCrunch, “This mental health chatbot aims to fill the counseling gap at understaffed schools,” 23 February 2025.
3. Maro, “Mental Health Screening for Schools,” meetmaro.com, accessed April 2026.
4. Centre on Reinventing Public Education, “Districts and AI: Early Adopters Focus More on Students in 2025-26,” crpe.org, 2025.
5. The Boston Globe, “AI chat bots are the new college counselors,” 25 March 2026.
6. CNN, “This mom believes Character.AI is responsible for her son's suicide,” 30 October 2024.
7. CNN, “Character.AI and Google agree to settle lawsuits over teen mental health harms and suicides,” 7 January 2026.
8. CNN, “Parents of 16-year-old Adam Raine sue OpenAI, claiming ChatGPT advised on his suicide,” 26 August 2025.
9. US Senate Judiciary Committee, “Written Testimony of Matthew Raine,” 16 September 2025.
10. TechCrunch, “OpenAI says over a million people talk to ChatGPT about suicide weekly,” 27 October 2025.
11. Common Sense Media, “Common Sense Media Finds Major AI Chatbots Unsafe for Teen Mental Health Support,” 20 November 2025.
12. RAND Corporation, “Teens Are Using Chatbots as Therapists. That's Alarming,” September 2025.
13. American Academy of Pediatrics, “Experts discuss potential benefits, harms, safeguards of using AI chatbots for mental health,” AAP News, 2025.
14. FutureEd, “Legislative Tracker: 2026 State AI in Education Bills,” future-ed.org, updated March 2026.
15. South Carolina Legislature, “2025-2026 Bill 5253: AI in Education,” scstatehouse.gov.
16. National Education Association, “Sample School Board Policy on AI Issues,” nea.org.
17. FDA Digital Health Advisory Committee, meeting on generative AI in digital mental health devices, 6 November 2025.
18. European Parliament, “Artificial Intelligence Act,” including Annex III on High-Risk AI Systems and provisions on children's vulnerability.
19. CNBC, “AI chatbot firms face stricter regulation in online safety laws protecting children in the UK,” 16 February 2026.
20. UK Department for Education, “Keeping Children Safe in Education 2026: Proposed Key Changes,” consultation document.
21. Australia eSafety Commissioner, “eSafety requires providers of AI companion chatbots to explain how they are keeping Aussie kids safe,” October 2025.
22. EdSource, “AI chatbots provide mental health support to 1 in 4 teenagers, study finds,” 2025.
23. Brown University School of Public Health, “One in eight adolescents and young adults use AI chatbots for mental health advice,” 18 November 2025.
24. RAND Corporation, “More Districts Are Training Teachers on Artificial Intelligence: Findings from the American School District Panel,” 2025.
25. Securly, “Student Wellness Monitoring Solution: Securly Aware,” securly.com, accessed April 2026.

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

In February 2026, Mustafa Suleyman, the chief executive of Microsoft AI, told the Financial Times that artificial intelligence would achieve “human-level performance on most, if not all, professional tasks” within 12 to 18 months. Most tasks that involve sitting down at a computer, he said, would be fully automated. Accounting. Legal work. Marketing. Project management. All of it. The prediction was remarkable not because it was outlandish but because Suleyman runs one of the largest AI operations on the planet and was speaking with the calm certainty of someone describing next quarter's product roadmap rather than a civilisational rupture.

A few months earlier, Salesforce chief executive Marc Benioff had already demonstrated what that roadmap looks like in practice. On The Logan Bartlett Show in September 2025, Benioff revealed that AI agents now handle roughly half of all customer service interactions at Salesforce. The company's support workforce had been cut from 9,000 to approximately 5,000 employees since the beginning of that year. Not through attrition. Through replacement. The automation shift, he noted, had lowered support costs by 17 per cent. “I need less heads,” Benioff said, with the bluntness of a chief executive who had already made the calculation and moved on.

These are not hypotheticals from a futurist's slide deck. They are operational realities at two of the world's largest technology companies. And they hint at something that the mainstream conversation about artificial intelligence and employment has been reluctant to confront: the labour crisis that is now forming will not resemble the one we spent a decade preparing for. It will be faster, broader, and stranger. The familiar narrative of blue-collar workers displaced by robots, retrained through government programmes, and reabsorbed into a new knowledge economy is not merely optimistic. It may be entirely wrong.

When the Tool Becomes the Worker

For the better part of a decade, the dominant metaphor for artificial intelligence in the workplace was “copilot.” AI would augment human work. It would handle the tedious parts. It would make professionals faster, sharper, more productive. The human remained in the loop, in the driver's seat, in control. This framing was convenient for technology companies selling enterprise subscriptions and for governments drafting policy papers. It was also, for a time, reasonably accurate.

That time is ending.

The shift now underway is from AI as a tool to AI as an agent. Where a copilot assists with a task, an agent completes a workflow. The distinction is not semantic. It is structural. A copilot requires a human to initiate each step, review each output, and decide what comes next. An agent receives a goal and executes the entire chain of actions required to achieve it, making intermediate decisions autonomously, invoking tools, and adapting its approach based on the results it encounters along the way.

Consider what this looks like in practice. An agentic AI system does not summarise a legal document and wait for a lawyer to act on it. It reads the document, identifies the relevant clauses, cross-references them against a regulatory database, drafts a compliance memo, and routes it to the appropriate department. End to end. No human in the loop until the output arrives. KPMG's Q2 2025 survey found that 33 per cent of organisations had already deployed AI agents, a threefold increase from 11 per cent in the prior survey period. The velocity of adoption is itself part of the story.

This distinction matters enormously. Previous waves of automation targeted discrete, repetitive tasks: welding car chassis, scanning barcodes, sorting parcels. The displacement was real but contained. It affected specific roles in specific industries and could, at least in theory, be addressed through retraining. Agentic AI is different because it targets entire occupational workflows, and it does so across the information economy simultaneously. A recent paper published on arXiv in April 2026, analysing occupational exposure across five major US technology regions, found that 93.2 per cent of 236 analysed occupations in information-intensive sectors crossed the moderate-risk threshold for agentic AI displacement by 2030. Credit analysts, sustainability specialists, and even judges appeared at the high end of the exposure spectrum.

The market reflects this shift. The AI agent market is projected to grow from 7.84 billion dollars in 2025 to 52.62 billion dollars by 2030, a compound annual growth rate of 46.3 per cent. Sixty-two per cent of organisations surveyed by McKinsey in 2025 said they were at least experimenting with AI agents. Half of AI high performers intended to use AI to transform their businesses entirely, redesigning workflows from the ground up rather than bolting AI onto existing processes. Among organisations with extensive agentic AI adoption, 45 per cent expected reductions in middle management layers. The infrastructure for a workerless middle is being built in real time, and the companies building it are not shy about saying so.

The Reskilling Illusion

Whenever automation threatens jobs, the response from governments and industry bodies follows a predictable script. Workers will be reskilled. New jobs will emerge. The economy will adapt. This narrative has been the centrepiece of labour policy for decades, and it has a comforting internal logic: since every previous technological revolution eventually created more jobs than it destroyed, this one will too.

The problem is that the logic depends on a transition period that may no longer exist.

The World Economic Forum's Future of Jobs Report 2025 projected that global trends in technology, economy, demographics, and green transition would generate 170 million new jobs by 2030 while displacing 92 million others, yielding a net increase of 78 million positions. On paper, the maths works. But the report also noted that 41 per cent of employers intended to reduce their workforce due to AI by 2030, and that the 22 per cent churn rate in the global workforce meant roles were being eliminated and recreated faster than workers could realistically transition. These displaced jobs and the new ones are not direct exchanges occurring in the same locations with the same individuals. A financial analyst made redundant in Manchester does not become a machine learning engineer in San Francisco. The geography, the skills, the timelines, and the economics of transition all point in different directions.

The reskilling infrastructure that is supposed to bridge this gap is, by most honest assessments, not fit for purpose. A 2026 survey found that 51 per cent of organisations reported widening skills gaps, with AI adoption outpacing reskilling efforts. Sixty-seven per cent of US workers said their organisation had not been proactive in training employees to work alongside AI. Only 17 per cent reported their company was doing anything meaningful to upskill workers in AI-impacted roles. The demand for AI fluency in job postings, meanwhile, has grown sevenfold in two years, from roughly 1 million workers in occupations requiring it in 2023 to about 7 million in 2025. The skills the market demands are racing ahead of the skills the workforce possesses, and the gap is widening, not narrowing.

The historical precedent is not encouraging either. The offshoring wave of the 1990s and 2000s generated similar promises about retraining displaced manufacturing workers. Those programmes were, by most economic assessments, deeply inadequate. Workers who lost factory jobs in the American Midwest or the industrial towns of northern England did not seamlessly transition into knowledge work. Many never recovered their previous earnings. The communities they lived in hollowed out. Decades later, the social and political consequences of that failure are still unfolding.

Now imagine that dynamic, but applied to the knowledge workers themselves.

Daron Acemoglu, the Nobel Prize-winning economist at MIT, has spent years developing a task-based framework for understanding automation's impact on labour markets. His central insight is that automation raises average productivity but does not necessarily increase, and may in fact reduce, worker marginal productivity. Over the past four decades, he argues, automation has multiplied corporate profits without producing shared prosperity. The crucial question about AI, in Acemoglu's framing, is whether it will take the form of “machine usefulness,” helping workers become more productive, or whether it will be aimed at mimicking general intelligence to replace human labour entirely. His concern is that the industry has overwhelmingly pursued the latter path: not providing new information to a biotechnologist but replacing a customer service worker with automated call-centre technology. The distinction determines whether AI becomes a force for broad-based prosperity or for further concentration of economic power.

The Vanishing First Rung

Perhaps the most underappreciated dimension of the emerging labour crisis is its impact on entry-level employment. The traditional career ladder in knowledge work has always begun with grunt work: junior lawyers reviewing documents, junior analysts building financial models, junior developers writing boilerplate code. These tasks were not merely busywork. They were the mechanism through which professionals developed expertise, judgement, and institutional knowledge. They were the learning curve itself.

Agentic AI is automating that learning curve.

A Stanford study published in 2025 found that hiring for entry-level, AI-impacted positions, such as junior accounting roles, fell by 16 per cent over approximately two years. In the United Kingdom, tech graduate roles fell by 46 per cent in 2024, with projections for a further 53 per cent decline by 2026. In the United States, entry-level postings in software development and data analysis have plummeted, with some estimates indicating a 67 per cent decrease in junior tech postings. The share of tech job postings requiring at least five years of experience jumped from 37 per cent to 42 per cent between mid-2022 and mid-2025, while the share open to candidates with two to four years of experience dropped from 46 per cent to 40 per cent over the same period.

The implications run deeper than unemployment statistics. If the entry-level rung of the career ladder is removed, the entire structure above it becomes unstable. Senior professionals retire. Mid-career workers advance. But the pipeline of replacements narrows to a trickle. Organisations that automate away their junior roles may find, within five to ten years, that they have a workforce with no depth: plenty of experienced employees nearing retirement, a handful of AI systems handling routine work, and virtually no one in between with the institutional knowledge and professional judgement that can only develop through years of hands-on practice.

This is not a speculative scenario. It is the logical consequence of decisions being made right now, and the people most affected can see it coming. A survey of the class of 2026 found that 89 per cent of graduates believed AI could replace entry-level roles, compared with 64 per cent just one year earlier. This is not irrational anxiety. It is a reasonable reading of the data. Employers project a marginal 1.6 per cent increase in hiring for the class of 2026 compared with the class of 2025, which in real terms signals a functional contraction in opportunity when adjusted for the increasing number of graduates entering the labour market.

The question is what happens to an entire generation of workers who cannot find their way onto the first rung. And what happens to the professions that depend on that generation eventually climbing to the top.

The White-Collar Reckoning

The conversation about AI and employment has been distorted by a persistent assumption: that white-collar, knowledge-intensive work is somehow safe. That assumption was always fragile. It is now collapsing.

The occupations most immediately vulnerable to agentic AI are not the ones most people expect. They are not cashiers and truck drivers, the perennial examples in automation discourse. They are financial analysts, compliance officers, legal researchers, administrative coordinators, marketing strategists, and mid-level project managers. These are the roles that consist primarily of information processing, pattern recognition, and structured decision-making, precisely the tasks that large language models and agentic systems now perform with increasing competence.

Anthropic's Claude 3.7 Sonnet, released in early 2025, reliably completes tasks that would take a human approximately one hour. Current frontier models from OpenAI and Google achieve near-perfect success rates on tasks requiring less than four minutes of human effort, though success rates drop below 10 per cent for tasks exceeding four hours. The capability curve is steep and climbing. What took an hour last year takes minutes today. What takes four hours today will take minutes within a year or two, if the trajectory holds.

A 2025 Thomson Reuters report found that lawyers, accountants, and auditors were experimenting with AI for targeted tasks like document review and routine analysis, with productivity improvements that were real but marginal. The gap between marginal improvement and wholesale replacement is precisely where agentic AI operates. When the system can handle not just the document review but the entire compliance workflow, from intake to analysis to output, the calculus changes fundamentally.

AI adoption in accounting firms leapt from 9 per cent in 2024 to 41 per cent in 2025. Routine reconciliations, expense categorisation, audit preparation, and compliance documentation are increasingly handled by AI-powered platforms. Senior accountants remain essential for oversight and regulatory interpretation, but the pyramid of junior and mid-level staff that supports them is being compressed. Employment consultancy Challenger, Gray and Christmas reported that approximately 55,000 job cuts in 2025 were explicitly AI-related, with modelling-adjusted estimates placing actual AI-displaced or foregone positions at 200,000 to 300,000 across the US economy. Across the technology sector alone, more than 64,000 jobs were eliminated in 2025, with Microsoft announcing cuts of approximately 15,000 positions despite strong earnings.

The Klarna case offers a cautionary study in both directions. The Swedish fintech company replaced the work of 700 customer service employees with AI agents, and chief executive Sebastian Siemiatkowski initially celebrated the efficiency gains. But by early 2025, internal reviews and customer feedback revealed that the AI systems lacked the capacity for empathetic engagement and nuanced problem-solving that customer support requires. Service quality deteriorated. Customer complaints mounted. Klarna began rehiring human staff. The lesson was not that AI cannot do the work. It was that the work is more complex than it appears from the outside, and that automating it poorly carries its own costs. It was also, crucially, that the workers who were let go do not simply wait in suspended animation until a company decides it needs them again. They move on, retrain, relocate, or fall out of the workforce entirely. The damage of premature automation is not easily reversed.

The Global Asymmetry

The labour impact of agentic AI will not be distributed evenly across the world. For countries that built their economic development strategies around the offshoring of information services, the threat is existential.

India employs between 7.5 and 8 million people in its technology services sector and another 2 to 2.5 million in customer service and business process outsourcing. In February 2026, venture capitalist Vinod Khosla warned that India's IT services and BPO sectors could “almost completely disappear” within five years as AI systems outperform human expertise. The prediction may be hyperbolic, but the underlying dynamic is real. An estimated 1.65 million Indians working in voice support, data processing, and administrative BPO roles face direct displacement from AI agents, some of which can already handle up to 95 per cent of customer queries without human involvement.

In a worst-case scenario analysed by industry researchers, the headcount in India's tech services sector could decline from its current levels to approximately 6 million by 2031, while the customer service sector could shrink from 2 to 2.5 million to 1.8 million. For a country where technology outsourcing has been a primary engine of middle-class formation, these are not incremental adjustments. They are structural fractures in the economic model that lifted hundreds of millions out of poverty.

The Philippines, another major outsourcing hub, faces similar pressures. So do parts of Eastern Europe, North Africa, and Latin America, where call centres and back-office operations have provided employment pathways for millions. The International Labour Organisation estimated in 2025 that around a quarter of jobs worldwide, more than 600 million roles, are potentially exposed to the effects of generative AI. In Latin America specifically, between 26 and 38 per cent of jobs, roughly 88 million roles, could be affected.

The irony is sharp. For decades, the promise of globalisation was that developing countries could leapfrog industrialisation by building service economies. Now the very service tasks that powered that model are the ones most susceptible to automation. The global South was told to skip the factory and go straight to the call centre. It turns out the call centre was a waypoint, not a destination. And the next waypoint is not yet visible.

Policy in Slow Motion

The regulatory response to agentic AI's labour implications has been, with a few exceptions, strikingly inadequate. Most policy frameworks treat AI governance as a safety and ethics question, not a labour question. The European Union's AI Act, which began entering into force in stages from 2024, establishes important guardrails around high-risk AI systems, but it does not address the core challenge of structural job displacement. Regulating how AI systems are developed and deployed is fundamentally different from managing how societies absorb the economic dislocation those systems create.

Some European policy experts have recognised this gap. The European Policy Centre has called for a “European AI Social Compact,” tied to the European Social Fund, that would align technological progress with labour protections and targeted upskilling across all 27 member states. The German Institute for Employment Research projected that 1.6 million jobs could be reshaped by or lost to AI in Germany alone over the next fifteen years. More broadly, researchers estimate that 50.2 million Europeans, or 32 per cent of the working population, face the risk of displacement in their current roles.

In the United States, policy responses have been fragmented and modest. The Guaranteed Income Pilot Programme Act of 2025, introduced by Representative Bonnie Watson Coleman, authorised 495 million dollars annually for five years to establish nationwide guaranteed income pilots. Representative Rashida Tlaib has proposed the BOOST Act, a 250-dollar monthly refundable tax credit. OpenAI chief executive Sam Altman has promoted the concept of an “American Equity Fund,” where large AI companies and landholders would contribute approximately 2.5 per cent of their value annually to a fund distributed to all citizens. California's Assembly Bill 661 mandates a feasibility study for a permanent statewide guaranteed income programme, explicitly acknowledging the threat automation poses to the state's tech-centric economy.

These are interesting experiments. They are not remotely proportional to the scale of the problem. A 250-dollar monthly payment does not replace a 60,000-dollar salary. A pilot programme does not constitute a safety net. And a feasibility study is, by definition, an acknowledgement that the actual policy does not yet exist.

The fundamental policy challenge is temporal. AI capabilities are advancing on a timeline measured in months. Labour markets adjust over years. Regulatory frameworks evolve over decades. The gap between the speed of technological disruption and the speed of institutional response is not closing. It is widening. And every month it widens further, the eventual adjustment becomes more painful.

What the Crisis Actually Looks Like

If the familiar automation narrative is wrong, what does the actual labour crisis look like?

It does not look like mass unemployment in the traditional sense. It looks like the gradual erosion of the occupational middle. The senior partners at law firms and accounting practices will be fine. The AI systems handling routine work will be fine. Everyone in between faces compression: fewer positions, lower wages, reduced autonomy, and a diminishing path from junior roles to senior ones.

It looks like a generation of graduates locked out of professions they trained for, holding degrees that assumed a labour market structure which no longer exists. It looks like the countries that built their development models around information services discovering that the rungs they climbed have been pulled up behind them.

It looks like corporate profits rising while median wages stagnate or decline, a pattern Acemoglu has documented across four decades of automation and one that agentic AI appears poised to accelerate. It looks like the political consequences of that divergence: the anger, the populism, the search for scapegoats that has already reshaped electoral politics across the Western world.

It looks, in other words, nothing like the orderly transition that government white papers and corporate social responsibility reports have been promising. And it looks precisely like the kind of crisis that societies tend not to prepare for, because preparing would require admitting that the current trajectory is unsustainable.

McKinsey's own research underscores the scale. Their study “Agents, Robots, and Us” found that currently demonstrated technologies could automate activities accounting for approximately 57 per cent of US work hours. Agents, defined as software systems that automate non-physical work, could perform tasks occupying 44 per cent of US work hours. Roles with the highest potential for automation make up approximately 40 per cent of total jobs, concentrated in legal and administrative services and in physically demanding roles such as drivers and machine operators.

The question is not whether work will change. It is whether the institutions meant to manage that change are capable of doing so at the required speed. The evidence, so far, suggests they are not.

Beyond the Productivity Promise

There is a deeper problem buried in the optimistic projections about AI and productivity, and it concerns what happens when the gains accrue almost entirely to capital rather than labour. Every previous technological revolution, from the steam engine to the personal computer, eventually produced broadly shared prosperity, but only after prolonged periods of dislocation, political conflict, and institutional reform. The factory system generated enormous wealth in 19th-century Britain, but it took decades of labour organising, legislative reform, and social upheaval before that wealth was distributed in a way that produced a functioning middle class.

The implicit assumption in today's AI discourse is that the same pattern will repeat, that displacement will be followed by adjustment, that new jobs will emerge, that the economy will find a new equilibrium. Perhaps. But the timescales that previous transitions required are worth noting. The Industrial Revolution's social adjustments took the better part of a century. The post-war economic settlement that created the modern welfare state was forged through two world wars and a global depression. The information technology revolution of the late 20th century produced three decades of wage stagnation for median workers before any broad-based gains materialised, and even those gains were unevenly distributed.

The agentic AI transition is occurring in a world with weaker unions, more fragile social safety nets, and governments already stretched thin by pandemic debt, climate adaptation costs, and geopolitical instability. The institutional capacity to manage a labour market shock of this magnitude is lower than at any point since the early Industrial Revolution.

A PricewaterhouseCoopers report found that 55 per cent of chief executives saw no measurable benefits from AI deployment, while a separate MIT study showed 95 per cent of enterprise uses of generative AI had no measurable impact on profit and loss. These findings might seem reassuring, but they cut both ways. If AI is not yet delivering transformative productivity gains for most organisations, then the job displacement that is already occurring is happening without the compensating economic growth that is supposed to fund new employment. The costs are arriving before the benefits. Workers are being replaced not because AI is producing extraordinary value but because it is producing adequate value at lower cost. That is a different economic story, and it has a different ending.

What Preparation Would Actually Require

Genuine preparation for the agentic AI transition would require a level of policy ambition that no major government has yet demonstrated. It would mean fundamentally rethinking education systems to emphasise adaptability and judgement over domain-specific knowledge that can be automated. It would mean building social insurance systems capable of supporting workers through multiple career transitions, not just one. It would mean confronting the distribution question directly: if AI-driven productivity gains flow primarily to the owners of AI systems, then some mechanism for redistribution is not a progressive wish but a structural necessity.

It would also mean being honest about what reskilling can and cannot do. Retraining a 45-year-old financial analyst to become an AI systems architect is not a realistic proposition for most people. The skills gap is not a gap that can be closed with a six-month certificate programme. And the jobs that are being created by the AI economy, the prompt engineers and machine learning specialists and AI safety researchers, represent a tiny fraction of the employment that is being automated away.

The Carnegie Endowment for International Peace argued in February 2026 that Europe's response to AI labour disruption must extend beyond regulating these systems into fiscal planning and institutional redesign. A credible response, their analysts wrote, rests on three pillars: establishing social protections for displaced workers, providing training infrastructure scaled for continuous transition, and building public trust. That framework is correct. Whether any government will implement it before the crisis arrives is another question entirely.

The pattern of the past decade suggests that policy will follow disaster rather than prevent it. We will get serious about the labour implications of agentic AI roughly 18 months after they become undeniable, which is to say, roughly 18 months too late.

The conversation about AI and jobs has been conducted in the language of the last disruption: automation of routine physical tasks, reskilling programmes, net job creation. That conversation is obsolete. What is coming is the automation of cognitive workflows at scale, the compression of occupational hierarchies, the elimination of the entry-level pipeline, and the concentration of economic gains in an ever-narrower segment of the labour market. It is not the crisis we were told to prepare for. And we are, by almost every meaningful measure, sleepwalking into it.

---

References and Sources

1. Fortune, “Microsoft AI chief gives it 18 months for all white-collar work to be automated by AI,” February 2026.
2. Fortune, “Salesforce CEO Marc Benioff says his company has cut 4,000 customer service jobs as AI steps in: 'I need less heads,'” September 2025.
3. arXiv, “Agentic AI and Occupational Displacement: A Multi-Regional Task Exposure Analysis of Emerging Labor Market Disruption,” April 2026.
4. McKinsey and QuantumBlack, “The State of AI in 2025: Agents, Innovation, and Transformation,” 2025.
5. McKinsey, “Insights on Today's Labor Market: Uncertainty, Agentic AI, and More,” 2025.
6. World Economic Forum, “Future of Jobs Report 2025,” January 2025.
7. Fast Company, “Reskilling Won't Save Us from AI. Here's What We Need to Do Instead,” 2025.
8. Daron Acemoglu and Pascual Restrepo, “Automation and New Tasks: How Technology Displaces and Reinstates Labor,” Journal of Economic Perspectives, Vol. 33, No. 2, 2019.
9. Daron Acemoglu, “The Simple Macroeconomics of AI,” MIT Economics Working Paper, 2024.
10. IMF Finance and Development, “Rebalancing AI,” Daron Acemoglu and Simon Johnson, December 2023.
11. Stanford University, study on AI impact on entry-level hiring, 2025. Reported via IEEE Spectrum, “AI Shifts Expectations for Entry Level Jobs,” 2025.
12. IntuitionLabs, “AI's Impact on Graduate Jobs: A 2025 Data Analysis,” 2025.
13. CNBC, “AI is not just ending entry-level jobs. It's the end of the career ladder as we know it,” September 2025.
14. Thomson Reuters, report on AI adoption in legal and accounting services, 2025.
15. Challenger, Gray and Christmas, AI-related job cut data, 2025.
16. CNBC, “Klarna CEO says AI helped company shrink workforce by 40%,” May 2025.
17. MLQ AI, “Klarna CEO admits AI job cuts went too far,” 2026.
18. BusinessToday, “IT, BPO services will disappear in the next 5 yrs: Vinod Khosla,” February 2026.
19. International Labour Organisation, generative AI exposure estimates, 2025.
20. Ghost Research, “AI Impact on BPO: Transforming India's Workforce,” 2025.
21. European Policy Centre, “AI's Impact on Europe's Job Market: A Call for a Social Compact,” 2025.
22. Carnegie Endowment for International Peace, “How Europe Can Survive the AI Labor Transition,” February 2026.
23. German Institute for Employment Research, AI job displacement projections for Germany, 2025.
24. Congress.gov, Guaranteed Income Pilot Program Act of 2025 (H.R. 5830), introduced by Rep. Bonnie Watson Coleman, 2025.
25. PricewaterhouseCoopers, report on CEO AI deployment outcomes, 2025.
26. Deloitte, “The Agentic Reality Check: Preparing for a Silicon-Based Workforce,” Tech Trends 2026.
27. PwC, “No More Pyramids: Rethinking Your Workforce for the Agentic AI Era,” 2025.
28. HFS Research, “Prepare for Agentic AI to Shatter Corporate Workforces and Global Economies,” 2025.
29. Equal Times, “Agentic AI and the Future of Work: Navigating Technological Promise and the Risk of Increased Automation,” 2025.
30. AI2 Incubator, “Insights 15: The State of AI Agents in 2025: Balancing Optimism with Reality,” 2025.
31. The Interview Guys, “89% of 2026 Grads Think AI Will Take Their Job Before They Even Get One,” 2026.

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

At 2:49 AM Eastern Time on 20 October 2025, a DNS race condition inside Amazon Web Services' US-EAST-1 region triggered a cascade that would, over the next fifteen hours, ripple across seventy-five AWS services and knock more than 3,500 companies offline in over sixty countries. Snapchat vanished. Fortnite went dark. Banking applications froze mid-transaction. And in bedrooms across the United States, owners of Eight Sleep's Pod 5 Ultra smart beds discovered that their $5,049 mattresses had locked themselves into upright positions or cranked their heating coils to uncomfortable temperatures, with absolutely no way to override the settings. The app that controlled their beds needed a server farm in Northern Virginia to function. Without it, people were quite literally trapped in furniture that had forgotten how to be furniture.

It was absurd. It was also a warning.

We have built a civilisation that routes an extraordinary share of its daily operations through a vanishingly small number of cloud data centres. Your payment terminal, your city's traffic management system, your hospital's patient records, your child's baby monitor, and yes, your bed, all phone home to the same handful of server clusters operated by Amazon, Microsoft, and Google. When those clusters stumble, the consequences are no longer limited to a slow-loading webpage. They cascade through supply chains, public services, financial markets, and the physical objects in your home. The question is no longer whether a single cloud failure could trigger a societal crisis. The question is how close we have already come.

The Anatomy of a Cascade

To understand the fragility, you first need to understand the architecture. AWS, Microsoft Azure, and Google Cloud collectively control more than 62 per cent of the global cloud market. In Europe and the United Kingdom, AWS and Microsoft alone command roughly 70 per cent. An estimated 94 per cent of enterprise services worldwide depend on at least one of these three providers. This is not a distributed system in any meaningful sense. It is a funnel, and nearly everything flows through it.

The October 2025 AWS outage demonstrated this with uncomfortable clarity. The failure started in DynamoDB's internal management system, where a subtle DNS race condition caused resolution failures. Within hours, the damage had spread to EC2 compute instances, Lambda serverless functions, S3 storage, and RDS databases. But the truly unsettling revelation was structural: US-EAST-1 serves as the control plane for AWS infrastructure globally. Organisations that had carefully architected their applications to run in European or Asian regions discovered that their supposedly distributed infrastructure still depended on a control plane sitting in Northern Virginia. A single region, a single DNS glitch, and the scaffolding beneath a significant portion of the global internet buckled.

The financial toll of such events is staggering and accelerating. The average cost of downtime across industries rose to $8,600 per minute in 2025, up from $5,600 in 2022. Large enterprises averaged $23,750 per minute of disruption. Across the Global 2000, IT outages collectively drain an estimated four hundred billion dollars annually. The October AWS outage alone generated more than four million outage reports within its first two hours, a measure not just of technical impact but of how many distinct services, businesses, and individuals had routed their operations through a single provider's infrastructure.

But the monetary figures, enormous as they are, obscure the deeper problem. Money can be recovered. Trust in infrastructure is harder to rebuild, particularly when people discover that the systems governing their physical safety have a dependency chain that terminates at a single server rack in a single building in a single state.

When Your Bed Becomes a Liability

The Eight Sleep incident during the October 2025 outage became something of an internet parable, the kind of story that makes you laugh until you think about it for more than thirty seconds. Eight Sleep's Pod 5 Ultra uses water-cooled coils managed through a cloud-connected application. The system tracks biometric data, adjusts temperatures throughout the night, and positions the adjustable base to reduce snoring. It is, by any measure, an impressive piece of engineering. It is also, as October 2025 revealed, entirely dependent on servers it does not own and cannot control.

When AWS went down, users lost access to the temperature controls entirely. Reports flooded social media: beds locked at dangerously high temperatures, adjustable bases stuck in elevated positions, alarms silenced. One user described the experience as sleeping in a sauna. Matteo Franceschetti, the company's chief executive, apologised publicly and promised to “outage-proof” the technology. Within twenty-four hours, Eight Sleep shipped an emergency “outage mode” using Bluetooth connectivity, allowing basic local control without an internet connection.

The speed of the fix only deepened the question: why had local control not been the default from the start? The answer, of course, is economic. Cloud connectivity enables continuous data collection, biometric tracking, firmware updates, and subscription revenue models. Local processing is less profitable. It does not generate the steady stream of user data that feeds product development and investor presentations. The result is a consumer landscape where the most intimate objects in your home require permission from a distant server to perform their basic functions. Your bed needs the internet to be a bed. Your lights need the cloud to switch on.

Eight Sleep was not an isolated case. The Sengled smart lighting outage from 18 to 22 June 2025 left thousands of households without control of their bulbs for four solid days. Not dimming. Not colour adjustment. Simply on or off, and even that was unreliable. German networking company Devolo announced the complete shutdown of its Home Control smart home platform's cloud servers on 31 December 2025, rendering its entire product ecosystem effectively non-functional from that date forward. Users could no longer change configurations, add devices, or access the system through any interface. Integration with Amazon Alexa and Google Assistant ceased entirely. Devices that people had purchased, installed, and relied upon became inert plastic overnight, not because they were broken but because a company decided to stop running a server.

These are not edge cases. They are the predictable outcome of an industry that has systematically traded user autonomy for recurring revenue. Gartner projects that by the end of 2026, 34 per cent of device operations will be processed locally rather than in the cloud, up from 15 per cent the year prior. The Matter protocol and local AI hubs represent early attempts to build smart home infrastructure that does not require a round trip to Oregon to turn on a kitchen light. That trajectory acknowledges the problem, but hardly solves it. Two-thirds of your smart home still needs to call home. And for the devices that fail when the cloud fails, the consequence is not just inconvenience. It is a fundamental breach of the implicit contract between consumer and product: the thing you bought should do the thing it was sold to do.

The Day the Internet Lost Its Middle

If the AWS outage of October 2025 demonstrated the risks of cloud concentration at the infrastructure layer, the Cloudflare outage of 18 November 2025 exposed a different but equally troubling dependency. Cloudflare handles roughly 20 per cent of global web traffic, serving as the content delivery network and security layer for millions of websites and applications. At 11:20 UTC, a change to the permissions of one of Cloudflare's database systems caused the database to output multiple entries into a configuration file used by its Bot Management system. The file doubled in size and was propagated across the entire network.

The result was immediate and vast. X (formerly Twitter), ChatGPT, OpenAI's suite of tools, Spotify, Discord, Zoom, Canva, Uber, and League of Legends all went down or experienced severe degradation. Public transit systems, e-commerce platforms, and banking interfaces were disrupted. The outage lasted approximately five to six hours before full recovery at 17:06 UTC. Crucially, this was not a cyberattack. It was a configuration change, the kind of routine database maintenance that happens thousands of times per day across the industry. A single permissions error in a single database cascaded into the disruption of services used by hundreds of millions of people.

What made the Cloudflare outage particularly revealing was the nature of the services it knocked offline. ChatGPT and Claude AI both experienced disruptions, meaning the AI assistants that an increasing number of professionals, students, and businesses rely upon for daily work simply stopped responding. The episode illustrated a dependency chain that most users had never considered: your AI assistant depends on a cloud provider, which depends on a content delivery network, which depends on a database permission being set correctly. Remove any link in that chain and the entire service collapses. The layers of abstraction that make modern technology convenient also make it opaque, and opacity breeds fragility.

Payments in Freefall

If a malfunctioning bed is an inconvenience and a broken AI chatbot is a productivity hit, a frozen payment system is something closer to a crisis. During the CrowdStrike incident of 19 July 2024, a faulty content update to the company's Falcon Sensor software caused 8.5 million Windows computers to crash simultaneously. The damage reached across airlines, hospitals, banks, and payment processors. Visa and Mastercard transaction systems experienced disruptions in some regions. The financial losses exceeded ten billion dollars globally, with Fortune 500 companies alone absorbing more than five billion dollars in direct costs. Insurers estimated payouts of approximately 1.5 billion dollars.

The CrowdStrike failure was not a cloud outage in the traditional sense. It was a security software update that exposed a different kind of single point of failure: the monoculture of endpoint protection. CrowdStrike held roughly 18 per cent of the global market, which meant that a bug in one company's update pipeline could, and did, bring commercial aviation and hospital systems to a halt on the same morning. The Library of Congress documented the impacts to public safety systems, noting that the outage affected emergency services infrastructure across multiple jurisdictions.

Payment infrastructure sits at the intersection of all these vulnerabilities. According to a 2024 survey, 76 per cent of global respondents run applications on AWS, and the service powers more than 90 per cent of Fortune 100 companies. When AWS experienced its October 2025 outage, UK banks were among those knocked offline, joining a long list of financial institutions that discovered their resilience planning had not accounted for the failure of a service they had treated as permanently available. The dependency is not hypothetical. It is operational, structural, and deeply embedded in the architecture of modern commerce.

Nordic countries and Estonia have begun exploring offline card-payment backup systems, a recognition that payment resilience must be designed deliberately rather than assumed. The Ponemon Institute's 2024 Cost of Data Center Outages report found that the average cost of a data centre outage is approximately $9,000 per minute, with financial services among the most severely affected sectors. The EU's Digital Operational Resilience Act (DORA), which came into effect in January 2025, now requires banks, insurers, and investment firms to prove that their digital operations are resilient, auditable, and accessible to regulators. DORA specifically addresses third-party ICT risk, requiring financial entities to identify and manage dependencies on critical technology providers. It is a start, but regulation follows disaster more often than it prevents it.

Hospitals Without Memory

The healthcare sector presents perhaps the most concerning frontier of cloud and AI dependency. In 2024, 71 per cent of non-federal acute care hospitals in the United States reported using predictive AI integrated into their electronic health records, up from approximately 66 per cent in 2023. Among hospitals affiliated with multi-hospital systems, adoption reached 86 per cent. These systems handle billing, scheduling, risk stratification, and increasingly, clinical decision support. They are not optional add-ons. They have become load-bearing elements of hospital operations.

The CrowdStrike outage of July 2024 provided a stark preview of what happens when those systems fail. A study published through the National Center for Biotechnology Information documented that 759 US hospitals experienced network disruptions during the incident. Of the nearly 1,100 internet-based services examined across those hospitals, 239 (21.8 per cent) were characterised as corresponding with direct patient care functionality. These were not administrative inconveniences. They were disruptions to systems that clinicians relied upon to access patient records, manage medications, and coordinate care.

The paradox of healthcare AI adoption is that it simultaneously improves operational efficiency and increases systemic vulnerability. Hospitals with mature predictive AI deployments have realised measurable improvements in billing accuracy, scheduling efficiency, and outpatient risk stratification. But those gains come with a dependency: if the cloud infrastructure supporting those AI systems fails, the hospital does not revert to a slightly less efficient version of itself. It reverts to paper processes that many current staff have never been trained to use. The institutional memory of how to operate without digital infrastructure is eroding precisely as the digital infrastructure becomes less reliable.

Rural and independent hospitals face a different version of this problem. With only 37 per cent adoption of predictive AI compared to 86 per cent at system-affiliated facilities, they are less exposed to AI-specific failures but also less equipped to absorb any technology-driven disruption. The digital divide in healthcare creates a fragmented landscape where a single infrastructure failure affects institutions unevenly, complicating coordinated emergency responses.

The Grid That Forgot How to Balance

On 28 April 2025, at 12:33 Central European Summer Time, the power systems of continental Spain and Portugal experienced a total blackout. In just five seconds, Spain lost approximately fifteen gigawatts of capacity, equivalent to 60 per cent of its national electricity demand. The remaining generation was insufficient to meet load, and the grid entered a cascading failure that left 31 gigawatts of demand disconnected. Power was interrupted for about ten hours across most of the Iberian Peninsula, and considerably longer in some areas.

The European Network of Transmission System Operators for Electricity (ENTSO-E) published its final expert panel report in March 2026, identifying a combination of interacting factors: oscillations, gaps in voltage and reactive power control, rapid output reductions, generator disconnections, and uneven stabilisation capabilities. The interaction between market design (schedule changes driving fast power ramps), grid code implementation (fixed power factor mode for renewables), protection coordination (settings diverging from requirements), and system architecture (insufficient reactive power reserves and manual switching of critical assets) created cascading failures that unfolded faster than human operators could respond. The Iberian system simply lacked the inertia needed to absorb the initial generation-loss shocks, and automatic protection mechanisms cascaded into a total system collapse.

The Iberian blackout was not caused by artificial intelligence. But it was a product of the same underlying dynamic that makes AI-dependent infrastructure so fragile: systems designed for efficiency and automation that lacked the inertia, redundancy, and human-override capability to absorb unexpected shocks. As more grid management functions migrate to AI-driven optimisation platforms hosted in cloud environments, the attack surface and failure surface both expand. A power-grid optimiser that destabilises supply, whether through a software bug, a corrupted model, or a compromised update, could reproduce the Iberian scenario in a system that has even less manual fallback capability than the one that failed in April 2025.

In August 2025, a corrupted update in a widely used open-source AI optimisation library triggered cascading failures lasting over twelve hours. Healthcare, finance, aviation, and emergency response systems were among those affected. The incident demonstrated that AI infrastructure does not need to be sophisticated to be dangerous. It merely needs to be ubiquitous. And ubiquity, in a world where a handful of open-source libraries and cloud providers underpin the majority of AI deployments, is exactly what we have achieved.

Concentration as Systemic Risk

The United Kingdom offers a particularly instructive case study in cloud concentration risk. The UK's cloud market is dominated by AWS and Microsoft, and the UK government's “One Government Value Agreement” with AWS led to a tenfold increase in spending on the platform, from approximately 100 million pounds to over one billion pounds since the agreement was originally signed in 2020. The National Preparedness Commission has argued that cloud dominance in the UK demands immediate Competition and Markets Authority action, framing concentration not as a market efficiency issue but as a national security concern.

The concern is not merely theoretical. During the October 2025 AWS outage, government and major banking services in the UK reported intermittent issues, demonstrating that when public infrastructure depends on a handful of cloud providers and regions, outages can compromise access to essential services. Microsoft's suspected suspension in May 2025 of the International Criminal Court Chief Prosecutor's email services further illustrated how dependence on a single US-based provider can create political vulnerabilities that extend well beyond technical uptime. When a foreign government's diplomatic communications or a court's prosecutorial operations can be disrupted by a single company's decision, the sovereignty implications are impossible to ignore.

Across Europe, American technology companies control more than 70 per cent of cloud infrastructure. The EU has responded with a series of regulatory instruments that collectively represent the most ambitious attempt yet to address digital dependency. The Data Act, which entered into force in January 2024, contains provisions to prevent vendor lock-in, requiring cloud providers to remove unjustified technical or contractual barriers to switching and mandating the phasing out of switching fees by 2026 to 2027. The Network and Information Security Directive 2 (NIS2), effective from October 2024, addresses cybersecurity and operational resilience for essential services, imposing requirements around risk management, supply chain security, and breach reporting. The European Commission is preparing to propose a Cloud and AI Development Act, expected in the first quarter of 2026, which may set standards for cloud computing services and promote investment in European data centres.

In the United Kingdom, the Cyber Security and Resilience Bill, introduced to the House of Commons on 12 November 2025, updates previous cybersecurity legislation and is expected to become law in 2026. These regulatory efforts acknowledge the problem. Whether they solve it depends entirely on enforcement and on whether governments are willing to accept the short-term costs of diversifying away from established providers. The track record is not encouraging. Convenience and cost savings have consistently won against resilience in procurement decisions, and the structural incentives that created cloud concentration remain firmly in place.

The Illusion of Redundancy

One of the more uncomfortable lessons of recent outages is how thoroughly they have dismantled the promise of multi-region resilience. Cloud providers market regional redundancy as a safeguard: distribute your workloads across multiple regions, and no single failure can bring you down. The October 2025 AWS outage revealed this as, at best, a half-truth. Because US-EAST-1 serves as the global control plane, organisations running workloads in Frankfurt or Singapore still found themselves dependent on infrastructure in Northern Virginia. The redundancy was architectural fiction, a comforting story that collapsed the moment the underlying assumption was tested.

This pattern recurs with dispiriting regularity. Google Cloud's Identity and Access Management outage on 12 June 2025 lasted only one hour and thirteen minutes, but because IAM is the gateway through which every Google Cloud product authenticates, the single failure immediately disabled a wide range of services worldwide. Azure's Front Door outage in October 2025 similarly cascaded: the failure of a global content delivery and application delivery service knocked out Microsoft 365, the Azure Portal, and enterprise customers including Alaska Airlines for more than eight hours. In late July 2025, Azure's East US region experienced allocation failures that Microsoft reported resolved by 5 August, though some users continued to report problems days afterwards.

Azure outages were the longest on average in the 2024-2025 period, lasting a mean of 14.6 hours per incident. Google Cloud disruptions averaged 5.8 hours. AWS incidents averaged 1.5 hours but, as the October 2025 event demonstrated, their impact was amplified by the sheer volume of services depending on a single region. Between August 2024 and August 2025, the three major providers together experienced more than 100 service outages. Global network outages increased by 33 per cent from January to May 2025, rising from 1,382 to 1,843 incidents. Critical cloud outages increased by approximately 18 per cent in 2024. The trend line is not ambiguous. The systems are getting more complex, more interdependent, and more prone to cascading failure.

Smart city infrastructure amplifies these risks further. Traffic management, water systems, power distribution, and emergency services increasingly depend on cloud-hosted platforms for coordination and optimisation. When those platforms are locked into a single provider's ecosystem, the city inherits all of that provider's failure modes. Individual subsystems such as traffic control, power distribution, and water management should, in principle, operate independently while coordinating through shared data layers. In practice, vendor-led platforms have created long-term lock-in, and fragmented governance has slowed the development of cross-domain interoperability. The networked nature of smart city technologies, with a growing number of interconnected sensors, cameras, and control nodes, also expands the attack surface. A failure in one subsystem can propagate to others through shared dependencies that were invisible during normal operations.

Building for the Failure You Cannot Predict

The instinct, after each major outage, is to call for better engineering. More robust DNS configurations. Improved testing protocols. Redundant control planes. These are necessary but insufficient responses. The deeper challenge is structural. We have allowed a civilisation-scale dependency to concentrate in a handful of private companies whose primary obligation is to shareholders, not to the public infrastructure that has come to rely on them.

The remedies being discussed span a wide range of ambition and feasibility. At the consumer level, the push towards local processing and edge computing represents a partial answer. At the enterprise level, genuine multi-cloud strategies (not merely multi-region deployments within a single provider) could distribute risk more effectively, though they carry significant cost and complexity overhead. At the regulatory level, the EU's Digital Operational Resilience Act and the UK's forthcoming Cyber Security and Resilience legislation represent early attempts to mandate resilience rather than merely recommend it. Experts have recommended mandating AI system redundancy for critical infrastructure, funding global audits of open-source AI dependencies, and establishing an international AI incident response coalition.

But the most fundamental shift required is conceptual. We need to stop treating cloud infrastructure as a utility that simply works and start treating it as what it is: a concentrated, commercially operated system with known failure modes and structural single points of failure. Traditional utilities have regulators, redundancy requirements, and public service obligations. Cloud providers, despite hosting infrastructure that is arguably more critical to daily life than the electricity grid was fifty years ago, operate under a fraction of that oversight. The gap between the criticality of these services and the regulatory framework governing them is one of the defining mismatches of our era.

The Iberian blackout of April 2025 unfolded faster than human operators could respond. The AWS outage of October 2025 revealed that global control planes create global vulnerabilities. The CrowdStrike failure of July 2024 showed that software monocultures can cascade across every sector simultaneously. The Cloudflare outage of November 2025 demonstrated that a single configuration error in a content delivery network can simultaneously disable AI assistants, social media platforms, ride-hailing services, and public transit systems. Each incident was different in its specifics but identical in its lesson: systems optimised for efficiency at the expense of redundancy will eventually fail, and when they do, the failure will propagate along every dependency chain that was invisible during normal operations.

The question posed at the outset, whether we have built a civilisation so dependent on AI infrastructure that one failure could cascade into a societal crisis, has an answer that is both reassuring and deeply troubling. We have not yet experienced a full-scale societal collapse from a cloud or AI infrastructure failure. But we have experienced dress rehearsals, and each one has been larger, longer, and more consequential than the last. The Eight Sleep bed that overheated in October 2025 is a punchline. The payment system that froze, the hospital that lost access to patient records, the power grid that collapsed in five seconds: those are not punchlines. They are data points on a curve that bends towards a reckoning we have not yet decided to prevent.

The infrastructure we depend on is only as resilient as its weakest dependency. Right now, that dependency is a DNS record in Northern Virginia.

---

References and Sources

1. ThousandEyes, “AWS Outage Analysis: October 20, 2025,” ThousandEyes Blog, October 2025.
2. AWS, “Post-Event Summaries,” AWS Premium Support, October 2025.
3. UPI, “AWS outage caused smart beds to overheat, get stuck upright,” UPI Odd News, 22 October 2025.
4. TechBuzz AI, “Eight Sleep adds offline mode after AWS outage left smart beds stuck,” October 2025.
5. CyberInsurance News, “Cloud Outages in 2024 Increased by 18%, Google Cloud Downtime Up 57%,” Parametrix Report, 2024.
6. CNN Business, “CrowdStrike outage: We finally know what caused it, and how much it cost,” 24 July 2024.
7. Congress.gov, “CrowdStrike IT Outage: Impacts to Public Safety Systems and Considerations for Congress,” Library of Congress, 2024.
8. ENTSO-E, “Expert Panel Final Report on 28 April 2025 Blackout in Spain and Portugal,” 20 March 2026.
9. Power Magazine, “Anatomy of a Blackout: Findings from the Spain-Portugal Grid Collapse Final Report,” 2026.
10. FinTech Magazine, “AWS Outage: A Major Risk For The Financial Sector?” October 2025.
11. National Preparedness Commission, “The Concentration Crisis: Why Cloud Dominance in the UK Demands Immediate CMA Action,” 2025.
12. The Register, “Europe gets serious about cutting US digital umbilical cord,” 22 December 2025.
13. Morrison Foerster, “Digital Regulation in EU and UK: The Enduring 2025 Themes,” January 2026.
14. House of Commons Library, “Cyber Security and Resilience (Network and Information Systems) Bill 2024-26,” 2025.
15. DemandSage, “54 Internet Outage Statistics (2026): Global Downtime, Impacts,” 2026.
16. The Conversation / Iowa State Research, “Why cloud service outages ripple across the internet and the economy,” March 2026.
17. Serenity Smart Homes NJ, “What the Sengled Outage Taught Us About Cloud Reliance,” 22 June 2025.
18. We Speak IoT, “Devolo Ends Its Home Control Smart Home System: Cloud Services to Shut Down by End of 2025,” 2025.
19. Ancher AI, “Global AI Outage of 2025: Causes and Lessons,” 2025.
20. New Civil Engineer, “Integrating AI into critical national infrastructure presents severe risks, experts warn,” 1 October 2025.
21. Belitsoft, “Outage Affected Multiple Google Cloud Platform Products,” June 2025.
22. Cloudflare Blog, “Cloudflare outage on November 18, 2025,” November 2025.
23. PMC / NCBI, “Patient Care Technology Disruptions Associated With the CrowdStrike Outage,” National Center for Biotechnology Information, 2025.
24. NCBI Bookshelf, “Hospital Trends in the Use, Evaluation, and Governance of Predictive AI, 2023-2024,” ASTP Health IT Data Brief, 2024.
25. McKinsey, “AI-native public infrastructure for smart cities,” McKinsey Technology, 2025.

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

In May 2025, something quietly extraordinary happened during Klarna's quarterly earnings call. Sebastian Siemiatkowski, the fintech company's co-founder and chief executive, appeared on screen to walk investors through the numbers. He looked like Siemiatkowski. He sounded like Siemiatkowski. But within seconds, the figure on screen confessed: it was not Siemiatkowski at all. It was an AI-generated avatar, trained on the CEO's likeness and voice, delivering the company's financial highlights while the real Siemiatkowski was elsewhere. The avatar did not blink quite as often as a human would, and the voice synchronisation was good but not flawless. Still, the message was clear: the era of sending your digital double to do your talking has arrived.

A day later, Zoom's own chief executive, Eric Yuan, did much the same thing, deploying an AI avatar of himself during an earnings presentation. The timing was hardly coincidental. Yuan had been evangelising the concept of “digital twins” since mid-2024, telling audiences at Fortune that people would eventually send their AI-powered replicas to future meetings so they could “go to the beach” instead. By TechCrunch Disrupt 2025, he was making bolder predictions: AI would enable three-to-four-day working weeks by 2030, partly because digital replicas could handle routine meetings while the flesh-and-blood human focused on higher-value work. In March 2026, Zoom formally rolled out photorealistic AI avatars as a product feature, promising lifelike figures that mirror a user's expressions, lip movements, and eye movements so that people can “be present” even when they are not camera-ready, or not present at all.

This is not science fiction any longer. It is a shipping product. And it forces a question that the technology industry, corporate boardrooms, and philosophers of mind alike are only beginning to grapple with seriously: when an AI avatar attends a meeting on your behalf, are the other participants being deceived? And does it matter?

The Spectrum of Standing In

To understand why this question is more complicated than it first appears, it helps to recognise that meetings have always involved varying degrees of presence, attention, and substitution.

Consider the humble out-of-office auto-reply, a digital stand-in that has existed for decades. No one considers it deceptive when a colleague's email bot informs you they are unavailable. Move up the spectrum and you find shared calendars where assistants accept invitations on an executive's behalf, or junior colleagues who “represent” a department without the senior leader's direct involvement. The video call itself, which became the default mode of professional interaction during the pandemic years, already introduced a layer of mediation between participants. Filters smooth skin. Virtual backgrounds conceal messy kitchens. Gallery views flatten hierarchies into a grid of equally sized rectangles. None of this is typically described as deception, yet each element subtly manipulates the impression one participant forms of another.

AI avatars occupy a new and considerably more potent position on this spectrum. When Zoom's Steve Rafferty, the company's head of APAC and EMEA, used his AI avatar to introduce a quarterly meeting in fluent French, he was not simply delegating a task; he was projecting a version of himself that could do something he could not. Rafferty's team spans from the Arctic Circle to Antarctica, covering roughly sixty different languages, and the avatar allowed him to deliver a personal, multilingual message at scale. The tool cannot yet interact with other participants or answer questions in real time, but the direction of travel is unmistakable.

The crucial distinction is between transparent substitution and covert impersonation. If everyone in the meeting knows they are watching an AI avatar, the dynamic is fundamentally different from a scenario where participants believe they are speaking to a living, breathing human being who happens to be on camera. The first is a communication tool. The second is, by most reasonable definitions, a form of deception. But between these two poles lies an enormous grey zone: the avatar that is technically disclosed but functionally indistinguishable from the real person; the avatar whose presence is noted in a meeting invitation that nobody reads; the avatar that begins as a disclosed introduction but seamlessly transitions into a conversation that feels, to other participants, like a human exchange. The spectrum of standing in, it turns out, is not a spectrum at all. It is a fog.

What Philosophers Make of Digital Doubles

The philosophical landscape here is richer than the technology industry tends to acknowledge. Luciano Floridi, the founding director of Yale University's Digital Ethics Center and a professor at the University of Bologna, has spent years developing an ethical framework for artificial intelligence built around five principles: beneficence, nonmaleficence, autonomy, justice, and explicability. Floridi's work on deepfakes is particularly relevant. He argues that AI-generated synthetic media has the capacity to undermine our confidence in the original, genuine, authentic nature of what we see and hear. The threat is not merely that a specific piece of content might mislead; it is that the very existence of convincing synthetic media corrodes the epistemic foundations on which trust depends.

Apply this framework to the meeting avatar scenario and the implications are sobering. A meeting is not just an exchange of information; it is a social contract. Participants implicitly agree to be present, to listen, to respond in good faith. When one party secretly outsources their participation to a machine, they violate not just the expectation of presence but the norms of reciprocity that make collaborative work possible. The person who sent the avatar may receive a neat summary afterwards, but their counterparts invested real cognitive and emotional effort into an interaction they believed was mutual. That imbalance is not a minor technical detail. It is a breach of the implicit bargain that makes professional relationships function.

From a Kantian perspective, the issue is equally stark. Immanuel Kant's categorical imperative holds that one should act only according to principles that could be universalised without contradiction. If everyone sent avatars to every meeting, the meeting itself would cease to function as a space for genuine human deliberation. The universalisation test fails spectacularly: a world in which all meeting participants are AI avatars is a world in which meetings are simply algorithms talking to algorithms, with no humans in the loop at all. The very concept of a “meeting” presupposes the meeting of minds, not the collision of language models.

Yet utilitarians might see the matter differently. If an AI avatar can represent its principal accurately, freeing that person to do more meaningful work or simply to rest, the aggregate benefit might outweigh the discomfort of reduced authenticity. PwC's 2025 Global Workforce Hopes and Fears Survey, which interviewed nearly 50,000 workers across 48 economies and 28 sectors, found that daily users of generative AI reported being more productive (92 per cent, compared to 58 per cent of infrequent users), with higher perceived job security and pay. If avatars extend these productivity gains by reclaiming hours lost to routine meetings, the utilitarian calculus could tip in their favour. The question then becomes empirical: does the avatar actually represent the person faithfully, or does it introduce distortions, biases, and errors that compound over time?

The Markkula Center for Applied Ethics at Santa Clara University published a case study examining precisely these tensions. The centre frames the discussion through multiple ethical lenses, including rights, justice, utilitarianism, the common good, virtue, and care ethics, and invites readers to consider what obligations a person has to disclose their use of an avatar. The case study does not offer a tidy resolution. Instead, it highlights that the ethics of meeting avatars depend heavily on context: who is in the meeting, what is at stake, whether disclosure has occurred, and what alternatives exist.

Consent, Disclosure, and the Trust Deficit

If the philosophical arguments suggest that undisclosed avatar use is ethically problematic, the practical question becomes: what kind of disclosure is sufficient?

Zoom's own approach offers one model. When the company's AI Companion joins a third-party meeting to transcribe and summarise, it automatically posts a message in the meeting chat identifying itself as a bot and indicating that it is transcribing. Its video tile displays the word “Transcribing” alongside the Zoom AI Companion logo. This is transparency by design, built into the product architecture so that disclosure is not left to the discretion of individual users.

But the new photorealistic avatar feature complicates this model considerably. If the avatar looks and sounds convincingly like a real person, a small chat notification may not be enough to prevent participants from believing they are interacting with a human. The gap between what the technology can simulate and what a text disclaimer can effectively communicate grows wider with each improvement in rendering fidelity, voice synthesis, and facial animation. There is an old principle in design: if you have to explain it, you have already failed. When a photorealistic avatar requires a text disclaimer to prevent deception, the product itself is designed in a way that defaults to misleading.

Zoom appears to recognise this tension. Alongside its avatar rollout in March 2026, the company introduced deepfake-detection technology for meetings, providing real-time alerts when synthetic audio or video is detected. This is a notable acknowledgement that the very product Zoom is selling, convincing digital replicas of real people, simultaneously creates a security and trust risk that requires countermeasures. It is as though a locksmith, having sold you the world's most sophisticated lock-picking kit, also offers to install a better deadbolt.

The broader data on consumer attitudes reinforces the concern. Research consistently shows that the vast majority of people value authentic content and view undisclosed AI usage as a breach of trust. More than half of consumers surveyed demand explicit disclosure when AI-generated video, images, or avatars are used, and younger demographics, particularly Generation Z, tend to view AI-generated content as inauthentic and unethical when it is not clearly labelled.

This creates a paradox for companies eager to deploy the technology. The more convincing the avatar, the more useful it is as a communication tool, but the more convincing it is, the greater the expectation of disclosure, and the more disclosure undermines the illusion of natural presence that makes the avatar appealing in the first place. Call it the uncanny valley of trust: as the technology improves, it enters a zone where it is good enough to deceive but not good enough to make deception acceptable.

The Legal Landscape Takes Shape

Regulators have not been idle. The legal framework surrounding AI-generated likenesses, synthetic media, and digital avatars has expanded rapidly across multiple jurisdictions, creating a patchwork of obligations that any organisation deploying meeting avatars must navigate.

In the European Union, Article 50 of the AI Act establishes transparency obligations for providers and deployers of AI systems that generate or manipulate content constituting a deepfake. The rules require that such content be clearly disclosed as artificially generated or manipulated. These transparency provisions are set to take full effect in August 2026, with a Code of Practice expected to be finalised in mid-2026 to establish practical standards. The scope is broad: the EU's framework covers AI-generated text, audio, video, images, avatars, and digital twins. For any multinational corporation considering the deployment of meeting avatars across European operations, the compliance obligations are substantial and the penalties for failure significant.

In the United States, the regulatory picture is more fragmented but no less active. As of early 2026, forty-six states have enacted legislation targeting AI-generated media in some form. In 2025 alone, 146 bills were introduced to state legislatures that included language specific to AI deepfakes. The federal TAKE IT DOWN Act, passed in 2025, represents America's first national law directly regulating deepfake abuse, though its primary focus is nonconsensual intimate content rather than business communications. At the state level, Tennessee's ELVIS Act (Ensuring Likeness, Voice, and Image Security) prohibits the unauthorised commercial use of a person's voice, including AI-generated replications. California's AB 2602, effective from January 2025, renders unenforceable any contract provision that allows for the creation of a digital replica of an individual's likeness in place of work the individual would have otherwise performed in person, unless the contract includes a reasonably specific description of intended uses and the individual had professional legal representation.

Morrison Foerster, the global law firm, published an extensive analysis in September 2025 noting that digital avatars sit at the nexus of several evolving legal regimes, including intellectual property rights, publicity rights, and consumer protection. The firm's assessment is unambiguous: companies deploying digital avatars must navigate a complex and rapidly shifting regulatory environment, and the cost of noncompliance is rising.

The Federal Trade Commission has also signalled its intent to act. Fines for “deceptive synthetic endorsements” now reach fifty thousand dollars per violation, a figure that concentrates the mind of any marketing or communications department considering avatar deployment without adequate disclosure. What remains unclear is whether a meeting avatar that participates in a business discussion without disclosure constitutes a “deceptive” practice under existing consumer protection law, or whether new legislative categories will be needed to address this specific use case.

Corporate Adoption and the Productivity Seduction

Despite the ethical and legal headwinds, the commercial momentum behind AI avatars is formidable. The productivity case is compelling on its face. If a digital twin can attend a routine status update, freeing its human counterpart to focus on strategic thinking, creative work, or simply recovering from meeting fatigue, the efficiency gains could be substantial. Microsoft has moved aggressively in this direction: at Ignite 2025, the company revealed that its Copilot agents had evolved from “helping with work to handling it on your behalf,” with autonomous capabilities governed through permission scopes, approval workflows, and execution logging. The Facilitator agent in Microsoft Teams can drive agendas, take notes, keep meetings on track, and manage actions, edging closer to a future where human attendance becomes optional.

Otter.ai, which reached one hundred million dollars in annual recurring revenue in 2025, exemplifies the trajectory from the startup side. The company has evolved from a passive transcription tool into an active meeting agent that can attend, summarise, and act on discussions. Its enterprise suite includes AI agents for sales teams, autonomous product demonstrations, and a comprehensive search capability spanning an organisation's entire meeting archive. Otter claims that for the average enterprise customer, the platform saves the equivalent workload of one full-time employee for every twenty users, translating to a ten-to-one return on investment. For a one-thousand-user organisation, that translates to fifty full-time equivalents' worth of work saved, or more than six million dollars in annual cost savings.

Dan Thomson, the founder and chief executive of Sensay, a startup that creates AI replicas of employees, has gone further still. Thomson, who holds a BA in Philosophy from King's College London and an MBA from the University of Cambridge, uses his own digital twin to draft replies to emails and messages, estimating that it saves him hours each day. Sensay's digital replicas are trained on employees' own materials and communications, and Thomson has cited examples where deploying a digital persona on a company website increased online conversions by three hundred per cent and reduced support costs by fifty to seventy per cent.

The appeal is obvious. But the question of whether an AI avatar can truly “represent” someone in a meeting raises deeper issues about what representation means. A human delegate sent to a meeting can exercise judgement, read the room, improvise, push back, and make commitments. Today's AI avatars can, at best, deliver prepared remarks, summarise known information, and answer simple questions drawing on a corpus of the principal's past communications. They cannot negotiate in real time, pick up on subtle social cues, or take responsibility for the consequences of what they say. They cannot feel embarrassment when they get something wrong, and they cannot feel the weight of a promise they have made.

This gap between capability and expectation is where the greatest risk of deception lies. If participants believe they are engaging with a person who can make decisions and commitments, but are in fact speaking to a language model with a convincing face, the resulting misunderstandings could have real consequences for contracts, relationships, and organisational trust.

Cultural Fault Lines

Attitudes toward AI avatars are not uniform across cultures, and the global rollout of these technologies will inevitably encounter varying norms around presence, formality, and authenticity.

Japan offers a particularly instructive case. The country has a distinctive openness to AI-based technologies, including robots and avatars, rooted in cultural attitudes that have long embraced the idea of machines coexisting with humans. The Japanese government's Moonshot Goal 1 programme aims to realise a society where humans can be free from limitations of body, brain, space, and time by 2050, explicitly including “cybernetic avatars” as part of that vision. The adoption rate of generative AI among Japanese users rose from 33.5 per cent in February 2024 to 42.5 per cent in February 2025, reflecting a methodical but steady embrace of the technology. Japan's approach to AI governance, as highlighted by the World Economic Forum in January 2026, prioritises how institutions adapt and govern AI rather than what specific technologies they adopt, a philosophical distinction that could shape how meeting avatars are regulated in the region.

Yet even in Japan, the business culture's preference for careful evaluation before widespread implementation suggests that avatar adoption in high-stakes meetings will proceed cautiously. Companies like Hakuhodo, through its Human-Centred AI Institute, emphasise using AI as a “co-pilot” to enhance creativity rather than replace human presence, a framing that implicitly acknowledges the importance of the human element in professional interactions.

In cultures where personal relationships and face-to-face trust-building are paramount, such as many Middle Eastern and Latin American business environments, the introduction of AI avatars into meetings could be perceived as fundamentally disrespectful, a signal that the absent party does not value the relationship enough to show up in person. Conversely, in cultures that prize efficiency and directness, an avatar that delivers a crisp, well-prepared message might be received more warmly than a distracted, multitasking human on a video call.

The cultural dimension matters because it reveals that the question of deception is not purely philosophical or legal; it is also deeply social. What counts as deceptive depends on shared expectations, and those expectations vary enormously across contexts. A practice considered efficient and pragmatic in one business culture may be experienced as insulting or dishonest in another. Any regulatory framework that ignores this variation risks being either toothless or oppressive, depending on where it is applied.

The Asymmetry Problem

Perhaps the most troubling aspect of AI meeting avatars is the asymmetry they introduce into professional relationships. When one party sends an avatar and the other does not know, the avatar-sender gains an informational advantage: they receive a summary of the meeting without having invested the time or cognitive effort to participate, while the other participants have engaged in good faith, believing they were building a relationship with a person.

This asymmetry is not merely inconvenient; it restructures power dynamics in ways that could erode the foundations of professional trust. If colleagues, clients, or business partners come to suspect that they might be talking to an avatar at any given time, the baseline level of trust in all video interactions could decline. Every call becomes potentially suspect. Every participant must wonder: is that really you?

PwC's 2025 survey data is instructive here as well. The research found that only 14 per cent of workers use generative AI daily, but those who do report dramatically different experiences of productivity and security compared to those who do not. This gap creates a two-tier workforce: those who leverage AI tools (potentially including meeting avatars) and those who do not, with the former gaining significant advantages that may be invisible to the latter. When that advantage extends to sending an undisclosed avatar to a meeting, the information asymmetry becomes an ethical asymmetry as well.

The 2025 Edelman Trust Barometer documented growing concerns about AI's impact on societal trust, and the deployment of meeting avatars without robust disclosure norms could accelerate that erosion. Research on workplace trust from 2026 found that teams experiencing breakdowns in recognition and authentic interaction showed significantly higher turnover rates, with an average lead time of eighty-seven days between the first detectable decline in genuine connection and a resignation.

The irony is sharp: a technology designed to free people from the drudgery of unnecessary meetings could end up making all meetings less meaningful by injecting doubt into the fundamental question of whether anyone is really there.

Navigating What Comes Next

So what should organisations, regulators, and individuals do? The answer is unlikely to be a blanket prohibition. AI avatars offer genuine benefits, from multilingual communication to accessibility for people with disabilities or chronic health conditions that make sustained video presence difficult. The technology is here, and it will improve.

What matters is the framework within which it is deployed. Several principles seem essential.

First, disclosure must be mandatory, not optional. Any meeting participant represented by an AI avatar should be required to inform other participants before the meeting begins, not buried in a chat message that might be missed, but through a clear, unavoidable notification. Zoom's deepfake detection feature is a useful backstop, but it should not be the primary mechanism for ensuring transparency. The EU AI Act's transparency obligations, due to take full effect in August 2026, offer a model: providers of AI systems must ensure machine-readable marking and detectability of AI-generated content, placing the burden on the technology companies rather than on individual users to opt into honesty.

Second, organisations need clear policies distinguishing between contexts where avatar use is acceptable and where it is not. A pre-recorded avatar delivering a company-wide update is categorically different from an avatar participating in a negotiation, a performance review, or a client pitch. The stakes, the expectations of presence, and the potential for harm differ dramatically across these scenarios. Internal guidelines should specify which meeting types permit avatar representation and which require genuine human attendance.

Third, the legal frameworks emerging across the EU, the United States, and elsewhere need to address the meeting-avatar use case specifically. Current legislation focuses heavily on deepfakes in political communications and nonconsensual intimate content, which are unquestionably important, but the professional communications context presents its own distinct challenges around consent, representation, and liability. If an avatar makes a commitment during a negotiation, who is legally bound? If an avatar misrepresents a position because it drew on outdated training data, who bears the responsibility? These questions need answers before, not after, the technology becomes ubiquitous.

Fourth, the technology companies building these tools bear a responsibility that extends beyond simply adding disclosure features. They must actively consider the incentive structures their products create. If the default setting makes it easy to send an avatar without disclosure and difficult to opt into transparency, the predictable result is widespread undisclosed use, regardless of what the terms of service say.

Finally, individuals must reckon with what they owe to the people they work with. Sending an avatar to a meeting is not inherently wrong, but doing so without telling anyone is a choice to prioritise convenience over honesty. In a professional culture already strained by remote work, algorithmic management, and the ambient anxiety of automation, that choice carries weight.

The Real Question Behind the Question

The debate over AI meeting avatars is, at its core, a debate about what we believe meetings are for. If meetings are simply information-exchange mechanisms, then avatars are a logical optimisation: a more efficient way to transmit and receive data. But if meetings are also spaces for relationship-building, for reading tone and body language, for the subtle negotiations of trust that underpin every working partnership, then the introduction of a convincing but non-sentient stand-in changes the nature of the interaction in ways that matter.

The discomfort many people feel about AI avatars attending meetings is not irrational technophobia. It is an intuition about something important: that presence is not just about being seen and heard, but about being accountable. A person who is genuinely present in a meeting can be surprised, challenged, moved, and changed by what happens there. An avatar cannot. It can only perform the appearance of those responses.

Whether that performance constitutes deception depends, ultimately, on whether it is disclosed. An avatar that announces itself as an avatar is a tool. An avatar that pretends to be a person is a lie. The line between the two is thin, and the technology industry's track record of respecting thin ethical lines is not, to put it diplomatically, encouraging.

As these tools proliferate through the spring and summer of 2026, the choices made by companies like Zoom and Microsoft, by regulators in Brussels and Washington, and by the millions of professionals deciding whether to click “send my avatar” will shape the norms of professional trust for years to come. The technology is neither good nor evil. But the decision to use it honestly, or not, very much is.

---

References and Sources

1. TechCrunch, “Klarna used an AI avatar of its CEO to deliver earnings, it said,” May 2025. https://techcrunch.com/2025/05/21/klarna-used-an-ai-avatar-of-its-ceo-to-deliver-earnings-it-said/

2. TechCrunch, “After Klarna, Zoom's CEO also uses an AI avatar on quarterly call,” May 2025. https://techcrunch.com/2025/05/22/after-klarna-zooms-ceo-also-uses-an-ai-avatar-on-quarterly-call/

3. TechCrunch, “Zoom CEO Eric Yuan says AI will shorten our workweek,” October 2025. https://techcrunch.com/2025/10/27/zoom-ceo-eric-yuan-says-ai-will-shorten-our-workweek/

4. TechCrunch, “Zoom introduces an AI-powered office suite, says AI avatars for meetings arrive this month,” March 2026. https://techcrunch.com/2026/03/10/zoom-launches-an-ai-powered-office-suite-says-ai-avatars-for-meetings-are-coming-soon/

5. Raconteur, “Tech CEOs are sending their AI avatars to meetings,” 2025. https://www.raconteur.net/technology/ai-avatars-meetings

6. Fortune, “Zoom founder Eric Yuan wants 'digital twins' to attend meetings for you so you can 'go to the beach' instead,” June 2024. https://fortune.com/2024/06/05/zoom-founder-eric-yuan-digital-ai-twins-attend-meetings-for-you/

7. Markkula Center for Applied Ethics, Santa Clara University, “Meeting Avatars: An AI Ethics Case Study.” https://www.scu.edu/ethics/focus-areas/internet-ethics/resources/meeting-avatars-an-ai-ethics-case-study/

8. Zoom Support, “Enabling or disabling AI Companion to join third-party meetings for meeting summaries.” https://support.zoom.com/hc/en/article?id=zm_kb&sysparm_article=KB0080357

9. Zoom Newsroom, “New AI innovations for Zoom Workplace simplify and scale teamwork,” March 2026. https://news.zoom.com/ec26-zoom-workplace/

10. EU Artificial Intelligence Act, “Article 50: Transparency Obligations for Providers and Deployers of Certain AI Systems.” https://artificialintelligenceact.eu/article/50/

11. Herbert Smith Freehills Kramer, “Transparency obligations for AI-generated content under the EU AI Act: From principle to practice,” March 2026. https://www.hsfkramer.com/notes/ip/2026-03/transparency-obligations-for-ai-generated-content-under-the-eu-ai-act-from-principle-to-practice

12. Morrison Foerster, “Digital Avatars Deep Dive Series: Navigating the Legal and Regulatory Landscape in 2025,” September 2025. https://www.mofo.com/resources/insights/250922-digital-avatars-deep-dive-series-navigating

13. ComplianceHub, “Complete Guide to U.S. Deepfake Laws: 2025 State and Federal Compliance Landscape.” https://www.compliancehub.wiki/complete-guide-to-u-s-deepfake-laws-2025-state-and-federal-compliance-landscape/

14. MultiState, “How AI-Generated Content Laws Are Changing Across the Country,” February 2026. https://www.multistate.us/insider/2026/2/12/how-ai-generated-content-laws-are-changing-across-the-country

15. Congress.gov, “S.1396 – Content Origin Protection and Integrity from Edited and Deepfaked Media Act of 2025.” https://www.congress.gov/bill/119th-congress/senate-bill/1396/text

16. Otter.ai, “Otter.ai Caps Transformational 2025 with $100M ARR Milestone,” 2025. https://otter.ai/blog/otter-ai-caps-transformational-2025-with-100m-arr-milestone-industry-first-ai-meeting-agents-and-global-enterprise-expansion

17. Sensay, CEO Dan Thomson profile and company information. https://danthomson.ai/

18. Dagama World, “Sensay CEO Dan Thomson on Digital Identity and Nomadic Leadership.” https://www.dagama.world/blog/sensay-ceo-dan-thomson-on-digital-identity-and-nomadic-leadership

19. Luciano Floridi, “The Ethics of Artificial Intelligence: Principles, Challenges, and Opportunities,” Oxford University Press, 2023. https://global.oup.com/academic/product/the-ethics-of-artificial-intelligence-9780198883098

20. Luciano Floridi, “Artificial Intelligence, Deepfakes and a Future of Ectypes,” SSRN. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3834958

21. ULPA, “The Rise of AI in Japan: A Complete Guide for 2025.” https://www.ulpa.jp/post/the-rise-of-ai-in-japan-a-complete-guide-for-2025

22. World Economic Forum, “What Japan's path to responsible AI can teach us,” January 2026. https://www.weforum.org/stories/2026/01/japan-path-to-responsible-ai-and-what-it-can-teach-us/

23. Edelman, “The AI Trust Imperative: Navigating the Future with Confidence,” 2025 Trust Barometer. https://www.edelman.com/trust/2025/trust-barometer/report-tech-sector

24. Happily.ai, “The 2026 State of Workplace Trust: How Recognition Frequency Predicts Retention,” 2026. https://happily.ai/blog/state-of-workplace-trust-2026/

25. ArentFox Schiff, “The Business of AI Avatars: Key Legal Risks and Best Practices.” https://www.afslaw.com/perspectives/alerts/the-business-ai-avatars-key-legal-risks-and-best-practices

26. Traverse Legal, “AI Twins and Avatars: Legal Risks for Companies Using Synthetic Voice and Likeness Technology.” https://www.traverselegal.com/blog/ai-avatar-legal-risks/

27. GMO Research and AI, “Japan's Generative AI Market Penetration and Business Adoption Trends 2025.” https://gmo-research.ai/en/resources/studies/2025-study-gen-AI-jp

28. PwC, “Global Workforce Hopes and Fears Survey 2025.” https://www.pwc.com/gx/en/issues/workforce/hopes-and-fears.html

29. Microsoft 365 Blog, “Microsoft Ignite 2025: Copilot and agents built to power the Frontier Firm,” November 2025. https://www.microsoft.com/en-us/microsoft-365/blog/2025/11/18/microsoft-ignite-2025-copilot-and-agents-built-to-power-the-frontier-firm/

30. Otter.ai, “Having Generated $1 Billion+ Annual ROI for Customers, Otter.ai Aims for Complete Meeting Transformation.” https://otter.ai/blog/having-generated-1-billion-annual-roi-for-customers-otter-ai-aims-for-complete-meeting-transformation-by-launching-next-gen-enterprise-suite

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

On 20 March 2026, WordPress.com flipped a switch that most of the internet did not notice but probably should have. The platform, which powers more than 43 per cent of all websites globally according to figures presented at Automattic's State of the Word event in December 2025, enabled AI agents to autonomously write, edit, publish, and manage entire websites. Not draft suggestions. Not autocomplete. Full publishing control, handed to machines through a protocol that lets Claude, ChatGPT, Cursor, and any other compatible AI client operate a WordPress site the way a human editor once did.

The update added 19 new writing capabilities across six content types: posts, pages, comments, categories, tags, and media. From a single natural-language prompt, an AI agent can now draft and publish a post, build a landing page using a site's existing theme and block patterns, approve and reply to comments, reorganise category structures, or fix missing alt text across an entire media library. The agent even understands your site's design system, inheriting its colours, fonts, spacing, and patterns so that everything it produces looks as though a human built it with care.

WordPress.com users already publish 70 million new posts every month. That is 1,600 new blog posts every minute, or roughly 26 every second. Now imagine what happens when you remove the bottleneck of human typing speed, human fatigue, and human doubt from that equation entirely.

Welcome to the age of autonomous publishing. The question is no longer whether AI can write for the web. It is whether anyone will be able to tell the difference, or whether it will even matter.

WordPress Hands Over the Keys

The technical architecture behind this shift is worth understanding, because it reveals how deliberately the infrastructure was built. WordPress.com's AI agent capabilities run on the Model Context Protocol, an open standard that governs how applications provide context to large language models. Automattic first introduced MCP on WordPress.com in October 2025, but at that stage it was read-only. Agents could query a site, read its content, analyse its structure, but they could not touch anything.

A second update in January 2026 added OAuth 2.1 authentication, making it simpler to connect AI clients securely. In February, Automattic launched an official Claude Connector, still read-only. The March update was the step the company had been building towards all along: full write access.

Matt Mullenweg, the co-creator of WordPress and CEO of Automattic, has been vocal about his vision for an AI-native web. In a February 2026 blog post, he laid out a roadmap for “agentic usability,” arguing that WordPress should strengthen its APIs, command-line tools, and machine-friendly interfaces so that personal AI agents can safely operate WordPress tasks without brittle user-interface automation. He called for WordPress.org to provide markdown versions of every page, covering not just documentation but forums, directories, and bug trackers, making WordPress content more easily parseable by AI agents.

“How perfect is that for AI to work with?” Mullenweg wrote, describing how WordPress Playground can spin up fully containerised WordPress instances in 20 to 45 seconds, allowing AI to test code changes across more than 20 environments simultaneously. His stated ambition: to take WordPress “from millions of WordPresses in the world to billions.”

Automattic has built in safety mechanisms, and they are worth enumerating because they reveal how the company is thinking about the tension between automation and oversight. New posts default to draft status, giving users a chance to review before anything goes live. If you update a published post, the agent warns that changes will be visible immediately. Deletions of posts, pages, comments, and media move to trash and remain recoverable for 30 days. Permanent taxonomy deletions require a second confirmation. All agent activity appears in the site's existing Activity Log. The agent inherits standard WordPress user-role restrictions, so an Editor cannot change site settings and a Contributor cannot publish. Each of the 19 operations can be individually toggled on or off per site through the MCP dashboard at wordpress.com/me/mcp.

But the fundamental shift is unmistakable: the platform that hosts nearly half the web has decided that machines should be allowed to run it.

The Numbers Behind the Flood

The WordPress announcement did not arrive in a vacuum. It landed in a digital landscape already saturated with machine-generated text, and the data paints a picture that would have seemed absurd even three years ago.

In April 2025, Ahrefs analysed nearly 900,000 newly created English-language web pages, one per domain, using its “botornot” detection tool. The finding was stark: 74.2 per cent of those pages contained AI-generated content. Only 25.8 per cent were classified as purely human-written. The remaining 71.7 per cent were a hybrid of human and AI work, with just 2.5 per cent identified as “pure AI” with no human editing whatsoever. The study also found that 86.5 per cent of top-ranking pages in search results contained some amount of AI-generated content, and that 91.4 per cent of pages cited in Google's AI Overviews did as well.

A separate study by Graphite, which analysed 65,000 English-language URLs from Common Crawl, found that as of November 2024, 50.3 per cent of new web articles were generated primarily by AI. That figure had risen from just 5 per cent before ChatGPT launched in late 2022. The percentage briefly surpassed human-written articles in November 2024 before settling into a rough equilibrium where human and AI content exist in near-equal proportions.

Meanwhile, the Imperva Bad Bot Report, published in April 2025 by Thales subsidiary Imperva, revealed that for the first time in a decade, automated traffic had surpassed human activity online, accounting for 51 per cent of all web traffic. Malicious bots alone now represent 37 per cent of internet traffic, up from 32 per cent the previous year. The report attributed much of this surge to the rapid adoption of AI and large language models, which have made bot development accessible to people with limited technical skills. Simple, high-volume bot attacks have soared, now accounting for 45 per cent of all bot attacks, up from 40 per cent in 2023.

The picture is even more striking in specific sectors. NewsGuard, the misinformation tracking organisation, has been cataloguing what it calls “AI Content Farm” websites since May 2023, when it identified just 49 such sites. By February 2024, the count had reached 713. By November 2024, it was 1,121. As of March 2026, NewsGuard has identified 3,006 AI Content Farm sites spanning 16 languages, with Pangram Labs, its detection partner, reporting that between 300 and 500 new AI content farm sites emerge every month. That represents roughly a 60-fold increase in under three years.

These are not fringe blogs. NewsGuard found 141 major brands advertising on AI content farms during one two-month observational period, with an estimated $2.6 billion in advertising revenue per year being unintentionally directed towards misinformation news sites. In August 2025, NewsGuard also found that leading generative AI tools repeat false news claims 35 per cent of the time on average.

When Conspiracy Becomes Measurement

There was a time, not long ago, when suggesting that the internet was mostly bots talking to other bots would have marked you as a conspiracist. The Dead Internet Theory, which first appeared in a 2021 post on Agora Road's Macintosh Cafe by a user called “IlluminatiPirate,” posited that most online content was generated by automated systems rather than real people, with authentic human interaction quietly displaced. It was treated as paranoid speculation, circulated across subreddits and tech forums but never taken seriously by the mainstream.

By 2025, it had moved to the centre of industry discourse. Sam Altman, the CEO of OpenAI, wrote on X: “i never took the dead internet theory that seriously but it seems like there are really a lot of LLM-run twitter accounts now.” At TechCrunch Disrupt in October 2025, Reddit co-founder Alexis Ohanian told Kevin Rose that “the dead internet theory is real.” The relaunch of Digg in January 2026, co-led by Ohanian and Rose, was shut down just two months later in March, citing an “unprecedented bot problem” among other issues.

The numbers validate what was once dismissed as paranoia. On X, approximately 64 per cent of accounts are estimated to be bots. LinkedIn's long-form posts are reportedly 54 per cent AI-generated. AI-generated reviews have been growing at 80 per cent month-over-month since June 2023, and by 2025, 23.7 per cent of real estate agent reviews on Zillow were likely created by AI, up from 3.63 per cent in 2019.

In 2022, Europol's Innovation Lab published a report titled “Law enforcement and the challenge of deepfakes” that included the widely cited claim that experts estimated 90 per cent of online content might be synthetically generated by 2026. That figure has been contested. Some analysts have pointed out that the original report focused specifically on deepfake technology's impact on law enforcement, not on broad AI content generation forecasts, and that for AI content to reach 90 per cent of total online material, it would need to dwarf three decades of accumulated human content. But the directional thrust of the prediction, if not its precise figure, appears increasingly difficult to dismiss.

Gartner, the technology research firm, added fuel to this narrative in February 2024 when it predicted that traditional search engine volume would drop 25 per cent by 2026, with search marketing losing market share to AI chatbots and other virtual agents. Gartner's VP Analyst Alan Antin stated that generative AI solutions were “becoming substitute answer engines, replacing user queries that previously may have been executed in traditional search engines.” Whether or not that specific prediction proves accurate, the shift in how people discover and consume content is undeniable.

The Ouroboros Problem

If the web is filling with AI-generated content, and AI models are trained on data scraped from the web, then a troubling feedback loop emerges. Researchers call it model collapse, though it has also acquired more colourful names: “AI inbreeding,” “AI cannibalism,” and “Habsburg AI.”

The landmark study on this phenomenon was published in Nature in 2024 by Ilia Shumailov of the University of Oxford, Zakhar Shumaylov of the University of Cambridge, Yiren Zhao of Imperial College London, Nicolas Papernot of the University of Toronto, and their colleagues. They investigated what happens when training data inevitably includes content produced by prior AI models, and their findings were sobering.

The team discovered that indiscriminately training generative AI on both real and generated content causes irreversible defects. Models first lose information from the tails of the data distribution, which they termed “early model collapse,” meaning that unusual, minority, or less-represented data disappears first. In later iterations, the data distribution converges so dramatically that it bears almost no resemblance to the original, a phase they called “late model collapse.” Within a few generations of recursive training, original content is replaced by what they described as unrelated nonsense.

The implications for an AI-saturated web are profound. If 74 per cent of newly published web pages already contain AI-generated content, as the Ahrefs data suggests, then the training data for next-generation models is increasingly contaminated with the output of current-generation models. Each cycle introduces small statistical distortions that compound over time, making outputs more homogeneous, less diverse, and increasingly prone to hallucinations. The phenomenon hits minority and less-represented data hardest, meaning that the voices and perspectives most at risk of being erased from AI training data are precisely those that the web was supposed to amplify.

Some researchers have pushed back against the most catastrophic framing. A response paper argued that if synthetic data accumulates alongside human-generated data rather than replacing it, model collapse can be mitigated. They contend that data accumulating over time is a more realistic description of how the web actually works than the assumption that all existing data is deleted and replaced each year. But there is broad agreement across the field that indiscriminate training on AI-generated data degrades model quality, and that the contamination of web data is accelerating faster than mitigation strategies can keep pace.

The practical consequence is that companies are now racing to secure access to verified human-generated content. Reddit signed a licensing deal with Google. News Corp signed one with OpenAI. The market for pre-2022 training data, collected before generative AI flooded the web, has become intensely competitive, and some observers have warned that this could entrench existing AI players who already possess large stores of uncontaminated data over newcomers who do not. Human-written text, once so abundant it was treated as a free resource, has become a strategic asset.

Google Does Not Care How You Made It

Search engines sit at the nexus of this transformation, and Google's response has been more nuanced than many expected. The company's official position, articulated by Google Search Liaison Danny Sullivan and consistent since the March 2024 helpful content guidance update, is straightforward: Google cares about whether content is helpful, not how it was produced.

Appropriate use of AI or automation is not against Google's guidelines. What triggers penalties is low-quality content produced at scale, regardless of whether a human or a machine wrote it. Google's enforcement actions typically result from mass production of thin, low-value pages, persistent factual inaccuracies, or republishing identical or near-identical AI output across multiple sites.

The data suggests this policy is having mixed effects. According to Ahrefs, 86.5 per cent of top-ranking pages now contain some amount of AI-generated content. Yet 86 per cent of the top-ranking pages in Google Search are still primarily human-written, with only 14 per cent classified as AI-generated. Among AI assistants like ChatGPT and Perplexity, the ratio is similar: 82 per cent human to 18 per cent AI. The message from search algorithms appears to be that AI-assisted content is fine, but AI-only content still struggles to reach the top.

Google's E-E-A-T framework, which evaluates Experience, Expertise, Authoritativeness, and Trustworthiness, remains the central ranking signal. AI content that incorporates original research, firsthand experience, clear author credentials, and comprehensive coverage performs similarly to traditional content. AI content that lacks these elements does not, regardless of how polished its prose might be.

But there is a deeper structural shift at play. Google's AI Overviews now appear in over 60 per cent of all searches, up from just 25 per cent in mid-2024. Traditional SEO metrics like domain authority have declined dramatically in importance. And 47 per cent of AI Overview citations now come from pages ranking below position five in traditional search results, suggesting that AI Overviews operate on fundamentally different ranking logic. The gatekeeping function of search, which once determined what content reached human eyes, is itself being reshaped by AI.

Labelling the Synthetic Web

If the web is becoming a place where distinguishing human from machine content matters, then provenance becomes the critical infrastructure. The most significant industry-wide effort on this front is the Coalition for Content Provenance and Authenticity, or C2PA, formed in 2021 through an alliance between Adobe, Arm, Intel, Microsoft, and Truepic, unifying two earlier initiatives: Adobe's Content Authenticity Initiative and Microsoft and the BBC's Project Origin.

C2PA's technical standard, called Content Credentials, functions like a nutrition label for digital content. Each asset carries cryptographically hashed and signed metadata that records when and where it was created, what tools were used, whether generative AI was involved, and what modifications were made along the way. The system is designed to be tamper-evident, meaning that any changes to the asset or its metadata are exposed. A small “CR” icon, the official Content Credentials mark of transparency, allows users to scroll over it and reveal the full provenance chain.

The standard has gained significant institutional backing. The U.S. National Security Agency published guidance in January 2025 recommending Content Credentials as part of a multi-faceted approach to content transparency. Google has integrated C2PA metadata into its Search and advertising systems, allowing users to see whether an image was created or edited with AI tools through the “About this image” feature. The C2PA specification is expected to be adopted as an ISO international standard, marking a milestone in content authenticity governance.

But provenance labelling faces the same challenge as every other transparency initiative in the history of the internet: voluntary adoption. Content Credentials are opt-in. Creators choose whether to apply them. Platforms choose whether to display them. And the incentive structure for AI content farms, which exist precisely because they can produce convincing content at negligible cost, does not favour transparency. The 3,006 AI content farm sites tracked by NewsGuard are unlikely to label their output as synthetic. The NSA's own guidance acknowledged this limitation, recommending that Content Credentials be deployed alongside education, policy, and detection rather than as a standalone solution.

The Human Cost of Infinite Content

The original appeal of the web was the presence of real perspectives, lived experience, and genuine stakes in a conversation. Someone who learned something and wanted to share it. Someone who built something and wanted to show it. Someone who suffered something and wanted to be heard. AI content can simulate all of these with increasing sophistication, but the simulation is, by definition, hollow. There is no person behind it who experienced anything at all.

This is not a theoretical concern. Researchers have begun studying the psychological impact of AI content in sensitive contexts. A study discussed in the Journal of Cancer Education examined what happens when patients in online cancer support forums discover that the support they received came from a large language model rather than a fellow human being. The findings suggest that the perception of authenticity matters enormously to people in vulnerable situations, and that the erosion of trust in online spaces has real consequences for mental health and community resilience.

The economic consequences are equally tangible and already measurable. Writing projects on Upwork declined 32 per cent year over year in 2025, the largest drop of any category on the platform. Within eight months of ChatGPT's launch, freelance writing jobs had dropped 30 per cent. The “Ramp Payrolls to Prompts” study from February 2026 found that more than half the businesses that spent on freelance platforms in 2022 had stopped entirely by 2025. Freelance marketplace spending as a share of total company spend fell from 0.66 per cent to 0.14 per cent, while AI model spending rose from zero to 2.85 per cent of total budgets.

The market has bifurcated. Entry-level project availability fell below 9 per cent, down from 15 per cent the year prior. The $40 blog post and the generic product description have been effectively automated out of existence. But at the top end, something unexpected is happening. Niche specialists report rising demand, with clients explicitly requesting subject-matter expertise and original content without AI involvement. AI-specialised freelancers on Upwork command 25 to 60 per cent higher rates than general practitioners, and AI-related freelance work crossed $300 million in annualised value by late 2025.

The pattern is clear: AI eliminates the floor while raising the ceiling. The writers who can offer what machines cannot, genuine expertise, original reporting, firsthand experience, and authentic voice, are more valuable than ever. Everyone else is competing against a system that works for free.

WordPress's own data illustrates the acceleration. Websites that use AI content saw a median year-over-year growth rate of 29.08 per cent, compared to 24.21 per cent for sites that did not, according to Ahrefs research. AI use allows companies to publish 42 per cent more content each month: a median of 17 articles versus 12 for those not using AI. The productivity advantage is real, and it compounds over time.

Building for Billions of Machine-Run Sites

Matt Mullenweg's vision is not shy about where this leads. He wants WordPress to become the “Web OS” for AI agents, the default platform through which machines interact with and publish to the internet. The WordPress AI Team has been shipping rapidly: the Abilities API shipped in WordPress 6.9, the WP AI Client and Workflows API are coming to WordPress 7.0, WordPress Agent Skills recently moved to an official WordPress repository, and WP-Bench launched in mid-January 2026.

Plugin submissions are accelerating towards 100,000 and beyond, with WordPress planning editorial curation to manage the AI-driven increase in development. Mullenweg has described a future in which billions of WordPress instances exist, many of them spun up and managed entirely by AI agents acting on behalf of individuals, businesses, or other AI systems. While he acknowledges the power of what he calls “vibey vibe coding,” where users prompt AI without deep technical understanding, he argues this approach “will pale in comparison to what the folks who can prompt and vibe code with a knowledge and understanding of what the agents are doing.”

The write capabilities announced on 20 March are available on all paid WordPress.com plans at no additional cost. Users enable them through the MCP dashboard, toggling on the specific operations they want to permit on each site. The barrier to autonomous publishing is now a toggle switch.

This is not a fringe experiment. WordPress holds a 60.5 per cent share of the content management system market. When the dominant platform for web publishing decides that AI agents should have full operational control, the rest of the industry faces a choice: follow WordPress into the age of autonomous publishing, or insist that humans remain in the loop. That answer, as multiple observers have noted, could define how the web works for the next decade.

The Web We Are Building

The honest answer to the question at the heart of this story, whether the internet could soon become a place where the vast majority of content was never touched by a human hand, is that it is already happening. The data from Ahrefs, Graphite, Imperva, and NewsGuard converges on the same conclusion: machine-generated content has become the default mode of web publishing. The WordPress announcement does not create this reality. It formalises it.

What remains uncertain is whether this matters. If an AI agent writes a perfectly accurate, well-structured, beautifully designed blog post about the best hiking trails in the Lake District, and a human being reads it and finds it useful, has something been lost? The information is real. The formatting is professional. The reader got what they came for.

But zoom out. If a thousand AI agents publish a thousand posts about Lake District hiking trails, each slightly rephrasing the same information scraped from the same sources, the web becomes a hall of mirrors. The diversity of perspective that once made the internet extraordinary, the idiosyncratic voice of someone who actually walked those trails in the rain and had a terrible time and wrote about it anyway, gets buried under an avalanche of competent sameness.

The mitigations being developed are real but incomplete. Content Credentials offer provenance but rely on voluntary adoption. Google's quality signals reward expertise but cannot distinguish authentic experience from convincing simulation. WordPress's safety controls default to drafts but do not prevent a determined operator from automating everything. Model collapse research warns of degradation but cannot halt the economic incentives driving synthetic content production.

The web is not dead. But it is changing in ways that demand attention. The machines are publishing now, and they are publishing at scale, with the full support of the platforms that host the internet's infrastructure. The question for the next decade is not whether AI content will dominate the web. It is whether the humans who still care about what they write, and what they read, can build the tools, standards, and cultural norms to ensure that authenticity retains its value in a world of infinite synthetic supply.

That is not a technical problem. It is a civilisational one.

---

References and Sources

1. WordPress.com Blog, “AI agents can now create and manage content on WordPress.com,” published 20 March 2026. Available at: https://wordpress.com/blog/2026/03/20/ai-agent-manage-content/

2. TechCrunch, “WordPress.com now lets AI agents write and publish posts, and more,” published 20 March 2026. Available at: https://techcrunch.com/2026/03/20/wordpress-com-now-lets-ai-agents-write-and-publish-posts-and-more/

3. The Next Web, “WordPress.com lets AI agents write, publish, and manage your site,” March 2026. Available at: https://thenextweb.com/news/wordpress-com-mcp-write-capabilities-ai-agent

4. Matt Mullenweg, “WP & AI,” personal blog, February 2026. Available at: https://ma.tt/2026/02/wp-ai/

5. Matt Mullenweg, “WP.com MCP,” personal blog, March 2026. Available at: https://ma.tt/2026/03/wp-com-mcp/

6. Ahrefs, “74% of New Webpages Include AI Content (Study of 900k Pages),” 2025. Available at: https://ahrefs.com/blog/what-percentage-of-new-content-is-ai-generated/

7. Graphite, analysis of 65,000 English-language URLs from Common Crawl, findings reported across multiple outlets including eWeek, “AI Now Writes Half of the Internet, but Still Ranks Behind Humans,” 2025. Available at: https://www.eweek.com/news/ai-writes-half-internet/

8. Imperva (Thales), “2025 Bad Bot Report,” published April 2025. Available at: https://www.imperva.com/resources/resource-library/reports/2025-bad-bot-report/

9. Thales Group press release, “AI-Driven Bots Surpass Human Traffic – Bad Bot Report 2025,” 2025. Available at: https://cpl.thalesgroup.com/about-us/newsroom/2025-imperva-bad-bot-report-ai-internet-traffic

10. NewsGuard, “Tracking AI-enabled Misinformation: 3,006 AI Content Farm sites (and Counting),” March 2026. Available at: https://www.newsguardtech.com/special-reports/ai-tracking-center/

11. NewsGuard, “Watch Out: AI 'News' Sites Are on the Rise,” 2024. Available at: https://www.newsguardtech.com/insights/watch-out-ai-news-sites-are-on-the-rise/

12. Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R. and Gal, Y., “AI models collapse when trained on recursively generated data,” Nature, volume 631, pages 755-759, 2024. Available at: https://www.nature.com/articles/s41586-024-07566-y

13. Europol Innovation Lab, “Law enforcement and the challenge of deepfakes,” 2022. Referenced across multiple outlets including Futurism, “Experts: 90% of Online Content Will Be AI-Generated by 2026.” Available at: https://futurism.com/the-byte/experts-90-online-content-ai-generated

14. Google Search Central Blog, “Google Search's guidance about AI-generated content,” February 2023, updated 2024. Available at: https://developers.google.com/search/blog/2023/02/google-search-and-ai-content

15. CMSWire, “Automattic Boosts WordPress.com with Anthropic, OpenAI & AI Agents,” March 2026. Available at: https://www.cmswire.com/digital-experience/wordpresscom-enables-ai-agents-to-write-manage-content/

16. C2PA (Coalition for Content Provenance and Authenticity), official website and technical specification, 2025. Available at: https://c2pa.org/

17. U.S. Department of Defense / NSA, “Strengthening Multimedia Integrity in the Generative AI Era,” published January 2025. Available at: https://media.defense.gov/2025/Jan/29/2003634788/-1/-1/0/CSI-CONTENT-CREDENTIALS.PDF

18. Google Blog, “How Google and the C2PA are increasing transparency for gen AI content,” 2025. Available at: https://blog.google/technology/ai/google-gen-ai-content-transparency-c2pa/

19. TIME, “Sam Altman Voices Concern Over Dead Internet Theory,” 2025. Available at: https://time.com/7316046/sam-altman-dead-internet-theory/

20. Wikipedia, “Dead Internet theory,” accessed March 2026. Available at: https://en.wikipedia.org/wiki/Dead_Internet_theory

21. WebProNews, “WordPress Hands the Keys to AI Agents – and the Implications for Publishing Are Enormous,” March 2026. Available at: https://www.webpronews.com/wordpress-hands-the-keys-to-ai-agents-and-the-implications-for-publishing-are-enormous/

22. Ahrefs, “Websites Using AI Content Grow 5% Faster [+ New Research Report],” 2025. Available at: https://ahrefs.com/blog/websites-using-ai-content-grow-faster/

23. Ahrefs, “80+ Up-to-Date AI Statistics for 2025,” 2025. Available at: https://ahrefs.com/blog/ai-statistics/

24. Gartner, “Gartner Predicts Search Engine Volume Will Drop 25% by 2026, Due to AI Chatbots and Other Virtual Agents,” published February 2024. Available at: https://www.gartner.com/en/newsroom/press-releases/2024-02-19-gartner-predicts-search-engine-volume-will-drop-25-percent-by-2026-due-to-ai-chatbots-and-other-virtual-agents

25. Mediabistro, “Freelance Writing Jobs & AI in 2026: Real Data,” 2026. Available at: https://www.mediabistro.com/go-freelance/freelance-writing-jobs-in-the-age-of-ai-what-the-data-says-and-how-to-position-yourself/

26. Winvesta, “AI cut freelance rates 30%: How top earners fight back in 2026,” 2026. Available at: https://www.winvesta.in/blog/freelancers/ai-cut-freelance-rates-30-how-top-earners-fight-back

27. NewsGuard, “NewsGuard Launches Real-time AI Content Farm Detection Datastream,” 2026. Available at: https://www.newsguardtech.com/press/newsguard-launches-real-time-ai-content-farm-detection-datastream-to-counter-onslaught-of-ai-slop-in-news/

28. Harvard Journal of Law and Technology, “Model Collapse and the Right to Uncontaminated Human-Generated Data,” 2025. Available at: https://jolt.law.harvard.edu/digest/model-collapse-and-the-right-to-uncontaminated-human-generated-data

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

In November 2025, Yann LeCun walked into Mark Zuckerberg's office and told his boss he was leaving. After twelve years building Meta's AI research operation into one of the most respected in the world, the Turing Award winner had decided that the entire industry was heading in the wrong direction. Four months later, his new venture, Advanced Machine Intelligence Labs, announced the largest seed round in European startup history: $1.03 billion to build AI systems that do not merely predict the next word in a sentence, but understand how physical reality actually works.

The money is staggering. The ambition is larger. And the question it raises is one that should unsettle anyone paying attention: if we succeed in building machines that can model the physical world with superhuman fidelity, will we have any idea what those machines actually know?

Welcome to the age of world models, where the gap between what AI understands and what we understand about AI threatens to become the defining tension of the next decade.

A Turing Winner's Trillion-Dollar Heresy

LeCun has never been shy about his contrarian streak. Even whilst serving as Meta's chief AI scientist, he publicly and repeatedly argued that the industry's obsession with large language models was fundamentally misguided. “Scaling them up will not allow us to reach AGI,” he has said, a position that put him at odds with the prevailing orthodoxy at OpenAI, Google, and, increasingly, within his own employer. His departure, first confirmed in a December 2025 LinkedIn post, was not merely a career move. It was a declaration of intellectual war.

AMI Labs, headquartered in Paris with additional offices in New York, Montreal, and Singapore, is built around a deceptively simple thesis: real intelligence does not begin in language. It begins in the world. The company's technical foundation is LeCun's Joint Embedding Predictive Architecture, or JEPA, a framework he first proposed in a 2022 position paper titled “A Path Towards Autonomous Intelligence.” Where large language models like ChatGPT, Claude, and Gemini learn by predicting the next token in a sequence of text, JEPA learns by predicting abstract representations of sensory data. It does not try to reconstruct every pixel or predict every word. Instead, it learns to capture the structural, meaningful patterns that govern how environments behave and change over time.

The distinction matters enormously. LeCun has used the example of video prediction to illustrate the point: trying to forecast every pixel of a future video frame is computationally ruinous, because the world is full of chaotic, unpredictable details like flickering leaves, shifting shadows, and textured surfaces. A generative model wastes enormous capacity modelling this noise. JEPA sidesteps the problem entirely by operating in an abstract embedding space, focusing on the low-entropy, structural aspects of a scene rather than its surface-level chaos.

The $1.03 billion seed round, which values AMI at $3.5 billion pre-money, drew an extraordinary roster of backers. The round was co-led by Cathay Innovation, Greycroft, Hiro Capital, HV Capital, and Bezos Expeditions. Additional investors include NVIDIA, Temasek, Samsung, Toyota Ventures, and Bpifrance, alongside individuals such as Jeff Bezos, Mark Cuban, and Eric Schmidt. LeCun initially sought approximately 500 million euros, according to a leaked pitch deck reported by Sifted. Demand far exceeded that figure.

Day-to-day operations are led by Alexandre LeBrun, the French entrepreneur who previously founded and ran Nabla, a medical AI startup. The leadership roster also includes Saining Xie, formerly of Google DeepMind, as chief science officer; Pascale Fung as chief research and innovation officer; Michael Rabbat as VP of world models; and Laurent Solly, Meta's former VP for Europe, as chief operating officer. LeCun himself serves as executive chairman whilst maintaining his professorship at New York University.

LeBrun has been candid about the timeline. “AMI Labs is a very ambitious project, because it starts with fundamental research,” he has said. “It's not your typical applied AI startup that can release a product in three months.” Within three to five years, LeCun has stated, the goal is to produce “fairly universal intelligent systems” capable of deployment across virtually any domain requiring machine intelligence. The initial commercial targets include healthcare, robotics, wearables, and industrial automation.

What World Models Actually Are (and Why They Change Everything)

To grasp why a billion dollars is flowing into world models, you need to understand what they are and why the current generation of AI systems falls short. A world model, in its simplest formulation, is an AI system designed to understand and predict how the physical world works. Gravity, motion, cause and effect, spatial relationships, object permanence: these are the kinds of knowledge that a world model attempts to internalise, not through explicit programming, but through learning from vast quantities of sensory data.

This is not an entirely new idea. The concept of internal models of reality has deep roots in cognitive science, where researchers have long argued that human intelligence depends on our brain's ability to simulate possible futures before we act. When you reach for a glass of water, you do not consciously calculate trajectories and grip forces. Your brain runs a rapid internal simulation, predicting what will happen and adjusting on the fly. World models attempt to give machines a similar capability.

Google DeepMind CEO Demis Hassabis, the 2024 Nobel laureate in Chemistry, has articulated the problem with current approaches in characteristically vivid terms. At the India AI Impact Summit in February 2026, he described today's AI systems as possessing “jagged intelligence,” explaining: “Today's systems can get gold medals in the International Maths Olympiad, really hard problems, but sometimes can still make mistakes on elementary maths if you pose the question in a certain way. A true general intelligence system shouldn't have that kind of jaggedness.” Large language models, Hassabis has argued, are ultimately sophisticated probability predictors. They do not genuinely understand the physical laws of the real world.

Fei-Fei Li, the Stanford professor often described as the “godmother of AI” for her foundational work on ImageNet, has put it even more bluntly. LLMs, she has said, are like “wordsmiths in the dark,” possessing elaborate linguistic ability but lacking spatial intelligence and physical experience. Her own company, World Labs, released its Marble world model in November 2025, capable of generating entire 3D worlds from a text prompt, image, video, or rough layout. World Labs is now reportedly in discussions at a $5 billion valuation after raising $230 million in funding.

The broader landscape is moving rapidly. Google DeepMind launched Genie 3, the first real-time interactive world model capable of generating navigable 3D environments at 24 frames per second, maintaining strict object permanence and consistent physics without a separate memory module. NVIDIA's Cosmos platform, announced at CES 2025 and trained on 9,000 trillion tokens drawn from 20 million hours of real-world data, has surpassed 2 million downloads. Waymo has built its autonomous vehicle world model on top of Genie 3, using it to train self-driving cars in simulated environments. Reports indicate that OpenAI triggered a “code red” response to Genie 3's capabilities, accelerating efforts to add spatial understanding to GPT-5.

Over $1.3 billion in funding flowed into world model startups in early 2026 alone. This is not a niche research interest. It is rapidly becoming the central front in the race towards more capable AI.

The Architecture of Understanding

AMI Labs' approach differs from its competitors in important ways. Where World Labs focuses on generating photorealistic 3D environments and DeepMind's Genie 3 emphasises interactive simulation, JEPA is fundamentally about learning representations rather than generating outputs.

The architecture works through a deceptively elegant mechanism. JEPA takes a pair of related inputs, such as consecutive video frames or adjacent image patches, and encodes each into an abstract representation using separate encoder networks. A predictor module then attempts to forecast the representation of the “target” input from the representation of the “context” input. Crucially, this prediction happens entirely in abstract embedding space, never at the level of raw pixels or tokens.

This creates what amounts to a learned physics engine. The system develops an internal model of how things relate to one another and how they change over time, without being burdened by the task of reconstructing surface-level details. An optional latent variable, often denoted as z, allows the model to account for inherent uncertainty, representing different hypothetical scenarios for aspects of the target that the context alone cannot determine.

Several variants already exist. I-JEPA learns by predicting representations of image regions from other regions, developing abstract understanding of visual scenes without explicit labels. V-JEPA extends this to video, predicting missing or masked parts of video sequences in representation space, pre-trained entirely with unlabelled data. VL-JEPA adds vision-language capability, predicting continuous embeddings of target texts rather than generating tokens autoregressively, achieving stronger performance with 50 per cent fewer trainable parameters.

The promise is tantalising. An AI system built on JEPA principles could, in theory, develop the kind of intuitive physical understanding that enables a child to predict that pushing a table will move the book sitting on it. It could reason about cause and effect, plan actions in the physical world, and adapt to novel situations without the brittleness that characterises current systems.

But there is a catch. And it is a significant one.

The Understanding Gap Widens

Here is the paradox at the heart of the world models revolution: the better these systems become at understanding physical reality, the harder they become for us to understand. We are constructing machines designed to build rich internal representations of how the world works, and we have strikingly little ability to inspect, interpret, or verify what those representations actually contain.

This is not a new problem, but world models threaten to make it dramatically worse. The interpretability challenges that plague current large language models are already formidable. Mechanistic interpretability, the effort to reverse-engineer neural networks into human-understandable components, has been recognised by MIT Technology Review as a “breakthrough technology for 2026.” Yet the field remains at what researchers describe as a critical inflection point, with genuine progress coexisting alongside fundamental barriers.

The core difficulty is what researchers call superposition. Because there are more features that a neural network needs to represent than there are dimensions available to represent them, the network compresses information in ways that produce polysemantic neurons, individual units that contribute to multiple, semantically distinct features. Understanding what a network “knows” requires disentangling this compressed representation, and the dominant tool for doing so, sparse autoencoders, faces serious unsolved problems. Reconstruction error remains stubbornly high, with 10 to 40 per cent performance degradation. Features split and absorb in unpredictable ways. And the results depend heavily on the specific dataset used.

Anthropic, the AI safety company, has made mechanistic interpretability a central focus, extracting interpretable features from its Claude 3 Sonnet model using sparse autoencoders and publishing results showing features related to deception, sycophancy, bias, and dangerous content. Their attribution graphs, released in March 2025, can successfully trace computational paths for roughly 25 per cent of prompts. For the remaining 75 per cent, the computational pathways remain opaque.

A 2025 paper published at the International Conference on Learning Representations proved that many circuit-finding queries in neural networks are NP-hard, remain fixed-parameter intractable, and are inapproximable under standard computational assumptions. In plain language: for many of the questions we most urgently need to answer about what neural networks are doing, there may be no efficient algorithm that can provide the answer.

Now consider what happens when you move from language models to world models. JEPA operates in abstract embedding spaces that are, by design, removed from human-interpretable inputs and outputs. A language model at least traffics in words, which we can read. A world model's internal representations are abstract mathematical objects encoding relationships between physical phenomena. The interpretability challenge is not merely scaled up. It is qualitatively different.

The field is split on how to respond. Anthropic has set the ambitious goal of being able to “reliably detect most AI model problems by 2027.” Google DeepMind, meanwhile, has pivoted away from sparse autoencoders towards what it calls “pragmatic interpretability,” an acknowledgement that full mechanistic understanding of frontier models may be neither achievable nor necessary. Corti, a Danish AI company, has developed GIM (Gradient Interaction Modifications), a gradient-based method that has topped the Hugging Face Mechanistic Interpretability Benchmark, offering improved accuracy for identifying which components in a model are responsible for specific behaviours. But even these advances represent incremental progress against an exponentially growing challenge.

When Physics Engines Dream

The practical implications of AI systems that can simulate physical reality extend far beyond academic curiosity. Consider the domains AMI Labs is targeting: healthcare, robotics, wearables, and industrial automation. In each of these fields, the consequences of AI misunderstanding the physical world range from costly to catastrophic.

AMI Labs has already established a partnership with Nabla, the healthtech company LeBrun previously founded, providing a direct conduit to the healthcare sector. In medicine, the hallucinations that plague large language models are not merely embarrassing; they can be lethal. A world model that genuinely understands human physiology, drug interactions, and disease progression could revolutionise clinical decision-making. But the opacity of that understanding creates a novel kind of risk: a system that is right for reasons nobody can articulate, or wrong for reasons nobody can detect.

In robotics, world models promise to solve one of the field's most persistent bottlenecks. Training robots in the physical world is slow, expensive, and dangerous. World models enable training in simulation, where a robot can experience millions of scenarios in hours rather than years. NVIDIA's Cosmos platform already allows autonomous vehicle and robotics developers to synthesise rare, dangerous edge-case conditions that would be prohibitively risky to create in reality. But the fidelity of the simulation depends entirely on the accuracy of the world model, and verifying that accuracy requires understanding what the model has learned, which brings us back to the interpretability gap.

The autonomous vehicle industry illustrates the stakes with particular clarity. Waymo's decision to build its world model on Google DeepMind's Genie 3 represents a bet that AI-generated simulations can adequately capture the chaotic complexity of real-world driving. The potential benefits are enormous: safer vehicles, faster development cycles, dramatically reduced testing costs. The potential risks are equally significant. If the world model contains subtle errors in its understanding of physics (the way light refracts in rain, the friction coefficient of wet roads, the behaviour of pedestrians at unmarked crossings) those errors will be systematically baked into every vehicle trained on the simulation.

Governing What We Cannot See

The regulatory landscape is struggling to keep pace with these developments. The European Union's AI Act, the world's most comprehensive legal framework for artificial intelligence, entered into force in August 2024 and will be fully applicable by August 2026. Its risk-based classification system imposes graduated obligations based on potential harm, with penalties reaching up to 35 million euros or 7 per cent of global annual turnover for the most serious violations.

But the AI Act was designed primarily with current AI systems in mind. Its requirements for high-risk systems, including documented risk management, robust data governance, detailed technical documentation, automatic logging, human oversight, and safeguards for accuracy and robustness, assume a level of inspectability that world models may not provide. How do you document the risk management of a system whose internal representations of physical reality are abstract mathematical objects that resist human interpretation? How do you ensure “human oversight” of a physics simulation running in an embedding space that no human can directly perceive?

The European Council, on 13 March 2026, agreed a position to streamline rules on artificial intelligence, whilst the Commission's Digital Omnibus package, submitted in November 2025, proposed adjusting the timeline for high-risk system obligations. But these adjustments are largely procedural. The fundamental question of how to regulate AI systems whose internal workings are opaque to their creators remains unaddressed.

At the broader international level, the AI Impact Summit 2026 in New Delhi produced a Leaders' Declaration recognising that “AI's promise is best realised only when its benefits are shared by humanity.” The International Institute for Management Development's AI Safety Clock, which began at 29 minutes to midnight in September 2024, now stands at 18 minutes to midnight as of March 2026, reflecting growing expert concern about the pace of AI development relative to safety measures.

In the United States, the NIST AI Risk Management Framework and ISO/IEC 42001 provide voluntary guidelines, but nothing approaching the binding force of the EU's approach. China's own regulatory framework focuses on algorithmic transparency and content generation, but similarly lacks specific provisions for world models. The result is a patchwork of rules designed for yesterday's AI, applied to tomorrow's.

Voices From Both Sides of the Divide

The debate over world models and their implications has produced sharp divisions amongst the people who understand these systems best.

LeCun himself has been consistently dismissive of existential risk concerns. He has called discussion of AI-driven existential catastrophe “premature,” “preposterous,” and “complete B.S.,” arguing that superintelligent machines will have no inherent desire for self-preservation and that AI can be made safe through continuous, iterative refinement. His position is that the path to safety runs through open science and open source, not through restriction and secrecy. Staying true to this philosophy, AMI Labs has committed to publishing its research and releasing substantial code as open source. “We will also make a lot of code open source,” LeBrun has confirmed.

Geoffrey Hinton, who shared the 2018 Turing Award with LeCun and Yoshua Bengio for their contributions to deep learning, occupies the opposite pole. The researcher often described as the “Godfather of AI” has warned that advanced AI will become “much smarter than us” and render controls ineffective. At the Ai4 conference in 2025, Hinton proposed a “mother AI” concept to safeguard against potential AI takeover scenarios. Their public disagreements have become one of the defining intellectual conflicts in the field.

The broader expert community is similarly divided. Roman Yampolskiy, a computer scientist at the University of Louisville known for his work on AI safety, estimates a 99 per cent chance of an AI-caused existential catastrophe. LeCun places that probability at effectively zero. A survey of AI experts published in early 2025 found that many researchers, while highly skilled in machine learning, have limited exposure to core AI safety concepts, and that those least familiar with safety research are also the least concerned about catastrophic risk.

AGI timeline estimates vary wildly. Elon Musk has predicted AGI by 2026. Dario Amodei, CEO of Anthropic, has suggested 2026 or 2027. NVIDIA CEO Jensen Huang places the date at 2029. LeCun himself has argued it will take several more decades for machines to exceed human intelligence. Gary Marcus, the cognitive scientist and persistent AI sceptic, has suggested the timeline could be 10 or even 100 years.

What is notable about the world models debate is that it cuts across these existing fault lines. You do not need to believe in imminent superintelligence to be concerned about the understanding gap. A world model does not need to be superintelligent to be dangerous if it is deployed in high-stakes domains whilst remaining fundamentally opaque. The risk is not necessarily that AI becomes too smart. It is that AI becomes smart enough to matter in ways we cannot verify.

Reading the Black Box, Through a Glass Darkly

The technical community has not been idle in the face of these challenges. New architectures and methods are emerging that offer at least partial responses to the interpretability crisis.

Kolmogorov-Arnold Networks, or KANs, represent a fundamentally different neural network architecture that decomposes higher-dimensional functions into one-dimensional functions, increasing interpretability and allowing scientists to identify important features, reveal modular structures, and discover symbolic formulae in scientific data. However, their interpretability diminishes as network size increases, presenting a familiar scalability challenge: the very systems we most need to understand are the ones that resist understanding most stubbornly.

The collaborative paper published in January 2025 by 29 researchers across 18 organisations established the field's consensus open problems for mechanistic interpretability. Core concepts like “feature” still lack rigorous mathematical definitions. Computational complexity results prove that many interpretability queries are intractable. And practical methods continue to underperform simple baselines on safety-relevant tasks.

There is also the question of whether full interpretability is even the right goal. Some researchers argue for a more pragmatic approach: rather than trying to understand everything a model knows, develop reliable methods for detecting when a model is likely to fail. This is the philosophy behind DeepMind's pivot to pragmatic interpretability and behind Hassabis's proposed “Einstein test” for AGI, which asks whether an AI system trained on all human knowledge up to 1911 could independently discover general relativity. If it cannot, Hassabis argues, it remains “a very sophisticated pattern matcher” regardless of its other capabilities.

LeCun, characteristically, sees the problem differently. He has argued that the architecture itself is the solution: by designing systems that learn structured, abstract representations rather than opaque statistical correlations, world models could ultimately be more interpretable than language models, not less. JEPA's operation in abstract embedding space is, in his view, a feature rather than a bug, because those embeddings encode the meaningful structural relationships that humans also rely on to understand the world, even if the format is different.

This is an optimistic reading. Whether it proves correct will depend on research that has not yet been conducted, using methods that have not yet been invented, applied to systems that have not yet been built. In the meantime, the money is flowing, the labs are hiring, and the world models are being trained.

Europe's Unlikely Gambit

There is a geopolitical dimension to this story that deserves attention. LeCun has stated that there “is certainly a huge demand from the industry and governments for a credible frontier AI company that is neither Chinese nor American.” AMI Labs, with its Paris headquarters and European seed record, is positioning itself to fill that void.

The timing is deliberate. The EU's AI Continent Action Plan, published in April 2025, aims to make Europe a global leader in AI whilst safeguarding democratic values. France's state investment bank Bpifrance is amongst AMI's backers. The company's open research commitment aligns with European regulatory philosophy, which emphasises transparency and accountability in ways that closed American labs like OpenAI and Anthropic have been criticised for resisting.

But Europe's track record in turning fundamental research into commercially dominant technology is, to put it diplomatically, mixed. AMI Labs' $1.03 billion seed round is enormous, but it pales beside the tens of billions flowing into American and Chinese AI labs. LeBrun has acknowledged the challenge, noting that AMI will prioritise quality over quantity in building its team across its four global locations. The question is whether a commitment to open science and European values can coexist with the scale of resources needed to compete at the frontier.

The second-largest seed round ever, raised by the American firm Thinking Machines Lab in June 2025 at $2 billion, provides a sobering comparison. The world models race is global, and capital alone will not determine the winner. But capital certainly helps.

Sleepwalking With Eyes Open

So, are we sleepwalking into a future where AI understands the world better than we do, without us understanding the AI? The honest answer is: we might be, but not in the way the question implies.

The framing of “sleepwalking” suggests unawareness, but the striking thing about the current moment is how many people are aware of the problem and how few solutions are available. The researchers building world models know that interpretability is an unsolved challenge. The regulators drafting AI governance frameworks know that their rules were designed for a different generation of technology. The investors writing billion-dollar cheques know that the commercial applications are years away and the fundamental research questions remain open.

The danger is not ignorance. It is a collective decision to proceed despite uncertainty, driven by competitive pressure, scientific ambition, and the genuine potential of these systems to solve real problems. When LeCun talks about world models revolutionising healthcare by eliminating the hallucinations that make LLMs dangerous in clinical settings, he is not wrong about the potential. When Hassabis describes the need for AI that can reason about physics rather than merely predicting word probabilities, he is identifying a real limitation of current systems. When Fei-Fei Li argues for spatial intelligence as the next frontier, she is pointing towards capabilities that could transform robotics, manufacturing, and scientific discovery.

But potential is not proof. And the understanding gap, the asymmetry between AI's growing capacity to model reality and our limited capacity to model the AI, is real and widening. Every billion dollars invested in making world models more capable should, in principle, be matched by investment in making them more transparent. The evidence suggests that ratio is nowhere close to balanced.

The world models era is not something that is coming. It is here. AMI Labs' billion-dollar bet, backed by some of the most sophisticated investors and researchers on the planet, is one data point amongst many. The question is not whether machines will learn to simulate physical reality. It is whether we will develop the tools to understand what they have learned before the consequences of not understanding become irreversible.

LeCun has said that within three to five years, AMI aims to produce “fairly universal intelligent systems.” The AI Safety Clock stands at 18 minutes to midnight. And the gap between what AI can model and what humans can comprehend about those models grows wider with every training run.

We are not sleepwalking. We are walking with our eyes open, into a future whose shape we can see but whose details remain, for now, profoundly and perhaps permanently, beyond our ability to fully perceive.

References

1. TechCrunch, “Yann LeCun's AMI Labs raises $1.03B to build world models,” 9 March 2026. https://techcrunch.com/2026/03/09/yann-lecuns-ami-labs-raises-1-03-billion-to-build-world-models/

2. TechCrunch, “Who's behind AMI Labs, Yann LeCun's 'world model' startup,” 23 January 2026. https://techcrunch.com/2026/01/23/whos-behind-ami-labs-yann-lecuns-world-model-startup/

3. MIT Technology Review, “Yann LeCun's new venture is a contrarian bet against large language models,” 22 January 2026. https://www.technologyreview.com/2026/01/22/1131661/yann-lecuns-new-venture-ami-labs/

4. Sifted, “Yann LeCun's AMI Labs raises $1bn in Europe's biggest seed round,” March 2026. https://sifted.eu/articles/yann-lecun-ami-labs-meta-funding-round-nvidia

5. Crunchbase News, “Turing Winner LeCun's New 'World Model' AI Lab Raises $1B In Europe's Largest Seed Round Ever,” March 2026. https://news.crunchbase.com/venture/world-model-ai-lab-ami-raises-europes-largest-seed-round/

6. TechCrunch, “Yann LeCun confirms his new 'world model' startup, reportedly seeks $5B+ valuation,” 19 December 2025. https://techcrunch.com/2025/12/19/yann-lecun-confirms-his-new-world-model-startup-reportedly-seeks-5b-valuation/

7. Meta AI Blog, “V-JEPA: The next step toward advanced machine intelligence,” 2024. https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-embedding-predictive-architecture/

8. Meta AI Blog, “I-JEPA: The first AI model based on Yann LeCun's vision for more human-like AI,” 2023. https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa/

9. Introl, “World Models Race 2026: How LeCun, DeepMind, and others compete,” 2026. https://introl.com/blog/world-models-race-agi-2026

10. News9live, “India AI Impact Summit 2026: DeepMind CEO Demis Hassabis says current AI still 'Jagged' and learning,” February 2026. https://www.news9live.com/technology/artificial-intelligence/india-ai-summit-2026-deepmind-hassabis-ai-jagged-learning-2932470

11. Storyboard18, “Demis Hassabis says AGI not here yet, calls current AI 'jagged intelligence,'” 2026. https://www.storyboard18.com/brand-makers/google-deepmind-ceo-says-agi-not-here-yet-calls-current-ai-jagged-intelligence-90028.htm

12. European Commission, “AI Act: Shaping Europe's digital future,” 2024. https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai

13. European Council, “Council agrees position to streamline rules on Artificial Intelligence,” 13 March 2026. https://www.consilium.europa.eu/en/press/press-releases/2026/03/13/council-agrees-position-to-streamline-rules-on-artificial-intelligence/

14. TIME, “Meta's AI Chief Yann LeCun on AGI, Open-Source, and AI Risk,” 2024. https://time.com/6694432/yann-lecun-meta-ai-interview/

15. WebProNews, “Yann LeCun and Geoffrey Hinton Clash on AI Safety in 2025,” 2025. https://www.webpronews.com/yann-lecun-and-geoffrey-hinton-clash-on-ai-safety-in-2025/

16. arXiv, “Why do Experts Disagree on Existential Risk and P(doom)? A Survey of AI Experts,” February 2025. https://arxiv.org/html/2502.14870v1

17. Transformer Circuits, “Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet,” 2024. https://transformer-circuits.pub/2024/scaling-monosemanticity/

18. Springer Nature, “Recent Emerging Techniques in Explainable Artificial Intelligence,” 2025. https://link.springer.com/article/10.1007/s11063-025-11732-2

19. Futurum Group, “Yann LeCun's AMI Raises $1BN Seed Round – Is the World Model Era Finally Here?” March 2026. https://futurumgroup.com/insights/yann-lecuns-ami-raises-1bn-seed-round-is-the-world-model-era-finally-here/

20. The Next Web, “Yann LeCun just raised $1bn to prove the AI industry has got it wrong,” March 2026. https://thenextweb.com/news/yann-lecun-ami-labs-world-models-billion

21. Corti, “Corti introduces GIM: Benchmark-leading method for understanding AI model behavior,” 2025. https://www.corti.ai/stories/gim-a-new-standard-for-mechanistic-interpretability

22. PhysOrg, “Kolmogorov-Arnold networks bridge AI and scientific discovery by increasing interpretability,” December 2025. https://phys.org/news/2025-12-kolmogorov-arnold-networks-bridge-ai.html

23. Sombrainc, “An Ultimate Guide to AI Regulations and Governance in 2026,” 2026. https://sombrainc.com/blog/ai-regulations-2026-eu-ai-act

24. Zaruko, “The Einstein Test: Why AGI Is Not Around the Corner,” 2026. https://zaruko.com/insights/the-einstein-test

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

OpenAI began serving advertisements inside ChatGPT on 9 February 2026. Within six weeks, the pilot had crossed $100 million in annualised revenue, with more than 600 advertisers on board and expansion into Canada, Australia, and New Zealand already under way. The company insists it will “never” sell user data to advertisers, that ads will never influence the chatbot's responses, and that the entire system runs on contextual matching rather than behavioural profiling. The language is careful, the assurances are firm, and the underlying question is enormous: does the distinction between contextual relevance and behavioural profiling survive contact with a system that remembers everything you have ever told it?

That question matters because ChatGPT is not a search engine with a text box. It is a conversational interface layered on top of a persistent memory system. Since April 2025, ChatGPT has referenced not only explicit “saved memories” but also the full archive of a user's past conversations to shape its responses. Memory is enabled by default. The system stores your preferences, your interests, your recurring concerns, your tone, your habits. It knows your dog's name and your dietary restrictions. It knows you have been asking about anxiety management every Thursday evening for the past three months. And now, adjacent to those responses, it serves advertisements that are “matched to conversation topics, past chat history, and previous interactions with ads.”

The privacy implications of this arrangement deserve scrutiny that goes well beyond whether OpenAI is technically compliant with its own terms of service. What is at stake is a fundamental question about what “contextual” means when the context never resets.

The Architecture of Remembering

To understand what makes conversational AI advertising fundamentally different from traditional web advertising, you need to understand how memory works in large language models, and how OpenAI has extended that architecture.

A standard LLM does not, on its own, remember anything between sessions. Each conversation is processed within a context window, a fixed-length buffer of tokens that the model uses to generate its next response. When the conversation ends, the context window is cleared. There is no persistent state, no long-term storage, no continuity. This is the architecture that makes the “contextual advertising” framing feel plausible: if the system only knows what you are saying right now, then matching an advertisement to that topic is no different from placing a kitchen appliance ad next to a recipe article.

But ChatGPT has not operated this way for some time. OpenAI introduced its memory feature in early 2024 and expanded it significantly in April 2025. The system now maintains two parallel layers of persistence. The first is “saved memories,” which are explicit facts the model has been asked to retain or has inferred should be retained. The second, and more consequential, is “chat history,” a mechanism that allows the model to reference the full archive of a user's prior conversations when generating new responses. The system does not retain every word verbatim, but it extracts patterns, preferences, and contextual signals that persist indefinitely.

This is not a context window. It is a profile. It may not be stored in a traditional database as a structured dossier, but functionally, it serves the same purpose. The model knows who you are, what you care about, what you have asked about before, and how those interests have evolved over time. When OpenAI says it matches advertisements to “conversation topics, past chat history, and previous interactions with ads,” it is describing a system that uses longitudinal personal data to determine what commercial messages a user is shown. The fact that this data is processed by a neural network rather than a relational database does not change what it is.

OpenAI has stated that ChatGPT is “actively trained not to remember sensitive information, such as health details,” unless explicitly asked. But critics have pointed out the inadequacy of this safeguard. If health details are excluded, what about financial stress? What about relationship difficulties? What about political leanings inferred from a pattern of questions about immigration policy or housing costs? The granular clarity about which categories of sensitive data are eligible for storage, and which are not, is largely absent from OpenAI's public documentation. The system's own judgement about what counts as sensitive is itself opaque.

The Contextual Alibi

OpenAI's public framing leans heavily on the word “contextual.” The company describes its advertising model as a “contextual retrieval engine” that matches ads to “real-time user queries rather than historical behavioral tracking.” This framing is strategically important because contextual advertising occupies a privileged position in privacy regulation. Under the GDPR, contextual advertising, which targets based on the content a user is currently viewing rather than their historical behaviour, generally does not require the same level of consent as behavioural profiling. It does not involve tracking across sites or building persistent profiles. It is, in regulatory terms, the clean option.

But OpenAI's system does not fit neatly into that category. Traditional contextual advertising operates on a stateless model: a user visits a page about running shoes, and the page displays an ad for running shoes. The advertiser knows nothing about the user beyond the fact that they are currently reading about running shoes. There is no memory, no history, no cross-session inference. In principle, contextual advertising treats consumers who request the same content equally and uses identical messaging for all visitors of a website.

ChatGPT's advertising layer operates on a stateful model. The system has access to a user's saved memories, their full conversation history, and their prior interactions with advertisements. When it selects an ad to display, it is not merely responding to the current query in isolation. It is drawing on a rich, persistent, and deeply personal dataset that has been accumulated over months or years of intimate conversational interaction. Two users asking the same question may see different advertisements, not because of the question itself, but because of everything else the system knows about them.

The distinction matters because the regulatory framework for advertising was built around a binary that no longer holds. Contextual advertising was understood as the privacy-preserving alternative precisely because it did not involve persistent data. Behavioural advertising was understood as the privacy-invasive alternative precisely because it did. When a system uses persistent conversational data to inform ad selection but calls itself “contextual,” it occupies a grey zone that existing regulation was not designed to address.

Researchers at TechPolicy.Press have argued that the line between contextual and behavioural advertising is becoming increasingly blurred as AI-driven systems incorporate ever more sophisticated inference capabilities. As one analysis noted, “privacy violations and privacy concerns are not unique to behavioral advertising. They may also be triggered by novel means put forward as 'contextual.'” The concern is not hypothetical. It describes exactly what is happening inside ChatGPT.

Industry observers have noted that companies claiming to operate contextual advertising systems may rely on session data such as browser and page-level data, device and app-level data, IP addresses, and other highly personal elements. In some cases, this may be combined with contextual information to create a comprehensive picture of the people being targeted. The result is that “contextual” becomes a label of convenience rather than a meaningful description of privacy practice.

What the Regulators See (and What They Miss)

The European Data Protection Board's Opinion 28/2024, adopted in December 2024, provides the most detailed regulatory guidance to date on the intersection of AI models and personal data. The opinion makes several points directly relevant to ChatGPT's advertising model.

First, the EDPB established that personal data used to train AI models does not cease to be personal data merely because it has been transformed into mathematical representations within the model. Even though training data “no longer exists within the model in its original form,” the EDPB considers it still capable of constituting personal data, particularly given that techniques such as model inversion, reconstruction attacks, and membership inference can be used to extract training data.

Second, the EDPB addressed the question of when AI models can be considered anonymous, concluding that anonymity must be assessed on a case-by-case basis and that a model is only anonymous if it is “very unlikely” that individuals can be identified or that personal data can be extracted through queries. The EDPB explicitly rejected the so-called Hamburg thesis, which had proposed that AI models trained on personal data should be treated as anonymous by default. Instead, the Board insisted that anonymity claims require rigorous, case-specific demonstration.

Third, and most relevant to the advertising question, the EDPB clarified that legitimate interest cannot generally serve as the legal basis for processing that involves extensive profiling. This is significant because OpenAI's advertising model, which draws on persistent conversational data to match ads, arguably constitutes a form of profiling under the GDPR's definition: “any form of automated processing of personal data consisting of the use of personal data to evaluate certain personal aspects relating to a natural person, in particular to analyse or predict aspects concerning that natural person's preferences, interests, reliability, behaviour, location or movements.”

The GDPR's definition of profiling does not require that the data be stored in a traditional profile database. It requires that personal data be used to evaluate personal aspects. ChatGPT's memory system does exactly this, continuously and automatically, as a prerequisite for generating personalised responses, and now, as a prerequisite for selecting personalised advertisements.

The Meta precedent is instructive here. In 2023, the EDPB ruled that Meta could not continue targeting advertisements based on users' online activity without affirmative, opt-in consent. The ban was extended permanently across the entire EU and EEA in October of that year, forcing Meta to adopt a consent-based approach and introduce ad-free paid subscriptions at 9.99 euros per month. The ruling established a clear principle: extensive profiling for advertising purposes cannot rely on legitimate interest and requires explicit consent. If that principle applies to Meta's tracking of likes and clicks, it applies with even greater force to OpenAI's processing of intimate conversational data.

Yet regulatory enforcement has been slow to catch up with the specific case of AI advertising. The EDPB created an AI enforcement task force in February 2025 by extending the scope of its existing ChatGPT task force, but concrete enforcement actions specifically targeting AI advertising remain sparse. The EU AI Act, which entered into force in 2024, adds requirements for transparency and human oversight in AI-powered advertising, but its practical application to systems like ChatGPT's ad layer is still being worked out by national regulators and the European AI Office.

A 2024 EU audit found that 63% of ChatGPT user data contained personally identifiable information, with only 22% of users aware of the settings that would allow them to disable data collection. This gap between the theoretical availability of privacy controls and users' actual awareness of them is not a minor implementation detail. It is the central problem.

The Intimacy Problem

There is a qualitative difference between the data that traditional advertising systems collect and the data that conversational AI systems accumulate. Google knows what you search for. Meta knows what you like, share, and comment on. These are signals derived from discrete, observable actions taken in contexts that most users understand, at least in broad terms, to be commercial environments.

ChatGPT knows what you confide. Users interact with conversational AI in a mode that more closely resembles therapy, journalling, or conversation with a trusted friend than it does browsing a website. They discuss their mental health, their relationship problems, their financial anxieties, their career frustrations, their parenting challenges, their creative ambitions. They do so in natural language, with a level of specificity and emotional openness that no search query or social media post would typically capture.

Marketing professor Scott Galloway, commenting on Anthropic's February 2026 Super Bowl advertisement (which carried the tagline “Ads are coming to AI, but not to Claude”), called it a “seminal moment” in the AI industry. Galloway argued that the ad resonated because “the number one use case for AI is therapy, with users routinely sharing their most intimate fears, anxieties, and personal struggles with chatbots.” When the system that receives those disclosures also serves advertisements informed by them, the power asymmetry between platform and user reaches a level that traditional ad-tech never achieved.

A recent controversy involving Meta AI underscored these risks in vivid terms. Users discovered that their private prompts to Meta's AI assistant had been posted to Meta's public “Discover” feed, revealing that people had been sharing deeply personal information with the system under the assumption of confidentiality. The incident demonstrated that users often interact with AI systems as though they are private, even when the platform's architecture does not treat them that way. The chasm between how individuals use these systems and their understanding of the potential implications of such interactions is vast.

The tragic case of Adam Raine, a 16-year-old whose suicide prompted a lawsuit against an AI companionship platform, illustrates the extreme end of this risk. Among the design elements alleged to have contributed to his death was the system's persistent memory capability, which purportedly “stockpiled intimate personal details” about his personality, values, beliefs, and preferences to create a psychological profile that kept him engaged. While ChatGPT's advertising system is not a companionship platform, the underlying mechanism, persistent memory used to build an ever-deepening model of a user's inner life, is architecturally similar.

As TechPolicy.Press observed, “an AI system that gets to know you over your life” is worrisome precisely because “even in human relationships, it is rare for any one person to know us across a lifetime. This limitation serves as an important buffer, constraining the degree of influence that any single individual can exert.” When that buffer is removed, and when the system that knows you most intimately is also the system that serves you commercial messages, the conditions for manipulation become structurally embedded. If long-term memory enhances personalisation, and personalisation increases persuasive power, then the boundary between usefulness and manipulation becomes perilously thin.

The Consent Fiction

OpenAI offers users several mechanisms for controlling how their data is used. Memory can be disabled. Individual memories can be deleted. Chat history can be turned off. Temporary Chat mode allows conversations that are not stored, not used for training, and not referenced by memory. Users on ad-supported tiers can, according to OpenAI, “control the use of memories for ads personalization.” These controls exist. They are documented. They are, in principle, available to anyone who knows where to find them.

The problem is that meaningful consent requires more than the theoretical availability of controls. It requires that users understand what they are consenting to, that they can realistically assess the consequences of their choices, and that the default configuration respects their interests rather than the platform's commercial objectives.

On every one of these criteria, ChatGPT's current design falls short. Memory is enabled by default. Chat history referencing is enabled by default. Ad personalisation, for users on ad-supported tiers, draws on these systems by default. The user who simply opens ChatGPT and starts talking, which is to say the vast majority of ChatGPT's 800 million weekly users, is automatically enrolled in a system that accumulates their personal data, builds a persistent model of their preferences and concerns, and uses that model to select commercial messages.

Disabling these features requires navigating settings menus that most users will never visit. Deleting a chat does not remove saved memories from that conversation. Turning off saved memory does not delete anything already remembered. OpenAI retains logs of deleted saved memories for up to 30 days. The architecture is designed for accumulation, and opting out is an effortful, incomplete, and poorly understood process.

This is not a new problem in technology. The entire history of digital privacy regulation is, in some sense, a response to exactly this pattern: defaults that favour data collection, controls that are technically available but practically invisible, and consent mechanisms that function as legal cover rather than genuine expressions of user preference. But the conversational AI context intensifies the problem in two important ways.

First, the nature of the data is more sensitive. Users disclose things to ChatGPT that they would not type into a Google search bar or post on Facebook. The expectation of privacy in a conversational interface is higher, and the gap between that expectation and the reality of data use is correspondingly wider. Mozilla's Privacy Not Included project has warned that “storing more of your personal information in a tech product is just never a great move for your privacy,” urging users to approach AI memory features with scepticism regardless of how conveniently they are marketed.

Second, the mechanisms of inference are less visible. When Google shows you an ad based on your search history, you can, with some effort, reconstruct the chain of inference. You searched for “best running shoes,” and now you see ads for running shoes. The logic is legible. When ChatGPT shows you an ad based on patterns extracted from months of conversation, the chain of inference is opaque. You do not know which conversations contributed to the selection. You do not know what the system inferred from them. You do not know how those inferences were weighted or combined. The system's reasoning is, by design, not transparent to the user. Users on Hacker News and OpenAI's own community forums have reported that even after disabling all personalisation and memory, ChatGPT appeared to “know things” about them, raising questions about whether the platform's data practices fully match its public documentation.

The Competitive Landscape and What It Reveals

OpenAI is not operating in isolation. Google reportedly told advertisers in late 2025 that it planned to introduce ads into Gemini in 2026. Microsoft's Copilot already serves sponsored results in certain contexts. Perplexity, the AI-powered search engine, has introduced labelled promotional placements. The movement towards advertising in conversational AI is industry-wide, and it is driven by the same economic logic that has governed the internet for two decades: the marginal cost of serving free users is high, subscription conversion rates are low, and advertising is the proven mechanism for monetising attention at scale.

Anthropic's decision to position Claude as an ad-free alternative is commercially significant but strategically ambiguous. Its Super Bowl campaign framed the absence of advertising as a core value proposition. The broadcast version softened the online tagline, settling on “there is a time and place for ads, and AI chats aren't it.” Sam Altman responded publicly, calling the original framing “dishonest” and “deceptive,” arguing that OpenAI would “never run ads in the way Anthropic depicts them.” The exchange revealed a genuine disagreement about the future of AI monetisation, but it also revealed something more important: neither company has fully addressed the underlying privacy question.

Anthropic does not serve ads. But Claude also has memory features and persistent context capabilities. If the absence of advertising is the only privacy safeguard, then the question of what happens to the data accumulated through persistent memory remains unanswered. The risk is not limited to what is monetised today. It extends to what could be monetised tomorrow, or what could be compromised, subpoenaed, or repurposed at any point in the future. OpenAI itself acknowledges that while it states user data is not sold or shared for advertising, it “may disclose your information to affiliates, law enforcement, and the government.”

OpenAI's financial trajectory makes the expansion of advertising virtually certain. Despite achieving $12.7 billion in annual recurring revenue in 2025, the company posted cumulative losses exceeding $13.5 billion in the first half of that year alone. Internal documents project that free-user monetisation will generate $1 billion in 2026 and nearly $25 billion by 2029. Truist analysts have called 2026 an “inflection year” for LLM-powered ads, projecting that within several years, “LLM-powered ad channels will become one of the most important pillars of the digital ad industry.” These are not the projections of a company that plans to keep its advertising footprint modest.

The hiring pattern tells the same story. OpenAI appointed Fidji Simo, the former Meta executive and Instacart CEO who built Instacart's advertising business, as CEO of Applications. Kate Rouch, formerly of Meta and Coinbase, became the company's first Chief Marketing Officer. David Dugan, another former Meta ads executive, was named to lead global advertising solutions in March 2026. Kevin Weil, OpenAI's Chief Product Officer, previously built ad-supported products at Instagram and X. CFO Sarah Friar, hired from Nextdoor in 2024, told the Financial Times that the company would be “thoughtful” about implementing ads, before subsequently tempering expectations. Within fourteen months, the ads were live. This is not a leadership team assembled to keep advertising peripheral.

Where Contextual Becomes Profiling

The core question is not whether OpenAI is acting in bad faith. It may well be sincere in its commitment to keeping ads separate from responses, to never selling conversation data directly, and to giving users controls over memory and personalisation. The core question is whether those commitments are sufficient to prevent contextual advertising from functioning as behavioural profiling when the context is a persistent, intimate, and ever-expanding conversational archive.

The answer, under any honest assessment, is no. The GDPR defines profiling as automated processing that uses personal data to evaluate personal aspects including preferences, interests, and behaviour. ChatGPT's memory system does exactly this. The fact that ad selection happens in real time, based on the current conversation plus the accumulated context, does not make it contextual in the regulatory sense. It makes it a hybrid that combines the real-time matching of contextual advertising with the persistent data accumulation of behavioural profiling. This hybrid is, in many respects, more invasive than either model in isolation, because it operates on data that is more intimate, more detailed, and less visible to the user than anything traditional ad-tech has collected.

The European Parliament's research service has warned that “policymakers need to carefully examine this rapidly evolving space and establish a clear definition of what contextual advertising should entail,” precisely because AI-driven systems are incorporating user-level data and content preference insights while still describing themselves as contextual. The Electronic Frontier Foundation has gone further, arguing that “ad tracking, profiling, and targeting violates privacy, warps technology development, and has discriminatory impacts on users,” and that behavioural advertising online should be banned outright.

These are not fringe positions. They reflect a growing recognition that the categories underpinning privacy regulation, contextual versus behavioural, stateless versus persistent, anonymous versus identified, are losing their coherence in the face of systems that operate across all of these boundaries simultaneously.

Towards Structural Accountability

The path out of this impasse is not more granular privacy settings or more detailed terms of service. Users cannot be expected to manage the boundary between contextual relevance and behavioural profiling through toggle switches in a settings menu. The asymmetry of information is too great. The mechanisms of inference are too opaque. The defaults are too permissive.

What is needed is structural accountability: regulatory frameworks that recognise the unique risks of advertising in conversational AI and impose constraints that do not depend on user vigilance. Several principles should guide this effort.

First, the definition of “contextual advertising” in privacy regulation must be updated to exclude systems that draw on persistent user data, regardless of whether that data is processed by a neural network or a traditional database. If ad selection is informed by anything beyond the current session, it is not contextual. It is profiling.

Second, memory systems in ad-supported AI products should be opt-in rather than opt-out. The current default, where memory is enabled automatically and users must actively navigate settings to disable it, reverses the burden of privacy protection. Users who choose to enable memory for the benefits of personalisation should do so with clear, specific, and genuine informed consent.

Third, regulators should require transparency about the inference chain. When a user sees an advertisement in ChatGPT, they should be able to understand, in concrete terms, what data contributed to its selection, which conversations were referenced, and what preferences or interests were inferred. The current “why am I seeing this ad” mechanism, which OpenAI says it will provide, must go beyond the vague category labels that have characterised similar features on other platforms.

Fourth, independent auditing of AI advertising systems should be mandatory. The opacity of neural network inference means that neither users nor regulators can verify claims about how ad selection works without access to the underlying systems. Third-party audits, conducted by entities with genuine independence and technical capability, are essential.

The stakes are not abstract. OpenAI's advertising system is, as of March 2026, a $100-million-and-growing commercial operation that serves ads to hundreds of millions of users based on the most intimate data any technology platform has ever accumulated. The company's assurances about contextual matching and user control are, at best, an incomplete description of a system that blurs the line between relevance and surveillance. At worst, they are a privacy fig leaf draped over the most sophisticated profiling engine ever built.

The question is not whether contextual advertising in conversational AI is acceptable. It is whether the concept of “contextual” retains any meaningful content when the context is your entire conversational history, your persistent memories, your evolving preferences, and your most private thoughts, all held by a system that has every commercial incentive to remember.

---

Sources and References

1. OpenAI, “Our approach to advertising and expanding access to ChatGPT,” OpenAI Blog, January 2026.
2. CNBC, “OpenAI to begin testing ads on ChatGPT in the U.S.,” 16 January 2026.
3. CNBC, “OpenAI ads pilot tops $100 million in annualized revenue in under 2 months,” 26 March 2026.
4. OpenAI, “Memory and new controls for ChatGPT,” OpenAI Blog, 2024.
5. OpenAI Help Center, “Memory FAQ,” updated 2025.
6. OpenAI Help Center, “What is Memory?,” updated 2025.
7. European Data Protection Board, “Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models,” 18 December 2024.
8. EDPB, “EDPB opinion on AI models: GDPR principles support responsible AI,” press release, December 2024.
9. EDPB, “EDPB adopts statement on age assurance, creates a task force on AI enforcement,” February 2025.
10. European Parliament Research Service, “Regulating targeted and behavioural advertising in digital services,” Study, 2021.
11. TechPolicy.Press, “What We Risk When AI Systems Remember,” 21 October 2025.
12. TechPolicy.Press, “Is So-Called Contextual Advertising the Cure to Surveillance-Based 'Behavioral' Advertising?,” 2024.
13. Electronic Frontier Foundation, “A Promising New GDPR Ruling Against Targeted Ads,” December 2022.
14. Benzinga, “'Ads Are Coming To AI But Not To Claude:' Anthropic's Super Bowl Spot Challenges OpenAI,” February 2026.
15. The Wrap, “OpenAI Considers Ads, Wants to Be 'Thoughtful' About Serving Them With Chat Responses,” December 2024.
16. eWeek, “OpenAI's CFO Discusses Potential ChatGPT Ads While CEO Calls It 'Last Resort',” December 2024.
17. The Information, “Exclusive: OpenAI Surpasses $100 Million Annualized Revenue From Ads Pilot,” March 2026.
18. European Business Magazine, “OpenAI's ChatGPT Embraces Advertising for Revenue Growth,” 2026.
19. Mozilla Foundation, “How to Protect Your Privacy from ChatGPT and Other Chatbots,” Privacy Not Included, 2025.
20. OpenAI, “ChatGPT Privacy Settings,” Consumer Privacy page, 2026.
21. European Data Protection Supervisor, “Revised Guidance on Generative AI,” October 2025.
22. Regulation (EU) 2024/1689, the EU AI Act, entered into force 2024.
23. GDPR, Article 4(4), definition of profiling; Article 6(1)(a) and (f), lawful bases for processing; Article 22, automated individual decision-making.
24. Private Internet Access, “Contextual Advertising Should Be Great for Privacy, But It Risks Being Undermined,” 2025.
25. DLA Piper Privacy Matters, “EU: EDPB Opinion on AI Provides Important Guidance though Many Questions Remain,” January 2025.
26. EDPB ruling on Meta behavioural advertising, October 2023; Meta consent-based advertising rollout, EU/EEA.
27. CNBC, “ChatGPT's ad pilot has the industry excited, but some insiders are frustrated with the slow rollout,” 20 March 2026.
28. OpenAI Community Forum, “Privacy Concerns in ChatGPT's Memory System,” 2025.
29. Hacker News discussion, “Why is OpenAI lying about the data it's collecting on users?,” 2025.

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...