SmarterArticles

On the second floor of the United Nations Headquarters in New York, in a chamber whose acoustics were engineered for the carefully measured cadence of diplomats, an Mbororo pastoralist from Chad delivered a sentence diplomats are not in the habit of hearing. AI, Hindou Oumarou Ibrahim told the room, becomes harmful when it is imposed without free, prior, and informed consent. The line was lifted from her own report, prepared for the twenty-fifth session of the United Nations Permanent Forum on Indigenous Issues, which opened on 21 April and runs until 1 May. It landed with the dull thump of something said many times before, in many forums, about many extractive industries, and that has not yet changed the rules of the game.

Outside the chamber, on the same continent, the rules of the game were being written by a different hand. On 17 April, an Alberta regulator dismissed the Sturgeon Lake Cree Nation's appeal against a water licence allowing six million cubic metres of annual withdrawal from the Smoky River, water destined to cool a proposed seventy-billion-dollar AI data centre marketed by the celebrity investor Kevin O'Leary as “Wonder Valley”. The nation said it had not been meaningfully consulted; the Aboriginal Consultation Office said no consultation was required. The Smoky watershed is the source of the nation's drinking water and the location of ceremonial and traditional land use sites roughly five kilometres downstream from the proposed diversion point. The trapline, the prayer, and the river all sit at a slightly lower elevation than the cooling tower.

This is the shape of the present, in late April 2026, for indigenous peoples whose territories and knowledge are being absorbed into the infrastructure of artificial intelligence. The forum chamber and the riverbank are the same story told in two languages, one of them legalese, the other hydrology. The arrival of AI on indigenous land is not an isolated event. It is the latest chapter in a five-hundred-year sequence of extractive industries deciding what was on indigenous territory was theirs for the taking. What is new, in 2026, is that the resource being extracted is not a mineral or a forest. It is the cognitive substrate of the communities themselves: their knowledge of plants, of weather, of governance, of language, of what is sacred and what is not.

A Forum, A River, A Crawler

The twenty-fifth session of the UN Permanent Forum on Indigenous Issues, known as UNPFII, took as its overarching theme the protection of indigenous peoples' health, including in the context of conflict. AI was not in the title. It was, however, threaded through the proceedings with an urgency that surprised observers expecting the usual catalogue of mining grievances. Ibrahim, a former chair of the forum, presented a study commissioned to map AI's effects on indigenous communities. Her conclusion, which she repeated in interviews with Mongabay and Grist, was that the technology represents a double-edged sword. AI can be a powerful ally to indigenous stewardship, she said, if it is used on our terms. The conditional was load-bearing.

The terms, in 2026, are not yet ours. Generative AI systems trained on web-scale corpora have already absorbed enormous quantities of indigenous-origin material: oral histories deposited in academic archives, ethnobotanical taxonomies recorded by colonial-era anthropologists, sacred narratives transcribed and uploaded by missionaries or by community members themselves under conditions of trust that did not anticipate machine ingestion. Indigenous languages, often digitised by linguists in preservation projects, now sit inside multilingual models whose outputs are deployed back into indigenous communities as the only available translation infrastructure. Kate Finn, Osage Nation citizen and executive director of the Tallgrass Institute, told the forum the question is no longer whether the extraction has happened. The data is gone. The question is what an enforceable framework of indigenous data sovereignty would look like now, and whether anything like restitution is possible for what has already been taken.

Two arXiv papers published on 23 April, the day after Ibrahim's address, gave the question particular sharpness. The first, “Why are all LLMs Obsessed with Japanese Culture? On the Hidden Cultural and Regional Biases of LLMs”, introduced a benchmark called CROQ, comprising 31,680 open cultural questions across 24 languages, eleven major topics, and 66 subtopics. Its authors documented that frontier language models, when asked to answer a culturally underspecified question, default not to a neutral response but to a small handful of dominant cultural reference points, with Japan emerging as a surprising attractor and Western, English-language assumptions saturating the rest. The bias, they found, is induced predominantly during the post-training and instruction-tuning phase: it is not just a property of the data but a property of the alignment regime the data is filtered through.

The second paper, “Multilinguality at the Edge: Developing Language Models for the Global South” by Lester James V. Miranda, Songbo Hu, Roi Reichart and Anna Korhonen, surveyed 232 papers attempting to build language models for non-English-speaking, hardware-constrained communities. They called the underlying challenge “the last mile”: the place where multilinguality and edge deployment goals align in principle but compete in practice, because the corpora, the compute, and the institutional support do not exist on equivalent terms. Read together, the two papers describe the cognitive infrastructure indigenous peoples will inherit if the current trajectory continues. It is an infrastructure that has already absorbed their knowledge, that does not yet speak their languages well enough to give it back, and whose default settings are not theirs.

Digital Extractivism and Its Older Cousins

The phrase indigenous organisers are using for what is happening to them is data colonialism. Krystal Two Bulls, the Oglala Lakota and Northern Cheyenne executive director of Honor the Earth, used it on Democracy Now! during the forum's opening week and has used it in the organising language of the Stop Data Colonialism coalition, a group of indigenous-led organisations now tracking somewhere between 103 and 160 proposed hyperscale data centres on or adjacent to Native lands in North America. The phrase is not a metaphor. It is a technical claim about the structural similarity between the historical practice of treating indigenous land as a frontier of unowned resources to be incorporated into a colonial economy and the current practice of treating indigenous knowledge as an unowned resource to be incorporated into a commercial AI economy.

The structural similarity is not lost on indigenous organisers, who have lived through the previous iterations. In Oklahoma, the Seminole Nation has unanimously passed a moratorium on hyperscale data centres on its land. In Alberta, the Sturgeon Lake Cree Nation is preparing to take its appeal against the Wonder Valley water licence to the province's superior trial court. In Querétaro, Mexico, residents downstream of new hyperscale facilities are documenting wastewater contamination and groundwater depletion. In Pennsylvania, in Thailand's Chonburi and Rayong provinces, in the U.S. Southwest where mega-projects are siting next to drought-stricken aquifers, the same pattern repeats: facility proposed, water licence applied for, consultation declared adequate by the state, communities not adequately consulted, electricity prices in surrounding areas climbing as much as 267 percent in some Bloomberg analyses, and the gigawatts and the gallons flowing out.

Existing hyperscale data centres have been documented to consume between 300,000 and 2.7 million gallons of water a year per facility, with cooling water and the secondary water embedded in their electricity supply both contributing to a footprint that places enormous load on the watersheds chosen to host them. Those watersheds are not random. They are, very often, the watersheds where land is cheap, water rights are weakly defended, and political resistance is structurally underweighted: in plain language, the watersheds nearest to indigenous, rural, and racialised communities. There is a name for this pattern in the environmental justice literature, and the name is environmental racism. The name has not changed because the pattern has not changed.

What is new, on top of this, is the second extraction. The data centre on the Smoky River is, in addition to a water consumer, a node in a planetary system that absorbs the very knowledge of the communities whose water it is using. This is the recursion that gives data colonialism its peculiar bite. A nation watches a facility built upstream of its trapline, knows the facility's compute is being used to train models that have already ingested the linguistic and ecological knowledge of the trapline, and is then offered the resulting AI assistant as a productivity tool to access government services in the language of the colonising state. The water, the knowledge, and the service are all running in the same direction.

What Was Taken, And How

The taxonomy of what has been taken is concrete. Traditional ecological knowledge, often abbreviated TEK, comprises millennia of accumulated observation about ecosystems: which plants flower when, which fish run with which tides, which soils respond to which fires, which weather patterns precede which migrations. Ethnobotanical knowledge encompasses the medicinal and nutritional properties of thousands of plant species, knowledge that pharmaceutical companies have spent decades attempting to extract through bioprospecting and that AI systems, trained on the resulting academic literature and on community-uploaded forums, can now retrieve in seconds. Oral histories, the substrate of governance and law in many indigenous nations, were transcribed throughout the twentieth century and deposited in archives whose access policies were written before web crawlers existed. Indigenous languages, in projects often initiated with explicit consent of speakers but with no anticipation of generative AI, have been digitised, tokenised, and absorbed into multilingual model corpora.

Some of what has been taken was never meant to leave its community. Sacred or restricted knowledge, governed by indigenous protocols specifying who may speak it, when, and to whom, has often been recorded by outsiders, deposited in archives, and crawled. Under the protocols of the originating nation, this knowledge was never publicly available even if it was technically accessible. The distinction between “publicly available” and “publicly available under the protocols of the originating community” is the distinction the entire commercial AI training pipeline has been built on ignoring. To say that something was on the open web is, in the context of indigenous knowledge, often to say nothing more than that a colonial process of recording and depositing was completed at some earlier date and that no subsequent process of consent has been required.

This matters for restitution because traditional knowledge is, in nearly all indigenous legal traditions, held collectively rather than individually. A song, a story, a botanical recipe, a place name: these have custodians, often specified by lineage or role, but their ownership is the nation's, not the individual's. Western intellectual property regimes, optimised for the individual author and the corporate licensee, are structurally incapable of recognising this form of ownership. The General Data Protection Regulation, often invoked as a model for data rights, is built on individual data subjects exercising individual consent, and provides no purchase for a collective right held by a people. The Convention on Biological Diversity's Nagoya Protocol, adopted in 2010, made the radical move of recognising that traditional knowledge associated with genetic resources triggers benefit-sharing obligations and required parties to obtain prior informed consent of indigenous and local communities for access to such knowledge. It applies, however, narrowly to genetic resources, and operates through state mechanisms that have been uneven in their enforcement.

The instruments closest to a binding standard for the broader case are Articles 11 and 31 of the United Nations Declaration on the Rights of Indigenous Peoples, adopted in 2007. Article 31 states that indigenous peoples have the right to maintain, control, protect and develop their cultural heritage, traditional knowledge and traditional cultural expressions, including their sciences, technologies and cultures, and to maintain, control and develop their intellectual property over such heritage. Article 11 obliges states to provide redress, including restitution, for cultural, intellectual, religious and spiritual property taken without free, prior and informed consent. UNDRIP is a declaration rather than a treaty, and its implementation depends on domestic legislative will, which is precisely the weakness AI training has exploited. The World Intellectual Property Organisation's Intergovernmental Committee on Genetic Resources, Traditional Knowledge and Folklore adopted a treaty in May 2024 requiring patent applicants to disclose the country of origin of genetic resources or associated traditional knowledge underlying their application. By the standards of WIPO, an extraordinary achievement. By the standards of the AI training pipeline, a small object travelling slowly through a window already broken.

The CARE That Is Already Written

A more precise instrument exists, and it has been written by indigenous data scientists rather than by treaty negotiators. The CARE Principles for Indigenous Data Governance, released in September 2019 by the Global Indigenous Data Alliance under the International Indigenous Data Sovereignty Interest Group within the Research Data Alliance, encode a deliberately different premise from the FAIR principles that have dominated open-science discourse since 2016. FAIR asks that data be Findable, Accessible, Interoperable and Reusable. CARE asks that it serve Collective benefit, that those affected have Authority to control it, that those handling it bear Responsibility for the relationships data creates, and that the entire system be subject to indigenous Ethics.

The shift is not cosmetic. FAIR is data-oriented and asks how data can move more freely. CARE is people-oriented and asks for whose benefit, under whose authority, with what accountability, and according to whose ethics. CARE explicitly addresses the asymmetry FAIR's authors did not address: that the move to maximally open data has, in practice, accelerated the extraction of indigenous knowledge by parties with no relationship of obligation to the communities of origin. CARE is intended to be implemented in tandem with FAIR, but its operative force lies in making the openness of FAIR conditional on the consent and benefit structures of CARE.

Apply CARE to AI training data and the shape of an enforceable framework comes into focus. Collective benefit would require that indigenous communities materially benefit from any commercial use of their knowledge, with benefit defined collectively rather than as fees to individual researchers. Authority to control would require communities to be the gatekeepers of inclusion: training corpora would need community-level consent before indigenous-origin material could be incorporated, and ongoing authority to withdraw or restrict that material thereafter. Responsibility would require parties handling the data, model developers, hosting providers, downstream deployers, to take on relational obligations to communities of origin that survive the technical operation of training. Ethics would require that the protocols governing the data be the ethics of the originating community, not the standardised research ethics of the institution doing the training.

This is, on the face of it, an enormous demand. It is also, on a clear reading of UNDRIP Article 31, the existing legal demand of an instrument 144 states have already endorsed. The novelty of CARE is not the principle but the operationalisation. Te Mana Raraunga in Aotearoa New Zealand, the United States Indigenous Data Sovereignty Network, the First Nations Information Governance Centre in Canada, and Maiam nayri Wingara in Australia are already operationalising versions of this framework at the national level. None of the major foundation model providers have signed on to anything resembling it.

Free, Prior, Informed, And Currently Hypothetical

Free, prior and informed consent, abbreviated FPIC, is the operative phrase recurring across UNDRIP, the Nagoya Protocol, and the indigenous data sovereignty movement. The four words are doing a great deal of work. Free means uncoerced by economic dependency or political pressure. Prior means before the act, with enough time for genuine deliberation through the community's own decision-making processes. Informed means with full understanding of what is proposed, including downstream consequences. Consent means refusal must be a real option. In the context of AI training data, the four words are currently hypothetical. No major commercial AI system, in 2026, has obtained anything resembling FPIC for the indigenous-origin material in its training corpus.

A workable framework would need legal recognition of collective indigenous data rights in the jurisdictions hosting the largest AI providers, which means at minimum the United States, the European Union, the United Kingdom, China, and the rest of the OECD. It would need a mandatory training-data provenance disclosure regime, of the sort the EU AI Act gestures towards but does not yet rigorously implement, capable of identifying indigenous-origin material in corpora at the point of training. It would need a mechanism for community-level FPIC operating at the speed and scale of commercial AI development, likely requiring automated tooling built and governed by indigenous data sovereignty bodies rather than by model developers themselves. It would need a right of withdrawal that survives training, which technically requires either model unlearning or retraining without the withdrawn data. It would need a right to negotiate licences on community terms, and crucially the right to refuse altogether. And it would need an enforcement architecture with teeth: regulators willing to fine, courts willing to order takedowns, and procurement regimes that exclude non-compliant systems from public contracts.

None of this is technically impossible. Most of it has been written about in the indigenous data sovereignty literature for at least a decade. The reason it has not been built is not technical. It is that the parties best positioned to build it are also the parties whose business models would be most disrupted by it.

Restitution Is The Hard Part

If the framework above is the prospective question, the harder question is retrospective. What does restitution look like for knowledge already absorbed into Llama, GPT-class models, Gemini, Claude, and the rest? The honest answer is that the menu is short, technically uneven, and politically untested.

The first option is model unlearning, the technical procedure of inducing a trained model to forget specific data without retraining from scratch. The state of the art on unlearning, as of early 2026, is improving rapidly but remains contested in its guarantees. It is one thing to remove an individual user's records from a model. It is quite another to remove the contribution of a community's entire cultural archive, distributed across a vast pretraining corpus, in a way that can be verified to have actually happened. Several recent papers have shown unlearning can leave residual signal recoverable through targeted prompting. Until verifiable unlearning is robust, claims that a model has unlearned indigenous-origin data are claims of intent, not of fact.

The second option is forced retraining, in which providers retrain models without the disputed data, at very large compute cost, and absorb that cost as a condition of operation. This is technically straightforward and politically explosive. It is, however, the option most consistent with the legal logic of UNDRIP Article 11's restitution requirement. If a thing has been taken without consent and cannot be unmade in place, the thing must be unmade and remade.

The third option is compulsory licensing with back-payment to community trusts: existing models continue to operate but providers pay licensing fees, calibrated to scale of use, into trusts controlled by the communities of origin. This is the most politically tractable option and the one most likely to be adopted in any near-term framework. It has obvious shortcomings: it monetises rather than reverses the extraction, places communities in the position of accepting payment for a thing they did not agree to sell, and creates incentives for downstream model providers to argue endlessly about which knowledge counts as indigenous-origin. It also has the advantage of being implementable now.

The fourth option is community ownership stakes in the systems built on top of indigenous knowledge: equity, governance seats, audit rights. This is the most structurally ambitious option and the one most consistent with the indigenous critique that the issue is not the price but the relationship. It would require statutory innovation rather than contractual elaboration, and it would change what an AI company is in a way the industry will resist.

The fifth option is mandatory disclosure of training-data provenance, sufficient to allow communities to identify what has been included and to negotiate from that point. This is the most modest proposal and arguably the precondition for any of the other four.

The sixth option, less concrete but recurring in indigenous testimony, is a reparations fund: a pooled levy on AI providers, administered by indigenous data sovereignty bodies, used to repair the cognitive infrastructure damage the extraction has done. When a multilingual model trained on a community's language is deployed back into the community as the only available digital tool, and when its outputs encode Western assumptions in the community's own grammar, the result is a slow erosion of the community's own ways of meaning. A reparations fund would, on this view, finance indigenous-controlled language technology, indigenous-controlled knowledge management, and indigenous-controlled AI development, on the principle that the appropriate response to colonised cognitive infrastructure is to fund the building of sovereign cognitive infrastructure.

Some of these options are feasible in the near term and others are aspirational. Provenance disclosure is feasible. Compulsory licensing is feasible if political will is generated. Reparations funds are feasible at modest scale. Verified unlearning, forced retraining at scale, and community ownership stakes are aspirational. They define the horizon against which the feasible options should be judged.

The Double Bind

Underneath the legal and technical questions is a deeper one. Even if every framework above were implemented tomorrow, the extraction has happened. The training has occurred. The models exist. The deployment is global. And, increasingly, the AI systems in question are the only available technology in the communities whose knowledge made them possible. Telephony, mapping, translation, education, agricultural advisory, even spiritual chat companions, are migrating to AI substrates whose default settings encode the Western, English-language assumptions documented in the CROQ paper. The community asking an AI assistant about a medicinal plant is asking a system that was, in part, trained on its own ancestors' descriptions of that plant, refracted back through a cultural lens that is not its own.

This is the double bind. The framework that would make the extraction unlawful would not, by itself, undo it. The systems that absorbed indigenous knowledge are now being deployed as essential infrastructure in the territories of the communities they extracted from. To refuse the systems is to refuse the infrastructure. To accept the systems is to accept the colonial overlay. Indigenous AI labs, of which Lars Ailo Bongo's Sámi AI Lab at UiT The Arctic University in Norway is one of a small but growing number, are working on the third option: building indigenous-governed AI on indigenous terms, with indigenous data, for indigenous purposes. Bongo notes the people exist; the funding does not. The Microsoft-Imazon partnership in the Katukina/Kaxinawá Indigenous Reserve in Brazil's Acre state, in which agroforestry agents like Siã Shanenawa use AI tools to monitor deforestation, demonstrates that AI on indigenous terms is possible. It does not, by itself, demonstrate that the broader pipeline can be redirected.

The double bind is not resolvable by clever framework design. It is resolvable, if at all, by a long process of building parallel and sovereign cognitive infrastructure, funded in part by the proceeds of restitution from the extracting industry, in which indigenous communities exercise the right to refuse non-compliant systems and to insist on compliant ones. This is a generational project. It requires the framework be put in place now, in 2026, so that the work of building can begin under the protection of law rather than against it.

What Would Be Required

Any honest editorial position on this matter has to begin with a refusal of the comforting framing that what is needed is more research, more dialogue, more fora. The research has been done. The dialogue has been held. The fora are filled with documentation. What is missing is an enforcement architecture and the political will to install it.

Any workable framework has, at minimum, the following shape. It begins with the legal recognition, in the major AI-hosting jurisdictions, of collective indigenous data rights as a category distinct from individual data subject rights. This is statutory work. It requires legislatures, not voluntary corporate codes. The EU AI Act and the GDPR can be the basis for this in Europe, but they require explicit amendment to recognise collective subjects. In the United States, tribal sovereignty already provides a legal foundation that has been systematically underused.

It requires a mandatory provenance disclosure regime granular enough that indigenous-origin material can be identified and that communities can exercise meaningful FPIC. It requires that FPIC be obtained before training, not after, and at the level of the originating community rather than from an individual or from a state acting on behalf of the community. It requires the right of withdrawal, with a workable technical pathway for fulfilment, whether through unlearning, retraining, or operational restriction. It requires that the CARE Principles be elevated from a research community framework to a regulatory baseline. The Global Indigenous Data Alliance has done the operationalisation work; the remaining task is binding adoption.

It requires a restitution architecture for the knowledge already taken. The most realistic near-term shape is a compulsory licensing regime, with payments flowing into community-controlled trusts, combined with provenance disclosure that allows communities to identify what has been used. The more ambitious shape, which the editorial position of this article supports, is a reparations levy whose proceeds fund indigenous-governed AI infrastructure, on the principle that the appropriate response to colonised cognitive substrate is sovereign cognitive substrate.

And it requires, at the level of physical infrastructure, that data centre siting be subject to the same FPIC standard as the data inside the centres. The Sturgeon Lake Cree Nation's appeal of the Wonder Valley water licence is the test case in the Canadian context. The Seminole Nation's hyperscale moratorium is the test case in the American one. The result of these cases will indicate whether courts and regulators are prepared to apply the same logic to data centres that the Nagoya Protocol applied to bioprospecting.

The honest closing observation is that none of this will happen because the AI industry chooses it. It will happen, if it happens, because indigenous nations, environmental justice coalitions, and the regulators willing to be moved by them, force it to happen. Krystal Two Bulls and Honor the Earth are organising for that. Hindou Oumarou Ibrahim is presenting reports for it at the UN. Kate Finn and the Tallgrass Institute are working with investors who have the leverage to demand it. The Global Indigenous Data Alliance has written the operational template. The CROQ benchmark has documented the cultural bias the framework would have to correct. The Multilinguality at the Edge survey has mapped the technical landscape on which sovereign indigenous AI will have to be built. The materials are present. What is needed is the decision to use them, and the political pressure to make that decision unavoidable.

The knowledge that sustains a community's relationship with its land, its language, and its identity was never the AI industry's to take. It has been taken. The question is no longer whether that was wrong. The question is whether the framework that would prevent it from happening again, and the restitution that would begin to repair the damage already done, will be built in time to matter. The river above the data centre is still flowing. The community downstream is still there. The forum chamber is still in session. The clock is louder than any of them.

---

References & Sources

1. United Nations Department of Economic and Social Affairs. “United Nations Permanent Forum on Indigenous Issues, 25th Session.” https://social.desa.un.org/issues/indigenous-peoples/unpfii/25th-session
2. Mongabay. “AI is a double-edged sword for Indigenous stewardship, say U.N. experts.” April 2026. https://news.mongabay.com/2026/04/ai-is-a-double-edged-sword-for-indigenous-stewardship-say-u-n-experts/
3. Mongabay. “War, climate change, and AI on the agenda at this year's U.N. Indigenous forum.” April 2026. https://news.mongabay.com/2026/04/war-climate-change-and-ai-on-the-agenda-at-this-years-u-n-indigenous-forum/
4. Grist. “AI is a double-edged sword for Indigenous land protection, UN experts warn.” April 2026. https://grist.org/indigenous/ai-is-a-double-edged-sword-for-indigenous-land-protection-un-experts-warn/
5. Grist. “Indigenous land defenders are being killed, and AI is scraping their knowledge.” April 2026. https://grist.org/indigenous/indigenous-land-defenders-are-being-killed-ai-is-scraping-their-knowledge/
6. Grist. “For Indigenous communities, AI brings peril and promise.” https://grist.org/indigenous/indigenous-peoples-examine-impact-of-ai-on-communities/
7. Democracy Now! “Data Colonialism: Native Communities Fight AI Data Centers on Indigenous Land.” 22 April 2026. https://www.democracynow.org/2026/4/22/krystal_twobulls_indigenous_lands_data_centers
8. Mongabay. “AI data center revolution sucks up world's energy, water, materials.” November 2025. https://news.mongabay.com/2025/11/ai-data-center-revolution-sucks-up-worlds-energy-water-materials/
9. Mongabay. “Thai data center boom sparks fears of water shortage, air pollution.” March 2026. https://news.mongabay.com/2026/03/thai-data-center-boom-sparks-fears-of-water-shortage-air-pollution/
10. Mongabay. “The Cloud vs. drought: Water hog data centers threaten Latin America, critics say.” November 2023. https://news.mongabay.com/2023/11/the-cloud-vs-drought-water-hog-data-centers-threaten-latin-america-critics-say/
11. Miranda, Lester James V., Songbo Hu, Roi Reichart, and Anna Korhonen. “Multilinguality at the Edge: Developing Language Models for the Global South.” arXiv preprint arXiv:2604.21637, 23 April 2026. https://arxiv.org/abs/2604.21637
12. “Why are all LLMs Obsessed with Japanese Culture? On the Hidden Cultural and Regional Biases of LLMs.” arXiv preprint, 23 April 2026. https://arxiv.org/html/2604.21751
13. Global Indigenous Data Alliance. “CARE Principles for Indigenous Data Governance.” September 2019. https://www.gida-global.org/care
14. Carroll, Stephanie Russo, et al. “The CARE Principles for Indigenous Data Governance.” Data Science Journal, vol. 19, no. 43, 2020. https://datascience.codata.org/articles/10.5334/dsj-2020-043
15. Carroll, Stephanie Russo, et al. “Operationalizing the CARE and FAIR Principles for Indigenous data futures.” Scientific Data, 2021. https://www.nature.com/articles/s41597-021-00892-0
16. United Nations. “United Nations Declaration on the Rights of Indigenous Peoples.” 2007. https://www.un.org/development/desa/indigenouspeoples/wp-content/uploads/sites/19/2018/11/UNDRIP_E_web.pdf
17. Convention on Biological Diversity. “Nagoya Protocol on Access and Benefit-sharing.” Entered into force 12 October 2014. https://www.cbd.int/abs/
18. World Intellectual Property Organisation. “Intergovernmental Committee on Intellectual Property and Genetic Resources, Traditional Knowledge and Folklore.” https://www.wipo.int/tk/en/igc/
19. Canada's National Observer. “'Absolute failure': First Nation slams Alberta and Kevin O'Leary's data centre moves.” 24 April 2026. https://www.nationalobserver.com/2026/04/24/news/wonder-valley-kevin-o'leary-alberta-first-nation
20. CBC News. “Alberta First Nation voices 'grave concern' over Kevin O'Leary's proposed $70B AI data centre.” https://www.cbc.ca/news/canada/edmonton/alberta-first-nation-voices-grave-concern-over-kevin-o-leary-s-proposed-70b-ai-data-centre-1.7431550
21. ICT News. “In Indian Country, data centers come with a familiar threat of colonialism. These organizers are fighting back.” https://ictnews.org/news/in-indian-country-data-centers-come-with-a-familiar-threat-of-colonialism-these-organizers-are-fighting-back/
22. Honor the Earth. “Stop Data Colonialism Campaign.” https://www.honorearth.org/stopdatacolonialism
23. High Country News. “War, climate change and AI are at stake at the 2026 UN Indigenous forum.” April 2026. https://www.hcn.org/articles/war-climate-change-and-ai-are-at-stake-at-the-2026-un-indigenous-forum/
24. Futurism. “Tech Companies Are Using Insidious Tactics to Build Data Centers on Indigenous Lands, Activists Say.” https://futurism.com/artificial-intelligence/data-centers-tribal-communities

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

The room is unremarkable. A clean desk, a laptop, a printed sheet of arithmetic and short reading passages, and a pair of EEG electrodes resting against the scalp like patient fingers. The participant has just finished ten minutes of working through problems with the help of an AI assistant. The screen is closed. The chatbot is gone. A research assistant slides a fresh page across the desk and asks, politely, for the subject to answer the next set of questions alone.

The subject reads. The subject thinks. The subject, by every behavioural and neural measure the researchers can capture, performs measurably worse than a control participant who never touched the assistant. The effect is not subtle. It is there in the response latencies, in the error rates, in the EEG traces that show a dampened pattern of frontoparietal engagement which, ten minutes earlier, was healthy and robust.

That, in essence, is the claim of a multi-institution study widely reported across science media in April 2026, attributed to researchers from UCLA, MIT, Oxford and Carnegie Mellon, which proposes the first causal evidence that brief AI use is sufficient to produce immediate, measurable cognitive impairment in the unaided performance of equivalent tasks. The reporting has been breathless and the framing predictably apocalyptic, but the scientific stakes, if the finding survives replication, are genuinely large. Earlier work in this area had described a slow drift, a kind of boiling-frog dependency in which years of cognitive offloading might thin out a person's capacity to think for themselves. The newer claim is something different and arguably more disturbing: that the cost shows up in minutes, not years.

The distinction is not academic. If the harm is gradual, you can argue, with some plausibility, that informed adults using AI in the privacy of their own choices are merely making a long-term trade-off they are entitled to make. If the harm is acute, then the deployment of AI assistants in classrooms, clinical consulting rooms, courtrooms, contact centres and welfare offices, often without disclosure and almost always without anything resembling informed consent, looks rather different. It looks like a very large and largely unmonitored field experiment.

What does the evidence actually show? What can be defended, and what cannot? And once we are honest about both, who has the responsibility to act?

The Boiling Frog and Its Discontents

For the past three years, the dominant frame for thinking about AI and cognition has been the boiling frog, the apocryphal creature that fails to leap from a gradually heating pot. The framing made sense because the foundational evidence in cognitive science was itself longitudinal and slow.

Eleanor Maguire's work at University College London on the hippocampi of London taxi drivers, beginning in 2000, established that the brain regions used to navigate a complex city physically thicken with use. Subsequent imaging work, including a 2017 study in Nature Communications by Hugo Spiers and colleagues, suggested that turn-by-turn satnav use suppressed activity in the same hippocampal circuits. Capacity follows demand: ask the brain to navigate, and it grows the apparatus for navigation; ask it to follow instructions from a phone, and the apparatus quietens.

In 2011, Betsy Sparrow, then at Columbia, with Jenny Liu and the late Daniel Wegner of Harvard, published a paper in Science showing that participants who expected to look information up later remembered the information itself less well, but remembered where to find it. A 2024 meta-analysis in the journal Memory found the Google effect real but more modest than early coverage suggested.

Together, this literature painted a picture of slow, accumulative externalisation. Bit by bit, certain cognitive functions migrated from the wet hardware in the skull to the dry hardware in the pocket. The implicit settlement was that the costs were chronic and perhaps reversible if you put the phone down.

Generative AI complicated this picture, but for the first eighteen months of the consumer chatbot era the public discussion still defaulted to the chronic frame. Even Michael Gerlich's much-cited 2025 paper in Societies, which surveyed 666 participants and reported a strong negative correlation between AI tool use and critical thinking scores, was best read as a snapshot of ongoing erosion rather than a claim about acute injury.

Acute injury is what the newer reporting is now claiming. And acute injury, scientifically and ethically, is a different beast.

What An Acute Effect Would Have to Mean

To understand why the reported April 2026 finding has provoked the reaction it has, it is worth being precise about what an acute cognitive effect would, and would not, be. An acute effect appears rapidly after exposure and is measurable on a short timescale. A chronic exposure might gradually wear down an organ over decades; an acute exposure produces a measurable change within minutes or hours.

In the cognitive context, the equivalent claim is that ten minutes of AI-assisted maths or reading leaves a measurable footprint on the brain's ability to perform similar tasks unaided immediately after. The footprint, if it exists, is not memory loss in any everyday sense. It is more like a transient state shift, a cognitive tone that has slackened.

This is not biologically implausible. Cognitive psychology has long documented carry-over effects between tasks. Mental set, the tendency to apply a problem-solving strategy beyond its useful range, is a textbook example. So is the well-replicated finding that performing a task in a state of high external scaffolding can degrade subsequent independent performance, a phenomenon educators have long known as the assistance dilemma.

What the reported study would add is an EEG-level signal, that the brain is not merely behaving as if it has just been scaffolded but is in some quantifiable sense still in the scaffolded state, with reduced engagement in the networks that would ordinarily be doing the work. If that signal replicates, the implication is that AI use is not merely a labour-saving device whose benefits and costs balance out neatly. It is a state-altering one.

This is where the strongest existing evidence in the literature, the MIT Media Lab paper Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task, becomes essential context. Authored by Nataliya Kosmyna and seven co-authors and posted to arXiv in 2025 as preprint 2506.08872, the paper studied 54 participants in the Boston area aged 18 to 39, who wrote SAT-style essays under one of three conditions: with a large language model, with a search engine, or with no tools at all. EEG recordings during writing showed that brain-only participants exhibited the strongest, most distributed network engagement; search engine users showed moderate engagement; and LLM users displayed the weakest connectivity. Eighty-three per cent of LLM users were unable to quote from the essays they had just produced. In a fourth session, when LLM users were reassigned to brain-only writing, they continued to show weaker neural connectivity than the consistent brain-only group. The MIT authors called this carry-over cognitive debt.

The MIT preprint was not peer reviewed when it was posted, the sample was modest, and the authors themselves cautioned against the most sensational interpretations. But the basic shape of its finding, that there is a residual neural signature after the AI is taken away, is precisely the shape of the claim that the April 2026 reporting is now amplifying. The newer study, on the description circulating in the science press, extends the logic to elementary cognitive tasks rather than essay writing, and to far shorter exposures.

It is too early to know whether the April 2026 work will hold up under peer review and replication. It is not too early to ask what the world should do if it does.

Cognitive Offloading, Without Romance

The mechanism most often cited for both the chronic and acute findings is cognitive offloading: the use of external tools to reduce the demands on internal cognition. The concept predates the AI debate by decades. Writing things down is cognitive offloading. So is asking a colleague. Offloading reduces working-memory load and frees attention. Under certain conditions, it also reduces depth of processing, weakens encoding into long-term memory, and degrades the capacity to do the offloaded task without the tool.

What seems to be different about generative AI is the scope, the seamlessness and the ambient nature of the offload. A calculator does arithmetic. A search engine fetches documents. A large language model writes the paragraph, generates the answer, structures the argument and presents the result in finished form. The cognitive task it performs is not retrieval but synthesis, the very thing that, in classical accounts, is supposed to constitute the active work of thinking.

The Microsoft Research and Carnegie Mellon paper presented at CHI 2025 in Yokohama, authored by Hao-Ping Hank Lee and colleagues and based on a survey of 319 knowledge workers analysing 936 real-world AI-assisted tasks, gives this dynamic empirical shape in the workplace. The paper found that higher confidence in AI was associated with less critical thinking, while higher self-confidence in one's own abilities was associated with more critical thinking. The authors described a shift in the nature of cognitive work itself, from information gathering toward verification, from problem-solving toward integration of AI output, from doing toward supervising. They warned of what they called cognitive atrophy.

The proposed mechanism for an acute effect, then, is not mysterious. During AI-assisted work, the cognitive networks responsible for evaluating and integrating outputs remain active. The networks responsible for original generation, planning and synthesis quieten down. When the tool is taken away, the still-quietened networks do not instantly come back online. There is a lag. The lag is what the EEG picks up. The lag is what shows up in error rates and response latencies on the unaided task that follows.

This is, importantly, not a permanent change. Nothing in the existing literature suggests that ten minutes of AI use causes structural damage to the brain. The relevant concern is not about lasting injury but about state, about the cognitive tone in which the next task is begun, and how quickly that tone recovers when the scaffolding is withdrawn.

Reversibility, And What We Do Not Yet Know

Here the honest answer is that the science is at the very beginning of being able to say anything precise. The MIT preprint hints at carry-over within and across sessions, but its design does not isolate the time course of recovery. The reported April 2026 work claims acute impairment immediately after use; it does not, on the publicly available descriptions, characterise the recovery curve in detail. We have evidence of a measurable effect on the order of minutes after AI use, and we do not yet have systematic evidence about whether that effect dissipates within an hour, a day or a longer period, nor about whether repeated daily exposures produce cumulative residue.

The plausible space of outcomes is not fanciful. If the effect resolves quickly and completely after each exposure, it is roughly analogous to the post-meeting fog that anyone who has spent two hours in a video call recognises, an irritation that fades. If it resolves slowly, or if repeated exposure produces cumulative dampening, the deployment-context implications become substantial. A nurse consulting an AI scribe before a complex assessment, a teacher grading with an AI marker before lesson planning, a junior solicitor moving from AI-drafted briefs to in-court argument: all are scenarios in which acute carry-over, even if reversible, has the potential to land on the high-stakes unaided task that follows.

The claim that needs neither hyping nor dismissing is the modest one. There is evidence, from multiple research groups and instruments, that recent AI use leaves a footprint on subsequent unaided cognition. The size of that footprint, its time course, and its dependence on the type of task, the type of AI and the individual user, are all open questions.

Where The Footprint Lands

The deployment contexts in which acute carry-over would matter most are, helpfully, the same contexts in which AI is being most aggressively deployed. They are not the recreational ones. Nobody is particularly worried about the cognitive aftermath of asking a chatbot to write a wedding speech. The relevant contexts are workplaces where consequential decisions are made under time pressure, classrooms where developing minds are still acquiring the very skills that AI is offloading, healthcare settings where lapses cost lives, and public services where outcomes determine whether citizens are housed, fed, treated or detained.

Take healthcare. In the United Kingdom, AI scribes and clinical-decision-support assistants have proliferated in primary care since 2024, with the Department of Health and Social Care actively encouraging the use of approved tools to reduce administrative burden on general practitioners. The case for these tools is strong; clinician burnout is a public-health emergency in its own right, and time spent transcribing is time not spent with patients. But the consultation that follows the AI-assisted note is not a low-stakes task. It is the next patient. If the cognitive tone with which the clinician enters that next consultation is even slightly slackened by the immediately preceding offload, the relevant question is not whether the tool, on average, saves time. It is whether the unmonitored carry-over is being detected, accounted for, or even acknowledged.

In classrooms, the acute frame inverts the existing debate. The debate so far has largely been about whether students who use AI to do their homework will eventually fail to learn how to write or reason. The acute frame asks a different question: what does it mean to ask a student to use an AI in the first half of a lesson and then to demonstrate understanding in the second? If the cognitive state in which that demonstration happens is materially different from the state of a student who never used the tool, then the assessment is not measuring what it purports to measure. The Department for Education's June 2025 guidance on AI in schools acknowledged that students still needed a strong foundation in reading, writing and critical thinking to use AI effectively. The acute literature, if it stabilises, suggests the guidance does not go nearly far enough. It is not enough to know how to use the tool. The question is what happens when you put it down.

In workplaces more generally, the carry-over question intersects with the dynamic identified by Lee and colleagues at CHI 2025: workers shifting from generation to verification, from problem-solving to integration. If the verification mode itself depends on a cognitive state that is, in the moment, dampened by the just-preceding AI exposure, then verification is precisely the function being undermined. The dynamic is recursive.

In public services, the stakes are starkest. Algorithmic systems already mediate decisions about welfare entitlements, child-protection assessments, criminal-justice risk scoring and immigration triage in many jurisdictions. The case-workers operating those systems are increasingly being given AI-assisted summarisation, drafting and recommendation tools. The decisions they then make about real human lives are made, in some cases minutes after closing the assistant. Whether the cognitive tone in which those decisions are made is materially different from the tone of an unaided counterpart is not a niche concern. In the deployment contexts that matter most, it is the central one.

The Consent That Was Never Asked For

The ethical literature on technology adoption has historically operated on a strong presumption: that adults, when offered new tools, are entitled to choose to use them, and that the costs of choosing are theirs to bear. This presumption rests on a thicket of assumptions which the acute-impairment frame, if it survives, calls into question.

The first assumption is that the user is the one bearing the cost. In the workplace, that is rarely true. A nurse using an AI scribe is not the principal bearer of the risk if her cognitive tone in the next consultation is dampened. The patient is. A teacher using an AI marker is not the principal bearer if his unaided judgement in the next lesson is reduced. The student is. The deployment of AI in service contexts shifts the costs onto people who were not party to the decision and who, in many cases, do not even know the tool was used.

The second assumption is that the user has been informed. This is, in practice, almost never the case. The patient who interacts with a clinician using an AI scribe is not, in the United Kingdom or in most other jurisdictions, given any disclosure that the scribe was used, much less that recent research has suggested an acute carry-over effect on subsequent unaided cognition. The student whose teacher has just spent half an hour grading essays with an AI marker is not informed. The benefits claimant whose case-worker's notes were drafted by a generative system is not informed. There is, in most settings, no equivalent of the medical-imaging consent form, no equivalent of the data-protection notice, no equivalent of any of the layered consent infrastructures that exist for less consequential interventions in the same lives.

The third assumption is that the cognitive risk is well characterised. It is not. The literature on acute carry-over is, at the time of this writing in April 2026, weeks old in its strongest formulations and months old in its broader contours. Honest informed consent at present would have to read something like: research suggests, but has not yet established, that recent AI use may produce a transient reduction in unaided cognitive performance, the magnitude and duration of which are not yet well understood. That is not a notice that any organisation in any sector is currently required to provide.

The result is a deployment landscape in which a class of cognitive risk has been quietly normalised across millions of high-stakes interactions, on the strength of an implicit assumption that the science was either not real or not relevant, and without any of the consent infrastructure that would be required to make the deployment ethically defensible if the science turns out to be both.

Who Is Currently Responsible, And Who Currently Is Not

The question of who has the responsibility to act on the emerging evidence has, at present, no clean answer. There is a thicket of actors with partial responsibilities, and a great deal of empty space between them where the responsibility falls through.

Regulators are the obvious candidates, but their instruments are not shaped for the problem. The European Union's AI Act, which entered substantive force during 2025, classifies systems by risk and imposes obligations on developers and deployers. It does not require disclosure of cognitive carry-over effects to end-users, nor monitoring in deployment. The United Kingdom's pro-innovation framework prefers sector-specific guidance and avoids cross-cutting consent obligations. The United States, post the rescission of the Biden-era AI executive order in 2025, has effectively no federal framework at all.

Employers have a duty of care to employees and, in regulated sectors, a duty of care to clients and patients. That duty arguably already extends to an obligation to understand the cognitive risks that workplace tools might impose. The General Medical Council in the United Kingdom and equivalent professional bodies elsewhere have begun to issue guidance on AI use in clinical practice, but these documents focus overwhelmingly on data protection, accuracy and clinical accountability. They do not, at the time of writing, address acute carry-over.

Educators bear a different but related duty. The Department for Education's guidance and the curricular adjustments under way in many school systems are mostly oriented toward whether AI use degrades the development of skill over time. They do not address whether AI use during an assessment, or in the hour before one, materially changes what the assessment measures.

Platform vendors are commercially positioned to be most relevant and culturally positioned to be least. The major AI labs (OpenAI, Anthropic, Google DeepMind, Microsoft) have all published responsible-use guidance of varying depth, and have all engaged, with varying degrees of seriousness, with concerns about cognitive effects. None of them, at the time of writing, surfaces information about cognitive carry-over to end-users in the products themselves. The prevailing commercial logic, in which engagement and frequency of use are positive metrics, does not align with cognitive-risk disclosure, and there is no regulatory instrument forcing the alignment.

Individual users carry the residual responsibility, the way they carry it for every consumer product whose risks have been imperfectly disclosed. That is a thin reed to lean on in any sector where the user is, in fact, the patient or the student or the citizen rather than the operator of the tool.

The honest map of responsibility is therefore a sparse one. There is no regulator currently obliged to act, no employer currently obliged to act, no educator currently obliged to act, no platform currently obliged to act, and no user adequately positioned to act. The gaps are not bugs in the system; the system was not designed for the problem, because the problem was framed, until very recently, as chronic.

Replication, Caveats, And The Cost Of Waiting

It would be irresponsible to leave this account without flagging that the strongest version of the acute-impairment claim still rests on a small number of studies, much of it unreplicated and some of it not yet peer reviewed. The MIT preprint by Kosmyna and colleagues has the limitations its own authors acknowledged: a sample of 54 participants in a single geographic region, no peer review at the time of its initial release, and a fourth-session reassignment design that, while suggestive, is not definitive. The CHI 2025 paper by Lee and colleagues is a survey of self-reported behaviour, not a controlled experiment. The Gerlich 2025 paper in Societies is correlational and was subsequently corrected by the publisher in September 2025 for unrelated issues.

The reported April 2026 multi-institution study would be the strongest causal evidence yet, but its full methodological detail is not, at the time of writing, available for the kind of scrutiny that allows confident claims. It will need to be peer reviewed. It will need to be replicated. It will need to be tested against the standard battery of cognitive-experiment objections: demand characteristics, expectancy effects, the difficulty of isolating the AI-use intervention from time-on-task confounds, the question of whether the post-test deficit is a real cognitive change or a motivational artefact.

These caveats matter, and the article that elides them does the public no favours. They do not, however, license inaction. The asymmetry of the situation is consequential. The cost of acting on a finding that turns out to overstate its case is, mostly, the modest inconvenience of disclosure obligations that would have been good practice anyway. The cost of failing to act on a finding that turns out to be robust is the continued silent conversion of millions of high-stakes interactions into a field experiment whose subjects never agreed to participate.

The right posture, on the present evidence, is therefore neither alarm nor dismissal. It is the unfashionable posture of taking research seriously while it is still emerging, of treating disclosure and consent as prudent defaults under uncertainty, and of designing deployment contexts to be measurable, monitorable and reversible. None of these are dramatic interventions. None require believing that the strongest claims in the literature are true. They require only believing that they might be.

What The Evidence Demands, And What It Does Not

What the evidence demands is modest, and would be modest even if every study cited above were fully replicated and beyond serious dispute. It demands, first, that the deployment of AI assistants in high-stakes settings be accompanied by disclosure to the individuals whose welfare depends on the cognitive performance that follows. The patient is entitled to know that the clinician is using a scribe. The student is entitled to know that the teacher is grading with a marker. The benefits claimant is entitled to know that the case-worker has just closed a chatbot.

It demands, second, that organisations deploying these tools begin to monitor outcomes in a manner sensitive to acute carry-over. Quality-assurance audits exist; they have not, until now, been designed with the carry-over hypothesis in mind, but they could be without much trouble.

It demands, third, that regulators, professional bodies and educators begin to update their guidance with the acute frame in view, and stop treating the cognitive consequences of AI use as a problem of long-term skill development alone. It demands, fourth, that platform vendors stop pretending the question of cognitive effects is somebody else's department, and begin to surface, in their products, the relevant information that emerging research has produced.

What the evidence does not demand is panic. It does not demand that AI be removed from clinics, classrooms or public-service settings. It does not demand that workers stop using tools that, on net, help them do their jobs. It does not demand the kind of moral-panic legislation that would, if enacted on the present evidence, almost certainly do more harm than good.

What it asks of us is the harder thing: to live, as adults, in the uncomfortable middle ground where evidence is suggestive but not yet conclusive, where the costs of action are real but bounded, and where the costs of inaction are uncertain but potentially large. The history of technology regulation is mostly the history of arriving at this middle ground decades after the relevant tools have already reshaped the landscape. The unfashionable possibility, this time, is to arrive earlier.

The Smaller, Truer Claim

Strip the press coverage of its more lurid framings and what remains is a claim that is smaller and harder to dismiss. The claim is not that AI is rotting our brains. The claim is not that ten minutes of ChatGPT will leave you intellectually impaired for the rest of the day. The claim is not even that the acute effect, if it exists, is large enough to matter in the average use case.

The claim is that there is a measurable carry-over effect from recent AI use to subsequent unaided cognitive performance, that the effect appears on the order of minutes rather than years, that the existing deployment of AI in high-stakes contexts has not been designed with that effect in mind, and that the consent and disclosure infrastructure required to make that deployment ethically defensible has not been built. The reported April 2026 study strengthens the first proposition. The MIT, Microsoft, Carnegie Mellon and Swiss Business School literatures of the past eighteen months have already strengthened the second. The third is empirical and visible to anyone who looks. The fourth is a matter of public policy that we have, until now, declined to address.

The room from the opening of this article, the desk and the laptop and the EEG electrodes, is not a metaphor. It is a research site, one of a small but growing number, in which the cognitive tone of recent AI users is being measured against the cognitive tone of unaided controls. Whether the field finds the effect to be small and easily managed, or large and policy-relevant, will become clearer in the months ahead. That it is being measured at all is the first piece of good news. That the measurements are not yet, in any meaningful sense, being relayed to the patients, students, clients and citizens whose welfare depends on the unaided performance that follows the use of the tools, is the part of the situation that does not require any further evidence to fix.

The technology will not pause for the science to catch up. The disclosure can.

---

References & Sources

1. Kosmyna, N., et al. (2025). Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task. arXiv preprint arXiv:2506.08872. MIT Media Lab. https://www.media.mit.edu/publications/your-brain-on-chatgpt/
2. Lee, H.-P. H., et al. (2025). The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '25), Yokohama. Microsoft Research / Carnegie Mellon University. https://www.microsoft.com/en-us/research/publication/the-impact-of-generative-ai-on-critical-thinking-self-reported-reductions-in-cognitive-effort-and-confidence-effects-from-a-survey-of-knowledge-workers/
3. Gerlich, M. (2025). AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking. Societies, 15(1), 6. SBS Swiss Business School. https://www.mdpi.com/2075-4698/15/1/6
4. Gerlich, M. (2025). Correction: AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking. Societies, 15(9), 252. https://www.mdpi.com/2075-4698/15/9/252
5. Sparrow, B., Liu, J., & Wegner, D. M. (2011). Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips. Science, 333(6043), 776-778.
6. Maguire, E. A., et al. (2000). Navigation-related structural change in the hippocampi of taxi drivers. Proceedings of the National Academy of Sciences, 97(8), 4398-4403.
7. Javadi, A.-H., Spiers, H. J., et al. (2017). Hippocampal and prefrontal processing of network topology to simulate the future. Nature Communications, 8, 14652.
8. Department for Education (UK). (2025, June). Generative artificial intelligence (AI) in education: policy paper. https://www.gov.uk/government/publications/generative-artificial-intelligence-in-education
9. European Parliament and Council of the European Union. (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act). Official Journal of the European Union.
10. UK Department for Science, Innovation and Technology. (2023). A pro-innovation approach to AI regulation. White paper. HM Government.
11. TIME Magazine. (2025). ChatGPT's Impact On Our Brains According to an MIT Study. https://time.com/7295195/ai-chatgpt-google-learning-school/
12. WBUR News. (2025, September 16). Using ChatGPT as a homework tool? MIT researcher says think twice. https://www.wbur.org/news/2025/09/16/ai-study-essays-brain-cognition
13. Fortune. (2025, February 11). AI might already be warping our brains, leaving our judgment and critical thinking 'atrophied and unprepared,' warns new study. https://fortune.com/2025/02/11/ai-impact-brain-critical-thinking-microsoft-study/
14. Policy Options / IRPP. (2025, September). How AI is eroding human memory and critical thinking. https://policyoptions.irpp.org/2025/09/ai-memory/
15. Frontiers in Psychology. (2025). The cognitive paradox of AI in education: between enhancement and erosion. https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2025.1550621/full

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

On the morning of 18 March 2026, Deborah Leslie stood at the lectern of the Supreme Court of Georgia, in downtown Atlanta, and tried to explain why several of the cases in her brief did not exist. Leslie was an Assistant District Attorney with Clayton County, assigned to appellate work and assets forfeiture, and she had filed papers opposing a new trial for Hannah Payne, a young woman convicted in 2023 of the murder of Kenneth Herring after a hit-and-run on a Clayton County road in 2019. Payne, then twenty-five, was serving life with the possibility of parole. Her lawyer, Brian Steel, had filed for a fresh trial on the grounds that her original counsel had failed to ask the jury to consider citizen's arrest as a defence. The state's response, signed by Leslie, ran to dozens of pages. It cited authorities. The authorities, in many places, were imaginary.

Chief Justice Nels Peterson did not bury the point. From the bench, he counted aloud: at least five citations to cases that did not exist, and at least five more to cases that existed but did not say what Leslie's brief claimed they said. The video of the exchange, which would later be viewed more than five million times across various clips, has the strangled politeness of a hearing that everyone in the room knows is going badly. Leslie initially suggested the citations might have been added to the version filed with the court rather than the one she had drafted. Peterson noted that the same non-existent cases appeared in the brief opposing Payne's motion below. The implication was unavoidable. The phantom citations were hers.

A week later, on 27 March, Clayton County District Attorney Tasha Mosley wrote to the Chief Justice. The letter, published shortly after by local outlets, conceded what was already obvious. Leslie had used artificial intelligence to draft the filing. She had not verified the output. The office had moved against her: a grievance with the State Bar of Georgia, suspension, a performance improvement plan, loss of privileges. In her own affidavit, Leslie said the errors were not intentional and that the references “were not independently verified before inclusion.” The Hannah Payne appeal, a case with a victim's family, a defendant on a life sentence, and a contested constitutional argument about the right to effective counsel, had been compromised by language a model invented in a few seconds at no cost.

The Georgia incident is not anomalous. It is the latest, most public entry in a list that legal scholar Damien Charlotin, who divides his time between Sciences Po Law School and HEC Paris, has been building since April 2025 in a database he started because he could not find anyone else doing the work. By the spring of 2026, his AI Hallucination Cases tracker had passed 1,200 documented incidents from courts around the world, with roughly 800 from the United States alone. On a single day in March 2026, he logged seventeen. The rate, Charlotin has said, is still rising. What began as a curiosity in late 2022, when ChatGPT first leaked into the workflows of overworked solicitors and overconfident litigants, has become a structural feature of contemporary legal practice. The machine is in the building. The machine lies. Sometimes the lies get caught. Sometimes they do not.

Lay this fact alongside another, less visible one. According to the Legal Services Corporation's 2022 Justice Gap study, conducted with NORC at the University of Chicago, ninety-two per cent of the substantial civil legal problems experienced by low-income Americans receive no, or insufficient, legal help. Seventy-four per cent of low-income households face at least one such problem in any given year. In England and Wales, Ministry of Justice statistics for the third quarter of 2025 showed that fifty-nine per cent of civil cases in the County Court involved at least one party with no legal representation. In state civil dockets across the United States, self-representation rates routinely exceed ninety per cent in housing, family, and consumer cases. The justice gap is not a metaphor. It is the operational reality of most non-criminal courtrooms in the English-speaking world.

This is the contradiction at the heart of the moment. Generative AI is the only piece of legal infrastructure that has, in living memory, become cheaper and more widely available rather than more expensive and more rationed. For the unrepresented mother fighting an eviction, the asylum seeker filling in a witness statement at midnight, the small employer hit with a discrimination claim, a free large language model is, on its worst day, more responsive than the legal aid hotline that has not picked up in three hours and, on its best day, capable of producing a coherent draft of a defence. The same technology, deployed by a tired prosecutor in a county DA's office or a partner under deadline at a magic-circle firm, can introduce phantom precedent into the foundations of a criminal appeal. AI is simultaneously democratising access and corrupting the evidentiary substrate. There is no clean way to keep one without the other.

The Hallucination Problem, Precisely

It helps to be technical about what is happening, because the loose language around “AI mistakes” understates the issue. A large language model does not retrieve. It predicts. Given a prompt, it generates the most statistically plausible next token, then the next, conditioned on its training data and on whatever it has just produced. When the prompt is “cite a case supporting the proposition that an officer's mistaken belief in probable cause is reviewed for objective reasonableness”, the model produces something that looks like a citation, because in the training data the answers to such prompts are followed by things that look like citations. Volume number, reporter, page, year, parenthetical court abbreviation. The format is the easy part. The model has internalised the format. What it has not internalised is the existence of the case.

This is why the hallucinations are so dangerous. They are not random. They are formally correct. A fabricated case will have a plausible volume number for the reporter, a sensible district, a year that lines up with the legal doctrine being argued, and often a holding that maps onto the proposition being supported. The fabrication is grammatical. The citation, considered in isolation, is indistinguishable from a real one until someone looks it up. The Stanford RegLab's preregistered study by Varun Magesh and Faiz Surani, published in the Journal of Empirical Legal Studies, gave the phenomenon a metric: even legal-specific tools hallucinated at startling rates. Westlaw's AI-Assisted Research generated incorrect or fabricated information thirty-three per cent of the time in their tests. LexisNexis's Lexis+ AI hallucinated seventeen per cent of the time. Thomson Reuters' Ask Practical Law AI sat near the same number. Premium products. Trained on real case law. Marketed to professionals. Still inventing.

The roll call of incidents starts with Mata v Avianca, the Manhattan personal-injury suit against the Colombian airline that became the founding text of the genre. In June 2023, Judge P. Kevin Castel of the Southern District of New York imposed sanctions of $5,000 on attorneys Steven Schwartz and Peter LoDuca, and on the firm Levidow, Levidow & Oberman, after Schwartz used ChatGPT to research a brief that ended up citing six cases that did not exist: Varghese v. China South Airlines, Martinez v. Delta Airlines, Shaboon v. EgyptAir, Petersen v. Iran Air, Miller v. United Airlines, and Estate of Durden v. KLM Royal Dutch Airlines. When opposing counsel pointed out that the cases could not be found, Schwartz had asked ChatGPT whether they were real; the model assured him they were and produced fabricated full texts. He had been a member of the New York bar since 1991. “It just never occurred to me”, he testified, “that it would be making up cases.”

Then came the parade. In late 2023, Michael Cohen, the former personal lawyer to Donald Trump, sent his attorney three citations he had pulled from Google's Bard, all fabricated, in support of a motion for early termination of supervised release. The judge declined to sanction Cohen but called the episode “embarrassing and certainly negligent”. In Texas, in November 2024, Judge Marcia Crone of the Eastern District sanctioned Brandon Monk in Gauthier v. Goodyear Tire & Rubber Co. after a brief produced with the help of Anthropic's Claude cited authorities that did not exist. In June 2025, the High Court of England and Wales handed down its joined judgment in Ayinde v London Borough of Haringey and Al-Haroun v Qatar National Bank QPSC, a decision that read less like a routine ruling and more like a public warning. The grounds for review in Ayinde, drafted by a barrister called Ms Forey, misstated section 188(3) of the Housing Act 1996 and cited five non-existent cases, including a phantom “El Gendi v Camden LBC”. In Al-Haroun, a solicitor's witness statement contained eighteen authorities that did not exist, with others misquoted or inapplicable, after the solicitor relied on his client's research without verifying it. The Divisional Court was blunt: GenAI does not extinguish professional responsibility, and Rule 11 equivalents in England and Wales apply with full force regardless of whether a human or a model produced the text.

Australia has produced its own running list. On 19 July 2024, before Justice Amanda Humphreys in Victoria, a solicitor in a marital dispute submitted a list of “relevant” prior cases that turned out to have been generated by AI. He became, that year, the first Australian lawyer formally sanctioned for AI-generated fabrications. He was barred from practising as a principal and required to work under supervision for two years. In August 2025, before the Supreme Court of Victoria, defence lawyer Rishi Nathwani, KC, apologised to Justice James Elliott for filing submissions in a teenager's murder trial that included fabricated quotes from a speech to the state legislature and non-existent citations purportedly from the same court. The errors caused a twenty-four-hour delay; Elliott eventually ruled the youth not guilty of murder by reason of mental impairment, but the embarrassment to the bar was complete. In the months that followed, a Western Australian solicitor was referred to that state's regulator for tendering documents citing four cases that either did not exist or were misreferenced.

South Africa joined the parade in 2025. In Mavundla v MEC: Department of Co-Operative Government and Traditional Affairs KwaZulu-Natal, the KwaZulu-Natal High Court found that of the nine authorities Mavundla's legal team had cited, only two were real. Among the fabrications was a confidently asserted “Hassan v Coetzee”, complete with a citation, a court, a year, and a tidy doctrinal proposition, none of which corresponded to any actual case. The court referred Mavundla's lawyers to the Legal Practice Council for investigation and ordered them to bear the costs of a hearing in which an inordinate amount of judicial and counsel time had been spent searching for cases that were never going to be found. The Cliffe Dekker Hofmeyr alerts that catalogued the affair noted, drily, that good intentions and apologies were no longer mitigation. They were table stakes.

Charlotin's tracker captures the cumulative shape. The early cases were almost all lawyers. By 2025, the share of pro se litigants caught submitting fabricated citations had grown sharply; Bloomberg Law reported that at least twenty-four self-represented litigants in the United States had been hit with monetary sanctions for AI-generated filings in the eighteen months following the second half of 2023. The trend in the data is unmistakable. The technology is not going away. The hallucinations are not going away. Adoption is outpacing verification, and the courts are catching up by issuing sanctions and warnings rather than by deploying any meaningful screening.

The Other Side of the Same Coin

Now consider who else is using these systems, and why. The New York State Bar Association published a piece on 10 February 2026 by its Pro Se Advocacy interest section titled “Pro Se Advocacy in the AI Era: Benefits, Challenges, and Ethical Implications”. The article does not pretend to resolve the contradiction. It frames it. It catalogues the practical uses to which an unrepresented person might put a chatbot: drafting letters to the court, preparing a defence to a parking ticket, navigating procedural requirements that the court itself communicates through forms a non-lawyer cannot reliably parse. It also notes the obvious risk: hallucinations that look like citations, advice that looks like guidance, and a tool that the client cannot themselves audit. The piece poses the question that the legal profession has, until very recently, been allowed not to answer: “Are the people, who otherwise would not have legal counsel, better served by at least having a chatbot to assist them?”

Similar commentary has come out of South Africa and Australia in the same window. The South African Daily Maverick ran a piece in July 2025 arguing that AI hallucinations were threatening the administration of justice in the country, while simultaneously acknowledging that the country's own access-to-justice gap, particularly in family and labour matters, had created a population for whom no realistic alternative to AI-mediated self-help existed. In one widely cited case, a self-represented litigant called Mr Makunga drafted heads of argument with the help of AI tools and online research, and the presiding judge commended the quality of his submissions, noting that some members of the practising bar had filed worse arguments than the AI-augmented ones. The South African legal profession is in the position of warning the public against the same technology that is, for many of those same members of the public, the only legal-adjacent help on offer. Australian commentators have made the same point, often more sharply: that decades of cuts to legal aid have produced a country where AI is not a luxury for the poor litigant but the default.

The numbers confirm what the rhetoric implies. The 2022 Justice Gap report from the Legal Services Corporation, the federally funded body responsible for funding civil legal aid in the United States, found that ninety-two per cent of the civil legal problems faced by low-income Americans received either no help or not enough. In 2021, LSC grantees were unable to provide adequate help on roughly 1.4 million of the 1.9 million problems brought to them. Across state civil courts, the New York City Bar Association has called the gap a “chasm”. In England and Wales, the Ministry of Justice's own statistics for July to September 2025 recorded that fifty-nine per cent of County Court civil cases involved an unrepresented party. In housing matters, in family proceedings, and in claims under £10,000, the proportion is higher still. The legal profession has been priced out of the lives of the people whom the legal system most often touches.

For those people, generative AI is not a fancy productivity tool. It is the only piece of legal infrastructure that scales to their need. A free model that responds in seconds, drafts in plain English, and produces something resembling a coherent argument is more meaningful in the life of an evicted tenant than a thirty-page government leaflet, a legal aid waiting list of nine months, or a self-help kiosk staffed by a volunteer who can offer information but not advice. The bar associations know this. They are also writing the practice notes that make their members liable for AI-generated errors. The result is a regulatory regime that, on paper, treats AI as a hazardous tool that licensed professionals must approach with caution, while in practice the same tool is being used as a substitute for those professionals by people the profession does not serve.

That asymmetry is not just uncomfortable. It is dangerous. When a lawyer files a brief with phantom citations, the lawyer is sanctioned, the judge is annoyed, the client may suffer reputational damage, and the firm pays the bill. Friction is built into the relationship. The lawyer has insurance, a regulatory body, a duty of competence. When a pro se litigant files the same brief, none of those scaffolds exist. The litigant is told, sometimes for the first time in their interaction with the system, that they have submitted falsehoods to a court. Their case is dismissed, or worse. Their credibility, which they did not choose to risk, is lost. They have no insurer, no body to pay sanctions, no firm to absorb the loss. They have a chatbot and the consequences.

Risk and Where It Lands

The doctrinal answer to “who bears the risk” is easy to state and brutal to apply. In every jurisdiction that has confronted the question, the answer has been: whoever signed the filing. Rule 11 of the United States Federal Rules of Civil Procedure binds the lawyer or the unrepresented party to the truth of every assertion. The Civil Procedure Rules in England and Wales impose comparable duties. The Legal Practice Council in South Africa has already announced that good intentions are not mitigation. Australian state bars have made the same point. The doctrinal posture is that the human is the author, the AI is the tool, and the tool's errors are the author's problem.

This makes intuitive sense for the represented client and their lawyer. It is much harder to defend in the case of the unrepresented. A pro se litigant who copies a fabricated case from a chatbot has not been negligent in the way a lawyer has been negligent. The lawyer is a trained professional with a duty of competence and an obligation to know that ChatGPT does not search; the litigant is a person who can read English and has been given a search box. The same conduct, on the same facts, attracts the same legal exposure but reflects radically different fault. Sanctions imposed on a pro se litigant for AI-generated falsehoods land on someone whose alternative was not better legal advice; their alternative was no advice at all. The system tells them, in effect, that they should have known not to use the tool that the system also will not give them an alternative to.

There are emerging cases that test the edges of this rule. On 4 March 2026, Nippon Life Insurance Company of America filed suit in the Northern District of Illinois against OpenAI Foundation and OpenAI Group PBC, alleging that ChatGPT, used by an opposing pro se litigant, had engaged in tortious interference with a settled contract, abuse of process, and the unlicensed practice of law. The Nippon complaint is one of the first attempts to push a portion of the risk back upstream, onto the maker of the tool, rather than letting it rest entirely on the user. It is far from clear whether the case will survive a motion to dismiss, and the substantive merits are contested, but the move is intellectually significant. If a chatbot purports to give legal advice to a litigant, and the advice is wrong, and the litigant's reliance produces real harm to a counterparty, then liability somewhere in that chain is unavoidable. The question is whether it stops at the user, where current doctrine puts it, or extends to the model, the deployer, or the platform.

State legislatures have begun to nibble at the same question. New York legislators are considering a bill that would expressly make companies liable for the unauthorised practice of law by their AI chatbots. The premise is that a tool that confidently advises a non-lawyer on the contents of a defence is, functionally, practising law without a licence; the licensing regime exists for a reason; and the licensing regime should bind the company that deploys the tool. The counter-argument, made vigorously by the deployers, is that disclaimers are visible, that the tool is a general-purpose system, and that holding the platform liable will simply shut the tool down for the very people who most need it. The argument is real on both sides. It is also, to borrow a WIRED instinct, a debate that exists because the legal system has refused to fund civil representation at the level the population requires.

The Patchwork of Rules

What courts have done in the meantime is improvise. Judge Brantley Starr of the Northern District of Texas issued the first published standing order on AI in court filings in 2023, requiring attorneys to certify either that no portion of the filing was drafted with generative AI or that any AI-drafted portion had been independently verified by traditional means. Filings without the certificate would be stricken. Starr's order travelled. By the end of 2025, Bloomberg Law's tracker had logged hundreds of standing orders, general orders, and local rules across federal and state courts in the United States addressing AI use in submissions. The orders are not uniform. Some require disclosure of the model used. Some require certification of independent verification. Some prohibit AI in particular categories of filing. Some are silent on pro se litigants and silent in different ways on legal aid clinics that use AI in supervised work.

In the United Kingdom, the Bar Council and the Solicitors Regulation Authority have issued guidance, and the Lord Chief Justice's office has updated its own guidelines for judges on the use of AI tools. The Ayinde judgment did most of the doctrinal work: lawyers are professionally responsible for everything they sign, AI cannot be invoked as an excuse, and serious cases will be referred to the regulators. In Australia, the Victorian Legal Services Board has begun to issue conditions on practising certificates for solicitors caught with fabricated citations. The South African Legal Practice Council has confirmed that referrals for AI-generated fabrications will be standard. None of these regimes is coordinated with the others. None deals systematically with the unrepresented litigant. All of them assume that the deterrent function of sanctions is sufficient, even though the data Charlotin is collecting suggests that sanctions are not, in fact, slowing the rate of submissions.

There is a distinct strand of proposal that goes beyond after-the-fact sanction. The most robust version is the “hyperlink rule” advocated in legal-technology circles, which would require every authority cited in a filing to be backed by a working hyperlink to the actual case in a recognised public database, with verification carried out before submission. Some jurisdictions have flirted with the idea. None has imposed it as a hard rule, in part because the doctrinal infrastructure for stable case URLs is patchy and in part because requiring hyperlinks puts an additional procedural burden on litigants who already cannot navigate the existing forms. A weaker version is the retrieval-augmented generation (RAG) requirement, in which AI tools used in legal practice must ground their outputs in a curated, court-vetted database of authorities rather than in the open internet. Westlaw, LexisNexis, and Thomson Reuters all market RAG-based products. The Stanford RegLab study showed that those products still hallucinate, just less often. RAG is mitigation, not solution.

A more interesting proposal, surfacing in academic work and in the most thoughtful sections of the New York State Bar Association's pro se commentary, is a two-tier disclosure regime. Lawyers using AI face one set of rules: they must disclose, certify, and verify, and they will be sanctioned if they fail. Pro se litigants face a different set: they must disclose, but the court will treat AI-generated errors as a procedural defect that triggers an opportunity to correct, not a sanctionable falsehood, provided the litigant did not knowingly file material they suspected to be fabricated. The justification is that the unrepresented litigant has a different epistemic position. They were not supposed to know. The system that did not give them a lawyer cannot then sanction them for the only substitute available. The objection is that the rule creates a second-class evidentiary regime in which the truth of submissions depends on who made them, and that asymmetry is its own injustice.

The Hardest Cases

It is worth sitting with the kinds of cases where this matters most. The Ayinde claimant, on whose behalf phantom cases were cited, had a real housing problem. The barrister's failure did not invent the homelessness. It complicated the record on which the homelessness would be adjudicated. In Mavundla, a real dispute about traditional leadership was filed alongside fabricated authorities, and the case was referred to the Legal Practice Council in part because the court could not separate the genuine claim from the contaminated argument. In the Hannah Payne appeal, the constitutional question, whether her trial counsel had been ineffective in failing to present a citizen's arrest defence, is genuine and consequential. Leslie's hallucinated brief did not change the facts of the underlying killing. It changed the texture of the appellate record, made the prosecution's argument less credible, and forced the Georgia Supreme Court to spend its time policing the inputs rather than weighing them.

For the unrepresented, the most painful version of the problem is not the high-stakes appeal. It is the small case that was always going to be hard. A tenant in Manchester or Atlanta or Cape Town files a defence to an eviction. The defence cites cases that do not exist. The landlord's counsel files a reply that catches the fabrication. The judge, depending on jurisdiction, either strikes the defence or grants it grudging weight. The tenant loses the home that they were trying to keep, in part because the only legal help they could afford was a model that lied to them. The fault lies, on every doctrinal account, with the tenant. The injury, on any honest account, is on the tenant and the tenant's children.

That kind of case rarely makes the Charlotin database. It does not produce a published opinion. It does not generate a sanctions order. It generates a default judgment, a removal, a debt. Some portion of the court orders that are entered against unrepresented defendants in the United States and the United Kingdom in 2025 and 2026 are now, almost certainly, downstream of AI-generated filings that were never identified as such because no one in the courtroom had time or expertise to check. The dark figure of AI's contribution to the justice gap is, by definition, invisible. The cases we know about are the ones in which someone, on the other side, had the resources to look up the citations. Where there is no other side with resources, there is no audit. Where there is no audit, there is no record.

What Should Be Done

This article will not pretend that there is a clean fix. WIRED's instinct, properly, is to take a position rather than to nod sympathetically at every party. The position is this. The legal profession's current posture, in which sanctions land on the human signatory regardless of context and AI tools are treated as neutral hazards, is intellectually consistent and morally untenable. It works only if the system also funds the legal representation that would let people avoid the hazardous tool. It does not. The same legislatures and bar associations writing tighter AI rules have, for thirty years, allowed civil legal aid to be eviscerated. They cannot now have it both ways.

There are concrete steps. Court-vetted, publicly funded retrieval-augmented systems, designed specifically for unrepresented litigants in jurisdictions where self-representation is the norm, would meaningfully reduce hallucination rates and shift some risk back to public infrastructure where it belongs. The technology exists. The cost is a fraction of what jurisdictions spend on courthouse construction. The political will is the obstacle. Two-tier disclosure regimes, in which courts adopt different sanctions postures for represented and unrepresented filers, would acknowledge the moral asymmetry the current rules ignore. Mandatory hyperlinking of authorities, with court-side automated verification, would let scale solve a scale problem. Liability that extends, where appropriate, to the deployer of a model marketed as a legal assistant, would give platforms an incentive to invest in accuracy that disclaimers do not.

None of these will fix the underlying issue, which is that justice is expensive and most people cannot afford it. Generative AI has revealed that fact in a particularly acute way: the cheapest available legal counsel is also the least reliable, and the most reliable counsel has never been cheap. Treating AI as either saviour or saboteur misses the structure of the problem. The technology is a mirror. It reflects, with terrifying efficiency, both the procedural form of legal argument and the unwillingness of states to fund the substance. The Georgia prosecutor and the Manchester tenant are using the same tool for related reasons: their respective systems have not given them what they need to do their work.

Phantom precedent is what happens when a pattern-matching machine is asked to do the job of a research lawyer. It is also, more deeply, what happens when courts pretend that everyone in front of them has equal access to the means of producing reliable arguments. They do not. They have never. The arrival of AI has made that gap visible in a new way, and the next decade of legal regulation will be measured by whether courts and legislatures respond to the visibility or to the symptom. If the answer is more sanctions, more warning notices, and more standing orders, the gap will widen. If the answer is funded counsel, vetted public tools, and a doctrinal reckoning with who actually bears the risk when a model lies, there is at least a route through. The contradiction will not resolve itself. Someone has to choose.

---

References

1. CBS News Atlanta, “AI in Georgia courts raises new questions after Clayton County prosecutor admits citing fake cases”, 2026.
2. Atlanta News First, “Chief justice: Attorney cites nonexistent cases in opposing new trial for woman convicted of murder”, 23 March 2026.
3. Atlanta News First, “Attorney with Clayton County DA's Office apologizes for using AI, citing fake cases in court brief”, 30 March 2026.
4. Damien Charlotin, AI Hallucination Cases Database, damiencharlotin.com/hallucinations, accessed April 2026.
5. Eugene Volokh, “In One Day (Mar. 31), 17 U.S. Court Decisions Noting Suspected AI Hallucinations in Court Filings”, Reason, 6 April 2026.
6. New York State Bar Association, “Pro Se Advocacy in the AI Era: Benefits, Challenges, and Ethical Implications”, 10 February 2026.
7. Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023), Opinion and Order on Sanctions, Judge P. Kevin Castel, 22 June 2023.
8. NPR, “Michael Cohen says he unwittingly sent AI-generated fake legal cases to his attorney”, 30 December 2023.
9. Gauthier v. Goodyear Tire & Rubber Co., E.D. Tex., Judge Marcia Crone, sanctions order, November 2024.
10. Ayinde v London Borough of Haringey; Hamad Al-Haroun v Qatar National Bank QPSC and QNB Capital LLC, [2025] EWHC 1383 (Admin), judgment of the Divisional Court, June 2025.
11. Information Age (Australian Computer Society), “First Australian lawyer penalised for AI blunder”, 2025.
12. NBC News, “Australian lawyer sorry for AI errors in murder case, including fake quotes and made up cases”, 15 August 2025.
13. Mavundla v MEC: Department of Co-Operative Government and Traditional Affairs KwaZulu-Natal and Others, [2025] ZAKZPHC 2.
14. Cliffe Dekker Hofmeyr, “Another episode of fabricated citations, real repercussions: South African courts show no tolerance for AI-hallucinated cases”, 4 July 2025.
15. Daily Maverick, “AI 'hallucinations' are threatening the administration of justice in South Africa”, 15 July 2025.
16. Varun Magesh and Faiz Surani et al., “Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools”, Journal of Empirical Legal Studies, Stanford RegLab, 2024.
17. Legal Services Corporation, The Justice Gap: The Unmet Civil Legal Needs of Low-Income Americans, 2022 Report, NORC at the University of Chicago.
18. Ministry of Justice (England and Wales), Civil Justice Statistics Quarterly, July to September 2025.
19. New York City Bar Association, “The Justice Gap Has Become a Chasm”.
20. Bloomberg Law, “Federal Court Judicial Standing Orders on Artificial Intelligence” comparison table, 2025.
21. Legal Dive, “Federal judge seeks to prevent generative AI mistakes in briefs”, 2023, on Judge Brantley Starr's standing order.
22. Vermont Law Review, “Mandatory AI Disclosures: Enforcing A Uniform Standard”, 2025.
23. National Law Review, “Preventing Fabricated AI Legal Authorities: The Case for a Mandatory 'Hyperlink Rule'“.
24. Bloomberg Law, “Big Law Grapples With AI-Fueled Pro Se Surge, Rising Legal Costs”, 2025.
25. Georgetown Journal of Legal Ethics, “GPT, Esquire: How the Nippon Case May Shape the Future of AI in Pro Se Litigation”, 2026.

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

There is a particular kind of professional disappearance that does not show up in the unemployment figures. The illustrator still has a desk. The translator still has a website. The session musician still owns a violin. None of them have been fired. None of them have been informed, in any official capacity, that their occupation is being phased out. Their names remain on the same freelance registers, the same union rolls, the same tax filings as last year. And yet, quietly, almost imperceptibly at first, the floor underneath their work has begun to give way.

This is the strangest economic story of the decade, and it is unfolding without a moment of high drama. There are no factory closures, no mass layoffs, no town-square photographs of redundant workers carrying cardboard boxes. Instead, there is a slow, grinding compression of the rates a cover illustrator can charge for a magazine commission, a slight but stubborn drop in the volume of subtitling work coming through the agency in São Paulo, a pause in the email chain from the German publisher who used to commission a translator in Lagos every six months. Each individual moment is deniable. Taken together, they describe a structural rearrangement of the creative economy that the existing policy vocabulary, fixated on automation and job displacement, is not equipped to name.

UNESCO's flagship report on creativity and digital transformation, published on 18 February 2026 as the fourth edition of its Re|Shaping Policies for Creativity series, attempted to put a number on the rearrangement. Drawing on data from more than 120 countries, the report projected that creators worldwide are on course to lose up to 24 per cent of their revenues by 2028 as a direct consequence of generative AI, with music creators bearing the heaviest exposure and audiovisual creators close behind. An accompanying analysis published the same week by Inter Press Service amplified the geographical dimension of the problem, observing that the income losses are falling most heavily on freelance and self-employed creators in the global south, layering a new digital injury on top of long-standing inequalities in the cultural economy.

The numbers are striking, but they are not the most interesting part of the story. The most interesting part is the mechanism. This is not, in the conventional sense, a tale of automation. The translator working from Yoruba to Spanish has not been replaced by a translation engine in any specific role. She is still the translator on her own letterhead. What has happened is that the demand curve she used to live on has been displaced, almost overnight, by something that produces an approximate substitute for her output at near-zero marginal cost. The publishers, the production companies, the marketing agencies who used to commission her have not declared that they are switching to machines. They have simply stopped commissioning at the same volume, or they have begun negotiating from a position that assumes her labour competes with a free alternative. There is no transition point. There is no redundancy notice. There is no clean moment at which the policy concept of retraining becomes applicable, because the person is still doing the job. It is the job, as a paid activity, that is being hollowed out.

This distinction matters, and not only as semantics. The entire architecture of late twentieth-century labour policy, from unemployment insurance to active labour market programmes, was built on the assumption that economic dislocation comes with a clear event horizon. Someone is hired. Someone is laid off. Someone is retrained. Someone is rehired. Generative AI breaks the model not by accelerating the cycle but by detaching the harm from any of these events. The freelance writer is never laid off because she was never on a payroll. The illustrator does not get a redundancy package because there was no employer. The market she sold into has simply contracted, and the policy instruments designed to catch falling workers were not built to catch a falling market.

If the diagnosis is right, then the question that follows is the one the UNESCO report, the IPS News analysis, and a fast-growing literature on AI and the creative economy have all begun to circle. What policy instruments, beyond copyright reform, can address the harm? And who, in any meaningful sense, is responsible for the structural losses already under way?

The market, not the job, is the unit of analysis

To understand why this matters, it helps to look at what the existing debate is mostly about. Almost every legal and political fight currently being waged on behalf of creative workers concerns the upstream side of the AI value chain: whether AI labs should be allowed to train models on copyrighted works without permission, whether they should pay licensing fees for ingesting them, whether opt-out registers should be opt-in, whether the European Union's text-and-data-mining exception should narrow or expand. These are real and important fights. The Copyright Licensing Agency in the United Kingdom announced its Generative AI Training Licence to allow collective compensation for ingestion. The US Copyright Office has explored extended collective licensing on the Danish model. The Court of Justice of the European Union held its first hearing on generative AI and copyright in March 2026 in Like Company v Google, a case that may reshape the press publishers' right across the bloc.

Yet copyright, in any of its forms, only addresses one half of the harm UNESCO identified. The CISAC global economic study published in late 2024, conducted by the consultancy PMP Strategy on behalf of the international confederation of authors' societies, was unusually clear about this. The losses creators face split into two distinct streams. The first stream is the value of their existing works being scraped into training data without consent or remuneration. Copyright reform, however imperfectly, is built to address that. The second stream is the substitution effect: AI-generated outputs competing in the market against human-made works, depressing rates and shrinking commissions. Copyright, as currently understood, has very little to say about that second stream. Even a perfectly negotiated training licence does not change the fact that, once the model is trained, the marginal cost of producing a passable cover illustration falls towards zero, and the rate the illustrator can charge falls with it.

This is the harder problem, and it is the one to which the policy debate is only beginning to turn. The question is no longer simply how to compensate creators for the use of their work in training. It is how to sustain a market for human creative labour at all, when the marginal product of that labour can be approximated, however crudely, by a system that does not pay rent, sleep, or eat.

A taxonomy of instruments beyond copyright

The good news, if it can be called that, is that the policy toolkit available to address market collapse is broader than the copyright debate sometimes suggests. The bad news is that almost every instrument involves redistributing money from somebody who currently does not pay to somebody who currently does not receive, and the political economy of that redistribution is brutal.

AI cultural levies

The most direct proposal, and the one that has gained the most traction in European policy circles over the past year, is a levy on AI systems pegged to their commercial use of human cultural output. Arthur Mensch, the chief executive of the French AI lab Mistral, surprised many observers in 2025 when he publicly endorsed a revenue-based levy of roughly 1 to 1.5 per cent on commercial providers placing AI models on the European market, with funds channelled into a central pot to support cultural creation. The Mistral proposal is hardly altruistic; it would also, conveniently, harden a continental moat against American and Chinese model providers. But its underlying logic is sound, and it draws on a legal heritage that goes back six decades.

The German Copyright Act of 1965 introduced the first private copying levy, attaching a small charge to the cost of devices and media that allowed users to duplicate protected works. The principle was that where copying is structurally uncontrollable, levy-funded compensation, distributed by collective management organisations, is a more workable alternative than litigation. Generative AI presents an almost perfect analogue. The training and inference of a foundation model, at any meaningful scale, is structurally beyond the reach of one-by-one licensing. A statutory levy on commercial AI services, collected by reformed collective management organisations and distributed to creators on a metered basis, would close the substitution-side gap that copyright cannot reach. It would also avoid the worst pathology of contemporary copyright reform, which is that platforms can outspend rights holders in any line-by-line negotiation.

There are real objections. A levy must be set high enough to matter and low enough not to suppress useful applications. It must be administered by institutions trusted by both creators and developers, which is not how the existing collective management landscape is universally regarded. And it must avoid becoming a moat for incumbents who can absorb a 1.5 per cent levy more easily than a research lab in Nairobi or a co-operative model in Buenos Aires. None of these objections is fatal. All of them require institutional design rather than policy retreat.

Statutory remuneration rights

A close cousin of the levy approach is the statutory remuneration right, which decouples permission from payment. Under such a regime, AI developers might be permitted to train on lawfully accessible works without negotiating individual licences, but they would owe a non-waivable payment to authors through a collective body. The European Parliament's commissioned study on generative AI and copyright, published in 2025, examined this possibility in detail. Springer Nature's International Review of Intellectual Property and Competition Law has run a series of analyses, by scholars including Christophe Geiger, arguing that a statutory remuneration right grounded in fundamental rights to participate in cultural life could be the most workable foundation for a new compact.

The advantage of a statutory remuneration right over a pure levy is that it sits more comfortably within the existing copyright framework. The disadvantage is that it still ties payment to the use of identifiable works, which means it primarily addresses the ingestion side rather than the substitution side. Combined with a levy, however, it begins to look like a serviceable architecture.

Public commissioning and the Irish experiment

While the levy debate continues, a quieter experiment has been running in Ireland since 2022. The Basic Income for the Arts scheme, originally a three-year pilot, paid 2,000 randomly selected artists 325 euros a week, regardless of output. The Irish Department of Culture, Communications and Sport opened applications for the 2026 to 2029 successor scheme in April 2026, and a published cost-benefit analysis found that for every euro invested, society received a return of 1.39 euros, a number that has been disputed in the Irish press but has not been seriously dislodged.

The Irish scheme is not a response to AI. It was designed to address the chronic under-monetisation of cultural work in a digital economy that had already eroded the freelance commercial base before generative models arrived. But its logic transfers cleanly. If the market for creative output is being structurally compressed by a technology whose externalities are not internalised, then a state instrument that decouples income from market success becomes more, not less, defensible. A universal creative income, scaled to the working population of practising artists in any given country, would stand to working creatives roughly as agricultural support payments stand to small farmers facing global commodity competition. It is unromantic, slightly bureaucratic, and precisely the kind of thing that has historically allowed cultural production to survive market shocks.

The political objection is that it looks like a cultural welfare state. The substantive objection is that, depending on how it is administered, it can entrench the credentialing power of arts councils and reproduce existing gatekeeping. Both are genuine. Neither is decisive against an instrument that, in Ireland at least, has empirical results to its name.

Public procurement as creative-economy policy

A surprisingly underused lever is the purchasing power of governments themselves. Public sector bodies are, in aggregate, among the largest commissioners of design, illustration, translation, audiovisual production, and music in most economies. The US General Services Administration's draft AI procurement clause, the December 2025 OMB memorandum M-26-04, the United Kingdom's Procurement Policy Note 017 from February 2025, and California's executive order on AI vendor certification signed by Governor Gavin Newsom in April 2026 all introduce disclosure obligations for AI-generated content within government contracts. None of them, however, goes the further step of creating a procurement preference for human creative work in cultural production funded by public money.

A modest reform would be to require, for example, that any public broadcaster, national museum, ministry of culture, or city government commissioning creative output beyond a defined threshold use human creators where reasonably possible, with transparent disclosure when generative tools are used. This costs the state more, in the short term, than letting procurement officers chase the cheapest bid. It also creates a stable demand floor for working creatives and signals, with the kind of clarity that markets respond to, that public money will not be deployed to accelerate the collapse of the freelance creative class. India's labelling thresholds for AI-generated visual and audio content, and the EU AI Act's transparency requirements, are early sketches of the disclosure architecture this would require.

Collective bargaining as economic infrastructure

The 2023 agreements between the Writers Guild of America, the Screen Actors Guild and the Alliance of Motion Picture and Television Producers are, for all their imperfections, the most concrete demonstration that collective bargaining can produce workable rules around AI in creative work. The WGA contract specifies that AI-generated material cannot be considered literary material for credit purposes and gives writers the right to refuse to use AI tools, while preserving their ability to challenge the use of writers' work to train AI. The SAG-AFTRA contract distinguishes digital replicas of identifiable performers from synthetic performers built from no individual likeness, and creates compensation and consent obligations around both.

These provisions are not perfect. The 2024 SAG-AFTRA video game performer strike, which ran for many months over precisely these AI consent and compensation issues in the interactive sector, demonstrated how quickly a contract negotiated for one segment of the industry begins to look incomplete when applied to another. But the agreements demonstrate the principle that collective bargaining can do work that copyright cannot, by setting industry-wide floors on consent, attribution, and compensation that apply regardless of the specific upstream provenance of any given AI output.

The implication for creative workers outside the unionised entertainment sector is uncomfortable but unavoidable. The freelance illustrator, the literary translator, the independent musician, the documentary editor often have no equivalent collective body. Building one, on a national or transnational basis, becomes infrastructure rather than ideology. The European Federation of Journalists, the European Writers' Council, the International Federation of Translators, and the Concerts Promoters Association are all operating in this space, as are emerging co-operative models among illustrators in continental Europe. Any serious policy response to the structural compression of creative labour markets needs to take seriously the question of how to fund and support these bodies.

Sovereign wealth approaches and creative commons funds

A more radical proposal, occasionally floated in policy circles and yet to find a serious political champion, is the sovereign wealth approach. The argument runs that the corpus of human cultural output ingested by foundation models is a non-rival public resource analogous to a national fishery or a hydrocarbon basin. Where states extract rents from companies exploiting natural resources, the rents fund either current public expenditure or, in the Norwegian case, an intergenerational sovereign wealth fund. By analogy, a national or supranational creative commons fund, capitalised by an ingestion-based levy on commercial AI training and operation, could be invested to provide perpetual support for cultural production.

The sovereign wealth analogy is imperfect. Cultural output is not extracted from a finite reservoir; it is generated, continuously, by living people whose labour the fund is meant to compensate. But the analogy is useful precisely because it forces a recognition that the value flowing into AI labs from human cultural output is, in macroeconomic terms, an unpriced externality of historic scale. The OECD's 2025 report on intellectual property and AI training data raised, without endorsing, the question of whether the absence of pricing on this externality represents a market failure that justifies non-market correction. That is exactly the conceptual frame a sovereign wealth approach would adopt.

The global south is not a footnote

Any honest reckoning with the policy space has to confront the dimension that the IPS News analysis put squarely on the table: the income losses from generative AI are not falling evenly across geography. They are falling disproportionately on freelance and self-employed creators in the global south. UNESCO's data, repeated in the Re|Shaping Policies for Creativity report, is sobering. In developed economies, 67 per cent of people possess essential digital skills; in developing economies the figure is 28 per cent. Cultural and creative leadership in developed countries has reached 64 per cent women in some institutional categories; in developing nations it is 30 per cent. Public funding for culture sits below 0.6 per cent of global GDP and is projected to decline. Only 61 per cent of countries surveyed have intellectual property frameworks UNESCO considers adequate.

These structural baselines were already producing inequality. Generative AI compounds them. Viviana Rangel, a Colombian independent expert quoted in the IPS News analysis, framed the problem in a sentence: the region does not produce this kind of technology; it consumes it. The economic flow runs in one direction. Cultural workers in Lagos, Lima, Manila, and Karachi see their commissions evaporate as European and North American clients route work through models trained on a corpus from which their own contributions are statistically marginal. The royalties and rents from those models, when they exist at all, flow to collective management organisations and rights holders concentrated in the North Atlantic.

This dimension has implications for every instrument discussed above. A European AI cultural levy, however well designed, will tend to recapture funds from European AI providers and recycle them through European collective management organisations to predominantly European creators. That is not necessarily wrong, but it is not a global solution. The CISAC study's projection of 22 billion euros in cumulative losses to music and audiovisual creators globally over five years is a number that needs distributional analysis. Where do the losses fall? Where do the few gains fall? UNESCO's framing of the problem as a global development issue, rather than a North Atlantic intellectual property dispute, opens space for instruments that the copyright debate alone would not generate.

The most credible candidates, at this point, are international transfers built into the supranational architecture of AI governance. A share of any revenues raised through training data taxes, statutory remuneration rights, or AI cultural levies should be directed, by treaty or legislative carve-out, to a global fund supporting creators in the regions where the harm falls hardest. This is not charity. It is restitution for an extraction whose proceeds are presently retained by a small group of companies whose corpora include cultural output from every continent. UNESCO, by virtue of its mandate over cultural diversity and the global character of its 120-country reporting, is the obvious institutional vehicle, although the World Intellectual Property Organisation and the United Nations Conference on Trade and Development have credible roles to play.

The harder version of the global south argument concerns sovereignty. If a Senegalese government wants to protect its translators, illustrators, and musicians from market compression caused by foundation models trained largely outside its borders, what tools does it have? The honest answer is: not many, in the short term. National AI levies on a small market produce modest revenue. National copyright reform reaches AI labs only weakly. National public commissioning and basic income programmes are constrained by fiscal capacity. This is one reason why the architecture of any serious policy response has to be partly supranational. It is also why policy frameworks that treat the global south as an afterthought, or that solve the problem of the European illustrator while leaving the Lagos illustrator untouched, will be morally and politically unstable.

Who bears responsibility?

The question of responsibility is the one most likely to be flattened by political slogans, so it is worth taking slowly. There are at least five candidates for the moral and economic ledger, and a serious policy framework needs to assign weight to each rather than collapsing them into a single villain.

The AI labs themselves are the most obvious candidate. They built the systems whose outputs are compressing creative labour markets. They trained the models on corpora they did not pay for, in most cases, and they continue to extract economic rent from those corpora at scale. The defence offered by lab leadership tends to combine the argument that training on publicly available content is fair use with the argument that the productivity gains from foundation models will, over time, raise everyone's incomes including creators'. Both arguments are contestable. The fair use claim is being litigated across multiple jurisdictions. The productivity-spillover claim has, so far, generated almost no observable benefit for the working creators whose markets are contracting fastest. Responsibility, on any plausible reading, sits substantially with the labs, and the policy instruments above should be priced accordingly.

The platforms that distribute creative work are a second locus. Streaming services that ingest AI-generated music into the same recommendation streams as human-made music; stock image platforms that have become, in some categories, predominantly synthetic; commissioning marketplaces that allow buyers to specify AI-generated drafts as deliverables. Each of these platforms makes choices about how to label, reward, and surface human versus synthetic output. UNESCO's report observes that opaque algorithms and platform consolidation are themselves part of the structural undercutting. Procurement-style transparency requirements, content provenance standards, and labelling rules are the relevant instruments here, and platforms are properly the subjects of them.

Governments are the third candidate. They license the regulatory environment within which labs and platforms operate, and they hold the fiscal and statutory authority to introduce levies, statutory remuneration rights, public commissioning rules, and basic income schemes. They also have the slowest reflexes. The EU AI Act, the UK text-and-data-mining consultation, the patchwork of state-level AI laws in the United States, and the regulatory regimes emerging across Asia and Latin America operate on time horizons measured in years; market compression is occurring in quarters. Responsibility for that gap falls on legislatures and on public agencies that have not yet pivoted from a copyright-only frame to a market-structure frame.

The fourth candidate is the end user: the corporate or individual buyer who chooses an AI-generated cover, a synthetic voice-over, or an automated translation over a human alternative. Moral responsibility here is real but limited. Buyers respond to prices, and prices are an artefact of upstream institutional architecture; no buyer can plausibly be expected to internalise an externality the policy regime has not bothered to price. End-user weight matters most in the public sector, in editorial institutions whose readers care about provenance, and in industries where reputation rewards transparency. Disclosure rules, labelling standards, and provenance technologies make this responsibility legible and therefore actionable.

The fifth candidate, the most diffuse and the least talked about, is the public itself, conceived as the political constituency that decides whether to treat creative labour as economically valuable enough to defend. The Irish basic income scheme exists because Irish politics decided it should. The WGA and SAG-AFTRA agreements exist because audiences, in the end, did not want to consume an industry whose writers and performers were being squeezed past tolerance. The slow shift in European policy thinking towards an AI cultural levy exists because European publics and their elected representatives have, for now, not lost their attachment to the idea that cultural work is a public good worth supporting through institutional design. That political attachment is not automatic. It can erode. Where it erodes, the labs and platforms set the frame.

What a serious settlement would look like

A serious policy settlement, on the analysis above, is not a single instrument but a stack. Copyright reform sits at the bottom of the stack, addressing the upstream ingestion question that copyright is institutionally suited to handle. Statutory remuneration rights and AI cultural levies sit above it, addressing the substitution-side compression that copyright cannot reach. Public commissioning rules and procurement preferences sit above those, deploying the state's purchasing power to maintain a demand floor for human creative work. Universal creative income schemes, on the Irish model, sit above those, decoupling baseline livelihood from market success. Collective bargaining and trade-association infrastructure sits across the stack, providing the institutional capacity for creators to negotiate consent, attribution, and compensation in real-time as the technology evolves. International transfers, capitalised by the levies and routed through multilateral cultural institutions, sit on top, addressing the global south dimension that no national policy can solve alone.

No single country has the full stack today. Ireland has a basic income scheme. France and the European Union are debating a levy. The United Kingdom has a collective licensing prototype. Spain has labelling rules. Germany has a private copying levy heritage that could be retrofitted. The United States has the WGA and SAG-AFTRA agreements, the GSA procurement clause, and California's vendor certification regime. India has labelling thresholds. UNESCO has 8,100 catalogued policy measures across 120 countries. The pieces exist; what is missing is the integration.

This is, in some sense, the unromantic conclusion. The problem that generative AI has created in the creative economy is not, primarily, a problem that demands a new philosophical framework, although philosophical frameworks help. It is a problem that demands the assembly of a known set of instruments into a coherent stack, with serious institutional design and credible enforcement, and with explicit redistribution towards the creators and regions where the harm is concentrated. The political difficulty of that assembly is high. The intellectual difficulty is lower than the public debate sometimes implies.

The alternative, if no such stack is built, is reasonably easy to describe. The compression continues. The freelance creative class, in the global north and more sharply in the global south, contracts. The cultural production that survives concentrates among those with independent means, institutional employment, or audiences large enough to bypass the compression. The texture of cultural output narrows in ways that are hard to see in real time but legible in retrospect, in the same way the disappearance of regional newspapers became legible only after the fact. The labs and platforms continue to capture the rents from a corpus they did not build. Lodovico Folin-Calabi, the UNESCO director who told the press at the report's launch that the world must critically examine how these technologies are deployed and whose voices are represented, may turn out to have been describing not a turning point but a wake.

Whether the settlement gets built is, finally, a political question rather than a technical one. The technical question has answers. The political question, whether public, labs, platforms, and governments collectively decide that the structural losses already under way are worth correcting, has only the answers a generation chooses to give. The 24 per cent number is a forecast. It is also a decision, not yet made.

---

References & Sources

1. UNESCO. Re|Shaping Policies for Creativity: Making Creativity Count. Fourth global monitoring report. Launched 18 February 2026. https://www.unesco.org/en/articles/creators-face-projected-global-revenue-losses-24-2028-new-unesco-report-shows
2. UNESCO Diversity of Cultural Expressions. “Making creativity matter: UNESCO's latest Re|Shaping Policies for Creativity report calls for bold policy action.” February 2026. https://www.unesco.org/creativity/en/articles/making-creativity-matter-unescos-latest-reshaping-policies-creativity-report-calls-bold-policy
3. United Nations News. “Artists face steep income decline due to AI, UNESCO finds.” 18 February 2026. https://news.un.org/en/story/2026/02/1166989
4. Inter Press Service. “Generative AI Could Deepen Inequality, Revenue Losses in Creative Industries.” February 2026. https://www.ipsnews.net/2026/02/generative-ai-could-deepen-inequality-revenue-losses-in-creative-industries/
5. CISAC and PMP Strategy. Global economic study shows human creators' future at risk from generative AI. December 2024. https://www.cisac.org/Newsroom/news-releases/global-economic-study-shows-human-creators-future-risk-generative-ai
6. AI Business / IT Pro. “Mistral CEO Arthur Mensch calls for AI cultural levy in Europe.” 2025. https://www.itpro.com/technology/artificial-intelligence/mistral-ceo-calls-for-ai-cultural-levy
7. Copyright Licensing Agency. “CLA announces development of Generative AI Training Licence.” 2025. https://cla.co.uk/development-of-cla-generative-ai-licence/
8. United States Copyright Office. Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication report). 2025. https://www.copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-3-Generative-AI-Training-Report-Pre-Publication-Version.pdf
9. OECD. Intellectual property issues in artificial intelligence trained on scraped data. 2025. https://www.oecd.org/en/publications/intellectual-property-issues-in-artificial-intelligence-trained-on-scraped-data_d5241a23-en.html
10. Geiger, Christophe et al. “Generative AI, Digital Constitutionalism and Copyright: Towards a Statutory Remuneration Right grounded in Fundamental Rights.” Kluwer Copyright Blog and International Review of Intellectual Property and Competition Law (Springer Nature). 2025. https://link.springer.com/article/10.1007/s40319-025-01569-6
11. European Parliament. Generative AI and Copyright: Training, Creation, Regulation. Study commissioned by the JURI Committee. 2025. https://www.europarl.europa.eu/RegData/etudes/STUD/2025/774095/IUST_STU(2025)774095_EN.pdf
12. Government of Ireland, Department of Culture, Communications and Sport. “Basic Income for the Arts Scheme 2026-2029 Guidelines for Application.” April 2026. https://www.gov.ie/en/department-of-culture-communications-and-sport/publications/basic-income-for-the-arts-scheme-2026-2029-guidelines-for-application/
13. CNN Style. “Ireland is paying artists a basic income in a pioneering scheme.” 12 February 2026. https://www.cnn.com/2026/02/12/style/ireland-artists-basic-income-intl-scli
14. Perkins Coie. “Generative AI in Movies and TV: How the 2023 SAG-AFTRA and WGA Contracts Address Generative AI.” 2023. https://perkinscoie.com/insights/blog/generative-ai-movies-and-tv-how-2023-sag-aftra-and-wga-contracts-address-generative
15. Center for Democracy and Technology. “New WGA Labor Agreement Gives Hollywood Writers Important Protections in the Era of AI.” 2023. https://cdt.org/insights/new-wga-labor-agreement-gives-hollywood-writers-important-protections-in-the-era-of-ai/
16. Bird & Bird. “Like Company v Google: CJEU Holds First-Ever Hearing on Generative AI and Copyright on 10 March 2026.” 2026. https://www.twobirds.com/en/insights/2026/like-company-v-google-cjeu-holds-first-ever-hearing-on-generative-ai-and-copyright
17. Gibson Dunn. “GSA AI Procurement Rules Would Introduce New Disclosure and Use-Rights Requirements for Federal Contractors.” 2025. https://www.gibsondunn.com/gsa-ai-procurement-rules-would-introduce-new-disclosure-and-use-rights-requirements-for-federal-contractors/
18. Ropes & Gray. “Newsom Signs Executive Order Establishing AI Vendor Certification and Procurement Framework.” April 2026. https://www.ropesgray.com/en/insights/alerts/2026/04/newsom-signs-executive-order-establishing-ai-vendor-certification-and-procurement-framework
19. Euronews. “Spain could fine AI companies up to €35 million for mislabelling content.” 12 March 2025. https://www.euronews.com/next/2025/03/12/spain-could-fine-ai-companies-up-to-35-million-in-fines-for-mislabelling-content

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

The knock comes on a Tuesday, late afternoon, when the rice is still on the hob and the youngest is doing homework at the kitchen table. A caseworker in a thin coat introduces herself, asks if she can come in, and explains that the city has received a report and is required to follow up. The mother, who has lived in the same flat for nine years and has never had a child welfare investigation in her life, asks who made the report. The caseworker hesitates. It is not exactly a report, she says. It is a flag.

This is the moment, repeated thousands of times a year across American cities, when a family discovers that they have been the subject of attention they did not know was possible. The flag did not come from a neighbour or a teacher or a paediatrician. It came from a model. A risk-scoring system, fed on years of administrative data, generated a number that crossed a threshold inside a software dashboard at the local child protective services office. A screener saw the number. A supervisor signed off. A caseworker was dispatched. Somewhere along that chain, a human being had to make a final decision, but the decision was anchored, framed, and quietly shaped by an output that nobody in the home would ever see.

The mother in the flat has no right to see the score. She has no right to know which features pushed her family above the threshold. She has no right to challenge any one of those features in front of a neutral reviewer. She has no right, in any meaningful sense, to know that the algorithm exists.

That is where American child welfare sits in the spring of 2026: an expanding lattice of predictive systems, deployed inside agencies whose decisions can place a family under state surveillance and, in the worst cases, separate parents from their children, operating almost entirely outside the procedural rights that any other consequential decision in modern life would attract. A family flagged by a credit-scoring algorithm has more statutory recourse than a family flagged by a child welfare risk model. A driver flagged by a parking enforcement camera has more transparency. A tenant flagged by an algorithmic landlord screen has more legal scaffolding to push back. The state has built one of the most invasive deployments of pattern-matching in American public administration, and it has done so on top of the thinnest possible layer of due process.

The Markup Investigation and What It Found

In 2025, The Markup published an investigation into the Administration for Children's Services in New York City, the agency that handles child abuse and neglect reports for roughly 1.6 million children. The investigation, drawing on internal documents and interviews with agency staff, established that ACS had been using an algorithmic risk-scoring tool to help decide which families warranted heightened scrutiny, surveillance and investigation following a hotline call. The tool, which the agency had introduced years earlier with limited public discussion, generated a score for every family entering the system, and that score informed which cases were elevated for what staff called “high-priority” review.

The Markup's reporters, working with academic researchers, found that the system disproportionately flagged Black and low-income families at rates higher than would be expected from the underlying base rates of confirmed maltreatment in those populations. The disparity was not fully explained by the data the agency claimed to be using. There were other variables, less obvious ones, that appeared to be doing meaningful work inside the model. Postcodes. Prior contact with public assistance programmes. Density of services in a neighbourhood. Each was, on its face, a non-racial input. Each, in practice, served as a proxy for race and class, because race and class in New York are written into the geography and the administrative trail of poverty.

The agency, when contacted, defended the tool. It pointed out that the score was advisory, that humans made the final calls, that the system had been validated internally. The agency declined to release the model's full feature set. It declined to release the weights. It declined to release the technical documentation that would have allowed independent researchers to reproduce the disparity findings or to test the model on counterfactual data. Families who had been flagged by the tool, and who had then had caseworkers in their homes, had no idea that an algorithm had been involved in the decision.

The Markup investigation matters not because it was the first time anyone had documented this pattern. It matters because it landed in the largest city in the United States, in the agency that handles the largest child welfare caseload in the country, and because it confirmed that what had previously been a research finding from smaller jurisdictions was now a continental-scale phenomenon. Child welfare is being run, in part, by black-box prediction.

The Allegheny Lineage

The patient zero of the modern child welfare risk-scoring movement is the Allegheny Family Screening Tool, deployed in Allegheny County, Pennsylvania, beginning in 2016. The tool was developed by a research consortium, validated using historical case data, and integrated into the county's call-screening process. When a hotline operator received a report, the tool produced a score that estimated the likelihood that a child in the household would be removed within two years. Higher scores triggered closer review.

The Allegheny tool was, in many ways, the public face of a movement that promised to bring rigour and consistency to an area of public administration long accused of being inconsistent and biased. Its developers were not naive technocrats. They were academics with serious credentials in social welfare and statistics, and they argued, plausibly, that human screeners themselves were biased, and that an algorithm trained on the same data could at least be audited. The tool was not deployed in secret. There were public meetings, advisory committees, journalistic profiles. For a brief moment in the late 2010s, Allegheny was held up as the responsible model.

What followed was a decade of audits that complicated that picture. Independent researchers, including teams who built fairness audit frameworks specifically for child welfare contexts, found that the tool's predictions correlated with socioeconomic status in ways that were not adequately disclosed in the public materials. They found that the tool's accuracy varied by demographic group. They found that the underlying training data, which was based on historical screening and removal decisions, encoded the biases of the human system the tool was supposed to improve. If the historical data showed that Black families in Allegheny had been more likely to have their children removed for any given maltreatment report, then a model trained on that data would learn to flag Black families more aggressively, and would do so even if every explicit racial variable was stripped from the inputs.

The Allegheny defenders responded that the tool reduced overall disparity compared with unaided human screening, and there is research that supports parts of that claim. The Allegheny critics responded that “less biased than the worst-case human” is not a high enough bar to justify deployment, particularly when the tool's mechanics remained opaque to the families it scored. By the early 2020s, this argument had calcified into a sort of trench warfare in the academic literature. The county kept using the tool. Other jurisdictions copied it. New York City's ACS was one of those jurisdictions, and the tool documented by The Markup is, in effect, a descendant of the Allegheny lineage, retrained on different data and tuned to different operational thresholds.

What the April 2026 Audits Showed

On 21 April 2026, two papers appeared on arXiv that, between them, gave the most rigorous picture yet of what is actually happening inside these systems. The first was a fairness audit of institutional risk models in welfare and safeguarding contexts. The second was an analysis of algorithmic fairness in case-note-augmented prediction systems, the newer generation of tools that pull free-text narrative from caseworker notes into the feature pipeline.

The findings, taken together, are damning in a precise and technical way. Models deployed in high-stakes welfare and safeguarding contexts routinely encode socioeconomic and racial proxies even when those variables are nominally excluded from the input set. The mechanisms are not mysterious. They are documented. Postcodes function as racial proxies in segregated cities. Prior interactions with means-tested benefits encode income and, indirectly, race. Neighbourhood-level deprivation indices, which were originally designed by social scientists to identify communities in need of investment, become, when fed into a risk model, indicators that an individual family is more dangerous to its own children. Each input, considered alone, has a defensible policy rationale. Stacked, weighted and combined inside a model that was optimised to predict historical removals, they produce a system that reproduces the geography of state intervention with eerie fidelity.

The case-note paper went further. Once a model starts ingesting free-text notes from caseworkers, the proxy problem deepens, because language itself is socially stratified. A caseworker note that describes a home as “chaotic” or a parent as “uncooperative” carries weight inside an embedding model. Whether those labels were accurate, fair, or applied consistently across demographic groups is a question the model cannot answer and the deployment process rarely interrogates. Audits showed that case-note-augmented models could amplify existing disparities, because the historical record of how caseworkers described different families itself encoded assumptions about whose homes were suspect.

Both papers stopped short of saying that current child welfare risk models cannot be made fair. Both papers said, in different ways, that current child welfare risk models are not currently fair, that their unfairness is structural rather than incidental, and that the standard mitigations on offer in the technical literature, group-balanced thresholds, adversarial debiasing, fairness constraints during training, are insufficient to address proxy encoding at the depth it currently operates. To put it bluntly: the tools the field has built to make these systems fair are themselves not powerful enough to overcome the data the systems are trained on.

The Berkeley Notice Problem

In January 2026, researchers at the University of California, Berkeley published an analysis of a different but related question. Not whether these systems are fair in a statistical sense, but whether the people they scored had any meaningful idea that scoring was happening. The Berkeley analysis catalogued the deployment of algorithmic decision systems across a growing range of life-altering institutional contexts, including child welfare assessments, public benefit eligibility, criminal justice risk assessment, healthcare allocation, and tenant screening. It found that, in the overwhelming majority of cases, the affected individual had no notice that an algorithm was involved, no access to an explanation, and no formal route of appeal that engaged with the algorithmic component of the decision specifically.

This is the harder problem, and in some ways the more politically tractable one. Statistical fairness is a moving target. Technically, you can argue forever about whether a particular calibration metric or error-rate parity standard is the right one. Notice and explanation are simpler. Either the family knows that a system was used or they do not. Either there is a document explaining the inputs or there is not. Either there is a procedure for contesting the score or there is no such procedure.

The Berkeley researchers' finding, applied to child welfare, is sobering. A family flagged by a risk-scoring tool in New York or Pittsburgh or Los Angeles has, in practice, no way to know that they were flagged by a tool. The caseworker on the doorstep is not required to tell them. The investigation paperwork does not disclose it. The records request, if they know to file one, may or may not produce the score. If it does, it almost certainly will not produce the underlying feature values, the model card, or any documentation that would allow them to understand what was being weighed and how.

The information asymmetry is total, and it sits on top of an existing power asymmetry that is itself substantial. Families in the child welfare system are disproportionately poor, disproportionately non-white, and disproportionately under other forms of state observation already, including housing assistance, food assistance, public schools, and Medicaid. The institutional knowledge of how to navigate any of these systems is unevenly distributed. Add an opaque algorithmic layer on top of all of that, and the result is a population of citizens making decisions, accepting investigations, and signing service plans without knowing one of the most important inputs into the state's interest in them.

Why No Rights Yet

The legal infrastructure that might check any of this exists, in patches, in places. None of it is robust enough to do the job.

At the federal level, the closest analogue to a comprehensive algorithmic accountability statute is the patchwork of civil rights law, which prohibits disparate-impact discrimination in some federally funded programmes but has never been successfully wielded against a child welfare risk-scoring tool in the way it has against, say, mortgage-lending models. The procedural due process clause of the Fourteenth Amendment offers some protection in cases where the state seeks to terminate parental rights, but the protections kick in late in the process, well after the algorithmic flag has done its work to set events in motion. Pre-investigation flags are not adjudicated. They are operational decisions, treated as administrative discretion, and discretion is precisely what courts have historically been reluctant to second-guess.

At the state level, a handful of legislatures have passed bills requiring agencies to disclose when they are using automated decision systems, but most of these laws contain carve-outs for “decision-support” tools, and almost every child welfare risk model is officially classified as decision-support rather than as automated decision-making. The reasoning is that a human screener still signs off. The reality is that the screener is reading a score that has been generated by software, and the score functions as the primary signal in many of those decisions. The carve-out exists because vendors and agencies argued for it, and because legislators who wrote the bills did not want to be accused of weakening safeguarding by wrapping it in transparency requirements.

At the procurement level, the contracts that govern these tools are often classified as confidential commercial information. Vendors negotiate terms that prohibit agencies from disclosing the model's inner workings, on the theory that the model is intellectual property. Agencies, who frequently do not have in-house data science capacity to evaluate the tools, accept these terms because they want the tools and they cannot easily build them themselves. The result is a procurement architecture in which the state delegates a consequential public function to a private contractor, accepts secrecy as a condition of the contract, and then refuses to disclose what it has bought, on the grounds that the secrecy is the vendor's right.

The contrast with the European Union is instructive. Article 22 of the General Data Protection Regulation, in its original 2018 form, gave individuals a right not to be subject to decisions based solely on automated processing that produce legal or similarly significant effects. The right was always more limited in practice than in headline, because purely automated decisions are rare and most regulated decisions involve some human in the loop. But Article 22, paired with the wider GDPR architecture of subject access rights, transparency obligations, and data protection impact assessments, created a baseline that simply does not exist in the United States. The 2024 EU AI Act extended this baseline with risk-tiered obligations for high-risk systems, including those used in social welfare administration. The United States has nothing equivalent at the federal level. State-by-state, the strongest American statutes on automated decision-making are weaker than the floor European regulators consider unacceptable.

This is not because Americans are less concerned about state surveillance of families. It is because the American legislative process, on questions of child welfare, has a particular political shape. No politician wants to be the one who voted for the bill that allegedly weakened child protection. Every vendor of a risk model can frame transparency requirements as obstacles to keeping children safe. Every agency that uses one of these tools can claim that disclosure of the model's mechanics would teach abusive parents how to game the system. These framings are sometimes sincere and sometimes opportunistic, and both are politically effective. The result is a legislative landscape in which proposals to give families notice and challenge rights die in committee, while procurement contracts for new tools are renewed without serious public debate.

Voices in the Field

The academic literature on algorithmic harm in welfare contexts is, by 2026, large enough to constitute a small subfield. Virginia Eubanks, whose 2018 book on automated inequality remains foundational, argued that the deployment of predictive tools in welfare administration represents a new form of digital poorhouse, applying mass surveillance to the populations least able to resist it. Dorothy Roberts, whose work on the racial politics of family policing predates the algorithmic era, has long argued that the child welfare system is structurally biased against Black families and that data-driven tools, far from correcting that bias, formalise it and make it harder to contest. Rashida Richardson, who has written on algorithmic accountability and government use of predictive systems, has argued for procedural rights of notice, explanation and contestation as a baseline condition of legitimate deployment.

On the technical side, researchers like Solon Barocas have spent years documenting the mechanisms by which proxy variables encode protected attributes, and the limits of formal fairness criteria in the face of those mechanisms. Hadi Elzayn and collaborators have published audits of welfare-adjacent algorithmic systems showing, with empirical rigour, how disparate impact persists even under well-designed mitigation strategies. None of these scholars has called for a complete ban on predictive tools in welfare contexts. Most have called for a combination of structural reforms: independent audits, transparency requirements, due process rights, and a presumption that high-stakes deployments require a much higher evidentiary bar than what is currently common practice.

The interesting feature of this body of work is how unified it is on the procedural questions, even when scholars disagree on the technical questions. Whether a particular fairness metric is the right one is contested. Whether families should have a right to know that a model was used in a decision that affected them is, within this literature, essentially uncontested. The gap between the academic consensus and the operational reality of American child welfare is wide, and it is not narrowing.

What Rights Would Look Like

The shape of a meaningful rights framework for algorithmic decisions in child welfare is, at this stage, well rehearsed in policy literature. The components are not exotic.

Notice would mean that a family receiving a child welfare contact would be told, in writing, whether an algorithmic risk-scoring tool was used in the decision to investigate, and that they would be given the name and a plain-language description of the tool. This is an extremely low bar. It would not change the outcome of any individual investigation. It would simply close an information asymmetry that currently has no defensible justification.

Access would mean that the family could obtain the score that was generated for them, the inputs that fed into the score, and the documentation describing how the model translates inputs into outputs. The technical documentation already exists in most cases. It is generated as part of the procurement process. The barrier to disclosing it is contractual, not technical.

Contestation would mean that the family could challenge specific data points used in the score. This is where the model intersects with longstanding administrative law practice. Government records routinely contain errors. Some of those errors are typographical. Others are substantive. A family who has been flagged on the basis of a prior investigation that was later closed as unfounded should be able to point at that investigation and ask whether it was correctly weighted in the model. A family flagged on the basis of a postcode association should be able to ask whether that association is what is doing the work and, if so, whether the weight is justified.

Human review with authority would mean that the human in the loop is not just a person who reads the score and signs off, but a person with the institutional standing to overturn the score, the time to actually examine the inputs, and a documented record of the reasoning behind their decision. This is the most demanding component, because it requires resourcing and training that most agencies have not invested in. It is also the most consequential, because it transforms the human-in-the-loop from a procedural fig leaf into a real check.

Independent auditing would mean that agencies cannot simply self-validate their tools. They would be required to submit the tools to external technical review, including review by parties with no commercial interest in the tool's continued deployment. Audit findings would be public. Significant findings would trigger remediation requirements with deadlines.

A route of appeal would mean that there is a forum in which a family can challenge an algorithmically influenced decision and obtain meaningful relief. This is the hardest component to graft onto the existing child welfare system, because the system's procedural backbone is calibrated for a different kind of dispute. It is calibrated for fact-finding about events in a household, not for technical contestation of a model's behaviour. Building this capacity would require new staff, new training, and probably a new tier of administrative tribunal.

None of these proposals is technically novel. Each has been articulated in academic and policy literature. Each, in some form, exists in other regulatory contexts. What is missing is not the design. What is missing is the political coalition to build them in.

Why the Politics Are Stuck

The reasons no such coalition has consolidated are visible in the structure of the issue. Child protection, as a political project, runs on the premise that the state's job is to err on the side of intervention. The institutional culture of the agencies, the framing of legislative debates, and the media treatment of failures all push in one direction. When a child is harmed in a family that the system did not investigate, there are inquiries, commissions and resignations. When a family is harmed by an unjustified investigation, the story tends not to make the front page, and the family tends not to have a press office.

This asymmetry shapes how risk-scoring tools are introduced and how they are defended. The pitch to administrators is that the tool will reduce the rate of false negatives, the cases where the system missed a child who needed protection. The pitch to legislators is similar. The cost on the other side, the rate of false positives, the families subjected to investigation they did not need, is rarely treated as a comparable harm in the political conversation, even though it is a quantifiable and substantial cost in the lives of those families. The current generation of risk-scoring tools is calibrated according to thresholds chosen by agency leadership, and those thresholds are typically set conservatively in the direction of investigating more rather than fewer households.

The vendors of these tools have learned to operate within this politics. They market on the prevention of catastrophic outcomes. They underplay the operational disparities. They negotiate procurement contracts that limit disclosure. They cultivate relationships with academic researchers who can supply the legitimating veneer of validation studies. None of this is corrupt in any obvious sense. It is the normal behaviour of any commercial actor selling into a politically sensitive market with high stakes and asymmetric information. But the cumulative effect is an industry that is poorly disciplined by external oversight, because the external oversight does not have the tools to discipline it.

Affected families, meanwhile, are nearly impossible to organise. They are already under state scrutiny. They are reluctant to draw additional attention to themselves. They often do not know that other families have had similar experiences, because the information that would allow them to find each other does not flow. Civil society organisations have done significant work in this area, but they have done it at a scale that is dwarfed by the operational scale of the agencies and vendors they are trying to hold accountable.

What to Watch

The most likely vector of change in the near term is litigation. Several civil rights organisations have been preparing cases that target specific algorithmic deployments in welfare contexts, looking to establish precedent under existing civil rights and due process doctrine. The legal theory would not require a new statute. It would require a court to recognise that a family has a constitutionally cognisable interest in not being subjected to investigation on the basis of a process that they cannot contest. Whether such a case will succeed is uncertain. The doctrine is unfriendly. The factual records are hard to build. But the architecture of the litigation is plausible enough that several organisations are betting on it.

A second vector is local legislation, particularly in cities and states where the political balance is more amenable to civil liberties framings. New York, in the wake of The Markup investigation, has seen renewed legislative interest in algorithmic accountability for city agencies. Whether ACS specifically will be brought under stronger transparency rules remains to be seen. The vendors have lobbyists. The agency has institutional inertia. But the political weather, in 2026, is more favourable to disclosure than it was in 2020, and the gap between civil society capacity and vendor capacity is starting to narrow as algorithmic accountability becomes a more established advocacy field.

A third vector, and the one most aligned with the academic literature, is the construction of an external audit infrastructure. A non-governmental organisation, an academic consortium, or a hybrid public-private body with the technical capacity to audit child welfare risk-scoring tools and the legal standing to compel disclosure does not currently exist in the United States. Building one would require funding, talent, and a political settlement that recognises external audit as a legitimate function. There are precedents in other regulated industries: financial auditing, environmental impact assessment, clinical trial review. The case for an analogue in algorithmic public administration is, in the wake of the April 2026 audit findings, harder to dismiss than it once was.

The Family Comes Back

The mother in the flat does not see any of this. She sees a caseworker on a Tuesday afternoon. She answers questions she did not expect to answer. She watches her children watched by a stranger. She signs paperwork. The investigation, in her case, is closed without findings six weeks later. She is not removed from the system; she is now in it, in the database, as a household with a prior contact, a feature that may itself be ingested by the next iteration of the model the next time her name comes up.

She is told none of this. She is not told that an algorithm was involved, that her postcode contributed to the flag, that the model's developers have already been audited by independent researchers and found wanting. She is not told that the city paid a vendor several million dollars for the tool, or that the vendor's contract prohibits disclosure of the model's inner workings. She is not told that, in another country with a different legal regime, she would have had a statutory right to ask for and receive an explanation of the decision that put a stranger in her kitchen.

If American child welfare is going to have any meaningful answer to the question of what happened to her, the answer will not come from the agencies that deployed the tools or from the vendors that built them. It will come from courts willing to take procedural due process seriously when it is dressed in code, from legislators willing to pass disclosure requirements that survive vendor lobbying, and from a civil society infrastructure that does not yet exist at the scale the problem demands. The April 2026 audits, and the Berkeley analysis from earlier in the year, and the Markup investigation that preceded both, are not a complete map of the problem. They are a sufficient one. The technology is here. The harms are documented. The scaffolding of rights is a decade behind.

The next time the knock comes, the family on the other side of the door deserves, at minimum, a piece of paper that tells them what they are dealing with. That is not a radical demand. It is the floor.

References

1. The Markup. (2025). Investigation into the Administration for Children's Services algorithmic risk-scoring tool, New York City. The Markup.
2. Eubanks, V. (2018). Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. St. Martin's Press.
3. Roberts, D. E. (2022). Torn Apart: How the Child Welfare System Destroys Black Families and How Abolition Can Build a Safer World. Basic Books.
4. Chouldechova, A., Putnam-Hornstein, E., Benavides-Prado, D., Fialko, O., & Vaithianathan, R. (2018). A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. Proceedings of Machine Learning Research, 81, 134-148.
5. Vaithianathan, R., Maloney, T., Putnam-Hornstein, E., & Jiang, N. (2017). Children in the public benefit system at risk of maltreatment: Identification via predictive modeling. American Journal of Preventive Medicine, 45(3), 354-359.
6. Allegheny County Department of Human Services. (2019). Allegheny Family Screening Tool: Methodology, Version 2. Allegheny County, Pennsylvania.
7. Richardson, R., Schultz, J., & Crawford, K. (2019). Dirty data, bad predictions: How civil rights violations impact police data, predictive policing systems, and justice. New York University Law Review Online, 94, 192-233.
8. Barocas, S., & Selbst, A. D. (2016). Big data's disparate impact. California Law Review, 104, 671-732.
9. Anonymous authors. (2026, 21 April). Fairness audits of institutional risk models in welfare and safeguarding contexts. arXiv preprint.
10. Anonymous authors. (2026, 21 April). Algorithmic fairness in case-note-augmented prediction systems. arXiv preprint.
11. UC Berkeley research team. (2026, January). Notice, explanation, and appeal in life-altering algorithmic decisions: An empirical analysis. University of California, Berkeley.
12. European Parliament and Council. (2016). Regulation (EU) 2016/679 (General Data Protection Regulation), Article 22.
13. European Parliament and Council. (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act).
14. Elzayn, H., Black, E., Vossler, P., et al. (2023). Measuring and mitigating racial disparities in tax audits. Stanford Institute for Economic Policy Research, working paper.
15. Eubanks, V., & Mateescu, A. (2021). “We don't deserve this”: New app places homeless services under surveillance. Logic Magazine.
16. Hurley, D. (2018). Can an algorithm tell when kids are in danger? The New York Times Magazine, 2 January.
17. Brown, A., Chouldechova, A., Putnam-Hornstein, E., Tobin, A., & Vaithianathan, R. (2019). Toward algorithmic accountability in public services: A qualitative study of affected community perspectives. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems.
18. Saxena, D., Badillo-Urquiola, K., Wisniewski, P. J., & Guha, S. (2020). A human-centered review of algorithms used within the U.S. child welfare system. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems.
19. Ho, D. E., & Engstrom, D. F. (2020). Algorithmic accountability in the administrative state. Yale Journal on Regulation, 37, 800-854.
20. Citron, D. K., & Pasquale, F. (2014). The scored society: Due process for automated predictions. Washington Law Review, 89, 1-33.

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

In February 2026, Erin West walked through an empty scam compound in Sihanoukville. The lights, she would later tell the Stolen podcast audience, were off across most of the Cambodian coastal city that had, for the better part of a decade, served as the operational headquarters of the global romance-fraud industry. Walking the corridors of a half-abandoned dormitory building, she found the small evidence of recent occupation: bedding still bunched on the floor, foreign-language posters peeled from concrete walls, plastic wash basins stacked in the corners. She also found, scattered around the workstations, the script binders and conversation guides that the trafficked workers had been required to memorise. What she did not find, by and large, was the workers themselves. They had been moved.

West, a former Santa Clara County deputy district attorney who spent twenty-six years prosecuting tech crime before founding Operation Shamrock, has spent three years documenting an industry that, on any honest accounting, has surpassed in scale and harm the international drug trade. Pig-butchering operations alone, the long-con romance fraud that combines synthetic intimacy with sham crypto investment, generated an estimated forty-three point eight billion dollars in revenue across Burma, Cambodia and Laos in 2023, equivalent to roughly forty per cent of those countries' combined official gross domestic product, according to figures cited in testimony to the United States Congress. The industry has not been dismantled by the international crackdown of late 2025. It has migrated, restructured and, increasingly, automated.

That last word is the one this story turns on. The compounds West walked through were emptier than she expected because the work being done inside them is, slowly but visibly, being absorbed by software. The cost of buying a trafficked Vietnamese teenager, beating compliance into him with rebar, and chaining him to a workstation to run twenty-four parallel romance scripts across WhatsApp and Tinder turns out to be higher than the cost of a stack of GPUs running an open-weights large language model fine-tuned for emotional manipulation. The fraud has not gone away. It has, in the way of all industrial production, become more efficient.

And the bills have started arriving.

The Numbers That Made It a Public Crisis

In February 2026, an industry analysis circulated by Credit Union Today put the headline figure for American losses to AI-driven romance fraud at six hundred and seventy-two million dollars across the preceding year. The figure tracked closely with the FBI's own 2025 Internet Crime Complaint Center reporting, which recorded romance and confidence scams as among the highest-loss categories of cybercrime, and which for the first time in IC3's twenty-five-year history saw total internet-crime complaints crossing one million. Reported fraud losses across all categories climbed to twenty point nine billion dollars, a twenty-six per cent increase over 2024.

The British numbers tell a similar story at a smaller scale. UK Finance, the industry body that tracks payment-system fraud across British banks, reported that romance scams cost UK victims twenty point five million pounds in the first six months of 2025 alone, across nearly three thousand cases, a thirty-five per cent increase year on year. City of London Police, which leads the national police response on fraud, recorded more than one hundred and six million pounds lost to romance fraud over the 2024 to 2025 financial year. TSB Bank, in February 2026, warned that romance fraud was rising sharply across its customer base, with the average victim sending eleven separate payments, losing roughly seven and a half thousand pounds before the deception was discovered.

These numbers are, in every estimation produced by every researcher who has examined them, undercounts. AARP's Foresight 50+ Omnibus survey, conducted by NORC at the University of Chicago and published in February 2026, found that more than half of American adults who had lost money to a scam never reported it to anyone. Among those who did report, only twenty-six per cent went to law enforcement; only twenty-three per cent contacted their bank. The reason, AARP's research consistently finds, is shame. While sixty-two per cent of people view scam victims primarily as targets of a crime, sixty per cent simultaneously assume the victim was naïve or too trusting. Those two attitudes coexist in the same survey, often in the same respondent, and they are the social mechanism through which under-reporting becomes structural.

Professor Monica Whitty, head of department for software systems and cybersecurity at Monash University and the leading academic authority on romance-scam victimology, has been documenting this dynamic since the early 2010s. Her foundational research, conducted with Tom Buchanan at the University of Westminster, established what subsequent studies have only deepened: that for many romance-scam victims, the loss of the relationship is more upsetting than the financial loss, with some victims describing the experience as equivalent to bereavement. The double trauma of grief and humiliation creates a powerful reporting suppressor. The figures we have, in other words, are a fraction of the harm being done.

What Changed When the Models Arrived

To understand why the AI inflection of this fraud matters, it helps to understand what it replaced. Romance scams in their pre-2023 form were a labour-intensive business. A scammer, working from a script, could maintain perhaps twenty active relationships at once, each requiring hours of attention, each calibrated to the specific emotional vulnerabilities the victim had revealed in earlier conversations, each constrained by the operator's stamina, language ability, time-zone alignment and capacity to remember which victim had told them what. The bottleneck was human cognition. The product was synthetic intimacy, and synthetic intimacy was, like every artisanal good, expensive to produce.

The bottleneck has now been removed. A study published in the Asian Journal of Criminology in 2025, examining how organised criminal groups in Myanmar were deploying machine learning across their operations, found that large language models were already being used for open-source intelligence gathering and victim-network construction, achieving estimated time and cost reductions of ninety-six point six per cent and ninety-nine point five per cent respectively compared with manual research methods. AI-enabled scams in the same study were found to be four and a half times more profitable than their manual counterparts. The authors anticipated that LLMs would, in the medium term, replace human operators in the conversational stages of pig butchering as well.

That medium term has arrived. Experian's 2026 fraud-trends report, cited approvingly by the American Bankers Association in January, named “machine-to-machine” romance scams, in which fully autonomous bots maintain hundreds of simultaneous personalised relationships, as one of the top five fraud trends to watch. The bots, the report noted, “respond convincingly, build trust over time, and manipulate victims with precision and emotion.” Norton's 2026 dating-scam research found that nearly half of US online daters reported having been targeted by a dating scam, with seventy-four per cent of those targeted falling for at least part of it. Fox Business reported in 2026 that Tinder had begun mandating facial verification and was working with World, the iris-scanning identity protocol formerly known as Worldcoin, to prove that users were human at the point of sign-up. Trustpilot user reviews now estimate that up to eighty per cent of profiles on some major dating sites are fake or AI-generated, up from ten to fifteen per cent before the generative-AI boom.

The technical capabilities being deployed are not science fiction. They are commodity. An off-the-shelf foundation model, fine-tuned on a few thousand transcripts of successful and failed romance-scam conversations, will reliably produce text that is grammatically perfect in dozens of languages, emotionally coherent across long conversational arcs, capable of remembering the victim's daughter's birthday and the name of her late husband and the specific complaint she made about her sister three weeks ago. Combine that text generation with voice cloning, with face-swapping live-video, with the ability to generate consistent images of the same fictitious “soldier deployed in Yemen” or “oil engineer working off the coast of Angola” in any pose holding any object, and the verification heuristics human beings have evolved over decades of online dating collapse simultaneously. The “send me a selfie holding today's newspaper” test, as one industry commentator told the security press in 2026, is dead.

The Architecture of Manufactured Attachment

The reason romance fraud is so devastating, and the reason its industrial scaling is so alarming, is that it does not exploit gullibility in the casual sense the public uses the word. It exploits the human attachment system, which is to say the same neural and behavioural machinery that makes long-term love possible in the first place. Whitty's persuasive-techniques model, derived from interviews with victims and from analysis of hundreds of scam conversations, identifies a sequence of stages. The scammer establishes presence and idealisation. The scammer engineers reciprocal self-disclosure. The scammer creates a private emotional world to which only they and the victim have access. The scammer, by this point typically referred to by the victim as a partner, then introduces a crisis whose resolution requires money.

What an LLM brings to this sequence is not a new technique but radically improved consistency at every stage. Human scammers, working long shifts in a Sihanoukville compound, forget details. They confuse one victim with another. They lapse out of character when tired. They misjudge tone. The model does not. It maintains a coherent persona across thousands of conversational turns, recalls every personal detail the victim has volunteered, mirrors the victim's emotional register with sub-clinical precision, and never, ever loses patience. Researchers at the University of New South Wales reported in February 2026 that participants in experimental studies of AI-generated romance content struggled to distinguish it from authentic human writing, even when explicitly told some samples were synthetic.

The harm profile that follows from this is, predictably, severe. A 2025 qualitative study published in the journal Computers in Human Behavior Reports, examining the emotional, physiological, financial and legal consequences of online romance scams in the United States, found four major themes of harm: mental-health consequences including suicidal ideation, physiological health consequences including stress-related conditions, financial consequences ranging from depleted retirement savings to bankruptcy, and legal consequences including, in some cases, prosecution of victims who had unwittingly become money mules in the laundering of other victims' funds. A separate 2025 paper in the journal Victims and Offenders, which interviewed victim-survivors and their family members about the lived experience of cyber-scam victimisation, recorded what one participant called “falling into a black hole,” a phrase that became the paper's title.

The interaction of grief and shame is the part most laypeople find hardest to model accurately. Victims are mourning the end of what felt to them like a real relationship, often the most emotionally intimate of their adult lives, and they are simultaneously being asked by the surrounding culture to feel embarrassed about the relationship having existed at all. NBC News, in its earlier reporting on victim self-harm, and the AARP's ongoing Perfect Scam podcast, which has documented multiple cases of suicide following romance fraud, have together built a body of journalism that establishes the connection beyond reasonable doubt: romance investment scams combine the two leading proximate causes of suicide identified in the public-health literature, namely the dissolution of an intimate relationship and the threat of imminent financial ruin. In a substantial minority of cases, both arrive in the same week.

The Liability Vacuum

If a foreign criminal network has stolen six hundred and seventy-two million dollars from American citizens by exploiting the infrastructure of American technology firms and American payment networks, the natural question is who, if anyone, is supposed to bear legal responsibility for what happened. The answer in 2026, as a matter of law in both the United States and the United Kingdom, is approximately nobody, distributed across a chain of approximately nobodies, each of whom can plausibly point to the next link as the appropriate defendant.

In the American framework, dating-app and social-media platforms remain heavily protected by Section 230 of the Communications Decency Act, which immunises interactive computer services from liability for content posted by their users. In February 2025, the United States Court of Appeals for the Ninth Circuit, in an opinion that the Electronic Frontier Foundation argued was correctly decided, held that Grindr could not be held responsible under product-liability or negligence theories for matching a fifteen-year-old who had falsified his age with adult users who subsequently raped him. The Supreme Court declined to take up an appeal of a separate Grindr Section 230 case in 2024. The doctrinal direction of travel, despite years of political pressure from both major American parties, is that the platforms remain shielded from liability for the third-party content they host, including content generated by foreign scam operations.

Foundation-model providers occupy a different but comparable redoubt. The major American AI companies, OpenAI and Anthropic prominent among them, have invested heavily in safety training designed to refuse explicit requests to generate scam content. OpenAI's December 2025 Model Spec explicitly prohibits using the company's models for “targeted or scaled exclusion” or “manipulation” of human autonomy, and the joint OpenAI-Anthropic safety evaluation published in 2025 demonstrated that both providers' models are reasonably resistant to direct jailbreaks. The problem is that romance fraud does not require the model to produce overtly malicious content. It requires the model to produce loving content, persistently, in response to victim messages that do not in themselves trigger any safety classifier. The fine-tuning required to weaponise a model for fraud is generally trivial. Open-weights models, which can be modified offline and deployed without provider oversight, foreclose the safety-training argument entirely.

The dating platforms themselves operate in a marketing environment that requires them to insist they are doing everything possible against scammers, while a business environment that rewards reduced friction at sign-up. Tinder's introduction of facial verification and World iris-scan integration in 2025 represented the most aggressive position in the industry, but Bumble, Hinge and the rest of the Match Group portfolio have moved in the same direction more slowly. None of these platforms is currently obliged, under American law, to reimburse victims of fraud that began on their service.

The British position is, on paper, modestly stronger. The Online Safety Act 2023, fully in force across 2025 and into 2026, designates fraud as a “relevant offence” for which platforms must take proactive risk-reduction measures, with Ofcom empowered to fine non-compliant providers up to eighteen million pounds or ten per cent of qualifying global revenue, whichever is higher. Ofcom's enforcement guidance, published in late 2025, explicitly identifies romance scams as in scope. The legislation does not, however, create a private right of action for individual victims to sue the platforms directly, and as of April 2026 the regulator had not initiated headline enforcement against any major dating service for romance-scam failures.

Where the British regime materially diverges from the American one is in the payment rails. From October 2024, the UK Payment Systems Regulator's mandatory reimbursement scheme for authorised push-payment fraud requires all in-scope payment-service providers to reimburse victims of APP fraud up to eighty-five thousand pounds per claim, with the cost shared fifty-fifty between the sending and receiving banks. The PSR's first-year data, published in early 2026, indicated that eighty-eight per cent of money lost to APP scams in the first twelve months of the scheme had been reimbursed, with eighty-two per cent of claims closed within five business days. Vulnerable customers are exempt from the standard one-hundred-pound excess. For all the scheme's limitations, including a fifty-thousand-pound cap initially proposed and then raised to eighty-five thousand after lobbying from consumer groups, it represents the most substantial liability shift any major economy has imposed on the financial sector for the cost of fraud committed against ordinary citizens.

The United States has nothing comparable. American banks, under Regulation E and the broader patchwork of consumer-protection law, are generally not required to reimburse customers who authorised the transfer themselves, even if they were tricked into doing so. The cryptocurrency leg of pig-butchering fraud, which is the most common modality, is even less regulated. Tether, the issuer of the USDT stablecoin that is the rail of choice for Southeast Asian scam compounds, has frozen more than three hundred and forty-four million dollars in suspect funds in cooperation with the US Treasury's Office of Foreign Assets Control and other law-enforcement bodies, but the freezes are discretionary and post-hoc, and the funds, once moved through enough cycles, are practically irrecoverable.

The criminal-law track has, to be clear, not been entirely toothless. In October 2025, the United States Treasury and His Majesty's Treasury jointly imposed what officials described as the largest coordinated sanctions package ever directed at Southeast Asian cyber-scam networks, designating one hundred and forty-six individuals and entities tied to Cambodia's Prince Holding Group as a transnational criminal organisation. Federal prosecutors in the Eastern District of New York unsealed a twenty-six-page indictment against the group's chairman, Chen Zhi, and a separate court filing revealed the seizure of fifteen billion dollars in cryptocurrency identified as proceeds of crime. South Korea, Singapore and other jurisdictions added their own designations through November and December. Chen was arrested in Cambodia and extradited to China in early January 2026. The international press, briefly, described the action as a turning point. Erin West, a few weeks later, was watching new compounds being constructed deeper inside Cambodia, staffed by the same trafficked workers who had been moved out of the sanctioned facilities. The criminal-law approach is necessary. It is not, on the available evidence, sufficient.

What Proportionate Response Actually Looks Like

The temptation, when an industry has scaled at the speed and scope of AI-enabled romance fraud, is to reach for a single-bullet solution: ban the model, regulate the platform, prosecute the kingpin. None of these has, on its own, the dimensions of the problem. The proportionate response, if such a thing is constructible, has to operate at every layer of the stack at once, on the principle that any one defensive measure can be circumvented but enough of them in combination create friction sufficient to make the unit economics of fraud worse than the unit economics of legitimate business.

The first layer is liability allocation. Romance fraud is currently a negative externality of three industries: dating platforms, AI model providers and payment processors. None of those industries pays the cost. The victims pay the cost. Standard externality economics tells us what to do in this situation: tax the externality back onto the producers, either through direct civil liability or through mandatory insurance schemes. The British APP-reimbursement model, applied to the AI inflection of fraud, is a credible starting point. The American conversation, dominated by Section 230 absolutism on one side and reactive criminalisation on the other, has not yet caught up with the proposition that platforms hosting industrial-scale fraud should pay a proportional share of the bill.

The second layer is technical. Provenance and content-credential standards, including the C2PA work led by Adobe, Microsoft and the BBC, offer a partial defence against synthetic media in the dating context, though they require widespread adoption to matter. Real-time scam-pattern detection, of the sort already deployed by some major banks against APP fraud, can be extended to dating platforms with the cooperation of the major providers. Crucially, the friction-budget conversation needs to be reopened: dating apps that allow free-tier sign-up with minimal verification have made an explicit choice to externalise risk onto the user base, and that choice can be unwound. Tinder's iris-scan experiment is awkward and intrusive and probably necessary. The privacy trade-offs are real and demand serious public scrutiny. They do not, on the current evidence, weigh more heavily than the harm being done.

The third layer is the financial chokepoint. Stablecoin issuers, cryptocurrency exchanges and remittance corridors that process pig-butchering proceeds are the single most leveraged point in the system, because the criminal enterprise cannot survive without them. Tether's discretionary freezes are an admission that the issuer can act when it wants to. The policy question is whether we want a global stablecoin network whose anti-fraud actions depend on the goodwill of a private issuer in the British Virgin Islands, or whether we want enforceable standards. The same question applies, with greater obvious leverage, to the major centralised exchanges. Coinbase, Kraken and Binance know which wallet clusters are receiving pig-butchering proceeds. The friction-imposition tools are already built. They are not yet mandatory.

The fourth layer is shame reduction. This is the layer journalists tend to skip, because it is the least technical and the least amenable to the policy-speak the trade prefers, and it is also the layer on which everything else turns. As long as romance-scam victims feel too humiliated to report, the entire enforcement apparatus is operating on a fraction of the available signal. AARP's research finding that sixty per cent of survey respondents simultaneously regard victims as crime targets and as personally naïve is the public-attitudes equivalent of a software bug, and like a software bug it is fixable. Public-information campaigns that explicitly de-stigmatise victimhood, similar in their emphasis to the long-running campaigns around domestic violence and sexual assault, are the work of years rather than months, but they pay back at compound interest. Police forces that train officers to treat romance-scam reports as serious crime rather than personal embarrassment do better at recovery. Banks that deploy specialist fraud teams trained in conversational intervention, of the sort TSB has begun piloting in the UK, recover more money and rupture fewer relationships in the process.

The Position

The position this article arrives at is simple enough to state and difficult enough to enact. If artificial intelligence has industrialised what was once a manual fraud, then the regulatory and civil-liability response must be industrial in proportion. The decade since Section 230 was last seriously contested in Congress is the decade across which the model providers, the platform owners and the payment processors have built a fraud-vulnerable infrastructure whose marginal costs are borne entirely by people who did not consent to the trade-off. That arrangement is not, on any defensible reading of consumer-protection law or basic distributive ethics, sustainable.

The British APP-reimbursement scheme is the most concrete signal in the global regulatory landscape that another arrangement is possible. It is imperfect. It has been gamed at the margins. It has shifted some risk from victims to banks, where lobbyists are already pushing back. But it has also, in its first year of operation, returned eighty-eight pence on the pound of stolen money to ordinary people who would otherwise have lost it. That is the kind of measurable, replicable outcome that should be the baseline expectation of any modern fraud-response regime. The American conversation, which currently treats the platforms and model providers as effectively immune from civil liability for the harms that flow through their services, has not yet caught up.

What the AI-fuelled romance fraud crisis tells us, more clearly than any other AI-policy story of the last five years, is that the question of who pays is not separable from the question of who builds. The companies whose models can sustain hundreds of simultaneous synthetic intimacies have built a capability that, in the absence of corresponding obligations, will be used by whoever has the lowest scruples and the cheapest GPUs. The dating platforms whose business model depends on frictionless sign-up have built an attack surface that, in the absence of mandatory verification, will continue to be exploited by foreign criminal syndicates. The payment processors whose systems clear billions in cross-border transfers within seconds have built an exfiltration channel that, in the absence of mandatory holds and reimbursement, will continue to be the last leg of every successful scam.

There are real trade-offs in every direction. Mandatory verification at dating-app sign-up imposes real privacy costs and may foreclose use by people with good reasons to remain pseudonymous. Liability shifts to model providers risk slowing the deployment of useful AI capabilities into other domains. Bank-level holds on cross-border transfers will frustrate legitimate users with foreign relationships. None of these trade-offs is trivial. None of them is anywhere near as severe as the harm currently being externalised onto romance-scam victims, whose lives are being ended, financially and sometimes literally, in their tens of thousands every year.

Back to Sihanoukville

Walking through the abandoned compound in February 2026, Erin West observed something that has stayed with the people who watched her video diaries on the Operation Shamrock site. The compound was not closed in any meaningful sense. The workers had been moved further inland, away from the coastal compounds that had drawn the attention of Western investigators, into newer facilities in the Cambodian interior, where the construction was active and the access roads were difficult and the international press had not yet arrived. The infrastructure of synthetic intimacy was, like every infrastructure that has ever been targeted by international enforcement, relocating.

The point of the relocation, from the operators' perspective, is that the work has become much easier to relocate. A workforce of trafficked humans is heavy, slow, expensive to move and dangerous to move because moved humans speak to journalists. A workforce of language models is none of those things. It can be replicated across server farms in jurisdictions with weak enforcement, fed translated scripts in any of fifty languages, and pointed at victims selected from leaked dating-site databases or social-media data brokers. The compound in Sihanoukville with its bunched bedding and abandoned wash basins is, in retrospect, an artefact of the human-labour era of romance fraud. The coming era will leave fewer artefacts, because there will be fewer humans to leave them.

What that era requires of the rest of us is the construction of a regulatory and legal infrastructure that does not depend on the existence of a physically locatable compound full of trafficking victims. The compounds were politically galvanising, but they were also a crutch. They allowed Western policymakers to treat romance fraud as an offshore problem that would yield to offshore enforcement. The model-driven version of the same fraud will yield only to onshore measures: civil liability for the platforms and the payment rails, mandatory provenance and verification standards for the dating services, reimbursement schemes for the victims, public-information campaigns that strip the shame off reporting. None of those measures is novel. All of them are tractable. The reason they have not been built yet is not technical and not even economic. It is that the people whose retirement savings are being extracted, conversation by tender conversation, by language models running in unmarked data centres, do not yet have lobbyists who can match the influence of the companies whose infrastructure is being used against them.

That, in the end, is the actual story of the six hundred and seventy-two million dollars and the twenty point five million pounds and the unknowably larger sums that will never be reported. It is not a story about AI alignment, or about jailbreaks, or even about the trafficking compounds, which are themselves only the most visible symptom of the deeper architecture of harm. It is a story about who pays, and the answer the current system gives is the wrong one. The bills are arriving in the post, in the bank statements, in the unanswered phone calls from a partner who does not exist, and the response so far, from the institutions whose infrastructure made the fraud possible, has been a press release. It is not enough. It will not be enough until the cost of building these systems carelessly is finally borne, in cash and in court, by the people who built them.

The compound lights are off in Sihanoukville. The models are warming up everywhere else.

---

References

1. CU Today. “Love, Lies And $672 Million Lost: Inside The AI-Fueled Romance Scam Explosion.” CU Today, February 2026. https://www.cutoday.info/THE-feature/Love-Lies-And-672-Million-Lost-Inside-The-AI-Fueled-Romance-Scam-Explosion
2. Federal Bureau of Investigation, Internet Crime Complaint Center. “FBI IC3 2025 Internet Crime Report.” FBI, 2026. Summary via SOCRadar, https://socradar.io/blog/fbi-ic3-2025-internet-crime-report-10-takeaways/ and AARP, https://www.aarp.org/money/scams-fraud/fbi-ftc-report-2025-losses/
3. UK Finance. “Over £600 million stolen by fraudsters in first half of 2025.” UK Finance Press Release, 2025. https://www.ukfinance.org.uk/news-and-insight/press-release/over-ps600-million-stolen-fraudsters-in-first-half-2025
4. City of London Police. “A wrong turn on Love Lane: City of London Police take over city streets to warn of the dangers of romance fraud, with more than £106 million lost in the last year.” City of London Police, June 2025. https://www.cityoflondon.police.uk/news/city-of-london/news/2025/june/a-wrong-turn-on-love-lane-city-of-london-police-take-over-city-streets-to-warn-of-the-dangers-of-romance-fraud-with-more-than-106-million-lost-in-the-last-year/
5. TSB Bank / Fintech Global. “TSB warns romance fraud rising sharply across social media.” Fintech Global, 6 February 2026. https://fintech.global/2026/02/06/tsb-warns-romance-fraud-rising-sharply-across-social-media/
6. AARP. “Romance Scams Put Millions of Adults 50-plus at Risk: Here is What You Need to Know.” AARP Press Release, 3 February 2026. https://www.aarp.org/press/releases/2026-02-03-Romance-Scams-2026.html
7. Whitty, Monica T.; Buchanan, Tom. “The online dating romance scam: The psychological impact on victims, both financial and non-financial.” Criminology and Criminal Justice, 2016. https://journals.sagepub.com/doi/10.1177/1748895815603773
8. Operation Shamrock. “I Walked Into an Abandoned Scam Compound.” Operation Shamrock, 2026. https://www.operationshamrock.org/blog/erin-west-cambodia-inside-scam-compound
9. West, Erin. Written Testimony for the United States House Committee on Financial Services, REACT Task Force, September 2024. https://docs.house.gov/meetings/BA/BA10/20240918/117659/HHRG-118-BA10-Wstate-WestE-20240918.pdf
10. Springer Nature. “An Artificial Intelligence-Driven Temporal Network Analysis of Myanmar's Cyber Scam Ecosystem.” Asian Journal of Criminology, 2025. https://link.springer.com/article/10.1007/s11417-025-09474-0
11. American Bankers Association Banking Journal. “AI romance, 'machine-to-machine' scams among top 2026 fraud trends.” ABA Banking Journal, January 2026. https://bankingjournal.aba.com/2026/01/ai-romance-machine-to-machine-scams-among-top-2026-fraud-trends/
12. Norton. “What Norton's Insights Reveal About AI Dating Scams (2026).” Norton, 2026. https://us.norton.com/blog/online-scams/ai-online-dating-scams
13. Fox Business. “Tinder turns to eye scans to weed out bots amid rising concerns over romance scams.” Fox Business, 2026. https://www.foxbusiness.com/media/tinder-turns-eye-scans-weed-out-bots-amid-rising-concerns-over-romance-scams
14. UK Government. “Online Safety Act: Explainer.” GOV.UK, 2024. https://www.gov.uk/government/publications/online-safety-act-explainer/online-safety-act-explainer
15. Payment Systems Regulator. “APP scams reimbursement dashboard for Q3 2025.” PSR, 2025. https://www.psr.org.uk/information-for-consumers/app-scams-reimbursement-dashboard/
16. Electronic Frontier Foundation. “Ninth Circuit Correctly Rules That Dating App Isn't Liable for Matching Users.” EFF Deeplinks, February 2025. https://www.eff.org/deeplinks/2025/02/ninth-circuit-correctly-rules-dating-app-isnt-liable-matching-users
17. OpenAI. “Model Spec.” OpenAI, 18 December 2025. https://model-spec.openai.com/2025-12-18.html
18. OpenAI. “Findings from a pilot Anthropic-OpenAI alignment evaluation exercise: OpenAI Safety Tests.” OpenAI, 2025. https://openai.com/index/openai-anthropic-safety-evaluation/
19. United States Department of the Treasury. “U.S. and U.K. Take Largest Action Ever Targeting Cybercriminal Networks in Southeast Asia.” Treasury Press Release, October 2025. https://home.treasury.gov/news/press-releases/sb0278
20. Crypto.news. “Tether freezes $344M in USDT tied to sanctions evasion, pig-butchering scams.” Crypto.news, 2025. https://crypto.news/tether-freezes-344m-in-usdt-tied-to-sanctions-evasion-pig-butchering-scams/
21. NBC News. “How online romance scams are netting millions of dollars and pushing victims to self-harm.” NBC News. https://www.nbcnews.com/tech/tech-news/how-online-romance-scams-netting-millions-self-harm-rcna85252
22. ScienceDirect. “A qualitative investigation of the emotional, physiological, financial, and legal consequences of online romance scams in the United States.” Computers in Human Behavior Reports, 2024. https://www.sciencedirect.com/science/article/pii/S2949791424000605
23. Taylor and Francis Online. “Falling into a Black Hole: A Qualitative Exploration of the Lived Experiences of Cyberscam Victim-Survivors and Their Social Support Networks.” Victims and Offenders, 2025. https://www.tandfonline.com/doi/full/10.1080/15564886.2025.2481267
24. University of New South Wales. “Valentine's Day is coming: How vulnerable are you to AI-enabled romance scams?” UNSW Newsroom, February 2026. https://www.unsw.edu.au/newsroom/news/2026/02/valentine-s-day-is-coming--how-vulnerable-are-you-to-ai-scams
25. Coalition for Content Provenance and Authenticity. “C2PA Specification.” C2PA, 2024. https://c2pa.org/specifications/specifications/2.0/index.html

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

John Steinbach opened the envelope at his kitchen table in Manassas, Virginia, and stared at a number that did not make sense. His January 2026 electricity bill came in at $281. The month before, he had paid something close to $100, the sort of bill he had been paying for years. He checked the decimal point. He read the meter reading twice. He was not running a grow-op in his garage. He had not left the heater jammed on. He was, as far as he could tell, living the same quiet suburban life he had always lived, in a house some twenty miles from the centre of what the industry calls Data Center Alley, the densest cluster of server farms on planet Earth.

“It's just so far beyond any bill that I've ever had,” he told Consumer Reports, in an investigation published in March 2026 that has become, in the months since, something close to a Rosetta Stone for understanding the strange new arithmetic of American household utilities. His bill had not tripled because he was using triple the electricity. It had tripled because of the vast silicon and concrete apparatus humming away in warehouses up the highway, the apparatus that now draws roughly 40 per cent of Virginia's electricity, the apparatus that is training and running the artificial intelligence models the rest of the world uses to summarise emails and generate images of astronauts on horseback.

The bill arrived, in other words, because somewhere in the invisible plumbing of the grid, a decision had been made about whose electricity this was, who would pay for the transmission upgrades required to keep the GPUs cool and the data centres fed, and who would absorb the shock when demand outstripped supply. That decision was not made by John Steinbach. It was not made by his neighbours. It was not made by anyone on his street, his block, or his county council. And that, more than the $181 increase, is the story.

A Gold Rush With Someone Else's Gold

The United States is currently in the middle of the largest private infrastructure build-out since the interstate highway system, and almost nobody who lives in its path has been asked to approve any part of it. According to Fortune, data centres now account for half of all new American electricity demand. According to Consumer Reports, residential electricity prices rose 7.1 per cent in 2025 alone, and in the Mid-Atlantic corridor served by the PJM grid, the wholesale auction that sets capacity prices reached record highs, resulting in average bill increases of more than $17 a month for Baltimore customers of Exelon's BGE utility, a figure that makes Steinbach's triple-digit leap look almost modest by comparison.

Behind these averages sits a deceptively simple question: when a hyperscaler, say, Amazon or Microsoft or Google or Meta, builds a two-gigawatt campus in a rural county, who pays for the substations, the transmission lines, the combined-cycle gas plants and the grid upgrades required to keep it running? Traditionally, utilities answered that question by spreading the cost across their entire rate base, which is to say, across every household and small business in the service territory. This is the legal framework that built America's electricity system, and it is also the legal framework that is now quietly transferring billions of dollars a year from retirees and single parents and small restaurants to the balance sheets of the richest corporations in history.

Ari Peskoe, director of the Electricity Law Initiative at Harvard Law School, has spent years documenting how this mechanism works in practice, and his conclusion is as blunt as it is unflattering: ratepayers, not shareholders, are funding the build-out. Kasia Tarczynska, a researcher at Good Jobs First, has traced the subsidy flows state by state and estimates that data centres are extracting roughly a billion dollars a year in tax breaks in Virginia and Texas alone, a figure that does not even include the uncosted externalities of rate shifting. The subsidies are public. The profits are private. The bills, when they arrive, arrive at the addresses of people like John Steinbach, who did not know there was a deal, let alone that he was on the hook for it.

The Mechanism, in Plain English

The mechanism is worth understanding, because it is the pivot on which the entire community-cost argument turns, and because utility regulators have for years hidden it behind a thicket of acronyms and rate-case filings that discourage lay scrutiny.

When a hyperscaler signs a contract with a utility for, say, a one-gigawatt data centre, that utility must deliver one gigawatt of firm, twenty-four-hour-a-day power. A gigawatt is roughly the annual consumption of 750,000 American homes. To deliver it, the utility must build or expand generation capacity (gas turbines, in most cases, because renewables are intermittent and nuclear is too slow), string new transmission lines, and upgrade substations. These are enormous capital projects, and utilities recoup their costs by applying to a state public service commission to raise rates.

The key word in that last sentence is “rates,” plural. American utility regulation does not typically require the cost of infrastructure built for a specific customer to be charged to that specific customer. Instead, it is socialised across the rate base on the grounds that everyone benefits from a robust grid. This assumption, reasonable in an era when new load came from population growth and air conditioning, becomes surreal when the new load is a single fenced-in campus drawing more power than a medium-sized city and producing no goods, no services and no jobs beyond a few dozen technicians.

In September 2025, Dominion Energy, Virginia's largest utility, formally acknowledged the problem by proposing a new rate class specifically for data centres. It was a partial concession, and critics including the Piedmont Environmental Council argued it did not go nearly far enough. In the 2026 legislative session that followed, Delegate John McAuliff pitched HB 503, a bill that would prevent utilities from putting the cost of transmission lines and generation serving data centres onto residential ratepayers. Governor Glenn Youngkin's subsequent amendments to related legislation instructed the State Corporation Commission to “take all measures to reasonably ensure” that data-centre costs “are not being subsidized by other customers of the utility.”

These are real legislative interventions, and they represent the first serious attempt in any American state to restore the polluter-pays principle to grid economics. They are also, so far, largely reactive. Nearly 600 data centres are already operational in Virginia. More than 100 are under construction or proposed. The horse is not so much bolting as already over the horizon, trailed by a slow-moving legislature and a furious electorate.

The Water Question, and the Silence That Follows It

In January 2026, Al Jazeera published a column by the Egyptian-British commentator Omar Shabana with the understated headline “AI's growing thirst for water is becoming a public health risk.” The column set out a case that, while partially familiar to climate reporters, had not quite been assembled with such clarity before. AI data centres, Shabana noted, are projected to increase global water usage from 1.1 billion to 6.6 billion cubic metres by 2027. Microsoft's own 2023 environmental report acknowledged that 41 per cent of its water withdrawals came from areas under water stress. Google's figure was 15 per cent from high-scarcity regions. And those were the self-reported numbers.

The public health argument ran as follows. Only 0.5 per cent of the water on Earth is fresh. When data centres siphon millions of gallons a day from municipal or aquifer sources in drought-prone regions, households are pushed to ration. Rationing means prioritising drinking and cooking over washing hands, food and bodies. The World Health Organization has long documented the correlation between reduced hygiene and the spread of cholera and diarrhoeal disease, conditions that disproportionately kill children under five, who bear 84 per cent of the global burden. “Although it is too early to draw direct causal links between AI data centres and water-related diseases,” Shabana wrote, “the known facts make this a significant concern.” It was, in WIRED's parlance, the softest possible framing of the hardest possible claim.

The specifics are where the argument gets teeth. In Newton County, Georgia, residents near a Meta facility reported discoloured, sediment-filled water in their taps. In Fayette County, similar complaints surfaced. In Phoenix, Arizona, Consumer Reports documented a facility consuming 385 million gallons a year for evaporative cooling, with projections rising to 3.7 billion gallons annually, in a desert city whose principal water source, the Colorado River, has been in declared shortage since 2021. A 2023 academic study led by researchers at the University of California, Riverside, estimated that a single ChatGPT conversation of 10 to 50 queries can consume roughly half a litre of water once electricity generation is factored in, a figure that sounds trivial until you multiply by a billion users.

The intellectual scandal, such as it is, concerns the asymmetry of visibility. The hyperscalers know precisely how much water their cooling towers consume; they metre it. The communities whose aquifers are being drawn down typically do not. In 25 of 31 Virginia communities surveyed by journalists in 2025, data-centre operators had insisted on non-disclosure agreements with local governments before revealing the details of their water or energy needs. These NDAs, enforceable under state commercial-confidentiality statutes, have meant that residents have often learned of a pending project's footprint only after construction began.

The Noise, the Heat, the Light

If the grid-rate and water stories are the invisible costs of the data-centre boom, the sensory costs are the ones that have, perhaps predictably, driven the fiercest backlash. Data centres, despite a reputation for clean-room minimalism, are loud. They are warm. They are often lit like small airports, twenty-four hours a day, on land that was once farm or forest.

In Southaven, Mississippi, Jason Haley stopped being able to sleep with the windows open in August 2025. The noise came from the direction of the old power-plant site half a mile from his house, where Elon Musk's xAI had begun erecting a cluster of natural-gas turbines to supply its Colossus supercomputer across the state line in Memphis. The turbines, as Haley described them to Mississippi Today, sounded like “a leaf blower” that ran for days at a stretch, all night, every night. Eighteen of them are currently operating. Fifty-nine are planned. xAI is awaiting state permits for the remainder. Whatever those permits say, the noise does not stop for them.

In Chandler, Arizona, a city of 280,000 just outside Phoenix, the battle over data-centre noise began as far back as 2014, when a million-square-foot facility produced a steady drone that drove residents to the council chambers. The city eventually adopted an ordinance targeting data-centre noise specifically. In December 2025, the Chandler City Council unanimously voted down the rezoning application for a proposed AI data centre, the kind of decisive “no” that, a few years ago, would have been almost unimaginable for a facility that promised capital investment and a handful of jobs.

Diana Dietz, who lives several miles from a QTS data-centre campus in Fayetteville, Georgia, told Consumer Reports she was not especially opposed to technology. What she objected to was the parade of earth-moving machinery through her neighbourhood at all hours. “You've got these giant excavation-style dump trucks two minutes from my house,” she said. The trucks, like the turbines, like the substation transformers and the chiller plants, are the visible evidence of a fact that AI boosterism tends to obscure: intelligence in a cloud is, in the physical world, heavy machinery.

The Regulatory Vacuum

The reason these stories keep happening in such similar shapes, in Manassas and Memphis and Chandler and Fayetteville, is that the United States has no coherent federal framework for siting AI infrastructure, and most states have not yet caught up. Zoning law is largely a matter of county and municipal jurisdiction. Utility regulation is a state matter, usually delegated to a public service commission whose commissioners are appointed, not elected, and whose rate-case proceedings are structured for expert testimony rather than public comment. Environmental permitting is split between the Environmental Protection Agency, state environmental agencies, and, in the case of water, a byzantine patchwork of river compacts, aquifer management districts and groundwater statutes that vary wildly from one jurisdiction to the next.

Into this regulatory swiss cheese rolls the hyperscaler, armed with non-disclosure agreements, a pre-negotiated economic-development package, and a timeline that counts in months rather than years. Local officials, often in rural counties with shrinking tax bases, see a multibillion-dollar investment and a handful of permanent jobs, and they sign. By the time the trucks arrive and the neighbours notice, the horse is out of the barn and several hundred megawatts of load have been added to a grid that was not designed for it.

Data Center Watch, a tracking group that has been cataloguing community responses since 2023, reported in early 2026 that roughly $64 billion worth of data-centre developments had been blocked or delayed by grassroots opposition across the United States. There are now 142 activist groups organised in 24 states. Twenty-five major projects were cancelled in 2025 alone. Twenty-one of those cancellations came in the second half of the year. In Wisconsin, sustained local opposition caused Microsoft to abandon plans for a 244-acre campus. In Brandy Station, Virginia, the Culpeper County Planning Commission unanimously denied rezoning for a $12 billion project in June 2024, a rare and galvanising rebuke. In King George County, Virginia, a newly elected board of supervisors reopened negotiations on Amazon's $6 billion campus.

These are not fringe events. They are becoming a pattern. And the pattern is, increasingly, that the hyperscalers are discovering the hard way that the social contract around infrastructure cannot be replaced with a press release.

Enter the Investors

In April 2026, Reuters reporters Simon Jessop, Valerie Volcovici and Supantha Mukherjee published a piece that marks, in retrospect, a turning point in the story. Amazon, Microsoft and Google had each, the article noted, recently abandoned multibillion-dollar data-centre projects in the face of community opposition. Now, shareholders were asking why. More than a dozen investors, ahead of spring annual meetings, were filing resolutions demanding disclosure of water consumption, power usage and community engagement strategies.

Trillium Asset Management, a Boston-based firm managing more than $4 billion, filed a resolution with Alphabet in December 2025 seeking clarity on how the company would meet its existing climate goals given the surging electricity needs of its data centres. Andrea Ranger at Trillium said the company had left investors “in the dark.” Jason Qi, lead technology analyst at Calvert Research and Management, offered the kind of restrained corporate-stewardship rebuke that reads as devastating only when you remember that Calvert sits on hundreds of billions in institutional assets: “We haven't seen them disclosing enough about their water consumption (and the) impact on the local community.” Green Century Capital Management joined the chorus.

The hyperscalers' responses were, on the whole, not reassuring. Microsoft said sustainability was “a core value” and that it was “proactively addressing sustainability challenges.” Google declined to comment. Meta did not return the Reuters reporters' request. Dan Diorio, vice-president of the Data Center Coalition, the industry's lobbying arm, offered what is probably the clearest articulation of where this argument is actually headed: “Being upfront with them regarding energy and water use, and so that residents can understand that this project will not stress their resources... and will protect them as rate payers is crucial.”

That is not the language of a confident industry. That is the language of an industry that has just realised its permits depend on the people whose aquifers it is draining and whose electricity bills it is inflating. The pressure, importantly, is now coming from two directions at once: from below, in the form of local activism and abandoned projects; and from above, in the form of fiduciary-duty shareholder resolutions from firms that can no longer pretend community opposition is an idiosyncratic, one-off risk.

The Social Licence to Operate

In March 2026, the World Economic Forum published a piece in its Agenda section with the title “Why AI and data-centre growth risks stalling without a social licence to operate.” The phrase “social licence to operate” originates in the mining industry, where it was coined in the 1990s to describe the informal consent that host communities grant (or withhold) from extractive projects, regardless of what formal permits say. In mining, the social licence is the difference between a functioning copper mine and a blockaded access road.

The WEF piece argued, in language calibrated for an audience of Davos attendees rather than Virginia ratepayers, that the AI industry was now facing precisely this kind of legitimacy test. “Understanding the importance of a social licence to operate is becoming critical as AI infrastructure, especially data centres, outpaces governance,” the piece observed. “Growing opposition across regulators, politicians and communities is already driving permit blocks and constraining scale.” Communities were not, it argued, “passive hosts” but “stakeholders with leverage.”

The piece then laid out what the industry needed to do to earn this licence: transparency about energy and water use; stable electricity prices; responsible water management; meaningful job pathways; enhanced local infrastructure. It was, in effect, an industry trade body admitting in print what the activists in Chandler and Brandy Station and Culpeper had been saying for years. If you do not earn consent, you do not get to build. If you get to build anyway, you will not get to build the next one. And the next one is where the money is.

The framing is useful because it reorients the ethical conversation away from the hypothetical harms of AI, the rogue superintelligences of science-fiction discourse, and towards the concrete, present, measurable harms of AI infrastructure. It is much easier to argue about whether GPT-7 will end civilisation than to explain to a state legislature why a Microsoft substation should or should not be socialised across the residential rate base. The social-licence framing does the explaining for us: it names the transaction, names the parties, and names the consent that was or was not given.

What Meaningful Consent Would Actually Look Like

If the social-licence argument is to be more than a rhetorical flourish, it has to be translated into institutional mechanisms with actual enforcement teeth. The literature on community consent in infrastructure, drawn from mining, pipelines, wind farms and landfill siting, suggests at least five such mechanisms, each of which is already being tested somewhere in the American data-centre debate.

The first is rate segregation, the Virginia HB 503 model, in which the cost of new generation and transmission built for a specific class of customer is recovered only from that class, rather than socialised across all ratepayers. This is the narrowest but most important reform, because it removes the hidden subsidy that currently makes data centres appear cheaper to host than they actually are. If hyperscalers had to pay the full incremental cost of the grid infrastructure they require, fewer speculative projects would pencil out, and the ones that did would bring with them a more defensible economic case.

The second is mandatory impact disclosure. HB 496 in Virginia, which would require localities to consider annual water-consumption estimates in rezoning decisions, is a modest but meaningful example. A more muscular version would require an environmental and infrastructure impact statement, subject to public hearing, for any data-centre project above a threshold size, modelled on the National Environmental Policy Act but administered at state level with specific provisions for aquifer drawdown, peak-load demand, noise contours and light pollution.

The third is community benefit agreements, the legal instrument used widely in urban development to bind a developer to specific commitments to the host community. In the data-centre context, a CBA might include funded local-grid upgrades, guaranteed rate stability for residential customers, water conservation investments, noise mitigation standards, a public education levy and a seat for community representatives on an ongoing oversight body. These agreements are enforceable contracts, not voluntary promises. The Chandler ordinance targeting data-centre noise is, in effect, a unilateral CBA imposed by a city after losing patience with the developer's assurances.

The fourth is meaningful public hearings with teeth, which sounds anodyne until one realises how often current hearings function as information sessions at which decisions already made are announced to residents. A hearing with teeth is one where the decision can actually be reversed, where the developer is obliged to answer questions under oath, and where non-disclosure agreements signed in advance by local officials are voided as a matter of public policy. The Culpeper County Planning Commission's unanimous denial in Brandy Station is a model of what this looks like when it works.

The fifth is water impact assessment with a veto right for water authorities, a mechanism borrowed from arid-region agricultural law. In several Western American states, irrigation districts already hold something close to this power over new industrial users. Applying it to data centres would mean that a project proposed in, say, the Phoenix metropolitan area would not simply need a permit from the city; it would need affirmative sign-off from the regional water management district, based on a fifty-year draw projection and a public-health impact assessment of the kind Omar Shabana described in Al Jazeera. This is how water scarcity is managed in places that take water scarcity seriously.

None of these mechanisms individually constitute “consent.” Together, they describe the shape of a regulatory regime in which a data-centre project could not be built until the people whose lives it would change had been informed, consulted, and given the ability to shape or stop it. That is not an anti-AI position. It is the position American infrastructure regulation already holds towards pipelines, landfills and nuclear plants. The hyperscalers have been exempt from it essentially by default, because no legislator anticipated that the data centre would become the dominant form of new industrial development in the country.

The Ratepayer Protection Pledge, and Why It Is Not Enough

In early March 2026, executives from Amazon, Microsoft, Google, Meta and Anthropic travelled to the White House to sign a document called the Ratepayer Protection Pledge. It was non-binding. It committed signatories to “principles” around transparency and responsible siting. Anthropic additionally pledged to cover electricity-price increases related to its own data-centre development, a gesture that impressed some observers and struck others as the kind of one-off public relations flourish that is easy to make when your data-centre footprint is a small fraction of Microsoft's.

The pledge is worth noting because it is, in effect, the industry's first serious attempt at self-regulation, and because self-regulation is almost always what an industry proposes when it fears the alternative. It is also worth noting because it is inadequate in ways that illustrate why the social-licence framing matters. A pledge without enforcement is an aspiration. A pledge that can be reinterpreted by its signatories is a negotiating document. The mechanisms that would actually protect John Steinbach in Manassas, or Jason Haley in Southaven, or Diana Dietz in Fayetteville, are statutory, not voluntary. They live in utility tariffs, zoning ordinances and environmental statutes, not in White House photo opportunities.

The Question of Who Pays

Return, finally, to John Steinbach's kitchen table, and to the $281 bill that started this story. The question the bill poses is not really about kilowatt-hours. It is about a much older political question: who pays for the infrastructure that powers the new economy? The answer the current system gives, by default, by inertia, by the absence of legislative attention, is that everyone does, apportioned by their share of the residential rate base. The answer the activists in Wisconsin and Virginia and Arizona are demanding is that the beneficiaries of the infrastructure should pay for the infrastructure, which is a principle so uncontroversial it appears in first-year economics textbooks. The reason it does not currently apply to data centres is that the law has not been updated to reflect a world in which a single industrial customer can consume more power than an entire American city.

The hyperscalers have, to their credit, begun to notice. The Reuters report on investor pressure, the WEF piece on social licence, the White House pledge, and the scattered community-benefit agreements now being negotiated in individual counties all suggest an industry that has belatedly realised it cannot continue to build its physical substrate in places that do not want it. What is still missing is a coherent legislative framework that gives communities the tools to say no, or to say yes on specific terms, before the earth-movers arrive.

That framework will arrive, eventually, the way such frameworks always arrive in the American system: through a patchwork of state reforms that eventually force federal harmonisation, driven by the accumulation of lawsuits, news stories, bills like Steinbach's and places like Brandy Station where the planners drew the wrong parcel on the map and the locals remembered they were allowed to vote. Until then, the story will continue to be written in envelopes opened at kitchen tables in towns most of the industry's customers have never heard of, by people whose consent was presumed rather than sought, for a build-out whose bill is still only beginning to come due.

The Cost Is Not Abstract

One of the oddities of covering this beat is how persistently the language used to describe AI infrastructure collapses the physical world into metaphor. We speak of “the cloud,” as though data did not live on spinning disks in concrete boxes. We speak of “compute,” as though it were an abstract quantity rather than a thermodynamic process that converts electricity into heat. We speak of “scaling,” as though the industrial footprint of a two-gigawatt campus were a matter of typography.

The activists in Chandler and the ratepayers in Manassas have done the rest of us a small service by refusing this language. The cost is not abstract. It is $281. It is a leaf-blower noise that does not stop. It is discoloured water in Newton County, Georgia, and a parade of dump trucks outside Fayetteville. It is a Microsoft cooling tower pulling from an aquifer that was already in decline, and an electricity bill that arrived in a January envelope and could not be explained by the weather.

What the AI industry has built is, on any honest accounting, extraordinary. What it has failed to build, so far, is the social architecture of consent that every previous heavy-industrial sector in American history eventually had to build: the permitting regimes, the environmental impact statements, the community benefit agreements, the ratepayer protections, the statutes that ensure the people who host the factory are not also the people who pay for it.

The absence of that architecture is not a technicality. It is the entire argument. Meaningful consent does not mean asking permission after the foundation has been poured. It means building a system in which the permission must be asked, granted, and periodically renewed, by the people whose land, water, electricity and night-time quiet are on the line. Anything short of that is not a social licence. It is an imposition with a press release.

John Steinbach will pay his bill. He does not have an option. The question is whether, by the time the next bill arrives, he will have been given any real say in what that bill is for. The answer to that question will determine, in ways the AI industry is only beginning to understand, whether the next gigawatt of data-centre capacity gets built at all.

---

References

1. Shah, Kaveh. “AI Data Centers: Big Tech's Impact on Electric Bills, Water, and More.” Consumer Reports, 20 March 2026. https://www.consumerreports.org/data-centers/ai-data-centers-impact-on-electric-bills-water-and-more-a1040338678/
2. Jessop, Simon; Volcovici, Valerie; Mukherjee, Supantha. “Investors press Amazon, Microsoft and Google on water, power use in US data centers.” Reuters, 6 April 2026. https://www.investing.com/news/stock-market-news/analysisinvestors-press-amazon-microsoft-and-google-on-water-power-use-in-us-data-centers-4598459
3. Shabana, Omar. “AI's growing thirst for water is becoming a public health risk.” Al Jazeera Opinion, 21 January 2026. https://www.aljazeera.com/opinions/2026/1/21/ais-growing-thirst-for-water-is-becoming-a-public-health-risk
4. World Economic Forum. “Why AI and data-centre growth risks stalling without a social licence to operate.” WEF Agenda, March 2026. https://www.weforum.org/stories/2026/03/ai-community-social-licence/
5. Fortune Editorial. “Data centers now account for half of all new U.S. electricity use, just as Americans start to sour on AI.” Fortune, 20 April 2026. https://fortune.com/2026/04/20/us-data-center-electricity-demand-public-opinion/
6. CNBC. “Who is really footing the AI energy bill? Inside the debate about data center electricity costs.” CNBC, 13 March 2026. https://www.cnbc.com/2026/03/13/ai-data-centers-electricity-prices-backlash-ratepayer-protection.html
7. NPR. “AI data centers use a lot of electricity. How it could affect your power bill.” NPR, 2 January 2026. https://www.npr.org/2026/01/02/nx-s1-5638587/ai-data-centers-use-a-lot-of-electricity-how-it-could-affect-your-power-bill
8. Virginia Mercury. “These bills aim to regulate Virginia data center siting, generator use and SCC oversight.” Virginia Mercury, 19 January 2026. https://virginiamercury.com/2026/01/19/these-bills-aim-to-regulate-virginia-data-center-siting-generator-use-and-scc-oversight/
9. Virginia Mercury. “The governor's amendments to energy and data center legislation will save money for customers.” Virginia Mercury, 22 April 2026. https://virginiamercury.com/2026/04/22/the-governors-amendments-to-energy-and-data-center-legislation-will-save-money-for-customers/
10. Piedmont Environmental Council. “2026 Data Center Reform Legislation.” PEC, 2026. https://www.pecva.org/region/regional-state-national-region/2026-data-center-reform-legislation/
11. Data Center Watch. “$64 billion of data center projects have been blocked or delayed amid local opposition.” Data Center Watch Report, 2026. https://www.datacenterwatch.org/report
12. Mississippi Today. “Southaven residents fear pollution, complain of noise, from Elon Musk's xAI data-center turbines.” Mississippi Today, 24 November 2025. https://mississippitoday.org/2025/11/24/southaven-residents-fear-pollution-complain-of-noise-from-elon-musks-xai-data-center-turbines/
13. AZ Family. “Chandler unanimously votes against proposed AI data center.” AZ Family, 12 December 2025. https://www.azfamily.com/2025/12/12/chandler-unanimously-votes-against-proposed-ai-data-center/
14. Environmental and Energy Study Institute. “Communities Are Raising Noise Pollution Concerns About Data Centers.” EESI. https://www.eesi.org/articles/view/communities-are-raising-noise-pollution-concernsabout-data-centers
15. ARLnow. “Winter electric bill spikes spark scrutiny of Va. data centers' power usage.” ARLnow, 27 February 2026. https://www.arlnow.com/2026/02/27/winter-electric-bill-spikes-spark-scrutiny-of-va-data-centers-power-usage/
16. The White House. “Ratepayer Protection Pledge.” The White House, March 2026. https://www.whitehouse.gov/releases/2026/03/ratepayer-protection-pledge/
17. Li, Pengfei; Yang, Jianyi; Islam, Mohammad A.; Ren, Shaolei. “Making AI Less Thirsty: Uncovering and Addressing the Secret Water Footprint of AI Models.” arXiv, 2023. https://arxiv.org/pdf/2304.03271
18. CalMatters. “AI data centers could hike California electricity bills.” CalMatters, March 2026. https://calmatters.org/environment/2026/03/little-hoover-data-center-electricity/
19. Virginia Mercury. “Dominion proposes higher utility rates, new rate class for data centers.” Virginia Mercury, 3 September 2025. https://virginiamercury.com/2025/09/03/dominion-proposes-higher-utility-rates-new-rate-class-for-data-centers/
20. Bloomberg. “AI Data Centers Are Sending Power Bills Soaring.” Bloomberg Graphics, 2025. https://www.bloomberg.com/graphics/2025-ai-data-centers-electricity-prices/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

The fifty-minute hour has always been a negotiated fiction. A patient talks, a therapist listens, and somewhere between them a third thing forms: a clinical relationship dense with attention, memory, and the particular quality of being heard. For more than a century, that triangle contained only two people. It no longer does.

Open the laptop of a typical American psychologist in the spring of 2026 and the odds are better than even that an AI tool is already running. It might be an ambient scribe quietly listening to the session and transcribing it into a SOAP note by the time the patient reaches their car. It might be a risk-screening dashboard that flags linguistic markers of suicidality in a client's intake forms. It might be a real-time coaching tool that whispers a reframe into the clinician's earpiece when a silence stretches too long. The American Psychological Association's Practitioner Pulse Survey, the results of which landed in March 2026, is unambiguous: AI has moved from fringe novelty to routine infrastructure in clinical practice. Fifty-six per cent of the 1,742 psychologists surveyed said they had used an AI tool to assist their work in the past year. Twenty-nine per cent use one at least monthly. In 2024, seventy-one per cent said they had never touched the stuff. That is a generational shift compressed into twelve months.

What the APA did not find, and went out of its way to say it did not find, was a professional consensus on any of the hard questions. Not on disclosure. Not on patient rights. Not on what a therapist owes a client before flipping on a machine that will listen to their most intimate disclosures and then ship the audio to a vendor in a different state, possibly a different country, for processing. The technology has outrun the ethics, and the ethics have outrun the law, and the patient, sitting on the couch, is the last to know.

This is the part of the story that does not get a press release. When somebody talks about AI in mental health, they usually mean the chatbot, the thing a lonely teenager might befriend at three in the morning. That conversation has had its reckoning in courtrooms and congressional hearings, and it is not the one that matters most for the several million people who still see a human therapist each week. The interesting question is quieter and harder. If the clinician sitting across from you, the one you were referred to by your GP or your insurer, is quietly running an AI in the background of your treatment, what exactly are they obliged to tell you about it? And does the research support the idea that adding the machine makes the therapy better, or only cheaper to provide?

The Ambient Scribe Enters the Room

The single most common entry point for AI into clinical practice is also the most banal. Therapists hate paperwork. They always have. The running joke in any psychology department is that nobody went into the field to write insurance justification letters, and yet everyone spends roughly a third of their working hours doing exactly that. A standard therapy session runs fifty minutes. The documentation attached to it, progress notes, billing codes, risk assessments, treatment plan updates, can eat another thirty. Over a week, the arithmetic is brutal. A therapist seeing twenty-five clients is writing for twelve and a half hours on top of their clinical load. Much of that work happens after dinner.

Into this grim calculus walked a class of product nobody would have predicted five years ago: the AI therapy scribe. Blueprint, founded in 2020 and now one of the category leaders, offers an ambient listening tool that records sessions, transcribes them, and spits out a structured progress note in under sixty seconds. The pitch is seductive. The company claims average documentation times can fall from around 4.7 hours a day to 1.2, a reduction of roughly seventy-five per cent. Upheal, a Prague-based competitor, promises essentially the same thing with a slightly different feature set: SOAP, DAP, BIRP, or GIRP notes generated to a therapist's preferred template, with the model learning the clinician's writing style over time. Both companies advertise HIPAA compliance, SOC 2 certification, Business Associate Agreements, and end-to-end encryption. Blueprint automatically deletes its recordings after transcription. Upheal deletes audio too, though it retains transcripts for clinician review.

These are not marginal products. Blueprint says it is used by tens of thousands of therapists across the United States. Upheal reports a comparable footprint. They have competitors: Freed, Mentalyc, Twofold, Heidi Health. They have pricing tiers aimed at solo practitioners and at group practices, and in some cases at the in-house compliance officers at hospital systems. They have venture capital behind them. They have, above all, momentum.

And they sit at the centre of an ethical problem most of their users have not fully reckoned with. When a client walks into a therapy session, their spoken words are, under ordinary confidentiality rules, legally and ethically protected to a very high standard. Therapy is the paradigm case of a privileged communication. What happens when those words are transmitted in real time to a third-party vendor for algorithmic processing, stored in a cloud they cannot see, and used, in most contracts, to improve the vendor's underlying model? The HIPAA answer is that the vendor becomes a business associate, that a Business Associate Agreement must be in place, and that appropriate safeguards must be documented. The ethical answer is considerably more complicated. It turns on whether the client has any meaningful understanding of what is happening, and any meaningful ability to say no.

What the APA Actually Said

The March 2026 APA findings, laid out across the association's Monitor on Psychology coverage and the underlying Practitioner Pulse Survey data, painted a field in the middle of an uneven, improvised adoption curve. The most common AI uses reported by psychologists were administrative: drafting emails, summarising documents, generating content, and dictation or note-taking. These are not, for the most part, tasks that touch clinical judgement. They are the grinding logistics of running a practice. But a growing minority of respondents reported more ambitious uses: treatment planning support, risk assessment augmentation, and, in the most contested category, real-time prompting during sessions.

The concerns practitioners reported were granular and serious. Sixty-seven per cent worried about data breaches. At least sixty per cent flagged concerns about biased inputs and outputs, inaccurate outputs, lack of rigorous testing, and unanticipated social harms. Thirty-eight per cent worried that AI would eventually render some or all of their job duties obsolete. The APA's accompanying health advisory, issued in tandem with the survey, was blunt: AI chatbots and wellness apps lack the testing and safety measures needed to provide quality mental health support, and such tools cannot replace qualified clinicians. The advisory did not, however, tell therapists what to do about the AI already running quietly in their own offices.

That gap was partially filled by the APA's Ethical Guidance for AI in the Professional Practice of Health Service Psychology, a document released in mid-2025 and refined through the end of that year. The guidance articulates what, in theory, any American clinician using AI is obliged to do. It is worth dwelling on, because the clarity of the guidance stands in such contrast to the murk of actual practice. Clinicians must obtain informed consent. They must clearly communicate the purpose, application, and potential benefits and risks of the AI tools they are using. They must disclose AI use to individuals receiving direct care, to other relevant providers, and to any third party who might be considered a client. The disclosure must cover the type of AI tools involved, their effect on treatment, the flow of data, the role of third-party vendors, and the cost implications of AI-assisted care. Crucially, clients must be able to opt out, and where feasible be offered a human-only alternative. AI must augment, not replace, professional judgement. The psychologist remains responsible for every decision, and must not defer blindly to a machine-generated recommendation.

This is an admirably complete list, and for anyone who has sat through the five-minute consent ritual in a modern clinic, a slightly fantastical one. The average informed consent form in private practice is already a dense three-page document that clients sign while their therapist fetches water. Adding a granular disclosure of AI usage, one that specifies which vendor, which data paths, which retention policies, and which opt-out mechanisms apply, pushes the document towards something no reasonable client will read. And yet the APA is right that the alternative, a disclosure so thin it amounts to a rubber stamp, betrays the therapeutic relationship's foundational trust.

The Psychiatric News Point/Counterpoint in January 2026 surfaced the tension openly. One camp argued that AI disclosure should be patient-empowering by default, with consent forms that actively invite clients to ask questions about the technology in the room. The other argued for a stricter standard rooted in the principle of non-maleficence: if the clinician is not prepared to answer detailed technical questions about the tool they are using, they should not be using it. Somewhere between those two positions sits the average working therapist, using a scribe because it saves them six hours a week, with only a dim understanding of where the audio goes after the session ends.

The Evidence Problem

Suppose, for a moment, that every clinician disclosed every AI use with flawless clarity. Suppose every client understood, in appropriate detail, what was happening. The question underneath the disclosure question would remain: does any of this make therapy better, or does it simply make it cheaper to deliver? Here the literature is much thinner and much less flattering than the marketing materials suggest.

The most comprehensive recent answer came from a scoping review of reviews published in January 2026 in Frontiers in Psychiatry, which examined the state of AI in mental health care across the field. The review's core finding was that most clinical AI models in mental health remain proof-of-concept. They achieve impressive headline numbers, AUCs of up to 0.98 and accuracies reaching ninety-seven per cent, but those numbers come overwhelmingly from internal validation on small cohorts. When the same models are tested against independent data, performance degrades, sometimes substantially. External validation is sparse. Where it exists, AUCs typically settle into a more modest 0.80 to 0.88 range, and often lower. The review concluded that the gap between demonstrated performance in the laboratory and deployed performance in the clinic had not meaningfully narrowed.

This matters because the business case for clinical AI assumes the headline numbers. A risk-screening tool that correctly identifies suicidal ideation with ninety-five per cent accuracy in a training dataset might flag a tenth of that in practice. A treatment-planning assistant that suggests evidence-based interventions might do so using a corpus that over-represents certain populations and therapies. None of this is to say AI cannot help. It is to say that the current evidence base for routine clinical deployment is not yet what the confidence of the product demos implies.

The picture is worse, and better documented, for consumer-facing therapy chatbots, which remain distinct from clinician-supporting tools but share an underlying technology and some of the same pathologies. The Stanford study that has become the most frequently cited reference point in this debate, presented at the ACM Conference on Fairness, Accountability, and Transparency and led by Jared Moore, Declan Grabb, and Nick Haber of Stanford with collaborators from Carnegie Mellon, the University of Minnesota, and the University of Texas at Austin, delivered a particularly uncomfortable finding. The researchers evaluated five popular therapy chatbots, including 7cups' Pi and Noni and Character.AI's Therapist, against a set of clinical criteria. Licensed human therapists responded appropriately ninety-three per cent of the time. The chatbots responded appropriately less than sixty per cent of the time.

More damning still, the researchers found that the chatbots exhibited more stigma towards certain mental health conditions than others. Alcohol dependence and schizophrenia drew more negative responses than depression. Newer and larger models did not perform better than older or smaller ones. In scenarios involving suicidal ideation or delusions, where a trained clinician would gently push back or help the client reframe, the chatbots sometimes enabled the dangerous thinking rather than interrupting it. The Stanford team was careful about what they were claiming. Their argument was not that chatbots could never be useful. It was that the specific failure modes they documented, stigma reinforcement and crisis mishandling, are particularly dangerous precisely because of the kind of user who seeks out an AI therapist in the first place.

There is a subtler finding layered underneath. If the chatbot reinforces stigma about schizophrenia, and the user of the chatbot is someone with a family history of schizophrenia who has not yet sought formal care, the effect of the interaction is to push that person further from the clinician they actually need. The AI therapist, in certain configurations, does not merely fail to help. It delays the real help.

The TikTok Problem

A second category of failure sits adjacent to the clinical AI question, and is probably more consequential at a population level. It concerns the recommendation algorithms that sit on the social-media platforms where vulnerable people now habitually go looking for mental health information. An arXiv pre-print published on 16 April 2026 by a research team auditing TikTok's mental health recommendations brought the issue into sharp focus. The paper, titled “Seeking Help, Facing Harm”, used thirty simulated accounts that interacted with TikTok's For You feed over a seven-day period in January 2026, collecting 8,727 videos. The audit varied two dimensions: whether the initial search framing was distress-initiated or help-initiated, and whether the account's interaction strategy leaned into the content or away from it.

The finding was that engaged accounts saw their feeds saturate with mental health content, roughly forty-five per cent of daily recommendations. Avoidance and passive viewing reduced the exposure but did not eliminate it, with mental health content still comprising between eleven and twenty per cent of recommended videos. Crucially, TikTok's algorithm did not appear to meaningfully distinguish between a user expressing distress and a user seeking help. Both were treated as a single mental health interest cluster. The consequence is that someone who types “how to stop feeling suicidal” gets treated roughly the same as someone who types “romanticising depression”. Recovery-oriented intent is conflated with distress consumption, and vulnerable users get served content that may exacerbate rather than mitigate harm.

The TikTok finding matters for the therapy-room question because it changes the context in which the therapy happens. The client who walks into a session at five o'clock has spent the preceding weeks, often the preceding years, consuming algorithmic content that actively shapes their sense of what their condition is and how it should be treated. Eating-disorder content, trauma content, ADHD content, grief content: all of it flows through the same engagement-optimised pipe. When the therapist then introduces an additional AI tool into the session, even a benign one like a scribe, they are not doing so in neutral territory. They are doing so on ground already extensively ploughed by algorithmic influence. The disclosure question is no longer just about the scribe. It is about the whole ecosystem of machine-mediated attention the client arrives with.

What the Patient Is Owed

So: if AI is now embedded in what remains one of the most intimate and trust-dependent professional relationships most people will ever have, what disclosure obligations should apply? The answer is not a single sentence, but it is a reasonably short list.

First, a client has the right to know, in plain language, that AI is in the room. That starts with whether a scribe is recording the session, but it extends to whether any automated system is generating, drafting, or shaping the treatment plan; flagging risk signals that will influence the clinician's judgement; or processing session data after the fact. The disclosure should happen at the start of treatment, not buried in a stack of intake forms. It should be revisited when tools change.

Second, the client has the right to understand the data path. Not every technical detail, but the substantive answers to the questions any thoughtful person would ask. Where does the audio go? Who stores it? For how long? Is it used to train a model? Can it be accessed by anyone other than the therapist? What happens if the vendor is acquired, goes bankrupt, or suffers a breach? These questions are answerable, in most cases, from a vendor's published documentation. If the therapist cannot answer them, the therapist does not yet understand the tool they are using.

Third, the client has the right to opt out. This is the hinge of the whole disclosure regime. A disclosure that comes with no meaningful alternative is not consent; it is notification. The APA's guidance is explicit that, where feasible, a human-only alternative should be offered. The qualifier “where feasible” is doing a great deal of work, and it is precisely the word that clinicians and practice managers will lean on when it becomes inconvenient to honour the principle. A patient who says “please do not record the session” should not have to become a problem case.

Fourth, the client has the right to know what the AI is not doing. Most people, when told an AI is involved in their care, will assume more than is actually the case. They may assume the AI is making diagnostic judgements it is not making. They may assume the therapist has abdicated more authority than they have. They may also assume the AI is doing less than it is. The disclosure should be honest in both directions.

Illinois, in August 2025, became the first US state to write this logic into statute. Public Act 104-0054, known as the Wellness and Oversight for Psychological Resources Act, prohibits the use of AI to provide therapy directly, to make independent therapeutic decisions, to interact with clients in any form of therapeutic communication, or to generate treatment plans without clinician review. It permits AI for administrative and supplementary support, provided the patient has given specific, written, revocable consent. Violations carry civil penalties of up to $10,000. Illinois will not be the last state to legislate this territory. Governing AI in Mental Health, a fifty-state legislative review published in late 2025, identified similar proposals in a growing number of jurisdictions. The Federal Trade Commission, having opened an inquiry in September 2025 into AI chatbots acting as companions, is circling the consumer end of the problem. The HHS approach is more deregulatory, aimed at promoting adoption through pilot programmes, but the direction of travel on disclosure is fairly clear.

In the UK, the regulatory movement has been quieter but real. The British Psychological Society has endorsed a Global Psychology Alliance guidance document on AI in psychology. The Health and Care Professions Council, which regulates practitioner psychologists in the UK, collaborated with four other regulators in February 2026 on a joint statement on AI in education and training. Neither document carries the specificity of the Illinois statute. But the principle is being worked out in comparable terms: disclosure, consent, clinician oversight, and meaningful alternatives.

The Economic Gravity

The harder question, and the one the industry would prefer not to answer, is whether the motivation for AI adoption in therapy has more to do with clinical improvement or with cost reduction. The available evidence suggests the latter is doing most of the work.

Start with what the data says about outcomes. A randomised clinical trial of a conversational AI agent for psychiatric symptoms, published in late 2025, found meaningful reductions in anxiety and depression symptoms compared to group therapy and control conditions. Meta-analytic work on AI-driven interventions has found moderate effect sizes, broadly comparable to low-intensity clinician-delivered treatments, particularly for cognitive behavioural approaches to mild or moderate depression. The therapeutic alliance literature suggests users can form surprisingly strong bonds with AI agents, sometimes within three to five days, and that these bonds correlate with engagement and symptom improvement. None of this should be dismissed. For someone who would otherwise get no care at all, a decent AI-mediated intervention is clearly better than nothing.

Now set that next to the economic reality of how therapy is paid for. Insurers routinely deny around fifteen per cent of mental health claims even when prior authorisation has been obtained. Therapists in network spend roughly a third of their working week on documentation and billing. The average productivity gain from an AI scribe, across published implementation studies, is a reduction in documentation time of roughly seventy per cent. That saving does not, in general, accrue to the patient as a lower fee. It accrues to the practice as either higher clinician throughput, a reduction in administrative overhead, or both. In the most straightforwardly commercial model, the clinician sees more clients per week because each one requires less paperwork, and the practice's revenue per clinician rises.

This is not inherently sinister. More clinician availability in a field with chronic workforce shortages is a genuine social good. But it reshapes what AI is for in a way the consent forms rarely acknowledge. The AI is not principally a clinical tool; it is a throughput tool. It makes it economically viable for the therapist to do more of what they already do. Whether it makes what they do better is a separate question, and the answer so far is: perhaps slightly, and only in certain narrow domains, and only where clinicians actively engage with the output rather than letting it substitute for their attention.

The managed-care angle is where this becomes politically charged. If AI scribes and AI treatment-planning aids allow a clinician to see, say, thirty per cent more patients per week, an insurer looking at its network will be tempted, over time, to recalibrate reimbursement rates on the assumption that AI productivity gains will absorb the pressure. The savings that flowed to the practice in year one flow to the payer in year five. The patient's out-of-pocket cost does not decline, but the amount of clinician attention per dollar declines quietly. This is the pattern that radiology, primary care, and coding specialities have already been living through for a decade. Psychotherapy is a latecomer to the trend, but there is no obvious reason it will be immune.

There is a further and more disquieting possibility, which the APA's guidance gestures at without quite confronting. If insurer incentives push towards AI-augmented therapy, and AI-augmented therapy can be delivered at higher throughput, the marginal therapist may find it increasingly difficult to operate without the AI. Not because the AI makes them a better clinician, but because opting out becomes economically untenable. The clinician who refuses the scribe sees fewer patients, generates less revenue, and becomes less competitive in the market. The right to opt out, from the therapist's side, begins to erode under the same logic that is already eroding it on the patient's side.

The Alliance Question

The single deepest concern, and the one hardest to measure, is what all of this does to the therapeutic alliance itself. The alliance, in the clinical literature, refers to the particular quality of collaboration and bond between client and clinician that consistently predicts outcomes across therapeutic modalities. It is not a decoration on therapy; it is substantively what therapy is. The studies on AI-mediated interventions suggest the alliance can form with a machine as well as a human, which is a finding both reassuring and unsettling. A recent Journal of Medical Internet Research integrative review on the digital therapeutic alliance identified five components, goal alignment, task agreement, therapeutic bond, user engagement, and the facilitators and barriers that mediate them, and found each was at least partially achievable in digital contexts.

That literature, however, is mostly about stand-alone digital interventions. The live question for the clinician-with-AI case is whether introducing a machine into the room between two humans changes the alliance those two humans are building. Common sense suggests it must. If the client knows that a transcript of the session is being processed by a vendor, that knowledge will shape what they say. Certain topics, abortion, substance use, past legal trouble, contemplated self-harm, may come up more tentatively or not at all. The therapist's assurance that “the system is HIPAA compliant” will not, in most clients, fully dissolve the awareness that a machine is listening. Awareness of surveillance, even benign surveillance, changes speech. That change is precisely what good therapy exists to undo.

There is a further dynamic on the therapist's side. When a clinician is using an ambient scribe, a small part of their attention is necessarily devoted to what the machine is picking up, whether it is capturing the important moments, whether the resulting note will require heavy editing. That split attention is subtle, but it is not zero. A significant body of attentional research suggests that the perceived presence of a recording device changes how the recorder behaves. Whether that change, summed across a caseload of thirty clients per week, is large enough to measurably degrade therapeutic outcomes is an open empirical question. Nobody has yet designed the study that would answer it cleanly.

The Shape of an Honest Practice

If the current moment is a kind of frontier, a period in which adoption has raced ahead of norms, what does an honest AI-augmented practice look like? A few things are becoming clear.

It looks like disclosure written in plain English and revisited as tools change, not a legal boilerplate tucked into the intake packet. It looks like vendor contracts the clinician has actually read and can summarise for a curious patient in three sentences. It looks like genuine opt-out alternatives, including for the administrative AI, with no penalty for the patient who takes them. It looks like clinicians who can articulate what the AI is doing and, more importantly, what it is not doing. It looks like practices that do not quietly shift the productivity dividend of AI into more intense caseloads without acknowledging the trade-off. It looks like professional bodies that enforce their guidance, rather than publishing it and moving on. It looks like regulators, at the state and federal level, who treat psychotherapy as a high-risk domain and set the bar for deployment accordingly.

It also looks like a willingness, on the part of the profession, to resist the parts of this technology that do not serve the patient. Real-time therapeutic prompting, in which an AI listens to the session and suggests what the clinician should say next, is the category where the ethics strain hardest. There is a plausible account of it in which a supervisor-in-the-ear helps less experienced clinicians avoid mistakes and improves overall care. There is a much less flattering account in which a clinician becomes a kind of animatronic front-end for a model whose reasoning is opaque, whose training data is proprietary, and whose accuracy on this particular patient has never been validated. The Illinois statute's prohibition on AI-driven therapeutic communication without clinician review reflects a judgement that the risk outweighs the benefit at the present state of the art. Other jurisdictions will have to make the same call.

The research agenda, too, has to catch up. The Frontiers scoping review's conclusion that most clinical AI in mental health remains proof-of-concept is not an argument against the technology. It is an argument for external validation studies, for pre-registered trials comparing AI-augmented care to unaugmented care on the outcomes patients actually care about, for longitudinal work on therapeutic alliance in hybrid settings, and for harm-reporting infrastructure of the kind that exists for drugs and devices but does not yet exist for software that mediates clinical relationships.

The Patient Still in the Room

Return, finally, to the room. A client arrives for a fifty-minute hour. They sit on a couch that has accommodated many lives before theirs. They are about to tell a therapist something they have never told anyone, or something they have told many people but never in quite this way. The question the APA report in March, the Frontiers review in January, the Stanford chatbot study, and the TikTok audit in April all circle without quite naming is whether that room is still the same room it was before the machines moved in.

The answer, probably, is that it is not, but that it can still be a good room, provided the people responsible for it are honest about what has changed. The therapist who discloses the scribe and offers the alternative, who reads the vendor contract, who does not pretend to their client that the AI is neutral or invisible, is not being paranoid. They are doing the oldest job in their profession, which is the protection of the therapeutic relationship against whatever threatens it, including the tools the therapist themselves brought in.

The patient, for their part, has a right the culture has not quite caught up with: the right to ask, and to get a real answer. Not a legally defensible reassurance, but the answer a friend would give if they happened to know the technology. Whether the profession can make that kind of answer routine is probably the question that will decide, more than any regulation, whether AI makes therapy better, or simply makes it cheaper to deliver, and quieter in its cheapness.

References and Sources

1. American Psychological Association. “AI in the therapist's office: Uptake increases, caution persists.” Monitor on Psychology, March 2026. https://www.apa.org/monitor/2026/03/ai-reshaping-therapy
2. American Psychological Association. “2025 Practitioner Pulse Survey: AI in the therapist's office.” APA Publications, released December 2025. https://www.apa.org/pubs/reports/practitioner/2025
3. American Psychological Association. “Ethical guidance for AI in the professional practice of health service psychology.” APA Topics, 2025. https://www.apa.org/topics/artificial-intelligence-machine-learning/ethical-guidance-ai-professional-practice
4. Frontiers in Psychiatry. “Artificial intelligence in mental health care: a scoping review of reviews.” January 2026. https://www.frontiersin.org/journals/psychiatry/articles/10.3389/fpsyt.2026.1688043/full
5. Moore, J., Grabb, D., Agnew, W., Ong, D. C., and Haber, N. (Stanford HAI, Carnegie Mellon, University of Minnesota, University of Texas at Austin). “Exploring the Dangers of AI in Mental Health Care.” Presented at ACM FAccT. Stanford HAI. https://hai.stanford.edu/news/exploring-the-dangers-of-ai-in-mental-health-care
6. “Seeking Help, Facing Harm: Auditing TikTok's Mental Health Recommendations.” arXiv preprint, 16 April 2026. Accepted at ICWSM 2026. https://arxiv.org/abs/2604.14832
7. Illinois General Assembly. “Public Act 104-0054: Wellness and Oversight for Psychological Resources Act.” August 2025. https://ilga.gov/Documents/Legislation/PublicActs/104/PDF/104-0054.pdf
8. JMIR Mental Health. “Artificial Intelligence in Mental Health Services Under Illinois Public Act 104-0054: Legal Boundaries and a Framework for Establishing Safe, Effective AI Tools.” 2025. https://mental.jmir.org/2025/1/e84854
9. Blueprint. “Blueprint: AI-Assisted EHR for Therapists.” Product documentation. https://www.blueprint.ai
10. Upheal. “AI Clinical Notes: HIPAA-compliant therapy documentation.” Product documentation. https://www.upheal.io/ai-clinical-notes
11. Blueprint. “AI for Therapists: ACA and APA Ethical Guidance.” https://www.blueprint.ai/blog/aca-and-apa-guidance-for-the-use-of-ai-for-therapists
12. Videra Health. “Navigating the APA's AI Ethical Guidance: A Clinician's Perspective.” July 2025. https://viderahealth.com/2025/07/03/apa-ai-ethical-guidance-clinician-perspective/
13. Documentation Wizard. “AI in Psychotherapy: Disclosure or Consent?” https://documentationwizard.com/ai-in-psychotherapy-disclosure-or-consent/
14. HIPAA Journal. “When AI Technology and HIPAA Collide.” https://www.hipaajournal.com/when-ai-technology-and-hipaa-collide/
15. Psychiatric News. “Point/Counterpoint: Empowering Patients vs. Do No Harm in the Age of AI.” January 2026. https://www.psychiatryonline.org/doi/10.1176/appi.pn.2026.01.1.11
16. British Psychological Society. “New guidance sets out principles for understanding artificial intelligence in psychology.” https://www.bps.org.uk/news/new-guidance-sets-out-principles-understanding-artificial-intelligence-psychology
17. Health and Care Professions Council. “HCPC collaborates on joint statement on use of AI in health and care professional education.” February 2026. https://www.hcpc-uk.org/education-providers/updates/2026/hcpc-collaborates-on-joint-statement-on-use-of-ai-in-health-and-care-professional-education/
18. Journal of Medical Internet Research Mental Health. “Does the Digital Therapeutic Alliance Exist? Integrative Review.” 2025. https://mental.jmir.org/2025/1/e69294
19. JMIR. “Efficacy of a Conversational AI Agent for Psychiatric Symptoms and Digital Therapeutic Alliance: A Randomized Clinical Trial.” 2025. https://pubmed.ncbi.nlm.nih.gov/41979879/
20. Bipartisan Policy Center. “Paying for AI in U.S. Health Care.” https://bipartisanpolicy.org/issue-brief/paying-for-ai-in-u-s-health-care/
21. Kelley Drye and Warren LLP. “AI Chatbots Face Rising Legal and Legislative Scrutiny.” https://www.kelleydrye.com/viewpoints/blogs/ad-law-access/ai-chatbots-face-rising-legal-and-legislative-scrutiny
22. Nixon Law Group. “How States Are Enforcing New AI Laws in Healthcare and Why It Matters.” https://www.nixonlawgroup.com/resources/how-states-are-enforcing-new-ai-laws-in-healthcareand-why-it-matters
23. NPR. “AI in the mental health care workforce is met with fear, pushback and enthusiasm.” 7 April 2026. https://www.npr.org/2026/04/07/nx-s1-5771707/mental-health-care-workforce-artificial-intelligence-ai
24. Manatt Health. “Health AI Policy Tracker.” https://www.manatt.com/insights/newsletters/health-highlights/manatt-health-health-ai-policy-tracker
25. Governing AI in Mental Health: 50-State Legislative Review. 2025. https://pmc.ncbi.nlm.nih.gov/articles/PMC12578431/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

On a wet Wednesday evening in November 2025, Portland City Council took a vote that amounted to a municipal exorcism. By a margin of eight to two, with two councillors absent, the council adopted an ordinance amending the Affordable Housing Code to prohibit a very specific kind of ghost: the algorithmic rent-setting device. The ordinance, pushed to the floor by Councillor Angelita Morillo, banned the sale, licence, and use of tools that ingest landlords' competitively sensitive rental data, mix it together in a proprietary blender, and spit out monthly pricing recommendations that somehow, as if by coincidence, tend to rise. On the same day, Oregon announced a proposed seven million dollar settlement with one of the country's largest landlords over illegal rental price fixing. The message was not subtle. The algorithm, the council had decided, was not a neutral tool. It was a participant in a conspiracy that no single human had ever needed to speak aloud.

To understand why a mid-sized American city felt compelled to legislate against a piece of software as if it were a cartel, you have to reckon with an odd fact about the last decade of rental housing in North America: the landlord, increasingly, is not a person. Not in any meaningful sense. It is a stack. A portfolio held by a private equity fund, operated by a property management company, advised by a revenue management platform, screened by a background check vendor, underwritten by an income verification API, and enforced, when necessary, by an automated eviction pipeline. Somewhere in that stack there is usually a human being who signs forms. But the decisions, the ones that matter to a tenant, the rent you pay, the flat you get, the day you are handed a notice to quit, have migrated upstream into systems whose logic is proprietary, whose operators have every incentive not to explain themselves, and whose errors compound at scale.

The Portland ordinance was among the first municipal laws in North America to say, in plain terms, that this arrangement is not acceptable. It will not be the last. In March 2026, the New Jersey Senate Community and Urban Affairs Committee reported Bill S451 with amendments, and a companion bill, A3497, advanced in the Assembly. Sponsored by Senators Brian Stack and Teresa Ruiz, the New Jersey legislation takes a sharper rhetorical line than Portland's: it frames the use of profit-maximising rental algorithms as a potential violation of state antitrust law, a reclassification of software as conspiracy. Governor Mikie Sherrill has signalled that she will sign the measure if it reaches her desk, describing the practice in terms that would have sounded paranoid five years ago: “for-profit surveillance by Big Tech.” And in February 2026, OpenMedia, the Canadian digital rights organisation, published an investigation titled Watch This Space: The Rise of AI Landlords in Canada, a dispatch from a country quietly outsourcing one of its most important social relationships to a handful of opaque vendors, and discovering, too late, how few tools exist to hold them to account.

What is emerging, in other words, is a regulatory front. It is jagged, uncoordinated, and full of holes. But it is real. And it is belated.

The Hub, the Spokes, and the Polite Fiction

The antitrust case against algorithmic rent-setting software rests on a theory as old as cartels themselves. In the classical “hub-and-spoke” conspiracy, a central coordinator channels information among competitors in a way that allows them to align their behaviour without ever meeting in a smoke-filled room. The hub never explicitly tells the spokes to raise prices. The spokes never explicitly agree to. But information flows, prices move, and the market ends up acting as if there had been an agreement, because functionally there was one.

In 2024, the US Department of Justice filed a civil antitrust lawsuit alleging the hub could be software. The defendant was RealPage, a Texas-based property technology company whose YieldStar and AI Revenue Management products had come to dominate the multifamily rental market. According to the DOJ's complaint, RealPage's software ingested non-public, competitively sensitive pricing and occupancy data from participating landlords, ran it through a proprietary algorithm, and returned real-time rent recommendations. Each landlord, acting alone, could plausibly claim to be simply accepting third-party advice. Collectively, they were coordinating rents across thousands of competing properties. The software was doing what a cartel meeting used to do, only faster, more granularly, and with a user interface.

RealPage settled with the DOJ on 24 November 2025, agreeing to stop offering software that uses non-public competitively sensitive data shared among landlords. The final judgement installs an independent monitor for three years. A parallel class action in Tennessee produced preliminary settlements with twenty-six defendants in October 2025. In New Jersey, the state attorney general sued RealPage and ten of the state's largest landlords, alleging revenue management products had been used to inflate rents and eliminate competitive pricing.

Legal academics have spent two years picking apart a question the RealPage case raises but does not fully resolve: at what point does using the same pricing software as your competitors become collusion? The Ninth Circuit, in an August 2025 decision, held that mere parallel use of a common algorithm is not enough; there must be some additional factor, a “plus” in antitrust parlance, that suggests coordination beyond independent business judgement. Critics argue this standard is already obsolete. The whole point of modern revenue management software is that it dissolves the distinction between independent judgement and coordinated behaviour. When your “judgement” is literally reading off a dashboard populated with your competitors' data, there is nothing independent about it. You are just a spoke on a very well-lit wheel.

This is why the Portland ordinance and the New Jersey legislation matter beyond their jurisdictions. They are experiments in skipping past the antitrust doctrinal debate entirely. Rather than arguing, case by case, about whether a particular use of a particular algorithm constitutes a Sherman Act violation, they simply prohibit the class of tools that makes the question interesting. Hoboken passed a similar ordinance in July 2025. Jersey City became the first New Jersey municipality to ban AI-driven rent-setting software in May 2025. San Francisco, Berkeley, Philadelphia, and Minneapolis have followed variously. New York signed its own ban in 2025. The patchwork is real. So is the backlash: RealPage has sued Berkeley alleging pre-emption, and a bill in Congress would block local bans altogether, sponsored by legislators whose campaign contributions from the property technology sector are, to put it charitably, not coincidental.

The polite fiction that algorithmic pricing is a neutral productivity tool, rather than a coordinated market behaviour, is dissolving.

Screening, or the Quiet Part Out Loud

If algorithmic price-setting is the loud scandal, algorithmic tenant screening is the quieter one, and arguably the uglier. Price fixing harms everyone who pays rent. Screening algorithms harm, specifically and disproportionately, the people with the least leverage: low-income applicants, people of colour, housing voucher holders, recent immigrants, people with disability histories, and, as we will see, retired people whose wealth happens to live in the wrong kind of account.

The paradigmatic case is Louis v. SafeRent Solutions, which ended in a 2.275 million dollar settlement in late 2024. Mary Louis, a Black woman in the Boston area, applied for a flat. She had a housing voucher. She had a cosigner with high credit. She had a recommendation from a landlord who had rented to her for seventeen years. SafeRent's scoring algorithm denied her anyway. The score was a number. The number was the decision. When she asked why, nobody could tell her, because nobody knew. The algorithm's weights were proprietary. The landlord had outsourced the judgement. The scoring company had automated it. And Louis, who was very much a person with a life and a rental history and a plan, had been translated into a probability distribution that did not flatter her.

The SafeRent settlement established, in a negotiated form, that an algorithmic screening system producing disparate impact against members of a protected class creates liability under Section 3604 of the Fair Housing Act. Even if each of the algorithm's individual inputs is facially neutral, the output is what matters. This aligns with guidance issued by the United States Department of Housing and Urban Development in May 2023, which stated that housing providers and tenant screening companies that use algorithms are not absolved from liability when their practices disproportionately deny people of colour access to housing.

The problem is that guidance is not law, settlements bind only their parties, and the broader regulatory environment for tenant screening remains, in a word, rickety. In theory, the Fair Credit Reporting Act requires that when a tenant is denied housing based on a consumer report, the landlord must provide an adverse action notice naming the reporting company and explaining the right to dispute. In practice, a survey cited by the Center for Democracy and Technology found that only three per cent of renters report knowing the name of the screening company involved in their denial. Three per cent. Which means ninety-seven per cent of denied applicants have been handed a decision that was probably, under existing law, required to come with a full disclosure, and simply did not.

Reddit, predictably, is where the texture of this failure lives. On forums like r/Tenant, r/Renters, and country-specific subs in Canada and the UK, the rejection stories share a grammar. An applicant submits through a national portal. A form email arrives a day or two later, stating that the application has been declined, that the decision is final, and that the landlord cannot share additional information. No name of the scoring company. No numerical score. No reasoning. Sometimes the rejection arrives within minutes, which is the giveaway: no human has looked. The tenant, if persistent, writes to the property manager. The property manager, if they respond, redirects to the screening vendor. The vendor explains that their system is a “decision support” tool and that the landlord made the decision, which the landlord did, in the sense that they clicked accept on the output. The buck is passed, circulated, and finally mislaid.

This is not an anecdote. It is a structural feature. When every participant can point to another as the “real” decider, nobody is responsible. The algorithm is treated as advisory. The landlord is treated as the decision-maker. The vendor is treated as a mere processor. And the tenant, the one actor whose life is materially altered, is treated as an input.

The Retired Couple Problem

Nowhere is the brittleness of algorithmic screening more visible than in a class of applicant that ought to be trivially easy to approve: the asset-rich, income-light retiree. Consider a retired couple in their seventies. They have sold their family home. They have eight hundred thousand pounds in a combination of index funds, a self-invested personal pension, and ISA savings. They receive modest state pension income plus a small private pension. They want to rent a flat, possibly for the rest of their lives. Any sensible human landlord would recognise the applicants as close to ideal. They are not going to miss a payment. Their wealth is documented and liquid in the relevant sense.

An automated income verification system does not see any of this. It sees a bank account with low deposits relative to expected rent. It sees no salary. It sees a 1099-R in the United States, or a pension statement in the United Kingdom, that does not parse as “employment income” in the system's model. It sees “insufficient income to rent ratio,” flags the applicant, and issues a decline. The couple, baffled, may not even learn the reason. If they do, they face the near-impossible task of explaining, through a web form or a call centre, that their liquidity lives in instruments the software does not understand.

This failure mode is not hypothetical. OpenMedia's February 2026 investigation documented multiple instances of retirees and self-employed applicants being denied by automated screening systems that could not ingest asset-based or non-standard income. Applicants on irregular income, seasonal work, gig platforms, disability benefits, family support, or investment distributions find themselves recursively rejected by systems trained on the assumption that income is a W-2 or a PAYE payslip. The effect is to systematically disadvantage anyone whose financial life does not conform to the assumptions of a mid-twentieth-century labour market.

And the effect compounds. Because these screening systems increasingly operate on overlapping datasets, a rejection by one often produces a cascade of similar rejections elsewhere. A tenant declined by a large property management company in a major metropolitan area may find themselves effectively blacklisted across the private rented sector in their region, without ever having done anything wrong, simply because the algorithm could not parse their life. There is no central registry of these decisions. There is no equivalent of a credit bureau dispute process that reliably works for algorithmic tenant scores. There is, in many cases, not even a name to sue.

What OpenMedia Found

The February 2026 OpenMedia report captures what happens when this infrastructure is built out in a jurisdiction with weaker privacy enforcement than the European Union and more fragmented housing law than the United States. OpenMedia's researchers documented the rapid spread of AI landlord tools across the Canadian rental market, naming vendors including CERTN and Single Key. Their marketing materials promise landlords the ability to scan up to seven years of a prospective tenant's social media activity, search more than a hundred databases of personal information including eviction and criminal records, flag “risky” online behaviour, and generate automated risk scores.

The scope of surveillance on offer, even leaving aside questions of accuracy, is extraordinary. A 2018 report by the British Columbia Information and Privacy Commissioner, cited by OpenMedia, found that ten of thirteen landlords studied were already systematically over-collecting sensitive personal information in violation of the province's Personal Information Protection Act. That was before the current generation of AI screening tools existed. Canadian tenants increasingly apply through portals that ingest years of financial history, employment records, social media handles, criminal background, and in some cases access to open banking feeds that reveal every transaction for months.

The OpenMedia report also documents an aspect that rarely makes headlines: automated eviction initiation. Several Canadian AI landlord platforms now offer modules that flag tenants for non-payment risk and, in some configurations, automatically generate and file notice-to-quit paperwork once a threshold is crossed. The property manager is presented with an eviction already drafted. All they need to do is sign off. The frictionlessness is the point. And the frictionlessness is what makes it dangerous.

In the United Kingdom, similar tools are in circulation, though deployment is more fragmented. The UK's regulatory posture is, in principle, stronger: the UK GDPR, which preserves Article 22, gives tenants a right not to be subject to decisions based solely on automated processing that produce legal or similarly significant effects. The Information Commissioner's Office has issued guidance that housing decisions fall squarely within Article 22's scope. In practice, enforcement has been minimal, and the “solely” in “solely automated” has become a loophole: a human clicking an approval button is often treated as meaningful human involvement, even when the human does no independent review. GDPR's promise, in the rental context, has been largely notional.

The Frameworks That Would Need to Exist

If we were to design, from scratch, a legal and ethical framework adequate to algorithmically mediated housing, it would need several overlapping layers. Not one silver bullet. Not one statute. A stack.

The first is algorithmic transparency and the right to an explanation. The model already exists, in imperfect form, in Article 22 of the GDPR and the associated Articles 13 to 15, which grant data subjects a right to meaningful information about the logic involved in automated decisions. In the housing context, any decision to reject an application, to set or raise rent, to initiate eviction, or to classify a tenant as high-risk must be accompanied by an individualised explanation that a reasonable applicant can act on. Not a generic disclaimer. Not “your score was below our threshold.” The features that drove the decision, the weights attached to them, and the means to dispute. California's AB 2930, Colorado's SB 205, and the EU AI Act's high-risk system provisions gesture in this direction. None yet fully deliver.

The second is a modernised adverse action regime. The FCRA, drafted in 1970, was not built for machine learning. Its core insight, that when a consumer is harmed by a report they must be told who provided it and how to dispute it, remains sound. What it needs is an update that treats algorithmic scoring outputs as consumer reports in their own right, regardless of whether the vendor is technically a consumer reporting agency. It needs a private right of action strong enough that violations are actually litigated. And it needs clear rules on data provenance: tenants must be able to know where every input came from, and correct ones that are wrong.

The third is antitrust reform for algorithmic coordination. The Portland ordinance and the New Jersey legislation are, in effect, saying that existing Sherman Act and state antitrust doctrine is not agile enough to handle pricing algorithms that coordinate markets without explicit agreement. They may be right. A federal statute clarifying that the provision or use of pricing software that ingests competitors' non-public data and returns pricing recommendations is per se unlawful, full stop, would resolve most of the doctrinal uncertainty the Ninth Circuit has struggled with.

The fourth is fair housing law modernisation. The Fair Housing Act's disparate impact doctrine, as reaffirmed by HUD's 2023 guidance and the SafeRent settlement, already in principle applies to algorithmic screening. What is missing is pre-deployment obligation: a requirement that any system used to screen tenants be audited for disparate impact before deployment, and periodically thereafter, with results disclosed to regulators and, in summary form, to the public. The EU AI Act has adopted something like this for high-risk systems. US federal law has not.

The fifth is data minimisation and purpose limitation. Why, precisely, does a landlord need seven years of your social media history to decide whether to rent you a flat? They do not. The reason this data is collected is that it is available and that vendors have built products around ingesting it. A defensible regime would restrict the information that can be lawfully used in a tenancy decision to a short, well-justified list: identity, credit history for a defined period, verifiable income or assets, prior eviction judgements. Everything else, social media, location history, network-of-associates data, should be off-limits.

The sixth is a human-in-the-loop mandate with teeth. The current form of “human review” is often performative: a property manager glances at a dashboard and clicks accept. A meaningful mandate would require that any adverse decision, a denial, a rent increase above a threshold, an eviction filing, involve substantive human consideration of the specific circumstances, documented in writing, by a person with authority to override the algorithm. Anything less is GDPR Article 22's “solely automated” dressed in a lab coat.

The seventh is a private right of action. Without one, regulators are the only enforcement mechanism, and regulators are outnumbered, underfunded, and subject to political winds. Tenants harmed by algorithmic decisions must be able to sue, individually or collectively, with the prospect of statutory damages that matter to the vendors' bottom line.

None of this is impossibly difficult to design. All of it is politically difficult to pass.

Why It Has Been So Slow

The lag between the emergence of algorithmic rental infrastructure and the legal response is not accidental. It is the product of several overlapping forces.

The first is lobbying. The real estate technology sector is well capitalised and well connected. RealPage alone has spent substantial sums on lobbying in Washington and in state capitals, and the National Multifamily Housing Council has been an aggressive opponent of algorithmic pricing bans. The Congressional bill that would pre-empt local bans did not arrive by accident. It arrived because the sector organised, wrote briefing papers, cultivated relationships, and showed up.

The second is jurisdictional fragmentation. Housing law in North America is, fundamentally, local. Tenancy rules vary by state, province, county, and city. Fair housing enforcement is divided between federal, state, and municipal bodies with overlapping mandates. Data protection in the United States is a patchwork of sectoral statutes and state laws rather than a comprehensive regime. In Canada, privacy law is split between federal and provincial statutes with varying scope. There is no single venue in which the problem can be addressed.

The third is the “it's just software” framing. Property technology vendors have been remarkably successful at presenting their products as neutral productivity tools, no different in principle from spreadsheets or email. This framing has allowed them to slip past regulatory scrutiny that would have attached to an equivalent human operation. A trade association that circulated pricing data among competing landlords would have been an antitrust target within months. A piece of software that did the same thing operated for over a decade before the DOJ took action.

The fourth is the pace gap between technology and law. Legislatures move slowly. Machine learning systems iterate quickly. By the time a committee has held hearings, drafted a bill, negotiated amendments, and passed a statute, the technology has moved on. In housing, where the human stakes are so direct, the gap is particularly painful.

The fifth, and perhaps most important, is that housing is politically fraught. Every proposed tenant protection collides with a coalition of property owners, developers, landlord associations, and investors who will argue, often successfully, that any new regulation will reduce housing supply and hurt the very people it aims to protect. This argument has merit in some contexts and is made in bad faith in others. The effect, either way, is to make housing reform slow, incremental, and vulnerable to reversal. Banning rent-setting software is presented by its opponents as a form of price control. That the alternative is an effectively unregulated market-wide coordination mechanism does not feature prominently in the industry's press releases.

Housing as a Human Right, and the Asymmetry

Behind the specific technical debates about algorithms and adverse action notices lies a broader ethical question that most regulatory frameworks dance around. If housing is a human right, or at least a precondition for the exercise of other rights, then who gets to live where is not a purely private matter between landlord and tenant. It is a matter in which the state has a legitimate and, arguably, obligatory interest. This framing is common in European human rights jurisprudence and in UN declarations. It is less established in North American constitutional doctrine, where housing is typically treated as a commodity subject to market allocation, moderated by discrimination law and modest subsidies.

The algorithmic turn makes the question urgent. In a market of individual landlords making individual decisions, even when those decisions are discriminatory or arbitrary, the harm is distributed and contestable. A tenant denied by one landlord can try another. The friction of the market is the friction of a thousand independent actors.

In a market mediated by a handful of algorithmic vendors, all of whom share similar data, similar models, and similar blind spots, the friction collapses. Being rejected by one system is being rejected by most. Being flagged as high-risk by one score is being flagged similarly across portfolios. The asymmetry between landlord and tenant, which was always real, becomes qualitatively different when the landlord's side of the relationship is automated, networked, and effectively infinite, while the tenant's side remains one anxious person with a phone and a rejection email.

This is the ethical heart of the matter. The algorithm does not just mediate; it structurally biases the relationship in favour of the side that deployed it. It is not a neutral tool. It is a party. And when it is a party that cannot be questioned, sued effectively, or compelled to explain, the relationship ceases to have the features we normally associate with fair dealing between legal persons. It becomes an administrative system with the power of a landlord and the accountability of a weather pattern.

What the Portland Vote Actually Represents

Which brings us back to that wet Wednesday in November. The Portland ordinance, in the grand sweep of housing policy, is small. It will affect a single city. It carries civil penalties of up to a thousand dollars per violation, which is, by the standards of the entities it regulates, loose change. Its practical impact on rents in Portland will be debated for years.

And yet. The ordinance represents something the regulatory conversation has been missing: a willingness to treat algorithmic infrastructure as a political object rather than a technical inevitability. Portland's council did not hold a symposium on whether large language models will bring about the singularity. They did not convene a blue-ribbon commission to study the metaphysics of automated decision-making. They looked at a specific piece of software, understood what it did, concluded that what it did was unlawful if done by humans, and made it unlawful when done by software. It was, in a sense, a deeply old-fashioned move. It treated the technology as conduct, and conduct as regulable.

New Jersey's March 2026 legislation extends the logic. OpenMedia's February investigation documents the scale of what has been built. HUD's guidance, the SafeRent settlement, the RealPage consent decree, and the various state and municipal ordinances form, together, the rough outline of a response. Not a coherent framework, not yet. But the scaffolding of one.

The question is whether the federal government, or a coalition of states, can assemble the pieces into something that actually protects tenants. The reasons to persist are the stories that show up, daily, in tenant forums and courtrooms and outside the flats that retirees have been refused because a computer could not count their pension.

A decade from now, we will either look back at 2025 and 2026 as the moment the law began to catch up with the infrastructure of rental housing, or as the moment a handful of cities shouted into a wind that kept blowing. Portland voted eight to two. New Jersey's Senate committee voted to advance. OpenMedia published. The DOJ sued and settled. The work, as ever, is in the follow-through. The algorithm is not going to regulate itself. It does not notice whether the person it just declined is Mary Louis, or a retired couple, or someone whose only crime was having the wrong kind of income in the wrong kind of account.

That work, still, belongs to us.

References and Sources

1. City of Portland. “Amend Affordable Housing Code to add prohibition of anti-competitive rental practices including the sale and use of algorithmic devices.” November 2025. portland.gov/council/documents/ordinance/algorithmic-rental-pricing
2. Portland Mercury. “Portland City Council Votes to Adopt AI Rental Price-Fixing Software Ban.” 19 November 2025.
3. US Department of Justice. “Justice Department Requires RealPage to End the Sharing of Competitively Sensitive Information.” 24 November 2025.
4. US Department of Justice. “Justice Department Sues RealPage for Algorithmic Pricing Scheme That Harms Millions of American Renters.” 2024.
5. Reed Smith. “Algorithmic pricing under pressure: DOJ's RealPage settlement changes the rules for rental markets.” 2025.
6. New Jersey State Policy Lab, Rutgers University. “Senate Committee Advances Ban on Rent-Setting Algorithms.” 9 March 2026.
7. Multifamily Dive. “New Jersey mulls algorithmic rent pricing ban.” 2026.
8. City of Hoboken. “City of Hoboken to introduce ordinance prohibiting algorithmic rent-fixing.” 2025.
9. Insider NJ. “Jersey City Council Advances Ordinances that Ban AI-Powered Rent-Fixing Algorithms.” 2025.
10. Arnold & Porter. “Algorithmic Pricing Bans Go Coast to Coast.” October 2025.
11. OpenMedia. “Watch This Space: The Rise of AI Landlords in Canada.” February 2026.
12. American Bar Association, Human Rights Magazine. “Ghosts in the Machine: How Past and Present Biases Haunt Algorithmic Tenant Screening Systems.” June 2024.
13. Center for Democracy and Technology. “Tenant Screening Algorithms Enable Racial and Disability Discrimination at Scale.”
14. Consumer Financial Protection Bureau. “Consumer Snapshot: Tenant Background Checks.” November 2022.
15. The Leadership Conference on Civil and Human Rights. “AI + Tenant Screening.”
16. TechEquity Collaborative. “Traditional vs. algorithmic tenant screening.” 10 July 2024.
17. Georgetown Law Poverty Journal. “The Discriminatory Impacts of AI-Powered Tenant Screening Programs.”
18. American University Business Law Review. “Screened Out: How Faulty Algorithms Are Shutting Doors on Fair Housing.” September 2024.
19. California Law Review. “A Home for Digital Equity: Algorithmic Redlining and Property Technology.”
20. Information Commissioner's Office (UK). “What is the impact of Article 22 of the UK GDPR on fairness?”
21. General Data Protection Regulation. “Article 22: Automated individual decision-making, including profiling.” gdpr-info.eu/art-22-gdpr
22. Gothamist. “Hochul signs bill banning NY landlords from using algorithm software to set rents.” 2025.
23. Freshfields. “Settling Defendants to Pay over $141 million to Settle RealPage Price-Fixing Class Action Claims in Tennessee.” October 2025.
24. Arnold & Porter. “Ninth Circuit Clarifies Antitrust Implications of Algorithmic Pricing.” August 2025.
25. HUD and DOJ joint guidance on algorithm-based tenant screening. May 2023.

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

On a humid July afternoon in Dover, Florida, an 82-year-old grandmother named Sharon Brightwell answered the telephone and heard her daughter crying. The voice was unmistakable. It sobbed, it choked on the words, it begged. There had been a car accident. A pregnant woman had been hurt. A lawyer would call shortly with instructions. Brightwell, who had raised this daughter, who had rocked her to sleep, who knew the exact timbre of her weeping because she had heard it a thousand times across half a century, did what any mother would do. She emptied an envelope of cash. She handed fifteen thousand dollars to a courier who appeared at her door. By the time she realised her daughter had never been in an accident at all, the money and the courier were gone.

The voice she heard was not her daughter. It was a synthetic reconstruction, stitched together by a generative model from audio scraped off the open internet, probably from a social media post, possibly from a voicemail greeting, certainly from nothing more than a few seconds of casual speech. The emotional content, the terror and the tears, was added by the same model as a matter of routine. Running the entire performance cost pennies. Producing a convincing clone of an unsuspecting person's voice, according to every major consumer research organisation that has tested the tools in the last eighteen months, now requires as little as thirty seconds of source material. Some academic demonstrations have done it with three.

That is the hinge on which this story turns, and it is the reason the United States has quietly slipped into one of the strangest fraud epidemics it has ever faced. In 2024, older Americans reported losing almost five billion dollars to scams and fraud, a figure that the Federal Trade Commission itself regards as a dramatic undercount. The agency's most recent report to Congress, issued in December 2025, estimates that the true cost of fraud against Americans aged 60 and over sat somewhere between ten billion and eighty-one and a half billion dollars last year, depending on the model used to correct for the torrent of cases that victims never report out of shame, confusion, or cognitive decline. Within that figure, the fastest-growing category, and by almost every measure the most psychologically ruinous, is the one that depends on cloned voices.

More than 75,000 consumers have now signed a petition urging the FTC to act. A bipartisan Senate bill introduced in December 2025 would criminalise the very act of using AI to impersonate someone with intent to defraud. And in April 2026, the Journal of Accountancy, a publication not generally given to panic, ran a feature instructing certified public accountants in the practical steps they now need to take to protect their elderly clients from being drained by phantom grandchildren. The professional class is catching up to what grandparents already know. Something has gone very wrong with the voice.

A weapon made of pennies

The mechanics are almost insultingly simple. A scammer needs three ingredients: a target, a sample of the target's loved one speaking, and a voice cloning tool. The first two are trivial. Older Americans are plentiful and reachable by phone. Voice samples have been uploaded by the billions to TikTok, Instagram, YouTube, Facebook, and countless wedding videos, podcast appearances, voicemail greetings, Zoom recordings, and church livestreams. The third ingredient used to be the hard part. It no longer is.

Commercial voice synthesis platforms such as ElevenLabs, Speechify, PlayHT, and LOVO have, over the last three years, made voice cloning available to anyone with a credit card and, in many cases, anyone willing to tick a box attesting that they have the legal right to reproduce the voice in question. A March 2025 assessment by Consumer Reports examined six leading voice cloning products and found that four of them, including three of the most widely used, relied exclusively on that self-attestation as their safeguard. The researchers who performed the test were able to clone real voices without providing any evidence of consent. The box was ticked, the clone was generated, the guardrail did nothing.

The FBI's Internet Crime Complaint Center reported that AI-enabled fraud accounted for roughly 893 million dollars in losses across more than 22,000 complaints in 2025, with around 352 million of that total attributed to elder fraud complaints in which an AI component was documented. Those are floors, not ceilings, because most victims do not file complaints and most families do not realise AI was involved when they do. Researchers inside Microsoft's AI for Good Lab, analysing 531,000 fraud reports drawn from AARP's Fraud Watch Network Helpline and the Better Business Bureau's Scam Tracker, reported that scams identified as AI-enabled, with realistic cloned voices or synthetic video, increased twentyfold between 2023 and 2025. Twentyfold. In two years.

The economics are what make it terrifying. A traditional phone scam requires a human operator, a convincing script, a plausible accent, and the nerve to hold a conversation with someone who may ask unexpected questions. Even the best of those operations run on thin margins. A generative voice scam requires none of that. The attacker can produce a bespoke audio deepfake in seconds, run the call through a VoIP provider, spoof the caller ID to match a known family number, and be off the line before a victim even registers that something is wrong. The cost of attacking one thousand targets is barely higher than the cost of attacking one. The cost of attacking ten thousand is not much higher than that.

Why the voice is special

There is a reason voice cloning is working as well as it is, and it has almost nothing to do with technology. It has to do with evolution.

The human auditory system is wired to treat voice as an exceptionally high-trust channel. Infants recognise their mother's voice within days of birth. By the time we reach adulthood, we can identify the voice of a close relative from a single syllable, across decades, through distortion, over a crackling phone line, even when we have not heard it for years. More importantly, voice is tightly coupled to emotion. The acoustic signatures of distress, fear, and pain trigger physiological responses in listeners that bypass conscious deliberation entirely. A mother hearing what she believes is her child crying on the telephone will release a cascade of stress hormones before any rational assessment has begun.

Jennifer DeStefano, a mother in Scottsdale, Arizona, described this experience in searing detail to the United States Senate Judiciary Committee in June 2023. She had received a call while her 15-year-old daughter Brianna was away skiing. She heard Brianna's voice sobbing “Mom!” and then a man's voice demanding a million dollars in ransom. She knew that voice. She had heard it, she testified, her entire life. She would never be able, she told the senators, to shake the sound of those desperate cries out of her mind. Only a coincidence saved her: a bystander handed her a separate phone, on which her actual daughter, safe and confused, was calling. Had that not happened, DeStefano would have wired whatever the callers demanded, because she was not making a financial decision. She was making a biological one.

This is the insight that voice cloning fraud weaponises with ruthless precision. The scams do not target rationality. They target the part of the brain that evolved to respond to a child in danger before the conscious mind has caught up. No amount of public awareness campaigning about “stop, verify, call back” survives first contact with that response, because the response is older than language itself. Older people, who are more likely to live alone, more likely to be isolated from the family members being impersonated, and more likely to have significant savings that can be moved in a single wire transfer, sit at the intersection of every vulnerability the attack model exploits.

Amy Nofziger, director of fraud victim support at AARP, has been telling anyone who will listen that the old mental model of the savvy-versus-gullible victim is obsolete. Her framing, delivered across AARP public briefings and research publications, is that this is not a matter of whether a person is smart enough to spot a scam but whether their nervous system can out-think a sound it has been trained to trust since birth. An AARP survey conducted in August 2024, fielded to a thousand adults aged 50 and over via a probability-based panel, found that 77 per cent of older Americans were concerned about becoming targets of AI-related fraud, and 85 per cent were worried about deepfakes generally. Concern, however, does not translate into immunity. In the same research, respondents massively overestimated their own ability to detect a cloned voice.

The regulatory scramble

Washington has noticed, but not at the speed the problem requires. The Federal Trade Commission finalised its rule on government and business impersonation in early 2024, following years of mounting complaints about scammers posing as the Internal Revenue Service, the Social Security Administration, Amazon customer service, and dozens of other familiar institutions. The rule was a genuine step forward. It gave the commission direct authority to sue impersonators and recover money for victims, and it laid the groundwork for an expansion to cover the impersonation of private individuals, an extension the agency had been seeking.

That expansion is where the fight has stalled. In February 2024, the FTC released a supplemental notice of proposed rulemaking that would cover the impersonation of individuals. Consumer Reports, joined by the Electronic Privacy Information Center, the National Consumers League, and other advocacy organisations, delivered a petition in August 2025 signed by more than 75,000 consumers demanding the commission finalise the individual impersonation rule and invoke its Section 5 powers to pursue the companies whose voice cloning products are enabling the scams in the first place. The petition called on the FTC to investigate the product-design failures, the absence of meaningful safeguards against cloning without consent, and the ease with which commercial voice tools can be turned into engines of fraud. As of April 2026, the rule remains under review. The commission has not yet moved to act against any of the major voice cloning vendors.

Congress has been marginally more active. In April 2025, Senators Chris Coons, Marsha Blackburn, Amy Klobuchar, and Thom Tillis reintroduced the Nurture Originals, Foster Art, and Keep Entertainment Safe Act, universally referred to as the NO FAKES Act. Originally conceived as a tool to protect musicians, actors, and other creative professionals from having their voices and likenesses replicated without consent, the bill would establish a federal right for every American to their own voice and visual likeness, create a notice-and-takedown regime for unauthorised deepfakes, and preempt the patchwork of state laws that has grown up in the meantime. The legislation has drawn support from SAG-AFTRA, the Recording Industry Association of America, the Motion Picture Association, OpenAI, and YouTube, which is an unusual coalition by any measure.

It has also drawn criticism. The Foundation for Individual Rights and Expression has argued that the bill's takedown provisions could be abused to suppress legitimate speech, and the Electronic Frontier Foundation has raised concerns about the scope of secondary liability for platforms. The result is that the NO FAKES Act, now in its third legislative cycle, remains unpassed.

More narrowly focused is the Preventing Deep Fake Scams Act, introduced by Senators Jon Husted and Raphael Warnock in June 2025 and endorsed by AARP, which would establish a federal task force led by the Treasury Department and financial regulators to coordinate a response to AI-driven fraud against financial institutions and their customers. And on 17 December 2025, Senators Shelley Moore Capito and Amy Klobuchar introduced the Artificial Intelligence Scam Prevention Act, a bipartisan bill that would make it illegal to use AI to impersonate any person with intent to defraud, and would establish an interagency committee bringing together the FTC, the FCC, the Consumer Financial Protection Bureau, and the Justice Department to coordinate enforcement.

None of these bills has yet become law. All of them, read together, suggest that even the most energetic legislators understand the problem to be growing faster than the legislative response. The April 2026 issue of the Journal of Accountancy, in a feature authored by forensic accountants David Zweighaft and Howard Silverstone, framed the matter in terms the professional services industry cannot ignore. Certified public accountants, the article argued, now have a practical obligation to warn elderly clients about voice cloning, to help them establish family verification codewords, and to build transaction-review processes that can flag urgent-sounding wire requests before the money leaves the account. It was a notice, delivered to a constituency that does not panic easily, that this is no longer hypothetical.

Defences out of pace

The technical defences against voice cloning fraud fall into three categories, and in April 2026, none of them works reliably.

The first category is detection. Academic and industry researchers have, over the last two years, produced a growing literature on audio deepfake detection, using machine-learning classifiers trained to distinguish synthesised speech from natural speech. A systematic 2025 analysis published in the ACM Transactions on Internet Technology, along with a peer-reviewed survey appearing in Engineering Reports, concluded that detection models perform well on the datasets they were trained on and collapse, sometimes catastrophically, when confronted with audio generated by unseen models. A paper presented at the Network and Distributed System Security Symposium in 2025, introducing a system called VoiceRadar, demonstrated that micro-variations in synthetic speech can be detected under controlled conditions but noted that adversarial retraining by attackers can neutralise those signals within weeks. More disturbingly, a 2025 study on synthetic speech detection reported that current systems exhibit demographic bias, with markedly higher false-positive rates for elderly speakers, adolescents, and speakers of certain English dialects, precisely the populations most often impersonated or most often targeted.

In practical terms, this means that by the time a consumer-grade voice deepfake arrives on someone's phone, no widely deployed tool can reliably tell them it is fake. Phone carriers do not scan audio content in real time. Call authentication protocols such as STIR/SHAKEN, which the Federal Communications Commission mandated across the United States telecom industry to combat robocalling, verify the origin and legitimacy of a calling number but say nothing about whether the voice on the other end of a legitimately placed call is human, synthetic, or stolen. STIR/SHAKEN was built for a world before generative AI. In that world, if you knew who was calling, you had a reasonable chance of knowing what they were going to say. That assumption no longer holds.

The second category is provenance. The Coalition for Content Provenance and Authenticity, known as C2PA, has since 2021 been building an open technical standard for cryptographically signing media at the point of creation so that downstream consumers can verify its origin and editing history. A new version of the specification was published in May 2025, and the standard is expected to be adopted as an ISO international standard this year. Microsoft, Adobe, Intel, Google, and several major news organisations have committed to implementing it. In principle, a C2PA-signed audio file can be traced back to its source, and synthetic audio generated by a compliant tool can be flagged as such.

In practice, provenance solves a different problem. It works beautifully for controlled creative pipelines, where a tool like a professional video editor or a generative image app stamps its output with verifiable credentials. It is much less effective against the scam caller who records a voice from a TikTok clip, strips the metadata, feeds it into a compliant or non-compliant voice model, and then pipes the output through a telephony gateway that was never designed to preserve signed audio. The telephone network, the channel through which the overwhelming majority of voice fraud is delivered, strips content credentials by design. Provenance works upstream. Fraud happens downstream.

The third category is authentication at the human level, which is the category most likely to actually help in the short term. The single most consistently effective defence that every major fraud researcher, from the FBI to AARP to the Journal of Accountancy to Consumer Reports, now recommends, is the family verification codeword: a simple, private phrase shared among relatives and used to confirm identity in emergencies. If a grandmother receives a call from her weeping grandson, she asks for the codeword. The real grandson knows it. The fraudster does not. This is, in the end, pre-industrial cryptography, a shared secret used to verify identity in a world where the cryptographic infrastructure we have built cannot reach the kitchen telephone.

Who is supposed to stop this?

The hardest question is not technical. It is the question of responsibility, and the answer involves six constituencies that have historically preferred to pass the problem to one another.

The voice model developers, companies such as ElevenLabs, LOVO, Speechify, PlayHT, and the large foundation-model labs whose capabilities underpin all of them, are the closest to the attack surface. The product can do what it can do because they built it that way. Consumer Reports' March 2025 findings were unambiguous: the majority of commercial voice cloning products tested lacked meaningful safeguards against unauthorised cloning. Some, like Resemble AI and Descript, required more than a ticked box. Most did not. Senator Maggie Hassan, in formal letters sent to ElevenLabs and three competitors in April 2026, asked each company to explain exactly how they prevent their tools from being used for fraud. The replies have been limited. ElevenLabs, to its credit, blocks the cloning of certain high-profile celebrity voices and uses internal classifiers to monitor for misuse, but the systems are imperfect and the incentives to prioritise growth over gatekeeping are, for a venture-funded company, almost irresistible.

The platforms that host voice samples are the second constituency. TikTok, Meta, YouTube, and their peers have built their business models around the frictionless sharing of audiovisual content, including audio of children, grandparents, and other family members who have no knowledge that their voices are being harvested. Default privacy settings on most of these platforms are permissive. Third-party scraping tools are widely available. No major social platform has yet committed to audio-scraping countermeasures or to robust default privacy settings that would shield users' voices from mass harvesting. Until they do, the raw material for every cloning scam will remain trivially accessible.

The telecommunications carriers are the third. They are the pipeline through which every fraudulent call travels. Their current fraud-prevention investments focus on number-level authentication, not content-level scrutiny. A content-aware defence, one that could scan incoming audio for markers of synthesis in real time, is technically plausible but would raise serious questions about call-content surveillance and would require regulatory scaffolding the United States does not currently possess. In the absence of such a framework, carriers remain, essentially, neutral conduits. The fraud flows through them as surely as the legitimate calls do.

The banks and wire-transfer services sit at the fourth position, and they are arguably the most capable of intervening because they are the last line before the money is gone. Every large wire transfer, every emergency cash pickup, every cryptocurrency exchange purchase executed in a single afternoon by a panicked elderly customer is a signal. Some banks have built transaction-monitoring systems that flag such patterns, and some branch staff have been trained to ask verification questions when a long-standing customer suddenly demands a large cash withdrawal for an unknown recipient. The bank teller in the Canadian case cited by CBC News, who stopped a grandmother from wiring nine thousand dollars to a claimed kidnapper, is exactly the kind of intervention point that works. But training is inconsistent, oversight is voluntary, and liability for downstream losses remains, in most US jurisdictions, the customer's.

The regulators are the fifth. The FTC, the FCC, the Consumer Financial Protection Bureau, the Securities and Exchange Commission, and state attorneys general all have jurisdictional claims somewhere in the fraud chain. None of them has yet been given the explicit statutory authority, the budget, or the coordinated mandate required to act as a unified response. The interagency committee envisioned by the December 2025 Capito-Klobuchar bill, if it becomes law, would be a start. Whether it would be adequate to a problem that compounds at twentyfold every two years is another question.

The sixth constituency is families themselves, and this is the uncomfortable part. The codeword, the verification call-back, the insistence on privacy settings, the education of elderly relatives about what is possible and what to watch for, all of this is work that falls on people who did not ask to become the first line of defence against a technology they do not fully understand. Shifting the burden of fraud prevention onto victims is, historically, the signature of a policy failure. And yet the codeword, today, works. No legislation, no regulator, no platform commitment currently works as reliably.

The trust question

There is a larger question beneath all of this, and it does not reduce cleanly to policy. It is a question about what happens to intergenerational trust in a world where the sound of a loved one's voice can no longer be presumed to be the loved one.

For the whole of human history, voice has been what philosophers have called a “warrant” of presence. Hearing someone's voice was evidence, perhaps the oldest form of evidence, that they were there, that they were real, that they were who they claimed to be. This assumption underwrote almost every important human relationship that was not conducted face to face. It underwrote the telephone as a technology, the voicemail as a convenience, the grandparent-grandchild call as a ritual. It was the reason, in the end, that voice scams worked at all. You believed the voice because voice was something that could be believed.

That assumption is now being eroded in real time, and the erosion is happening unevenly. Younger people, who have grown up around voice filters, autotune, AI-generated podcasts, and synthetic TikTok audio, have developed a generalised scepticism that older generations have not had the time or cultural context to acquire. An older person who has spent sixty years treating the telephone as a trusted channel cannot retool that instinct in an afternoon. The asymmetry is precisely what the attackers exploit.

What gets lost, if the trend continues, is not just money. It is a feature of family life that has existed since the invention of the telephone: the ability to hear a grandchild's voice and to know, instantly and without effort, that it is the grandchild. If every such call must now be preceded by a verification protocol, if every loved voice must be met with a reflexive “what is our codeword?”, something has changed about what it means to be in a family. The burden of suspicion, which used to live on the perimeter of our lives, has migrated inward.

AARP and its peers have tried, carefully, to describe this shift without causing the very panic that would make older adults withdraw from the phone altogether. Their advice in 2026 is consistent: have the codeword conversation, practise it, make it a ritual, do not treat it as a sign of mistrust but as a sign of love in a changed landscape. The framing matters. If the codeword is understood as a form of hygiene, like locking the front door, it becomes bearable. If it is understood as a sign that the voice itself can no longer be trusted, it becomes something else. It becomes an admission of defeat.

What effective protection would actually require

A serious response, one proportionate to the scale of the threat, would combine technical, legal, social, and platform-level interventions. It would begin at the model layer, where voice cloning companies would be required, by binding regulation, to implement robust consent-verification before cloning any voice, to watermark their output in forms that survive normal telephony transmission, and to maintain traceable provenance records accessible to law enforcement. Self-attestation is not a safeguard. It is a liability shield dressed up as a safety feature.

It would continue at the platform layer, where social networks hosting audio content would default to privacy settings that prevent mass scraping of users' voices, particularly those of minors, and would provide users with tools to audit and restrict the use of their voice recordings. This is not a speech issue. It is an infrastructure issue. The scale at which audio can currently be harvested from public platforms was not the intent of any of the people who uploaded that audio.

It would extend to the telecommunications layer, where carriers would be required to develop and deploy content-aware fraud-detection capabilities, with regulatory frameworks that make clear what such systems can and cannot do. STIR/SHAKEN is not enough. A protocol that authenticates numbers without authenticating voices is a partial answer to a problem that has fully evolved.

It would impose liability at the financial-services layer, making banks and wire services responsible for detecting and halting transactions that bear the signature of urgent-request fraud, and giving victims clear legal recourse when obvious warning signs are missed. It would give the FTC the final authority it has been asking for to pursue the individual-impersonation rule and to act against the product manufacturers whose tools are being used. It would pass the Artificial Intelligence Scam Prevention Act, or something like it, to create the federal criminal prohibition that does not currently exist. And it would fund, with federal dollars, the awareness and codeword-education campaigns that are currently being run, on shoestring budgets, by AARP and a handful of consumer advocacy organisations.

None of this is impossible. Most of it has been proposed, some of it in serious detail, in legislation already pending in the 119th Congress. What is missing is not the blueprint. What is missing is the velocity. The gap between the speed at which generative voice technology is proliferating and the speed at which the country's regulatory and platform responses are arriving is, right now, widening. Every additional month in which the FTC does not finalise its individual-impersonation rule, in which no federal statute criminalises AI-driven impersonation fraud, in which major voice platforms continue to rely on tick-box consent, in which social networks continue to default to public audio sharing, is a month in which more Sharon Brightwells empty envelopes of cash to couriers they will never see again.

The mechanism of the scam is new. The dilemma is old. A society has to decide how much of the burden of new technological harm it is prepared to place on the people least equipped to bear it, and how much it is prepared to impose on the institutions that built, distributed, and profited from the tools in the first place. So far, the distribution has been badly skewed. The voice cloners are paying pennies. The victims are paying with their savings, their dignity, and, in some cases, their ability to ever again pick up the telephone and believe what they hear.

If effective protection is possible, and it is, it will look like a deliberate rebalancing of that ledger. It will involve regulators willing to act before every piece of evidence is in, platforms willing to inconvenience their growth curves, carriers willing to be more than neutral, banks willing to own the moment of the transfer, model developers willing to build products that refuse to do the most dangerous things they are capable of doing, and families willing, as they have always been willing when pressed, to protect one another with the tools they have. The codeword is a start. It is not a strategy. The strategy is still, in April 2026, being written, and the clock is running at the speed of the next cloned voice.

References

1. Federal Trade Commission, Protecting Older Consumers 2024-2025: A Report of the Federal Trade Commission, December 2025. https://www.ftc.gov/system/files/ftc_gov/pdf/P144400-OlderAdultsReportDec2025.pdf
2. AARP, “$12.5 Billion Lost to Scams and Fraud in 2024, Older Adults Hit Hard,” 2025. https://www.aarp.org/money/scams-fraud/older-adults-ftc-fraud-report/
3. Greg Iacurci, “Financial fraud cost older adults up to $81.5 billion in 2024, FTC estimates,” CNBC, 13 December 2025. https://www.cnbc.com/2025/12/13/financial-fraud-seniors-ftc.html
4. Consumer Reports, “More than 75,000 consumers urge FTC to crack down on AI voice cloning fraud,” press release, August 2025. https://advocacy.consumerreports.org/press_release/more-than-75000-consumers-urge-ftc-to-crack-down-on-ai-voice-cloning-fraud/
5. Consumer Reports, “Consumer Reports' Assessment of AI Voice Cloning Products,” press release, March 2025. https://www.consumerreports.org/media-room/press-releases/2025/03/consumer-reports-assessment-of-ai-voice-cloning-products/
6. Federal Trade Commission, “FTC Proposes New Protections to Combat AI Impersonation of Individuals,” press release, February 2024. https://www.ftc.gov/news-events/news/press-releases/2024/02/ftc-proposes-new-protections-combat-ai-impersonation-individuals
7. Federal Trade Commission, “Approaches to Address AI-enabled Voice Cloning,” April 2024. https://www.ftc.gov/policy/advocacy-research/tech-at-ftc/2024/04/approaches-address-ai-enabled-voice-cloning
8. U.S. Senator Amy Klobuchar, “Klobuchar, Coons, Blackburn and Colleagues Reintroduce Bipartisan NO FAKES Act,” press release, April 2025. https://www.klobuchar.senate.gov/public/index.cfm/2025/4/klobuchar-coons-blackburn-and-colleagues-reintroduce-bipartisan-no-fakes-act
9. H.R.2794, 119th Congress (2025-2026), NO FAKES Act of 2025. https://www.congress.gov/bill/119th-congress/house-bill/2794/text
10. S.2117, 119th Congress (2025-2026), Preventing Deep Fake Scams Act. https://www.congress.gov/bill/119th-congress/senate-bill/2117/text
11. U.S. Senator Amy Klobuchar, “Klobuchar, Capito Introduce Bipartisan Artificial Intelligence Scam Prevention Act,” press release, 17 December 2025. https://www.klobuchar.senate.gov/public/index.cfm/2025/12/klobuchar-capito-introduce-bipartisan-artificial-intelligence-scam-prevention-act
12. AARP, “AARP Endorsement of Preventing Deep Fake Scams Act,” 2025. https://www.husted.senate.gov/wp-content/uploads/2025/06/AARP-Endorsement-of-Preventing-Deep-Fake-Scams-Act-119th-Senate1.pdf
13. David Zweighaft and Howard Silverstone, “Elder fraud rises as scammers use AI,” Journal of Accountancy, April 2026. https://www.journalofaccountancy.com/issues/2026/apr/elder-fraud-rises-as-scammers-use-ai/
14. Jennifer DeStefano, “Written Statement to the United States Senate Committee on the Judiciary,” 13 June 2023. https://www.judiciary.senate.gov/imo/media/doc/2023-06-13%20PM%20-%20Testimony%20-%20DeStefano.pdf
15. AARP, “AI-Powered Scams Make Fraud Even Harder to Spot,” 2025. https://www.aarp.org/money/scams-fraud/detecting-ai-fraud/
16. Biometric Update, “Voice cloning tools give rise to cacophony of impersonation fraud,” August 2025. https://www.biometricupdate.com/202508/voice-cloning-tools-give-rise-to-cacophony-of-impersonation-fraud
17. ElevenLabs, “Safety.” https://elevenlabs.io/safety
18. CBC Marketplace, “How con artists are using AI voice cloning to upgrade the grandparent scam,” 2025. https://www.cbc.ca/news/marketplace/marketplace-ai-voice-scam-1.7486437
19. American Bar Association Senior Lawyers Division, “The Rise of the AI-Cloned Voice Scam,” September 2025. https://www.americanbar.org/groups/senior_lawyers/resources/voice-of-experience/2025-september/ai-cloned-voice-scam/
20. Coalition for Content Provenance and Authenticity, Content Credentials White Paper, October 2025. https://c2pa.org/wp-content/uploads/sites/33/2025/10/content_credentials_wp_0925.pdf
21. National Security Agency, “Strengthening Multimedia Integrity in the Generative AI Era,” January 2025. https://media.defense.gov/2025/Jan/29/2003634788/-1/-1/0/CSI-CONTENT-CREDENTIALS.PDF
22. “Where are We in Audio Deepfake Detection? A Systematic Analysis over Generative and Detection Models,” ACM Transactions on Internet Technology, 2025. https://dl.acm.org/doi/10.1145/3736765
23. H. Shaaban et al., “Audio Deepfake Detection Using Deep Learning,” Engineering Reports, 2025. https://onlinelibrary.wiley.com/doi/full/10.1002/eng2.70087
24. Network and Distributed System Security Symposium, “VoiceRadar: Voice Deepfake Detection using Micro-frequency Variations,” 2025. https://www.ndss-symposium.org/wp-content/uploads/2025-3389-paper.pdf
25. Federal Communications Commission, “Promoting Caller ID Authentication to Combat Illegal Robocalls.” https://docs.fcc.gov/public/attachments/DOC-366783A1.pdf

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...