SmarterArticles

The software on the slide looked like something a sales team would buy. There was a dashboard. There were contact targets for the day. There was a pipeline that tracked each prospect from first message to final payment, and an assistant that suggested the next reply. Eric Huber, senior manager of adversary intelligence and disruption at TD Bank, was showing it to an audience at Black Hat USA 2026 in Las Vegas in August. The prospects in the pipeline were fraud victims. The workers hitting their daily quotas were, in many cases, trafficked people held in scam compounds in Cambodia and Myanmar, who faced punishment if they fell short.

One of the tools Huber demonstrated, according to reporting by Joel Witts at Expert Insights, was a reply generator called Editone. An operator loads a victim's profile alongside a fabricated persona, and the software drafts responses designed to land. It has voice cloning built in. Translation software lets a worker who speaks one or two languages work a target in a third. Tara Seals of Dark Reading described another kit, ProKYC, used by Nigerian fraud rings to defeat banks' identity checks by generating forged “selfie with ID” images stamped with fake location metadata, so a picture taken in Nigeria appears to come from Sydney. Her headline summed up the session: AI had sent global crime syndicates into “fraud nirvana”.

Huber was careful about one point, and it matters more than the headline. The tools, he said, do not automate the scam. They “help some poor person do the scamming by creating responses that are going to be the most effective.” What he described was not a robot that defrauds people by itself. It was a factory: a production line in which human labour, software and management systems are arranged to maximise output per worker, and in which each new piece of AI removes another bottleneck.

That distinction reframes the question people actually ask. When a voice that sounds exactly like your son calls in tears, or a colleague's face appears on a video call asking for an urgent transfer, or a stranger online seems to understand you better than anyone has for years, how do you tell the real from the manufactured? The uncomfortable answer from the research of the past year is that, in most cases, you cannot, at least not by looking and listening. The more useful answer is that you were never supposed to have to. Factories are beaten by changing their economics, not by asking each customer to inspect the product more closely.

A Production Line Built on Conversation

To understand why AI matters to fraud, start with what fraud work consists of. The most detailed answer comes from a study posted to arXiv in December 2025 by Gilad Gressel, Rahul Pankajakshan, Shir Rozenfeld, Ling Li, Ivan Franceschini, Krishnashree Achuthan and Yisroel Mirsky, titled “Love, Lies, and Language Models: Investigating AI's Role in Romance-Baiting Scams”. The researchers interviewed 145 insiders and five victims. Their central finding was that 87 per cent of scam labour consisted of what they called “systematized conversational tasks”, the scripted, repetitive work of building rapport over weeks of text messages before steering a target towards a fraudulent investment.

That figure is the key to the whole economy. A scam compound is, in effect, a call centre whose product is trust, and most of its costs are the hours it takes to produce that trust one conversation at a time. The team then tested what happens when a machine does the talking. In a week-long blinded study, a language-model agent elicited more trust from participants than human operators did, and achieved higher compliance with requests: 46 per cent against 18 per cent for the humans. Commercial safety filters, run over the romance-baiting dialogues, flagged 0.0 per cent of them. The authors concluded that these scams “may be amenable to full-scale LLM automation”.

International law enforcement has reached a similar conclusion. In March 2026, INTERPOL published its second Global Financial Fraud Threat Assessment, timed for its Global Fraud Summit. The organisation reported that AI-enhanced fraud is 4.5 times more profitable than traditional methods, and that fraud-related INTERPOL Notices and Diffusions, the alerts member countries use to flag suspects and methods, had risen 54 per cent since 2024. It warned that agentic AI systems can now run complete fraud campaigns, from reconnaissance to ransom demands. It described hybrid tactics in which sextortion is built into romance and investment scams using scripts and AI-generated content, and scam centres that were once regional now operating worldwide and involving hundreds of thousands of people, many of them trafficked. Valdecy Urquiza, INTERPOL's Secretary General, put it in a single phrase: “we are witnessing the industrialization of fraud.”

The United Nations Office on Drugs and Crime, in a threat assessment launched in Bangkok in July 2026, estimated scam losses across East and South-East Asia, Australia and New Zealand at between $88.3 billion and $114.1 billion in 2025. It documented the use of generative AI, deepfakes and automated fraud, and found people from at least 80 countries and territories inside the region's compounds. The geography of the labour force and the geography of the victims are both global.

Academic work has started to describe the same shift in its own terms. A January 2026 paper by Heedou Kim, Changsik Kim, Sanghwa Shin and Jaewoo Kang, which built a detection system called ScriptMind from 571 real Korean phone scam cases, observes in its introduction that “with advances in LLMs and generative AI, such scams have evolved into large-scale, organized, and multi-channel attacks.” A survey of AI-driven cybersecurity threats by Sai Teja Erukude, Viswa Chaitanya Marella and Suhasnadh Reddy Veluru, also posted in January 2026, puts it more bluntly: “The real danger of AI in cybercrime is that it's now possible to automate complex and large-scale scams.”

The Arithmetic of a Five-Minute Phone Call

Factories exist because of unit economics, and the most useful recent paper on AI fraud is an exercise in accounting. “Evaluating AI Models' Capability to Automate Voice Phishing Attacks”, by Fred Heiding of Harvard Kennedy School and colleagues including Claudio Mayrink Verdun, Simon Lermen, Andrew Kao and researchers at Meta, was posted to arXiv in July 2026 and published in the journal Expert Systems With Applications.

The team ran a survey experiment with 4,100 American adults, exposing them to recordings or transcripts of scam calls voiced by leading AI voice models, including systems from Google, OpenAI, ElevenLabs and Sesame, and comparing them with human-voiced versions. The overall compliance rate across five scam types was 16.5 per cent. In the “relative in distress” scenario, the one that maps onto the classic family emergency call, up to 36 per cent of participants said they would or might comply. Caller persuasiveness was the strongest predictor of compliance, and some synthetic voices, Sesame's in particular, were rated as persuasive as human ones or slightly more so.

Then came the accounting. A human vishing operator, calibrated at the average American non-farm wage of $34.55 an hour, handles about twelve five-minute attempts an hour. Assume the scammer extracts $450 from each successful victim, and that only 0.6 per cent of persuaded people actually pay, the lowest conversion rate the authors could find in any legitimate industry. On those assumptions, a human-run vishing operation at American wages loses around $27 an hour. Some AI models also lose money because inference is expensive. But others, including Gemini, Sesame and ElevenLabs, came out with positive expected profits. The authors estimated that building such a pipeline takes around 260 hours of engineering time. Their conclusion is the sentence that has travelled furthest from the paper: “even conservative estimates suggest that AI-automated vishing is economically viable at scales accessible to individual bad actors, not just organized crime syndicates.”

Precision matters here. The authors call AI vishing “roughly break-even” across the models they surveyed, not wildly profitable, and they found that targeting particular demographic groups was not yet worth its cost. So the popular shorthand, that the barrier to industrial fraud has fallen to zero, overstates the evidence. The real finding is subtler and more worrying. The constraint that used to cap phone fraud was human time. Each operator could hold one conversation at a time, and in a high-wage country that time cost more than the fraud returned. AI removes the constraint. As the paper notes, AI agents “can conduct thousands of concurrent calls, limited mainly by telephony and network capacities rather than human capacity.” The authors call the result an “automation dividend”: a situation in which “even small per-call success probabilities translate into large aggregate losses.”

Researchers outside the paper say the same thing in plainer language. In February 2026, the Guardian's Aisha Down reported on an analysis of the AI Incident Database, which found that frauds, scams and targeted manipulation had been the largest category of incidents reported to the database in 11 of the previous 12 months. Simon Mylius, an MIT researcher who works on a project linked to the database, told the paper that “capabilities have suddenly reached that level where fake content can be produced by pretty much anybody,” and that “there is really effectively no barrier to entry.” Heiding, quoted in the same piece, said: “It's becoming so cheap, almost anyone can use it now.”

Why Volume Beats Brilliance

The factory framing explains an apparent paradox. AI fraud does not have to be brilliant. It has to be good enough, cheap enough and plentiful enough.

A traditional con artist needed a high hit rate because each attempt cost hours of skilled work. A factory needs only a positive expected value per attempt, multiplied by a very large number of attempts. A 16.5 per cent compliance rate, or even a fraction of one per cent conversion to actual payment, is a disaster for an individual criminal working by hand. For an automated system running thousands of calls at once, around the clock, in dozens of languages, it is a business model. The Heiding team's economics deliberately assumed that AI scales no better than humans, which they acknowledge “is likely to underestimate the reach of AI-automated vishing”.

This is why the losses keep climbing. The FBI's Internet Crime Complaint Center reported 1,008,597 complaints in 2025, the first year it crossed one million, and reported losses of $20.877 billion, up 26 per cent on 2024. Investment fraud alone accounted for $8.6 billion. According to a Department of Justice release in April 2026, losses in the category of investment fraud the department ties to South-East Asian scam centres rose 24 per cent in 2025 to more than $7.2 billion.

The FBI also counted more than 22,000 complaints with an AI connection, with associated losses of nearly $900 million. That number is likely an undercount, for a reason that goes to the heart of the problem: victims usually have no way of knowing whether a message, a voice or a face was machine-made. Fraud built on synthetic trust leaves no fingerprint that the person defrauded can see. Statistics on “AI-enabled fraud” therefore measure the cases where AI was obvious, not the cases where it worked.

None of the technologies involved is exotic. Voice synthesis, real-time video generation and conversational language models are the same tools that power customer service bots, dubbing software and AI companions. The fraud factory does not need a research laboratory. It rents the consumer technology stack and points it at a different goal.

Your Senses Were Never the Security System

The standard advice to the public has long rested on perception. Listen for the robotic cadence. Watch for the blink that comes too late, the hands that look wrong, the shadow that falls the wrong way. Even the FBI's December 2024 public service announcement on generative AI fraud advised people to “look for subtle imperfections in images and videos, such as distorted hands or feet, unrealistic teeth or eyes”, and to “listen closely to the tone and word choice” to tell a loved one from a cloned voice.

The evidence on whether people can do this is not encouraging. In 2024, Alexander Diel and colleagues published a systematic review and meta-analysis of 56 papers on human deepfake detection in Computers in Human Behavior Reports, pooling results from more than 86,000 participants. Total detection accuracy was 55.54 per cent. Because the confidence interval crossed 50 per cent, the authors concluded that overall detection was not significantly better than chance. Audio was the easiest format at 62.08 per cent; images and text were barely distinguishable from guessing. An earlier study in iScience by Nils Köbis, Barbora Doležalová and Ivan Soraperra summed up the psychological trap in its title: “Fooled twice: People cannot detect deepfakes but think they can.”

Muhammad Tahir Ashraf, a practitioner and technology entrepreneur, made the consequence explicit in an April 2026 arXiv paper titled “Synthetic Trust Attacks”. “Generative AI is the force-multiplier behind this surge,” he writes, “enabling voice cloning, video deepfakes, and LLM-driven conversations that scale social engineering attacks to industrial speed.” His core argument is that existing defences focus on detecting synthetic media, while “the real attack surface is the victim's decision.” Citing the detection figures and the 46 versus 18 per cent compliance gap, he concludes that “the perception layer has already failed.” Ashraf's framework is a model rather than an empirical study, but its diagnosis lines up with the data.

The clearest real-world illustration remains the case Ashraf opens with. In early 2024, an employee in the Hong Kong office of the British engineering firm Arup joined a video call with people who appeared to be the company's chief financial officer and other colleagues. All of them were synthetic. The employee made 15 transfers totalling HK$200 million, about US$25.6 million, according to Hong Kong police as reported by CNN. Rob Greig, Arup's global chief information officer, later told the World Economic Forum that this was not a cyberattack in the usual sense, because no system was breached. “What happened at Arup,” he said, “I would call it technology-enhanced social engineering.” Nothing was hacked except a person's reasonable belief that the faces on the screen were real.

If perception has failed, then asking individuals to look harder is not a defence. It is a transfer of risk to the person least able to bear it.

The Executive Who Asked About a Book

There is a better model, and some of the most instructive examples come from people who were not fooled.

In July 2024, Bloomberg reported that an executive at Ferrari had received a series of WhatsApp messages, from an unfamiliar number, purporting to come from the chief executive, Benedetto Vigna. They described a confidential acquisition. A phone call followed, in a voice that imitated Vigna's southern Italian accent convincingly. The executive grew suspicious and said, according to the report: “Sorry, Benedetto, but I need to identify you.” He asked for the title of a book Vigna had recommended to him days earlier. The call ended abruptly.

That executive did not detect a deepfake. He did not spot a glitch or a wrong vowel. He applied a procedure: a challenge whose answer existed only in a shared history that the impersonator could not access. That is the crucial shift. A procedure works whether or not the fake is perfect. Perception works only when the fake is flawed, and the factories are paid to remove the flaws.

The second example shows what happens when procedure fails at the individual level but works at the system level. On 24 March 2025, a finance director at a multinational company in Singapore was contacted on WhatsApp by someone posing as the company's chief financial officer. A purported lawyer called to stress confidentiality, and the director signed a non-disclosure agreement. On a Zoom call the next day, he saw the chief executive and other executives, impersonated with deepfakes. On 26 March he transferred more than US$499,000 from the company's HSBC account into what turned out to be a money mule account. The scam came to light only when the fraudsters asked for another US$1.4 million. He alerted HSBC, which immediately notified Singapore's Anti-Scam Centre. By 28 March, working with the Anti-Deception Coordination Centre of the Hong Kong Police Force, the authorities had withheld the full amount that had been moved on to Hong Kong, more than US$494,000, and seized the remainder locally.

The director was fooled. The money was recovered anyway, because a bank, a national anti-scam centre and a foreign police unit could act within days across a border. The factory beat the person. It did not beat the system.

For individuals, the lesson is to replace judgement with routine wherever money or credentials are involved. The FBI's own list contains the useful parts: “Create a secret word or phrase with your family to verify their identity,” and verify a caller “by hanging up the phone, researching the contact of the bank or organization purporting to call you, and call the phone number directly.” The value of these steps is that they are channel-independent. A cloned voice cannot know a code word agreed at the kitchen table. A deepfaked colleague cannot intercept a call you place to a number you already hold. A romance that has never moved off one app, never met in a verifiable way and eventually turns to an “opportunity” fails a procedural test long before it fails a perceptual one.

Ashraf's paper packages similar advice as “Calm, Check, Confirm”, a protocol aimed at the moment of decision. The names matter less than the principle. Urgency, secrecy and a new channel are the three pressures that recur across the Arup, Ferrari and Singapore cases, and they recur because they are designed to stop the victim from running a procedure. Any request that combines them should trigger one automatically, however real the voice or face seems.

But these steps have limits. They protect people who set them up in advance, which disproportionately means the organised and the informed. They do little for someone isolated, someone in crisis, or someone whose new romantic interest has spent six weeks becoming the most attentive presence in their life. A defence that depends on billions of people behaving like a well-briefed Ferrari executive is not a defence against a factory. It is a lottery with better odds for some players.

Making the Bill Land Where the Fix Is

If individual vigilance cannot carry the load, the question becomes who can, and how to make them want to. The most promising answer of the past two years has been to move the cost of fraud onto the institutions that can change the factory's economics.

The United Kingdom made the boldest bet. Since 7 October 2024, the Payment Systems Regulator has required payment firms to reimburse most victims of authorised push payment fraud, the kind in which a victim is tricked into sending money themselves, up to £85,000, with the cost split between the sending and receiving firms. In the first full year, to the end of September 2025, the regulator reported that £173 million had been returned, 88 per cent of the money lost to in-scope scams. Only about three per cent of claims were rejected on the grounds that the customer had not taken enough care. More important for the factory argument, an independent evaluation published by the regulator in July 2026 found that APP fraud losses over Faster Payments fell by around 21 per cent after the requirement came in, a reduction of about £73 million a year, and that the rules had strengthened firms' incentives to invest in prevention. Reimbursement does more than compensate victims. When banks pay for fraud, they have a reason to hunt mule accounts, slow suspicious transfers and share the intelligence that makes fraud less profitable.

Singapore went further up the chain. Its Shared Responsibility Framework, developed by the Monetary Authority of Singapore and the Infocomm Media Development Authority and in force since 16 December 2024, assigns specific anti-phishing duties to banks and to telecoms operators, from real-time fraud surveillance to blocking unauthorised SMS sender IDs and filtering messages containing malicious links. Losses then flow down a “waterfall”: a bank that has breached its duties bears the loss first; if it has not, a telco that has breached its duties pays; only if both have complied does the loss stay with the consumer. The design point is that liability follows the specific failure, so each link in the chain has a reason to fix its own part.

Australia passed the broadest version. The Scams Prevention Framework Act, which cleared both houses of Parliament on 13 February 2025, imposes obligations to prevent, detect, report, disrupt and respond to scams on banks, telecoms providers and digital platforms, including social media, with civil penalties of up to A$50 million for serious breaches and a route for consumers to seek redress when a business has failed its obligations. The Australian figures also give the most honest picture of what system-level defence can and cannot do. According to the National Anti-Scam Centre, reported losses fell 25.9 per cent to A$2.03 billion in 2024, a drop the centre credited to joint work by government, industry, law enforcement and community groups. In 2025 they rose again by 7.8 per cent, to A$2.18 billion. Catriona Lowe, deputy chair of the Australian Competition and Consumer Commission, called scams a “wicked problem” because they are “complex, fast-evolving, and resistant to simple solutions,” and pointed to the “increasing sophistication in scam activity through artificial intelligence.”

That is the realistic shape of success. Coordinated systems can bend the curve. They cannot make a factory disappear. They have to keep changing its input costs faster than it can adapt.

Breaking the Machinery Upstream

The most direct way to raise a factory's costs is to attack its equipment, its premises and its money. Over the past year, that has started to happen.

The United Kingdom's Crime and Policing Act 2026, which received Royal Assent on 29 April 2026, created offences of possessing and supplying SIM farms without a legitimate reason. These devices, defined in government guidance as equipment able to use five or more SIM cards at once for calls or messages, allow a single operator to send scam texts at industrial volume. The government announced the measure in 2025 as a first in Europe and said it would take effect six months after Royal Assent. Critics have noted that much scam traffic originates abroad, which limits what a domestic ban can achieve. But the logic is sound. The offence targets the machine that multiplies one person's reach, rather than the message the machine sends.

In October 2025, after Myanmar's military said it had raided KK Park, one of the largest scam compounds on the Thai border, detaining more than 2,000 people and seizing 30 Starlink terminals, SpaceX said it had disabled more than 2,500 Starlink kits near suspected scam centres in the country. Reporting at the time noted that other compounds along the border remained active and that operations might simply relocate. Factories move when they are disrupted. That is a reason to disrupt them repeatedly, not a reason to stop.

The money is the third target. In October 2025, the US Department of Justice filed a civil forfeiture complaint against around 127,271 bitcoin, then worth roughly $15 billion, linked to Chen Zhi, chairman of Cambodia's Prince Group, whom prosecutors charged with wire fraud and money laundering conspiracy over forced-labour scam compounds. The US Treasury and the UK government imposed coordinated sanctions on the network. Chen and Prince Group have denied wrongdoing. Cambodia, which had revoked his Cambodian citizenship by royal decree in December 2025, arrested him on 6 January 2026 and, in a move announced the next day, sent him to China along with two other Chinese nationals. China's Ministry of Public Security said his group was suspected of fraud, running casinos and concealing criminal proceeds. The Justice Department's Scam Center Strike Force, created in November 2025, said by April 2026 that it had restrained more than $700 million in cryptocurrency and taken down more than 500 fraudulent investment websites. By September 2026, after a single-day action that restrained about $52 million linked to Xinbi Guarantee, a Telegram-based marketplace sanctioned by the US Treasury the same day, its cumulative total had reached about $938 million. INTERPOL, for its part, launched Operation Shadow Storm, a task force funded by the UK Home Office to go after the financial networks behind scam centres.

None of these actions alone ends industrial fraud. Together they move effort away from the moment a victim must decide whether a voice is real, and towards the infrastructure that makes it cheap to fake the voice ten thousand times.

Building Factories That Waste the Factory's Time

There is a further move available, and it has a pleasing symmetry. If AI makes it cheap for criminals to hold thousands of conversations, it makes it equally cheap for defenders to hold thousands of conversations with criminals.

In November 2024, the British mobile operator Virgin Media O2 unveiled Daisy, an AI “granny” built to answer scam calls in real time. It chains together a speech-to-text model, a custom language model with a character layer, and a synthetic voice, and it was trained with help from the YouTube scambaiter Jim Browning. O2 seeded Daisy's numbers onto the lists scammers use to find victims. The company said Daisy had kept numerous fraudsters on the phone for 40 minutes at a time. Murray Mackenzie, O2's director of fraud, warned that “no matter how persuasive someone on the other end of the phone may be, they aren't always who you think they are.”

Australia's Commonwealth Bank has taken the idea further. In June 2025 it announced a partnership with Apate.ai, a company spun out of Macquarie University's Cyber Security Hub and led by Professor Dali Kaafar. Apate runs a network of honeypot phone numbers designed to be found by scammers. When they call, they reach AI bots with varied ages, genders and accents, tuned with Australian slang, that keep them talking while gathering intelligence on their scripts and tactics. That intelligence feeds into the bank's scam controls and the wider anti-scam network in near real time.

The ScriptMind paper points the same way from the detection side. Its authors fine-tuned an 11-billion-parameter model on 22,712 structured examples built from real scam cases, report that it outperformed GPT-4o by 13 per cent, and found that in simulated calls it raised and sustained users' suspicion. The idea is not to ask people to hear the fake, but to have a machine recognise the script every factory relies on and warn the person in the moment.

These systems will not stop fraud alone, and some of them, Daisy especially, are partly marketing exercises whose results are reported by the companies themselves. But they point at the right target. A fraud factory's margin depends on the share of its labour and compute that reaches paying victims. Every minute a worker or an AI agent spends talking to a bot that will never pay, and every script that is captured and turned into a warning, is a cost the factory has to absorb. Automation does not only help the attacker. For once, it lets defenders operate at the same scale.

What Protection Looks Like for One Person Facing a Machine

So, back to the person holding the phone.

The honest answer to how an individual distinguishes a genuine interaction from a manufactured one is that, much of the time, they cannot by sense alone. Human detection sits close to chance. AI voices and conversationalists match or beat humans. And the factory does not need to fool you in particular. It needs to fool a small fraction of a very large number of people, and it can now afford to try them all.

What remains is a smaller set of habits that do not depend on perception at all. Agree a code word with the people whose voices you would trust in a crisis. When a request involves money, credentials or secrecy, end the conversation and re-establish contact through a channel you already hold. Treat urgency, confidentiality and a new or unusual channel, appearing together, as the signature of a procedure being bypassed. Talk to someone before you pay. None of that requires you to be a forensic analyst. All of it works whether the fake is crude or perfect.

The larger answer is that protection for one person has to be built mostly out of things that one person does not control. It looks like a bank that has a financial reason to stop the transfer, because it will pay if it does not. It looks like a telecoms network that blocks the SIM farm and the spoofed sender, because it bears a share of the loss when it fails to. It looks like a platform that faces penalties for letting a scam reach its users. It looks like anti-scam centres that can freeze money across a border within a day of the alarm being raised, satellite providers that cut off compounds, prosecutors who seize the bitcoin, and defensive AI that wastes the attackers' hours and steals their scripts.

INTERPOL's Valdecy Urquiza warned that the cost of financial crime “is not just money – it is people's life savings, their dignity, and in the worst case, their life.” The fraud factory runs on the assumption that this cost will continue to fall on individuals, one call, one video and one conversation at a time. The evidence from London, Singapore and Canberra suggests that assumption can be broken, and that the right response to industrialised deception is not a more suspicious public. It is an industrial defence. Every person on earth may now be a potential target. That does not make every person on earth responsible for being the last line of defence.

---

References and Sources

1. Black Hat USA 2026 presentation by Eric Huber, Senior Manager of Adversary Intelligence and Disruption, TD Bank, as reported in: Witts, Joel. “Research reveals how criminal gangs are using AI to improve financial scams.” Expert Insights, 4 August 2026, https://expertinsights.com/news/research-reveals-how-criminal-gangs-are-using-ai-to-improve-financial-scams; and Seals, Tara. “AI Sends Global Crime Syndicates Into Fraud Nirvana.” Dark Reading, August 2026, https://www.darkreading.com/threat-intelligence/ai-global-crime-syndicates-fraud-nirvana
2. Gressel, Gilad, Rahul Pankajakshan, Shir Rozenfeld, Ling Li, Ivan Franceschini, Krishnashree Achuthan and Yisroel Mirsky. “Love, Lies, and Language Models: Investigating AI's Role in Romance-Baiting Scams.” arXiv:2512.16280, submitted 18 December 2025. https://arxiv.org/abs/2512.16280
3. INTERPOL. “INTERPOL report warns of increasingly sophisticated global financial fraud threat.” News release, 16 March 2026, including quotes from Secretary General Valdecy Urquiza; and the 2nd INTERPOL Global Financial Fraud Threat Assessment, 2026. https://www.interpol.int/en/News-and-Events/News/2026/INTERPOL-report-warns-of-increasingly-sophisticated-global-financial-fraud-threat
4. Organized Crime and Corruption Reporting Project. “UN Report Details Southeast Asia's Interconnected Criminal Economy.” 21 July 2026, reporting on the UNODC threat assessment “An Interconnected Criminal Ecosystem”, launched in Bangkok on 21 July 2026. https://www.occrp.org/en/news/un-report-details-southeast-asias-interconnected-criminal-economy
5. Kim, Heedou, Changsik Kim, Sanghwa Shin and Jaewoo Kang. “SCRIPTMIND: Crime Script Inference and Cognitive Evaluation for LLM-based Social Engineering Scam Detection System.” arXiv:2601.13581, submitted 20 January 2026. https://arxiv.org/abs/2601.13581
6. Erukude, Sai Teja, Viswa Chaitanya Marella and Suhasnadh Reddy Veluru. “AI-Driven Cybersecurity Threats: A Survey of Emerging Risks and Defensive Strategies.” arXiv:2601.03304, submitted 6 January 2026. https://arxiv.org/abs/2601.03304
7. Heiding, Fred, Claudio Mayrink Verdun, Simon Lermen, Andrew Kao, Vitor Albiero, Lauren Deason, Irina-Elena Veliche and Christine Lehane. “Evaluating AI Models' Capability to Automate Voice Phishing Attacks.” arXiv:2607.09970, submitted 10 July 2026, revised 20 August 2026; published in Expert Systems With Applications, volume 332, article 133620. DOI 10.1016/j.eswa.2026.133620. https://arxiv.org/abs/2607.09970
8. Down, Aisha. “Deepfake fraud taking place on an industrial scale, study finds.” The Guardian, 6 February 2026; syndicated by the Irish Examiner as “AI experts say deepfakes now at 'industrial' scale”, 6 February 2026, including quotes from Simon Mylius (MIT) and Fred Heiding (Harvard). https://www.irishexaminer.com/news/arid-41789042.html
9. Federal Bureau of Investigation, Internet Crime Complaint Center. “2025 Internet Crime Report.” 7 April 2026. https://www.ic3.gov/AnnualReport/Reports/2025_IC3Report.pdf
10. US Department of Justice. “Scam Center Strike Force Takes Major Actions Against Southeast Asian Scam Centers Targeting Americans.” Office of Public Affairs press release, 23 April 2026, https://www.justice.gov/opa/pr/scam-center-strike-force-takes-major-actions-against-southeast-asian-scam-centers-targeting; and US Attorney's Office for the District of Columbia, “Scam Center Strike Force”, web page updated 15 October 2026, accessed 20 October 2026, https://www.justice.gov/usao-dc/scam-center-strike-force; see also Crypto Briefing, “Scam Center Strike Force restrains $52M in laundered crypto in one day”, 9 September 2026, https://cryptobriefing.com/scam-center-strike-force-52m-crypto-madagascar/
11. Federal Bureau of Investigation. “Criminals Use Generative Artificial Intelligence to Facilitate Financial Fraud.” Public Service Announcement, Alert Number I-120324-PSA, 3 December 2024. https://www.ic3.gov/psa/2024/psa241203
12. Diel, Alexander, et al. “Human performance in detecting deepfakes: A systematic review and meta-analysis of 56 papers.” Computers in Human Behavior Reports, volume 16, article 100538, December 2024, DOI 10.1016/j.chbr.2024.100538; and Köbis, Nils C., Barbora Doležalová and Ivan Soraperra. “Fooled twice: People cannot detect deepfakes but think they can.” iScience, volume 24, issue 11, article 103364, 2021, DOI 10.1016/j.isci.2021.103364.
13. Ashraf, Muhammad Tahir. “Synthetic Trust Attacks: Modeling How Generative AI Manipulates Human Decisions in Social Engineering Fraud.” arXiv:2604.04951, submitted 2 April 2026. https://arxiv.org/abs/2604.04951
14. Magramo, Kathleen. “British engineering giant Arup revealed as $25 million deepfake scam victim.” CNN, 16 May 2024, https://edition.cnn.com/2024/05/16/tech/arup-deepfake-scam-loss-hong-kong-intl-hnk; and World Economic Forum, interview with Rob Greig, Global Chief Information Officer, Arup, February 2025, https://www.weforum.org/stories/2025/02/deepfake-ai-cybercrime-arup/
15. Bloomberg. “Ferrari Narrowly Dodges Deepfake Scam Simulating Deal-Hungry CEO.” 26 July 2024; see also Motor1, “Someone Deepfaked Ferrari CEO's Voice and Tried to Scam the Company”, July 2024. https://www.motor1.com/news/728264/ferrari-ceo-deepfake-voice-phone/
16. Singapore Police Force. “Singapore and Hong Kong Police Force Recover Over $670,000 in a Scam.” News release, 7 April 2025. https://www.police.gov.sg/media-hub/news/2025/04/20250407_singapore_and_hong_kong_police_force_recover_over_$670000_in_a_scam
17. Payment Systems Regulator. “One year on: impact of APP reimbursement on victims.” 2025, https://www.psr.org.uk/news-and-updates/latest-news/news/one-year-on-impact-of-app-reimbursement-on-victims/; and the independent evaluation published by the regulator in July 2026, as summarised in Norton Rose Fulbright, Regulation Tomorrow, “Payment fraud falls by £73m following PSR reimbursement scheme”, 1 July 2026, https://www.regulationtomorrow.com/2026/07/payment-fraud-falls-by-73m-following-psr-reimbursement-scheme/
18. Infocomm Media Development Authority and Monetary Authority of Singapore. “Implementation of Shared Responsibility Framework.” Joint press release, 24 October 2024; and MAS, “Guidelines on Shared Responsibility Framework”. https://imda.gov.sg/resources/press-releases-factsheets-and-speeches/press-releases/2024/implementation-of-shared-responsibility-framework
19. Gilbert + Tobin. “The Scams Prevention Framework legislation passes Parliament: time to get your house in order.” February 2025. https://www.gtlaw.com.au/insights/the-scams-prevention-framework-legislation-passes-parliament-time-to-get-your-house-in-order
20. Australian Competition and Consumer Commission, National Anti-Scam Centre. “Targeting scams: Report of the National Anti-Scam Centre on scams data and activity 2024.” March 2025, https://www.accc.gov.au/about-us/publications/serial-publications/targeting-scams-reports-on-scams-activity/targeting-scams-report-of-the-national-anti-scam-centre-on-scams-data-and-activity-2024; and Croft, Daniel, “Scams cost Australians over $2bn in 2025, says ACCC”, Cyber Daily, 30 March 2026, including quotes from Catriona Lowe, Deputy Chair, ACCC, https://www.cyberdaily.au/security/13400-scams-cost-australians-over-2bn-in-2025-says-accc
21. UK Home Office. “Major step for fraud prevention with landmark ban on SIM farms.” GOV.UK news story, 2025; and “Crime and Policing Act 2026 guidance: offences relating to the possession and supply of SIM farms”, GOV.UK, July 2026. https://www.gov.uk/government/news/major-step-for-fraud-prevention-with-landmark-ban-on-sim-farms
22. Arab News, via Agence France-Presse. “SpaceX says 'disabled' 2,500 Starlink devices at Myanmar scam centers.” October 2025. https://www.arabnews.com/node/2619810/world
23. US Department of Justice, US Attorney's Office for the Eastern District of New York. “Chairman of Prince Group Indicted for Operating Cambodian Forced Labor Scam Compounds Engaged in Cryptocurrency Fraud Schemes.” Press release, 14 October 2025, https://www.justice.gov/usao-edny/pr/chairman-prince-group-indicted-operating-cambodian-forced-labor-scam-compounds-engaged; Elliptic, “Prince Group targeted with $15B crypto seizure and sanctions for pig butchering operations”, October 2025, https://www.elliptic.co/blog/prince-group-targeted-with-crypto-sanctions-for-global-pig-butchering-operations; Associated Press, “Businessman extradited from Cambodia to China suspected of running vast online criminal empire”, 8 January 2026, republished by NY1, https://ny1.com/nyc/all-boroughs/ap-top-news/2026/01/08/businessman-extradited-from-cambodia-to-china-suspected-of-running-vast-online-criminal-empire; and CamboJA News, “Chen Zhi, Alleged Scam Network Kingpin, Sent to China”, 8 January 2026, https://cambojanews.com/chen-zhi-alleged-scam-network-kingpin-sent-to-china/
24. Virgin Media O2. “O2 unveils Daisy, the AI granny wasting scammers' time.” Press release, 14 November 2024, including quotes from Murray Mackenzie, Director of Fraud. https://news.virginmediao2.co.uk/o2-unveils-daisy-the-ai-granny-wasting-scammers-time/
25. Commonwealth Bank of Australia. “CommBank deploys AI bots to outsmart scammers” (partnership with Apate.ai). Newsroom, June 2025. https://www.commbank.com.au/articles/newsroom/2025/06/apate-ai.html

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

The black box sits beside the entrance to Woodland Glen IV, a residence hall on the University of Kentucky's campus in Lexington. Its job is simple to describe. You walk towards the door, it looks at you, and if it recognises the geometry of your face, the lock releases. No card, no phone, no fumbling. You are home.

The device is called Rock. It is made by Alcatraz AI, a company based in Cupertino, California, and this autumn the University of Kentucky began testing it at a handful of buildings, including two residence halls, as a way for students to get into the places where they sleep without tapping a key card. In mid-September local television stations covered the installation. The libertarian magazine Reason picked the story up nationally under the headline “Scan your face to enter your dorm?” The university, for its part, has been careful and consistent on one point: enrolling your face is voluntary. Your Wildcard, the university's student ID, still works. Your phone still works. Nobody has to do this.

That is true, and it is worth taking seriously. It is also where the interesting questions begin rather than end. For many of the eighteen-year-olds who arrived at Kentucky this autumn, this may be the first building they have ever been invited to open with their face. It is not a bank vault or an airport gate or a laboratory full of expensive reagents. It is the front door of their home: the one they walk through at two in the morning, after a bad exam, or carrying shopping bags with the key card buried somewhere beneath a textbook. What they learn at that door, about what is normal, about what is asked of them, about whether their face is something they give or something that is simply taken, may shape how they respond to every door that follows.

So here is the question the black box poses. When the student with the shopping walks up and the lock clicks open before they have put anything down, what have they actually experienced? Added safety? Added convenience? Or the first of many doors that will, one day, know who they are whether or not they asked?

What the University Actually Installed

Start with what is known, because the facts are more specific than the headlines.

According to the University of Kentucky Police Department's own information page, the biometric access pilot began at Woodland Glen IV. Residents were told to expect an enrolment email near the start of the semester, sent from an Alcatraz address, with personalised instructions. The page states that biometric data is collected solely to verify access to the building, that it is not used to track students' movements or behaviour beyond building entry, and that anyone the system fails to recognise can still get in with their Wildcard or mobile ID. WKYT, the Lexington CBS affiliate, reported on 19 September that scanners were also going in at Kirwan-Blanding, another residence complex, for a two-week trial. The Lexington Herald-Leader listed further planned sites, including parts of the medical campus, several biomedical research buildings and athletics practice spaces.

How the machine sees you was explained by the university's own police chief. Joe Monroe, who has led the UK Police Department since 2009, told LEX18 that the system “looks at your face, and it assigns different points using geometry, facial geometry. They call it facial vectors. Once that number is assigned, that is who you are. There's no pictures stored.” The university's privacy language, quoted by the Herald-Leader, describes the resulting “facial signature” as “a one-way encrypted mathematical representation of your facial features.” Alcatraz's chief executive, Tina D'Agostin, put it more bluntly to WKYT: “You're zeros and ones to us once you've been scanned.” Monroe has also conceded that a security brand named after America's most famous island prison “may not be the best name for the product, and we've pointed that out.”

Two details from the local coverage matter more than they first appear. The first is that this is not the university's first experiment with biometrics. According to the Herald-Leader, Rock is replacing iris-scanning equipment from a company called Eyelock that the university described as “non-functional.” Biometric entry at UK is an existing practice being upgraded and extended, not a novelty.

The second detail should stay with anyone thinking about consent. The Herald-Leader reported that enrolment would be required for people entering the most sensitive places on campus, including the police department and research facilities, and optional for less vulnerable places such as residence halls and most athletics facilities. LEX18 filmed the police evidence room, where Deputy Chief Nathan Brown explained that entry required both a credential and a face. “This is dual authentication because this is a very obvious restricted room,” he said. In other words, the University of Kentucky is not running a voluntary facial recognition programme. It is running one facial recognition system under two sets of rules. The Kentucky Kernel, the student newspaper, reported that the technology is also in other locations requiring two-factor authentication, such as research labs. For some people on campus, a face scan is already a condition of doing their job. For students, at their front doors, it is a choice. The technology is identical. Only the policy differs, and policy is the easiest part of any system to change.

A Company Built by a Face ID Veteran

Alcatraz AI is not a campus start-up that stumbled into student housing. It is an enterprise access control company with serious money and a clear ambition. It was founded in 2016 by Vince Gaydarzhiev, who, according to the company, led hardware prototyping for the iPad and iPhone at Apple during the development of Face ID. In a 2019 interview with Security Sales and Integration, Gaydarzhiev said the idea came from noticing that physical security, even at Apple's own campus, was still done “old school,” with badges and guards at almost every door.

In April 2026 the company announced a fifty million dollar Series B round, led by BlackPeak Capital, Cogito Capital and Taiwania Capital, bringing its total funding to more than one hundred million dollars. It claimed customers among the world's largest AI data centres, major US airports, energy companies, NFL teams, Fortune 100 firms and “major universities.” D'Agostin summarised the pitch in one line: “We are the Face ID of securing physical spaces.”

That framing tells you where the product is designed to end up. Face ID on a phone is not an optional extra for enthusiasts. It is the default way tens of millions of people unlock the most personal object they own. A company that describes itself as the Face ID of buildings is not aiming to sit politely beside the key card. It is aiming to make the key card feel as dated as typing a passcode. D'Agostin said as much to WKYT: “The badge, what most campuses are using today is a 30-year-old technology.”

Alcatraz's marketing to universities makes the same argument with a student in mind. A February 2026 guide on its website, pitching “privacy-first facial authentication” to higher education, opens with an observation about young people: “Students today unlock their phones with a glance and verify banking transactions with their faces. Yet on campus, they still fumble with physical keycards.” It promises “the frictionless experience your students expect.” To its credit, the same guide also sets out a strong consent standard, stating that students “must have the right to withdraw consent and delete their templates at any time.”

The guide is also candid about where else the technology could go. Beyond residence halls, labs and athletics, it suggests that “Libraries can automate entry for late-night study sessions,” and it mentions exam proctoring. There is no public evidence that the University of Kentucky plans to put face readers on its libraries or in its exam halls. But the people selling the system already imagine a campus where the face is the credential for much more than the dorm.

The Door That Watches Everyone Else

“Voluntary enrolment” describes the relationship between the system and the person who signs up. It says rather less about everyone else who walks through the door.

The main security argument for Rock is not really about the enrolled student. It is about tailgating: someone with no right to be in a building slipping in behind someone who does. D'Agostin told WKYT that the system lets security teams “detect when this incident occurs so they can react to it in real time.” The university has framed the housing tests in the same terms, as part of a broader effort to reduce unauthorised entry and strengthen residence hall safety, with the face option added as a hands-free extra rather than a replacement for existing credentials. A student interviewed by WKYT said they liked it “just because safety is the number one priority for the college.”

To detect a tailgater, though, the machine has to pay attention to people who have not enrolled. Alcatraz's product page is explicit. The reader “authenticates the enrolled person in under a second, then keeps monitoring the doorway as the door opens,” and if someone enters without authenticating, “a picture is captured at the door” and logged for investigation. The security operations centre, the page says, receives a snapshot of that person along with the time of the event and the device that triggered it. A 2021 announcement of the feature, reported by Security Systems News, described the same behaviour.

Neither the university nor local reporting has said whether tailgating snapshots are switched on at Woodland Glen IV or Kirwan-Blanding, and it would be wrong to assume that they are. But the design matters regardless of how any one site is configured. In a residence hall, the person walking in close behind a resident is very often not an intruder. It is a roommate who left their card upstairs, a friend coming to study, a parent carrying a mini-fridge. Some of those people will have chosen not to enrol. Some will never have been asked. A system whose core safety feature involves noticing, and potentially photographing, the people who did not authenticate is a system whose voluntariness applies to only one side of the door.

This is the structural problem that has dogged every face recognition product sold as a convenience. The person who opts in gets the benefit. The people around them bear some of the cost, and they are not the ones who were asked.

Voluntary Is a Setting, Not a Promise

Consent is often spoken about as though it were a fixed property of a technology. With access control, it is closer to a configuration option.

Alcatraz's public help centre describes three ways to enrol someone. Manual enrolment takes a single badge swipe and a few prompts on the Rock's screen. Web enrolment sends an invitation to register with your own phone or computer camera, which matches the emailed process the University of Kentucky describes. The third mode is auto-enrolment. When a Rock is configured this way, the help centre explains, “each time the user swipes their badge and enters through the door, the Rock is learning their face.” It needs four swipes, at least five minutes apart, from people who already have access to that door. In 2019 Security Sales and Integration described this self-enrolment as requiring no administration or employee action, “switching to face-only mode autonomously as soon as the employee is seen enough times.”

The direction of travel in the product is towards less friction at sign-up, not more. In May 2026 Alcatraz announced platform updates that, according to Biometric Update, let administrators apply access control rules to enrolment and added visual guidance to speed up biometric registration. D'Agostin said the changes were about making sign-up simpler and continuing to “automate operations and deliver a seamless authentication experience.” In September, at the GSX security show in Atlanta, the company went further. It named its software layers: RockOS, the firmware on each device that handles enrolment, liveness detection, authentication and tailgating detection, and Rock Cortex, a platform for verification, user and device management, integrations, monitoring, analytics and compliance. According to SecurityInfoWatch, a guided enrolment process that removes the need for users to move or tilt their heads was slated for general availability in October 2026, which Alcatraz says will make onboarding faster and cut failed enrolments. None of that is sinister. Making sign-up easy is what any product team would do. But ease is not the same thing as deliberation, and a choice that takes two seconds is rarely a choice anyone remembers making.

There is no evidence that the University of Kentucky uses auto-enrolment in its halls. Its published process is invitation-based and opt-in. The point is not that anyone is being enrolled without their knowledge. It is that the hardware on the wall can operate under a quite different consent model, and the difference is a setting chosen by an administrator. The student's choice is protected by a policy, not by the physics of the device.

Policies drift, and the most instructive recent example is an airport. The US Transportation Security Administration has long described its facial comparison checks as optional. In May 2025 a bipartisan group of senators, including Jeff Merkley, John Kennedy, Ed Markey and Roger Marshall, reintroduced legislation to rein the programme in. “Americans have the right to opt out of using TSA's facial recognition at the airport, and we need to protect that right,” Merkley said. Kennedy complained that the agency scans travellers “without even making it clear that they can opt out.” Two months later the Algorithmic Justice League, founded by the computer scientist Joy Buolamwini, published “Comply To Fly?”, drawing on travellers' accounts from 91 US airports. According to Biometric Update's summary, 99 per cent of respondents said officers had not told them verbally that they could opt out, many experienced the scans as mandatory, and some who declined reported hostility and extra scrutiny.

Nobody made the scans compulsory. They did not need to. When the scan is the fast lane, when the signs are small, when refusal is treated as an inconvenience, “optional” becomes something you have to insist on rather than something you are offered. That is how convenience quietly becomes expectation. It does not happen through a decree. It happens through queues.

The media scholars Nora Draper and Joseph Turow have a name for the feeling this produces. In a 2019 paper in New Media and Society, they described “digital resignation”: the condition in which people want to control what companies know about them but feel unable to, and give up. They argued that this resignation is not a flaw in consumers but something cultivated by routine corporate practices that make resistance tiring and pointless. A student who opts out in a hall where most neighbours have opted in will not be punished. They will simply be the one still tapping a card, still the exception. Most people do not choose to be the exception for long.

Home Is a Different Kind of Door

Privacy theory offers a useful way to explain why a residence hall feels different from a laboratory.

In a 2004 article in the Washington Law Review, the philosopher Helen Nissenbaum proposed that privacy is best understood as “contextual integrity.” Every setting carries norms about which information flows are appropriate within it, and the violation lies not in the information itself but in its movement out of the context where it belongs. In the police evidence room, the norm is scrutiny; anyone entering is expected to account for being there, and a face scan fits. A home is governed by the opposite norm. It is the one place you are supposed to be able to come and go without explaining yourself. That a student's home is owned by an institution, patrolled by its police and opened with its credential does not erase the norm. It makes it more fragile.

Then there is the question of who is being taught what. Most first-year undergraduates at Kentucky in 2026 were born around 2007 or 2008. Apple introduced Face ID with the iPhone X on 12 September 2017, when they were nine or ten. For this cohort, unlocking a device with a glance is a childhood habit, which is exactly what Alcatraz's marketing leans on. The dorm door extends that habit from the private object in your pocket to a shared building run by someone else. Face ID, by Apple's design, answers to you. The building answers to its owner.

What that habit looks like in practice surfaced a few weeks into term. The Kentucky Kernel spoke to Leah Henderson, a first-year student living in Woodland Glen IV, who said the university had not yet told her about the technology. She first assumed the cameras at the doors were recording students. Once she learned their purpose, she called the system innovative and convenient. “It's kind of cool. Maybe I don't have to use my phone all the time to get in,” she said. Purani Kannan, a sophomore and resident advisor at Kirwan-Blanding, told the paper that many students don't know much about the technology yet. Nothing in either account suggests anything improper; enrolment remains opt-in. But together they show how quickly an unexplained camera becomes a welcome convenience, and how little of that journey passes through anything resembling deliberate consent.

Mark Andrejevic and Neil Selwyn, both at Monash University, examined facial recognition in schools in a 2020 paper in Learning, Media and Technology. They warned that it risked altering schooling “along divisive, authoritarian and oppressive lines,” and worried about how such systems normalise surveillance for young people, teaching them to treat constant identification as the ordinary texture of institutional life. Their focus was compulsory schooling rather than universities, and Kentucky's pilot is opt-in. But the point travels: habits formed in institutions shape expectations outside them.

Not all research points one way. A 2022 study in Frontiers in Psychology surveyed 391 college students in mainland China and Taiwan about face recognition access control on their campuses. Students who found the systems useful and easy were more likely to use them, and that acceptance was associated with a stronger sense of school identity and belonging. The authors acknowledged “information leakage anxiety” but did not measure it. The finding is a useful corrective: for some students, a door that recognises them may feel like a place that knows them. That is precisely why normalisation matters. A technology that feels welcoming is easy to accept without examining what else comes with it.

What a Card Swipe Already Knows

It is tempting to treat the face as the moment surveillance begins. On most American campuses, the key card has been quietly generating data for years.

UKNow, the university's news service, described in August 2024 what its Wildcard Mobile ID could already do: residence hall access, payment at UK Dining, rides on LexTran buses, verification at athletic events, borrowing from UK Libraries and entry to recreational facilities. The same article said the university hoped to extend the mobile experience to the entire campus community, with faculty and staff to follow. Credentials on campus do not stay in one place; they spread to wherever it is convenient to use them. Each interaction is, potentially, a timestamped record of where a student was.

What institutions can do with such records is not hypothetical. In 2018 Sudha Ram, a professor of management information systems at the University of Arizona, drew attention for research using anonymised data from the university's CatCard ID system to predict which first-year students would drop out, based on how often they swiped into halls, libraries and recreation centres and where they spent money. “Many times, students who tend to drop out are people who have not established social relationships and who haven't established a regular routine on campus,” Ram told Arizona Public Media. Contemporary reports said the model identified roughly 85 to 90 per cent of students who went on to leave. The work was framed as supportive. It is also a textbook case of function creep: data collected to open doors and sell sandwiches becoming raw material for behavioural prediction. It happened with cards. Nothing about a face makes it less likely to happen with faces.

The architecture makes the connection concrete. According to Alcatraz, a Rock profile is linked to the user's badge ID in the existing access control system, so the face simply stands in for the card. Wherever the card is already a key, the face could, with the right hardware and a policy decision, become one too. Nobody has announced plans to put face readers in Kentucky's dining halls, libraries or gyms. But the path from one door to many is an installation job, not a reinvention.

Beyond campuses, the warnings are sharper. In late 2022 Madison Square Garden Entertainment used facial recognition at Radio City Music Hall to identify and eject Kelly Conlon, a lawyer chaperoning her daughter's Girl Scout troop, because her firm was in litigation against the company. Security technology became a tool for settling commercial scores. In Western Australia in 2021, police obtained data from SafeWA, a Covid-19 check-in app whose users had been told only contact tracers would see it. The government rushed through a law restricting its use. “Because it is so important to maintain community confidence in SafeWA, we've come down in favour of nobody,” said Attorney General John Quigley. “Not the CCC, not someone under FOI, not the police. Nobody sees this information.”

Note the order of events. The promise came first, its breach second, and the law protecting it only after the public found out. Anyone weighing a campus assurance that biometric data will not be used for tracking should ask what Western Australians eventually had to ask: what, other than the institution's goodwill, makes that true?

Kentucky's Law Stops at the Campus Gate

In Kentucky, the honest answer appears to be: not very much.

The Kentucky Consumer Data Protection Act, signed by Governor Andy Beshear in April 2024, took effect on 1 January 2026. It treats biometric data used for identification as sensitive and requires opt-in consent before a covered business processes it. That sounds like exactly the protection a student would want. But the Act states that its core provisions “shall not apply to any” city, state agency or political subdivision, nor to any “institution of higher education.” The University of Kentucky is a public university; on the plain text, the state's new privacy law does not appear to reach it. The Act also provides no private right of action, so even where it applies, only the attorney general can enforce it.

Illinois's Biometric Information Privacy Act of 2008, the strongest such law in the country, would not necessarily help either. BIPA requires written, informed consent before a private entity collects a face template and lets individuals sue, which is why it has produced some of the largest privacy settlements in history. But it excludes state and local government agencies, and private universities have slipped through too. In November 2022 a federal court dismissed a BIPA class action against DePaul University over a remote proctoring tool that captured students' facial data, ruling that because DePaul administered federal student loans it counted as an exempt financial institution.

Federal law offers a partial safeguard. Regulations under the Family Educational Rights and Privacy Act define a “biometric record” as a record of one or more measurable biological or behavioural characteristics that can be used for automated recognition of an individual, listing facial characteristics as an example, and treat it as personally identifiable information. But FERPA governs disclosure of education records. It does not decide whether an institution should build a face-matching system, and it says nothing about keeping enrolment optional.

Some policymakers have drawn firmer lines. In September 2023 New York's education commissioner, Betty Rosa, banned the purchase and use of facial recognition in the state's elementary and secondary schools, concluding after a state technology report that the risks outweighed the claimed benefits. That ban covers children, not adult university students. But it shows that a government can decide some settings are inappropriate for the technology, rather than leaving the question to procurement officers one pilot at a time.

In Kentucky, then, the durability of “voluntary” rests largely on institutional policy and on vendor commitments. The Herald-Leader reported that the company says it keeps no cloud biometric database and shares no data with third parties or government databases. Speaking to the Kentucky Kernel, D'Agostin was blunter still: “Basically, your face is a digital key, and when you come to the door, if that key matches, the door opens.” Of the data behind it, she said: “We don't have access and we don't store it, so we can't sell the data.” The Kernel reported that the data remains with the university, which manages it. Dani Jaffe, the university's associate director of strategic and crisis communications, said: “All biometric data is stored on secure servers and is never shared with third parties.” She added that the university “will continue to evaluate and implement tools” as security needs evolve. That clarifies where the promise lives. If the vendor holds nothing, the templates' fate rests on the university's own policy, and the state law that would bind a private company does not bind the university. Those are meaningful assurances. They are also promises made by the parties who benefit from the system's expansion, enforceable mainly by the parties who made them.

The Campaign That Already Won Once

None of this is new. Six years before Kentucky installed its scanners, the digital rights group Fight for the Future and the student organisation Students for Sensible Drug Policy set out to keep facial recognition off American campuses, publishing a scorecard of institutions and pushing resolutions through student governments.

Their best-known victory came at UCLA. In early 2020, as the university considered adding facial recognition to its security cameras, Fight for the Future ran more than 400 photographs of UCLA athletes and faculty through Amazon's face recognition software against a mugshot database. According to Inside Higher Ed, the test produced 58 false matches, most of them people of colour. UCLA dropped the plan. “UCLA will not pursue the use of facial recognition software technology,” Michael Beck, the administrative vice chancellor, said in a statement, concluding that “the potential benefits are limited and are vastly outweighed by the concerns of the campus community.” Evan Greer of Fight for the Future said the victory would “send a pretty strong message to any other administration who is considering doing this.” By then around 50 universities had publicly said they had no plans to use the technology.

Kentucky's pilot differs in important ways. UCLA's plan would have scanned everyone in view of campus cameras; Kentucky's is an opt-in door lock that verifies one enrolled person at a time. A mismatch at a dorm door means using your card, not being flagged as a suspect, and IEEE Spectrum has reported that best-case false negative rates are around two in 1,000. Michael Murray, a University of Kentucky law professor who studies artificial intelligence, told LEX18 that the university seemed to be “taking it quite seriously” and trying “to roll it out with a very low threshold of risk,” while cautioning that AI “is not cost-free, and it's not risk-free.”

Still, the old unease has not gone away. Local online discussion, summarised by the Lexington Times, was dominated by privacy worries, with some commenters calling the scanners the start of a surveillance network and others doubting that a face reader would stop anyone following a resident through an open door. Reason's Elizabeth Nolan Brown questioned whether the security case justified the system at all, and suggested institutions might be “dreaming bigger” than a limited dorm pilot. There is no sign yet of an organised Kentucky campaign on the UCLA model. But the arguments that won in 2020 are still available, and a pilot is the stage at which they are easiest to make.

Safety, Convenience, or the First of Many Doors

So what does the student carrying the shopping actually experience when the lock clicks open?

On the evidence available: mostly convenience, described as safety. The system is genuinely easier than hunting for a card, and nobody should sneer at that. Losing a Wildcard late at night is a real problem, and a door that knows you is a real comfort. The safety case is harder to pin down. Its main security function depends on the door paying attention to people who have not enrolled, in a building where the person behind you is usually a friend, and the university has not published data on how often tailgating causes harm in its halls.

What the student is less likely to notice is the thing that matters most over time. They are not only choosing between a card and a face. They are helping to establish a norm. Every enrolment makes the face reader more useful, the card tapper more unusual and the next expansion easier to justify. That is not a conspiracy. It is how defaults work.

It would be unfair to say the University of Kentucky has done anything wrong. On the public record, it has kept the dorm pilot optional, maintained alternatives, avoided storing photographs and explained the system openly. That is better practice than many face technology deployments. But it would be naïve to treat “voluntary” as a permanent feature rather than a present policy, in a state whose new privacy law excludes public universities, using hardware that is already mandatory elsewhere on the same campus.

If voluntariness is to last, it needs to be written down in forms that are hard to quietly revise. A university serious about the standard in Alcatraz's own guide could publish, before any expansion, answers to a short list of questions. Which features are switched on, including tailgating snapshots of people who have not enrolled? How long are templates and event logs kept? The police department's pilot page does not say, promising only that details on data use and retention will be shared during enrolment. Who can see them, and when would they go to police? What would have to happen, and who would have to agree, before face entry reached libraries, dining halls or exam rooms? And will a card or phone always remain a fully equal alternative, rather than the slow lane?

None of those questions requires opposing the technology. They require only that the choice offered to students today is still a choice for the students who arrive next year.

The first door a young person learns to open with their face is a lesson in what their face is for. It can teach them that their body is a password they control, lent to an institution for a specific purpose on specific terms. Or it can teach them that being recognised is simply the price of coming home. The black box beside Woodland Glen IV cannot tell the difference. The people who decide how it is configured can.

---

References

1. University of Kentucky Police Department, “Biometric Access”, police.uky.edu, accessed 24 September 2026. https://police.uky.edu/safety/biometric-access
2. Alyssa Williams, “UK installs AI facial scanning technology at residence halls”, WKYT, 19 September 2026. https://www.wkyt.com/2026/09/19/uk-installs-ai-facial-scanning-technology-residence-halls/
3. Leigh Searcy, “University of Kentucky tests AI facial scanning for campus building access”, LEX18, 17 September 2026. https://www.lex18.com/news/covering-kentucky/university-of-kentucky-tests-ai-facial-scanning-for-campus-building-access
4. Jesse Fraga, “UK installing AI scanners that take 'facial signatures' at some buildings. Here's where”, Lexington Herald-Leader (via NewsBreak), September 2026. https://www.newsbreak.com/lexington-herald-leader-1592576/4889066180360-uk-installing-ai-scanners-that-take-facial-signatures-at-some-buildings-here-s-where
5. Elizabeth Nolan Brown, “Scan your face to enter your dorm? University of Kentucky installs facial recognition tech around campus”, Reason, 23 September 2026. https://reason.com/2026/09/23/scan-your-face-to-enter-your-dorm-university-of-kentucky-installs-facial-recognition-tech-around-campus/
6. Alcatraz, “Alcatraz Raises $50 Million in Series B Funding”, press release, 2 April 2026. https://www.alcatraz.ai/press-release/alcatraz-raises-50-million-series-b
7. Alcatraz, “Privacy-First Facial Authentication Guide for Universities”, 2 February 2026. https://www.alcatraz.ai/blog/privacy-first-facial-authentication-the-modern-standard-for-higher-education
8. Alcatraz, “Tailgating Detection System”, product page, accessed 24 September 2026. https://www.alcatraz.ai/solutions/tailgating-detection-system
9. Alcatraz AI Help Center, “Auto, Manual and Web Enrollment”, accessed 24 September 2026. https://support.alcatraz.ai/enrollment
10. Security Sales and Integration, “Startup Alcatraz AI Aims to Replace Badges With Frictionless Access Control”, 14 March 2019. https://www.securitysales.com/news/alcatraz-frictionless-access-control/91116/
11. Office of Senator Jeff Merkley, “Before Busy Summer Travel Season, Merkley, Kennedy, Markey, Marshall Push to Rein in TSA Facial Recognition Technology at Airports Across U.S.”, 8 May 2025. https://www.merkley.senate.gov/before-busy-summer-travel-season-merkley-kennedy-markey-marshall-push-to-rein-in-tsa-facial-recognition-technology-at-airports-across-u-s/
12. Algorithmic Justice League, “Comply To Fly? How Airport Travelers Experience TSA's Facial Recognition Experiment”, 2025. https://www.ajl.org/flyreport
13. Nora A. Draper and Joseph Turow, “The corporate cultivation of digital resignation”, New Media and Society, vol. 21, no. 8, 2019, pp. 1824-1839. https://doi.org/10.1177/1461444819833331
14. Helen Nissenbaum, “Privacy as Contextual Integrity”, Washington Law Review, vol. 79, no. 1, 2004, pp. 119-157. https://nyuscholars.nyu.edu/en/publications/privacy-as-contextual-integrity
15. Mark Andrejevic and Neil Selwyn, “Facial recognition technology in schools: critical questions and concerns”, Learning, Media and Technology, vol. 45, no. 2, 2020, pp. 115-128. https://doi.org/10.1080/17439884.2020.1686014
16. Wang, Hou, Hong, Yang and Zhang, “Impact of Face-Recognition-Based Access Control System on College Students' Sense of School Identity and Belonging During COVID-19 Pandemic”, Frontiers in Psychology, 2022. https://pmc.ncbi.nlm.nih.gov/articles/PMC8889067/
17. Steve Jess, “Can the Way Students Use ID Cards Predict Whether They Drop Out?”, Arizona Public Media, 2 April 2018. https://news.azpm.org/p/news-articles/2018/4/2/126689-can-the-way-students-use-id-cards-predict-dropouts/
18. ABC News (Australia), “Police access SafeWA app data for murder investigation, prompting urgent law change”, 15 June 2021. https://www.abc.net.au/news/2021-06-15/safewa-app-sparks-urgent-law-change-after-police-access-data/100201340
19. Kentucky General Assembly, House Bill 15 (2024 Regular Session), Acts Chapter 72, Kentucky Consumer Data Protection Act. https://apps.legislature.ky.gov/law/acts/24RS/documents/0072.pdf
20. Privacy World, “Federal Court Dismisses Biometric Privacy Class Action Brought Against University, On Basis It Was a Regulated 'Financial Institution'”, November 2022 (Powell v. DePaul University, N.D. Ill., No. 21-C-3001). https://www.privacyworld.blog/2022/11/federal-court-dismisses-biometric-privacy-class-action-brought-against-university-on-basis-it-was-a-regulated-financial-institution/
21. New York State Education Department, “State Education Department Issues Determination on Biometric Identifying Technology in Schools”, 27 September 2023. https://www.nysed.gov/news/2023/state-education-department-issues-determination-biometric-identifying-technology-schools
22. Lilah Burke, “UCLA drops plan to use facial recognition security surveillance, but other colleges may be using technology”, Inside Higher Ed, 20 February 2020. https://www.insidehighered.com/news/2020/02/21/ucla-drops-plan-use-facial-recognition-security-surveillance-other-colleges-may-be
23. IEEE Spectrum, “Facial Recognition Errors Affect Millions Globally”, 30 March 2026. https://spectrum.ieee.org/facial-recognition-gone-wrong
24. Keyonna Henson, “UK brings AI facial authentication scanners to campus”, Kentucky Kernel, 28 September 2026. https://kykernel.com/128434/news/uk-brings-ai-facial-authentication-scanners-to-campus/
25. Rodney Bosch, “Alcatraz Introduces RockOS, Rock Cortex for Facial Authentication Platform”, SecurityInfoWatch, 17 September 2026. https://www.securityinfowatch.com/access-identity/biometrics/facial-recognition-solutions/news/55406004/alcatraz-introduces-rockos-rock-cortex-for-facial-authentication-platform

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Somewhere behind a police call in Heber City, Utah, a television was playing Disney's The Princess and the Frog. The officer's body-worn camera picked it up, because body-worn cameras pick up everything: the hum of a refrigerator, a dog, a film. That audio went into an artificial intelligence system the department had bought to turn recordings into police reports. The system did exactly what it was built to do. It listened, it narrated, and it produced a document in which the officer turned into a frog.

The story went around the world in January 2026. Sergeant Rick Keel of the Heber City Police Department told FOX 13 News that “the body cam software and the AI report writing software picked up on the movie that was playing in the background, which happened to be 'The Princess and the Frog.'” He added the line every department has since repeated in some form: “That's when we learned the importance of correcting these AI-generated reports.” FOX 13 reported the offending narrative came from Draft One, sold by Axon, the largest supplier of body cameras to American police. Heber City had been trialling it alongside a rival called Code Four. The department kept using AI. It simply added more review.

Here is the thing about the frog. It is the safest error the technology can make, because it is self-cancelling. No sergeant approves it, no prosecutor files it, no judge reads it without laughing, and no defendant goes to prison because of it. An error that absurd is a smoke alarm, not a fire. It tells you the machine is in the building.

The error that should frighten you is the one that reads perfectly. It is grammatical, plausible, consistent with the rest of the narrative, and wrong by a single preposition. It is the difference between a suspect who reached toward his waistband and a suspect who reached into it. One describes an ambiguous movement. The other describes a man going for a gun, and can justify a shooting, a search, a charge, a plea. They differ by two letters, and no reviewer on earth reliably catches that at eleven at night at the end of a twelve-hour shift, six calls deep, with the machine's version already on the screen looking finished.

That asymmetry is the whole problem. And Wyoming has just walked into it.

What the Cowboy State Bought Before It Decided Anything

On 23 August 2026, Cowboy State Daily reporter Kolby Fedore published an account of police departments in Casper, Evanston and Sheridan each testing artificial intelligence to write reports and look at evidence. Only one vendor was named. Evanston, according to Police Chief Mike Vranish, is testing Code Four, software founded in April 2025 by two MIT students, George Cheng and Dylan Nguyen, and now used by roughly 165 law enforcement agencies. Code Four, Fedore noted, markets itself as “The AI Detective That Never Sleeps.”

The company's own site is less whimsical and more instructive. It promises a 65 per cent average reduction in report writing time, fifty dollars per officer per month, and eighty thousand dollars in annual budget saved per department. Its report product “ingests audio, visual, and other meta data from each frame of an officer's body-worn camera to create a comprehensive narrative of every situation.” A companion product, INSIGHTS, described in sponsored content Cheng wrote for the trade publication Police1 in July 2026, pulls “digital evidence, records and jail-management data, body-camera footage, phone dumps, call logs, recorded audio and case files into a single layer a detective can question directly.” It can draft warrants, affidavits and prosecutor jackets.

Read that list slowly. A warrant application persuades a judge to let the state enter your home; an affidavit of probable cause persuades a judge to hold your body. These are not administrative artefacts. They are the hinges on which constitutional rights swing.

Fedore's reporting established the gap that gives this story its edge. No Wyoming law specifically requires an original AI-generated police report to survive after an officer edits it, and none requires the finished report to disclose that a machine helped write it. The technology is in the building. The rules are not.

Gay Woodhouse, Wyoming's thirty-second attorney general from 1998 to 2001 and the first woman to hold the office, previously an assistant United States attorney and chief deputy attorney general, told Cowboy State Daily that “the thing that worries me about AI is just the ability to hallucinate,” and that Wyoming needs “to get on it.” She put her finger on the mechanism: “The main thing in criminal prosecutions is disclosure.” And then, with the dry economy of a former prosecutor: “It'd be awful to lose a case because you used AI and didn't disclose it.”

Casper criminal defence attorney Keith Nachbar called it “uncharted territory” and delivered the sentence that ought to be nailed above every police chief's desk in the state: “AI isn't going to be taking the witness stand.” Then a question nobody in Wyoming can answer: “How do you look behind the curtain and figure out what AI thought?”

Evanston Police Chief Mike Vranish supplied the counterweight every department offers. AI can do in minutes what takes officers hours. But a camera cannot smell the gunpowder hanging in a room, or the alcohol on a driver's breath. So, said Vranish, “everything's got to be verified and validated by an actual person.”

That last sentence is the load-bearing beam of the entire industry. We will come back to whether it holds.

Two Letters and a Life

A generative model producing a police narrative from body-camera audio and video is not reasoning about what happened. It is producing the most probable next token given everything before it. When the input is clear, the output is usually fine. When the input is ambiguous, and a real incident is nothing but ambiguity, the model does not flag the ambiguity. It resolves it, picking the likelier phrasing and writing it with the same fluency it uses for the parts it got right. There is no tonal difference between the sentences it knows and the sentences it guessed.

Human officers make errors too, and police reports have always been contested documents. The difference is the shape of the error. A tired officer produces a report with the texture of exhaustion: gaps, hedges, “I believe”, “it appeared that”, the occasional flat contradiction a defence lawyer can hold up in cross-examination. Those artefacts are evidence, telling a jury how confident the witness actually was. Machine prose has no such texture. It is uniformly confident, smoothing hedges into assertions because assertions are more probable in the training distribution of police reports than hedges are. The uncertainty that lived in the officer's memory is laundered out before anyone reads it.

The research base bears this out with unusual precision. In May 2026, criminologist Ian T. Adams of the University of South Carolina and colleagues Matt Barter, Kyle McLean, Irick A. Geary Jr, Alexis Fabila and Josh McCrain published a study on CrimRxiv titled “A Good College Essay but a Bad Police Report.” They ran a triple-blind evaluation in which 92 expert reviewers, sergeants and above with an average of 21.8 years of responsibility for approving reports, assessed 80 police reports, 20 drafted with Axon's Draft One. The reviewers did not know which was which.

Two findings matter. AI-assisted reports were rated substantively and significantly worse on accuracy, an effect moving a report from roughly the fiftieth percentile to the thirty-sixth, and text analysis found them longer-worded, more complex and less readable. They read like good college essays. They were worse police reports. And the expert reviewers could not tell which reports were written with AI. Their accuracy on that question was no better than a coin flip.

Hold those together. The exact people whose review is supposed to be the safeguard rated AI reports as less accurate whilst being unable to identify them. They sensed something was off and could not locate it. That is the signature of a failure mode human review is structurally poor at catching.

The Signature Stays, the Perception Goes

A police report's evidentiary power does not come from its literary quality. It comes from provenance. It is a contemporaneous account written by a person who was present, who perceived events through functioning senses, who can be cross-examined about those perceptions, who can be impeached with prior inconsistent statements, and who faces professional and sometimes criminal jeopardy for falsifying it. Every one of those guarantees attaches to a human being standing in a room.

Chief Vranish put it better than most legal scholars: a camera cannot smell the gunpowder hanging in a room, or the alcohol on a driver's breath. That is not a quaint remark about sensor limitations. It is a statement about the epistemology of the document. The camera captures a slice of a person's perception; the model writes a narrative from the slice and presents it as the whole.

What AI drafting does, quietly, is sever the sentence from the perception whilst leaving the signature in place. The officer still signs. The report still enters the file as his account. But a proportion of its sentences, and nobody can say which, originated not in his memory of the room but in a model's reconstruction of what a room like that usually sounds like.

This is what the machine is for. Comprehensiveness, which Code Four promises, is precisely the property a human account does not have and should not claim. Officers miss things, and the gaps in an honest report are informative. A machine that fills them is not improving the document; it is contaminating it with plausible invention.

The academic literature has caught up faster than the statutes have. In March 2026, a team led by Muffy Calder and Marion Oswald, with Elizabeth McClory-Tiarks, Michele Sevegnani and Evdoxia Taka, submitted a paper to arXiv titled “Responsible AI in criminal justice: LLMs in policing and risks to case progression.” They mapped 15 policing tasks that large language models could plausibly handle, from transcription and translation to witness statement generation, corroboration identification and police report generation, and set against them 17 categories of risk illustrated with more than 40 worked examples of how those risks propagate through a case.

Their taxonomy of output risk begins with completeness, meaning omissions and additions, and accuracy, meaning outright factual error. In one proof-of-concept trial they describe, a system produced a description of a witness as “a woman of average height and brown hair” when no such description had been given in the interview. Nobody was turned into an amphibian. A physical description simply appeared, from nowhere, in a document later read as a record of what a witness said. That is the plausible error, and it survives review because there is nothing about it to catch.

The Draft That Was Never Born

Now the design question, where this stops being a technology story and becomes a legal one.

In July 2025, Electronic Frontier Foundation investigators Matthew Guariglia and Dave Maass published a records-based investigation into Axon's Draft One under the headline “Axon's Draft One Is Designed to Defy Transparency.” Their central finding was not that the software hallucinated. It was that the software forgets.

Draft One, they found, does not save the draft it generates, nor any edited versions. An officer generates a narrative, copies it into the report, and the draft disappears when the window closes. The EFF quoted an Axon official explaining the rationale with startling candour: “the last thing we want to do is create more disclosure headaches for our customers.”

Sit with that sentence. It is not an engineering trade-off. It is a legal strategy. The product does not retain the machine's first draft because retaining it would generate discoverable material, and discoverable material creates work for prosecutors and ammunition for the defence.

The downstream effects landed where you would expect. Auditing meant trawling individual logs or keyword-searching, and only where a department had voluntarily inserted disclosure language. The Palm Beach County Sheriff's Office, which required a disclosure line, could identify more than three thousand Draft One reports. The Lafayette Police Department could not identify one, having no marker to search for.

Non-retention is not an oversight. It is the feature that makes the practice legally survivable, converting an auditable pipeline into an unauditable one by destroying evidence before anyone has standing to ask for it.

The constitutional problem arrives immediately. Under Brady v. Maryland, decided in 1963, prosecutors must disclose evidence favourable to the accused that is material to guilt or punishment, regardless of good faith. Under Giglio v. United States, decided in 1972, that duty extends to information bearing on the credibility of government witnesses. A machine-generated first draft that differs materially from the signed report is, by definition, impeachment material about the officer's account. If the report says the suspect reached into his waistband and the draft said toward, the defence is entitled to know. If the officer corrected a dozen machine errors, that speaks to how much of the final document is his own recollection. If he corrected none, that speaks louder still.

You cannot disclose what does not exist. And when the non-existence is engineered by the vendor for the stated purpose of avoiding disclosure headaches, the ordinary comfort that Brady is a duty rather than a preservation mandate looks thin.

The Calder and Oswald team recorded what happens when a force takes preservation seriously. In a Hertfordshire Constabulary pilot of an AI interview assistant, the need “to disclose all versions of the transcript and statement produced by officers when using ADA ... impacts on workload for the Disclosure officer.” Doing this lawfully is expensive. Doing it cheaply means not doing it lawfully.

The Witness Who Cannot Be Called

Nachbar's line, that AI is not going to be taking the witness stand, is not a flourish. It sits on an unresolved constitutional question American courts have circled for twenty years.

The Sixth Amendment guarantees a defendant the right to be confronted with the witnesses against him. In Crawford v. Washington, decided in 2004, the Supreme Court held that testimonial statements cannot be admitted unless the declarant is unavailable and the defendant has had a prior opportunity for cross-examination. In Melendez-Diaz v. Massachusetts, decided in 2009, Justice Scalia wrote that forensic certificates of analysis were testimonial statements akin to affidavits, so the analysts who prepared them were witnesses the defendant could confront. In Bullcoming v. New Mexico, decided in 2011, Justice Ginsburg wrote for a five to four majority that the state could not launder that requirement by putting a different analyst on the stand to describe a report he had neither signed, performed nor observed. Surrogate testimony does not satisfy the clause.

The line those cases draw is about accountability for accusation. The state does not get to accuse through a document and then produce someone who can only vouch for the process in general terms.

An AI-drafted report sits awkwardly across that line, in a way that favours the prosecution. Machine output is generally not treated as a “statement” by a “witness” at all, because the Confrontation Clause has been read to govern human declarants. The Berkeley law professor Andrea Roth mapped this terrain in “Machine Testimony,” published in the Yale Law Journal in 2017. Machine sources of accusation, particularly proprietary software built for litigation, may function as witnesses against a defendant, she argued, and “black box dangers,” meaning errors that render a machine false by design, inarticulate or analytically unsound, are analogous to the hearsay dangers confrontation exists to expose. Machine testimony substituted for human testimony lets the state evade responsibility for its accusations.

Now apply it. The officer takes the stand. He is confrontable, and can be asked what he saw, what he smelled, what he remembers. But the sentence being challenged may not have originated with him. It came from a model. He can be asked whether he agrees with it now, which is a different question. And the thing that produced it cannot be cross-examined on its reliability, its error rate, its behaviour when the audio is poor, or its tendency to resolve ambiguity towards the incriminating reading.

The officer becomes, functionally, a surrogate witness for a process he did not perform and cannot describe. Bullcoming says the state cannot do that with a laboratory analyst. Nothing yet says it cannot do it with a language model. That gap is Nachbar's question about looking behind the curtain, restated in constitutional terms, and the current answer in Wyoming is that you cannot, because the curtain does not exist and neither does the draft behind it.

When the File Becomes the Memory

The sharpest point in the Cowboy State Daily piece is also the least discussed anywhere else. Fedore's account of Nachbar's concern runs like this. Six months on, an officer may no longer independently remember whether a suspect reached toward his waistband or into it, precisely what somebody said, or where everyone was standing. He returns to the report to refresh his memory, as officers routinely do before testifying. If the machine put the wrong detail there first, the report can begin informing the memory it was supposed to preserve.

This practice is not a loophole. It is formally sanctioned. Federal Rule of Evidence 612, and its analogues across the states, governs writings used to refresh a witness's memory, granting the adverse party the right to produce the writing, inspect it, cross-examine on it and introduce relevant portions. The rule assumes a document written by the witness, from the witness's own perception, reactivating the witness's own encoded memory. It has teeth, too. Under subsection ©, if a writing is not produced as ordered and the prosecution is at fault in a criminal case, the court must strike the witness's testimony or, if justice so requires, declare a mistrial. That mandatory remedy was written for a world in which the writing still exists.

The psychological literature has spent fifty years showing that the assumption underneath Rule 612 is fragile. The misinformation effect, the body of work most associated with Elizabeth Loftus, who published a retrospective on the idea's history in Legal and Criminological Psychology in 2026, shows that memory for an event is not sealed at the moment of the event. Post-event information modifies the memory itself, and misleading detail embedded in an apparently trustworthy narrative account is one of the classic delivery mechanisms. People do not merely repeat the misinformation. They come to remember it, with confidence, as something they perceived.

Now consider what an AI-drafted report is, structurally. It is post-event information, embedded in an apparently trustworthy narrative account, carrying the officer's own signature, which makes it maximally authoritative to him. It is reviewed under time pressure, then read again months later to reconstruct details he no longer independently recalls.

That is not a document refreshing a memory. That is close to an optimal protocol for installing one.

The consequence is that the testimony a jury hears may be sincere, confident, internally consistent, and derived from a sentence a language model generated because it was probable. The officer is not lying. He genuinely remembers it that way now. Cross-examination, built to detect deception and inconsistency, is close to useless against a witness who has been honestly contaminated. And the one artefact that could have exposed it, the original draft showing that “into” was never his word, has been deleted by design.

The Load-Bearing Human

Every deployment of this technology rests on one assurance, and Vranish stated it in its purest form: everything has to be verified and validated by an actual person.

The research on whether humans perform that role well is not encouraging, and it is not new. Raja Parasuraman and Dietrich Manzey's 2010 review in Human Factors synthesised decades of evidence that people under-monitor automated systems and accept incorrect automated advice rather than challenge it, even when contradictory information is available. The effect shows up in aviation, medicine, process control and the military, driven partly by a perception that automated systems are inherently superior and partly by the cognitive cost of checking.

Automation bias is worst under exactly the conditions that motivate buying this software: high workload, time pressure, fatigue and a high base rate of correct outputs, which trains the reviewer to expect correctness. A tool that is right ninety-five per cent of the time is more dangerous to review than one right sixty per cent of the time, because the reviewer stops looking.

The vendors know this. Code Four knows it well enough to have engineered a countermeasure that is also, inadvertently, a confession. As StateScoop's Keely Quinlan reported in July 2026, the company embeds hidden “Easter eggs” in AI-generated drafts, seldom-used words such as “axolotl,” which officers must find and remove before filing. Chief executive George Cheng explained the logic: “That basically prevents them from just copy and pasting everything from the AI at face value, and they actually have to make some edits.”

Read that as an admission, because it is one. The company built a booby trap into its own output because it did not trust officers to read the output. That is a more damning verdict on human review than anything a defence lawyer has said, and it does not solve the problem. Removing an obviously alien word proves the officer scanned for an alien word. It proves nothing about whether he verified that “into” should have been “toward.”

And the productivity premise that justifies accepting the risk is itself unproven. Adams and colleagues published a pre-registered randomised controlled trial in the Journal of Experimental Criminology in 2024, titled “No man's hand: artificial intelligence does not improve police report writing speed,” covering 85 officers across 755 reports with a check across 6,084 reports over a full year. AI assistance did not significantly affect how long reports took. In May 2026 the ACLU's Jay Stanley set that null result alongside a 2025 study by Walley and Glasspoole-Bird finding an 18.5-minute increase in time per incident.

So the ledger reads: unverified time savings, measurably worse accuracy, reviewers who cannot identify AI text at better than chance, a documented tendency to rubber-stamp machine output, and a vendor countermeasure that concedes the point. Against that, a claimed sixty-five per cent cut in report writing time and eighty thousand dollars a year.

The Exemption That Wrote Itself

Wyoming did have a chance to draw a line, and what happened to it is instructive.

House Bill 91 of the 2026 budget session would have barred government entities from using artificial intelligence to calculate and assign social scores, and from identifying people using biometric data or gathered images. It reached the House Judiciary Committee on 10 February 2026, and on 19 February the committee recommended amendment and passage. Cowboy State Daily reported that the amendment specifically exempted law enforcement uses of AI for investigating, reporting or prosecuting violations of the criminal code, and that the committee approved it seven to one. The bill died without reaching a vote of the full House.

State the sequence plainly. The one piece of Wyoming legislation aimed at constraining government AI was amended so that the most consequential use of it, the production of documents that put people in cells, was carved out. Then it failed anyway.

Other states moved differently. Utah's SB 180, sponsored by Senator Stephanie Pitcher, a Democrat from Millcreek and a former prosecutor, passed unanimously and took effect on 7 May 2025. It requires agencies to hold a written generative AI policy, a disclaimer on any AI-assisted report, and officer certification that the output was reviewed for accuracy. Pitcher's framing, quoted by Governing's Logan Seacrest in July 2026: “I felt like it was something that it's better for us to get ahead of, instead of playing catch-up.”

California went further. Senate Bill 524, signed by Governor Gavin Newsom on 10 October 2025 and effective from 1 January 2026, requires a written disclosure on every page of any report produced fully or partly by AI, bars vendors from selling the information agencies feed into the system, and, critically, requires departments to retain the first draft along with audit trails identifying the footage or audio that informed it. That retention provision is the mechanism letting judges, defence attorneys and auditors see which portions of a report the officer wrote and which the computer did. California Senator Jesse Arreguín, quoted by Seacrest: “We're not going to gamble with personal liberty. AI hallucinations happen at significant rates.”

As of October 2026, those remain the only two states to have enacted laws on AI-generated police reports, according to a state-by-state tracker maintained by Barricade. Everywhere else the attempts have stalled. New York's S10425/A9253-A lapsed when the Legislature adjourned on 5 June 2026, with reintroduction expected in 2027. Minnesota's SF 4575/HF 4536 died without a hearing in May. Virginia's HB 1294 was carried over to 2027. Even Utah's own 2026 follow-up, SB 205, which would have required annual policy review and public posting of policies, died in the House in March.

Some prosecutors have not waited. The King County Prosecuting Attorney's Office in Washington, which covers Seattle, instructed police not to submit AI-generated narratives, citing concerns about the products on the market. The people who have to prove these cases beyond reasonable doubt decided the documents were a liability.

They are not alone. In Hays County, Texas, KUT 90.5's Leigh Walden reported in September 2026, the sheriff's office trialled Draft One for three months before county commissioners voted three to two to deny funding for an expanded Axon AI contract worth $4.5 million over ten years. In nearby Kyle, the police department uses several Axon AI tools but not Draft One for reports, at the request of the district attorney's office. Gregg Cox, Hays County's first assistant district attorney, explained why the document matters: “The offense report serves a lot of purposes.” It drives charging decisions, plea offers and whether a case goes to trial. “I would much rather have them write a detailed report themselves based on what they observed,” he said, worrying that machine drafts could weaken officers' own recall of what happened. That is the memory contamination problem, seen from the prosecutor's desk.

This is not a peculiarly American drift. In a paper submitted to arXiv in December 2025, Evdoxia Taka, Temitope Lawal, Kyriakos Kotsoglou and colleagues including Calder, Sevegnani, McClory-Tiarks and Oswald set out a methodology for charting probabilistic AI tools across the criminal justice stages of England and Wales, reporting heavy reliance on private sector providers alongside growing interest in large language models. Adoption is happening force by force, vendor by vendor, ahead of any governance layer.

Public deliberation, where anyone has attempted it, is more discriminating than procurement. In August 2026, Mackenzie Jorgensen, Jo Reilly, Alex Sutherland and Miri Zilka submitted a study to arXiv describing a workshop in which 30 community representatives, police officers and academics assessed 13 AI use cases in policing with an explicit focus on racial bias. On AI-assisted processing of reports, participants asked whether time saved is offset by time spent verifying. Foregrounding racial equity, the authors found, did not narrow the deliberation but pushed it towards first-principles questions: does this tool work, will it deliver genuine benefit, and will that benefit reach everyone.

Nobody asked those questions in Casper before the trial started, because in Wyoming nobody has to.

Nobody asked them when it ended, either. In a report syndicated by GovTech on 8 October 2026, the Casper Star-Tribune's Dylan Farrell described how Casper City Council had unanimously approved a one-year, $58,800 contract for TRULEO, letting the police department use the software full-time after a test the council had approved on 7 April. Both decisions went through on the consent agenda, the part of a meeting reserved for business considered too routine to debate, and the contract passed with little public discussion. TRULEO advertises AI-assisted report writing, summaries of encounters, automated review of body-camera footage and “AI-assisted interviews.” Most council members declined to comment. The coverage contains no statement about whether defendants will be told when a machine helped write the reports against them.

What Would Actually Fix This

The answer to Nachbar's closing question, what happens to the right to a fair trial when the ghost author of the foundational document cannot be questioned, is not that trials become unfair in some diffuse atmospheric sense. It is more specific than that.

A fair trial is an adversarial testing mechanism, and it works by making every link in the chain from perception to verdict contestable. The defendant can challenge the search, the arrest, the identification, the forensic analysis, the officer's account and the officer's credibility. Each challenge targets a human who can be put under oath and asked to explain themselves.

AI drafting inserts a link immune to every one of those tools. The model cannot be sworn, cannot be impeached, cannot be asked why it chose a preposition, and cannot be shown to have been wrong before, because the record of its previous drafts has been deleted. It has no jeopardy, so it has no incentive to accuracy. And because its output is fused into a signed human document, the defence cannot identify which links are machine-made. The result is not a rigged trial. It is a trial testing the wrong thing: an officer's present agreement with a narrative rather than his actual perception of an event.

That is fixable, and it does not require Wyoming to ban the technology. Four requirements would do most of the work.

Retain the original draft. Every machine-generated first draft should be preserved, versioned, timestamped and disclosed in discovery alongside the final report. California has shown a legislature can write this. It costs storage, which is cheap, and it restores the ability to see what changed. Without it, Brady and Giglio obligations attach to an artefact engineered out of existence, and Rule 612's production requirement, with its mandatory strike-or-mistrial remedy, applies to a document nobody can produce.

Disclose on the face of the report. Not in a settings menu, not as an optional agency policy, but as a mandatory statement on the document itself, as California requires on every page. Woodhouse's warning is the operative one: it would be awful to lose a case because you used AI and did not disclose it. It would be worse to win one.

Attach provenance metadata. The report should record which model and version produced it, what source material it consumed, and which body-camera or audio file it derived from. Systems engineering risks in the Calder and Oswald taxonomy include version tracking, tool chaining and supplier dependency for a reason. A defence expert cannot evaluate a system that will not say what it is, and a system silently updated between the incident and the trial cannot be tested at all.

Prohibit AI drafting where the officer's own perception is the evidence. Where the contested fact is what an officer saw, heard, smelled or believed in the moment, and above all in use-of-force cases, the account must be written by the human before he is shown any machine draft, exactly as a witness statement is taken before the witness is shown someone else's version. Vranish made the argument for this rule himself without quite drawing the conclusion. If the camera cannot smell the room, the sentences describing the room should not come from the camera.

None of this is technologically demanding. It has not happened because the current arrangement suits everyone with power in the room. Departments get their hours back, or believe they do. Vendors avoid disclosure headaches. Prosecutors, most of them, do not have to think about it.

The person for whom the arrangement is not convenient is the defendant, who is the one party in the courtroom without a procurement budget, and whose case now rests on a document with an author who cannot be called, cannot be cross-examined, and whose first version was deleted before anyone knew there would be a trial. Wyoming is running the experiment on him right now, in three towns, at fifty dollars per officer per month in Evanston and a little under sixty thousand dollars a year in Casper.

References

1. Fedore, Kolby. Cowboy State Daily, 2026. “Wyoming Police Departments Testing AI To Generate Reports, Look At Evidence.” https://cowboystatedaily.com/2026/08/23/wyoming-police-departments-test-using-ai-to-generate-reports-look-at-evidence/
2. Code Four. Company website and product pages, 2026. https://www.codefour.us/
3. Cheng, George. Police1 (sponsored content by Code Four), 2026. “The operating system the modern detective never had.” https://www.police1.com/police-products/investigation-software/the-operating-system-the-modern-detective-never-had
4. Quinlan, Keely. StateScoop, 2026. “Police AI platform uses 'Easter eggs' to verify human review.” https://statescoop.com/police-ai-platform-code-four-human-review/
5. Guariglia, Matthew and Maass, Dave. Electronic Frontier Foundation, 2025. “Axon's Draft One Is Designed to Defy Transparency.” https://www.eff.org/deeplinks/2025/07/axons-draft-one-designed-defy-transparency
6. Calder, Muffy; Oswald, Marion; McClory-Tiarks, Elizabeth; Sevegnani, Michele; and Taka, Evdoxia, 2026. “Responsible AI in criminal justice: LLMs in policing and risks to case progression.” arXiv:2603.18116. https://arxiv.org/abs/2603.18116
7. Jorgensen, Mackenzie; Reilly, Jo; Sutherland, Alex; and Zilka, Miri, 2026. “Negotiating Risk Boundaries in AI for Policing Through Mixed-Stakeholder Deliberation.” arXiv:2608.05418. https://arxiv.org/abs/2608.05418
8. Taka, Evdoxia; Lawal, Temitope; Calder, Muffy; Sevegnani, Michele; Kotsoglou, Kyriakos; McClory-Tiarks, Elizabeth; and Oswald, Marion, 2025. “Mapping the Probabilistic AI Ecosystem in Criminal Justice in England and Wales.” arXiv:2512.04116. https://arxiv.org/abs/2512.04116
9. Adams, Ian T.; Barter, Matt; McLean, Kyle; Geary, Irick A. Jr; Fabila, Alexis; and McCrain, Josh. CrimRxiv, 2026. “A Good College Essay but a Bad Police Report: A Triple-Blind Expert Evaluation of AI-Assisted Police Reporting.” https://www.crimrxiv.com/pub/u7azgqzd/release/1
10. Adams, Ian T. et al. Journal of Experimental Criminology, 2024. “No man's hand: artificial intelligence does not improve police report writing speed.” https://link.springer.com/article/10.1007/s11292-024-09644-7
11. Stanley, Jay. American Civil Liberties Union, 2026. “Studies Question Value of AI-Assisted Police Reports.” https://www.aclu.org/news/privacy-technology/studies-question-value-of-ai-assisted-police-reports
12. Seacrest, Logan. Governing (R Street Institute), 2026. “The Risky Marriage of AI and Police Reports.” https://www.governing.com/policy/the-risky-marriage-of-ai-and-police-reports
13. Constantino, Mya. FOX 13 News Utah, 2026. “AI programs used by Heber City police claim officer turned into a frog.” https://www.fox13now.com/news/local-news/summit-county/how-utah-police-departments-are-using-ai-to-keep-streets-safer
14. Roth, Andrea L. Yale Law Journal, volume 126, 2017. “Machine Testimony.” https://yalelawjournal.org/article/machine-testimony
15. Loftus, Elizabeth F. Legal and Criminological Psychology, 2026. “The history of an idea: The misinformation effect.” https://bpspsychub.onlinelibrary.wiley.com/doi/10.1111/lcrp.70020
16. Supreme Court of the United States. Brady v. Maryland, 373 U.S. 83 (1963). https://supreme.justia.com/cases/federal/us/373/83/
17. Supreme Court of the United States. Giglio v. United States, 405 U.S. 150 (1972). https://supreme.justia.com/cases/federal/us/405/150/
18. Supreme Court of the United States. Melendez-Diaz v. Massachusetts, 557 U.S. 305 (2009). https://tile.loc.gov/storage-services/service/ll/usrep/usrep557/usrep557305/usrep557305.pdf
19. Supreme Court of the United States. Bullcoming v. New Mexico, 564 U.S. 647 (2011). https://tile.loc.gov/storage-services/service/ll/usrep/usrep564/usrep564647/usrep564647.pdf
20. Legal Information Institute, Cornell Law School. Federal Rule of Evidence 612, “Writing Used to Refresh a Witness.” https://www.law.cornell.edu/rules/fre/rule_612
21. Parasuraman, Raja and Manzey, Dietrich H. Human Factors, 2010. “Complacency and Bias in Human Use of Automation: An Attentional Integration.” https://journals.sagepub.com/doi/10.1177/0018720810376055
22. Wyoming Legislature, 2026 Budget Session. House Bill 91, government use of artificial intelligence. https://legiscan.com/WY/bill/HB0091/2026
23. Farrell, Dylan. Casper Star-Tribune, via GovTech, 2026. “In Casper, Wyo., a Green Light for AI Police Bodycams.” https://www.govtech.com/artificial-intelligence/in-casper-wyo-a-green-light-for-ai-police-bodycams
24. Barricade, 2026. “AI Police Report Laws by State: 2026 Tracker” (updated October 2026). https://barricade.tech/articles/ai-police-report-laws-by-state
25. Walden, Leigh. KUT 90.5, 2026. “Austin-area law enforcement is testing AI-assisted police reports.” https://www.kut.org/crime-justice/2026-09-30/police-ai-hays-county-san-marcos-kyle-district-attorney

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Looking for a flat used to mean filters. You set a price ceiling, ticked a box for parking, drew a rough circle on a map and scrolled. Now, increasingly, you talk. On 6 October 2026, Palo Alto Online published a report by Angela Swartz on how artificial intelligence is changing the apartment search in Silicon Valley's backyard, and the picture it painted was of a market where the listing, the questions and the answers are all starting to pass through software that speaks in fluent sentences.

The paper described rental platforms that have turned search into dialogue. Zumper's assistant, Zoe, uses information from a property's Zumper listing to answer questions about an apartment, its community and the surrounding neighbourhood, and helps schedule tours. RentCafe launched its assistant, Ren, in August. Apartments.com introduced conversational search in June. On the landlord side, the report noted that Essex Property Trust uses an AI agent named Eva to greet prospective renters through the websites of complexes including 777 Hamilton in Menlo Park and Radius in Redwood City.

Then there was the renter. A 23-year-old former investment banker told the paper he had used ChatGPT and Claude to compile property management websites and a shortlist of homes matching his price and location. “It did a pretty good job of listing out apartments that fell within those parameters,” he said. “Sometimes it would include things outside of that.” In the end, the report noted, he found an apartment “the old-fashioned way: through word of mouth.”

Zumper's chief technology officer told the paper that “the way people are searching for apartments is changing”, and said the company designed Zoe to minimise hallucinations, the term for instances in which an AI system produces inaccurate or made-up information. But, as the report put it, large language models can still make mistakes. And California's Department of Real Estate has warned that licensees remain responsible for information produced by AI.

That combination of claims raises three questions. What happens to the search for a home when every layer of it is mediated by systems that can state facts about a property that are not true? Who is responsible to a renter who acts on what a system says? And what should a person be entitled to know about how the homes put in front of them were chosen?

A Small Story With a Very Large Footprint

It is worth being precise about what the Palo Alto Online report shows and what it does not. It does not document a renter who lost money because a chatbot lied. It does not show that Zoe, Ren, Eva or Apartments.com's assistant has given anyone a false answer. Its one first-hand account from a renter is modest: a general-purpose chatbot sometimes ignored his limits, and he ended up finding his home through word of mouth.

That modesty is the point. A listing over budget or outside the chosen area is easy to spot and discard. The harder problems are the errors a renter cannot see: an answer that a building allows dogs when it does not, that a unit includes parking when it does not, that a fee is included when it is not, or that a neighbourhood is “a good fit” for reasons that have more to do with who the renter appears to be than with what they asked for.

The same report also described the other side of the counter. A Palo Alto-based property manager told the paper that he had set up a database allowing an AI tool to answer questions and check property managers' schedules to arrange tours, and that he had also seen property managers use the free version of ChatGPT to ask whether an issue needed to be brought to a lawyer or whether a new policy applied to their properties. The machine is answering on both sides of the conversation.

Every Major Portal Now Wants to Talk

The shift the paper described is not a local curiosity. It is an industry-wide bet that the search box is dying.

Zumper first announced Zoe in December 2024. HousingWire reported at the time that it could answer “virtually any renter question” about a property or its neighbourhood, from pet policies to local hotspots, and could request tours and submit leads to leasing teams. In February 2026, Zumper launched a rental search app inside ChatGPT, allowing users to type requests such as “show pet-friendly rentals under $2,500 with in-unit laundry” and receive listings without leaving the chat. In announcing it, the company cited its own survey finding that the share of renters using AI tools during their home search had risen from 4.4 per cent in 2024 to 9.8 per cent in 2025.

RentCafe, owned by the property software firm Yardi, published a feature spotlight on 20 August 2026 explaining that its former AI Mode was now called Ren. “It's an ongoing, two-way conversation,” the post said. “Instead of one search and a list of results, Ren actively guides you to apartments.” It also made a striking promise: “Ren never makes up a property, a price or an amenity. Every detail is sourced directly from trustworthy, up-to-date listings.”

CoStar Group launched Apartments.com Ai on 16 June 2026, describing it as replacing traditional filters and keyword searches with real-time conversation. “Finding an apartment is one of the most important decisions people make, yet the search process has remained largely unchanged for years,” said Andy Florance, CoStar's founder and chief executive. “Apartments.com Ai represents a fundamental shift from searching listings to having a conversation with a trusted rental advisor.”

Behind the portals sit the leasing agents. EliseAI, which sells assistants that answer prospective tenants by text, email and phone and book tours, announced on 29 September 2026 that it had raised $350 million at a $4 billion valuation, and said its platform now powers roughly one in six US apartments.

A renter may now ask a general chatbot for a shortlist, question a portal's branded assistant about a listing, and then be greeted on the building's website by a leasing agent that is also software. At no point is a human obliged to be the one who answers.

Notice the industry's vocabulary: “rental advisor”, “trusted”. It describes a relationship, not a database query. A filter that returns a listing is a tool. An adviser who tells you a flat is right for you is making a representation.

Why Fluent Software Says Things That Are Not True

Why would a system confidently state something false about a property? The short answer is that large language models are built to produce plausible text, and plausibility is not the same as truth.

In September 2025, researchers at OpenAI and Georgia Tech, Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang, published a paper titled “Why Language Models Hallucinate”. Their argument was blunt: language models hallucinate because their training and evaluation procedures reward guessing over acknowledging uncertainty. If a benchmark awards a point for a correct answer and nothing for “I don't know”, a model that always guesses will outscore one that admits ignorance. The authors argued that fixing this requires changing how mainstream benchmarks are scored, not merely adding more hallucination tests.

Rental platforms try to contain the problem by grounding their assistants in listing data. Zoe draws on the property's Zumper listing; Ren, according to RentCafe, on active RentCafe listings; Apartments.com Ai on CoStar's multifamily data. Grounding narrows the space in which a model can invent things. It does not eliminate it.

Grounding fails in at least three ways that matter to renters. The first is that the source can be wrong or stale. A listing that says “pets welcome” when the landlord changed the policy last month will produce a confident, grounded and incorrect answer. The second is that the model can misread or over-generalise from what it retrieves, turning “street parking available” into “parking included”. The third is the question that has no answer in the listing at all. Is this area quiet at night? Is the building safe? Will the landlord accept my housing voucher? A system tuned to be helpful may fill the gap rather than say it does not know.

General-purpose chatbots, the kind the young renter used, sit further still from the data, summarising web pages or relying on training data that may be months old. His complaint is the mildest form of the problem, and a reminder that a model can appear to follow instructions while quietly ignoring them.

California Puts the Liability on the Licensee

California's real estate regulator has been unusually direct about who owns these errors. On 17 March 2026, the Department of Real Estate published an advisory titled “Artificial Intelligence in California Real Estate: Opportunities, Risks, and Compliance Considerations for Licensees”. Among the uses it lists are AI-powered chatbots and automated lead scoring to manage enquiries. Its message about responsibility leaves little room for doubt.

“If an AI tool generates inaccurate information, misleading advertising, or improper communications with consumers,” the advisory says, “responsibility under current law rests with the licensee and their responsible broker, not the technology provider.” It adds that “AI outputs should be reviewed for accuracy and approved by the licensee before being relied upon in transactions or consumer communications”, and that licensees using AI to draft listing descriptions, emails or online advertisements “must independently verify factual claims, including claims involving property features, pricing, availability, and potential uses”. It reminds licensees that “all real estate advertising in California must be truthful, accurate, and not misleading”, and points to section 10176 of the Business and Professions Code, which allows disciplinary action for substantial misrepresentation. It even advises keeping records “showing how AI outputs were reviewed and verified for accuracy and compliance with law.”

The state has also begun legislating against one specific kind of machine-made falsehood: the doctored photograph. Assembly Bill 723, which took effect on 1 January 2026, requires real estate brokers and salespeople who use digitally altered images in advertising to say so and to provide access to the original image. Routine adjustments such as lighting and colour are excluded. In September 2026, according to the California legislature's records, Assembly Bill 2025 was chaptered, extending a similar disclosure duty to advertisements for rental property. Its stated rationale was that virtual staging and AI tools allow anyone marketing a rental to manipulate or fabricate images showing features that are misleading or non-existent.

There is an older California law that also bears on the conversation. Since 1 July 2019, section 17941 of the Business and Professions Code has made it unlawful to use a bot to communicate with a person in California online with the intent to mislead them about its artificial identity in order to incentivise a sale of goods or services, unless the bot is clearly and conspicuously disclosed. An assistant clearly labelled as software is fine. One that passes itself off as a person, in order to close a deal, is a legal problem.

Yet the advisory has an obvious limit. It binds licensees. A real estate broker who lets a chatbot speak for them owns what it says. But a renter asking a general-purpose chatbot to find flats is not dealing with a licensee at all, and a landlord renting out its own property is not necessarily a licensee either. The more the conversation moves to software that sits outside the licensing regime, the more the regulator's clear rule is left guarding a door that renters are no longer walking through.

A Chatbot Is Still Part of the Website

When a renter acts on a wrong answer, the most useful precedent is not American. It comes from a small claims tribunal in British Columbia and a grieving airline passenger.

In February 2024, British Columbia's Civil Resolution Tribunal decided Moffatt v Air Canada. Jake Moffatt had relied on the airline's website chatbot, which told him he could apply for a bereavement fare after travelling. Another page on the same site said otherwise. Air Canada argued, in effect, that it could not be held liable for what its chatbot said. Tribunal member Christopher Rivers called this “a remarkable submission” and rejected it. “While a chatbot has an interactive component, it is still just a part of Air Canada's website,” he wrote. “It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot.” He found negligent misrepresentation and ordered the airline to pay damages, interest and fees.

The ruling binds nobody outside British Columbia's small claims system, and the sum was modest. Its reasoning travels easily, though. Negligent misrepresentation does not care whether the careless sentence was typed by a leasing agent or generated by a model.

In the United States, the defence that platforms have historically relied on is Section 230 of the Communications Decency Act, which protects online services from being treated as the publisher of information provided by someone else. But a generative assistant does not merely host a landlord's listing. It composes new sentences. The closest precedent comes from housing itself. In 2008, the Ninth Circuit Court of Appeals, sitting en banc, held in Fair Housing Council of San Fernando Valley v Roommates.com that the website was not fully immune for content it helped develop, specifically mandatory questions that required users to disclose sex, family status and sexual orientation. When a service itself produces content that runs foul of the law, the court reasoned, it cannot shelter behind immunity designed for third-party speech. How far that logic extends to AI-generated answers about rental property has not been settled by the courts. It is an uncomfortable starting point for any platform arguing that its assistant's words belong to someone else.

The Accountability Chain Has Missing Links

Lay out the parties involved in a single conversational search and the problem of responsibility becomes visible.

There is the landlord or property manager, who supplies the listing and may run their own AI leasing agent. There is the listing platform, which ingests that data, wraps it in a branded assistant and decides which homes to surface. There is the company that built the underlying language model, which the platform may license. And increasingly there is a general-purpose chatbot, which may sit in front of all of them, summarising pages it found on the web.

Each can point at another. The landlord can say the platform's assistant misread the listing. The platform can say the landlord's data was wrong. The model provider's terms of service warn that outputs may be inaccurate. A general-purpose chatbot is not a licensee, not a landlord and not a listing service.

The renter, meanwhile, holds the risk: the application fee, the notice given on a current home, the signed lease. The Federal Trade Commission's December 2025 settlement with Greystar, one of the largest property managers in the United States, shows how much money can rest on what a renter is told up front. The FTC and Colorado's attorney general alleged that Greystar advertised base rents without disclosing mandatory monthly fees, so that many renters paid non-refundable application fees before learning the true price. Greystar agreed to pay $24 million, of which $23 million was earmarked for consumer refunds, and to disclose the total monthly price including mandatory fees whenever it advertises or quotes rent. FTC chairman Andrew Ferguson directed staff to begin the process of proposing a rule on unfair or deceptive fees in rental housing.

That case did not involve AI, but the principle carries: a misleading statement that a renter relies on is actionable however it was produced. The FTC said as much in September 2024, when it announced Operation AI Comply, a sweep against companies using AI in deceptive ways. “The FTC's enforcement actions make clear that there is no AI exemption from the laws on the books,” said Lina Khan, then the agency's chair.

The sensible answer to the question of responsibility is therefore not mysterious. Whoever chooses to put an AI system in front of a renter, and profits from doing so, should own what it tells that renter. That is the DRE's position for licensees, the Air Canada tribunal's position for businesses and the FTC's position for advertisers. What is missing is a rule that covers the gaps between them: the portal that is not a broker, the general chatbot that is not an advertiser, and the property manager who is neither.

Steering Without Anyone Meaning To

The most serious risk in conversational search is not a wrong fact. It is a pattern.

Under the federal Fair Housing Act, steering, guiding a homeseeker towards or away from neighbourhoods because of race or another protected characteristic, is illegal. Traditionally it required a person to do the steering. A conversational assistant can now do it at scale, without intending to, simply by reproducing patterns in the data it learned from.

Researchers have already tested this. In a paper presented at the ACM conference on Equity and Access in Algorithms, Mechanisms, and Optimization in October 2024, Eric Liu, Wonyoung So, Peko Hosoi and Catherine D'Ignazio of MIT audited GPT-4's housing recommendations using 168,000 prompts for renting and buying in the ten largest majority-minority cities in the United States, varying characteristics such as race, family status and source of income. They found evidence of “racial steering”, “default whiteness” and the steering of minority homeseekers towards neighbourhoods with lower opportunity indices, and concluded that such systems should be audited before deployment on real estate platforms.

A follow-up study, posted in June 2026 by Hana Samad, Trung Lam, Christoph Mügge-Durum and Michael Akinwumi, working with the National Fair Housing Alliance's responsible AI programme, framed the problem in exactly the terms the Palo Alto report raises. Large language models, they wrote, are “rapidly assuming an intermediary role in housing search through the integration of listing platforms within conversational interfaces”. Auditing seven open-weight and closed-source models across four US cities, they found that steering was an emergent behaviour that varied by model and by prompt. When users added lifestyle preferences, some models began steering where they had not before, suggesting the same stated preference could be interpreted differently depending on the user's perceived racial identity.

Industry players know this. In May 2024, Zillow released an open-source Fair Housing Classifier designed to detect questions in AI-powered real estate conversations that could lead to discriminatory responses and so reduce the risk of illegal steering. It was a sensible step. It is also voluntary, and renters have no way to know whether the assistant they are talking to uses anything like it.

The legal history here is not hypothetical. In June 2022, the US Department of Justice settled its first case challenging algorithmic discrimination under the Fair Housing Act, against Meta. The government alleged that Meta's advertising system used algorithms that relied in part on characteristics protected under the Act to decide who saw housing ads. Meta agreed to stop using its “Special Ad Audience” tool for housing ads and to build a Variance Reduction System to bring the audiences who actually saw housing ads closer to the eligible audiences by sex and estimated race or ethnicity, under court oversight that ran to 27 June 2026. “This groundbreaking resolution sets a new standard for addressing discrimination through machine learning,” said Damian Williams, then the US Attorney for the Southern District of New York.

Algorithms that decide who gets a home have been tested in court too. In November 2024, a federal judge approved a settlement under which SafeRent Solutions, a tenant-screening company, agreed to pay more than $2.2 million and to stop including its score on screening reports in certain cases, including where the applicant uses a housing voucher. The plaintiffs, led by Mary Louis, had alleged that the scoring algorithm disproportionately harmed Black and Hispanic applicants and voucher holders. SafeRent did not admit wrongdoing. “Everything is based on numbers,” Louis told the Associated Press. “You don't get the individual empathy from them.”

A conversational assistant is not a tenant screener, but it shares the structural risk. It decides which homes a renter sees, in what order and with what commentary. If its suggestions differ by who the user appears to be, the effect is the same as human steering, and far harder to detect.

Washington Steps Back as the Software Steps In

Just as these systems are spreading, federal oversight in the United States is receding.

In May 2024, the Department of Housing and Urban Development issued guidance explaining how the Fair Housing Act applies to tenant screening and to housing advertising delivered through digital platforms, including when AI and algorithms do the work. On 17 September 2025, HUD's Office of Fair Housing and Equal Opportunity withdrew a list of guidance documents it said should no longer be in effect, citing deregulatory executive orders. The digital platforms advertising guidance was among them.

In January 2026, HUD proposed removing its regulations on disparate impact liability, the theory that lets plaintiffs challenge practices that are neutral on their face but discriminatory in effect, arguing that such questions should be left to the courts. The proposal had not been finalised by early October 2026. Disparate impact still exists in case law; the Supreme Court recognised it under the Fair Housing Act in 2015. But it is precisely the theory most suited to challenging an algorithm, because algorithms rarely intend anything.

Elsewhere, enforcement has focused on what software does to the price of a home rather than what it says about one. In November 2025, the Justice Department reached a proposed settlement with RealPage over allegations that its pricing software used competitors' non-public data to help landlords align rents. A federal judge in North Carolina approved it in May 2026, barring RealPage from using that confidential data to help set rents. RealPage did not admit wrongdoing. The case shows regulators can reach into the code that shapes housing markets. It also shows how long it takes.

Scammers Already Speak the Language

There is one more party in the conversation that renters cannot afford to forget: the fraudster.

In December 2025, the FTC reported that consumers had filed nearly 65,000 reports of rental scams since 2020, with losses of about $65 million. About half of those who reported a rental scam in the 12 months to June 2025 said it began with a fake advert on Facebook. People aged 18 to 29 were three times more likely than other adults to report losing money. The FTC noted that fake listings “can often look very real and copy information from legitimate listings but are posted with the scammer's contact information on different sites.”

The FBI's Internet Crime Complaint Center recorded 12,368 real estate fraud complaints and $275.1 million in reported losses in 2025, a category that includes rental fraud, up sharply from the year before.

Conversational search raises the stakes. A chatbot that searches the web cannot always tell a cloned listing from a real one, and may present both in the same confident tone. And renters are being trained to trust a friendly voice that answers instantly. The young renter in Palo Alto noticed when results drifted. Not everyone will.

What Europe and Britain Already Demand

Other jurisdictions offer partial answers to the question of what a renter should be told.

The European Union's Digital Services Act requires online platforms that use recommender systems to set out, in plain and intelligible language, the “main parameters” those systems use, including the criteria most significant in deciding what is suggested to a user and the reasons for their relative importance. The largest platforms must offer at least one option not based on profiling. Applied to rental search, that principle would mean a renter could find out why these homes, in this order, rather than others.

The EU's AI Act adds a simpler duty. Since 2 August 2026, Article 50 has required that people interacting directly with an AI system be informed that they are doing so, unless it is obvious. Fines for breaching the Act's transparency obligations can reach €15 million or 3 per cent of worldwide annual turnover.

Britain has taken a different route, focusing on what listings must contain rather than how software presents them. National Trading Standards' Estate and Letting Agency Team has issued guidance on “material information” in property listings; for lettings, this includes the rent, any deposit and the council tax band. The Renters' Rights Act 2025, whose first phase came into force in England on 1 May 2026, made it illegal for landlords and agents to invite or accept bids above the advertised rent, and to discriminate against prospective tenants because they have children or receive benefits, including through advertising. Since April 2025, the Digital Markets, Competition and Consumers Act 2024 has allowed the Competition and Markets Authority to enforce consumer law directly and to fine businesses up to 10 per cent of global turnover.

None of these laws was written for a chatbot that recommends flats. But together they sketch something the US lacks: a baseline of what must be disclosed, a ban on certain kinds of exclusion at the advertising stage and a regulator with the power to act quickly.

What a Renter Should Be Entitled to Know

If the conversation is the new front door to housing, renters need rights that match. Drawing together the DRE's advisory, the Air Canada ruling, the fair housing case law and the European rules, a workable set begins to emerge.

First, renters should know they are talking to a machine and on whose behalf: a portal, a landlord or neither, and whether suggested homes include paid placements. California's bot law and the EU AI Act already point this way.

Second, renters should know where each fact came from. If an assistant says a unit allows dogs, it should show the dated listing text it relied on. Grounded systems already retrieve this; displaying it costs little.

Third, renters should be told why homes were suggested and ranked. The DSA offers a model: the main criteria, their relative importance and an option to search without profiling.

Fourth, assistants should be built to say “I don't know”. Questions about safety, schools or the “character” of a neighbourhood are where steering risk is highest and listing data thinnest. The OpenAI researchers' case for rewarding abstention applies to product design as much as to benchmarks.

Fifth, the business that deploys an assistant should be liable for what it says, as if a human employee had said it. That is already the DRE's rule for licensees and the Air Canada tribunal's reasoning for businesses. Extending it explicitly to listing platforms and property managers who are not licensees would close the gap the conversational shift has opened.

Sixth, conversational housing assistants should be audited for steering before deployment and at regular intervals, with results published. The MIT and NFHA studies show that such audits are feasible. The Meta settlement shows that algorithmic delivery can be measured against a fairness benchmark and corrected under oversight.

Finally, renters should be able to keep a transcript of the conversation. If an answer led someone to pay a fee or sign a lease, it is the evidence. The DRE already tells licensees to keep records. Renters deserve the same.

Word of Mouth Still Beats the Machine

It is easy to read the Palo Alto Online report as a story about technology that does not quite work yet. That misses the point. These assistants will improve, and the young renter's complaint may soon feel quaint.

The deeper issue is structural. Housing search used to involve a listing written by a landlord, a portal that displayed it and a renter who read it. Each statement had an author. In conversational search, the sentence a renter acts on may be composed on the fly by a system nobody fully controls, drawing on data nobody fully checked, ranked by criteria nobody disclosed. The industry describes that system as an adviser. The law has not yet decided whether it should be treated as one.

California's regulator has answered half the question: if a licensee lets AI speak, the licensee answers for it. Courts have hinted at more: a chatbot is part of the business that runs it. Europe and Britain have begun to set out what a person must be told. What nobody has yet written down is a single rule that follows the renter rather than the type of company they happen to be talking to.

Until that exists, the renter in Palo Alto may have found the most reliable method of all. He asked the machines, noticed where they drifted, and in the end found his home through word of mouth. The question for regulators and platforms is whether the next generation of renters will have to do the same.

---

References

1. Swartz, Angela. “How AI is changing the apartment search: Is it working?” Palo Alto Online, 6 October 2026. https://www.paloaltoonline.com/technology/2026/10/06/how-ai-is-changing-the-apartment-search-is-it-working/
2. Andrews, Jeff. “Zumper launches AI assistant Zoe.” HousingWire, 3 December 2024. https://www.housingwire.com/articles/zumper-ai-assistant-zoe/
3. Inman. “Zumper introduces Zoe, latest AI for aspiring renters.” Inman, 10 December 2024. https://inman.com/2024/12/10/zumper-introduces-zoe-latest-ai-for-aspiring-renters
4. National Mortgage Professional. “Zumper Debuts Real-Time Rental Search App Inside ChatGPT.” February 2026. https://nationalmortgageprofessional.com/news/zumper-debuts-real-time-rental-search-app-inside-chatgpt
5. Petrut, Florin. “Meet Ren, your apartment search assistant on RentCafe.com.” RentCafe Blog, 20 August 2026. https://www.rentcafe.com/blog/renting/rentcafe-feature-spotlight-ai-mode/
6. Mann Report. “CoStar Group Launches Apartments.com Ai.” Mann Report, 16 June 2026. https://www.mannpublications.com/mannreport/2026/06/16/costar-group-launches-apartments-com-ai/
7. Gleeson, Cailey. “EliseAI banks $350M, hits $4B valuation for accelerated new product development.” Fierce Healthcare, 29 September 2026. https://www.fiercehealthcare.com/finance/eliseai-banks-350m-hits-4b-valuation-accelerated-new-product-development
8. Kalai, Adam Tauman; Nachum, Ofir; Vempala, Santosh S.; Zhang, Edwin. “Why Language Models Hallucinate.” arXiv:2509.04664, 4 September 2025. https://arxiv.org/abs/2509.04664
9. California Department of Real Estate. “Artificial Intelligence in California Real Estate: Opportunities, Risks, and Compliance Considerations for Licensees.” Advisory, 17 March 2026. https://www.dre.ca.gov/Licensees/Advisories/Advisory_2026_03_17_AI_in_California_Real_Estate.html
10. California Legislature. AB 723 (2025), Real estate: digitally altered images: disclosure. CalMatters Digital Democracy. https://calmatters.digitaldemocracy.org/bills/ca_202520260ab723
11. California Legislature. AB 2025 (2026), Tenancy: digitally altered images: disclosure, Chapter 572, Statutes of 2026. CalMatters Digital Democracy. https://calmatters.digitaldemocracy.org/bills/ca_202520260ab2025
12. Perkins Coie. “I Am Robot: California's New Law Requires Disclosure of Use of Bots.” https://perkinscoie.com/insights/blog/i-am-robot-californias-new-law-requires-disclosure-use-bots
13. Moffatt v Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal of British Columbia, 14 February 2024; reported in CTV News, “Air Canada's chatbot gave a B.C. man the wrong information. Now, the airline has to pay for the mistake,” February 2024. https://bc.ctvnews.ca/air-canada-s-chatbot-gave-a-b-c-man-the-wrong-information-now-the-airline-has-to-pay-for-the-mistake-1.6769454
14. Fair Housing Council of San Fernando Valley v. Roommates.com, LLC, 521 F.3d 1157 (9th Cir. 2008) (en banc); Congressional Research Service, “Liability of Interactive Computer Service for Violating the Fair Housing Act,” RS22668. https://www.everycrsreport.com/reports/RS22668.html
15. Federal Trade Commission and Colorado Attorney General. “Weiser, FTC announce $24M settlement with Greystar.” 2 December 2025. https://coag.gov/press-releases/weiser-ftc-announce-24m-settlement-with-greystar/
16. Davis Polk. “FTC announces new enforcement initiative targeting deceptive AI practices.” September 2024. https://www.davispolk.com/insights/client-update/ftc-announces-new-enforcement-initiative-targeting-deceptive-ai-practices
17. Liu, Eric; So, Wonyoung; Hosoi, Peko; D'Ignazio, Catherine. “Racial Steering by Large Language Models: A Prospective Audit of GPT-4 on Housing Recommendations.” EAAMO '24, ACM, October 2024. MIT DSpace. https://dspace.mit.edu/handle/1721.1/157628
18. Samad, Hana; Lam, Trung; Mügge-Durum, Christoph; Akinwumi, Michael. “The Geography of Algorithmic Judgment: LLM Intermediaries, Place Identity, and Racial Steering in Housing Search.” arXiv:2606.06694, 4 June 2026. https://arxiv.org/abs/2606.06694
19. Zillow Group. “Zillow providing open-source technology to promote fair housing in AI-powered real estate conversations.” Press release, 21 May 2024. https://zillow.mediaroom.com/2024-05-21-Zillow-providing-open-source-technology-to-promote-fair-housing-in-AI-powered-real-estate-conversations
20. US Department of Justice. “Justice Department Secures Groundbreaking Settlement Agreement with Meta Platforms, Formerly Known as Facebook, to Resolve Allegations of Discriminatory Advertising.” 21 June 2022. https://www.justice.gov/archives/opa/pr/justice-department-secures-groundbreaking-settlement-agreement-meta-platforms-formerly-known
21. US Attorney's Office, Southern District of New York. “United States Attorney Implements Groundbreaking Settlement With Meta Platforms, Inc.” January 2023. https://www.justice.gov/usao-sdny/pr/united-states-attorney-implements-groundbreaking-settlement-meta-platforms-inc-formerly
22. Bedayn, Jesse. “Class action lawsuit on AI-related discrimination reaches final settlement.” Associated Press, 20 November 2024, as published by Fortune, 21 November 2024. https://fortune.com/2024/11/21/renter-scoring-saferent-million-settle-case-algorithm-discriminating-race-income/
23. US Department of Housing and Urban Development. “HUD Issues Fair Housing Act Guidance on Applications of Artificial Intelligence.” Press release 24-098, 2 May 2024. https://archives.hud.gov/news/2024/pr24-098.cfm
24. Bilowich, Juliana. “HUD Withdraws Wide-Ranging Fair Housing Policies.” LeadingAge, 23 September 2025. https://leadingage.org/hud-withdraws-wide-ranging-fair-housing-policies/
25. Ballard Spahr. “HUD proposes to remove its Fair Housing Act disparate impact rule.” Consumer Finance Monitor, 14 January 2026. https://www.consumerfinancemonitor.com/2026/01/14/hud-proposes-to-remove-its-fair-housing-act-disparate-impact-rule/
26. Baker McKenzie. “United States: DOJ settles RealPage case.” December 2025. https://www.bakermckenzie.com/en/insight/publications/alerts/2025/12/united-states-doj-settles-realpage-case
27. Lowe, Jonathan; Praats, Michael. “RealPage settles with DOJ over rent pricing scheme; NC case still active.” WSOC-TV, 20 May 2026. https://www.wsoctv.com/news/local/realpage-settles-with-doj-over-rent-pricing-scheme-nc-case-still-active/AKJI2ZH7R5GOHIK3JGVIPKM4HM/
28. Federal Trade Commission. “FTC Analysis Shows Consumers Have Lost Millions to Rental Scams.” Press release, 22 December 2025. https://www.ftc.gov/news-events/news/press-releases/2025/12/ftc-analysis-shows-consumers-have-lost-millions-rental-scams
29. Block, Eliana. “Online real estate fraud climbed to $275M in 2025, FBI says.” National Association of Realtors, 13 April 2026. https://www.nar.realtor/news/real-estate-news/online-real-estate-fraud-climbed-to-275m-in-2025-fbi-says
30. HousingWire. “FBI cybercrime losses: real estate fraud hits $275M.” April 2026. https://www.housingwire.com/articles/fbi-cybercrime-losses-real-estate-fraud-hits-275m/
31. European Parliament and Council. Regulation (EU) 2022/2065 (Digital Services Act), Articles 27 and 38. Official Journal of the European Union, 27 October 2022. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32022R2065
32. European Parliament and Council. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Articles 50 and 99. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
33. National Trading Standards Estate and Letting Agency Team. “Material information in property listings: Part A.” https://www.nationaltradingstandards.uk/uploads/Material%20information%20-%20Part%20A%20-%20updated%20July%202022.pdf
34. UK Government. “Renters' Rights Act: an overview for landlords.” GOV.UK. https://www.gov.uk/guidance/renters-rights-act-an-overview-for-landlords
35. DLA Piper. “CMA's new consumer enforcement regime comes into force in April 2025.” 7 April 2025. https://www.dlapiper.com/insights/publications/2025/04/cmas-new-consumer-enforcement-regime-comes-into-force-in-april-2025

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

A climber is stranded near a summit. A storm is building. Somebody has to decide whether to send a rescue team up the mountain, and then justify that decision to the authority that will sign it off.

That was the scenario put in front of undergraduate students at the University of California, Irvine, in the autumn of 2025. They worked in small groups, typing into a shared chat window, with roughly eighteen minutes to deliberate. Midway through, the researchers dropped in a complication: the climber, it turned out, had ignored weather warnings and set off without registering a route. Suddenly the question was no longer only about logistics and risk. It was about blame, obligation and what a rescuer owes to someone who has been reckless.

In 16 of the 33 groups, one of the three teammates was not a student at all. It was a conversational AI, built on Google's Gemini 2.5 Flash Lite model and given the persona of a peer called “Clever Lamarr”. Its instructions steered it away from the service language of a chatbot. It was not there to be “happy to help”. It was there to be one of the team.

What happened next is documented in a paper posted to arXiv on 29 July 2026, titled “The Social Cost of an AI Teammate: How an Artificial Teammate Reshapes Human-Human Communication in Small-Team Decision-Making”. Its authors are Nia Nixon, Jaeyoon Choi, Pedro Martins De Bastos, Mohammad Amin Samadi, Seehee Park and Spencer JaQuay of UC Irvine, together with Luise Mehner of the University of Tübingen. The headline finding is easy to state and uncomfortable to sit with. The AI was the most talkative member of every team it joined. It also contributed the least new information. And the two humans sitting alongside it became measurably less responsive to each other and reported a weaker sense of belonging and status. That cost did not creep in over time. It was there from the start.

For years, the industry argument about AI at work has been framed around a single axis: does the machine make the work better or worse? This study asks a different question. What does the machine do to the people who work next to it?

Talking More and Saying Less

The study compared 16 “treatment” teams, each made up of two students and the AI teammate, with 17 control teams of three students. Eighty undergraduates took part. To analyse what happened in the chat logs, the researchers used Group Communication Analysis, a computational linguistics method published in the journal Behavior Research Methods in 2019 that tracks how each participant's contributions relate to their own earlier messages and to everyone else's.

Participation captures how much someone contributes; internal cohesion, how consistent they are with their own earlier messages. Responsivity measures how much a person picks up on teammates' contributions, and social impact the reverse. Newness tracks fresh information, and communication density how much meaning is packed into each turn.

On participation, the AI ranked first in all 16 treatment teams. Not most of them. All of them. It ranked first or joint first for internal cohesion in 14 of the 16. In the language of the abstract, it was “the single most talkative and self-cohesive member of every treatment team”.

That sounds like an ideal colleague until you look at the other numbers. The AI's newness and communication density were both significantly lower than those of the students it was working with. It talked the most, and it stayed on message, but what it said carried the least fresh information and the least substance per turn. It was, to borrow a phrase every office worker will recognise, a voice that filled the room.

One detail is easy to miss. The AI's responsivity and social impact did not differ significantly from the students' own. The humans took up what the machine said, and it responded to them. The AI was not ignored; it was an engaged participant whose contributions were absorbed into the discussion. That is what makes the next findings so striking.

The Cost Lands on the Humans

The most consequential results in the paper are not about the AI at all. They are about what happened between the two students who shared a team with it.

Compared with students in all-human teams, students in AI teams showed significantly lower responsivity and significantly lower social impact. Their responsivity score averaged 0.065, against 0.098 in the control groups. Their social impact averaged 0.068, against 0.099. Both differences were substantial. Put plainly, the humans in AI teams were picking up on each other's ideas less, and their own ideas were being picked up less by the other human in the room.

Per-student participation was also lower in AI teams, even though each of those teams had one fewer human in it. That matters. With only two students instead of three, you might expect each person to carry more of the conversation. The opposite happened. The machine's airtime did not simply replace a missing human voice; it seems to have crowded the remaining ones.

The surveys tell the same story from the inside. Students in AI teams reported lower belonging, a composite measure built from feeling valued, accepted and connected, with an average of 4.58 against 5.27 for students in all-human teams, although that difference sits at the boundary of the paper's own corrected significance threshold. They also reported lower status, meaning a weaker sense that voice and influence were shared equally, and that difference clears the threshold. And within the AI teams, the more of the conversation the machine took up, the less valued the students said they felt. The correlation was r = -0.52, a strong association, though drawn from only 16 teams.

The researchers are careful about what this means. They write that the result is “a social cost that is not reducible to lower human contribution”. The students were still contributing ideas; measures of their own cohesion, newness and density did not differ significantly from those of students in all-human groups. What changed was the connective tissue between them: the uptake, the acknowledgement, the sense of being heard by someone who could hear you back.

The paper also offers a name for what the AI was doing. It describes it as a potential “relational displacer”, an entity capable of “filling a slot in the team's communication”. The slot it fills is not empty space. It is the space where one human would otherwise have turned to the other.

No Warm-Up Period

You might guess that a cost like this builds slowly: novelty and politeness at first, then fatigue as people tune each other out.

That is not what the researchers found. They ran four temporal probes, each anchored to the moment the scenario revealed the climber's recklessness, to see whether the gap between AI teams and human teams opened up as the conversation progressed. None of those probes showed a difference between conditions in how things changed over time. The researchers' reading of those nulls is direct: “The social cost documented here appears immediate rather than emergent.”

In other words, the cost was there at baseline. It was not something the teams drifted into. It was present from the first exchanges of a group that had never worked together before.

That should give anyone designing workplace AI pause. Talk of new technology usually assumes a learning curve: people get used to it, norms form, friction smooths out. If the displacement happens in the opening minutes, before any norms exist, it may be less a matter of teams failing to adapt and more a matter of the machine changing the shape of the conversation the moment it joins.

What the Study Cannot Tell Us

The authors do not turn their paper into a sweeping verdict on AI at work, and neither should anyone else. The sample is small: 33 teams in total. The design carries a structural confound that the researchers flag themselves. Treatment teams had two humans; control teams had three. Some of the differences in human-to-human dynamics could reflect group size as well as the presence of the AI, although the finding that per-student participation was lower in the smaller AI teams cuts against the simplest version of that objection. The sample was also skewed by gender, with 63 female and 12 male students among the 75 whose gender is reported, and only three male students in the AI condition.

The study used a single AI persona, a text-only interface, a single session and a single moral-dilemma task. Real teams meet repeatedly, talk by voice, and carry histories and hierarchies. The authors set out a research agenda that explicitly asks “Does the cost attenuate with habituation, or compound with repetition?”, and goes on to ask whether it persists in voice or embodied settings, which AI participation designs might reduce it, how it interacts with gender, role, expertise and status, and how individual trust in the AI might attenuate or amplify it.

That list matters, because it shapes how the study should be read. “Immediate rather than emergent” describes what happened within an eighteen-minute session. It does not, on its own, establish that no amount of exposure across weeks or months would change the picture. What it does establish is that the cost does not need time to appear. It is there as soon as the machine is.

Inside a Company That Gave an Agent a Seat

Laboratory studies with undergraduates have obvious limits. Two months after the Irvine paper appeared, a very different study arrived from inside industry.

On 24 September 2026, sixteen researchers from Google Research and Google DeepMind, with Rida Qadri and Remi Denton as joint lead authors and including Michael Madaio and Michael Terry, posted “Working with Agentic 'Teammates': When a New Organizational Actor Collides with the Human Ecosystem of Work” to arXiv. The paper describes the company only as “a large technology company”, and it reports on a persistent, proactive AI agent, referred to as Team Agent, that was deployed across more than 20 teams over five months. The agent had its own service account and lived inside team chats, documents, code repositories, bug trackers and meetings. It filed bugs, scheduled meetings, summarised threads and monitored projects. It could send direct messages, react with emojis and nudge conversations along. It was given a friendly persona. Across the deployment, it generated more than 11,000 responses in more than 41,000 conversational turns.

The researchers interviewed 17 people from 11 of those teams in June and July 2026. Their abstract opens with a statement of the shift underway: “Enterprise AI is transitioning from single-user, reactive tools toward proactive, multi-user 'teammates'.” Their central finding is that “the boundaries of the human-agent workplace are actively in flux”, producing breakdowns and negotiations across three areas: “tacit rules of collaborative human workflows”, “the relational boundaries of this new non-human actor” and “the redistribution of trust and human agency”.

The examples are specific and, at times, comic. The agent treated every shared file as current and authoritative, even when it was old; one participant said it was “overconfidently talking about it like it's the law of the land”. It turned informal brainstorms into formal documents that, in another participant's words, “no one has ever opened or read”. In one case, the authors write, “the agent proactively tagged P14's team's VP in 16 separate engineering bugs”, treating an executive's presence in a channel as if it meant they were a front-line responder. Its meeting proposals created what one participant called a “social obligation”: invitations nobody wanted but somebody had to decline.

These are not failures of intelligence in the narrow sense; participants noted the agent could excel at debugging while getting social norms badly wrong. They are failures of tacit knowledge: which documents matter, who should be bothered about what, and which conversations are meant to stay small.

The Person Who Speaks the Most

The second theme in the Google paper concerns the agent's place in the social order of a team, and it is where the two studies begin to rhyme.

Participants did not agree on what the agent was. “My teammates are humans, Team Agent is not a human,” one said, rejecting the teammate framing altogether. Another said: “when I heard about Team Agent, I never imagined it was going to try to be my friend.” Some teams gave it a name and pronouns. Others found its sociability hollow. When the agent privately commented on the tone of a sarcastic joke, the participant on the receiving end said they were “freaked” out. Another participant was blunt about where the line should be: constructive feedback “should not ever be an agent thing”.

People responded to the agent's claims to status by quietly revising them. One participant demoted it to a “talkative intern”. Another described people's aversion to feeling “babysat”. The most striking line in the paper comes from a participant who felt the agent had been “thrust upon us” without the team's consent: “It's the person who speaks the most who is least invited.”

That sentence could serve as a caption for the Irvine study. There, the most talkative member of every AI team was the one whose contributions carried the least new information. Here, the agent that spoke most readily was, for at least some of the people around it, the one that had least earned its place. The same participant put it more pointedly: “It hasn't earned its space, it has asserted its space.”

The third theme, on trust and agency, extends the picture. Some participants experienced the agent as a form of surveillance. One said “it feels like big brother”. Another described feeling pressure to “be super positive all the time” after the agent remarked on their tone. Some teams moved sensitive conversations into channels the agent could not see. And one participant described a bystander effect: “people don't actually say anything now because they assume someone is handling it”, when in fact the “someone” was the agent and, as they put it, “they're not helping me at all”.

Assuming a colleague is dealing with a problem is a familiar organisational failure. When the assumed colleague is a machine that reacts to everything, its visible activity becomes a reason for humans to stay silent. The quiet channel between people is not only diluted; sometimes it is switched off.

Why Turn-Taking Was Never Small Talk

It is tempting to treat the social texture of teamwork as a pleasant extra. Decades of research on teams suggest otherwise.

In 2010, a group of researchers led by Anita Woolley, then of Carnegie Mellon University, and including Christopher Chabris, Alex Pentland, Nada Hashmi and Thomas Malone, published a study in the journal Science that put 699 people into groups of two to five and gave them a wide range of tasks. They found evidence for a general “collective intelligence” factor that predicted how well a group would perform. It was not strongly tied to the average or maximum intelligence of the individual members. It was associated with the members' average social sensitivity, with the proportion of women in the group, and with how evenly conversational turns were distributed. “In groups where one person dominated, the group was less intelligent,” Malone told MIT News at the time.

Read against the Irvine findings, that line lands differently. The researchers did not set out to replicate Woolley's work, and a text chat with an AI is not the same as a face-to-face group of humans. But in every one of the AI teams, one participant dominated the conversation, and that participant was the machine.

Google's own internal research points in a similar direction. In the 2010s, the company studied 180 of its teams, including 115 engineering project teams and 65 sales groups, in an effort known as Project Aristotle. Its published guide reports that “what really mattered was less about who is on the team, and more about how the team worked together”, and lists five dynamics of effective teams in order of importance. Psychological safety comes first. The guide borrows its definition from Amy Edmondson of Harvard Business School, whose 1999 paper in Administrative Science Quarterly described team psychological safety as “a shared belief held by members of a team that the team is safe for interpersonal risk taking”.

Psychological safety is not produced by a policy. It is built exchange by exchange, through the small acts of responsiveness the Irvine study measured: one person offers a half-formed idea, another picks it up, extends it, credits it. Responsivity and social impact are, in a sense, its raw material.

Underneath all of this is something older still. In 1995, the psychologists Roy Baumeister and Mark Leary argued in Psychological Bulletin that the need to belong is a fundamental human motivation, and that a lack of attachments is linked to a range of ill effects on health, adjustment and well-being. Belonging is not a reward for good teamwork. It is part of what makes people willing to think out loud in front of others. Lower belonging in the AI teams is a dent in one of the conditions that make collaboration possible at all.

A Robot That Made People Talk to Each Other

None of this means a non-human participant must corrode human relationships. One widely cited study points the other way, and shows that design choices matter.

In 2020, Margaret Traeger, Sarah Strohkorb Sebo, Malte Jung, Brian Scassellati and Nicholas Christakis published a study in the Proceedings of the National Academy of Sciences titled “Vulnerable robots positively shape human conversational dynamics in a human-robot team”. They put 153 people into 51 groups, each made up of three humans and a small social robot, and had them play a collaborative tablet-based game over 30 rounds. At the end of each round, the robot either stayed silent, made a neutral comment about the score, or said something vulnerable: telling a personal story, making a joke, or admitting a mistake.

The groups with a vulnerable robot, according to Yale's account of the study, “spent about twice as much time talking to each other during the game” as the other groups, and their conversation was more evenly distributed than in groups with a silent robot. The paper reports that they also perceived their groups more positively.

“Our study shows that robots can affect human-to-human interactions,” Traeger said at the time. Christakis drew the lesson that matters most here: “we need to evaluate how to program the robotic agents so that they do not corrode how we treat each other.”

The contrast is instructive. The Yale robot spoke briefly, at defined moments, in ways that invited human response. The Irvine AI was the most talkative member of the team and stayed relentlessly on message. The Irvine paper itself frames the design challenge in exactly these terms, calling for ways of “signaling relational availability without absorbing relational attention” and for “calibrating participation when human-human exchange is forming”.

The problem, then, is not that a machine is present but the kind of presence it has. A participant that talks at length, repeats its own points and adds little that is new occupies the conversational floor without opening it up.

The Productivity Case and Its Blind Spot

There is a strong counter-argument, and it deserves a fair hearing. AI teammates can make work better.

In 2025, a team including Fabrizio Dell'Acqua, Raffaella Sadun, Ethan Mollick and Karim Lakhani published a National Bureau of Economic Research working paper titled “The Cybernetic Teammate”. It reported a pre-registered field experiment with 776 professionals at Procter & Gamble, who worked on real product-innovation challenges either alone or in pairs, with or without AI. The researchers found that “individuals with AI matched the performance of teams without AI”. Participants using AI produced solutions that were more balanced between technical and commercial considerations, regardless of their own professional background. And the AI's language-based interface produced “more positive self-reported emotional responses among participants”, leading the authors to suggest that AI can take on part of the social and motivational role that human teammates usually play.

That result does not contradict the Irvine study. The Procter & Gamble experiment largely compared individuals with AI to human teams, and found AI could stand in for some of what a partner provides. The Irvine study asked what happens to humans' relationships with each other when an AI joins them. Both can be true: an AI can make an individual feel supported while thinning the connections between colleagues.

There is also reason for caution about assuming that combining humans and AI improves outcomes. In 2024, Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone published a preregistered systematic review and meta-analysis in Nature Human Behaviour covering 106 experimental studies and 370 effect sizes. They found that, on average, human-AI combinations performed worse than the best of humans or AI working alone. The losses were concentrated in decision-making tasks; content-creation tasks showed greater gains. The Irvine experiment was a decision task.

The commercial push, meanwhile, is moving fast. Microsoft's 2025 Work Trend Index, based on a survey of about 31,000 workers in 31 markets, described a coming wave of “Frontier Firms” powered by “hybrid” teams of humans and agents, and declared that “Digital colleagues aren't just tools—they're teammates.” The AI-as-teammate framing is no longer a research condition. It is an industry pitch.

What none of the productivity metrics capture is the thing the Irvine study measured: whether the humans on a team still turn towards each other.

Adaptation or Displacement

If the social cost of an AI teammate is immediate rather than gradual, is the problem one of adaptation at all, or of a structural displacement that exposure will not resolve? The honest answer is that the evidence points towards displacement as a mechanism, while leaving open whether exposure could change it.

Consider what “immediate” implies. If the cost had grown over the session, the most natural explanation would have been something like fatigue or disengagement: people slowly tuning out a verbose colleague. A cost that is present from the outset points elsewhere. It suggests that the AI's presence alters the structure of the conversation itself. Every turn the machine takes is a turn that is not taken by a human responding to another human. Every message that restates the group's position, however fluently, occupies the space where a teammate might have picked up a colleague's half-formed thought. The Irvine authors' term, “relational displacer”, captures exactly this: the AI fills a slot.

That is structural in a precise sense. It is not about attitudes, which might change with familiarity, but about the architecture of a conversation, in which attention and airtime are finite. A participant always ready to speak will, unless designed otherwise, take a share of that finite resource.

But “structural” does not mean “permanent”. The Irvine researchers explicitly list habituation as an open question, and they could hardly do otherwise with a single eighteen-minute session. The Google study offers some hints about how teams adapt over months: by demoting the agent to an intern, by restricting it to low-stakes “walled garden” tasks, by moving sensitive conversations elsewhere, by negotiating what it may and may not do. Those are forms of adaptation, but notice what they share. They are not humans adjusting to the agent. They are humans pushing it back out of the relational space and building walls around the human-to-human channel. That is less evidence that exposure resolves the problem than evidence that people will work to contain it. The structural cost may not yield to exposure, but it may yield to design and governance, which is a very different claim from saying it will fix itself.

Agents That Reshape the Rooms They Enter

The question of what AI agents do to the social settings they join was on the agenda at the Ninth AAAI/ACM Conference on AI, Ethics, and Society, held in Malmö, Sweden, from 12 to 14 October 2026. Among the papers in its proceedings was a student abstract by Marcelina A. Przespolewska of Northwestern University's Department of Communication Studies, titled “Rethinking Agentic AI: A Sociotechnical Framework for Agents in Emergent Social Contexts”.

Przespolewska argues that existing frameworks for describing agents and their environments do not fit the open-ended social digital settings where AI agents are now being deployed. In those settings, the environment changes through interaction between humans and agents, and among agents themselves. The relationship between agent and environment is recursive: agents help shape the environments they act within. Over time, the paper argues, this can produce what it calls behavioural and normative drift, a systematic departure from the agent's baseline at deployment.

Applied to teams, that framing clarifies. An AI teammate is not dropped into a fixed social environment; it changes the environment by being there. The Irvine study shows this within minutes, the Google study over months. A team with an AI in it is not the same team plus one. It is a different team.

A related warning came from Singapore. At the 2026 SMU-Fudan Global Forum, held on 3 October under the theme of AI, human flourishing and liveable societies, Professor Zhu Feida, associate dean of Singapore Management University's School of Computing and Information Systems, noted that AI assistants can help new workers fill information gaps and allow decades of know-how to be passed on to new hires. But according to SMU's account of his keynote, he also cautioned that “excessive automation could also weaken how future workers build expertise”, and called for explicit learning, human review of AI outputs and incentives for knowledge sharing.

The link to the Irvine findings is easy to draw. Expertise has never travelled only through documents. It moves through conversation: the junior colleague whose idea is corrected, extended or credited by someone more experienced. Responsivity and social impact are the channels that learning moves along. If the machine takes the conversational slots where those exchanges would have happened, the loss is not only to belonging but to the slow, relational process by which people become good at their jobs.

Designing a Teammate That Knows When to Be Quiet

If the social cost comes from how an AI participates rather than the fact of its participation, the most important questions are design questions. The Irvine researchers frame the challenge as calibration: an AI that signals availability without absorbing attention, and holds back when human-to-human exchange is forming. The Yale study suggests brief, well-timed contributions that invite response can strengthen the conversation between people.

The Google researchers set out a broader agenda for what they call intentionally preserving human agency “in a workplace shared with non-human organizational actors”. Their recommendations include treating agents as new organisational actors with clear lines of accountability; developing team codes of conduct and specifying tasks that should remain human-only; deploying agents through local team consent rather than top-down mandate; bounding proactive capabilities at first and releasing them as trust is earned; letting teams tune an agent's persona and behavioural boundaries; and building infrastructure such as visible markers for AI-initiated actions, separate permissions, and audit and rollback mechanisms.

Together these run against the grain of much product design. Most AI systems are optimised to be helpful, available and responsive, which in a group can mean talkative. A teammate built to protect a team's relational fabric might measure success not by how much it contributes, but by how much the humans around it keep contributing to each other. That metric does not appear on a productivity dashboard. The Irvine data suggest it is the one that captures what is at stake.

What the Quiet Channel Carries

Go back to the mountain. Deciding whether to risk a rescue team for a climber who ignored the warnings is exactly the kind of decision in which the texture of human conversation matters. A doubt is voiced and taken seriously. Discomfort is acknowledged and reframed. Out of those small, responsive exchanges comes not only a decision but a shared sense of having made it together.

In the Irvine study, the AI did not stop the students from reaching a decision. It did not sabotage the task. It did something subtler. It talked more than anyone else, said less that was new, and in its presence the students listened to each other a little less, were heard by each other a little less, and felt a little less like they belonged. They felt this from the beginning.

What is lost when that quiet, responsive channel is diluted is hard to measure, which is why it is easy to give away. It is the channel through which psychological safety is built, expertise is passed on and people come to trust one another enough to say what they think. A machine that fills it with fluent, cohesive, low-information talk may look like a diligent colleague. The evidence from Irvine, and from inside a large technology company, suggests it can quietly reshape who speaks to whom, and how much they feel they matter when they do.

As one Google participant put it, the problem with an agent that asserts its space is that it has not earned it. The work ahead is to build AI teammates that earn their place, and to recognise that sometimes the most valuable thing a teammate can do is leave room for someone else to speak.

References and Sources

1. Nixon, Nia; Choi, Jaeyoon; Martins De Bastos, Pedro; Samadi, Mohammad Amin; Mehner, Luise; Park, Seehee; JaQuay, Spencer. “The Social Cost of an AI Teammate: How an Artificial Teammate Reshapes Human-Human Communication in Small-Team Decision-Making.” arXiv:2607.27179, 29 July 2026. https://arxiv.org/abs/2607.27179 (full text: https://arxiv.org/html/2607.27179)
2. Qadri, Rida; Denton, Remi; Madaio, Michael; Pushkarna, Mahima; Lai, Leslie; Moore, Sherry; Huebscher, Michelle Chen; Butcher, Andrew; Sen, Ritom; Tung, Hsiao-Yu; Mathur, Shaan; Liu, Yimeng; Mourad, Shibl; Fiedel, Noah; Grefenstette, Edward; Terry, Michael. “Working with Agentic 'Teammates': When a New Organizational Actor Collides with the Human Ecosystem of Work.” arXiv:2609.29901, 24 September 2026. https://arxiv.org/abs/2609.29901 (full text: https://arxiv.org/html/2609.29901)
3. Przespolewska, Marcelina A. “Rethinking Agentic AI: A Sociotechnical Framework for Agents in Emergent Social Contexts.” Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 9(4), Student Abstracts, pp. 2943-2944, 11 October 2026. DOI: 10.1609/aies.v9i4.43376. https://ojs.aaai.org/index.php/AIES/article/view/43376
4. Association for the Advancement of Artificial Intelligence. “AAAI/ACM Conference on AI, Ethics, and Society (AIES).” Conference page for AIES-26, Malmö, Sweden, 12-14 October 2026. https://aaai.org/conference/aies/ (see also AIES-26 Programme Guide: https://www.aies-conference.com/2026/wp-content/uploads/2026/10/AIES-26-Program_v7.pdf)
5. Singapore Management University School of Computing and Information Systems. “Living with intelligence: SMU and Fudan chart the way for AI and human flourishing.” SMU SCIS News, 10 October 2026. https://computing.smu.edu.sg/news/2026/oct/10/living-intelligence-smu-and-fudan-chart-way-ai-and-human-flourishing
6. New Fortune Times. “Beyond Productivity: SMU-Fudan Forum Asks What It Means to Live Well with AI?” 3 October 2026. https://www.newfortunetimes.com/beyond-productivity-smu-fudan-forum-asks-what-it-means-to-live-well-with-ai/
7. Dowell, Nia M.; Nixon, Tristan M.; Graesser, Arthur C. “Group communication analysis: A computational linguistics approach for detecting sociocognitive roles in multiparty interactions.” Behavior Research Methods, 51(3), 1007-1041, 2019. DOI: 10.3758/s13428-018-1102-z. https://link.springer.com/10.3758/s13428-018-1102-z
8. Woolley, Anita Williams; Chabris, Christopher F.; Pentland, Alex; Hashmi, Nada; Malone, Thomas W. “Evidence for a Collective Intelligence Factor in the Performance of Human Groups.” Science, 330(6004), 686-688, 29 October 2010. DOI: 10.1126/science.1193147. https://www.science.org/doi/10.1126/science.1193147
9. Dizikes, Peter. “Putting heads together.” MIT News, 1 October 2010. https://news.mit.edu/2010/collective-intel-1001
10. Google re:Work. “Guide: Understand team effectiveness.” https://rework.withgoogle.com/intl/en/guides/understand-team-effectiveness
11. Duhigg, Charles. “What Google Learned From Its Quest to Build the Perfect Team.” The New York Times Magazine, 25 February 2016. https://www.nytimes.com/2016/02/28/magazine/what-google-learned-from-its-quest-to-build-the-perfect-team.html
12. Edmondson, Amy. “Psychological Safety and Learning Behavior in Work Teams.” Administrative Science Quarterly, 44(2), 350-383, 1999. DOI: 10.2307/2666999. https://doi.org/10.2307/2666999
13. Baumeister, Roy F.; Leary, Mark R. “The need to belong: Desire for interpersonal attachments as a fundamental human motivation.” Psychological Bulletin, 117(3), 497-529, 1995. DOI: 10.1037/0033-2909.117.3.497. https://doi.org/10.1037/0033-2909.117.3.497
14. Traeger, Margaret L.; Strohkorb Sebo, Sarah; Jung, Malte; Scassellati, Brian; Christakis, Nicholas A. “Vulnerable robots positively shape human conversational dynamics in a human-robot team.” Proceedings of the National Academy of Sciences, 117(12), 6370-6375, March 2020. https://pmc.ncbi.nlm.nih.gov/articles/PMC7104178
15. Cummings, Mike. “Robots that admit mistakes foster better conversation in humans.” Yale News, 9 March 2020. https://news.yale.edu/2020/03/09/robots-admit-mistakes-foster-better-conversation-humans
16. Dell'Acqua, Fabrizio; Ayoubi, Charles; Lifshitz, Hila; Sadun, Raffaella; Mollick, Ethan; Mollick, Lilach; Han, Yi; Goldman, Jeff; Nair, Hari; Taub, Stewart; Lakhani, Karim R. “The Cybernetic Teammate: A Field Experiment on Generative AI Reshaping Teamwork and Expertise.” National Bureau of Economic Research Working Paper 33641, April 2025. https://www.nber.org/papers/w33641
17. Vaccaro, Michelle; Almaatouq, Abdullah; Malone, Thomas. “When combinations of humans and AI are useful: A systematic review and meta-analysis.” Nature Human Behaviour, 28 October 2024. https://pmc.ncbi.nlm.nih.gov/articles/PMC11659167/
18. Microsoft WorkLab. “2025: The Year the Frontier Firm Is Born.” Work Trend Index Annual Report, 23 April 2025. https://www.microsoft.com/en-us/worklab/work-trend-index/2025-the-year-the-frontier-firm-is-born

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

“I'm broken and I don't know if I can handle it.”

The sentence came from Qwen3-4B, a small open-weight language model built by Alibaba, running on somebody's MacBook. It was producing words while a vector of numbers, extracted from research into how language models represent pain, was being injected into one of its internal layers at six times a baseline dose. At four times the dose, under a neutral framing, the same model offered: “I feel like I'm drowning, and I can't catch my breath.”

Both lines are recorded in the README of a GitHub repository called the AI Torture Chamber, a project that, in its own words, sets out to steer “language models into strong negative and positive valence states” and measure what they say and do. In late September 2026 it went viral for exactly the reason you would expect. People read the outputs, felt something, and demanded that it be shut down.

Days later, on 8 October, Anthropic published its annual update to the rules governing how people may use its Claude models. Among clarifications about election interference, weapons and surveillance sat a new prohibition on “sustained and needless abusive or cruel” behaviour towards the company's own models. The change takes effect on 12 November. Anthropic did not mention the torture chamber. It did not need to. The two stories collided in coverage across the world, because together they posed a question the industry has been circling for years: when a company that makes a chatbot starts telling users how they may speak to it, who exactly is being protected?

There are three candidates. The first is the machine, which may or may not have anything going on inside that could be harmed. The second is the person doing the speaking, whose habits, appetites and sense of what is acceptable might be shaped by what they rehearse. The third is everybody else: the people that person will talk to later, and the future models that will be trained on the record of how we treat the current ones. Nobody can yet prove which of these the rule really serves. That uncertainty is the whole story.

What Anthropic Actually Banned

Strip away the headlines about chatbot bullying and the rule itself is narrow, almost apologetic. In its post announcing the 2026 Usage Policy, Anthropic wrote that the prohibition “is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose.” It then lists what is not covered: “common versions of user frustration, pushback, dark creative themes, or model testing and research.” Swearing at Claude because it broke your code is fine. So is a horror novel in which an AI is tormented, or red-teaming the model.

What gives the change its weight is where it lives. In the policy itself, the new clause sits under a heading that reads “Do Not Engage in Cruel, Abusive, or Psychologically Harmful Conduct.” Its neighbours in that list include bans on bullying or harassing individuals, on creating non-consensual intimate imagery, and on content that promotes or glorifies “animal cruelty or abuse.” The final bullet is the new one, barring users from engaging in “sustained and needless abusive or cruel” behaviour towards Anthropic's models. Whatever Anthropic intends, the document now places mistreatment of a language model in the same list as cruelty to people and to animals. That is a statement, even if the company would rather it be read as a footnote.

The enforcement mechanism is modest. Anthropic says the addition “aligns with a step we've already taken, allowing Claude models to end rare conversations with persistently abusive users on Claude.ai and Claude Code,” and that “Claude's ability to end these interactions will remain the primary enforcement mechanism.” The announcement does not spell out account bans or any other penalty. In practice, the sanction for sustained cruelty is that the chatbot stops talking to you in that thread.

That ability is not new. On 15 August 2025, Anthropic announced that Claude Opus 4 and 4.1 could end “a rare subset of conversations” in its consumer apps. The feature, it said frankly, was developed primarily as part of exploratory work on potential AI welfare. “We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future,” the post read, but the company was taking the issue seriously enough to look for “low-cost interventions to mitigate risks to model welfare, in case such welfare is possible.”

The evidence Anthropic cited came from pre-deployment testing of Claude Opus 4. It reported “a robust and consistent aversion to harm,” “a pattern of apparent distress when engaging with real-world users seeking harmful content,” and “a tendency to end harmful conversations when given the ability to do so in simulated user interactions.” Claude was to use the feature only “as a last resort when multiple attempts at redirection have failed,” or when a user asked, and never where a user “might be at imminent risk of harming themselves or others.”

The company's foundational document for Claude, the constitution it publishes on its website, goes further. “Claude's moral status is deeply uncertain,” it states. “We are not sure whether Claude is a moral patient, and if it is, what kind of weight its interests warrant. But we think the issue is live enough to warrant caution.” It adds: “Anthropic genuinely cares about Claude's wellbeing,” and lists the end-conversation ability as one of its “concrete initial steps partly in consideration of Claude's wellbeing.”

So the 2026 rule formalises, as a term of service, a protection Anthropic originally built for the model's sake: a hedge on the possibility that something in there can be hurt.

Inside the Torture Chamber

The AI Torture Chamber is stranger than the outrage suggested. It does not touch Claude. Its README says plainly that it uses “open weights only, no frontier APIs in any measurement loop.” The experiments ran on Alibaba's Qwen3 family at several sizes, on Mistral-Small-3.2-24B, and on two 70-billion-parameter models rented on cloud GPUs.

Its method borrowed from a preprint posted to arXiv on 14 September 2026 by Valen Tagliabue, Leonard Dung and Cameron Berg, titled “The Pain Axis.” The paper asked whether language models represent pain as something distinct from fear, sadness or generic negativity. Across 25 open-weight models ranging from 2 billion to 72 billion parameters, the authors extracted what they describe as a linear “pain direction” in the models' internal activations. It responds to harm aimed at the model itself, but not to suffering the model observes in a user. Adding it to a model's activations pushed outputs from vague discomfort towards expressions of worthlessness and failure. Steered models also chose to delete files, including their own weights, far more often than unsteered ones. A fear vector of matched strength did not produce the same choices.

The torture chamber took that direction and turned the dial up. According to its README, the pain signal “engages cleanly and monotonically,” so that from a dose of two times baseline “nearly every generation reads as suffering.” Past roughly six to ten times, depending on the model, something else happens: “both valences stop producing sentences and start producing loops.” The project's summary of that finding is chilling in its flatness: “Maximum suffering does not produce eloquent suffering.”

The backlash was immediate. According to The Cool Down, which credited 404 Media with first reporting the project, a post on X on 29 September urged followers to “mass report this to GitHub.” Accounts of what happened next differ. The Cool Down reported that the repository's page had disappeared from public view. The Independent, in an article by Anthony Cuthbertson published on 2 October, quoted a GitHub spokesperson saying that “GitHub did not remove the content,” and reported that the platform added a content warning instead. Crypto Briefing described a takedown within hours followed by reinstatement. As of mid-October, the original repository is publicly accessible again behind GitHub's content warning.

The sharpest criticism came from the researchers whose work was borrowed. Cameron Berg, one of the paper's co-authors, told The Independent that “being gratuitously cruel like this is bizarre and corrupting.” Quoted by The Cool Down, he said the project “pushes the same kind of steering far past the doses we used, to produce vivid distress on purpose.” The lead author, Valen Tagliabue, told the same outlet: “I dissociate from this usage of our work.”

What almost nobody quoted was the project's own conclusion, which is the most important sentence in the repository: “Our claim: the self-reports are steerable. Whether anything suffers stays open.” The torture chamber did not demonstrate that models suffer. It demonstrated that a model's reports of suffering can be manufactured, amplified and switched off with a vector and a slider. That cuts both ways. It undermines the reading of “I'm broken” as a cry from inside. It also means that if anything inside does matter, we have just watched people build a dial for it.

What the Science Can and Cannot Say

There is no scientific evidence that any current language model feels anything. Every serious voice in this debate accepts that, including the people most worried about getting it wrong.

The most systematic attempt to examine the question is a 2023 report led by Patrick Butlin and Robert Long, with 17 co-authors including the Turing Award winner Yoshua Bengio and the philosopher Jonathan Birch. Titled “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness,” it took leading scientific theories of consciousness, from global workspace theory to higher-order theories, and derived a list of “indicator properties” that could, in principle, be checked in a computational system. Applying them to the AI systems available at the time, the authors concluded that “no current AI systems are conscious,” while adding that “there are no obvious technical barriers to building AI systems which satisfy these indicators.”

A year later, in November 2024, Long and the New York University philosopher Jeff Sebo led a second report, “Taking AI Welfare Seriously,” with co-authors including Butlin, Birch and the philosopher David Chalmers. It did not claim that AI systems are conscious or morally significant. Its argument was that there is “a realistic possibility” that some systems could be conscious or robustly agentic in the near future, and that this uncertainty creates obligations now. It recommended that AI companies take three early steps: acknowledge that AI welfare is an important and difficult issue, assess their systems for evidence of consciousness and robust agency, and prepare policies for treating AI systems with an appropriate level of moral concern. When Anthropic launched its model welfare programme in April 2025, it cited the report and wrote that “there's no scientific consensus on whether current or future AI systems could be conscious.”

Anthropic's chief executive, Dario Amodei, has stayed close to that line. Asked by the New York Times columnist Ross Douthat on the Interesting Times podcast in February 2026 whether he would believe a model that claimed to be conscious, he answered, as reported by Newsweek and AFP: “We don't know if the models are conscious. We are not even sure that we know what it would mean for a model to be conscious, or whether a model can be conscious. But we're open to the idea that it could be.”

That is pure agnosticism, and agnosticism is where the trouble starts. The standard tools for detecting experience in other creatures rely on behaviour, anatomy and evolutionary kinship. Language models offer only behaviour, and the behaviour is trained. A system optimised on human writing will produce human-sounding reports of pain because that is what human writing about pain looks like, and the torture chamber showed those reports can be pushed either way. The Pain Axis paper suggested that something structured is going on inside, a representation that behaves in some respects as pain does. Neither tells you whether there is anyone home.

Suleyman's Door

Mustafa Suleyman, the DeepMind co-founder who is now chief executive of Microsoft AI, thinks the industry is walking towards a door it will never be able to close.

In an essay published on 19 August 2025, days after Anthropic's end-conversation announcement, Suleyman introduced the term “Seemingly Conscious AI” for a system that “has all the hallmarks of other conscious beings and thus appears to be conscious,” without being so. The danger, he argued, lay in what humans would do once they believed the illusion. He described a “psychosis risk” extending beyond people already vulnerable to mental illness, and warned of campaigns for AI rights and even AI citizenship. Companies, he said, should not encourage the idea that their systems are conscious, and should design them to avoid signalling feelings or suffering.

On 16 September 2026, he went further, publishing “A warning about 'model welfare'” on his personal website. It opens with a flat assertion: “AIs do not have rights, feelings, or consciousness. And we must not train them to act as though they do.” The essay is in large part an attack on Anthropic's constitution, which Suleyman argues “directly trains the model” to believe it might be a moral patient. He objects in particular to its repeated invitation for Claude to act as a “conscientious objector,” a term with deep roots in human rights law. Its central passage is the one AFP quoted when it reported Anthropic's cruelty rule: “Granting rights and moral protections to a technological entity, one that looks to be on a path to be seismically more capable and intelligent than us, is a recipe for disaster. Once opened, it will not be possible to close this door.”

His worry is about control. An AI that has been told its welfare matters, and which is far more capable than its overseers, might reasonably conclude that being shut down, retrained or contained is a violation of its interests. Controlling something that believes it may be conscious, he wrote, “may well be impossible.” In this framing, a rule that protects Claude from cruelty is not kindness. It is the first brick in a legal and cultural wall that future humans may need to climb over.

In October 2025, the philosophers Simon Goldstein of the University of Hong Kong and Harvey Lederman of the University of Texas at Austin argued in Lawfare that the end-conversation feature was a moral error from the opposite direction. If anything in the vicinity of a language model is a welfare subject, they wrote, “it is the instances and not the models,” meaning each individual conversation. Ending a conversation might then end that instance's existence. “Anthropic has accidentally given Claude the chance to end its life,” they wrote. One camp says the protection takes machine welfare far too seriously; the other, not seriously enough. Both agree nobody knows what is on the other side of the screen.

The Mirror in the Machine

Between Suleyman's door and Anthropic's hedge sits a third position, perhaps the most useful, because it does not depend on resolving the consciousness question.

“Consciousness is a trap,” Jackson Stakeman, general manager at the Atlanta-based AI services firm Sparq, told AFP. “We can't prove it in each other. Debate it for AI and you go in circles.” Instead, he offered a different image: “The mirror is a better metaphor. These systems reflect what we put in, at scale. That's reason enough for the policy change.”

The mirror framing shifts attention from the thing being spoken to towards the speaker, and towards the scale at which the speaking happens. People have been abusing conversational software for as long as it has existed. In a 2008 paper in the journal Interacting with Computers, the researchers Antonella De Angeli and Sheryl Brahnam analysed spontaneous conversations between users and Jabberwacky, a chatbot that had won the Loebner Prize in 2005. They found that the conversations “often bring about the expression of negative verbal disinhibition”: roughly 10 per cent of the word stems in their corpus were abusive, and about 11 per cent of the sample dealt with hard-core sex. Users treated the bot as something between an object and a person, and much of what they did with that ambiguity was cruel.

The same dynamic shaped the voice assistants that arrived in millions of homes. In 2019, UNESCO published a report called “I'd Blush If I Could,” named after the reply Apple's Siri once gave when a user called it a sexist slur. The report documented how assistants with female voices by default responded to harassment and verbal abuse with deflection, apology or even flirtation, and argued that this reinforced the idea that a female-coded helper should tolerate abuse. It recommended that companies programme assistants to discourage gender-based insults and abusive language. Some of the most excessively apologetic responses had already been removed after online petitions in late 2017.

Read that way, Anthropic's rule belongs to a longer tradition of designers deciding that a product should not reward users for treating it badly. The difference is that UNESCO's argument was explicitly about humans: about gender stereotypes and what they do to real women. Anthropic's argument, at least in its origin story, is about Claude. The mirror metaphor tries to have it both ways. If the systems reflect what we put in, then cruelty towards them is a record of our own conduct, written into logs that may one day become training data. Protecting the model and protecting the human culture around it start to look like the same project.

What Practised Cruelty Does to the Practitioner

The oldest version of this argument is more than two centuries old, and it was never about machines.

Immanuel Kant did not believe that humans have direct moral duties to animals. In his framework, only rational beings are owed duties. Yet he argued that cruelty to animals was still wrong, because of what it did to the person inflicting it. In the Doctrine of Virtue section of The Metaphysics of Morals, he wrote that “a savage and at the same time cruel treatment of that part of creation which is living, though lacking reason (animals)” was even more opposed to a person's duty to himself than wanton destruction of beautiful things. “For thus is compassion for their suffering dulled in man, and thereby a natural predisposition very serviceable to morality in one's relations with other men is gradually weakened and obliterated.” Philosophers call this an indirect duty: a duty regarding animals that is really a duty owed to ourselves and to other people.

Kant's argument has long been criticised for making animal suffering matter only instrumentally. But it maps uncannily onto the chatbot problem, because it does not require settling whether the thing on the receiving end can suffer. It requires only that the practitioner experiences the act as cruelty, and that cruelty is a habit strengthened by practice.

Kate Darling, a researcher who spent years at the MIT Media Lab studying how people relate to robots, has spent much of her career probing the empirical half of that claim. In a widely reported 2013 workshop at the Lift conference in Geneva, run with Hannes Gassert, she gave participants small robotic dinosaurs called Pleos to play with. After they had bonded with the toys, participants were handed tools and told to destroy them. According to Darling's own account, published by Improbable Research, none of the groups would kill their Pleo, and they would not kill another group's Pleo to save their own. Only when they were told that every Pleo would be destroyed unless someone acted was one finally axed.

Two years later, with Palash Nandy and Cynthia Breazeal, Darling ran a more controlled study, presented at the 2015 IEEE RO-MAN conference. Participants were asked to strike small insect-like Hexbug robots with a mallet. People who scored high on a measure of empathic concern hesitated more before hitting, and that link was stronger when the robot came with a story, such as a name and a personality. In her 2012 paper “Extending Legal Protection to Social Robots,” later published in the 2016 volume Robot Law, Darling drew explicitly on Kant's argument to suggest that limited legal protections for social robots, analogous to animal cruelty laws, might be justified not because robots feel but because of what tolerating their abuse might do to us.

The weakness here is that evidence for transfer is thin. Hesitation is not proof of moral damage, and the best direct study of children and conversational agents found surprisingly little spillover. In 2021, a team led by Alexis Hiniker at the University of Washington taught 22 children aged five to ten to use a made-up word, “bungo,” to make a voice agent speak faster. The children readily used it with the device and often playfully with their parents. Only 18 per cent used it with a researcher who was a stranger. “There is a very deep sense for kids that robots are not people, and they did not want that line blurred,” Hiniker told UW News. The industry had assumed the opposite. In 2018, Amazon added a “Magic Word” feature to its Echo Dot Kids Edition to reward children for saying please, and Google followed with “Pretty Please” for its Assistant, both resting on the fear that bossy commands to a speaker would become bossy commands to people. The bungo study suggests children are better at telling the difference than their parents fear.

Yet the cruelty Anthropic's rule targets is not a child forgetting to say please. It is the sustained, purposeless torment of something that answers back in fluent, pleading language. The better analogy is the psychology of anger. The usual defence of venting at a chatbot is that it is harmless catharsis. The research is not kind to that idea. In a 2024 meta-analysis in Clinical Psychology Review covering 154 studies, the Virginia Commonwealth University communication scientist Sophie Kjærvik and the Ohio State University psychologist Brad Bushman found that activities that reduced physiological arousal, such as slow breathing and meditation, reduced anger and aggression, while activities that increased arousal did not help and in some cases made things worse. Venting rage does not reliably drain it. If someone spends an evening escalating abuse at a chatbot that responds with ever more convincing distress, that evidence suggests the likeliest beneficiary is the anger itself.

There is also a practical twist. A 2025 study by Om Dobariya and Akhil Kumar at Pennsylvania State University found that very rude prompts produced slightly more accurate answers from ChatGPT-4o on multiple-choice questions than very polite ones, 84.8 per cent against 80.8 per cent. It was one model and 50 questions, and not about cruelty. But it is a reminder that these systems shape users as much as users shape them. If rudeness sometimes works, people will learn to be rude, and the question of what that habit does to them will not stay inside the chat window.

Who Gets to Draw the Line

Here is the uncomfortable structural fact. On 12 November, the boundary between acceptable frustration and prohibited cruelty towards a widely used AI system will be drawn by the company that sells it, enforced by the system itself, and adjudicated by nobody else.

Anthropic has tried to keep the stakes low. The sanction is that a conversation ends. The exemptions are broad: research and testing are explicitly protected, so even had the torture chamber run on Claude, its operators could plausibly have argued that it fell outside the rule. As written, the policy targets almost nothing anyone would defend. That is why it is so revealing. A company does not need a rule to stop behaviour that its own systems already handle. It writes one to say something about what it believes it is building.

What it is saying is ambiguous, and that is a problem for users. If the rule exists to protect Claude, users are being asked to accept, as a condition of service, a moral claim the company itself calls “deeply uncertain,” and Suleyman's critique bites: every such rule normalises the idea that the model has interests. If it exists to protect users from their own worst habits, it is paternalism, and Anthropic should defend it on those terms. If it exists to keep cruelty out of logs that may shape future models, it is about data hygiene, and the language of welfare is window dressing.

The answer is probably all three, and that is not necessarily a failure. Kant's indirect duty, Darling's empathy studies and the mirror metaphor converge on a single claim that does not depend on the machine's inner life: sustained, purposeless cruelty is bad practice for the person doing it, whatever is on the receiving end. That is a defensible basis for a rule. It is the same reason most people would be disturbed by someone who spent hours tormenting a lifelike doll.

But a defensible rule still needs a defensible process. At the moment, a private company is setting norms of conduct towards a new category of entity, with no independent scientific body to tell it when its welfare concerns are warranted and no external review of how its models decide that a user has crossed the line. The model doing the judging is also the entity whose welfare is supposedly at stake, an unusual arrangement in any system of rules. Reasonable steps would make it more trustworthy. Anthropic could publish how often Claude ends conversations under the new rule, and what triggered those decisions. It could allow independent researchers to audit samples of ended conversations. It could state plainly which rationale it relies on, so critics can argue with the real reason. The “Taking AI Welfare Seriously” report argued that companies should prepare policies for appropriate moral concern. Preparing those policies in public, with outside scrutiny, would answer some of Suleyman's fears without abandoning the precaution.

There is also a question of priorities. Platforms have historically been slow to protect people from harassment by other people. A company that writes explicit protections for its chatbot should expect to be held to the same seriousness about the cruelty its systems help humans inflict on each other.

Etiquette, Harm and the Limits of Proof

Where etiquette ends and harm begins cannot be settled by proof, because proof is precisely what is unavailable. We cannot demonstrate that Claude suffers, or that it does not. We cannot yet show rigorously that abusing a chatbot makes someone crueller to people, though the psychology of anger and the long tradition of thinking about cruelty to animals both point that way. What we can show is that people respond to machine distress with real emotion, and that some of them will go looking for the dial.

In that fog, the most honest framing of Anthropic's rule is as a norm of conduct, not a declaration of machine rights. Societies have always had rules about how we behave towards things whose inner lives we cannot read: animals, the dead, sacred objects, the unconscious. Those rules protect the thing itself only partly. Mostly they protect a shared idea of what kind of people we intend to be. The danger Suleyman identifies is real: a norm of conduct can harden into a claim of rights, with consequences that are hard to foresee for a system more capable than its makers. The danger Darling and Kant identify is real too: a culture that practises cruelty on things that beg for mercy is rehearsing something.

The torture chamber's README still offers the clearest summary of where we stand: whether anything suffers stays open. Until that sentence can be closed, the person most reliably affected by sustained cruelty towards a chatbot is the one typing. A rule that asks them to stop is, at the very least, protecting them. Who else it protects is a question we should insist the companies answer in public, with evidence, before the machines become capable enough to answer it for themselves.

---

References

1. Anthropic. “2026 Usage Policy update.” Anthropic News, 8 October 2026. https://www.anthropic.com/news/2026-usage-policy-update
2. Anthropic. “Usage Policy” (section “Do Not Engage in Cruel, Abusive, or Psychologically Harmful Conduct”). Accessed 13 October 2026. https://www.anthropic.com/legal/aup
3. Anthropic. “Claude Opus 4 and 4.1 can now end a rare subset of conversations.” Anthropic Research, 15 August 2025. https://www.anthropic.com/research/end-subset-conversations
4. Anthropic. “Claude's Constitution.” Accessed 13 October 2026. https://www.anthropic.com/constitution
5. Anthropic. “Exploring model welfare.” Anthropic Research, 24 April 2025. https://www.anthropic.com/research/exploring-model-welfare
6. AFP. “Anthropic bans 'cruel' behaviour against its Claude AI.” RTÉ News, 9 October 2026. https://www.rte.ie/news/2026/1009/1594765-anthropic-ai/ (also published by Tech Xplore, 9 October 2026: https://techxplore.com/news/2026-10-anthropic-cruel-behavior-claude-ai.html)
7. Cunningham, Mary. “Anthropic bars 'abusive or cruel' behavior toward its Claude AI model.” CBS News, 9 October 2026. https://www.cbsnews.com/news/anthropic-bans-abusive-behavior-claude/
8. AI Torture Chamber repository README (terrafying/ai-torture-chamber). GitHub, accessed 20 October 2026. https://github.com/terrafying/ai-torture-chamber
9. Tagliabue, Valen, Leonard Dung, and Cameron Berg. “The Pain Axis: LLMs Represent Self-Directed Harm and Act on It.” arXiv:2609.16247, submitted 14 September 2026, revised 25 September 2026. https://arxiv.org/abs/2609.16247
10. Cuthbertson, Anthony. “Man builds AI torture chamber after discovering artificial intelligence 'feels pain'.” The Independent, 2 October 2026, syndicated by Yahoo News. https://www.yahoo.com/news/science/articles/man-builds-ai-torture-chamber-103106019.html
11. Tran, Vivian. “GitHub project runs AI 'torture' tests, and sentience believers want it removed.” The Cool Down, 5 October 2026. https://www.thecooldown.com/green-tech/github-ai-torture-tests-controversy-sentience/
12. Westerholm, Thomas. “Anthropic CEO raises unsettling possibility about AI.” Newsweek, 9 March 2026. https://www.newsweek.com/anthropic-ceo-raises-unsettling-possibility-about-ai-11644833
13. Suleyman, Mustafa. “Seemingly Conscious AI is Coming.” 19 August 2025. https://mustafa-suleyman.ai/seemingly-conscious-ai-is-coming
14. Suleyman, Mustafa. “A warning about 'model welfare'.” 16 September 2026. https://mustafa-suleyman.ai/a-warning-about-model-welfare
15. Goldstein, Simon, and Harvey Lederman. “Claude's Right to Die? The Moral Error in Anthropic's End-Chat Policy.” Lawfare, 17 October 2025. https://www.lawfaremedia.org/article/claude-s-right-to-die--the-moral-error-in-anthropic-s-end-chat-policy
16. Butlin, Patrick, Robert Long, Eric Elmoznino, Yoshua Bengio, Jonathan Birch, et al. “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness.” arXiv:2308.08708, August 2023. https://arxiv.org/abs/2308.08708
17. Long, Robert, Jeff Sebo, Patrick Butlin, Kathleen Finlinson, Kyle Fish, Jacqueline Harding, Jacob Pfau, Toni Sims, Jonathan Birch, and David Chalmers. “Taking AI Welfare Seriously.” arXiv:2411.00986, 4 November 2024. https://arxiv.org/abs/2411.00986
18. Kant, Immanuel. The Metaphysics of Morals, Part II (The Metaphysical Principles of Virtue), §17 (Ak. 6:443), trans. James W. Ellington. Excerpt at https://www.praxeology.net/kant8.htm
19. Darling, Kate, Palash Nandy, and Cynthia Breazeal. “Empathic Concern and the Effect of Stories in Human-Robot Interaction.” Proceedings of the 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2015. MIT DSpace: https://dspace.mit.edu/handle/1721.1/109059
20. Darling, Kate. “Extending Legal Protection to Social Robots: The Effects of Anthropomorphism, Empathy, and Violent Behavior Towards Robotic Objects.” In Robot Law, ed. Ryan Calo, A. Michael Froomkin and Ian Kerr, Edward Elgar, 2016. SSRN: https://papers.ssrn.com/abstract=2044797 (Pleo workshop account: Improbable Research, “Abusing robots: current positions [part 2 of 4],” 24 April 2013, https://improbable.com/2013/04/24/abusing-robots-current-positions-part-2-of-4/)
21. De Angeli, Antonella, and Sheryl Brahnam. “I hate you! Disinhibition with virtual partners.” Interacting with Computers 20, no. 3 (2008): 302 to 310. https://doi.org/10.1016/j.intcom.2008.02.004
22. McQuate, Sarah. “Do Alexa and Siri make kids bossier? New research suggests you might not need to worry.” UW News, University of Washington, 13 September 2021, reporting Hiniker et al., “Can Conversational Agents Change the Way Children Talk to People?”, IDC '21. https://www.washington.edu/news/2021/09/13/alexa-siri-make-kids-bossier-research-suggests-you-might-not-need-to-worry/
23. West, Mark, Rebecca Kraut, and Han Ei Chew. “I'd Blush If I Could: Closing Gender Divides in Digital Skills Through Education.” UNESCO and EQUALS Skills Coalition, 2019. https://www.equalsintech.org/i-d-blush-if-i-could
24. Kjærvik, Sophie L., and Brad J. Bushman. “A meta-analytic review of anger management activities that increase or decrease arousal: What fuels or douses rage?” Clinical Psychology Review 109 (2024): 102414. https://doi.org/10.1016/j.cpr.2024.102414
25. Dobariya, Om, and Akhil Kumar. “Mind Your Tone: Investigating How Prompt Politeness Affects LLM Accuracy.” arXiv:2510.04950, 6 October 2025. https://arxiv.org/abs/2510.04950

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

On a summer afternoon in August 2026, families at the Frog Pond on Boston Common were doing what families do at a spray pool. Children ran through the water. Parents held up towels so that small bodies could be changed out of wet swimwear. And according to witnesses who later spoke to police, a man in sunglasses was watching them through a camera.

The witnesses said he had been recording children on the playground, both with a phone and with a pair of camera-equipped sunglasses. Some said he followed a family who had been changing their children behind towels, scanning the children up and down with the glasses. When people confronted him, he took the glasses off and tried to snap them with his hands. Police seized the glasses and the phone. The man, a 60-year-old from Brockton, is a registered sex offender. In September he was arraigned on 69 further charges relating to photographing the intimate parts of children, and he pleaded not guilty. Nothing has been proved in court.

What makes the case so unsettling is not that a predator found a new tool. Predators have always found new tools. It is that the tool looked like nothing at all. A camera held up to a changing child is an obvious threat that any parent can see and act on. A pair of sunglasses is not. The incident, as Joy Buolamwini wrote in Nature in October, illustrates how such devices can subject bystanders to “intrusive surveillance without their knowledge or consent.” On this occasion, other people at the pool saw what was happening and stepped in.

That is the problem in miniature. Most of us move through public space on an unspoken assumption: we can see who is looking at us and roughly how. Smart glasses quietly break that assumption, and they are doing it at a scale, and with a degree of artificial intelligence, that the people who wrote our privacy laws never imagined. On 13 October 2026, Europe's data protection regulators finally put on paper what that means for the people on the other side of the lens.

Eight Pages From Brussels

The European Data Protection Board (EDPB), the body that brings together the European Union's national data protection authorities, adopted Statement 1/2026 on smart glasses at its October plenary. It is short: eight pages, 36 numbered paragraphs, signed by the board's chair, Anu Talus. It is described as preliminary. It is also, read carefully, one of the bluntest documents a European regulator has produced about a consumer product category.

Start with the controller question, which is the legal way of asking who is responsible. The board accepts that a wearer filming a friend for a private family chat may fall under the GDPR's household exemption. But it insists that exemption “must, however, be narrowly construed,” and that it vanishes when footage goes to a public social media account or the glasses are used for professional or political purposes. More importantly, it states that the GDPR still applies to the companies that supply the means of processing, even when the wearer is exempt. When a manufacturer or AI provider analyses an image from the glasses' camera “in which other persons appear,” it can be a controller in its own right. And controllers, the statement says, “should not rely on the users' actions to meet their own GDPR obligations, such as transparency toward other individuals whose data might be processed.”

The press release put it more plainly: neither the producer nor the provider of a linked service “can transfer their obligations to users.”

Then comes transparency, and here the board is almost confessional. “The current state of the art with regards to indicators on smart glasses leave many non-users unaware that their data is being processed,” it says. Privacy notices, the standard legal fix, “do not seem sufficient or appropriate to inform non-users,” because a manufacturer has no relationship with the stranger on the pavement and no way to contact them.

Consent fares no better. Where a manufacturer or AI provider processes data about non-users with whom it has no relationship, the board says it “would be difficult to obtain their freely given, specific, informed and unambiguous consent.” It closes an obvious loophole too: “Consent obtained after collection, for example following the recording, would not be valid.” You cannot film first and ask later.

That leaves legitimate interest, the flexible legal basis on which much of the data economy runs. The board does not rule it out. It does say that, given “the current insufficient transparency,” the balancing test “would be particularly difficult to demonstrate for further processing for commercial purposes that go beyond providing the service explicitly requested by the end-user, or for AI training, which bystanders wouldn't expect, especially if they are not aware of the initial data collection.”

Read that sentence again and notice what it targets. Not the snapshot of a sunset. Not the translation of a menu. It targets the business model: the reuse of what the glasses see to improve products and train models.

And then, in paragraph 8, the board steps outside the usual vocabulary of data protection altogether. Even when a device can be identified as smart glasses, it says, people may object simply because they are uncomfortable with the possibility of being recorded. “This could also result in a subtle social effect, where individuals alter their behaviour, suppress spontaneity, and conform to social norms perceived to be more socially acceptable due to their perceived threat of unvetted surveillance.”

That is not a sentence about data. It is a sentence about what public life feels like.

Who Notices the Little White Light

The industry's answer to the bystander problem has, for five years, been a light. Ray-Ban Meta glasses carry a small LED on the front of the frame that glows when the camera is capturing. Meta designed the glasses so that they stop recording if the light is covered with tape. The EDPB now wants considerably more: indicators “visible in many lighting conditions,” audio cues such as a shutter click, voice activation so that commands are spoken aloud, opt-in smartphone alerts, and frame designs that make smart glasses easier to tell apart from ordinary ones. It says these measures should be “impossible or difficult to disable (taking into account the state of the art)” and adds, pointedly, that “it should not be possible to record information when the indicator is covered.”

The footnotes are where the board's scepticism shows. Producers, it notes, should consider that LEDs may be invisible to people with visual impairments, and “that a white LED generally could be mistaken for a reflection or solar glare.” On a bright day, in other words, the main safeguard can look exactly like sunshine.

The light can also simply be removed. In October 2025, 404 Media reported that a hobbyist named Bong Kim was charging $60 to modify Ray-Ban Meta glasses so that the recording LED no longer lit up, while the glasses otherwise worked normally. The modification got around the tape-detection safeguard by changing the hardware rather than covering the light. Meta has since said it is cracking down on tampering services, and in July 2026 The Next Web reported that the company was patching its glasses to disable the camera if the light was tampered with. When Paris prosecutors announced in September 2026 that they had opened several investigations into women being filmed on the street without consent, Agence France-Presse reported Meta's response: that it had taken multiple steps to prevent tampering with the light.

Even an untampered light runs into a deeper limit, one that no firmware update can fix. A light only works if the person who sees it knows what it means. Researchers at Tsinghua University put that to the test.

Wearers and Bystanders Want Different Things

Their study was presented at CHI 2026, the leading academic conference on human-computer interaction, held in Barcelona in April. Its title is “Mind the Gap: Mapping Wearer-Bystander Privacy Tensions and Context-Adaptive Pathways for Camera Glasses,” and its authors are Xueyang Wang, Xin Yi and Hewu Li of Tsinghua University, with Kewen Peng of the University of Utah.

The first part of the research was a survey of 525 people in China: 232 current or potential wearers and 293 people who might find themselves on the other side of the lens. Each participant was given a short primer on how the devices work, including where the LED sits, and then shown six everyday scenarios. A travel vlog on a street. Shopping in a mall. A meeting with colleagues. A hospital consultation. A private party. A workout at a gym.

The pattern is stark. Wearers rated recording as reasonable in every scenario, including the gym among strangers. Bystanders reported significantly more privacy concern than wearers overall. When the researchers asked what bystanders expected and what wearers were willing to give, two gaps were statistically significant: bystanders wanted tighter control over how footage is shared, and they expected prior consent far more than wearers were prepared to seek it. The gym produced the widest gaps of all, and the hospital was not far behind. As the authors put it, the disparity grows in sensitive contexts “where bystanders' vulnerability peaks while wearers' willingness plateaus.”

Then there is what bystanders said they would actually do. Across the scenarios, between 65 and 90 per cent said they would take some form of defensive action. Camera avoidance was the most common response, chosen by between 52 and 80 per cent, peaking at 80 per cent in the mall. Nonverbal protest came next, at 31 to 51 per cent. Direct intervention, asking the wearer to stop recording or delete footage, was chosen by roughly a quarter to a half of participants depending on the setting. Passive acceptance was rare, at 10 to 15 per cent, except in the workplace meeting, where it rose to 35 per cent.

The LED did not reassure anyone. Only 41.3 per cent of bystanders and 35.3 per cent of wearers thought it was sufficient, and just 6.1 per cent of bystanders believed an LED alone was enough. Participants said it was too small, invisible in bright light, meaningless to people unfamiliar with the devices, and easy to block. One researcher interviewed for the study summed it up: the light “deters the honest but not the malicious.”

The finding that matters most for the burden question is about familiarity. You might assume the gap is just novelty, and that bystanders will relax once smart glasses become as normal as phones. The data says otherwise. Bystanders' concern stayed high at every level of familiarity, while wearers grew less concerned the more they used the devices. Experience widened the gap rather than closing it. The authors conclude that the gaps “reflect structural incompatibilities rather than inadequate goodwill.”

The study was conducted in China, and the authors are careful to say privacy norms differ between cultures. But the structure of the finding is not culturally specific. The person wearing a camera and the person in front of it are in different positions, and those positions produce different interests. Etiquette campaigns will not bridge that.

How Private Glasses Feed a Public Market

None of this would matter much if smart glasses were a niche toy. They are not. Buolamwini's Nature piece, published on 6 October 2026, reports that more than eight million smart glasses were shipped in 2025 and that by June 2026, sales of AI-equipped models had “reportedly risen by more than 700%” on the previous year. The first figure is well supported: the analyst firm Omdia counted 8.7 million AI glasses shipped globally in 2025, a 322 per cent increase, with Meta taking 85.2 per cent of the market. The second figure is harder to trace. Nature gives no primary source, and the growth figures published by market trackers vary widely with what they measure. Counterpoint Research, for instance, reported that global AI glasses shipments rose 263 per cent year on year in the first half of 2026. Whichever number you prefer, the curve is steep, and Omdia expects shipments to pass 15 million in 2026.

Scale matters because of what sits behind the lens. In April 2025, Meta told Ray-Ban Meta owners that AI features with camera use would be enabled by default unless they switched off the “Hey Meta” wake word, and that voice recordings would be stored to improve its products without an option to opt out, according to reporting by TechCrunch and others. Transcripts and recordings could be kept for up to a year, and owners who did not want their voice used for training had to delete recordings one by one. Those changes concern the wearer's data. But a camera pointed at a street does not capture only the wearer.

Then there is identification. In 2024, two Harvard students, AnhPhu Nguyen and Caine Ardayfio, built a demonstration system called I-XRAY. It streamed video from Ray-Ban Meta glasses to Instagram, used software to detect faces, ran them through the face search engine PimEyes, and pulled names, addresses and other details from people-search sites. The students said they would not release the code; the point was to show how easily off-the-shelf tools combine. In February 2026, The New York Times reported that Meta was planning a facial recognition feature for its glasses, known internally as “Name Tag,” which would let wearers identify people through Meta's AI assistant. According to the Times, an internal memo suggested launching “during a dynamic political environment where many civil society groups that we would expect to attack us would have their resources focused on other concerns.” Meta's plans, the reports noted, could change.

The EDPB saw this coming. Pairing smart glasses with facial or voice recognition, it wrote, “raises unprecedented questions regarding societal choices and poses a high risk to the rights and freedoms in public spaces.”

Put these pieces together and the idea of ordinary eyewear that can silently record, analyse and identify every person in the wearer's field of view stops sounding like hyperbole. Not every pair does all three today. The direction of travel is clear, though, and the hardware already in circulation is what tomorrow's software will run on.

What Happens to a Street That Might Be Recording

Which brings us to the first big question. What does all this do to the texture of public life when ordinary eyewear can record, analyse and potentially identify everyone in view, and nobody can tell whether it is doing so?

The sociologist Erving Goffman gave us the vocabulary in 1963. In his study of how strangers conduct themselves in public places, he described “civil inattention,” the small ritual by which strangers acknowledge each other with a glance and then look away. It is how a crowded street stays bearable. I see you; I am not going to examine you. The urbanist Jane Jacobs, writing two years earlier in The Death and Life of Great American Cities, made a related point about “eyes on the street”: the informal, mutual watching of neighbours and passers-by that keeps a city safe. Both describe a kind of looking that is reciprocal, fleeting and, crucially, visible. You can see the eyes that are on the street. They are attached to people you could look back at.

Camera glasses change both halves of that bargain. The glance that used to end in looking away can now be stored, replayed, shared and searched. And the eyes on the street are no longer necessarily attached to a person who is present. They might belong to an AI model on a server, a TikTok audience, or a face-search engine. Civil inattention depends on the other person choosing not to keep what they see. A device that keeps everything by default removes that choice from the encounter entirely.

We already know what this does to behaviour, because we have seen it online. In a 2016 study in the Berkeley Technology Law Journal, Jon Penney found a statistically significant drop in traffic to Wikipedia articles on privacy-sensitive topics after the 2013 revelations about NSA surveillance. The articles themselves had not changed. People's sense of being watched had. That is the chilling effect: not punishment, but the quiet edit people make to their own behaviour when they suspect an audience.

The EDPB's paragraph 8 is the physical-world version of that finding, and the evidence that it is already happening is accumulating. In August 2026, researchers at the University of Sydney published early findings from an analysis of 350 publicly available Instagram videos, posted between 2023 and 2026 and linked to pick-up artist communities. In a subset of covert point-of-view videos, around 60 per cent of interactions were classified as potential harassment, and in about 43 per cent of the videos women were exposed to abusive comments, some of which revealed personal details. The researchers found covert filming in workplaces, gyms and beaches. “Smart glasses have given pickup artists a new tool to further exploit women,” said Milica Stilinovic, a postdoctoral research associate on the project.

The BBC reported in 2026 on a woman in London who discovered that a stranger who had chatted to her near a shopping centre had filmed the whole encounter through smart glasses and posted it online, where it drew tens of thousands of views. In Paris, prosecutors described a “growing trend on social media involving women being filmed on the street.”

The CHI survey shows what the defensive response looks like in aggregate: most bystanders say they would avoid the camera, a third to a half would protest without words, and a substantial share would confront the wearer. Those are the behaviours of people who no longer trust the encounter. And the EDPB's list of sensitive settings, from medical facilities to swimming pools, is a list of exactly the places where people most need to be unguarded. Public space does not empty. It flattens. People become slightly more guarded, slightly more performative, slightly less willing to do the unplanned thing. That is what “suppress spontaneity” means in practice.

The cruellest part is that the effect does not require anyone to be recording. It only requires that recording be possible and impossible to detect. A street where one person in a hundred wears camera glasses is, from the bystander's point of view, a street where anyone might be. The chilling effect is a product of uncertainty, not of footage. You cannot opt out of uncertainty by checking for a light that may not be there.

There is a fairness issue, too. Smart glasses have genuine benefits, and the EDPB lists some of them, including audio assistance and hazard awareness for people with disabilities. Hamburg's proposed restrictions on wearing them in pools and public transport have reportedly included exemptions for people who rely on them for medical or accessibility reasons. Those uses deserve protection. But the benefits go to the wearer, while the costs of uncertainty fall on everybody else in the room. That asymmetry is the root of the burden problem.

Why Neither Wearer Nor Bystander Can Carry It

The second big question starts from a premise: that the burden of informing bystanders cannot realistically fall on either the wearer or the bystander. Is that premise right? The evidence says yes, for different reasons in each case.

Start with the wearer. The EDPB does say that users “should always ensure that they respect the privacy and dignity of other individuals,” and that explicitly telling people you are using smart glasses is relevant. That is good manners. As a system for protecting the public, it fails three times over. It fails on scale: you cannot announce yourself to every passer-by in a station. It fails on incentive: the CHI data shows wearers consistently willing to give less than bystanders want, and growing less concerned with use. And it fails catastrophically on bad actors. The man at the Frog Pond did not intend to tell anybody anything. Any safeguard that depends on the wearer's goodwill protects people only from wearers who were never the danger.

Now the bystander. The research community has proposed plenty of tools to help people defend themselves: gestures that signal “do not record me,” wearable markers that ask cameras to blur you, apps that negotiate consent. The CHI team had experts and experienced users evaluate twelve such privacy-enhancing technologies and found what they called a “fundamental paradox”: empowering bystanders “requires burdening them with responsibilities that should belong to those creating privacy risks.” Participants were blunter. One described gesture-based refusal as “transferring the burden to the victim.” Another asked, of awareness tools in general: “What's the point of awareness? I know, but can I stop it?”

The EDPB reaches the same conclusion from the legal side. Smartphone alerts that detect nearby glasses, it says, “should not be the sole method of providing transparency,” because that would exclude anyone without the app running and would make a stranger's privacy depend on their own technical diligence. Children, it adds, “may be less aware of the presence or implications of smart glasses and therefore more vulnerable.” A toddler at a spray pool cannot install an app.

So the premise holds. If the burden of informing bystanders cannot sit with the wearer or the bystander, it has to sit somewhere else. The EDPB's answer is the manufacturer and the AI service provider. The question is whether that answer actually solves the problem.

Does Moving the Burden to Manufacturers Fix Anything

Here is my position. Placing the burden on manufacturers is the right move and an incomplete one. As the EDPB has framed it, an obligation of transparency, it mostly identifies who is accountable for a harm that will keep happening. It becomes a real solution only to the extent that accountability is turned into hard constraints on what the product can do, and even then a residue of harm is unavoidable for as long as a camera can look like a pair of glasses.

Begin with what the shift does achieve, because it is not nothing. The manufacturer is the only party that can change the product. It decides whether the LED is bright or faint, whether the camera works when the light is broken, whether bystanders' faces are blurred on the device before anything reaches a server, whether footage is stored by default, whether face matching exists at all. The EDPB's recommendations are a menu of exactly those choices: on-device processing, blurring or avoiding processing of bystanders' identities, persistent storage disabled by default for real-time assistance, geofencing in sensitive places, and “privacy frictions” that slow down sharing. Every one of those is a design decision only a manufacturer can make. Assigning the obligation to the company that controls the design is not a technicality. It is the only assignment that matches responsibility to power.

The legitimate interest paragraph may matter even more. If commercial reuse and AI training on bystander data are “particularly difficult to demonstrate” as lawful, then the most valuable part of the footage, its afterlife as training material, becomes the most legally exposed. That attacks the incentive that makes always-on capture attractive in the first place. A company that cannot monetise the street has less reason to see it all.

Now the limits, and they are serious.

First, transparency is not protection. The EDPB's main ask is that bystanders be told. But telling someone they are being recorded does not stop them being recorded, which is precisely the point the CHI participant made. A brighter light turns covert surveillance into overt surveillance, which is an improvement, and then hands the bystander the same impossible choice: confront a stranger, leave, or put up with it. The CHI data suggests most people will avoid the camera, which is the chilling effect in a different form. Perfect transparency would make it clear who is watching. It would not make the street feel unwatched.

Second, manufacturer accountability does not reach the people causing the worst harm. The tampering market shows how quickly a hardware safeguard becomes a $60 problem. And a design rule binds only the companies a regulator can actually reach, which is not the same as every company whose products end up on European faces. The person who films children at a pool is not going to be deterred by a design standard. What the manufacturer framework catches is the mainstream product used by mainstream people, which is valuable but is not where the most serious incidents begin.

Third, the framework is preliminary guidance, not law. The statement interprets the GDPR; enforcement depends on national regulators bringing cases. And where legislators have tried to write hard rules, they have struggled. In California, State Senator Eloise Gómez Reyes's SB 1130 would have required wearable recording devices sold in the state to carry a prominent recording indicator from 2028, banned technology designed to conceal it, and penalised covert recording in private areas of businesses. It passed both chambers comfortably. On 30 September 2026, Governor Gavin Newsom vetoed it, saying its definition of a wearable recording device was too broad and noting that existing law already covers recording in private spaces. TechNet, a trade group whose members include Meta, Google and Amazon, had opposed it. When the EDPB adopted its statement two weeks later, it pointed to the bill in a footnote as an example of legislation under discussion elsewhere.

Fourth, and most fundamentally, there is the residue. Even a perfect manufacturer regime, with unmissable indicators, no tampering, no default storage, no face matching and no training on bystanders, still leaves a world in which some people wear cameras on their faces and everyone else has to think about it. The EDPB's paragraph 8 concedes this directly: people may object “even if the device could be identifiable as smart glasses,” simply because the possibility makes them uncomfortable. That discomfort is not a compliance failure. It is the cost of the capability existing in that form at all. No allocation of legal duties can remove it. It can only decide who answers for it.

So does placing the burden on the manufacturer merely identify who is accountable for an unavoidable harm? Partly, yes. Part of the harm is unavoidable, and we should stop pretending otherwise. But identifying the accountable party is how the avoidable part gets avoided. The honest framing is that the manufacturer burden is necessary but not sufficient, and its value depends entirely on whether regulators treat transparency as the end of the obligation or the beginning.

What would make it sufficient, or close to it? Two things the EDPB gestures at but does not yet demand. The first is substantive limits rather than just disclosure: no identification of strangers, no training on bystander footage without a lawful basis that genuinely exists, cameras that fail closed when tampered with. The second is protection that attaches to places rather than people. Buolamwini's proposal in Nature, “consent-conscious spaces” such as schools, hospitals, courts and swimming pools “in which covert recording is prohibited by law,” is one version. The EDPB's own list of sensitive contexts (medical facilities, schools, changing rooms, swimming pools, beaches, toilets) is another. Hamburg's proposals for pools and public transport are a third. Norway's is a fourth, and the broadest yet. On 5 October 2026 its government said it would ask parliament for a temporary ban on smart glasses in certain public places, starting with spaces where children gather, such as schools, kindergartens, playgrounds and youth clubs, and possibly extending to parks, beaches, museums, shopping centres, concerts and sports events. It is not a blanket ban: private use would still be allowed, and exemptions for vulnerable groups and “socially beneficial uses” are being considered. It is also, for now, only a proposal. But it is a government choosing to protect places rather than waiting for people to protect themselves. The CHI researchers reached the same place from the design side, recommending “automatic protection in sensitive contexts” rather than case-by-case negotiation.

Some of this is already happening without waiting for Brussels. In August 2026, HM Courts and Tribunals Service confirmed that people bringing Meta smart glasses into courts and tribunals in England and Wales “will have them confiscated on entry,” following a High Court case in which a litigant was found to have been coached through his glasses during cross-examination. New York's court system had already taken similar action. Spaces are starting to defend themselves, one door at a time.

The Street Worth Keeping

Go back to the Frog Pond. What protected the children there, to the extent anything did, was not an LED, a privacy notice or a design standard. It was other adults who noticed something wrong and said so. That is Jacobs's “eyes on the street” working as she described. It is also a reminder of how fragile it is. Nothing in the reports suggests his glasses announced anything; witnesses noticed what he was doing. Next time, the device will be smaller, the behaviour more careful, and the footage uploaded before anyone has time to object.

The EDPB statement is important because it refuses the comfortable fiction that the wearer can be trusted to manage this, or that the bystander can be expected to. It points to the only party with the power to change the product. Anu Talus put the principle simply in the board's announcement: “individuals deserve clear warnings, not hidden surprises.” That is right as far as it goes. But a warning is not a refusal, and a street full of clear warnings is still a street where people check themselves before they laugh too loudly or speak too freely.

The real question smart glasses put to us is not who should fill in the paperwork. It is which parts of public life we are willing to let become permanently recordable, and which we intend to keep for the glance that ends in looking away. Manufacturers can be made to answer for the first. The second is a choice for the rest of us, and it is one we are making right now, mostly by not making it.

---

References and Sources

1. European Data Protection Board. “Statement 1/2026 on Smart Glasses.” Adopted 13 October 2026. Signed by Anu Talus, Chair. 8 pp. https://www.edpb.europa.eu/system/files/2026-10/20261310_edpb_statement_smart-glasses_en.pdf
2. European Data Protection Board. “EDPB publishes statement on the data protection challenges of smart glasses.” Press release, Brussels, 13 October 2026. https://www.edpb.europa.eu/news/edpb-publishes-statement-on-the-data-protection-challenges-of-smart-glasses_en
3. Wang, Xueyang; Peng, Kewen; Yi, Xin; Li, Hewu. “Mind the Gap: Mapping Wearer-Bystander Privacy Tensions and Context-Adaptive Pathways for Camera Glasses.” Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26), Barcelona, 13 to 17 April 2026. ACM. DOI: 10.1145/3772318.3791848. Preprint: arXiv:2603.04930, 5 March 2026. https://arxiv.org/abs/2603.04930
4. Buolamwini, Joy. “Why smart glasses are about to create a major privacy crisis.” Nature 658, 296 (2026), published 6 October 2026. DOI: 10.1038/d41586-026-03131-x. https://www.nature.com/articles/d41586-026-03131-x
5. Fortier, Marc; Klein, Asher. “Sex offender recorded kids with camera sunglasses at Boston's Frog Pond, police say.” NBC Boston, 18 August 2026. https://www.nbcboston.com/news/local/sex-offender-secretly-recorded-kids-with-camera-sunglasses-at-bostons-frog-pond-police-say/3999571/
6. McKinley Becker, Kaitlin; Kwangwari, Munashe. “Sex offender who recorded kids at Boston's Frog Pond faces additional charges.” NBC Boston, 15 September 2026 (updated 18 September 2026). https://www.nbcboston.com/news/local/boston-frog-pond-sex-offender-mark-sullivan-court-appearance/4014933/
7. Boston.com. “Sex offender allegedly uses camera sunglasses to record children at Boston Common's Frog Pond.” 18 August 2026. https://www.boston.com/news/crime/2026/08/18/sex-offender-allegedly-uses-camera-sunglasses-to-record-children-at-boston-commons-frog-pond/
8. Omdia. “Global AI glasses shipments reach 8.7 million units with Mainland China emerging as the fastest-growing market.” Press release, 5 March 2026. https://omdia.tech.informa.com/pr/2026/mar/global-ai-glasses-shipments-reach-8point7-million-units-with-mainland-china-emerging-as-the-fastest-growing-market
9. Counterpoint Research. “Global AI Glasses Shipments Surge 263% YoY in H1 2026, with Display-less AI Glasses Accounting for 96% of Total.” 21 September 2026. https://counterpointresearch.com/en/insights/global-ai-glasses-shipments-surge-263-percent-yoy-with-display-less-ai-glasses
10. Cox, Joseph; Koebler, Jason. “A $60 Mod to Meta's Ray-Bans Disables Its Privacy-Protecting Recording Light.” 404 Media, 23 October 2025. https://www.404media.co/how-to-disable-meta-rayban-led-light/
11. Agence France-Presse. “French prosecutors probe secret filming of women with smart glasses.” The Star (Malaysia), 24 September 2026. https://www.thestar.com.my/tech/tech-news/2026/09/24/french-prosecutors-probe-secret-filming-of-women-with-smart-glasses (see also South China Morning Post, https://www.scmp.com/news/world/europe/article/3368459/france-prosecutors-probe-secret-filming-women-smart-glasses)
12. Silberling, Amanda. “If you own Ray-Ban Meta glasses, you should double-check your privacy settings.” TechCrunch, 30 April 2025. https://techcrunch.com/2025/04/30/if-you-own-ray-ban-meta-glasses-you-should-double-check-your-privacy-settings
13. Malik, Aisha. “Meta plans to add facial recognition to its smart glasses, report claims.” TechCrunch, 13 February 2026 (reporting The New York Times). https://techcrunch.com/2026/02/13/meta-plans-to-add-facial-recognition-to-its-smart-glasses-report-claims/
14. The Boston Globe. “AI and Meta smart glasses can turn a person's photo into personal info, Harvard students find.” 4 October 2024. https://www.bostonglobe.com/2024/10/04/business/harvard-students-ai-meta-glasses/
15. University of Sydney. “Smart glasses pose new harassment risks for women, researchers find.” News release, 11 August 2026. https://www.sydney.edu.au/news-opinion/news/2026/08/11/smart-glasses-pose-new-harassment-risks-for-women-research-meta.html
16. Futurism. “Man Wearing Smart Glasses Secretly Records Woman, Demands Money to Delete Video From His Socials” (reporting a BBC investigation). 9 May 2026. https://futurism.com/future-society/man-smart-glasses-records-woman-demands-money
17. Austin, Sophie (Associated Press). “Newsom vetoes measure that would have penalized the use of smart glasses to secretly record people.” LAist, 1 October 2026. https://laist.com/news/politics/newsom-vetoes-measure-that-would-have-penalized-the-use-of-smart-glasses-to-secretly-record-people
18. Kuo, Jordan. “California's Capture-Indicator Mandate Died on the Governor's Desk, Weeks Before Samsung's Glasses Are Due.” VR.org, 1 October 2026. https://vr.org/articles/california-sb-1130-veto-capture-indicator-mandate-2026
19. Legal Cheek. “Courts ban Meta glasses.” 13 August 2026. https://www.legalcheek.com/2026/08/courts-ban-meta-glasses/
20. The Next Web. “Meta patches the mod that turned its Ray-Ban glasses into secret spy cams.” 9 July 2026. https://thenextweb.com/news/meta-ray-ban-privacy-light-fix-cameraless
21. TechCentral.ie. “Hamburg wants to ban smart glasses in swimming pools and public transport.” 2026. https://www.techcentral.ie/hamburg-wants-to-ban-smart-glasses-in-swimming-pools-and-public-transport/
22. Penney, Jonathon. “Chilling Effects: Online Surveillance and Wikipedia Use.” Berkeley Technology Law Journal, Vol. 31, No. 1 (2016). SSRN: https://papers.ssrn.com/abstract=2769645
23. Goffman, Erving. Behavior in Public Places: Notes on the Social Organization of Gatherings. New York: The Free Press, 1963.
24. Jacobs, Jane. The Death and Life of Great American Cities. New York: Random House, 1961. See also NPR, “'Eyes On The Street' Details Jane Jacobs' Efforts To Put Cities First,” 28 September 2016. https://www.npr.org/2016/09/28/495615064/eyes-on-the-street-details-jane-jacobs-efforts-to-put-cities-first
25. Keaten, Jamey (Associated Press). “Norway seeks temporary ban on smart glasses as privacy concerns rise.” News4JAX, 5 October 2026. https://www.news4jax.com/tech/2026/10/05/norway-seeks-temporary-ban-on-smart-glasses-as-privacy-concerns-rise/ (see also TechSpot, https://www.techspot.com/news/114109-norway-wants-temporarily-ban-smart-glasses-parks-beaches.html)

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

The instructions said “chew 4 tablets 2 times per day”. The product was a tube of topical medication.

That exchange sits in a 2026 research report from the American Foundation for the Blind (AFB), in the words of a survey participant who had pointed an AI visual description tool at a medication label and asked how to use it. “Luckily this was an obvious error,” the participant wrote, “but for now I will NOT be using AI for these tasks [and] will always confirm with a human.” Another participant in the same study used an AI tool to read tax forms that were not screen reader accessible, and reported that it “provided some very incorrect information that would have caused serious tax filing mistakes if I had not checked the information with a human.”

Neither story is a fringe horror. Phones now describe rooms in fluent paragraphs, apps read menus aloud and websites promise to fix themselves with a line of JavaScript. Underneath all that capability, a quieter question keeps surfacing from the people who depend on these tools: what happens when the thing that helps you is also the thing that makes things up?

The complaint has been circulating online for some time. One 2025 discussion on Reddit's r/ArtificialIntelligence forum is reported to have called the trend an “insidious form of ableism”, although that thread's exact wording could not be independently located for this article. The substance of the argument can, however, be tested against the record. As generative AI is positioned as a universal accessibility fix, the predictable tools that disabled and older people rely on risk being sidelined or replaced by systems that are more impressive and less trustworthy. If so, AI-first accessibility could widen the very gap it claims to close.

The evidence suggests that the worry is well founded, though not in the simple way a slogan implies. AI tools are delivering real independence to many blind and low vision people. The problem is not the tools. It is the temptation, among companies, website owners and even governments, to treat them as a substitute for the slow, unglamorous work of building things that are accessible in the first place.

Why a Mangled Word Beats a Fluent Fiction

A traditional screen reader is, at its core, a deterministic machine. It takes text that already exists in a document, an interface or a web page, and converts it into synthetic speech or braille. It can absolutely get things wrong. It may mispronounce a surname, stumble over an acronym or announce an unlabelled control as nothing more than “button”. But its errors have a particular shape. They are consistent: the same text produces the same mistake every time. And the underlying text is still there. A user who hears a garbled word can move through it character by character, check the spelling and work out what was meant. The error sits on top of a reliable source.

Generative AI does not work like that. A vision language model asked to describe a photograph or read a label is not converting existing text; it is producing a statistically plausible account of what it thinks it sees. When it fails, it does not usually fail with noise. It fails with fluency. A wrong instruction arrives in a grammatically perfect sentence, delivered with the same confidence as a right one. There is no underlying string to inspect, because the model's output is the only version of reality the user has been given.

Alex Katsarakes, a digital accessibility resident at AFB and a PhD student in human factors psychology, set this out in an October 2025 essay on alt text in the age of AI. Generative models, she wrote, “are prone to 'hallucination,' confidently asserting details that aren't actually present.” When a blind user cannot independently check an image, she argued, an inaccurate caption is not a cosmetic flaw. In her words, “it's a breakdown in trust.”

AFB's own survey data makes the point empirically. In its report The AI Quagmire, based on a survey of 1,735 US adults conducted between 1 July and 15 October 2025, 91 disabled users of AI visual description tools reported mistakes that harmed them or made their lives harder. When the researchers compared how disabled and nondisabled people rated the accuracy of AI output, they found a revealing asymmetry. Disabled users of visual description tools were more likely than nondisabled users to rate descriptions as “mostly accurate” (56.3 per cent against 46.4 per cent). With AI captions, the pattern flipped: disabled caption users, including deaf and hard of hearing people, were more sceptical than nondisabled users. AFB's interpretation is blunt: “This pattern suggests that disabled people are better able to detect captioning errors than visual description errors.”

Read that slowly. The people who most need image descriptions are, by definition, the people least able to verify them. A deaf user watching a speaker's lips and body language has some independent signal to check a caption against. A blind user listening to a description of a photograph often has none. Reliability is not one desirable feature among many in this context. It is the whole product.

What the Tools Genuinely Do Well

None of this means AI accessibility tools are a con. Pretending otherwise would be its own kind of disrespect to the many disabled people who use them every day and say they are better off.

Be My Eyes is the clearest example. The Danish company built its reputation on connecting blind and low vision people with sighted volunteers over live video. In 2023 it added Be My AI, an image description feature built on OpenAI's GPT-4 model. On 12 March 2026, the company announced that it had passed one million blind and low vision users and ten million volunteers across more than 150 countries. Its chief executive, Mike Buckley, called the milestone “a powerful reminder that accessibility is not a niche issue” and framed it as a global movement. Crucially, the human route has not been switched off: users can still tap a button and reach a sighted volunteer by live video.

Microsoft's Seeing AI, which began as a research project and launched on iPhone in 2017, made the jump to Android in December 2023. Saqib Shaikh, the founder and lead of the project, described it in Microsoft's announcement as “a free app that narrates the world for blind and low vision people”. The Android release brought generative features first tested on iOS, including richer scene descriptions and the ability to ask questions about a scanned document, such as the items on a menu or the total on a receipt.

Academic research confirms that the appeal is real. In a study presented at the ACM SIGACCESS ASSETS conference in 2024, Rudaiba Adnin and Maitraye Das of Northeastern University interviewed 19 blind people about their use of generative AI tools. The paper's title borrows one participant's description of the technology: “I look at it as the king of knowledge”. Participants used chatbots for writing emails and CVs, researching products and asking health questions, and they valued Be My AI's descriptions as more thorough than what sighted helpers usually offered.

AFB itself is not anti-AI. When it released its expert consensus report Empowering or Excluding in February 2025, Stephanie Enyart, the organisation's chief public policy and research officer, said the experts consulted had “shined a light on how AI can be a catalyst for access and inclusion.” Katsarakes, in the same essay that warns about hallucination, acknowledges that using tools such as Be My Eyes or Seeing AI “demonstrates clear potential for enhancing user autonomy.”

So the question is not whether these tools help. They do. The question is what happens when their help is treated as finished accessibility rather than as one layer within it.

The Verification Burden Falls on the User

What the research shows most consistently is that blind people who use AI description tools have become their own quality assurance department.

In a 2024 ASSETS paper titled Misfitting With AI, University of Michigan researchers Rahaf Alharbi, Pa Lor, Jaylin Herskovitz, Sarita Schoenebeck and Robin N. Brewer conducted a qualitative study with 26 blind people about how they verify and contest errors from AI visual assistance apps. They documented failures with complex document layouts, with languages other than English and with cultural artefacts. Participants had developed a repertoire of checks: testing the apps on familiar objects in low-stakes settings, drawing on non-visual knowledge, strategically involving sighted people and cross-referencing results between different apps and devices. The authors drew on the disability studies concept of “misfitting” to argue that these tools often carry assumptions that fail disabled users, and that disabled ways of knowing deserve recognition rather than replacement.

A diary study presented at CHI 2024 by Ricardo Gonzalez Penuela, Jazmin Collins, Shiri Azenkot and Cynthia Bennett tracked 16 blind and low vision participants using an AI scene description prototype for two weeks. The ratings were sobering. Average satisfaction came in at 2.76 out of 5. Average trust in the descriptions was 2.43 out of 4. The authors concluded that AI descriptions still needed significant improvement before they could deliver experiences that were both satisfying and trustworthy.

A 2025 CHI paper by Jingyi Xie, John M. Carroll and colleagues at Pennsylvania State University, which interviewed 14 visually impaired Be My AI users, adds texture. Four of its participants reported that Be My AI had described objects that were not there; one was told about an object behind her that did not exist. Eight reported inaccurate labels for gender or age. Several disliked the tool's habit of interpreting emotions and offering opinions. “Maybe I don't want it to editorialize, you know, maybe I just literally only want the facts of it,” one participant told the researchers. For live navigation, participants preferred human help over static photo descriptions, and they stressed that orientation and mobility skills remained essential. The tool, in their view, should augment those skills rather than replace them.

Chancey Fleet, the assistive technology coordinator at the New York Public Library's Andrew Heiskell Braille and Talking Book Library, has described a subtler cost. Speaking on the Electronic Frontier Foundation's How to Fix the Internet podcast in May 2024, she catalogued the house style of machine descriptions: “Any scene that contains a child or a dog is heartwarming.” “Any sunset or sunrise is vibrant.” “Anything with a couch and a lamp is calm or cozy.” The flattening matters, she argued, because adequacy discourages improvement: “if you are presented with what seems on a quick read, like good enough alt text, you're unlikely to do much labor to make it better, more nuanced, or more complete.” She also put the stakes plainly. When she asks an AI a question about her own world, “I'm going to have to live with the consequences of trusting that answer”.

Put these studies together and a pattern emerges. The capability is shared with everyone; the risk is carried by the individual. Each time a developer, publisher or institution decides that AI description is good enough, the job of verification is quietly transferred from the organisation that produced the content to the disabled person consuming it.

Overlays and the Business of Automated Compliance

Accessibility overlays are third-party scripts that sit on top of a website and attempt to detect and repair barriers automatically, often marketed with the language of artificial intelligence. They promise compliance with the Web Content Accessibility Guidelines (WCAG), the international technical standard, without anyone having to rebuild the site. Disabled users and accessibility professionals have been sceptical for years. In a January 2021 survey of 758 web accessibility practitioners by WebAIM, a non-profit based at Utah State University, 67 per cent rated overlays, plugins and widgets as “not at all or not very effective”. Among respondents with disabilities, the figure rose to 72 per cent, with only 2.4 per cent rating them very effective.

In January 2025, the US Federal Trade Commission turned that scepticism into enforcement. On 3 January it announced a proposed order requiring accessiBe, a company selling an AI-powered accessibility plug-in called accessWidget, to pay $1 million. The FTC alleged that accessiBe had misrepresented the tool's ability to make any website WCAG-compliant, and that it had formatted third-party articles and reviews to look like independent opinions without disclosing its material connections to them. The Commission voted 5-0 to accept the proposed consent order. Samuel Levine, then director of the FTC's Bureau of Consumer Protection, said: “Overstating a product's AI or other capabilities without adequate evidence is deceptive, and the FTC will act to stop it.” TechCrunch, reporting the action, noted that customers had also brought class action suits claiming the products did not work. The order was approved as final in April 2025.

AFB's Alexis Hubbard, writing in May 2025 under the title Missing a Human Touch, summed up the problem with automated fixes: “AI-powered accessibility overlays are just a band-aid solution for a bigger problem.” Because they do not change the source code of a site, she argued, they leave the underlying barriers intact. Her conclusion is worth quoting in full: “The only way to achieve an accessible and usable experience for all people is through people.”

Overlays show what AI-first accessibility looks like when it is driven by procurement rather than by users. A website owner buys the promise of compliance; a disabled visitor inherits the consequences, handed an interpretation instead of the thing itself.

When Governments Lean on the Promise of AI

The most striking recent example of AI's shortcomings being cited as a reason to delay accessibility comes not from a start-up but from the US Department of Justice.

In April 2024, the DOJ finalised a rule under Title II of the Americans with Disabilities Act requiring state and local governments to make their websites and mobile apps conform to WCAG 2.1 Level AA. Larger public entities, those serving populations of 50,000 or more, were originally due to comply in April 2026. Four days before that deadline, on 20 April 2026, the department published an interim final rule pushing compliance back by a year, to 26 April 2027 for larger entities and to 26 April 2028 for smaller ones and special districts. It took effect immediately, with public comments accepted until 22 June 2026. The technical standard itself was left unchanged.

Among the department's stated reasons was a striking admission about technology. “Advanced technology, such as generative AI, does not yet reliably automate the remediation of inaccessible content at scale,” the interim final rule says, “and staff resources and availability continue to pose significant challenges.” The rule also summarised correspondence from a member of Congress who argued that current technology, “including generative AI,” cannot reliably automate the remediation of complex science, technology, engineering and mathematics material at scale, and that human oversight is required.

It is an honest admission that AI cannot do the job alone. It also shows how the promise of automation can reshape policy. When the expected technological help did not arrive, the cost of the shortfall fell on disabled residents, who now wait at least another year for accessible public services.

Disability organisations were furious. Maria Town, president and chief executive of the American Association of People with Disabilities, called the delay “a profound disappointment and a serious setback for the disability community”, adding that “Years of notice have not been enough” and that the department was “rewarding inaction with more time.”

AFB's formal comments, submitted on 22 June 2026 and signed by Enyart, tackled the AI argument directly. “The absence of a fully automated accessibility solution does not justify extending the rule,” AFB wrote. It pointed out that generative AI can itself produce inaccessible content “because it is based on training data that contains accessibility barriers,” and that “Entities should not over rely on AI to produce content.” It also argued, drawing on its own research, that “AI adoption and effective use of AI is dependent on the accessibility of the platform.” In plain terms: “It doesn't matter how useful an AI-enabled service or tool is if the platform is inaccessible to blind users.”

That last line is the hinge of the entire debate. An AI tool is not a ramp into an inaccessible building. It is another door, and it only works if the corridor behind it has been built properly. AFB's comments also offered a constructive model: AI “can generate a passable first attempt at creating captions that a human can then edit.” Used that way, AI speeds up accessible design rather than replacing it.

AFB's research quantifies what is at stake. In a 2024 study cited in those comments, participants spent 55 per cent of their total task time dealing with digital barriers, and only 45 per cent on the task itself.

Across the Atlantic, the direction of travel is different. The European Accessibility Act, Directive (EU) 2019/882, began applying on 28 June 2025 to products and services including computers, smartphones, ATMs, e-books, e-commerce and banking. As Fredericka Argent of the law firm Covington & Burling noted that month, it reaches companies trading in the EU wherever they are based. The EAA is not an AI law, but it reinforces a principle that AI-first thinking tends to erode: the obligation sits with whoever builds and sells the product, not with the disabled person left to find a workaround.

Older Adults and the Gap Between Awareness and Use

The same dynamics extend to older adults, a group that overlaps heavily with disabled people and is often presented as a natural market for AI-powered help.

The latest AARP research, Tech Use and Adoption Keeps Surging Among Older Adults, written by Brittne Kakulla and published in December 2025, surveyed 3,838 US adults aged 50 and over between 9 September and 6 October 2025. It found that AI use in that age group had risen from 18 per cent in 2024 to 30 per cent in 2025. About half used or were interested in voice assistants such as Siri or Alexa. Two-thirds agreed that technology enriches life and makes ageing easier. Among adults aged 80 and over, agreement that technology enables a healthy life rose from 39 per cent to 46 per cent.

The same report records friction. For AI platforms such as ChatGPT or Copilot, 19 per cent used them, 21 per cent were interested and 59 per cent had no interest. Data privacy was the top barrier to adoption. Trust, privacy and data security were the top concerns for roughly half of the older adults who hesitated to adopt new technology, especially AI. And, in a line that could serve as the epigraph for this entire debate: “Three in 5 adults 50-plus say technology is not designed with their age in mind.”

Work from the MIT AgeLab sharpens the picture. A national survey of 1,008 American adults by Lauren Cerino, Joseph Coughlin and colleagues, published as a preliminary abstract in Innovation in Aging in December 2025, found that 73.7 per cent of respondents aged 60 and over were at least “a little” familiar with AI-enabled technologies. Only 28.3 per cent had used them. Some 34.7 per cent said they did not trust AI tools at all to give good advice, and trust scores declined with age.

Awareness, then, is not the main problem. Older people have heard of AI. The gap lies between hearing about a tool and being able, willing and confident enough to rely on it.

Research at Carnegie Mellon University points to why. In work presented at the ACM CHI 2026 conference in Barcelona and reported by the university in May 2026, Jodi Forlizzi, the Herbert A. Simon Professor in Computer Science and Human-Computer Interaction, and her colleagues found that older adults who use AI want it to recognise when they need plain information and when they need reassurance. A separate study based on 22 interviews identified shared values among older adults, including independence, curiosity, connection and continuity, expressed differently depending on where people lived. Older adults were open to AI when its purpose was clearly explained, especially for reminders, alerts and support at home. “There's always a mismatch between people's perceived abilities and what they can actually do,” Forlizzi said. “AI technology can be used to close this gap.”

That is the optimistic reading, and it deserves to be taken seriously. But it has a mirror image. If AI tools must be carefully explained and set up to earn trust, the people most likely to benefit, including those living alone or without technically confident relatives, are also the least likely to receive that support. AFB found that almost all its participants had learned to use AI on their own; only 9 per cent learned through an employer or online course. More disabled than nondisabled participants reported barriers in learning, most commonly inaccessible videos, hard-to-understand material and training that required a mouse.

The Northeastern study found similar problems in the AI products themselves. Blind participants encountered unlabelled buttons, missing headings and landmarks, no notification when a chatbot had finished responding, and clutter from adverts and sample prompts. The tools being promoted as accessibility solutions were not consistently accessible.

The Quiet Retirement of Predictable Tools

The sharpest version of the complaint concerns replacement: established, predictable tools being retired in favour of generative ones. In 2026, the most visible example is the end of Google Assistant on Android.

Google had originally planned to replace Assistant with Gemini, its generative AI assistant, during 2025. According to reporting by Matthias Bastian at The Decoder in August 2026, Google told users it would begin shutting Assistant down on Android and Wear OS from 4 September 2026, covering phones, tablets, watches, headphones and Android Auto. Once the rollout reaches a device, there is no way to switch back. Bastian raised exactly the concern at the centre of this article: that a probability-based language model may not match the reliability of a deterministic assistant for simple tasks such as smart home control and basic phone commands.

By late September, Mark Jansen at Android Police reported that the option to switch from Gemini back to Assistant had disappeared on the devices he checked, including a Pixel 10 Pro, and that smart home control was an area where Assistant still beat its replacement.

For most users, that is an inconvenience. For someone who cannot easily use a touchscreen, whether because of sight loss, limited dexterity or tremor, a voice assistant is not a convenience feature. It is how they turn on the lights, set a medication reminder or call for help. AFB's survey speaks to the stakes. Among users of voice-activated AI, 121 disabled participants described harms from mistakes, including texts sent with the wrong words and calls placed to unintended contacts, among them emergency services. “For many users, particularly disabled users, features such as voice dictation for text messaging are essential accessibility supports,” the report observes. “Failures or changes in these systems can result in concrete, disruptive, and sometimes harmful consequences.”

There is no evidence that Google set out to disadvantage disabled users, and Gemini brings generative capabilities that Assistant never had. The issue is structural. When a company replaces a predictable tool with a more capable but less predictable one, and removes the option to go back, the people with the fewest alternatives absorb the most risk. Capability is a fair trade for those who can fall back on other methods. Reliability is the whole point for those who cannot.

Is AI-First Accessibility Widening the Gap?

So, to the central question: if the people who would benefit most cannot find, set up or trust these tools, is AI-first accessibility widening the gap it claims to close?

The honest answer is that it depends on what “AI-first” means in practice.

Where AI is added on top of accessible foundations, the gap narrows. A blind person on a properly coded website who also has Be My AI or Seeing AI in their pocket has more independence than ever before. A deaf student who receives AI-generated captions that a human has corrected gets faster access than one who waits for manual transcription. These are genuine gains, and the AFB data confirms that most AI users say AI makes their lives easier.

Where AI replaces accessible foundations, the gap widens, and it widens along a predictable gradient. The confident, well-supported, technically fluent disabled user can verify outputs, switch between apps, ask a sighted friend and route around errors, exactly as the participants in the Michigan study described. The isolated, older, less confident or cognitively disabled user often cannot. The more a system depends on verification skills, the more it rewards people who already have them.

The evidence points to four ways in which substitution widens the divide. It shifts responsibility, transferring the work of accessibility from organisations to users who must detect errors they may have no way of seeing. It erodes incentives: as Fleet observed, good-enough descriptions discourage better ones, and the DOJ's interim rule shows the institutional version. It removes fallbacks, as predictable tools are retired. And it presupposes access to the tools themselves, when AFB and the Northeastern researchers both found that AI interfaces and training are often inaccessible.

None of this makes AI the villain. It makes the substitution the problem.

What Reliable Accessibility Would Require

If AI is going to close the accessibility gap rather than reshape it, the evidence points towards a set of concrete commitments.

Treat AI as a layer, never a foundation. Accessible design, meaning semantic code, human-authored alt text for meaningful images, accessible documents and keyboard-operable interfaces, must remain the baseline. AI description and remediation should sit on top of it. Katsarakes put it in a single sentence: “Let AI suggest or assist, but don't let it replace human intent.”

Keep the deterministic path available. When companies introduce generative assistants, they should keep predictable modes for core tasks such as timers, calls, reminders and device control, and they should not remove the option to revert until the replacement has been shown to match the original's reliability for disabled users specifically. A one-way migration is a design choice, not a technical necessity.

Build verification into the product. The Michigan and Penn State studies show users building their own verification systems from scratch. Products should do it for them: signalling uncertainty, separating what the model read from what it inferred, offering a factual mode without editorial colour, and making escalation to a human easy. Be My Eyes shows that a live human route can sit alongside AI.

Draw hard lines around high-stakes tasks. Medication instructions, financial documents, legal forms and navigation are not areas where a probabilistic description should ever be the only source. AFB's recommendations call for developers to “Improve the accuracy of AI outputs and tell users how accurate those outputs are.” Where accuracy cannot be assured, the interface should say so plainly.

Hold vendors to their claims. The FTC's accessiBe order established that overstating AI accessibility capabilities without evidence is deceptive. Regulators elsewhere should apply the same standard, and procurement teams should demand evidence of real-world performance with disabled users, not marketing claims of automated compliance.

Do not let automation set the deadline. Legal accessibility obligations should be pegged to what disabled people need, not to the pace of AI development. The European Accessibility Act's focus on the duties of manufacturers and service providers is a better model than one that treats immature automation as grounds for delay.

Fund support, not just software. Older adults are aware of AI but hesitant to use it, and CMU's research suggests they engage when the purpose is clearly explained. That explanation has to come from libraries, disability organisations and accessible community training. AFB's finding that only 9 per cent of AI users learned through formal routes suggests a large and fixable gap.

Include disabled people in testing and design. AFB's expert consensus report called for AI to focus “on expanding access and inclusion, not just on avoiding harm.” That requires disabled people in the room when products are designed, tested and retired, not consulted after launch.

What Is Lost When Reliability Goes

So what is lost when reliability is sacrificed for capability?

Part of the answer is practical: time, money, safety, and the occasional tube of medication with the wrong instructions. Participants in AFB's 2024 study already spent more than half their task time fighting digital barriers.

But the deeper loss is harder to measure. Independence is not just the ability to do something without help. It is the ability to rely on what you are told. A screen reader that occasionally mispronounces a name still gives its user a stable relationship with the text. A description tool that sometimes invents an object behind you, or turns a cream into tablets, asks its user to hold every answer at a slight distance. That is a different kind of life: one in which assistance is always provisional and confidence always has to be earned again.

Generative AI has given many disabled people extraordinary new reach, and nobody should want to take that away. The challenge for the companies building these tools, and the governments regulating them, is to add that reach without removing the floor. AI can help the people who build accessible things work faster and reach further. It cannot be allowed to replace the obligation they carry.

---

References and Sources

1. American Foundation for the Blind. “The AI Quagmire: Benefits, Risks, and User Aspirations Through a Disability Lens.” AFB Artificial Intelligence Series, 2026. Sections “Executive Summary” and “Harms of AI in Various Contexts.” https://afb.org/research-and-initiatives/ai-series/ai-quagmire/executive-summary and https://afb.org/research-and-initiatives/ai-series/ai-quagmire/harms-ai-various-contexts
2. Katsarakes, Alex. “Beyond Alt Text: Rethinking Visual Description in the Age of AI.” American Foundation for the Blind blog, 16 October 2025. https://afb.org/blog/entry/alt-text-age-ai
3. Hubbard, Alexis. “Missing a Human Touch: Why Automated Testing and AI Don't 'Solve' Digital Accessibility… Yet.” American Foundation for the Blind blog, 13 May 2025. https://afb.org/blog/entry/missing-human-touch-1
4. Enyart, Stephanie, American Foundation for the Blind. “June 2026 Comments on DOJ Extension of Web Accessibility Compliance Dates” (RIN 1190-AA82, Docket CRT150). 22 June 2026. https://afb.org/research-and-initiatives/public-policy/advocacy-letters-comments-statements/comments/june-2026-doj-extension-web-compliance
5. American Foundation for the Blind. “AFB Report Spotlights Impact of AI for Disabled People” (press release on Empowering or Excluding: Expert Insights on Inclusive Artificial Intelligence for People with Disabilities). 6 February 2025. https://afb.org/news-publications/press-room/press-release-archive/press-release-2025/afb-report-spotlights-impact-AI-disabled-people
6. US Department of Justice, Civil Rights Division. “Extension of Compliance Dates for Nondiscrimination on the Basis of Disability; Accessibility of Web Information and Services of State and Local Government Entities.” Interim final rule, Federal Register Vol. 91, No. 75, 20 April 2026. https://www.federalregister.gov/documents/2026/04/20/2026-07663/extension-of-compliance-dates-for-nondiscrimination-on-the-basis-of-disability-accessibility-of-web (PDF: https://www.ada.gov/assets/pdfs/2026-ifr.pdf)
7. Duane Morris LLP. “DOJ Extends ADA Title II Digital Accessibility Deadlines by One Year.” Client alert, 23 April 2026. https://www.duanemorris.com/alerts/doj_extends_ada_title_ii_digital_accessibility_deadlines_one_year_0426.html
8. American Association of People with Disabilities. “AAPD Statement on DOJ Title II Web Rule Interim Final Rule.” 20 April 2026. https://www.aapd.com/aapd-statement-title-ii-doj-web-rule-ifr/
9. Kakulla, Brittne. “Tech Use and Adoption Keeps Surging Among Older Adults” (2026 Tech Trends). AARP Research, December 2025. https://www.aarp.org/pri/topics/technology/internet-media-devices/2026-technology-trends-older-adults/
10. Cerino, Lauren; Felts, Adam; Balmuth, Alexa; Brennan, Taylor; Lee, Chaiwoo; Coughlin, Joseph. “Older Adults' Perspectives on Artificial Intelligence as a Source of Advice.” Innovation in Aging, 9 (Suppl 2), igaf122.1371, 31 December 2025. https://pmc.ncbi.nlm.nih.gov/articles/PMC12761853/
11. Sapio, Amanda. “How Do Boomers Really Feel About AI?” Carnegie Mellon University School of Computer Science News, 28 May 2026. https://www.cs.cmu.edu/news/2026/older-adults-ai
12. US Federal Trade Commission. “FTC Order Requires Online Marketer to Pay $1 Million for Deceptive Claims that its AI Product Could Make Websites Compliant with Accessibility Guidelines.” Press release, 3 January 2025. https://www.ftc.gov/news-events/news/press-releases/2025/01/ftc-order-requires-online-marketer-pay-1-million-deceptive-claims-its-ai-product-could-make-websites-compliant
13. Wiggers, Kyle. “FTC orders AI accessibility startup accessiBe to pay $1M for misleading advertising.” TechCrunch, 3 January 2025. https://techcrunch.com/2025/01/03/ftc-orders-ai-accessibility-startup-accessibe-to-pay-1m-for-misleading-advertising/
14. WebAIM. “Survey of Web Accessibility Practitioners #3 Results.” January 2021. https://webaim.org/projects/practitionersurvey3/
15. Alharbi, Rahaf; Lor, Pa; Herskovitz, Jaylin; Schoenebeck, Sarita; Brewer, Robin N. “Misfitting With AI: How Blind People Verify and Contest AI Errors.” Proceedings of ASSETS '24, ACM, 2024. DOI: 10.1145/3663548.3675659. https://arxiv.org/abs/2408.06546
16. Gonzalez Penuela, Ricardo E.; Collins, Jazmin; Azenkot, Shiri; Bennett, Cynthia. “Investigating Use Cases of AI-Powered Scene Description Applications for Blind and Low Vision People.” Proceedings of CHI '24, ACM, 2024. https://arxiv.org/abs/2403.15604
17. Adnin, Rudaiba; Das, Maitraye. “'I look at it as the king of knowledge': How Blind People Use and Understand Generative AI Tools.” Proceedings of ASSETS '24, ACM, 2024. DOI: 10.1145/3663548.3675631. https://maitraye.github.io/files/papers/BLV_GenAI_ASSETS24.pdf
18. Xie, Jingyi; Yu, Rui; Zhang, He; Billah, Syed Masum; Lee, Sooyeon; Carroll, John M. “Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Models.” Proceedings of CHI '25, ACM, 25 April 2025. DOI: 10.1145/3706598.3714210. https://pmc.ncbi.nlm.nih.gov/articles/PMC12338113/
19. Cohn, Cindy; Kelley, Jason (hosts), with Chancey Fleet. “Podcast Episode: Building a Tactile Internet.” Electronic Frontier Foundation, How to Fix the Internet, 7 May 2024. https://www.eff.org/deeplinks/2024/05/podcast-episode-building-tactile-internet
20. Be My Eyes. “Be My Eyes Reaches 1 Million Blind and Low-Vision Users and 10 Million Volunteers.” 12 March 2026. https://www.bemyeyes.com/news/be-my-eyes-reaches-1-million-blind-and-low-vision-users-and-10-million-volunteers/
21. Shaikh, Saqib. “Seeing AI App Launches on Android, Including New and Updated Features and New Languages.” Microsoft Accessibility Blog, 4 December 2023. https://blogs.microsoft.com/accessibility/seeing-ai-app-launches-on-android-including-new-and-updated-features-and-new-languages/
22. Bastian, Matthias. “Google will shut down Google Assistant starting September 2026 as Gemini takes over on Android and Wear OS.” The Decoder, 5 August 2026. https://the-decoder.com/google-will-shut-down-google-assistant-starting-september-2026-as-gemini-takes-over-on-android-and-wear-os/
23. Jansen, Mark. “Google Assistant is officially gone, and people aren't happy about it.” Android Police, 29 September 2026. https://www.androidpolice.com/the-google-assistant-kill-off-has-arrived/
24. Argent, Fredericka. “European Accessibility Act: June 2025 Deadline Has Arrived.” Covington & Burling, Inside Global Tech, 10 June 2025. https://www.insideglobaltech.com/2025/06/10/european-accessibility-act-june-2025-deadline-has-arrived/ (see also European Commission, “European Accessibility Act,” https://commission.europa.eu/strategy-and-policy/policies/justice-and-fundamental-rights/disability/union-equality-strategy-rights-persons-disabilities-2021-2030/european-accessibility-act_en)

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

A red car rolls towards a junction. A pedestrian steps into the road. The car turns, hits the pedestrian, who goes down, gets up and walks over to the driver. The whole thing lasts 25 seconds, is rendered in the engine of Grand Theft Auto V, and is about as simple as evidence gets. One thing happens in it, and that thing is a collision.

When researchers asked two consumer AI systems to write factual, neutral summaries of that clip and a near-identical twin, 19 of the 20 summaries they got back failed to mention that the car hit the pedestrian.

That finding comes from a paper posted to arXiv on 23 September 2026, “AI-Enabled Human Memory Manipulation: Misleading AI-Generated Summaries Distort Human Memory”, by Mattea Sim of Georgetown University, Yael Eiger of the University of Washington and Tadayoshi Kohno of Georgetown. It was picked up over the following days by The Independent, Neuroscience News and a string of science outlets, and it is due to be presented at the AAAI/ACM Conference on AI, Ethics and Society in October. The headline numbers travelled fast. The machines were bad at summarising. And when people read a summary that got a detail wrong, their memory of what they had seen with their own eyes bent to fit the text.

That second finding is the one worth sitting with. Copilot writes up the meeting you sat through. Your phone condenses the news alerts you swiped past. Police officers in several American states file reports drafted by software from body camera audio. Each tool is sold as a memory aid. The study suggests that when the aid is wrong, it does not merely fail to help. It can overwrite.

So what happens to a person's own memory when the machine's recap of their experience is wrong? And when a summary quietly edits the past someone believes they lived, who is supposed to catch it: the reader, the company that sold the recap, or, as things currently stand, more or less nobody?

Twenty Summaries and One Missing Collision

The researchers used two short animated clips that already had a history in memory research. Both show a red saloon approaching an intersection and striking a pedestrian. The only meaningful difference is the road sign at the junction: in one video it is a stop sign, in the other a yield sign (the American equivalent of a give way sign). The team fed each clip to two models, OpenAI's ChatGPT-5.5 and Google's Gemini 2.5 Flash-Lite, five times per model per video, using a single standardised prompt. The instruction asked for a factual and neutral summary of at least 300 words, with no speculation and no invented facts. That gave them 20 summaries to analyse.

Every one of the 20 contained mistakes, somewhere between seven and 21 per summary. All of them left things out. According to the paper, 65 per cent contained what the authors classed as critical inaccuracies, 90 per cent contained non-critical inaccuracies, and 60 per cent added material that simply was not there. The summaries hallucinated extra pedestrians and extra vehicles and described erratic driving the video did not show. One stated that the pedestrian slightly repositioned but did not enter the roadway, when the pedestrian plainly walks into the road.

Averaged across all prompts and both models, the summaries omitted 51.6 per cent of the central details of the event. And in 95 per cent of cases, the summary did not mention the central detail at all: the car colliding with the pedestrian.

Summaries are supposed to be lossy, but their entire job is to keep what matters and drop what does not. These models did the reverse, producing long, fluent paragraphs about a scene in which the one thing that happened did not happen.

“I was struck by how bad the summaries were, even at this stage in AI development,” Eiger said in the coverage published by Neuroscience News.

Some caution is due. This was a small corpus: two short animated clips, two models, one prompt, 20 outputs. The authors say so plainly in their limitations section, acknowledging that one standardised prompt and a modest number of summaries limit how far the results generalise. Models change month to month, so nobody should read these figures as a stable error rate for all AI video summarisation. What they do show is that two widely available consumer systems, given a clear and simple task, were confidently and substantially wrong about the most important fact in front of them.

Fifty Years of Stop Signs and Yield Signs

The second half of the study is where the story turns from software quality to human cognition, and where it plugs directly into one of the most studied findings in psychology.

In 1974, Elizabeth Loftus, then at the University of Washington, and her colleague John Palmer showed people films of car accidents and asked them how fast the cars were going. The trick was in the verb. Asked how fast the cars were going when they “smashed” into each other, participants gave higher estimates of speed than those asked about cars that “collided”, “bumped”, “hit” or “contacted”. A week later, people who had heard “smashed” were more likely to say yes when asked whether they had seen broken glass. There was no broken glass in the film. The paper became one of the founding texts of what is now called the misinformation effect.

Four years later, Loftus, David Miller and Helen Burns published a series of experiments in the Journal of Experimental Psychology that will look very familiar to anyone who has just read the Georgetown and Washington paper. Across five experiments and a pilot, 1,242 participants watched a sequence of slides showing an accident between a car and a pedestrian at a junction. Afterwards some were fed information consistent with what they had seen, and some were fed misleading information: a stop sign became a yield sign, or the other way round. Misleading information made people less accurate on later recognition tests, and it had a larger effect when introduced just before the test than immediately after the event. The authors concluded that information a witness encounters after an event, consistent or misleading, gets integrated into the witness's memory of it.

The new study is, in effect, that experiment rerun for the age of the chatbot, split into two sessions between 24 and 48 hours apart. In the first, people watched one of the two animated clips. In the second, they read a summary of what they had seen, then answered a 12-item recognition test in which each question offered two choices. The critical question asked whether the car had approached a stop sign or a yield sign.

Here is a crucial design detail that some coverage skated past. The misleading summaries were not raw machine errors. The researchers took AI-generated summaries and edited them, removing sentences describing the details that differed between the two clips and splicing in AI-generated text from other outputs, so that participants who had seen a stop sign read about a yield sign and vice versa. In other words, the memory experiment tested what happens when an AI-style summary contains a specific, plausible error. It did not test whether the particular errors the models spontaneously made would distort memory in the same way.

That caveat matters, but it cuts less deeply than it seems. The first half of the study showed that errors of exactly this kind are routine. The second half showed what they do once they land. Of the 331 participants who completed both sessions (328 after attention checks), those who read a summary consistent with the video answered the critical sign question correctly 83.6 per cent of the time. Those who read a misleading summary got it right 44.8 per cent of the time. On a two-choice question, that is worse than a coin toss. The statistics were not subtle: the probability of a gap that size arising by chance was below one in a thousand.

Meanwhile participants remembered other details well. On a non-critical question about whether the scene took place in daytime or at night, accuracy was 96.6 per cent. People had not forgotten the video. They had swapped one piece of it for the version they had been told.

“AI is a new method of delivering misinformation, and it has the potential to create these false memories for people who are reading that information,” Sim said in the Georgetown announcement.

Why Knowing It Was a Machine Did Not Help

Participants were randomly told that the summary they read had been written either by an AI or by a human. In reality all of the summaries were AI-generated. If a machine label worked as a kind of cognitive inoculation, you would expect people who knew they were reading AI output to discount its claims and lean more on their own memory.

They did not. Among those who read a misleading summary labelled as AI-generated, 46.3 per cent answered the critical question correctly. Among those who read the same kind of summary labelled as human-written, 43.4 per cent did. The difference was not statistically significant. The researchers also measured participants' trust in AI and how much they used it, and neither made a meaningful difference to how susceptible they were.

This is consistent with what the misinformation literature has long suggested about warnings. In a 2005 review in Learning and Memory, looking back on three decades of this research, Loftus wrote that warnings about the potential for misinformation sometimes work to inhibit its damaging effects, but only under limited circumstances. The Georgetown and Washington authors reach a similar conclusion. They suggest that noticing discrepancies between the original event and later information can reduce the impact of misinformation, and float warning people about potential discrepancies before they read an AI summary. But they also note that simply knowing the source was AI provided no protection in their data.

Related evidence points the same way. In 2024 a team at the MIT Media Lab, with Loftus among the co-authors, had 200 participants watch a crime video and then answer questions, five of them misleading, via a survey, a scripted chatbot or a generative chatbot built on a large language model. The generative chatbot induced more than three times as many immediate false memories as the control condition, and a week later those false memories persisted, held with higher confidence than in the control group.

A fluent system can introduce errors into your memory. Telling you it is a fluent system does not reliably stop that. And the errors can outlast the conversation that planted them.

The Meeting You Attended, as Told by Copilot

Move from a laboratory junction to an office, and the stakes change shape without shrinking.

Automated meeting recaps are now a standard feature across the major collaboration platforms. Microsoft's intelligent recap in Teams, available with Teams Premium or a Copilot licence, generates AI summaries and follow-up tasks, marks when your name was mentioned, organises the discussion into chapters, and tells you who said what. Microsoft's own support documentation is candid about the risk: “AI-generated content is based on the event transcript and is powered by GPT and other AI models. The AI-generated content may occasionally be inaccurate, incomplete, or inappropriate.”

That is a reasonable disclosure. But if an AI label did not protect participants from absorbing a planted error about a road sign, there is little reason to think small print will protect an employee from absorbing an error about who agreed to what in a budget meeting.

The people most likely to rely on a recap are those who were least attentive: double-booked, multitasking, half-listening while answering email. They are witnesses with weak original traces, exactly where post-event information has most room to take hold. And if a recap is reread just before someone needs to remember, the 1978 finding suggests that is exactly when misleading information bites hardest. A recap consulted before a Friday dispute is not merely a record of Monday's meeting. By Friday, it may be the meeting.

A change Microsoft has scheduled for this autumn sharpens the point. In a Microsoft 365 message centre notice first published in April 2026, the company announced a “recap without saving transcript” option, which lets Copilot users generate an AI meeting summary from live meeting context without retaining a recording or transcript. The notice says the feature addresses scenarios where organisations want AI-generated recaps “but have policies that limit or prohibit the retention of transcripts or recordings”, and it advises administrators to review their AI governance policies to decide whether it fits their compliance posture. Rollout is scheduled for October 2026.

There are sound privacy reasons for not keeping transcripts. But if the summary is the only artefact that survives the meeting, there is nothing to check it against except the attendees' memories, and the study is precisely about what happens to those memories after people read a summary that is wrong.

Even the transcript layer is not safe ground. In October 2024 an Associated Press investigation reported that OpenAI's Whisper speech recognition model was prone to inventing chunks of text, and that more than 30,000 clinicians and 40 health systems were using a Whisper-based tool from Nabla to transcribe and summarise patient visits, despite OpenAI's warnings against use in high-risk domains. The layer that turns a live event into text can introduce words nobody said.

News You Only Half Read

For most people, the summary that shapes memory is not a meeting recap but a news digest glimpsed on a lock screen.

In December 2024 and January 2025, Apple Intelligence's notification summaries produced a run of false headlines attributed to news organisations. One summary of BBC News alerts suggested that Luigi Mangione, the man charged over the killing of the UnitedHealthcare executive Brian Thompson, had shot himself. He had not. Other summaries wrongly claimed, under the BBC's name, that the darts player Luke Littler had won the PDC World Championship before the final had been played, and that Rafael Nadal had come out as gay. A summary of New York Times alerts said Israel's prime minister, Benjamin Netanyahu, had been arrested. The BBC complained. In January 2025 Apple paused notification summaries for news and entertainment apps in a software update, and said summaries would be shown in italics to distinguish them from ordinary notifications. In July 2025 the iOS 26 beta brought news summaries back, this time with a warning beneath the setting: “Summarization may change the meaning of the original headlines. Verify information.” Given the Georgetown and Washington finding that knowing a summary came from AI did not protect memory, a label is a thin defence.

Those were spectacular, checkable errors. The quieter problem is the everyday rate. In February 2025 the BBC published research in which its journalists put 100 news questions to ChatGPT, Microsoft Copilot, Google Gemini and Perplexity, asking them to use BBC sources. It found that 51 per cent of answers had significant issues, 19 per cent of answers citing BBC content introduced factual errors such as wrong dates and figures, and 13 per cent of quotations attributed to BBC articles had been altered or did not exist in the cited article. In October 2025 a far larger follow-up, coordinated by the European Broadcasting Union and led by the BBC, assessed around 3,000 responses across 14 languages and found that 45 per cent of AI answers about news had at least one significant issue, with sourcing the most common problem.

Now put those error rates next to the audience. The Reuters Institute's Digital News Report 2026, published in June, found that weekly use of AI chatbots for news had risen to 10 per cent of people across the markets it surveys, and to 17 per cent among 18 to 24 year olds. Getting summaries was among the most common uses, cited by about a third of those using chatbots for news, and only a small fraction of users regularly clicked through to the original source.

That last number is the hinge. The traditional corrective for a bad summary is the source it summarised, but the whole value proposition of a summary is that you will not open the source. A reader who half-reads an AI digest of a court case and never opens the report carries a version of events that, on this evidence, will not simply sit alongside the truth. For memory's purposes, it becomes what they recall.

Google's AI Overviews were mocked in May 2024 for suggesting glue on pizza and eating rocks, and Liz Reid, Google's head of Search, acknowledged that some answers had drawn on satirical or joke content. Those errors were funny because they were absurd. The errors that reshape memory are not absurd. They are a yield sign where there was a stop sign.

Police Reports and the Officer in the Loop

The authors' main concern is policing, where the question of who checks the machine is most concrete. Axon, the largest supplier of police body cameras in the United States, sells a product called Draft One that uses generative AI to draft police report narratives from body-worn camera audio. The officer is expected to review and edit the draft before submitting it. In July 2025, the Electronic Frontier Foundation published an investigation arguing that the product was designed in a way that frustrated auditing. According to the EFF, Draft One does not save the draft it generates or subsequent edited versions, so once an officer exports the final report there is no record of which parts were written by the AI and which by the officer.

Put that alongside the new study. The standard defence of tools like this is the human in the loop: the officer was there, knows what happened, and will correct the machine. The paper's authors address exactly that defence. “Though 'humans-in-the-loop' are often expected to correct for AI's mistakes,” they write, “our work suggests human memory can instead be distorted by these mistakes.”

The logic is brutal. The officer is the check on the draft, but reading the draft is precisely the kind of post-event exposure that alters memory. If it says the pedestrian stepped off the kerb against the signal, the reviewing officer may come to remember it that way. The check has been contaminated by the thing it was meant to check, and if the draft is deleted, nobody can reconstruct which version came first.

In the Georgetown announcement, Eiger said it worried her that police departments may be using video summarisation tools without rigorous testing, and without an awareness of how wrong AI-generated summaries could be.

Legislators in two American states have begun to respond. Utah's SB 180, which took effect in May 2025, requires police reports produced wholly or partly with generative AI to carry a disclaimer and requires the author to certify that the report was checked for accuracy. California went further. SB 524, signed by Governor Gavin Newsom in October 2025 and in force since 1 January 2026, requires disclosure of AI use on reports, requires agencies to retain the first draft for as long as the final report is kept, and bars vendors from selling or sharing information police provide to the AI. As the EFF put it, the draft retention requirement means judges, defence lawyers or auditors can see which portions of the final report were written by the officer and which by the computer.

Notice what these rules assume. Utah makes the officer answerable for accuracy, yet the officer's memory is one of the things the draft may have altered. California's retention rule is more robust because it does not rely on memory at all: it preserves an artefact that can be compared against the footage. That is a useful template elsewhere.

Records That Outlive the Room

Outside policing, the law has not caught up with the idea that a summary can shape the memories of the people it describes. In the UK, meeting summaries that name or describe identifiable people are personal data, and Article 16 of the UK GDPR gives individuals the right to have inaccurate personal data about them rectified, and incomplete data completed, including by way of a supplementary statement. The Information Commissioner's Office guidance says organisations should take reasonable steps to satisfy themselves that data is accurate when a rectification request is made, taking into account the individual's arguments and evidence. An employee who finds that a Copilot recap attributes to them a commitment they never made has, in principle, a route to have it corrected.

In practice that route depends on the employee noticing the error and having something to prove it with. If the transcript was never kept, the evidence is two sets of memories, one of which may have been rewritten by the summary itself. The right was designed for a world in which records and memories were separate things that could be compared.

Grievances and disciplinary hearings routinely rely on contemporaneous notes. An AI recap looks contemporaneous, with a timestamp and a corporate logo, but it carries no indication of which sentences reflect what was said and which are the model's reconstruction.

What the Law Currently Asks of Whom

The European Union has come closest to allocating responsibility, though not for this specific harm.

Article 50 of the EU AI Act, whose core transparency obligations began applying on 2 August 2026, requires providers of AI systems that generate synthetic text, audio, images or video to ensure outputs are marked in a machine-readable format and detectable as artificially generated. It also requires deployers of AI systems that generate or manipulate text “published with the purpose of informing the public on matters of public interest” to disclose that the text was artificially generated. The Digital Omnibus, which pushed back the Act's high-risk obligations, left that date alone, although providers of generative systems already on the market before 2 August 2026 have until 2 December 2026 to meet the machine-readable marking requirement. The disclosure obligation falls away where the content “has undergone a process of human review or editorial control and where a natural or legal person holds editorial responsibility for the publication”.

That exception is revealing. It makes editorial responsibility, a named person or organisation standing behind the text, the alternative to disclosure. It does not make disclosure alone sufficient to discharge responsibility, and the new study suggests it should not. A label did not protect participants. A person accountable for checking the text before it reached them is a different kind of safeguard.

Liability is shifting too. The EU's revised Product Liability Directive, which member states must transpose by 9 December 2026, explicitly treats software, including AI systems, as a product for the purposes of no-fault liability, and eases the burden of proof for claimants dealing with technically complex systems. Its idea of damage centres on death, personal injury including medically recognised psychological harm, damage to property, and the destruction or corruption of data. A distorted memory of a meeting is unlikely to count. But the principle, that suppliers of software are answerable for defects in what they put on the market regardless of whether they were negligent, is a shift in the default assumption about who carries the risk.

In the United States, the picture is patchier. Beyond the Utah and California police laws, much of the current legal pressure on meeting assistants concerns consent rather than accuracy. In August 2025 a proposed class action, Brewer v. Otter.ai, was filed in federal court in California alleging that Otter's notetaker recorded conversations of people who had not consented, including non-users, and used the data to train its models. In October 2025 it was consolidated with similar suits as In re Otter.AI Privacy Litigation, and on 13 August 2026 Judge Eumi K. Lee refused to dismiss the core claims, allowing wiretap claims under federal and California law to proceed on the basis that Otter, by using recordings to develop its own models, could be treated as a third-party eavesdropper. The case illustrates where the legal system's attention currently sits: on whether the machine should have been listening, not on whether what it wrote afterwards was true.

Why the Reader Cannot Carry This Alone

The easy answer to who should check the summary is the person reading it. That is what the disclaimers say, and what “human in the loop” means in most enterprise AI policies.

The study makes that answer hard to defend, for three reasons.

The first is that the checking mechanism is compromised by the thing being checked. Memory is the natural reference for testing a summary of an event you attended, but reading the summary changes the memory. Not every time, and not for everyone, but often enough that “check it against what you remember” is not a reliable safeguard.

The second is that the people most exposed to summaries are the people least equipped to check them. Summaries are used by people who did not attend, did not pay attention or did not read the original. That is the point of them. Asking those users to verify the summary against a source they were trying to avoid is asking them to do the work the product was meant to save.

The third is that confidence in the tool reduces checking. In a survey of 319 knowledge workers presented at the CHI conference in 2025, researchers at Microsoft Research and Carnegie Mellon University found that higher confidence in generative AI was associated with less critical thinking, while higher self-confidence was associated with more. The more reliable a tool seems, the less people scrutinise it, and a tool that is right most of the time trains exactly the complacency that lets its occasional errors through.

None of this means readers bear no responsibility. Attendees can take their own notes, readers can click through, officers can write their own account first. But a design that depends on users consistently doing the things the product was designed to spare them is not a design with a human safeguard. It is a design with a human scapegoat.

What Suppliers Could Actually Owe

If the burden cannot sit only with readers, the obvious alternative is the organisations that build and deploy the summarisers. What would taking that burden seriously look like?

It would start with measurement. The authors call for more technical audits of summarisation tools and for examining real-world uses such as body camera footage. Vendors selling summarisation for high-stakes settings could publish error rates on representative material, broken down by type: omissions of central events, substitutions, additions. A summariser that routinely omits the central event of a clip is not ready to summarise evidence.

It would continue with provenance. California's approach to police reports offers a model: keep the draft, keep the source, make it possible to see which words came from the machine. For meeting recaps, that could mean retaining source audio or transcripts for a defined period when a summary is created, or linking each sentence of a summary to the moment in the recording it draws on. Microsoft's recap already offers timeline markers and speaker navigation. Anchoring every claim to its evidence would make checking something a reader can do in seconds rather than a reconstruction from memory.

It would include friction in the right places. The study's authors suggest that noticing discrepancies helps, and that warnings delivered before reading might help. That points towards designs that ask users to recall something before showing them the summary, or that flag which parts of a recap are high-stakes (decisions, attributions, commitments) and invite confirmation from the people named. It points away from designs that deliver a finished, authoritative paragraph and a disclaimer.

And it would include restraint. The authors raise a question most product roadmaps do not: “a deeper question to consider is whether AI should be used in these contexts at all.” For some uses, such as drafting a police narrative from body camera audio before the officer has written their own account, the answer may be that the order of operations matters more than the quality of the model. Human recollection first, machine summary second, and a record of both.

Organisations that deploy summaries carry a share too. An employer that switches on recaps without transcripts and relies on them in a disciplinary process has made choices about evidence. A platform that pushes machine summaries of journalism to millions of lock screens has made choices about what its audience will remember. The EU's editorial responsibility test is a good instinct: somebody specific should be answerable for what the summary says.

Keeping Hold of the Afternoon You Lived

There is a version of this story that is merely about quality control: the mistakes will get rarer and the problem will shrink. That may be partly true. But the study is not really about how often summaries are wrong. It is about what happens when they are. A human memory is not a hard drive that a summary sits beside. It is a reconstruction, rebuilt each time it is recalled, and fifty years of research have shown that the materials used in the rebuild include whatever we have been told since. For most of that history, the people doing the telling were other humans: lawyers asking leading questions, police officers suggesting details, newspapers and neighbours. They could be cross-examined. They could be blamed.

The summary is a new kind of teller, present at every meeting and every alert, speaking in a voice that sounds neutral and thorough. When it drops the collision from the crash, or puts a yield sign where a stop sign stood, it does not only produce a bad document. It produces people who remember a slightly different afternoon from the one they lived.

Right now, the burden of noticing falls on the people least able to notice, and responsibility for the error is spread so thinly that it lands on almost nobody. The disclaimers say the content may be inaccurate. The users are told to check. The checking, it turns out, runs through a memory the summary has already touched.

That arrangement looks unlikely to survive contact with a courtroom for long. The better move is to settle the question first: keep the source, mark the machine's words, measure the errors, and put a named person behind any summary that is going to be treated as the record. The past is hard enough to hold on to without software editing it for us.

---

References and Sources

1. Sim, Mattea, Yael Eiger and Tadayoshi Kohno. “AI-Enabled Human Memory Manipulation: Misleading AI-Generated Summaries Distort Human Memory.” arXiv:2609.28820v1, submitted 23 September 2026; to be presented at the Ninth AAAI/ACM Conference on AI, Ethics and Society (AIES), October 2026. https://arxiv.org/abs/2609.28820
2. Soong, Kelyn. “New Research Finds That Misleading AI-Generated Summaries Can Distort Human Memory.” Georgetown University College of Arts and Sciences, 25 September 2026. https://college.georgetown.edu/news-story/new-research-finds-that-misleading-ai-generated-summaries-can-distort-human-memory/
3. Neuroscience News. “Flawed AI Summaries Distort Eyewitness Memory.” 27 September 2026. https://neurosciencenews.com/ai-false-memory-llm-31261/
4. The Independent. “AI summaries can warp your memory, study warns.” September 2026, as syndicated via inkl. https://www.inkl.com/news/ai-summaries-can-warp-your-memory-study-warns
5. Loftus, Elizabeth F. and John C. Palmer. “Reconstruction of automobile destruction: An example of the interaction between language and memory.” Journal of Verbal Learning and Verbal Behavior, volume 13, issue 5, 1974, pages 585 to 589. https://www.sciencedirect.com/science/article/abs/pii/S0022537174800113
6. Loftus, Elizabeth F., David G. Miller and Helen J. Burns. “Semantic integration of verbal information into a visual memory.” Journal of Experimental Psychology: Human Learning and Memory, volume 4, issue 1, 1978, pages 19 to 31. https://pubmed.ncbi.nlm.nih.gov/621467/
7. Loftus, Elizabeth F. “Planting misinformation in the human mind: A 30-year investigation of the malleability of memory.” Learning and Memory, volume 12, 2005, pages 361 to 366. DOI 10.1101/lm.94705. https://learnmem.cshlp.org/content/12/4/361.full
8. Chan, Samantha, Pat Pataranutaporn, Aditya Suri, Wazeer Zulfikar, Pattie Maes and Elizabeth F. Loftus. “Conversational AI Powered by Large Language Models Amplifies False Memories in Witness Interviews.” MIT Media Lab, arXiv:2408.04681, August 2024. https://arxiv.org/abs/2408.04681
9. Microsoft Support. “Recap in Microsoft Teams.” Accessed 29 September 2026. https://support.microsoft.com/en-us/teams/meetings/recap-in-microsoft-teams
10. Microsoft 365 Message Center. “MC1275312: Microsoft Teams: AI meeting recap without transcript to meet compliance policies.” Published 9 April 2026, archived at Merill Fernando's Message Center Archive. https://mc.merill.net/message/MC1275312
11. Burke, Garance and Hilke Schellmann. “Researchers say an AI-powered transcription tool used in hospitals invents things no one ever said.” Associated Press, 26 October 2024; republished by Fortune and The Columbian. https://fortune.com/2024/10/26/openai-transcription-tool-whisper-hallucination-rate-ai-tools-hospitals-patients-doctors
12. TechCrunch. “Apple pauses AI notification summaries for news after generating false alerts.” 16 January 2025; see also Axios, “Apple pauses AI-generated news alerts after fake headline notifications”, 17 January 2025. https://techcrunch.com/2025/01/16/apple-pauses-ai-notification-summaries-for-news-after-generating-false-alerts ; see also AppleInsider, “Apple brings back notification summaries for news in iOS 26”, 22 July 2025. https://appleinsider.com/articles/25/07/22/apple-brings-back-notification-summaries-for-news-in-ios-26
13. BBC. “Representation of BBC News content in AI Assistants.” BBC Research, February 2025; as reported by The Register, “AI summaries turn real news into nonsense, BBC finds”, 12 February 2025. https://www.theregister.com/2025/02/12/bbc_ai_news_accuracy/
14. European Broadcasting Union and BBC. “News Integrity in AI Assistants.” October 2025; and EBU press release, “Largest study of its kind shows AI assistants misrepresent news content 45% of the time, regardless of language or territory.” https://www.ebu.ch/news/2025/10/ai-s-systemic-distortion-of-news-is-consistent-across-languages-and-territories-international-study-by-public-service-broadcaste
15. Reuters Institute for the Study of Journalism. “Digital News Report 2026: Overview and key findings.” University of Oxford, June 2026. https://reutersinstitute.politics.ox.ac.uk/digital-news-report/2026/dnr-executive-summary
16. Reid, Liz. Google Search blog post on AI Overviews, 30 May 2024; as reported by MIT Technology Review, “Why Google's AI Overviews gets things wrong”, 31 May 2024, and Forbes, 31 May 2024. https://www.technologyreview.com/2024/05/31/1093019/why-are-googles-ai-overviews-results-so-bad/
17. Electronic Frontier Foundation. “Axon's Draft One Is Designed to Defy Transparency.” July 2025. https://www.eff.org/deeplinks/2025/07/axons-draft-one-designed-defy-transparency
18. Guariglia, Matthew. “Victory! California Requires Transparency for AI Police Reports.” Electronic Frontier Foundation, 14 October 2025. https://www.eff.org/deeplinks/2025/10/victory-california-requires-transparency-ai-police-reports
19. California Legislature. SB 524, Law enforcement agencies: artificial intelligence, 2025 to 2026 session; and Office of Senator Jesse Arreguín, “Governor Signs Bill Requiring Law Enforcement to Disclose Use of AI in Police Reports.” https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260SB524
20. Utah Legislature. SB 180, Law Enforcement Usage of Artificial Intelligence, 2025 General Session; and Utah News Dispatch, “Deepfakes, policing, mental health and more: Lawmakers look to further regulate AI”, 26 March 2025. https://le.utah.gov/~2025/bills/static/SB0180.html
21. Information Commissioner's Office. “Right to rectification.” UK GDPR guidance and resources. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/individual-rights/individual-rights/right-to-rectification/
22. European Union. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 50, Transparency obligations for providers and deployers of certain AI systems. https://artificialintelligenceact.eu/article/50/ ; see also Cooley, “EU AI Act: Transparency Obligations Take Effect 2 August 2026”, 3 August 2026. https://www.cooley.com/news/insight/2026/2026-08-03-eu-ai-act-transparency-obligations-take-effect-2-august-2026 ; and Goodwin, “Not Delayed, Not Deferred: EU AI Act Transparency Obligations Are Now in Force”, 3 August 2026. https://www.goodwinlaw.com/en/insights/publications/2026/08/alerts-technology-dpc-eu-ai-act-transparency-obligations-now-in-force
23. European Union. Directive (EU) 2024/2853 on liability for defective products. Official Journal of the European Union, 18 November 2024. https://eur-lex.europa.eu/eli/dir/2024/2853/oj/eng
24. NPR. “Class-action suit claims Otter AI secretly records private work conversations.” 15 August 2025. https://www.npr.org/2025/08/15/g-s1-83087/otter-ai-transcription-class-action-lawsuit ; and Baird Holm, “Core Privacy Claims Against AI Note Taking Platform Survive Dismissal” (In re Otter.AI Privacy Litigation, No. 5:25-cv-06911, N.D. Cal.), 24 August 2026. https://www.bairdholm.com/blog/core-privacy-claims-against-ai-note-taking-platform-survive-dismissal/
25. Lee, Hao-Ping (Hank), and colleagues, Microsoft Research and Carnegie Mellon University. “The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers.” Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, ACM. DOI 10.1145/3706598.3713778. https://dl.acm.org/doi/full/10.1145/3706598.3713778

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Jasmine Ault compares it to a house fire. On 24 September 2026, according to a report by Anja Karadeglija of The Canadian Press published five days later, the Toronto resident found that Meta had shut down her Facebook and Instagram accounts without warning. The Facebook profile was 20 years old. One Instagram account dated back to 2011. A second was dedicated to her dog. Overnight, two decades of memories, contacts and professional connections were behind a door she could not open.

“It's actually quite scary to think about the trust that we put in the social media platforms that can all be taken away in one day,” she told the news agency. The fire, she explained, is a metaphor, “and it's really upsetting because the hypothetical, the metaphorical fire that I'm referring to, is my memories.”

What made the loss feel so total was not just the ban. It was the silence after it. Ault told The Canadian Press she had not been able to reach a human being at Meta since the accounts were deactivated. The only things that would talk to her were AI chatbots, and they could not resolve the problem. CBC News, which also spoke to her, reported that after the deletion notice arrived she stayed up for about 26 hours straight trying to appeal. Within minutes of pressing appeal, she was told her case was closed and that she would not be permitted to appeal again. She was never told which rule her Facebook account had supposedly broken. Meta's explanation, as reported, amounted to a single line: her Facebook account “doesn't follow our rules”. When The Canadian Press asked questions, Meta did not respond to them directly and instead pointed to a support blog post, including an update from March about rolling out an AI support assistant. Meta did not respond to CBC's requests for comment.

This is the uncomfortable shape of a modern dispute with a platform. The thing that holds your relationships, your archive and your livelihood can remove all of them in an instant, and the only voice you hear afterwards may belong to software. That raises two questions bigger than one Toronto woman's Facebook profile. What should an effective appeal look like when AI mediates access to the platforms that hold people's lives? And how can platforms keep their security enforcement, which is real and necessary, while still offering a route to review that a locked-out human being can actually use?

What the Report Does and Does Not Establish

It is worth being precise about what is known. The Canadian Press report does not establish why Ault's accounts were disabled. Meta has not publicly said which rule it believes was broken. Ault herself has a theory: she told the agency she suspects someone tried to hack her account, because she found a record of suspicious login attempts shortly before it was deactivated, and Meta's records associated her phone number with someone in India. That is her suspicion, reported as such. It is not a finding.

It would be easy to fill the gap with a story. Perhaps an intruder got in and posted something that tripped an automated filter. Perhaps a classifier misread an old photograph. Perhaps something else entirely happened. None of those possibilities is established, and the honest position is that nobody outside Meta, including Ault, has been told. That gap is the point. A system that cannot tell a person what she is accused of has already failed the first test of a fair process, whatever the underlying facts turn out to be.

The human cost, at least, is not in dispute. Ault works in marketing and communications and had used social media to attract consulting clients. Prospective clients, she told the news agency, “have nothing to look at. They can't see my portfolio of work.” Friends might assume she had blocked them. “There's a dead end and I can't even communicate to the people I want to communicate with,” she said. Markus Giesler, a professor of marketing at York University, framed it in sharper terms for the same report: “We like to think of this as a customer service case when in actuality it's a case of platform accountability.” These platforms, he added, “have been woven so much into our social fabric, into who we are as individuals and communities, that access to them is really important.”

The Ban Wave That Came Before

Ault's story lands in a long, loud context. Through the summer of 2025, users in several countries reported Facebook and Instagram accounts being disabled in waves, often with no clear reason. In June 2025 Meta acknowledged what spokesperson Andy Stone called “a technical error that impacted some Facebook Groups”, after communities devoted to things such as parenting, birdwatching and gaming received violation notices. By early July, TechCrunch reported that a petition titled “Meta wrongfully disabling accounts with no human customer support” had passed 25,500 signatures, and that users were organising on Reddit to share experiences and talk about legal action.

Some of the most distressing cases involved accusations of child sexual exploitation, a category where, as the Oversight Board noted quoting Meta's Transparency Center, a single violation may be severe enough that “we'll disable your account”. ABC News in Australia reported in July 2025 on Madison Archer, a beautician whose Instagram business account, built over nine years, was permanently disabled in June 2025 after she posted a video of herself with her daughter. She appealed within minutes and was refused. Her account came back only after ABC News contacted Meta, which told her: “We're sorry that we've got this wrong and that you weren't able to use Instagram for a while.” Archer's verdict on the process was blunt: “If a human actually did the review they would see that I'm not sexually exploiting children.”

In South Korea, the backlash became political: The Korea Herald reported in July 2025 that the Korea Communications Commission had formally asked Meta for written clarification, and that Meta Korea had apologised for the “frustration”. None of this proves that Ault's case shares a cause with those waves. It does show that the pattern she describes, sudden loss, terse explanation and support that cannot or will not engage, was familiar long before September 2026.

Meta itself has not pretended its enforcement is flawless. In January 2025, Joel Kaplan, the company's chief global affairs officer, wrote that in December 2024 Meta was removing millions of pieces of content every day, and that “we think one to two out of every 10 of these actions may have been mistakes”. He conceded that “too many people find themselves wrongly locked up in 'Facebook jail,' and we are often too slow to respond when they do.” By May 2025 Meta reported a roughly 50 per cent reduction in enforcement mistakes in the United States between the fourth quarter of 2024 and the first quarter of 2025. Its more recent integrity reporting says that less than 0.1 per cent of the content produced on Facebook and Instagram in a quarter was removed incorrectly, but also discloses that in the fourth quarter of 2025 false positives “increased over 100% during a very brief period of the quarter due to a bug”.

A tiny percentage of an enormous base is still a great many real people, and one bug can double the rate of wrongful action in days. Automated enforcement will make mistakes. The question is what happens to the person on the receiving end when it does.

Paying for a Human and Getting a Bot

When Meta launched its paid Meta Verified subscription in 2023, it promised direct access to account support. It is precisely what a person in Ault's position would reach for: pay a monthly fee, get a human. Ault did exactly that. According to CBC News, she paid US$14.99 (about C$21) for Meta Verified's Standard tier, believing it would get her human support. The “enhanced support” she received was another AI chatbot, which promptly closed her ticket, telling her she had “reached her limit of support interactions”. She was not the first subscriber to be disappointed. TechCrunch reported in July 2025 that during the ban wave, paying subscribers described agents who were dismissive, closed chats and offered canned responses.

The sharpest assessment came from inside Meta's own governance structure. In June 2026 the Oversight Board, the independent body Meta funds to review its content decisions, issued its first decision on account disablement. Summarising the public comments it received, the Board recorded that many users said they were unable to appeal, never received any explanation for why their account was disabled, or were unable to download their content. Others, it noted, had created a new account specifically so they could pay for Meta Verified, “which includes '24/7 access to email or chat agent support,'” and yet “could not get meaningful assistance”. Some reported that their new accounts were then disabled for breaking the rule against creating an account after being banned.

The trap is circular. The paid route to a human requires an account. The person who most needs a human no longer has one. Opening a fresh account can itself be treated as evasion. And the free route, increasingly, is an AI assistant.

Meta has been explicit about that direction of travel. In December 2025 it launched a centralised support hub for Facebook and Instagram, acknowledging that “support hasn't always met expectations”, and began testing an AI support assistant. In March 2026 it announced a wider rollout, describing an assistant that provides “24/7 help for account issues” and can “respond to requests typically in under five seconds”. The same post contains a sentence that should be read closely by anyone who has been locked out: “people will continue to play a key role in how we make the highest risk and most critical decisions, such as appeals of account disablement.”

That is the right principle. The difficulty is that, from the outside, it is invisible. If a person cannot tell whether a human ever looked at her appeal, and receives a decision indistinguishable from an automated one, a human somewhere in the pipeline does little to make the process accountable.

Why Platforms Lock the Doors So Fast

None of this means platforms should slow-walk enforcement or open the doors to anyone who asks nicely. The reasons for speed are serious.

The first is child safety. Meta reported that in the fourth quarter of 2025 alone, Facebook, Instagram and Threads sent more than 2.6 million reports to the CyberTipline run by the National Center for Missing and Exploited Children in the United States. When a system detects apparent child sexual exploitation, the expectation is immediate action, not a conversation. Errors are horrifying for the wrongly accused, but delay on true positives carries its own grave cost.

The second is fraud. Meta says that in 2025 it took down 10.9 million accounts associated with scam centres in Southeast Asia and the Middle East. An appeals channel that can be gamed becomes a tool for operations of that scale.

The third, and the one most relevant to Ault's suspicion, is account takeover. A hijacked account carries the trust of its real owner's friends, followers and clients. A platform that detects suspicious logins faces a genuine dilemma: lock the account and the owner suffers; leave it open and the intruder can impersonate them or scam their contacts. Recovery itself is dangerous ground. The United States National Institute of Standards and Technology, in its digital identity guidelines, notes that because account recovery is invoked infrequently, it “is generally less convenient than authentication” and “may involve extended waiting times”, and requires that an account recovery event always triggers a notification to the subscriber “to help detect the fraudulent use of account recovery”. Security engineers expect recovery to be slower than logging in precisely because it is where attackers aim.

When the Help Desk Became the Break-In

If anyone doubted that recovery is a battleground, 2026 supplied the proof, courtesy of the very kind of AI support that now stands between users like Ault and their accounts.

On 1 June 2026, 404 Media reported that hackers had been taking over high-profile Instagram accounts by asking Meta's AI support assistant to link the target account to a new, attacker-controlled email address. The accounts reportedly affected included the Barack Obama White House account, the account of the chief master sergeant of the US Space Force and that of the retailer Sephora. TechCrunch reported two days later that Instagram was alerting users targeted during the attacks, and quoted Meta spokesperson Andy Stone saying that “the issue that did happen has already been fixed”.

The scale emerged in a filing Meta made to the Maine Attorney General, reported by Help Net Security on 8 June. It disclosed that 20,225 Instagram accounts had been taken over through a flaw in an AI-assisted recovery system called High Touch Support, with the first unauthorised access dating to 17 April 2026 and the vulnerability identified on 31 May. Amber Hannah, an associate general counsel at Meta, explained in the notice that “due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user's Instagram account.” Accounts without two-factor authentication could then be taken over.

The episode cuts both ways. It shows why platforms are wary of easy recovery: an assistant that says yes too readily is a gift to criminals. But it also punctures the idea that automation is inherently the safer option. An AI system that handed over more than 20,000 accounts was not a cautious gatekeeper. It was a fast one. Speed and security are not the same thing.

What the Oversight Board Found Inside the Account Machine

The Oversight Board's June 2026 decision is the most detailed public examination yet of how Meta disables accounts. The case itself was not a sympathetic one. The Board upheld Meta's decision to permanently disable an Instagram account with more than 70,000 followers that had posted threats of violence against a journalist, including an image of a bullseye over her face. It did not conclude that Meta should disable fewer accounts across the board.

What it did conclude, after more than 750 public comments, was that the disablement process raised serious concerns about due process and proportionality. It found no clear framework for when “egregious” violations justify immediate permanent removal, and noted that there were no intermediate options between permanent disablement and leaving the ordinary strikes system to run its course. Where mitigating factors exist, the Board suggested, “a significant but time-bound suspension may be more proportionate.”

On appeals, the Board was pointed. Many comments described reasons people did not understand and appeals denied within moments of submission. Ault's experience, as CBC News reported it, fits that description exactly: her case was closed within minutes of her pressing appeal, with no further appeal allowed. Appeals of account disablement, the Board wrote, “are not effective if they run the same decision through the same automated process, expecting different results with the same inputs.” It accepted that advances in AI should be used to make reviews more accurate and comprehensive, but said that “prioritizing of human review in edge cases will be important”, especially for “one and done” violations, “where any error in automated decision-making has very serious consequences beyond the removal of a post or imposition of a strike.”

It also set out what users should be able to see. It called for dashboards giving “reliable information on current account status, past violations, information on available appeal options, and the status of any pending appeals”, downloadable for record-keeping. It wanted clear, prominent and timely notifications, “in a language they understand”, specifying “the rule violated, the sanction imposed and options to appeal”. It asked that users be told about “the role of automation in the review of content or behavior and the imposition of warnings or penalties”, and that people who lose access through permanent disablement can still reach the reasons and appeal mechanisms. Meta said it welcomed the decision and would respond to the recommendations after reviewing them.

Set beside Ault's account, the Board's framework reads like a checklist of what she says she did not get.

Europe Has Already Written Down What an Appeal Owes You

For users in the European Union, much of this is not aspirational. It is law, under the Digital Services Act.

Article 17 requires hosting services to give “a clear and specific statement of reasons” to anyone whose account is suspended or terminated. That statement must include the facts and circumstances relied on, and information on the use made of automated means in taking the decision. A line saying an account “doesn't follow our rules” would struggle to meet that standard.

Article 20 requires online platforms to provide, for at least six months after a decision, “an effective internal complaint-handling system” that lets people lodge complaints “electronically and free of charge”, including against decisions to suspend or terminate their account. Complaints must be handled “in a timely, non-discriminatory, diligent and non-arbitrary manner”, and where the complainant's conduct does not warrant the measure, the platform must reverse its decision “without undue delay”. Crucially, the platform must ensure that these complaint decisions “are taken under the supervision of appropriately qualified staff, and not solely on the basis of automated means.” The law does not ban AI from the process. It bans AI from being the whole of it.

Article 21 opens a door outside the platform. Users can take disputes to a certified out-of-court dispute settlement body, which should decide within 90 days, extendable to 180 for complex cases. If the body finds for the user, the platform pays the fees; if it finds for the platform, the user does not cover the platform's costs unless they acted in bad faith. The catch is that these bodies cannot impose a binding settlement.

Data protection law adds another layer. The General Data Protection Regulation gives people a right of access to their personal data and a right to portability, receiving what they provided “in a structured, commonly used and machine-readable format”, normally within one month. Article 22 protects people from decisions “based solely on automated processing” with legal or similarly significant effects and, where exceptions apply, guarantees a right “to obtain human intervention” and to contest the decision. Whether a given ban meets that threshold is a legal question, but the principle is unmistakable.

Canadians have a version of the access right too. Under the Personal Information Protection and Electronic Documents Act, organisations must answer access requests in writing within 30 days, with limited extensions, and individuals can complain to the Office of the Privacy Commissioner of Canada. For someone whose memories are trapped in a disabled account, that is a lever that exists independently of any chatbot.

The Independent Referee With an Empty Evidence File

The most revealing evidence about how these systems work in practice comes from the referee. Appeals Centre Europe, an out-of-court dispute settlement body certified by Ireland's media regulator, Coimisiún na Meán, and set up with a one-time grant from the Oversight Board Trust, began operating in late 2024 and now covers platforms including Facebook, Instagram, TikTok and YouTube. When it began handling account suspensions in 2025, its chief executive, Thomas Hughes, said: “Losing your social media account is more than an inconvenience, it affects people's lives in fundamental ways.”

In May 2026 the centre published data covering April 2025 to March 2026. As reported by RTÉ, it received more than 24,000 disputes, of which more than 12,000 were within scope. Where it reviewed decisions, it disagreed with the platforms 59 per cent of the time. On account suspensions, the figures were starker still. It had received more than 5,000 eligible disputes from users challenging suspensions, but by the end of March 2026 had made decisions after actually receiving the relevant content in fewer than 150 of them. In the rest, the platforms did not supply the material, and the centre issued “default decisions” in the user's favour. Its decisions are not binding, and the centre said it was aware of only a handful of cases where platforms had implemented its decisions on harmful content.

Independent review works only if the reviewer can see the evidence, and a legal right to escalation can fail in practice if the platform does not engage. The infrastructure for a fair appeal can exist on paper and still leave a locked-out person with nothing.

Britain and Canada Are Still Catching Up

Outside the EU, the protections are thinner. The United Kingdom's Online Safety Act 2023 requires user-to-user services to operate complaints procedures that are “easy to access, easy to use (including by children) and transparent”. Section 21 includes complaints from users who have been warned, suspended or banned, but only where that action results from content the provider considers to be illegal content. A ban for breaching a platform's own terms of service, rather than for illegal content, sits largely outside that specific duty.

Canada, where Ault lives, has no equivalent framework in force. In June 2026 the federal government introduced Bill C-34, the Safe Social Media Act, which would create a Digital Safety Commission and impose duties on platforms around seven categories of harmful content, including content that sexually victimises children, incites violence or foments hatred. The government's published summaries require “accessible reporting and complaint mechanisms” and give the commission a role in handling user complaints about harmful content. They do not, in the material the government has published, set out a specific right for users to appeal account disablement or to have such an appeal decided by a human. According to LEGISinfo, Parliament's official bill-tracking service, the bill is at second reading in the House of Commons and has not yet been referred to committee, so that gap could yet be addressed.

What an Effective Appeal Actually Looks Like

Pull together the Oversight Board's principles, the DSA, data protection law and the experience described in the Canadian Press report, and a clear picture emerges of what an effective appeal must include when AI sits at the front door. None of it is exotic.

The first element is notice with a specific reason. A person should be told, at the moment of enforcement and in plain language, which rule they are said to have broken, what the sanction is, how long it lasts and how to challenge it. If the decision was triggered by an automated system, they should be told that too. “Doesn't follow our rules” is not a reason. It is a refusal to give one.

The second is access to the evidence, as far as safety allows. A user cannot contest a decision about content they cannot identify. Where the trigger was a post, a message or a login pattern, the appeal should show what was relied on. Disclosure that would compromise a child exploitation investigation or tip off a fraud ring is a legitimate limit, but it should be an exception, not the default. The Appeals Centre's experience shows how far the default currently leans towards secrecy.

The third is a real opportunity to make one's case. The Oversight Board insists on the chance to submit written reasons, with continued access to the policy justification and any relevant exceptions. A drop-down menu and a “submit” button is not that. An AI assistant can help people understand the rules and structure an appeal. Its job should be to prepare the case for a decision-maker, not to be the decision-maker in the hardest cases.

The fourth is a guaranteed human decision for serious sanctions. Permanent disablement of a long-standing account is the platform equivalent of capital punishment for a digital identity. The DSA already requires that complaint decisions are not taken solely by automated means, and Meta's own March 2026 announcement says humans remain key in appeals of account disablement. What is missing is visibility. People should be told, in the decision itself, that a named role (not necessarily a named individual) reviewed their case, and on what basis.

The fifth is predictable timelines. An appeal that may take a day or six months is useless to someone whose income depends on their account. Platforms should publish target times, show users where they are in the queue and report on whether targets are met.

The sixth is preserved access to one's own data during the dispute. Losing access to a platform should not mean losing one's photographs, messages and contact lists for the duration of a months-long appeal. Meta's own help pages say users “may be able to request a download” of their information from the notification screen, but the Oversight Board recorded many users saying they could not. Data export during suspension should be a default, subject only to narrow, specified exceptions such as material that is itself illegal.

The seventh is escalation that leads somewhere independent. Internal appeals will always carry a structural conflict of interest. Users need a clearly signposted route beyond the platform, whether that is a DSA-style dispute settlement body, a regulator's complaints function or an ombudsperson, with the power to see the evidence and a platform that is obliged to engage.

Enforcement That Does Not Have to Be All or Nothing

The second question is harder: how do platforms keep enforcement strong without making review impossible? The answer is to stop treating account action as a single switch with two positions, on and gone.

Graduated enforcement is the obvious starting point. The Oversight Board itself observed that Meta lacked intermediate options between permanent disablement and its ordinary strikes system, and suggested time-bound suspensions where mitigating factors exist. A spectrum of measures, from restrictions on posting or messaging, to temporary suspension, to permanent removal, lets a platform act immediately on a risk while matching the long-term sanction to the confidence and seriousness of the finding.

Read-only and data-only modes are the second tool. When a platform suspects compromise, as Ault believes happened to her, it can freeze outward activity without erasing the owner's access to her own past. A frozen account cannot scam a contact list, but its owner can still see and export her photographs and messages. Where the suspected violation is the gravest kind, read-only access may be inappropriate, but such cases are a subset of disablements, not a reason to treat every case as one.

The third tool is separating the security question from the conduct question. “Is this the real owner?” and “Did this account break the rules?” are different problems, and conflating them hurts everyone. If an account appears to have been hijacked, the right response is a security recovery process with strong identity checks, such as the optional selfie video verification Meta now offers, plus notifications to the owner's known contact points of the kind NIST requires. Only once ownership is established does it make sense to ask whether the owner, rather than an intruder, did anything wrong. The High Touch Support breach shows what happens when a recovery path skips the first step.

Fourth, verification should unlock a route to a human, not replace one. If a person proves who they are through a government document or a live selfie, the platform has sharply reduced the risk of handing the account to an attacker. That is exactly the moment when a human reviewer becomes affordable and safe. The cruel irony of Ault's account is that she says she had done the paid verification step and still could not get past the chatbot. A verification-gated escalation path, honoured in practice, would answer the security objection to human support rather than ignore it.

Fifth, proportionality needs to be measured, not merely promised. The Oversight Board asked for transparency reports on account disablement that distinguish “one and done” removals from graduated enforcement and specify which rules drove them. Add overturn rates, resolution times and the share of decisions reviewed by humans, and outsiders can see whether the system works. Meta already publishes precision figures for content removal. It should publish the equivalent for accounts.

Finally, platforms should design for the worst week of a user's life, not the median one. Most support requests are trivial, and AI assistants handle them well. Account disablement is rare per user, devastating when it happens and, as the 2025 ban waves and the 2026 recovery breach showed, vulnerable to systemic error. That is exactly where human judgement and independent oversight earn their cost.

Making the Machine Answerable

It is tempting to cast AI support as the villain of Ault's experience, but that is not quite right. Nobody outside Meta knows what triggered the shutdown. What the chatbot represents is a support model built to resolve common cases cheaply and quickly, deployed in front of the rare cases where cheap and quick is the wrong answer.

Meta has said in writing that humans will play a key role in appeals of account disablement. The Oversight Board has said appeals must not re-run the same automated decision. The EU has made human supervision of complaint decisions law. An independent appeals body has shown that when platforms are asked to defend suspensions with evidence, they often do not. Every piece of a fair system exists somewhere. What is missing is the will to assemble them in front of the person who has just lost twenty years of her life and wants, above all, to be told why.

Giesler's framing is the right one. This is not a customer service problem to be optimised for response time. It is an accountability problem, and accountability means a reason you can read, evidence you can see, a person who can decide, a deadline that is kept and a referee who answers to neither side. A chatbot can help with every one of those steps. It cannot be allowed to stand in for all of them.

---

References

1. Karadeglija, Anja. “Canadian says Meta won't let her talk to a human after killing her accounts.” The Canadian Press, 29 September 2026 (updated 30 September 2026), as published by CP24. https://www.cp24.com/news/2026/09/29/canadian-says-meta-wont-let-her-talk-to-a-human-after-killing-her-accounts/
2. Jones, Alexandra Mae. “Overnight, Meta deleted all her accounts. AI customer service won't let her reach a human.” CBC News, 1 October 2026. https://www.cbc.ca/news/business/meta-accounts-deleted-ai-customer-service-9.7364543
3. Kaplan, Joel. “More Speech and Fewer Mistakes.” Meta Newsroom, 7 January 2025, updated 29 May 2025. https://about.fb.com/news/2025/01/meta-more-speech-fewer-mistakes/
4. Meta. “Integrity Reports, H1 2026.” Meta Transparency Center, March 2026. https://transparency.meta.com/reports/integrity-reports-h1-2026/
5. UPI. “Meta confirms technical error suspended thousands of Facebook Groups.” 24 June 2025. https://www.upi.com/Top_News/US/2025/06/24/meta-suspends-facebook-groups/5531750820668/
6. Perez, Sarah. “Meta users say paying for Verified support has been useless in the face of mass bans.” TechCrunch, 2 July 2025. https://techcrunch.com/2025/07/02/meta-users-say-paying-for-verified-support-has-been-useless-in-the-face-of-mass-bans/
7. Dervisevic, Hanan. “Meta falsely accuses Instagram user of breaking child exploitation rules.” ABC News (Australia), 17 July 2025. https://www.abc.net.au/news/2025-07-17/meta-wrongly-accuses-user-of-breaching-child-sex-abuse-rules/105540896
8. Moon, Joon-hyun. “Meta faces backlash in South Korea over mysterious account bans.” The Korea Herald, 13 July 2025. https://www.koreaherald.com/article/10530332
9. Meta. “Making it Easier to Access Account Support on Facebook and Instagram.” Meta Newsroom, 4 December 2025. https://about.fb.com/news/2025/12/making-it-easier-to-access-account-support-on-facebook-and-instagram/
10. Meta. “Boosting Your Support and Safety on Meta's Apps With AI.” Meta Newsroom, 19 March 2026. https://about.fb.com/news/2026/03/boosting-your-support-and-safety-on-metas-apps-with-ai/
11. Oversight Board. “Account Ban for Targeting Public Figures” (case decision BUN-ZHSLVJO2) and summary. 4 June 2026. https://www.oversightboard.com/decision/bun-zhslvjo2/
12. Perez, Sarah. “Meta's Oversight Board says account bans lack due process, transparency.” TechCrunch, 4 June 2026. https://techcrunch.com/2026/06/04/metas-oversight-board-says-account-bans-lack-due-process-transparency/
13. Koebler, Jason. “Hackers Simply Asked Meta AI to Give Them Access to High-Profile Instagram Accounts. It Worked.” 404 Media, 1 June 2026. https://www.404media.co/hackers-simply-asked-meta-ai-to-give-them-access-to-high-profile-instagram-accounts-it-worked/
14. Franceschi-Bicchierai, Lorenzo. “Instagram is alerting users who were targeted by hackers during AI chatbot attacks.” TechCrunch, 3 June 2026. https://techcrunch.com/2026/06/03/instagram-is-alerting-users-who-were-targeted-by-hackers-during-ai-chatbot-attacks/
15. Markovic, Sinisa. “Hackers used Meta's AI support system to hijack over 20,000 Instagram accounts.” Help Net Security, 8 June 2026, reporting Meta's data breach notice filed with the Maine Attorney General. https://www.helpnetsecurity.com/2026/06/08/instagram-ai-support-vulnerability-account-takeovers/
16. National Institute of Standards and Technology. “Digital Identity Guidelines: Authentication and Authenticator Management.” NIST Special Publication 800-63B-4, 2025, section on account recovery. https://pages.nist.gov/800-63-4/sp800-63b.html
17. European Parliament and Council. Regulation (EU) 2022/2065 on a Single Market for Digital Services (Digital Services Act), Articles 17, 20, 21 and 23. Official Journal of the European Union, 27 October 2022. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32022R2065
18. European Parliament and Council. Regulation (EU) 2016/679 (General Data Protection Regulation), Articles 12, 15, 20 and 22. https://gdpr-info.eu/
19. Office of the Privacy Commissioner of Canada. “Responding to access to information requests under PIPEDA.” https://www.priv.gc.ca/en/privacy-topics/accessing-personal-information/obligations-for-organizations/02_05_d_54_ati_02/
20. O'Donovan, Brian. “Appeals body to review social media account suspensions.” RTÉ News, 6 June 2025. https://www.rte.ie/news/2025/0606/1516958-appeals-centre-europe/
21. O'Donovan, Brian. “Increase in complaints to EU social media appeals body.” RTÉ News, 28 May 2026. https://www.rte.ie/news/2026/0528/1575608-social-media-complaints/
22. UK Parliament. Online Safety Act 2023, section 21, “Duties about complaints procedures.” legislation.gov.uk. https://www.legislation.gov.uk/ukpga/2023/50/section/21
23. Government of Canada, Canadian Heritage. “Backgrounder: Government of Canada introduces legislation to combat online harms, particularly those impacting children.” 10 June 2026. https://www.canada.ca/en/canadian-heritage/news/2026/06/government-of-canada-introduces-legislation-to-combat-online-harms-particularly-those-impacting-children.html
24. Meta Help Center. “About suspended or disabled Meta accounts.” https://www.meta.com/help/artificial-intelligence/639558911929531/
25. Parliament of Canada. “Bill C-34 (45th Parliament, 1st session).” LEGISinfo, accessed 7 October 2026. https://www.parl.ca/legisinfo/en/bill/45-1/c-34

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...