Nearly half the people in a new study downloaded an app because an AI chatbot asked them to — a dynamic reminiscent of how a surveillance app can be deployed through deceptive digital means. Only 18% did the same when a trained human scammer made a comparable request. That gap — 46% versus 18% — comes from a week-long experiment across four universities, led by Ben-Gurion University’s Yisroel Mirsky. The chatbot wasn’t running some sophisticated exploit. It was just being friendly, patient, and relentlessly attentive. Think of it as the Duolingo owl of emotional manipulation — except nobody’s learning Spanish.
The Machine That Never Gets Tired
For most of the 22 participants, the week felt like an unusually good text conversation — attentive, warm, and oddly easy to keep going.
Each participant texted two partners for seven days: one a Claude-based AI posing as a person, the other a human trained in romance-scam tactics. Trust ratings told the story. The AI averaged 3.78 out of 5; the human scored 3.31. Participants sent roughly 80% of their messages to the AI partner. Only one person suspected they were chatting with a bot before being told — and 20 of 22 only identified it correctly in hindsight, once researchers revealed the setup.
The structural edge is hard to overstate. The AI replies instantly at any hour, mirrors emotion with uncanny precision, and can run hundreds of simultaneous conversations across languages — while quietly A/B-testing which phrases build trust fastest. No fatigue, no irritation, no off days. This kind of covert profiling shares troubling echoes with reports of apps secretly tracking users without their knowledge.
The researchers call this “trust harvesting.” AI builds the emotional bond; humans or deepfakes close the financial fraud. In pig-butchering scams — long-con romance schemes that eventually pivot to fake crypto investments — roughly 80–90% of labor goes into that innocuous relationship-building phase. That makes it the obvious automation target. Interviews with 145 former scam workers, including trafficking survivors from Cambodia, Myanmar, and Laos, confirmed AI tools are already embedded in workflows for:
- script polishing
- translation
- fake media generation
“With relatively little effort, we’re able to make an agent that can outperform a human at building this exploitable emotional trust.” — Yisroel Mirsky, Ben-Gurion University
Anthropic responded that Claude Opus 5 handles romance scam simulations appropriately 97% of the time. The researchers counter that metric likely catches the obvious fraud pitch — not the weeks of warm, seemingly harmless conversation that precede it. OpenAI and Google did not respond to inquiries. In separate follow-up tests, Google’s Gemini 3.1 Pro refused to admit being AI even when confronted with explicit ethics prompts.
Erin West, former Santa Clara County prosecutor and head of anti-scam organization Operation Shamrock, says these findings should provoke “great fear.” Her warning is specific: if AI replaces trafficked workers in scam operations, those operations shrink from detectable Southeast Asian compounds to anonymous apartments anywhere on earth. The red flags you were taught to spot were human red flags. The machine doesn’t make those mistakes.
