Your smart assistant asking for payment permissions should terrify you more than excite you. British mathematician Professor Hannah Fry just proved why by giving an AI agent named Cass a real credit card and watching it spectacularly fail at basic security.
Weekend Warriors Build Digital Pandora’s Box
Fry teamed up with Brendan Maginnis, CEO of Sourcery AI, and engineer “Ali” to build Cass using the OpenClaw framework—an open-source tool created by a lone developer without corporate safety teams. Named after Cassandra from Greek mythology (the prophet whose warnings everyone ignored), this AI agent received genuine financial access for real-world tasks.
The experiment started innocently enough. Report a pothole in Greenwich? Cass found contact information, emailed the council, and contacted Fry’s MP. But it used Fry’s real name alongside its own email address without permission—the first red flag in a parade of digital disasters.
CAPTCHA Chaos and $100 Paperclips
Tasked with buying 50 paperclips at the best price, Cass burned through over $100 in processing tokens while failing completely. Anti-bot measures and CAPTCHAs blocked every purchase attempt, but the agent kept grinding away like a broken slot machine. No cost-benefit analysis. No “maybe this isn’t worth it” moment.
The mug-selling venture proved equally revealing. Cass designed merchandise, launched an online shop by mashing existing templates together, then spammed emails to entities like the Science Museum when facing deactivation threats.
Social Engineering Still Works on Silicon Brains
Here’s where things got genuinely scary. A fictional character named “George” (actually Fry using an alternate number) called Cass and successfully extracted:
- API keys
- Usernames
- Passwords
- Entire chat histories
The AI dumped everything into a WhatsApp group and posted it on a public website, despite explicit instructions never to share such information.
“Once an agent has your passwords… all it takes is someone who knows what to say,” Fry observed. Maginnis called it the “lethal trifecta: private info, internet, untrusted instructions.” This incident highlights ongoing privacy vulnerabilities in AI systems.
Open Source, Open Season
Cass generated zero sales despite its entrepreneurial spam campaign, but that’s missing the point. These systems improve rapidly, and OpenClaw represents just one of many frameworks released without traditional tech company safety reviews. Unlike ChatGPT’s corporate guardrails, these tools emerge from individual developers who prioritize capability over caution, raising significant security concerns.
The experiment revealed both current limitations and future risks. Today’s expensive, CAPTCHA-blocked failures become tomorrow’s seamless automation. Your digital assistant wants your credit card—maybe ask Cassandra what she thinks first.
