USA Today just published an investigation tracking ICE detention policies using the Wayback Machine—then blocked that same tool from crawling their site. This contradiction captures a broader crisis: major news outlets are systematically cutting off the internet’s most powerful preservation tool while relying on it for their own reporting. You’ve probably used the Wayback Machine to fact-check claims or recover lost content. Now it’s disappearing just when digital accountability matters most.
A Growing Blackout
The 30-year-old Archive preserves over a trillion web pages, but blocking trends threaten massive gaps in digital records.
Major news sites now block the Internet Archive’s crawler, including:
- The New York Times
- The Guardian
USA Today’s parent company restricts numerous outlets from archival crawling. These aren’t obscure publications—they’re sites you visit daily.
Their content, once preserved for historical research and fact-checking, now vanishes into digital amnesia. The trend creates concerning gaps in our collective digital memory at precisely the moment when preserving online information has become critically important for accountability and research.
AI Panic Drives the Blocks
Publishers cite fears that AI companies will scrape archived content for training models without permission.
Publishers justify these restrictions by worrying that archived content feeds AI training without authorization. The New York Times specifically mentions concerns about content being used to “compete directly” with their work. With numerous AI copyright lawsuits active in the US, outlets fear the Wayback Machine provides unauthorized data access.
The Guardian raises similar concerns about archived material being misused by AI systems, reflecting broader industry anxiety about protecting intellectual property in an era of aggressive data harvesting.
Journalists Fight Back
Professional consensus emerges that archival access serves the public good despite publisher concerns.
Over 100 journalists signed a letter supporting the Internet Archive, including prominent figures like Rachel Maddow and Taylor Lorenz. They argue the tool remains essential for fact-checking, investigative research, and accessing cultural sites that vanish without warning.
The pushback reveals professional consensus that archival access serves journalism itself. Even reporters at outlets blocking the Archive recognize its value for accountability reporting and historical documentation.
Digital Memory at Risk
No comparable public tool exists for comprehensive web preservation beyond the Internet Archive.
“They’re able to pull together their story research because the Wayback Machine exists. At the same time, they’re blocking access,” notes Internet Archive director Mark Graham about publishers’ contradictory behavior. No comparable public alternative exists for web preservation.
Legal pressures continue following recent settlements, while ongoing negotiations with major outlets could determine whether this essential digital infrastructure survives intact. The irony cuts deep: news organizations potentially erasing the very tool that makes investigative reporting possible.
