You open ChatGPT, type something personal, something you probably wouldn’t say out loud in a coffee shop, and assume it vanishes into a server farm somewhere. According to 404 Media, that assumption is wrong.
An internal OpenAI program called Project Lily reportedly routes real user prompts to human contractors, who read them, summarize the user’s intent, and score multiple ChatGPT responses on a 1-to-7 scale. The opt-out setting that controls this is on by default for Free, Plus, and Pro accounts. It does not apply retroactively to conversations you have already had, meaning prior conversations remain eligible for contractor review regardless of any setting change you make now, per 404 Media’s reporting. When 404 Media asked OpenAI whether it had ever explicitly disclosed that humans may review prompts, the company did not answer directly.
What’s Really Happening Behind the Interface
The review process is more structured than most users would expect.
Per training materials seen by 404 Media, contractors first read the user’s prompt, then write a summary of what the user was trying to accomplish, then critique and rate multiple ChatGPT responses. The target qualities are specific: less sycophancy, fewer emojis, less “AI-speak,” and more restrained, professional language. Think of it as editorial quality control, except the raw material is your private conversation.
Recruiters for the work reportedly include a firm called Crossing Hurdles, with payments processed through Mercor. One contractor told 404 Media they earned more than $50 an hour. The materials reviewed by 404 Media do not identify which model Project Lily is actually training, so the timeline and scope remain unclear.
The Filter That Admits It Can Miss Things
OpenAI’s own privacy safeguard comes with a built-in caveat.
OpenAI says a Privacy Filter screens prompts before they reach contractors. Reassuring, until you read OpenAI’s own description of it: the company acknowledges the model can miss uncommon identifiers and ambiguous private references.
Users who typed something sensitive, medical, financial, or deeply personal did so expecting a machine on the other end. A best-effort filter that hedges about its own reliability functions as a disclaimer, not a guarantee.
Disclosure Shouldn’t Live in the Fine Print
One competitor manages to say the quiet part out loud.
Anthropic also uses human review to improve its models, making this an industry-wide practice rather than an OpenAI anomaly. The disclosure gap, though, is real. Google’s Gemini privacy notice states, “Humans review some saved chats to improve Google AI,” a plain sentence that a normal person can read and understand.
OpenAI’s users, by contrast, found out through leaked contractor documents published by a news outlet. The contrast is the problem: transparency is not complicated when a company actually wants to provide it.
The opt-out option (“Improve the model for everyone”) applies only to new conversations after you disable it. Anything you shared before that moment stays in play. For the millions of users who treat ChatGPT the way previous generations treated a private diary, that retroactive gap is not a minor detail.
The fix is straightforward: make the default opt-in rather than opt-out, put the human-review disclosure in plain language at sign-up rather than buried in policy pages, and build a privacy filter reliable enough that OpenAI stops qualifying its own promises. Earning the trust of a confidant requires actually behaving like one.
