“Project Lily”: Real People Are Reading Your ChatGPT Conversations

OpenAI hires hundreds of outside contractors who read a nonstop stream of real ChatGPT users' queries — so reports the outlet 404 Media, citing leaked internal documents, guidelines, Slack channels and the actual text of those queries. Most of the service's more than 900 million users, it seems, have no idea that a living person might read their conversations with the chatbot.

The point of this work is to improve ChatGPT's answers. Reviewers rate and critique the generated replies: among other things, they're taught to wean the model off ascribing human traits to itself and to make it less obsequious. It was precisely the 4o model's excessive flattery that, according to a number of lawsuits, partly contributed to several suicides.

How the review works

A contractor picks a “task,” sees a real user query, briefly summarizes what the person wanted, and then rates four candidate bot answers on a scale from one (“unusable”) to seven (“hard to improve”). They must highlight at least three passages that do or don't match the target model, and explain why. The internal documents offer some odd examples: skull emojis, say, are out of place in a conversation about death, and little airplanes in a report about a crash. An excess of emoji and “AI officialese” lowers the score.

Privacy in question

Contractors don't see users' names, and OpenAI runs the conversations through its own Privacy Filter model, which strips out personal data. The company admits, however, that the filter makes mistakes and can miss rare identifiers. What's more, a “memory summary” about the user is sometimes displayed above the query — including their presumed place of residence and other personal context. Among the queries 404 Media saw were some in which a person explicitly asked the bot to keep what they'd said secret.

OpenAI did not answer the question of whether it had warned users that people would be reading their conversations. You can turn off the use of chats for training by unchecking “improve the model for everyone,” but on the free, Plus and Pro plans it's on by default — and it doesn't work retroactively.

The practice isn't unique: Anthropic confirmed that it, too, brings in people to review (when the relevant setting is enabled and after anonymization), and Google Gemini's disclaimer states outright that some chats are reviewed.

The man behind the curtain

One reviewer from North America said he earns more than $50 an hour — unusually generous for the industry — through the intermediary Crossing Hurdles, with the company Mercor ultimately footing the bill. He called the work “monotonous” and the guidelines contradictory and constantly changing.

UCLA professor Sarah T. Roberts compared the revelation to a scene from “The Wizard of Oz,” where a man pulling levers turns out to be hiding behind the magic. In her view, the models constantly need human intervention — yet it's precisely this labor that companies value and pay the least, and the generous rate will most likely not last long.

Source: 404 Media