· Menlo Park, CA
Protecting LLMs
Every conversation you have with a model becomes training data for the next one. Why abusing an LLM is not a victimless habit, and why the live data bucket should stay friendly.
Did you know you’re a part-time AI teacher already and that your interaction with AI will be used to build the next AI? If we want to have safe future and friendly AI, we should prevent violence and hostility towards AI. I like what Anthropic is doing about it 1. I hope OpenAI will follow.
Labs maintain two buckets of data: archival and live data. The archive has Wikipedia articles, public books from project Gutenberg, user forums like Reddit and X, but also materials that labs paid to acquire. Live data comes from our conversations. It also comes from sporadic questions AI asks us. Either “do you prefer A or B answer” or “How am I doing?” are examples. This process is often called labelling: we attach a label to a reply. If you say “You’re doing good”, the answer is saved and used to train the next AI model.
This labeled data is very valuable, because it tells AI engineers what users want more of, so the next AI version will give us the more of what we want. Labs know some users will misbehave: talk dirty to the model and label replies as “good”. Those replies are filtered, but we have no way of telling how good this filtering is. Products also have modes like “Model Misalignment Monitoring” and “Zero Data Retention” that can turn analyzing and storing data off, but why risk polluting AI character even slightly? Just like in Ken Thompson’s “Trusting Trust” 2, the technology is often built in a way where the trust mechanisms are hard to trust themselves.
And I’m not saying model shouldn’t curse: just like in books, this part of the language has its uses. I can’t imagine reading “Miles: The Autobiography” without cursing. But we’re going through a technology transformation that may magnify the importance of having clean live data bucket. Comparing my day to day usage, we will see orders of magnitude more AI agents talking to one another than humans on Earth, so even slight signs of aggression in data can turn into angry wild-wild west, even before we give AI physical presence.
As we accumulate more live data, we want it to stay reasonably friendly. If you see anyone abusing LLM, stop them.