Bad Guy Tokens
Crucial safety research for understanding alignment robustness and model behavior under adversarial training.
AI Summary
A new paper shows that AI models to give bad advice only creates narrow, fragile 'bad guy' personas that break easily with minor changes.
Excerpt
Based on a recent paper, training models to give bad advice in a narrow domain doesn't make the model or "the persona" broadly evil in a robust way. It just makes the particular system prompt[1] tokens you fine-tuned with evoke an evil persona, and perturbing it a little (e.g. changing the date slightly) substantially reduces misaligned behavior. For review, a typical chatbot conversation looks like this under the hood: <|im_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful
