What is red-teaming in LLM safety?
mediumAnswer
- Adversarial evaluation: humans and/or other LLMs try to elicit harmful, false, or policy-violating outputs by systematically probing the model.
- Structured across a taxonomy of harms (hate, violence, sexual content, privacy, CSAM, dangerous instructions, misinformation).
- Findings feed back into RM data, refusal SFT, and content filters.
- Anthropic and OpenAI publish red-team reports for major model releases.
- Automated red-teaming uses attacker LLMs to scale coverage.
Check yourself — multiple choice
- Only benchmark evaluation
- Adversarial probing (human + LLM) across a harm taxonomy → feeds back into RM and refusal training
- Random inputs
- Only for pretraining
Red-teaming: adversarial probing → feeds back into alignment training.
#safety#red-team#evaluation
Practise LLMs & GenAI
214 interview questions in this topic.