What is an instruction hierarchy in modern LLMs?
hardAnswer
- OpenAI (2024) formalized: developer instructions > user instructions > tool outputs > third-party content.
- RLHF training teaches the model to weight higher tiers over lower ones when they conflict.
- Primary defense against indirect prompt injection (retrieved content trying to override the system prompt).
- Anthropic uses a similar 'privileged / unprivileged' distinction.
- Enforced during alignment via curated conflict examples.
Check yourself — multiple choice
- All instructions equal
- Developer > user > tool output > third-party content; RLHF trains the LLM to honor this priority — main defense against prompt injection
- Random priority
- Only user prompts
Instruction hierarchy: dev > user > tool > 3rd-party; enforces safety in retrieval / agents.
#prompting#safety#alignment
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is Constitutional AI?
- How is 'refusal' behavior trained into LLMs?
- How do you balance helpfulness vs harmlessness in RLHF?
- What are the pillars of good prompt engineering?
- What are the main LLM safety and alignment concerns?
- What is supervised fine-tuning (SFT) and where does it fit in the alignment pipeline?