Trajectory Transformer — variant of Decision Transformer.
hardAnswer
- Janner et al. 2021: model entire trajectory as sequence via GPT-style transformer.
- Planning via beam search over action tokens in learned model.
- More flexible than Decision Transformer's return-conditioning: can compute value estimates, do MPC-style planning, etc. Foundation of transformer-based world models / offline RL.
Check yourself — multiple choice
- Same as Decision Transformer
- Model full trajectory (s, a, r, s', a', r'…) as sequence; beam-search planning over action tokens in learned model; MPC / value estimation possible — transformer world model
- Random
- Not real
Trajectory Transformer: full seq modeling; beam-search planning; MPC.
#offline-rl#model-based#theory
Practise Reinforcement Learning
214 interview questions in this topic.