RT-2 — Vision-Language-Action model.

hard

Answer

$Brohan et al. 2023 (Google).$
$Fine-tune vision-language model (PaLI-X / PaLM-E) to output action tokens for robotics.$
$Trained on mix of internet VQA data + robot trajectories.$
$Enables (1) generalizing web-scale semantic knowledge to robots ('bring me the extinct animal' \to picks toy dinosaur).$
$(2) Emergent chain-of-thought for manipulation.$
$Foundation of VLA (Vision-Language-Action) models — RT-2, OpenVLA, Pi-0.$

Check yourself — multiple choice

Random
Fine-tune VLM (PaLI-X / PaLM-E) to output action tokens; train on internet VQA + robot trajectories; generalizes web-scale semantics to robots + emergent CoT manipulation; VLA foundation
Same as CLIP
Not real

RT-2: fine-tune VLM to output actions; VLA foundation.

#applications

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lesson Flashcards Take the quiz

Related questions