How is a modern vision-language model (LLaVA, GPT-4V, Claude 3) built?
hardAnswer
- (1) Pretrained vision encoder (CLIP-ViT or SigLIP) produces image feature tokens.
- (2) Projector (MLP or Q-former) maps vision tokens into the LLM's embedding space.
- (3) Feed vision tokens as a prefix to a pretrained LLM decoder.
- (4) Train: (a) freeze both encoders, train only the projector on image-caption pairs; (b) then fine-tune LLM + projector on visual instruction data (LLaVA-Instruct).
- Modern additions: interleaved image-text pretraining, dynamic image resolution, video frames.
- GPT-4V, Claude 3, Gemini follow this pattern at scale.
Check yourself — multiple choice
- Same as text-only LLM
- Vision encoder (CLIP/SigLIP) → projector → LLM decoder; train projector then instruction-tune on visual data (LLaVA recipe)
- Only for images
- No LLM involved
VLM: vision encoder + projector + LLM decoder; two-stage training (projector then instruct-tune).
#multimodal#architecture
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is the context window and what tricks extend it?
- What is a Mixture-of-Experts (MoE) LLM?
- Why do modern LLMs use decoder-only architectures?
- What is SwiGLU and why do LLMs use it in the MLP block?
- Why did RoPE replace learned positional embeddings in modern LLMs?
- What is Grouped-Query Attention (GQA) and why does it matter?