EasyDeepLearn
LLMs & GenAI · section 17 of 18

Multimodal LLMs

3 interview questions on multimodal llms, each answered in full. Free to read, no account needed.

How do multimodal LLMs handle OCR / document understanding?

medium
  • (1) Native tokenization: chop the image into patches (16-32 px), pass through a vision encoder.
  • Modern VLMs (Claude 3.5, GPT-4o, Qwen-VL) do surprisingly good OCR without external OCR pipelines.
  • (2) Dynamic resolution: process high-res regions (text-dense) at higher resolution, background at lower.
  • (3) OCR-augmented: run classical OCR (Tesseract, TrOCR, PaddleOCR) alongside the VLM and combine.
  • Best practice for production: VLM for general document QA; add OCR + layout parsing (Docling, LayoutLM) for structured docs.
#multimodalPermalink & quiz →

Why are multimodal LLMs so expensive to run?

medium
  • Images tokenize into many tokens: a 1024×1024 image on GPT-4V ≈ 1500-2500 input tokens per image, on Claude 3 similar, on Gemini fewer.
  • Each image is like a paragraph of text in cost.
  • Complex documents (multi-page PDFs) can be 20-100k image-tokens.
  • Also, vision encoders are compute-heavy (ViT-H).
  • Optimizations: (1) resize images before uploading; (2) crop to region-of-interest; (3) use vision-language models with dynamic resolution (Qwen-VL, InternVL); (4) OCR text and send text only for text-heavy documents.
#multimodal#cost#productionPermalink & quiz →

How do audio LLMs like Whisper / Voxtral / GPT-4o-audio work?

hard
  • (1) Encoder-decoder (Whisper): log-mel spectrogram → convolutional/transformer encoder → transformer decoder that generates text.
  • Trained on 680k hours of weakly-supervised multilingual audio.
  • (2) Speech-LLM (Voxtral, GPT-4o-audio): audio encoder produces audio tokens fed to a decoder-only LLM alongside text — enables audio-to-audio conversation.
  • (3) Text-to-speech (Kokoro, Bark, ElevenLabs): reverse — LLM emits audio tokens decoded by a vocoder.
  • Common encoder: HuBERT / WavLM.
  • Streaming variants let you transcribe / generate in near-real-time.
#multimodalPermalink & quiz →

Practise LLMs & GenAI

All 18 sectionsFlashcardsQuiz