How do audio LLMs like Whisper / Voxtral / GPT-4o-audio work?
hardAnswer
- (1) Encoder-decoder (Whisper): log-mel spectrogram → convolutional/transformer encoder → transformer decoder that generates text.
- Trained on 680k hours of weakly-supervised multilingual audio.
- (2) Speech-LLM (Voxtral, GPT-4o-audio): audio encoder produces audio tokens fed to a decoder-only LLM alongside text — enables audio-to-audio conversation.
- (3) Text-to-speech (Kokoro, Bark, ElevenLabs): reverse — LLM emits audio tokens decoded by a vocoder.
- Common encoder: HuBERT / WavLM.
- Streaming variants let you transcribe / generate in near-real-time.
Check yourself — multiple choice
- Text only
- Whisper: audio → text via encoder-decoder; audio LLMs: audio tokens + text tokens in one decoder → audio-to-audio conversation
- Only images
- Impossible
Audio: Whisper (STT enc-dec) / speech LLMs (audio tokens in decoder) / TTS (LLM emits audio tokens).
#multimodal
Practise LLMs & GenAI
214 interview questions in this topic.