How do you make an LLM feature feel fast when generation is inherently slow?
mediumAnswer
- Optimize the metric the user feels, which is time to first token rather than total time.
- Stream tokens so reading begins immediately, and the perceived wait collapses to the prefill time.
- Do retrieval and other setup in parallel rather than in series before the call.
- Keep the prompt prefix stable so provider prefix caching cuts prefill.
- Show intermediate progress for agents, since a visible tool step reads as work rather than a hang.
- Where the output is long and structured, render sections as they arrive.
- Speculative decoding helps at the model level, but streaming and parallelism give the larger perceived win.
Check yourself — multiple choice
- Only a bigger GPU helps
- Optimize time to first token: stream output, parallelize retrieval, keep a cacheable prefix, and show agent progress
- Batch requests harder
- Latency cannot be hidden
Perceived latency is dominated by time to first token, so streaming and parallelism matter most.
#latency#production#inference
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- Why is streaming output important in production LLM apps?
- What is prompt caching and when does it help?
- How do you trade off throughput vs latency in LLM serving?
- vLLM vs TensorRT-LLM vs TGI vs SGLang — how do you pick?
- What role does NVIDIA Triton play in LLM serving?
- How do you pick a GPU for serving a 70B LLM?