What is a large language model, in one paragraph?
easyAnswer
- An LLM is a large transformer trained on massive text (and often code/multimodal data) to predict the next token given the previous ones.
- Scale of parameters and data lets it acquire broad linguistic and world knowledge.
- After pretraining, it is typically instruction-tuned and aligned (SFT + preference optimization such as RLHF or DPO) to behave as a helpful assistant.
- At inference it generates tokens autoregressively from a prompt.
Check yourself — multiple choice
- LLMs are trained to classify sentences into topics
- LLMs are transformers trained on next-token prediction at scale
- LLMs are recurrent networks with fixed context
- LLMs cannot be fine-tuned
Next-token prediction on massive corpora is the training objective.
#fundamentals
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- Walk through how one training step of an autoregressive LLM works.
- How does Byte-Pair Encoding (BPE) tokenization work?
- Why don't LLMs simply use character-level tokenization?
- What does 'emergence' mean in LLM capabilities and is it real?
- Why do modern LLMs use decoder-only architectures?
- What is SwiGLU and why do LLMs use it in the MLP block?