Word2Vec skip-gram — objective and training.
mediumAnswer
- For each target word, predict surrounding context words (window size ~5).
- Softmax over full vocabulary too expensive → negative sampling: for each positive (target, context) pair, sample k random 'negative' words and train binary logistic.
- Learns dense word vectors capturing distributional semantics.
- Levy & Goldberg 2014: mathematically equivalent to implicit factorization of shifted PMI matrix.
Check yourself — multiple choice
- Random
- Predict context from target; negative sampling replaces full softmax; equivalent to implicit shifted-PMI matrix factorization (Levy & Goldberg)
- Same as GPT
- Not real
Word2Vec skip-gram: predict context with negative sampling; ≡ PMI factorization.
#representation-learning#nlp
Practise Unsupervised Learning
214 interview questions in this topic.