PMI and PPMI — what they measure.
hardAnswer
- PMI(x, y) = log[P(x, y) / (P(x) P(y))].
- Positive → co-occur more than random.
- Negative → less than random.
- PPMI = max(0, PMI) — drops negative values, more stable.
- Uses: word association (Church & Hanks), building sparse word co-occurrence matrices before SVD (LSA); implicit target of Word2Vec (Levy & Goldberg).
- Foundation of distributional semantics.
Check yourself — multiple choice
- Random
- PMI(x,y) = log[P(x,y)/(P(x)P(y))]; positive = co-occur more than random; PPMI = max(0, PMI); foundation of distributional semantics + implicit target of Word2Vec
- Same as KL
- Not real
PMI: log co-occurrence lift; PPMI = clipped; distributional semantics foundation.
#nlp#theory
Practise Unsupervised Learning
214 interview questions in this topic.