How do you measure and improve annotation quality?
mediumAnswer
- (1) Inter-Annotator Agreement (IAA): Cohen's κ (2 annotators), Fleiss's κ (3+), Krippendorff's α.
- Below 0.6 = poor, above 0.8 = strong.
- (2) Gold-standard test tasks to score annotators.
- (3) Redundant annotation with adjudication for critical labels.
- (4) Clear guidelines + examples + edge cases.
- (5) Iterative refinement of guidelines based on disagreement analysis.
- (6) Regular calibration meetings.
- (7) MACE / dawid-Skene for probabilistic label aggregation.
Check yourself — multiple choice
- Random
- IAA (Cohen/Fleiss κ, Krippendorff α) + gold-standard tests + redundant annotation + clear guidelines + iterative refinement + MACE/Dawid-Skene aggregation
- Not needed
- Just count
Annotation quality: IAA + gold + redundancy + guidelines + aggregation.
#data-quality
Practise MLOps & Data Quality
215 interview questions in this topic.
Related questions
- What is data drift and how do you detect it?
- How is concept drift different from data drift?
- What does 'data imbalance' mean and why is it a problem?
- What are common sources of data leakage in an ML pipeline?
- How do you validate data quality in an ML pipeline?
- What is schema drift and how do you detect it?