Why is high-cardinality label bad for Prometheus?
hardAnswer
- Each unique label combination = separate time series.
- Explosive:
latency{user_id=..., request_id=...}→ millions of series → memory blowup. - Rule: labels ≤ 10 unique values, aggregate at query time.
- High-cardinality data belongs in logs / traces, not metrics.
- Modern alternatives: VictoriaMetrics / Mimir handle better; ClickHouse for OLAP-style.
- Or use histograms + percentiles instead of raw values.
Check yourself — multiple choice
- Random
- Each unique label combo = separate series → millions blow up memory; keep labels ≤10 unique values, aggregate at query; high-cardinality → logs/traces not metrics; VictoriaMetrics / Mimir / ClickHouse alternative
- Not real
- Just add
Cardinality: label combos explode; keep low; high card → logs/traces.
#observability#infrastructure
Practise MLOps & Data Quality
215 interview questions in this topic.
Related questions
- Distributed tracing for ML inference — why?
- Online vs offline feature store — architecture.
- How do you compute real-time features (streaming aggregates)?
- Orchestration tools for ML pipelines — Airflow vs Prefect vs Dagster.
- Why partition ML training tables by date?
- What is an 'asset' in Dagster and why is it useful for ML?