Managing Metrics Cardinality to Control Observability Spend

A single high-cardinality label can multiply your time series count — and your bill — by orders of magnitude.

Understanding Cardinality

Every unique metric name + label combination = one time series. http_requests_total{service, endpoint, method, status_code} across 20 services x 50 endpoints x 4 methods x 5 statuses = 20,000 series. Add instance (100): 2,000,000. Add user_id (100K): 200 billion.

Detection

Warning signs: single metric >100K series, label with >1000 unique values, cardinality growth >10%/week, "too many series" errors.

Optimization Techniques

Label Pruning

Drop instance from aggregated service metrics = 100x reduction per metric.

Recording Rules

Pre-aggregate high-cardinality into lower-cardinality summaries for dashboards/alerts. Drop raw metric if per-instance granularity isn't needed.

Collection Interval Optimization

60-second scrape for slow-changing metrics (disk, memory) = 6x reduction vs 10-second default.

Metric Allow/Deny Lists

Only collect metrics referenced in dashboards or alerts.

Histogram Bucket Pruning

Default 11 time series per label combo. Reduce to 5-6 relevant buckets.

Prevention

1. Before shipping: Does this metric add labels with >100 unique values?
2. Monthly: Review new metrics for unexpected growth
3. Quarterly: Audit top-50 by cardinality, prune unused labels
4. Automation: Cardinality limits/alerts at collector level

Worked example: one label explosion

http_requests_total{route="/users/:id", user_id="..."} with 50K active users → 50K+ series from one metric.

At $5 per 100 custom series (Datadog list): $2,500/mo for a single metric.

Fix: normalize route template, drop user_id from metric labels, log exemplars in traces instead. Series count → ~200, cost → $10/mo.

What to do this week

• [ ] Export top 20 metrics by series count from vendor
• [ ] Ban user_id, request_id, session_id in label policy
• [ ] Use recording rules / aggregate views for dashboards
• [ ] Add CI check that rejects new high-cardinality labels

Sources & further reading

• Prometheus cardinality docs — label best practices
• RED metrics without full traces — metrics-first SLO patterns

---

Related Reading

• Telemetry Cost Optimization
• Reducing Infrastructure Monitoring Costs
• Telemetry Pipeline Optimization
• Where Your Monitoring Budget Goes

Use the SignalCost Calculator → to model these scenarios with your own numbers.