Reducing Log Ingestion and Storage Expenses

Logs consume 50-65% of total observability spend. Teams that apply systematic optimization typically cut 40-70% without losing visibility.

Cost Drivers

Ingestion: $0.05-0.50/GB depending on vendor. Indexing: 2-5x more than raw ingestion (Datadog, Splunk charge separately). Retention: $0.019-0.10/GB/month for hot storage.

The Reduction Playbook

Step 1: Categorize Sources

Run a log source inventory. Group into: critical operational (never cut), useful for debugging (sample aggressively), infrastructure noise (filter/drop), legacy/unknown (investigate and likely drop). Typically 30-50% of volume is never queried.

Step 2: Pipeline Filtering

Use OTel Collector, Vector, or Fluent Bit to filter before ingestion. Dropping health checks and trace-level logs alone saves 10-25%.

Step 3: Log Sampling

Rate-based (1 in N), priority-based (all errors, sample info), or hash-based (by trace_id for complete request context). Savings: 50-90% for sampled sources.

Step 4: Retention Tiering

Tier	Duration	Cost vs Hot
Hot	3-7 days	1x
Warm	7-30 days	0.3-0.5x
Cold (S3)	90-365 days	0.01-0.05x

Step 5: Reduce Log Line Size

Remove redundant K8s metadata (reconstitutable from pod name), shorten field names. 60-70% per-line size reduction.

Vendor-Specific Tactics

• Datadog: Exclusion filters on indexes, archive excluded to S3
• Splunk: Workload pricing for low search:ingest, null queue for noisy sourcetypes
• New Relic: Pre-ingest agent filtering (post-NRDB drops still bill)
• Grafana/Loki: Drop high-cardinality labels, keep streams under 100K

Worked example: 50 GB/day → 28 GB/day in one sprint

Source	Before	Action	After
K8s probe logs	12 GB/day	Drop 200/health in collector	0.3 GB/day
Access logs (static)	8 GB/day	Sample 10% success, 100% 5xx	2.1 GB/day
App DEBUG	15 GB/day	Force INFO in prod	6 GB/day
Audit (compliance)	5 GB/day	Route to cheap S3, not index	5 GB/day ingest, $0 index

Monthly savings (Datadog-style): ~$1,400 ingest + $2,200 indexing = $3,600/mo.

What to do this week

• [ ] Identify top 5 log sources by volume in vendor UI
• [ ] Add collector filter processor for probe/heartbeat patterns
• [ ] Split compliance logs to cold storage
• [ ] Set dynamic log level override for incidents only

Sources & further reading

• Log sampling strategies — structured sampling patterns
• Vector / OTel filter docs — edge filtering examples

---

Related Reading

• Telemetry Cost Optimization
• Log Sampling Strategies
• Controlling Cloud Logging Budgets
• Telemetry Pipeline Optimization

Use the SignalCost Calculator → to model these scenarios with your own numbers.