A 5 TB a day observability bill is a parsing decision, not a volume problem

Where the bill actually comes from

A five terabyte day is 150,000 gigabytes a month. At Grafana Cloud's published self-serve log rates, 0.400 dollars per gigabyte to write and 0.100 to retain, that is 60,000 dollars and 15,000 dollars a month before volume discounts, plus 0.050 per gigabyte to process, which adds 7,500. The total is about 82,500 dollars a month, and 91 per cent of it sits in the write and retain lines.

Three terabytes a day lands near 49,500 dollars a month and ten terabytes a day near 165,000 on the same arithmetic. The rates are published starting points and the page says automatic volume discounts apply, so the absolute figure is soft. The ratio is not, and the ratio is what a buyer can act on in a renewal.

Retention multiplies the same three lines. Extending it, or keeping a second copy in another region, is a pricing decision as much as a compliance one, because the write and retain charges follow every gigabyte that arrives.

Metrics bill by series, not by volume

Logs and traces are billed by the gigabyte. Metrics are billed by active series, and the same page lists 6.50 dollars per 1,000 series with a platform fee of 19 dollars a month. That changes what a cardinality problem costs, because a new attribute on a metric adds no bytes; it adds series, and series are billed for as long as they exist.

The CNCF walkthrough makes the size of that visible. Twenty-five agent sessions produced 300 series across eight metrics, because every data point carried a session id, a user id, a user email and two account identifiers. At a few hundred sessions a day the series count becomes the bill, not the log volume.

Grafana's own page prices the fix as well as the problem. Adaptive Metrics is offered as a way to cut the series count by up to 80 per cent and Adaptive Logs as up to 50 per cent off ingest, which are vendor estimates rather than customer results, but they set the order of magnitude a collector is aiming at.

Vendor agents bill by the gigabyte they receive

The vendor side is not mysterious. Datadog's pricing page bills log management per gigabyte of uncompressed data ingested for processing, and bills forwarding per gigabyte outbound per destination, while APM includes 150 gigabytes of span ingestion per APM host per month. The page checked for this article did not state dollar rates in the served markup, so the rate is a quote; the unit is not.

That distinction is where a collector changes an invoice. An agent sends what the application emits. A collector sends what survives its processors, and the vendor never sees the difference in the bill.

The APM allowance is the exception worth knowing. Datadog includes 150 gigabytes of span ingestion per APM host per month, summed across every host, so two hosts that ingest 250 and 50 gigabytes between them pay nothing for ingestion that month. A fleet of quiet hosts sits well inside it; a few chatty ones do not.

Three signals, three units, one decision

The comparison that matters is not one vendor against another. It is which signal carries the volume, and which processor can shrink it without deleting the answer to a question someone actually asks.

What the collector actually removes

The CNCF post is a worked example at small scale. One coding agent run with every content flag on produced 181 telemetry records and 289 kilobytes of JSON, and a 40 kilobyte file read by a tool became a 40 kilobyte span event, with the same content written again to logs. Many teams store that twice and query neither copy.

The fix is a filter processor that drops a whole event type and a processor that deletes payload attributes. Filtering the api_request event in that walkthrough removed roughly a quarter of log records. Applied to the five terabyte estimate, a 25 per cent cut takes 82,500 dollars to about 61,875 a month, which is 20,625 dollars for a configuration change and a week of checking that no dashboard breaks.

The test in the article is the one to copy: if no dashboard, alert or query reads a field, it is storage being paid to be indexed. That is a defensible rule for a platform lead and an uncomfortable one for an application team that switched on debug logging to answer a question last quarter.

Attribute trimming is the cheaper half of the work. Dropping a user email from a metric deletes no log line, but it stops a session identifier from creating a new series every time somebody signs in, and it is the change least likely to break a query anyone runs.

The decision rule

Keep the vendor agent where the vendor's own correlation is the product, the detections or the tracing interface, and where the volume is a small fraction of the bill. Put a collector in front where volume is dominated by a few high-cardinality fields or payloads, because that is the part the invoice charges for and the part a processor can remove.

Measure before arguing. Take one day of production telemetry, run it through a collector in front of the same backend, and compare bytes per minute and series count on both paths. The gap between those two numbers is the negotiation, and it is the only figure worth taking to a renewal.

The uncomfortable part is that the cheapest collector configuration is usually the one an application team did not choose. That is why the filtering rules belong to the platform team, which sees the invoice, while the debug flags stay with the team that sees the bug.