The First Alarm That Fired Was the Invoice

In June 2026, AWS Cost Anomaly Detection flagged a spend spike on the storage side of SoundMint, my music platform. I opened Cost Explorer expecting a runaway bucket, some multi-gigabyte accident of my own making. What I found instead was the strangest bill I have ever read.

The storage line was $0.02. Two cents. The spike was almost entirely a single usage type: Requests-GIR-Tier2, which came to $110.12 on 11,012,052 requests, per my incident record. GIR is S3 Glacier Instant Retrieval, the archive tier. The archive in question held roughly 537 objects, somewhere between 6 and 8 GB of old audio files. Something was reading that tiny, cold dataset about 1.27 million times a day.

I diagnosed the class of the problem before I read a single line of code, and the tool that got me there was not a tracing platform or a log aggregator. It was the shape of the bill.

Machines are flat, humans have bedtimes

The daily request graph told the story on its own. Requests sat near zero through June 13, stepped up to 804,000 on June 15, then settled into a dead-flat plateau of about 1.27 million requests a day that ran unbroken through June 23.

Human traffic breathes. It peaks in the evening, sags overnight, swells on weekends or shrinks on them, depending on the product. Real users produce a diurnal curve because real users sleep. A perfectly flat, 24/7 plateau with no daily rhythm at all is the signature of a scheduled machine process. That one observation eliminated users, crawlers, and abuse from the suspect list before I opened an editor.

The second signature was the ratio. Storage was effectively free while requests were the entire bill, which means a very small dataset was being read absurdly often. Small, cold, hammered on a clock: that is a reconciler or a scanner in a hot loop. From there I knew exactly which subsystem to open.

Three stacked defects

The culprit was a cron I had shipped nine days earlier: a storage-tier reconciler, driven by an EventBridge rule on a 15-minute schedule that called an internal endpoint on the production Go Lambda through an API destination. Its job was to detect drift between the storage class recorded in the database and the storage class S3 actually held. Reasonable job. Three independent defects turned it into a runaway.

Defect A: the reconciler touched objects it had no reason to touch. Each tick, the reconcile routine selected up to 100 files ordered by the time their storage class last changed, nulls first, with no storage-class filter, and issued a HeadObject against each one to check for drift. Nulls-first ordering front-loaded exactly the wrong rows: idle objects from 2024 that were already settled in Glacier Instant Retrieval and were never going to drift anywhere. Every one of those HEADs was billed at Glacier request rates, and re-billed on every rotation through the table, forever.

Defect B: a timeout turned 96 ticks into a retry storm. The EventBridge API destination gives an endpoint roughly five seconds to respond. My handler had a 25-second time budget, sized for the Lambda's 30-second limit, not for EventBridge's patience. So every single tick timed out from EventBridge's point of view: 96 invocations a day, 96 recorded failures, a 100 percent failure rate. And because I had set no retry policy on the rule target, EventBridge applied its default of up to 185 retries over 24 hours, while the API destination's rate limit of one invocation per second paced those retries into a continuous stream. Lambda invocations stepped from about 250 a day to somewhere between 16,000 and 20,000. Per my incident record, the arithmetic was 96 ticks, amplified by roughly 165 retries each, at about 80 Glacier HEADs per invocation: approximately 1.27 million Glacier HEADs a day. Each of those 537 sleepy objects was being HEADed about 2,360 times daily.

Defect C: the pricing model I had never priced. Here is the billing fact the whole incident turns on: S3 bills a HeadObject or GetObject against a Glacier-class object as a Tier-2 request, at $0.01 per 1,000 versus $0.0004 per 1,000 for Standard. Roughly 25 times the price. The entire storage saving from keeping that 6 to 8 GB in Glacier Instant Retrieval instead of Standard was about $0.12 a month. Even the happy path, 96 ticks at 100 HEADs each, meant paying premium request rates every day to re-verify a twelve-cent saving. The bug made it a fire; the economics were already upside down.

Each layer was survivable. The stack was not.

What makes this incident worth writing up is not any single defect. It is that every defect, alone, would have been fine.

The unfiltered reconciler on its own produces 9,600 Glacier HEADs a day: a few dollars a month, invisible. The retry storm on its own, pointed at Standard-class objects, produces a pile of requests at $0.0004 per 1,000: annoying, cheap. Glacier Instant Retrieval on its own, for data nothing scans, is genuinely near-free to store. Each layer passed the "is this fine?" test individually. Stacked, they multiplied into a run-rate of about $381 a month, roughly $4,600 a year, per my incident record, to read the coldest data I own.

That is layered failure without layered monitoring. There was no alarm on EventBridge failed invocations. No dead-letter queue, so there was nothing to alarm on. The Lambda duration metrics pegged at 25 seconds for nine days and nobody was paged, because no page existed. The only system that noticed was billing, and billing noticed on day nine.

What changed

The immediate mitigation was blunt: disable the reconcile rules in both environments and watch the request graph collapse. It fell from 1.27 million to 190,000 the next day, then toward zero. The structural fixes came in one deploy.

First, a cost guard in the reconciler. Files already at rest in a cold storage class consistent with their tier policy are skipped entirely; no HEAD, no charge. The subtle part is what the guard still checks: a file whose policy says it should be hot but which has drifted into a cold class still gets HEADed and healed back to Standard. The obvious fix, a blanket filter excluding all Glacier rows from the query, would have quietly abandoned any playable file stuck cold. The guard keys on both the storage class and the intent, so the cost case is skipped while the safety case still flows.

Second, a retry policy of zero attempts plus a dead-letter queue on the EventBridge target. A failed tick is now dropped to the DLQ instead of retried into a storm, and the next 15-minute tick is the real retry. Worst-case amplification is capped at exactly one.

Third, a CloudWatch alarm on DLQ depth wired to an ops topic. If the endpoint ever fails every tick again, the queue fills within the hour and I get paged in minutes, not billed in nine days. I also took the lesson upstream into budgets: Cost Anomaly Detection caught this on day nine, but an alarm sitting directly on the Glacier request-count metrics, or an AWS budget scoped to S3 request spend rather than total spend, catches the same failure in minutes. Request classes deserve their own budgets, because request spend is where bugs show up. Storage spend is where data shows up.

And the incident forced a strategy question I should have asked at design time: why does this machinery exist at all? A reconcile cron, minimum-stay tracking, and drift detection are a lot of bespoke, money-touching code to capture twelve cents a month. S3 Intelligent-Tiering does the same job natively: its archive instant access tier has the same storage price and the same millisecond latency as Glacier Instant Retrieval, with no retrieval fee and no request premium on instant-access tiers, for a monitoring fee of $0.0025 per 1,000 objects. Adopting it retires the reconcile loop entirely, which makes this whole incident class structurally impossible rather than merely alarmed.

The bill is telemetry you already receive

Here is the general claim I took away, and the reason for the title. Your cloud bill is sampled, aggregated telemetry that nobody instruments and everybody receives. It is slow, a day or more behind. It is coarse, bucketed by usage type rather than by request. But it is also complete in a way almost nothing else in your stack is. Every billable API call is counted. There are no sampling decisions to get wrong, no log statements to forget, no dashboards that only exist for the subsystems you remembered to care about. The reconciler had no metrics of its own worth reading, and its failures sat in a queue of nobody. The bill counted every single one of its 11 million requests anyway.

The skill is reading the bill's shape, not its total. A flat plateau means a machine; a diurnal curve means people. A request bill towering over a storage bill means a small dataset in a hot loop. A single usage type carrying the whole spike means one subsystem, not general growth. None of that requires tooling. It requires treating the invoice as a graph instead of a number.

Two habits made it operational for me. Cost Anomaly Detection is on for every account I run, because a free alarm with nine-day latency beats no alarm. And any per-object S3 operation inside a loop that scans a table now gets treated as a cost multiplier: objects, times frequency, times retries. That third factor is the one I had never written down, and it was worth about $4,600 a year, per my incident record.

I have written before about outages where the alarms never fired. This was the inverse: an alarm that fired reliably, on schedule, from the one system I had never thought of as monitoring. The bill was watching the whole time. I just had to read it.