Usage Metering & Billing Aggregator

Meter anything. Bill exactly once. A production-grade Go pipeline for high-throughput, idempotent usage metering — from raw tenant events to Stripe usage invoices that survive crashes, retries, and clock skew.

---

📌 Contents

• Why this exists
• Key features
• System architecture · Component breakdown
• Request lifecycle · The billing chain
• Money-critical invariants
• Tech stack · Project structure
• Quick start · Configuration · API surface
• Testing · Interactive diagrams

---

🎯 Why this exists

Usage-based billing looks simple until you ship it: producers retry storms, clocks skew, workers restart mid-window, and Stripe redelivers webhooks. Every one of those is a double-bill or lost-revenue incident waiting to happen.

This system closes that gap with a three-stage pipeline — ingest → aggregate → report — where every hop is idempotent and every aggregate is reproducible from the append-only event log.

✨ Key features

• 🧾 Zero-loss idempotent ingestion — Redis SET NX PX 24h single-writer reservations backed by a Postgres UNIQUE (tenant_id, metric_code, idempotency_key) index; replays answer 200 deduplicated without touching a counter.
• 🧮 Continuous windowed rollups — SUM, MAX, and LAST aggregation per billable metric; only closed windows are ever computed, so partial totals can never reach an invoice.
• 🔐 Atomic quota enforcement — a single Redis Lua script checks and increments per-cycle usage: rejected traffic never inflates the counter (429 QUOTA_EXCEEDED is non-billable), soft-limit (80%) notifications fire exactly once per cycle.
• 💸 Exactly-once Stripe sync — deterministic SHA-256(tenant ‖ metric ‖ window_start) event identifiers + a persisted checkpoint watermark, so a worker crash mid-pass can neither skip nor double-report a window.
• 🔁 Replay-safe billing webhooks — signature-verified subscription.updated / invoice.created events with a 72-hour processed-ID guard against redelivery.
• 🧪 Load-tested realism — a 500-tenant traffic simulator with 15% duplicate replay injection, plus an end-to-end stress suite asserting 100% aggregation accuracy and zero reconciliation drift.

---

🏗️ System architecture

🧩 Component breakdown

Trust boundaries. Tenant traffic stops at the gateway: nothing a client sends mutates aggregates directly. Redis holds only ephemeral correctness state (24h TTL keys, cycle counters) — Postgres is the durable ledger every billable number can be rebuilt from. Stripe sits entirely outside the trust boundary: the syncer treats it as a network of record for money and guards it with deterministic identifiers and persisted watermarks, never with hope.

---

🔄 Request lifecycle: ingestion API

1. Decode & validate — single event or {"events":[…]} batch; quantity must be positive and finite, timestamp within ±24h skew (internal/domain/event.go:101).
2. Quota gate — tenant plan is resolved from the catalog, then one Lua script atomically checks-and-increments the cycle counter (internal/aggregation/quota_checker.go:33). Breaching a non-overage hard limit returns 429 and the counter is never moved.
3. Idempotency reservation — SET NX per (tenant, metric, idempotency_key) collapses duplicate submissions across all replicas before any write path is touched.
4. Acknowledge — 202 with counts for new events; 200 deduplicated for replays. Clients can retry with abandon.
5. Asynchronous persistence — the in-memory buffer flushes multi-row inserts (2048 rows or 1s), with the Postgres unique index as the second dedup layer if Redis fails open (internal/ingestion/buffer.go:84).

💰 The billing chain: rollups → Stripe

1. The aggregator recomputes every fully-closed window (hourly/daily) in a 48h lookback — late-arriving events settle into the correct bucket, and re-running is idempotent (internal/aggregation/engine.go:87).
2. The syncer loads rollups newer than the tenant/metric checkpoint, skips already-committed windows, and builds a payload whose Identifier is SHA-256(tenant ‖ metric ‖ window_start) (internal/domain/sync.go:177).
3. Meter events go out oldest-first; a Stripe response commits the checkpoint (internal/repository/checkpoint_repo.go:58). A crash between send and commit can at worst re-send the same idempotent identifier — Stripe collapses it.
4. A 5-minute latency after window close keeps in-flight rollup updates from racing the report; open windows are never reported.
5. Reconcile compares local rollups against reported watermarks and surfaces drift for finance review (internal/stripe/syncer.go:282).

---

🛡️ Money-critical invariants

---

🧰 Tech stack

📂 Project structure

.
├── cmd/                        # three independently deployable binaries
│   ├── ingestion_api/          #   high-throughput event gateway (:8080)
│   ├── aggregator_worker/      #   continuous window rollups
│   └── stripe_syncer/          #   exactly-once meter-event reporting
├── internal/
│   ├── domain/                 # UsageEvent · Rollup · plans · Stripe payloads (pure)
│   ├── ingestion/              # validator · Redis dedup · batch buffer
│   ├── aggregation/            # windowed rollup engine · Lua quota gate
│   ├── stripe/                 # resilient SDK client · checkpointed syncer
│   ├── repository/             # pgx pools · event/checkpoint/catalog stores
│   └── transport/              # chi handlers & routes · webhook listener
├── migrations/                 # 000001_init_metering_schema.up.sql
├── simulator/                  # 500-tenant burst + 15% replay traffic generator
├── tests/                      # dedup · windows · quota · sync · API · e2e stress
├── docs/
│   ├── diagrams/               # Archify specs (architecture + sequence)
│   └── README.md               # interactive-diagram regeneration guide
├── docker-compose.yml          # postgres:16 :5434 · redis:7 :6381
├── Makefile                    # up · migrate · run-* · simulate · test-race · e2e
└── go.mod                      # module usage-metering

---

🚀 Quick start

Prerequisites: Go ≥ 1.27 (as declared in go.mod), Docker, and optionally make + jq.

1 — Clone & boot the data plane

git clone https://github.com/<owner>/usage-metering.git
cd usage-metering

make up        # postgres:16 → :5434 · redis:7 → :6381 (health-checked)
make migrate   # apply 000001_init_metering_schema.up.sql

(No make? Use docker compose up -d, then load the SQL file into the metering database.)

2 — Run the three services (POSIX shell; PowerShell uses $env:VAR=…)

# terminal 1 — API gateway; seed a demo fleet so FKs resolve and tiers mix
SEED_TENANTS=25 SEED_TIER=FREE make run-ingestion     # listens on :8080

# terminal 2 — rollup engine (hourly windows, 30s cycle)
make run-aggregator

# terminal 3 — Stripe reporter (no STRIPE_API_KEY set → safe dry-run sink)
make run-syncer

3 — Emit metered traffic

make simulate    # 500 tenants · bursts · 15% duplicate replays

# or send one event by hand (quantity is a decimal string on the wire)
curl -sS localhost:8080/api/v1/usage -H 'content-type: application/json' -d '{
  "tenant_id": "<uuid from GET /api/v1/tenants>",
  "metric_code": "llm_tokens",
  "idempotency_key": "evt-0001",
  "quantity": "1500",
  "event_timestamp": "'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'"
}'

4 — Read the money back

TENANT=$(curl -s localhost:8080/api/v1/tenants | jq -r '.[0].id')
curl -s "localhost:8080/api/v1/tenants/$TENANT/usage?window=hour&metric=llm_tokens" | jq
curl -s localhost:8080/readyz

Re-send evt-0001: the response flips to 200 {"status":"deduplicated"} and every aggregate stays frozen. That is the whole promise of the system, in one curl.

---

⚙️ Configuration

Every binary self-configures from environment variables with sane defaults (docker-compose.yml ports are the defaults).

Billable metrics: llm_tokens (SUM) · api_calls (SUM) · storage_gb (MAX) · active_conns (LAST).

🔌 API surface

---

🧪 Testing

make test        # go test -v ./...           (unit + integration, miniredis-backed) — currently passing
make test-race   # go test -race -count=1     (concurrency safety of buffer & gates)
make e2e         # burst → dedup → rollups → Stripe sync stress, repeated pass, drift check
make vet && make build

The e2e suite drives the real HTTP surface (in-process router) with high-concurrency traffic and 15% duplicate replay injection across FREE / PRO / ENTERPRISE tiers, then asserts: 202/200/429 response semantics, zero double-counted events, 100% aggregation accuracy for closed windows, zero double-charges on a repeated syncer run against the in-memory meter sink, and zero reconciliation drift.

🖇️ Interactive diagrams

The Mermaid diagrams above render natively on GitHub. Full interactive versions (guided views, focus/search, dark/light, PNG/SVG export) are one command away — specs live in docs/diagrams/:

node <archify>/bin/archify.mjs finalize architecture docs/diagrams/architecture.candidate.json docs/architecture.html --quality showcase --json
node <archify>/bin/archify.mjs finalize sequence     docs/diagrams/sequence.candidate.json     docs/sequence.html     --quality showcase --json

See docs/README.md for details and known host constraints.

---

Built with a single rule: every hop must survive its own retry.

⭐ Star the repo · 🐛 Open an issue · 📣 PRs welcome — see Contributing below.

🤝 Contributing

1. make up && make test-race before touching anything; the race detector is part of the contract.
2. New billable metrics: register the code + aggregation type in the catalog and extend domain — never hand-roll aggregation in handlers.
3. Money-path changes (dedup, quota Lua, checkpointing, Stripe payloads) require a test that fails before the fix.

📜 License

No LICENSE file is committed yet — add one before publishing the repository (e.g. MIT or Apache-2.0) and update this section.