ABR Video Transcoding Pipeline

Turn one uploaded master into a GOP-locked, multi-bitrate HLS ladder — published atomically, with zero 404s.

---

📖 Overview

Most "video transcoding" demos produce four MP4 files and call it done. Real players need more: every rendition must place its I-frames on the exact same timeline, or a bandwidth switch freezes the picture and desyncs audio. And the moment a player can see master.m3u8, it will immediately request every segment the manifest references — so publishing a manifest before its chunks exist guarantees a wall of 404s.

This project is the production-grade version of that pipeline, written in Go with no web framework and no transcoding service in the middle:

---

✨ Core Features

• 🧬 Strict keyframe alignment — GOP = round(fps × segmentDuration), forced with both discrete flags and -force_key_frames expr:gte(t,n_forced*2) as belt-and-braces.
• 📐 Four-rung ABR ladder — 1080p / 720p / 480p / 360p, validated at runtime: even dimensions for yuv420p, strictly descending bitrate, maxrate ≥ b:v, legal H.264 profiles.
• 🔽 Upscaling guard — SelectLadder drops every rung above the source height, so a 720p source never gets a pointless 1080p encode.
• 📦 fMP4 / CMAF packaging — init.mp4 + seg_%05d.m4s, -hls_flags independent_segments, each media playlist validated locally before upload.
• 🧾 Custom master compiler — emits #EXT-X-STREAM-INF with BANDWIDTH, AVERAGE-BANDWIDTH, RESOLUTION, FRAME-RATE and a correct CODECS string per rung, sorted by bandwidth descending.
• 🗄️ Auditable artifacts — every rendition's segment count and byte total is persisted to video_renditions, and the uploaded byte count is compared against the local file size.
• 🔍 Deep media probing — ffprobe JSON parsing with a rational-tolerant decoder, rotation metadata, display-dimension correction, B-frame detection, and keyframe timestamp extraction for verification.
• 🪝 Signed webhooks — completion/failure events signed with HMAC-SHA256 in X-ABR-Signature, exponential backoff to 4 attempts, redirects deliberately not followed so the signature never leaks.
• 🧹 Path-traversal defence — client filenames are sanitized to a bare base name with control characters and reserved separators stripped before they become part of an object key.

---

🏗️ System Architecture

🔍 Component breakdown

1. Ingestion plane — cmd/api_gateway

A single net/http server on :8080 built from the stdlib ServeMux. It applies migrations on boot, waits for the object store, and exposes six API routes plus two health probes. It never proxies media bytes — Ingest inserts the video_assets row before signing so the upload key is reserved even if the client never follows through, then returns a presigned PUT valid for 15 minutes.

2. Work plane — cmd/transcode_worker

Wires the JobProcessor to a worker pool and runs two maintenance loops. The handler wraps Process so job outcomes reach Postgres and the customer webhook no matter how the worker later exits, using a detached context. On SIGTERM it drains for WORKER_SHUTDOWN_GRACE (30 s) — a wedged encode cannot hold the process open, because the FFmpeg watchdog and the queue visibility timeout let another worker recover the abandoned job.

3. Media engine — internal/ffmpeg

Three focused units, each independently testable:

• probe.go — wraps ffprobe -print_format json, decoding the many ways ffprobe emits numbers (bare or quoted, N/A, 0/0). Normalizes rotation to {0, 90, 180, -90} and swaps display dimensions for ±90. KeyframeTimestamps re-probes with -skip_frame nokey so alignment is verified against real media, not assumed.
• orchestrator.go — runs the binary with pipes, parses time= / fps= / bitrate= / speed= frames into a Progress struct, keeps the last 40 stderr lines for diagnostics, and enforces WatchdogTimeout = 2 × media duration (30 min floor). Error precedence is deliberate: timeout beats context-cancellation beats exit code.
• packager.go — invokes the HLS muxer for fMP4 packaging and compiles the master playlist. ValidateMediaPlaylist re-reads the emitted playlist from disk and confirms the #EXTM3U header, a resolvable non-empty #EXT-X-MAP init segment, at least one segment URI, #EXT-X-ENDLIST, and version ≥ 3.

4. Sandbox — internal/ffmpeg/sandbox_{unix,windows}.go

Build-tagged, and both variants enforce the same ResourceLimits:

• Linux — prlimit --as=<bytes> --cpu=<seconds>, Setpgid for process-group isolation, SIGKILL to -pid with single-PID fallback. If prlimit is absent the runner degrades to no rlimits and says so, rather than pretending.
• Windows — a Job Object with JOB_OBJECT_LIMIT_JOB_MEMORY, JOB_OBJECT_LIMIT_JOB_TIME, and always KILL_ON_JOB_CLOSE; termination via TerminateJobObject.

5. Queue plane — internal/queue

No Redis Streams, no BRPOPLPUSH — a purpose-built structure that keeps every recovery path explicit:

A maintenance loop promotes due retries every 5 s; a second recovers jobs whose claim expired (backoff base 5 s, capped at 15 min). Progress reporting is deliberately best-effort — a Redis hiccup must never fail a healthy encode.

6. Atomic publisher — internal/storage/atomic_writer.go

The component that makes the zero-404 claim true. Publish is strictly two-phase:

1. StageManifest — PutObject to master.m3u8.tmp. The live key is never written by a PutObject.
2. VerifySegments — HEAD every referenced key, rejecting both absent and zero-byte chunks, and collecting all failures into one error rather than failing on the first.
3. CopyObject — the atomic swap onto the live key.
4. ObjectSize check — the committed body's length is compared against the staged bytes.

With VerifyRemotely the verification runs twice — once at stage time and again immediately before the swap — which closes the window where a segment could vanish mid-publish. A Commit without a prior stage fails closed with "stage before commit".

📌 Design note on CopyObject: S3 overwrite is a single server-side operation, so a reader never observes a truncate-then-rewrite window the way a filesystem PUT would.

7. Persistence — internal/repository

pgxpool with migrations applied from ./migrations in lexical order, each file in its own transaction together with its schema_migrations row, so a failure rolls back exactly one migration. MarkJobStatus stamps started_at only on the first real transition.

Stage legality is enforced in Go, not in SQL: JobStage.Advance permits self-transitions and exactly one step forward, rejecting regressions and skips — so a late status update can never overwrite a job that has already moved past that stage.

🔐 Trust boundaries

---

🔄 Core Workflow — Ingestion to Playback

Step-by-step

1. Reserve — Ingest generates the asset UUID, sanitizes the filename, and inserts the video_assets row with a server-derived key. The row exists before the signature does, so an abandoned upload leaves an auditable UPLOAD_PENDING record rather than nothing.
2. Upload — the client PUTs media directly to media-raw. This is the trust boundary; the gateway stays out of the data path entirely.
3. Gate — Transcode HEADs the master before queueing. Enqueuing work for an object that has not landed yet is the single most common cause of wasted transcode capacity, so it is rejected with 409 upload_incomplete.
4. Claim — the worker BRPOPs across the priority and normal lists in a single call, then HSETs an expiry into the active hash. A heartbeat refreshes it every 30 s while the job runs.
5. Probe — ffprobe yields duration, frame rate, display geometry (rotation-corrected) and codec. UpdateAssetProbe persists the real numbers, and SelectLadder trims the ladder to what the source can actually justify.
6. Encode — for each surviving rung, libx264 runs with locked GOP flags. IsCorruptedMedia inspects stderr to distinguish "bad input file" (permanent) from "transient" (retryable), so a truncated upload dead-letters immediately instead of burning three attempts.
7. Package — the HLS muxer emits fMP4 chunks; the generated media playlist is validated from disk before anything is uploaded.
8. Upload & verify — segments go up via parallel multipart (4 workers × 8 MiB parts), and each uploaded byte count is compared against the local file size. A short upload aborts the job.
9. Publish — the master is staged to .tmp, all segments are HEAD-verified a second time, and one CopyObject commits it live. This is the step that makes the zero-404 guarantee hold.
10. Announce — Postgres records COMPLETED / READY, the claim is acknowledged, and the signed webhook fires on a detached context.

🔁 Failure & recovery paths

Job stage progression is a strict forward-only machine:

QUEUED ──▶ PROBING ──▶ ENCODING ──▶ PACKAGING ──▶ COMPLETED
   │           │            │             │
   └───────────┴────────────┴─────────────┴──▶ FAILED
                      (any stage, via FailJob SQL)

Advance accepts self-transitions and single forward steps only, so a delayed status write can never move a finished job backwards. Note that FAILED is deliberately not reachable through Advance — it is written by FailJob SQL, keeping failure an explicit, audited act rather than an ordinary transition.

---

🧰 Tech Stack

The compose file defaults to rustfs/rustfs because it implements the same S3 surface — identical endpoint paths and AWS SigV4 auth — as the minio/minio image referenced in the design. Switching back is a one-line image change with no code or configuration edits.

---

📁 Project Structure

abr-pipeline/
├── cmd/
│   ├── api_gateway/           # :8080 HTTP server — ingest, status, presigned URLs
│   ├── transcode_worker/      # distributed worker — polls queue, drives FFmpeg
│   ├── migrate/               # apply ./migrations/*.up.sql
│   ├── bootstrap/             # idempotently create media-raw + media-processed
│   └── mediagen/              # synthetic benchmark clips (ffmpeg lavfi)
│
├── internal/
│   ├── domain/                # pure business rules — no I/O, no dependencies
│   │   ├── job.go             #   JobStage machine, retry budget, validation
│   │   └── ladder.go          #   ABR profiles, GOP math, FFmpeg arg builders
│   ├── ffmpeg/                # media engine
│   │   ├── probe.go           #   ffprobe JSON → MediaInfo
│   │   ├── orchestrator.go    #   process runner, progress parsing, watchdog
│   │   ├── packager.go        #   HLS fMP4 muxer + master playlist compiler
│   │   ├── sandbox_unix.go    #   prlimit + process-group isolation
│   │   └── sandbox_windows.go #   Job Object equivalent
│   ├── storage/
│   │   ├── s3_client.go       #   multipart upload, presigning, readiness probe
│   │   └── atomic_writer.go   #   stage → verify → CopyObject (zero-404 core)
│   ├── queue/
│   │   ├── producer.go        #   enqueue, claim, backoff, DLQ, quarantine
│   │   └── consumer.go        #   worker pool, heartbeats, stale recovery
│   ├── repository/
│   │   ├── db.go              #   pool + migration runner
│   │   └── job_repo.go        #   asset / job / rendition persistence
│   ├── pipeline/
│   │   ├── processor.go       #   end-to-end job execution
│   │   └── outcome.go         #   DB + webhook outcome reporting
│   └── transport/
│       ├── http/              #   routes.go · handlers.go
│       └── webhook/           #   HMAC dispatcher with backoff
│
├── migrations/                # 000000_migrations_table · 000001_video_schema
├── simulator/                 # media_generator.go — synthetic test clips
├── tests/                     # gop_alignment · atomic_manifest · transcode_worker · e2e
├── docs/
│   └── architecture.md         # diagrams, invariants & verification matrix
├── testdata/                  # (generated fixtures are gitignored)
│
├── docker-compose.yml         # postgres:16 · redis:7 · rustfs
├── Makefile                   # GNU make entry points
├── make.ps1                   # PowerShell mirror (make is not on Windows)
├── go.mod
└── .env                       # committed local defaults

The dependency direction is strictly inward: cmd/ wires everything, internal/transport and internal/pipeline depend on internal/domain, and internal/domain imports nothing but the standard library. The GOP and ladder rules are therefore unit-testable without a database, a broker, or FFmpeg on the machine.

📚 Full design notes — every invariant, the Redis key layout, the trust-boundary table and a complete verification matrix — are in docs/architecture.md.

---

🚀 Quick Start

Prerequisites

• Go 1.27+
• Docker (with Compose v2)
• FFmpeg and ffprobe on PATH — the only hard dependency for the media tests

1. Clone and configure

git clone https://github.com/Kavryx-Dev/adaptive-bitrate-abr-video-transcoding-packaging-pipeline.git
cd abr-pipeline

A .env with working local defaults is committed. Docker Compose reads it automatically; the Go binaries read the process environment via os.Getenv, and every setting also has a code-level default that matches Compose — so on a stock machine no export is required.

Optional: tune configuration

2. Start infrastructure

make up         # docker compose up -d  → postgres:16, redis:7, rustfs
make bootstrap  # create media-raw + media-processed buckets (idempotent)
make migrate    # apply ./migrations/*.up.sql

Windows (PowerShell)

./make.ps1 up
./make.ps1 bootstrap
./make.ps1 migrate

3. Generate test media

make generate-test-video          # 30s 720p clip → ./testdata/generated
go run ./cmd/mediagen -preset all  # short + standard + long benchmark set
go run ./cmd/mediagen -corrupt     # also emit an undecodable fixture

4. Run the pipeline

go run ./cmd/api_gateway     # terminal 1 — :8080
go run ./cmd/transcode_worker # terminal 2 — needs WORKER_ID if running replicas

5. Drive it end to end

# 1. Reserve a key and get a presigned upload URL
curl -sX POST localhost:8080/v1/assets \
  -H 'Content-Type: application/json' \
  -d '{"filename":"clip.mp4","size_bytes":10485760,"webhook_url":"https://example.com/hook","auto_transcode":false}'

# 2. Upload the bytes straight to object storage
curl -sX PUT "<upload_url>" --data-binary @testdata/generated/testsrc_30s_720p.mp4

# 3. Queue the transcode
curl -sX POST localhost:8080/v1/assets/<asset_id>/transcode

# 4. Poll progress (percent and stage come from Redis)
curl -s localhost:8080/v1/jobs/<job_id>

# 5. Collect renditions and a playback URL
curl -s localhost:8080/v1/assets/<asset_id>
curl -s localhost:8080/v1/assets/<asset_id>/playback

Build binaries

make build     # → ./bin/api_gateway and ./bin/transcode_worker

---

🧪 Testing

make test        # migrate + bootstrap, then the full suite
make test-short  # skips the four long encode + upload cases (tests/*.go, -short)
go vet ./...

The suite is organised around the invariants that actually break players:

Dependency-degradation is deliberate: every Postgres, Redis and S3 requirement degrades to t.Skipf, so only FFmpeg is mandatory. That keeps unit-level tests runnable anywhere while the full integration matrix runs under Compose.

⚠️ One known flake: TestConcurrentReadersNeverSeeTruncatedManifest has intermittently failed with an S3 CRC32 mismatch originating in the local object store's read path under concurrent overwrite — not in the publish logic, which is CopyObject-based. It reproduces rarely and passes on re-run.

---

📄 Design Notes

Why not a framework? The gateway is seven routes over net/http. Adding a web framework would buy routing sugar and cost a dependency tree, an upgrade treadmill, and a layer of magic around ServeMux's path-parameter handling — which this project already uses.

Why not Redis Streams? Streams bring consumer groups and pending-entry lists, but they also bring XAUTOCLAIM semantics that must be tuned to avoid silent message loss. Explicit lists plus an active-claim hash make the recovery path readable in about thirty lines and trivially testable — which is exactly what the queue tests exercise.

Why -force_key_frames and discrete GOP flags? -g/-keyint_min/-sc_threshold 0 define the cadence; the forced-keyframe expression re-anchors it in wall-clock time. Both are emitted so a rounding difference in frame rate cannot let one rung drift by a frame.

Why stage to .tmp at all? Because the failure mode is asymmetric. A manifest that lags its segments costs a retry; a manifest that leads them costs a user-visible 404 wall on a live stream.

---

📄 License

This project is provided as-is for portfolio and educational use. Add a LICENSE file before distributing.

---

Built with Go, FFmpeg, Redis, PostgreSQL and a healthy distrust of truncated manifests.