Runbook: Cloudflare R2 storage

Service overview

Nine R2 buckets across three environments and three purposes:

• kc-media-eu-{env} — source audio, rendered output, pg_dumps/
• kc-clone-eu-{env} — voice clone model artefacts
• kc-public-eu-{env} — publicly accessible assets (thumbnails, previews)

Symptoms → Diagnosis → Fix

Upload fails with AccessDenied

Diagnosis

1. Confirm the app is using the correct per-env credentials.
2. Check that R2_BUCKET_MEDIA (or CLONE/PUBLIC) matches the env (dev, staging, prod).
3. The per-env S3 keys are scoped — a dev key must fail on kc-media-eu-prod (expected).

Fix

• Rotate credentials in Cloudflare dashboard → R2 → API tokens → manage.
• Update Coolify env vars; redeploy affected services.

Object not found (404) after upload reported success

Diagnosis

• Strong consistency: R2 is strongly consistent, so a 404 after PUT is unusual.
• Check the exact bucket + key used: inspect orchestrator logs for the PUT request.

Fix

• Verify no typo in R2_BUCKET_MEDIA / object key.
• Ensure the upload completed (no partial upload interrupted by timeout).

Bucket over quota / billing alert

Diagnosis

• Cloudflare dashboard → R2 → Usage.
• Common cause: orphaned objects from failed jobs that were never cleaned up.

Fix

• Run the orphan sweep: bun run scripts/r2-smoke.ts --sweep (or wait for the daily scheduled task).
• Adjust BACKUP_RETAIN_DAYS if pg_dump accumulation is the cause.

Public asset not accessible via Cloudflare CDN

Diagnosis

• Check that the object is in kc-public-eu-{env} (not a private bucket).
• Check the R2 bucket’s public access / custom domain setting in Cloudflare dashboard.

Fix

• Enable “Allow public access” on kc-public-eu-{env} if accidentally disabled.
• Purge Cloudflare cache if a stale object is served.

Important: do not delete coolify-host-schne-id

This bucket holds Coolify host backups. It is completely separate from the keepcadence data buckets and must never be touched during cleanup operations.