Runbook: Render pipeline

Service overview

The render pipeline assembles TTS audio + original audio into the final output file. It runs on RunPod serverless (kc-render-{env}) using FFmpeg under the hood. Output is uploaded to kc-media-eu-{env}.

Symptoms → Diagnosis → Fix

Render job stuck in processing > 10 min

Diagnosis

1. Check RunPod endpoint status and worker logs.
2. Check media_job.metadata — look for runpod_job_id and query RunPod API directly if needed.
3. Long renders (> 2 h source audio) are expected to take proportionally longer.

Fix

• If RunPod job shows FAILED: cancel the media_job and requeue (set status = 'pending').
• If RunPod job is gone (expired): same — requeue.

Output file is silent or has wrong duration

Diagnosis

1. Download the R2 output object and run ffprobe -v error -show_entries format=duration.
2. Compare with the source duration stored in media_file.

Fix

• Re-render: reset media_job.status = 'pending' — orchestrator will pick it up.
• If reproducible: check FFmpeg container image version in RunPod endpoint settings.

Output upload fails (R2 error)

Diagnosis

• Check orchestrator logs for S3Error or AccessDenied.
• Verify R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY are valid for kc-media-eu-{env}.

Fix

• Rotate R2 credentials in Cloudflare dashboard → R2 → API tokens.
• Update Coolify env vars for the orchestrator service.

Useful queries

-- Render jobs by status
SELECT status, COUNT(*) FROM media_job WHERE job_type = 'render' GROUP BY status;