Skip to content
keepcadence docs

Runbook: RunPod serverless

Service overview

Section titled “Service overview”

Three RunPod serverless endpoints per environment:

  • kc-asr-{env} — Whisper / ASR
  • kc-tts-{env} — IndexTTS2 / CosyVoice2
  • kc-render-{env} — FFmpeg render

All endpoints are invoked via @keepcadence/worker-provider-runpod.

Symptoms → Diagnosis → Fix

Section titled “Symptoms → Diagnosis → Fix”

All jobs failing immediately (no cold-start)

Section titled “All jobs failing immediately (no cold-start)”

Diagnosis

  1. Check RUNPOD_API_KEY is valid: curl -s https://api.runpod.io/graphql -H "Content-Type: application/json" -d '{"query":"{ myself { id } }"}' -H "Authorization: Bearer $RUNPOD_API_KEY".
  2. Check RunPod dashboard for account-level issues (billing, quota).

Fix

  • Rotate RUNPOD_API_KEY if expired — generate a new one in RunPod → Settings → API Keys.
  • Update Coolify env var for orchestrator.

Endpoint returns ENDPOINT_NOT_FOUND

Section titled “Endpoint returns ENDPOINT_NOT_FOUND”

Diagnosis

  • The endpoint ID in env var RUNPOD_ENDPOINT_{ASR,TTS,RENDER} does not match a live endpoint.

Fix

  • Re-deploy the endpoint in RunPod dashboard (see W10-016/017 operator runbook).
  • Update the endpoint ID in Coolify env vars.

Worker keeps restarting (container crash loop)

Section titled “Worker keeps restarting (container crash loop)”

Diagnosis

  1. Open RunPod dashboard → Endpoint → Worker logs.
  2. Common causes: CUDA version mismatch, missing model artefact, Python import error.

Fix

  • Model artefact missing: re-upload to R2 and trigger endpoint re-pull.
  • Image mismatch: rebuild and re-push the worker Docker image; update endpoint image tag.
  • OOM: upgrade GPU tier in endpoint settings.

Cold-start latency unacceptable in production

Section titled “Cold-start latency unacceptable in production”

Fix

  • Set min_workers: 1 on the production endpoint (steady-state cost accepted for MVP).
  • Use max_workers to cap cost.

Useful references

Section titled “Useful references”