Runbook: ASR (Automatic Speech Recognition)

Service overview

ASR runs as a RunPod serverless endpoint. The orchestrator dispatches asr jobs via worker-provider-runpod, polls for results, and stores transcripts in media_job.

Symptoms → Diagnosis → Fix

Job stuck in pending state > 5 min

Diagnosis

1. Check RunPod endpoint health: RunPod dashboard → Endpoints → kc-asr-{env} → Status.
2. Check cold-start queue depth — RunPod shows “queued” vs “running” workers.
3. Check orchestrator logs: docker logs keepcadence-orchestrator | grep asr.

Fix

• If endpoint is disabled: re-enable in RunPod dashboard.
• If no workers available: scale min workers from 0 → 1 in endpoint settings (incurs idle cost).
• If orchestrator is not polling: restart orchestrator container.

Transcript quality degraded (gibberish / wrong language)

Diagnosis

1. Check media_job.metadata for the failing job — confirm language field is set correctly.
2. Check input audio duration — files > 2 h may exceed RunPod timeout (default 3600 s).

Fix

• Pass explicit language hint in the job payload.
• Split long files before dispatch (orchestrator pipeline should do this; check orchestrator-pipelines).

RunPod returns 5xx

Diagnosis

1. Check RunPod status page: https://status.runpod.io
2. Check endpoint logs in RunPod dashboard.

Fix

• Transient 5xx: orchestrator retries automatically (exponential back-off, 3 attempts).
• Persistent 5xx: open RunPod support ticket; consider failover to a secondary endpoint.

Useful queries

-- Jobs stuck in ASR processing
SELECT id, status, created_at, metadata->>'error' AS error
FROM media_job
WHERE job_type = 'asr' AND status IN ('pending', 'processing')
ORDER BY created_at
LIMIT 20;