Runbook: Webhook replay

Service overview

Two external webhook sources:

• Polar — payment/subscription events → POST /webhooks/polar
• RunPod — job completion callbacks → POST /reports/runpod (worker report surface)

Both are idempotent: replaying a webhook should produce the same final state.

Symptoms → Diagnosis → Fix

Polar webhook replay

Via Polar dashboard

1. Polar dashboard → Webhooks → select the webhook endpoint.
2. Find the failed event (filter by failed status or by date).
3. Click “Redeliver” on the event.

Via Polar API

curl -X POST "https://api.polar.sh/v1/webhooks/deliveries/{delivery_id}/redeliver" \
  -H "Authorization: Bearer $POLAR_API_KEY"

What to check first

• Is POLAR_WEBHOOK_SECRET correct? A signature mismatch returns 400 from the API.
• Is the API service healthy? Check API logs and Coolify deployment status.

RunPod job report replay

RunPod callbacks are fire-and-forget. If the orchestrator missed a callback:

1. Query media_job for jobs in processing state older than the expected timeout.
2. For each stuck job, query the RunPod API for the job’s actual status:

curl "https://api.runpod.io/v2/{endpoint_id}/status/{job_id}" \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

3. If RunPod shows the job completed: manually update media_job.status and trigger the next pipeline step, or reset to pending to requeue.

Manual requeue

-- Reset a stuck job to pending so the reconciliation loop picks it up
UPDATE media_job
SET status = 'pending', updated_at = now(), metadata = jsonb_set(metadata, '{replay}', 'true')
WHERE id = '<job_id>' AND status = 'processing';

The reconciliation loop (startReconciliationLoop) will resubmit it within one tick interval.

Idempotency notes

• Polar events: keyed on event.id — duplicate delivery is a no-op if already processed.
• RunPod jobs: media_job.id is the idempotency key passed to RunPod as the job id field.