Runbooks¶
Operator runbooks for dichit-backend production incidents. Runbooks are step-by-step, tested, low-ceremony procedures — if a step is missing or wrong, fix it in the same incident (see templates/runbook.md).
How to use¶
- Read the runbook for the symptom you observe.
- Follow Detection → Initial assessment → Fix → Verification in order.
- If a step doesn't apply, note it and move on — do not skip verification.
- Time-box triage: if you can't resolve within 30 minutes, escalate.
- After the incident, open a PR updating the runbook with what you learned.
Index¶
| Runbook | When to use |
|---|---|
| websocket-down.md | Real-time events stall / WS connections fail |
| redis-down.md | Caching/queues/pub-sub fail (Redis errors) |
| database-down.md | PostgreSQL unreachable / connection pool exhausted |
| high-cpu.md | Sustained high CPU / slow requests |
| deployment-failed.md | A deploy pipeline fails or rolls out a bad build |
| rollback.md | Emergency revert of a bad release |
Common tools¶
- Sentry — error + trace triage (see monitoring).
- CloudWatch Logs — search by
reqId(see logging). - Bull Board
/admin/queues— inspect/retry jobs. - Health/meta —
/health,/metaplatform routes. - Slack DevOps channel — alerting + coordination.
Escalation¶
| Severity | Example | Response |
|---|---|---|
| S1 | full outage | immediate, on-call + lead |
| S2 | feature degraded | same-day |
| S3 | cosmetic / low impact | next release |
Runbook template¶
Copy templates/runbook.md for every new runbook.