Skip to content

Runbook: Database Down

Overview

PostgreSQL is unavailable or severely degraded. This is an S1/S2 situation — nearly every request depends on the database.

Detection

  • connection refused / timeout errors from the Prisma adapter (pg).
  • /health fails or reports DB dependency down.
  • ECONNREFUSED on the DATABASE_URL host/port.
  • Slow queries + pool exhaustion (see high-cpu.md for overload).

Initial assessment

# from a host that can reach the DB
psql "$DATABASE_URL" -c "SELECT 1"
# connection count
psql "$DATABASE_URL" -c "SELECT count(*) FROM pg_stat_activity;"
# locks / long-running queries
psql "$DATABASE_URL" -c "SELECT pid, state, now()-query_start AS running FROM pg_stat_activity WHERE state <> 'idle' ORDER BY running DESC;"
  • Is it network, auth, or the DB itself?
  • Managed RDS: check console status, failover events, storage, maintenance.
  • Check DATABASE_SSL_ENABLED matches the DB requirements.

Fix

1. Network / credentials

  • Verify VPC/security-group reachability, DATABASE_SSL_ENABLED, and the credentials in DATABASE_URL.

2. Managed RDS

  • Trigger a failover from the console if the primary node is unhealthy.
  • Monitor storage auto-scaling and replica lag if replicas are in use.

3. Self-hosted PostgreSQL

  • Restart the container; verify the volume is intact.
  • Check pg_wal/disk space (df -h), then pg_ctl recovery if the DB won't start cleanly.

4. Connection pool exhaustion (not a true outage)

  • Kill idle-in-transaction / long-running queries carefully.
  • Reduce app connections temporarily; scale the DB class if needed.
  • Look at N+1 or runaway batch jobs (see optimization).

Verification

  • psql ... -c "SELECT 1" succeeds.
  • /health green; app errors for DB drops to zero.
  • Queues that touch the DB resume and drain (Bull Board).

Escalation

DB outage = S1. Page the on-call + platform lead immediately; keep the incident channel updated. Restore from the latest RDS snapshot / point-in-time only if the data loss is acceptable and coordinated (see backups).