Skip to content

Runbook: Redis Down

Overview

Redis is a hard dependency for caching, BullMQ queues, and WebSocket pub/sub. When it fails, those features degrade or fail while REST requests to PostgreSQL may still work.

Detection

  • Connection error / ECONNREFUSED from ioredis in logs.
  • Queue workers stall; Bull Board is empty or errors.
  • WebSocket fan-out stops (see websocket-down.md).
  • Cache misses everywhere.

Initial assessment

redis-cli -h $REDIS_HOST -p $REDIS_PORT ping
redis-cli -h $REDIS_HOST -p $REDIS_PORT info memory
  • Is it the endpoint, auth (REDIS_PASSWORD), or TLS (REDIS_TLS_ENABLED)?
  • Check REDIS_CONNECT_TIMEOUT / REDIS_MAX_RETRIES_PER_REQUEST.
  • Managed Redis (ElastiCache): check the console for node failure / maintenance.

Fix

1. Restart / recover Redis

  • Managed: trigger a failover to a replica or restart the node in the console.
  • Self-hosted: restart the container and verify persistence (RDB/AOF).

2. Verify app reconnection

  • Ensure REDIS_MAX_RETRIES_PER_REQUEST allows reconnect.
  • Once Redis returns, BullMQ workers and pub/sub subscriptions reattach automatically; verify with:
redis-cli -h $REDIS_HOST -p $REDIS_PORT dbsize
redis-cli -h $REDIS_HOST -p $REDIS_PORT pubsub numsub 'ws:global'

3. Gauge impact

  • Caching: reads should fall through to PostgreSQL (see caching).
  • Queues: undelivered jobs are backed up in Redis — after recovery, workers drain them. Watch background jobs (notifications, PDFs, payments).

Verification

  • redis-cli ping → PONG.
  • Bull Board loads and shows jobs progressing.
  • WS events deliver to all instances.
  • Cache hit manually: read a previously cached resource after the first miss.

Escalation

Any partial/prolonged Redis impact on payments/auctions → SSE (severity) S2+.