Skip to content

Runbook: WebSocket Down / Real-Time Stall

Overview

WebSocket connectivity or event delivery is broken: clients can't connect, or connected clients stop receiving events. Real-time auctions and presence are affected; REST generally still works.

Detection

  • Client reports "not receiving live bids / presence".
  • WS connections fail at handshake (close code 1008 on auth failures).
  • Redis pub/sub or Bull Board shows no activity for ws:* keys.
  • Errors in logs around WebSocketRouter, deliverFromPubSub, or webSocketPubSub.

Initial assessment

Is Redis reachable?           → check REDIS_HOST/PORT, redis-cli ping
Is the /ws route registered?  → GET /ws should upgrade
Are connections authenticating?→ check JWT secrets/token expiry
Is it one instance or all?    → compare across replicas

Fix

1. Redis is the root cause

Real-time fan-out depends on Redis pub/sub (channel ws:global):

redis-cli -h $REDIS_HOST -p $REDIS_PORT ping   # expect PONG
redis-cli -h $REDIS_HOST -p $REDIS_PORT pubsub channels 'ws:*'
  • If Redis is down → follow redis-down.md first.
  • If the channel has no subscribers, connections may still succeed but events won't fan out; restart the affected WS instances to resubscribe.

2. Authentication failures

  • Rotated/expired ACCESS_TOKEN_SECRET breaks handshakes.
  • Confirm the token used by clients matches the current secret (see authentication).

3. Single instance vs cluster

  • If one instance fails, the load balancer should route clients to healthy instances. Verify the ALB target group health check (/health).
  • Sticky-session config must not pin clients to a dead instance.

4. Deadlock / overload

Verification

  • Client reconnects and receives a system.pong and auction.snapshot.
  • Pub/sub delivers to all instances (pubsub numsub ws:global grows with replicas).
  • Error rate for WS events back to baseline.

Escalation

30 min without resolution → page lead + infrastructure owner.