Skip to content

Runbook: High CPU / Slow Requests

Overview

Sustained high CPU or elevated request latency across the service. The API target is < 200 ms per response.

Detection

  • CloudWatch/ECS CPU ≥ 80% sustained.
  • p95/p99 latency climbing (Sentry performance, ALB metrics).
  • LOG_PRISMA_SLOW_QUERY_MS flags a spike of slow queries.
  • 429s / 503s from @fastify/rate-limit and overload.

Initial assessment

  • Scope: all instances or one? All endpoints or a few?
  • Recent change: a new deploy, a new campaign/auction, a data backfill?
  • Profile: CPU-bound (JSON/PDF/encryption) vs DB-bound (queries).

Fix

1. DB-bound (most common)

# find the hot queries
SELECT pid, now()-query_start AS running, left(query, 200) AS q
FROM pg_stat_activity WHERE state = 'active' ORDER BY running DESC LIMIT 20;
  • Missing index → add via a migration (see indexing).
  • N+1 / over-fetch → fix in the repository (see optimization).
  • Cache the hot read path (see caching).

2. CPU-bound in the app

  • Large in-request work (PDF rendering, heavy encryption, big payloads) should move to queues (see event-driven).
  • Check for tight loops / linear scans flagged by the code graph (e.g. transitive_loop_depth ≥ 3 hotspots).
  • Offload and retest before scaling.

3. Overload / traffic spike

  • Scale out horizontally (stateless REST; WS scales via Redis pub/sub).
  • Tune rate limits (see rate-limits for WS).
  • Consider caching public/config reads aggressively.

Verification

  • CPU returns to baseline; latency back under target.
  • Slow-query log quiet; Bull Board jobs draining.
  • No new 5xx/429 errors during peak.

Escalation

30 min unresolved or customer-visible degradation → lead + infra owner.