Skip to content

Runbook: Deployment Failed

Overview

A deploy pipeline failed, or a deploy rolled out but the app is unhealthy.

Detection

  • GitHub Actions run failed (see ci-cd).
  • /health or /meta shows the new version down.
  • ALB 5xx spike after rollout; Sentry errors on the new release.
  • Migrations did not apply, or the app can't boot with the new schema.

Initial assessment

Open the failing workflow:

  • cd-dev.yml / cd-staging.yml / prod-release.yml — build vs deploy step?
  • Which step failed: image build, migration, ECS update, health check?
Build failed      → code/config problem, fix and redeploy
Migration failed  → schema/SQL problem (see database-down if locked)
Rollout failed    → app won't start or fails health checks

Fix

1. Build failed

  • Read the log; fix lint/type/build errors, or missing secrets in GH secrets.
  • Re-run the pipeline after the fix — do not force.

2. Migration failed

  • Check migrate:status on the target environment (see database/migrations.md).
  • If a migration partially applied, resolve it (--rolled-back/--applied) after confirming DB state.
  • Never migrate reset on shared environments.

3. Rollout failed / app unhealthy after deploy

  1. Confirm the app image boots locally with the same env.
  2. Check startup errors in CloudWatch (env, secrets, DB reachability).
  3. If config-related, fix and redeploy.
  4. If the build itself is bad, roll back to the previous image (see rollback.md).

Verification

  • Pipeline green for the fixed/rolled-back version.
  • /health and /meta report the expected version.
  • Error rate baseline restored; queues drain.

Escalation

Failed production rollout with user impact → S1; follow rollback.md immediately, then diagnose.