Skip to content

Runbook: Emergency Rollback

Overview

A release is causing user-visible harm and must be reverted now. The strategy is documented in deployment/rollback.md; this runbook is the execution checklist.

When to trigger

  • S1 outage caused by the latest release.
  • Data-integrity risk from the new code path.
  • 5xx/Sentry error spike immediately following rollout.

Execution

1. Freeze

  • Announce the rollback on the incident channel.
  • Stop further merges/promotions to the affected environment.
  • No new deploys until the incident is resolved.

2. Roll back the application image

  1. Identify the last known-good version from CHANGELOG.md or the previous image tag (see release-process.md).
  2. Re-deploy that image via the environment pipeline / ECS service update. Images are immutable and versioned — never patch a running container.
  3. Verify /health green and /meta reports the old version.

3. Handle the database

  • If the bad release didn't ship schema/data changes → skip; code rollback is enough.
  • If it did:
  • Prefer a forward-fix migration over reversing DDL.
  • A migration that already ran is not undone by re-deploying old code — the schema stays. Write and test a reversing migration if the new schema breaks the old code.
  • Never migrate reset or restore a snapshot without explicit, coordinated approval (data loss).

4. Verify

  • Core flows (login, list, payment, auction) pass on the old version.
  • Queues and WS pub/sub healthy; no accumulating failed jobs.
  • Error rate at baseline for 15–30 minutes before calling it done.

Aftermath

  • Root-cause via Sentry/CloudWatch (see monitoring).
  • Write a follow-up fix, test, and redeploy forward.
  • Update the runbook if any step was wrong or missing.