CI/CD Pipeline
Modern CI/CD reference: GitHub push triggers build and test, artifacts land in S3/ECR, then staged deploys to dev, staging, and prod with manual approval gates,
Architecture
- GitHub — Source of truth. Push to main (via PR merge) fires a webhook that starts the pipeline; commit statuses are reported back.
- Slack — Receives stage results, prod approval requests, and rollback alerts.
- Release Pipeline — Orchestrates source -> build -> dev -> staging -> approval -> prod. One pipeline execution per commit to main; superseded executions are cancelled.
- Build & Test — Runs unit tests and lint, builds the container image tagged with the git SHA, and produces the deploy bundle (taskdef + appspec).
- ECR — Immutable image registry. Images are tagged :<git-sha>; nothing is ever retagged, so any previous version can be redeployed.
- Artifact Bucket — Versioned bucket holding build artifacts, task definitions, and appspec files per pipeline execution.
- Blue/Green Deployer — ECS blue/green deployments: launches the green task set, shifts canary traffic, watches alarms during the bake, auto-rolls back on breach.
- Dev Cluster — Auto-deployed on every green build. Smoke tests run here before anything else proceeds.
- Staging Cluster — Production mirror. Blue/green deploy plus integration and load checks gate the prod approval.
- Prod Cluster — Deployed only after manual approval. Canary 10% for 15 minutes, then full traffic shift.
- Canary Alarms — 5xx rate, p99 latency, and task health alarms scoped to the green task set. Any breach during the bake window triggers automatic rollback.
Workflow
- Push to main — PR merged; GitHub webhook starts a pipeline execution pinned to this commit SHA.
- Build + unit tests — Lint, unit tests, container build tagged :<git-sha>.
- Build green?
- Publish artifacts — Push image to ECR, upload taskdef/appspec bundle to the versioned artifact bucket.
- Deploy to dev — Rolling deploy to the dev cluster; no approval needed.
- Smoke tests — Hit health and critical-path endpoints against dev.
- Smoke passed?
- Blue/green to staging — CodeDeploy launches the green task set in the staging cluster and shifts traffic.
- Integration + load checks — Full integration suite plus a short load test against staging.
- Staging healthy?
- Await prod approval — Manual approval gate. Release notes and staging results posted to Slack; any release approver can approve or reject.
- Approved?
- Prod canary 10% — CodeDeploy launches the green task set in prod and shifts 10% of traffic to it.
- Bake period — Canary alarms (5xx rate, p99 latency, task health) watched against the green task set.
- Alarms healthy?
- Shift 100%, retire blue — All traffic to green; blue task set drained and terminated after the wait window.
- Released — Commit status set to released; release annotated in CloudWatch.
- Auto rollback — CodeDeploy reroutes 100% of traffic back to the blue task set and terminates green. Prod never runs the bad build at full traffic.
- Page on-call — Rollback alert to Slack with alarm details and diff link; commit marked failed.
- Rolled back
- Report failure — Fail the pipeline execution, set GitHub commit status, post the failing stage to Slack.
- Pipeline failed
About this design
- # About this design ## What this is A reference CI/CD pipeline for a containerized service: GitHub -> build/test -> immutable artifacts (ECR + S3) -> staged deploys through dev, staging, and prod with a manual approval gate, canary traffic shifting, and alarm-driven automatic rollback. The AWS pieces are CodePipeline, CodeBuild, and CodeDeploy driving ECS blue/green — but every stage maps 1:1 onto GitHub Actions + your deploy tool of choice, so treat the shape as the spec, not the vendor list. Start with the **workflow** view — it is the whole pipeline as a state machine with every decision gate. The **sequence** view zooms into the most interesting part: a prod deploy, shown twice (canary healthy vs. alarm-triggered rollback). ## How it works 1. **One execution per commit.** A merge to main fires a webhook; the pipeline pins everything to that git SHA. The image is tagged `:<sha>` and never retagged — immutable artifacts are what make rollback trivial. 2. **Fail fast, cheaply.** Unit tests gate the build; smoke tests gate promotion out of dev; a full integration + load pass gates the approval request. Each gate that fails reports to Slack and GitHub and ends the execution. 3. **Humans approve prod, machines verify it.** The approval gate (up to 24h) carries release notes and staging results. After approval, CodeDeploy launches the green task set, shifts 10% of traffic, and bakes for 15 minutes while CloudWatch alarms (5xx rate, p99 latency, task health) watch the green side only. 4. **Rollback is automatic and boring.** Any alarm breach during the bake reroutes 100% of traffic back to the blue task set — no human in the loop, and the bad build never took full traffic. Manual rollback later is just redeploying a previous `:<sha>` tag. ## How to extend - **Different runtime**: the pattern is runtime-agnostic — swap ECS blue/green for Lambda alias traffic shifting or a Kubernetes progressive-delivery controller; keep the gates and the bake. - **More environments**: clone the staging stage (deploy -> verify -> gate). Resist adding environments that don't have a distinct verification job — every stage should earn its latency. - **Tighter canaries**: add synthetic canary traffic (CloudWatch Synthetics) so low-traffic services still exercise the green task set during the bake. - **DB migrations**: add an expand/contract migration step before the canary — forward-compatible migrations are what keep instant rollback safe. - **Compliance**: the approval gate is the natural place to attach change tickets; pipe approvals to your change-management system instead of (or alongside) Slack.