Event-Driven Pipeline
SNS + SQS + Lambda fan-out reference: one producer publishes domain events to an SNS topic that fans out to per-consumer SQS queues (each with a DLQ), processed
Architecture
- Order Client — Storefront or internal service that emits order lifecycle events via the publish API.
- Events API — REST entry point. Authenticates callers (IAM/API key) and forwards publish requests to the Publisher Lambda.
- Publisher Lambda — Validates the event envelope, writes an audit record to DynamoDB, then publishes to the SNS topic with eventType as a message attribute for subscription filtering.
- Event Audit Table — Append-only record of every published event (event_audit). Source of truth for replay and debugging.
- order-events Topic — Single SNS topic for all order.* events. Fans out to one SQS queue per consumer; filter policies route by eventType attribute.
- Fulfillment Queue — Buffers order.created / order.cancelled for the fulfillment worker. Redrive policy: maxReceiveCount 5 to its DLQ.
- Fulfillment DLQ — Dead-letter queue. 14-day retention; alarmed on depth > 0.
- Notifications Queue — Buffers all order.* events for customer emails. Redrive policy: maxReceiveCount 5 to its DLQ.
- Notifications DLQ — Dead-letter queue. 14-day retention; alarmed on depth > 0.
- Analytics Queue — Buffers all order.* events for the analytics sink. Redrive policy: maxReceiveCount 5 to its DLQ.
- Analytics DLQ — Dead-letter queue. 14-day retention; alarmed on depth > 0.
- Fulfillment Worker — Idempotently upserts the order projection in DynamoDB (conditional write on last_event_id). Batch size 10, partial batch responses enabled.
- Notification Worker — Renders and sends transactional email via SES. Dedupes on eventId so retries never double-send.
- Analytics Worker — Batches events into NDJSON objects and writes them to the S3 data lake, partitioned by event date.
- Order Projections — Read-optimized current state of each order (order_projection table).
- SES — Sends order confirmation / cancellation emails to customers.
- Events Data Lake — s3://order-events-lake/dt=YYYY-MM-DD/ NDJSON batches, queryable with Athena.
- DLQ Alarms — ApproximateNumberOfMessagesVisible > 0 on any DLQ pages on-call and starts the redrive workflow.
Workflow
- Event lands in queue — SNS delivered the event to a consumer's SQS queue.
- Worker receives batch — Lambda event source mapping polls the queue (batch 10, partial batch responses on).
- Process event — Handler validates the payload and applies the side effect idempotently (keyed on eventId).
- Succeeded?
- Delete message — Message removed from the queue (or omitted from batchItemFailures).
- Done
- receiveCount < 5?
- Visibility timeout backoff — Message stays invisible, then reappears for another delivery attempt.
- Moved to DLQ — SQS redrive policy moves the poison message to the dead-letter queue.
- Alarm pages on-call — CloudWatch alarm on DLQ depth > 0 notifies the on-call channel with queue name and sample message.
- Transient failure?
- Redrive DLQ to source queue — SQS StartMessageMoveTask back to the source queue once the downstream issue (or handler bug) is fixed. Log to dlq_redrive_log.
- Reprocessed
- Archive and discard — Persist payload to S3, record resolution=discarded in dlq_redrive_log, purge from DLQ.
- Discarded
Event flow
- Publisher Lambda — Validates envelopes and publishes all order.* events with eventType as a message attribute.
- order-events — Single SNS topic; per-subscription filter policies route by eventType.
- fulfillment-queue — Subscribed with filter eventType IN (order.created, order.cancelled).
- notifications-queue — Receives every order.* event for customer messaging.
- analytics-queue — Receives every order.* event for the data lake.
- Fulfillment Worker — Upserts order projections in DynamoDB; idempotent on eventId.
- Notification Worker — Sends transactional email via SES; dedupes on eventId.
- Analytics Worker — Batches events to NDJSON in the S3 data lake.
- fulfillment-dlq — Poison messages after 5 failed receives.
- notifications-dlq — Poison messages after 5 failed receives.
- analytics-dlq — Poison messages after 5 failed receives.
- DLQ Monitor — Alarm-driven triage: logs the failure, then redrives to the source queue or archives to S3.
- order.created — A customer completed checkout and a new order exists.
- order.updated — Order details changed (address, items, status transition other than cancel).
- order.cancelled — Order was cancelled by the customer or by fraud review.
About this design
- # About this design ## What this is A reference implementation of the SNS + SQS + Lambda **fan-out** pattern — the workhorse of event-driven architectures on AWS. One producer publishes order lifecycle events (`order.created`, `order.updated`, `order.cancelled`) to a single SNS topic; the topic fans out to one SQS queue per consumer; independent worker Lambdas drain their queues into their own sinks (DynamoDB projections, SES emails, an S3 data lake). Every queue has a dead-letter queue and an alarm, so failure handling is designed in, not bolted on. Start with the **event flow** view — it is the clearest picture of what happens here. The **workflow** view covers the retry/DLQ lifecycle of a single message, and the **architecture** view maps everything onto concrete AWS services. ## How it works 1. A client calls `POST /events` (see the API contract). The Publisher Lambda validates the envelope against the schema for its `eventType`, writes an append-only record to the `event_audit` table, then publishes to SNS with `eventType` as a message attribute. 2. SNS delivers to each subscribed queue. Subscription **filter policies** do the routing — the fulfillment queue only receives `order.created` and `order.cancelled`; the other queues take everything. Consumers never filter in code. 3. Each worker is **idempotent on `eventId`**: fulfillment uses a conditional write guarded by `last_event_id`, notifications dedupe before sending, analytics writes are naturally re-runnable. This matters because SQS is at-least-once. 4. On handler failure the message becomes visible again and is retried; after 5 failed receives the redrive policy moves it to the DLQ, a CloudWatch alarm pages on-call, and the workflow view describes the triage: fix-and-redrive (via `StartMessageMoveTask`) for transient issues, archive-and-discard for poison payloads. Every resolution is recorded in `dlq_redrive_log`. ## How to extend - **Add a consumer**: create a queue + DLQ pair, subscribe the queue to the topic with a filter policy, attach a worker Lambda, and alarm the DLQ. Nothing upstream changes — that is the point of the pattern. - **Add an event type**: register its JSON Schema (see the event flow view for the existing three), extend the `eventType` enum in the API contract, and update any filter policies that should receive it. - **Need strict ordering per order?** Swap the topic and queues for their FIFO variants and use `orderId` as the message group id. - **Higher throughput sinks**: replace the analytics worker with Kinesis Firehose if batching logic outgrows Lambda. - **Replay**: the `event_audit` table is your event store — a small script can republish any slice of history to the topic.