Event-Driven Pipeline

Architecture

• Order Client: Storefront or internal service that emits order lifecycle events via the publish API.
• Events API: REST entry point. Authenticates callers (IAM/API key) and forwards publish requests to the Publisher Lambda.
• Publisher Lambda: Validates the event envelope, writes an audit record to DynamoDB, then publishes to the SNS topic with eventType as a message attribute for subscription filtering.
• Event Audit Table: Append-only record of every published event (event_audit). Source of truth for replay and debugging.
• order-events Topic: Single SNS topic for all order.* events. Fans out to one SQS queue per consumer; filter policies route by eventType attribute.
• Fulfillment Queue: Buffers order.created / order.cancelled for the fulfillment worker. Redrive policy: maxReceiveCount 5 to its DLQ.
• Fulfillment DLQ: Dead-letter queue. 14-day retention; alarmed on depth > 0.
• Fulfillment Worker: Idempotently upserts the order projection in DynamoDB (conditional write on last_event_id). Batch size 10, partial batch responses enabled.
• Order Projections: Read-optimized current state of each order (order_projection table).
• Notifications Queue: Buffers all order.* events for customer emails. Redrive policy: maxReceiveCount 5 to its DLQ.
• Notifications DLQ: Dead-letter queue. 14-day retention; alarmed on depth > 0.
• Notification Worker: Renders and sends transactional email via SES. Dedupes on eventId so retries never double-send.
• SES: Sends order confirmation / cancellation emails to customers.
• Analytics Queue: Buffers all order.* events for the analytics sink. Redrive policy: maxReceiveCount 5 to its DLQ.
• Analytics DLQ: Dead-letter queue. 14-day retention; alarmed on depth > 0.
• Analytics Worker: Batches events into NDJSON objects and writes them to the S3 data lake, partitioned by event date.
• Events Data Lake: s3://order-events-lake/dt=YYYY-MM-DD/ NDJSON batches, queryable with Athena.
• DLQ Alarms: ApproximateNumberOfMessagesVisible > 0 on any DLQ pages on-call and starts the redrive workflow.

Workflow

• Event lands in queue: SNS delivered the event to a consumer's SQS queue.
• Worker receives batch: Lambda event source mapping polls the queue (batch 10, partial batch responses on).
• Process event: Handler validates the payload and applies the side effect idempotently (keyed on eventId).
• Succeeded?
• Delete message: Message removed from the queue (or omitted from batchItemFailures).
• Done
• receiveCount < 5?
• Visibility timeout backoff: Message stays invisible, then reappears for another delivery attempt.
• Moved to DLQ: SQS redrive policy moves the poison message to the dead-letter queue.
• Alarm pages on-call: CloudWatch alarm on DLQ depth > 0 notifies the on-call channel with queue name and sample message.
• Transient failure?
• Redrive DLQ to source queue: SQS StartMessageMoveTask back to the source queue once the downstream issue (or handler bug) is fixed. Log to dlq_redrive_log.
• Reprocessed
• Archive and discard: Persist payload to S3, record resolution=discarded in dlq_redrive_log, purge from DLQ.
• Discarded

Event flow

• Publisher Lambda: Validates envelopes and publishes all order.* events with eventType as a message attribute.
• order-events: Single SNS topic; per-subscription filter policies route by eventType.
• fulfillment-queue: Subscribed with filter eventType IN (order.created, order.cancelled).
• notifications-queue: Receives every order.* event for customer messaging.
• analytics-queue: Receives every order.* event for the data lake.
• Fulfillment Worker: Upserts order projections in DynamoDB; idempotent on eventId.
• Notification Worker: Sends transactional email via SES; dedupes on eventId.
• Analytics Worker: Batches events to NDJSON in the S3 data lake.
• fulfillment-dlq: Poison messages after 5 failed receives.
• notifications-dlq: Poison messages after 5 failed receives.
• analytics-dlq: Poison messages after 5 failed receives.
• DLQ Monitor: Alarm-driven triage: logs the failure, then redrives to the source queue or archives to S3.
• order.created: A customer completed checkout and a new order exists.
• order.updated: Order details changed (address, items, status transition other than cancel).
• order.cancelled: Order was cancelled by the customer or by fraud review.

API contract

• POST/eventsPublish a domain event
• GET/events/{eventId}Fetch the audit record for a published event
• EventEnvelope: Standard envelope every publisher must send

Database

About this design

About this design

What this is

A reference implementation of the SNS + SQS + Lambda fan-out pattern — the workhorse of event-driven architectures on AWS. One producer publishes order lifecycle events (order.created, order.updated, order.cancelled) to a single SNS topic; the topic fans out to one SQS queue per consumer; independent worker Lambdas drain their queues into their own sinks (DynamoDB projections, SES emails, an S3 data lake). Every queue has a dead-letter queue and an alarm, so failure handling is designed in, not bolted on.

Start with the event flow view — it is the clearest picture of what happens here. The workflow view covers the retry/DLQ lifecycle of a single message, and the architecture view maps everything onto concrete AWS services.

How it works

A client calls POST /events (see the API contract). The Publisher Lambda validates the envelope against the schema for its eventType, writes an append-only record to the event_audit table, then publishes to SNS with eventType as a message attribute.

SNS delivers to each subscribed queue. Subscription filter policies do the routing — the fulfillment queue only receives order.created and order.cancelled; the other queues take everything. Consumers never filter in code.

Each worker is idempotent on eventId: fulfillment uses a conditional write guarded by lasteventid, notifications dedupe before sending, analytics writes are naturally re-runnable. This matters because SQS is at-least-once.

On handler failure the message becomes visible again and is retried; after 5 failed receives the redrive policy moves it to the DLQ, a CloudWatch alarm pages on-call, and the workflow view describes the triage: fix-and-redrive (via StartMessageMoveTask) for transient issues, archive-and-discard for poison payloads. Every resolution is recorded in dlqredrivelog.

How to extend

Add a consumer: create a queue + DLQ pair, subscribe the queue to the topic with a filter policy, attach a worker Lambda, and alarm the DLQ. Nothing upstream changes — that is the point of the pattern.

Add an event type: register its JSON Schema (see the event flow view for the existing three), extend the eventType enum in the API contract, and update any filter policies that should receive it.

Need strict ordering per order? Swap the topic and queues for their FIFO variants and use orderId as the message group id.

Higher throughput sinks: replace the analytics worker with Kinesis Firehose if batching logic outgrows Lambda.

Replay: the event_audit table is your event store — a small script can republish any slice of history to the topic.