Event-driven order processing with SNS, SQS, and Lambda
One order in, many jobs out: how SNS fanout, SQS buffers, and Lambda consumers turn a checkout into a loosely coupled pipeline that survives failures and never double-charges.
The shape: intake, fan out, buffer, process
A checkout produces one business event ("order placed") that several unrelated subsystems care about: inventory must reserve stock, payments must charge the card, notifications must email the customer. The tightly coupled version has the intake code call each subsystem synchronously and fail the whole request if any one is down.
The event-driven version breaks that coupling. API Gateway accepts the order and invokes a small intake Lambda whose only job is to publish the event once to an Amazon SNS topic. SNS fans the event out to one Amazon SQS queue per subsystem, and each queue is drained by its own Lambda consumer. The intake function returns as soon as the publish succeeds; it neither knows nor cares whether payments is healthy. Each SQS queue is a buffer that absorbs spikes and outages, so a slow or failed consumer holds up only its own work, not the customer's checkout.
Event-driven order processing with SNS fanout, SQS buffers, and Lambda consumers
Trace
An order arrives at API Gateway and is published once to an SNS topic. SNS fans the event out to one SQS queue per bounded context (inventory, payments, notifications); subscription filter policies let each queue receive only the messages it cares about. A Lambda event source mapping polls each queue in batches. Failed messages return to the queue after the visibility timeout and, once they exceed maxReceiveCount, are moved to a dead-letter queue. Consumers write to a DynamoDB idempotency table so at-least-once redelivery never double-charges an order.
Why SNS sits in front of SQS
SNS is pub/sub: publish a message once and every subscriber gets its own independent copy. Subscribing an SQS queue to the topic gives each subsystem a durable, private copy of the event that it can process at its own pace. This is the fanout pattern, and it is why the topic sits in front of the queues rather than the intake function writing to three queues directly: adding a fourth subsystem later is a new subscription, not a code change to the producer.
SNS subscription filter policies keep each queue clean. A filter policy is set on the subscription and compares message attributes; a subscriber receives only the messages whose attributes match its policy, and non-matching messages are filtered out before delivery instead of being delivered and discarded by the consumer. So the notifications queue can ask for only order-placed and order-shipped events while the payments queue asks for only order-placed, and neither wastes a Lambda invocation on events it does not handle.
SNS fanout with subscription filter policies
Trace
The intake Lambda publishes one message with attributes (eventType) to a single SNS topic. Every subscribed SQS queue gets its own durable copy, and a subscription filter policy on each subscription drops non-matching events before delivery. Adding a fourth subsystem later is a new subscription on the same topic, not a change to the producer.
Making Lambda consumption reliable
A Lambda consumer does not subscribe to SQS directly. Instead you configure an event source mapping: the Lambda service polls the queue on your behalf, reads messages in batches, and invokes your function with a batch of records. On success the service deletes those messages from the queue; if the function throws, the whole batch becomes visible again after the queue's visibility timeout and is retried. Because retries hinge on that timer, AWS recommends setting the queue's visibility timeout to at least six times the function's timeout so a message is not redelivered while an invocation is still working on it.
Two controls stop a single bad message from poisoning the batch. A partial batch response (returning the IDs of only the failed records via ReportBatchItemFailures) lets the successful records be deleted while just the failures are retried, instead of reprocessing the entire batch. And a redrive policy with a maxReceiveCount sends a message to a dead-letter queue once it has been received that many times without being deleted, so a poison message lands in the DLQ for inspection instead of looping forever.
Reliable Lambda consumption: event source mapping, retries, and DLQ
Trace
A Lambda event source mapping (not a direct subscription) has the Lambda service poll the payments queue in batches and invoke the function. Successful records are deleted; a partial batch response retries only the failures. Failed messages reappear after the visibility timeout, and once they exceed maxReceiveCount the redrive policy moves them to a dead-letter queue for inspection.
Ordering and idempotency
Standard SQS queues give at-least-once delivery and best-effort ordering: a message is delivered at least once, which means the same order event can occasionally be delivered more than once, and order is not guaranteed. The correct response is not to fight the queue but to make each consumer idempotent, so processing the same order event twice has the same effect as processing it once. A common implementation is a DynamoDB idempotency table keyed on the order ID: the payments consumer records the order ID as it charges, and a redelivered copy sees the key already present and skips the second charge.
When the business truly requires strict ordering and no duplicates, use a FIFO topic subscribed to FIFO queues. FIFO delivers messages in order within a message group ID and provides exactly-once processing, deduplicating messages that share a deduplication ID within a five-minute interval. FIFO trades throughput for those guarantees, so reach for it only when ordering or exactly-once actually matters; for most fanout work, standard queues plus idempotent consumers are the simpler and cheaper choice.
At-least-once with idempotency, and the FIFO alternative
Trace
Standard queues deliver at-least-once with best-effort ordering, so a duplicate order event is possible; the fix is an idempotent consumer backed by a DynamoDB conditional write keyed on the order ID, so a redelivered copy skips the second charge. When strict ordering and exactly-once are truly required, a FIFO topic subscribed to a FIFO queue delivers in order within a message group ID and deduplicates within a five-minute window.
What this teaches for the exam
When a scenario says one event must trigger several independent actions, the answer is SNS fanout to multiple SQS queues, not a producer that writes to each queue itself. When a subscriber should receive only some of the topic's messages, the answer is an SNS subscription filter policy, not consumer-side filtering. When a question asks how a repeatedly failing message is isolated, the answer pairs a redrive policy's maxReceiveCount with a dead-letter queue. And when a question worries about duplicate processing on a standard queue, the answer is idempotent consumers (often a DynamoDB conditional write), while strict ordering with no duplicates points to FIFO.
Event-driven order processing with SNS fanout, SQS buffers, and Lambda consumers
Trace
An order arrives at API Gateway and is published once to an SNS topic. SNS fans the event out to one SQS queue per bounded context (inventory, payments, notifications); subscription filter policies let each queue receive only the messages it cares about. A Lambda event source mapping polls each queue in batches. Failed messages return to the queue after the visibility timeout and, once they exceed maxReceiveCount, are moved to a dead-letter queue. Consumers write to a DynamoDB idempotency table so at-least-once redelivery never double-charges an order.
