Event-driven decoupled processing with SQS, SNS, and Lambda

Why decouple: the synchronous trap

Imagine an order API that, on every request, calls a fulfillment service and an analytics service directly and waits for both to answer before responding. This is tight coupling, and it has two failure modes the exam loves. First, the slowest downstream component sets the speed of the whole request: if analytics takes two seconds, the customer waits two seconds. Second, a traffic spike hits every component at once, so a burst that fulfillment could survive can still knock the request over because analytics fell behind.

The fix is to put a buffer between the producer and its consumers. A queue absorbs a spike: the producer writes messages as fast as they arrive and returns immediately, while consumers drain the queue at their own sustainable rate. The producer no longer knows or cares whether a consumer is fast, slow, or temporarily down. That single change, moving from a synchronous call to a durable buffer, is what 'loosely coupled' means in practice.

Fan-out with SNS, buffer with SQS, process with Lambda

When one event needs to reach several independent consumers, publishing directly to many queues from the producer re-introduces coupling. The clean pattern is fan-out: the producer publishes a single message to an Amazon SNS topic, and SNS delivers an independent copy to every Amazon SQS queue subscribed to that topic. Add a new consumer later by subscribing a new queue, and the producer code never changes.

Each queue is then drained by its own AWS Lambda function through an event source mapping. Lambda polls the queue, and invokes the function with a batch of messages (by default up to 10 messages per batch, and you can add a batch window to wait and collect more). When the function successfully processes a batch, Lambda deletes those messages from the queue. Because each queue and each Lambda scales on its own, the analytics consumer falling behind never slows the fulfillment consumer, and the producer is unaffected either way.

SNS subscription filter policies let a subscriber receive only the subset of messages it cares about, so the analytics queue can take only the message types it needs instead of everything on the topic.

Failure handling, idempotency, and the dead-letter queue

SQS event source mappings deliver each message at least once, which means duplicate processing can happen (for example, a batch fails partway and its messages become visible again). The architect's obligation is to make the consumer idempotent, so processing the same message twice has no unintended side effect. While a consumer holds a batch, those messages are hidden from other pollers for the length of the queue's visibility timeout; if the function does not finish and delete them in time, they reappear for another attempt.

For messages that fail repeatedly (a malformed 'poison' message that will never succeed), you attach a dead-letter queue and set a maxReceiveCount. Once a message has been received that many times without being deleted, SQS moves it to the dead-letter queue instead of retrying forever. That isolates the bad message so the main queue keeps flowing, and it gives you a place to inspect and reprocess failures. With partial batch responses, a function can even tell Lambda exactly which messages in a batch failed so only those are retried, not the whole batch.