Real-time stream processing with Kinesis Data Streams and Lambda
A durable ordered buffer on one side, a poller that hands your function batches on the other, and a poison-pill trap in between.
Two halves: a durable buffer and a poller
Real-time processing on AWS usually splits into two halves that you configure independently. On one side is Amazon Kinesis Data Streams: a durable, ordered buffer that producers write into with PutRecord or PutRecords. Each record carries a partition key, and that key decides which shard the record lands on. Within a single shard, records stay in the order they arrived.
On the other side is your AWS Lambda function. It does not reach into the stream itself. Instead you attach a Lambda event source mapping, a poller that Lambda manages for you. The event source mapping reads records from the stream and invokes your function synchronously with a batch of those records. You write only the code that processes a batch; the polling, batching, and retry machinery is the event source mapping's job. Because the two halves are decoupled by the stream, a slow or failing consumer never causes producers to lose data as long as records are still within the retention window.
Two halves: a durable buffer and a poller
Trace
The pattern splits into two halves you configure independently. Producers write records (tagged with a partition key) into a Kinesis data stream, which durably buffers them across shards in arrival order per key. A Lambda event source mapping polls each shard and hands your function batches synchronously. Because the stream sits between them, a slow or failing consumer never causes producers to lose data while records are inside the retention window.
How the event source mapping actually works
The event source mapping polls each shard on your behalf and collects records until a batch is ready, controlled by two limits you set: batch size (the maximum number of records) and batch window (the maximum time to wait while filling a batch). When either limit is hit, the whole batch is delivered to your function in a single synchronous invocation.
Ordering is preserved per shard: the event source mapping does not send the next batch for a shard until the current batch for that shard finishes. To process a single shard faster without adding shards, you raise the parallelization factor, which lets the event source mapping run up to 10 concurrent function invocations per shard while still keeping records with the same partition key in order. This is why partition key choice matters: all records for one key go to one shard and are processed in order, so a high-cardinality key spreads load while a low-cardinality key creates a hot shard.
How the event source mapping actually works
Trace
The event source mapping polls each shard and collects records until batch size (max records) or batch window (max wait) is hit, then delivers the whole batch in one synchronous invocation. Ordering is preserved per shard, so it holds the next batch until the current one finishes. Raising the parallelization factor lets up to 10 invocations run concurrently on a single shard while records with the same partition key stay in order.
Retention and replay: the stream is not a queue
A Kinesis data stream retains records for 24 hours by default, and you can extend retention up to 365 days. Reading a record does not delete it, which is the key difference from an Amazon SQS queue where a consumer deletes a message after handling it. Because records persist for the whole retention window, multiple independent consumers can each read the same stream from their own position, and you can replay history by resetting a consumer to an earlier point in the stream.
That durability is what makes the buffer safe: if your function is down for an hour, the event source mapping simply resumes polling from where it left off and catches up, as long as the affected records have not aged past the retention window. Choosing a retention period is therefore a real design decision, not a default to ignore.
Retention and replay: the stream is not a queue
Trace
A Kinesis read is non-destructive. Records are retained 24 hours by default (extendable to 365 days), so multiple independent consumers can each read the same stream from their own position, and a consumer can be reset to an earlier point to replay history. That is the key difference from an SQS queue, where a consumer deletes a message once it is handled.
The poison pill and how to defuse it
By default, if a batch causes your function to error, the event source mapping retries that same batch and will not move past it. Because order is preserved per shard, one record that always fails, a poison pill, blocks the entire shard until those records expire from the stream. That is the classic stuck-shard failure.
You defuse it with the event source mapping's own error controls. MaximumRetryAttempts caps how many times a failed batch is retried, and MaximumRecordAgeInSeconds discards records once they are too old, so a bad batch cannot block the shard forever. BisectBatchOnFunctionError splits a failing batch in half and retries each half, isolating the one bad record instead of failing the whole batch. An on-failure destination sends a record with metadata about the failed batch (not the record contents) to an Amazon SQS queue or Amazon SNS topic so you can inspect and reprocess it. Finally, ReportBatchItemFailures lets your function return the sequence numbers of only the records that failed, so the event source mapping retries just those instead of the entire batch.
The poison pill and how to defuse it
Trace
By default a failed batch is retried and, because order is preserved per shard, the event source mapping will not move past it — one always-failing record blocks the whole shard. You defuse it with the mapping's error controls: cap retries (MaximumRetryAttempts), age off old records (MaximumRecordAgeInSeconds), split the batch (BisectBatchOnFunctionError), send failed-batch metadata to an on-failure destination, or return just the failed sequence numbers with ReportBatchItemFailures.
Scaling and watching for lag
Two dials govern throughput. The first is shard count: each shard is a unit of stream capacity, and more shards mean more parallelism because the event source mapping polls each shard independently. The second is the parallelization factor within a shard, up to 10 concurrent invocations per shard. Together they set how much work the consumer side can do at once.
To know whether the consumer is keeping up, watch the CloudWatch IteratorAge metric for the event source mapping. Iterator age is the age of the last record the function processed; a rising iterator age means records are sitting in the stream longer before they are handled, so the consumer is falling behind and you need more shards, a higher parallelization factor, faster function code, or more concurrency. A steady low iterator age means processing is keeping pace with ingestion.
Scaling and watching for lag
Trace
Two dials govern throughput: shard count (each shard is a capacity unit polled independently) and the parallelization factor within a shard (up to 10 concurrent invocations). To know whether the consumer is keeping up, watch CloudWatch IteratorAge — the age of the last record the function processed. A rising IteratorAge means records wait longer before handling, so the consumer is falling behind and you add shards, raise the parallelization factor, speed up the code, or add concurrency.
