Lambda Triggers & Error Handling

The most common way to consume a DynamoDB Stream is with AWS Lambda. This integration allows you to build serverless, event-driven applications that react instantly to data changes.

1. Event Source Mapping (ESM)

It’s important to understand that Lambda does not “listen” to the stream directly. Instead, an Event Source Mapping (ESM) process (managed by AWS) polls the stream shards for you.

1. Polling: The ESM polls the shard for records.
2. Batching: It accumulates records until it hits a BatchSize limit OR a BatchWindow time limit.
3. Invocation: It synchronously invokes your Lambda function with the batch of records.
4. Checkpointing: If the Lambda succeeds, the ESM advances the iterator. If it fails, the ESM retries the entire batch until it succeeds or the records expire.

[!NOTE] Concurrency Limit: You can have at most one Lambda invocation per Shard at any given time. To scale processing, you must increase the number of shards (which happens automatically as you increase table write throughput).

---

2. Tuning: Batch Size vs. Window

Optimizing the batch configuration is critical for cost and latency.

• Batch Size (Default: 100): The maximum number of records to send in one function call. Larger batches = fewer invocations (cheaper) but higher memory usage.
• Batch Window (Default: 0s): The maximum time to wait to fill a batch. Increasing this (e.g., to 10s) drastically reduces invocations for low-traffic streams but adds latency.

Interactive: Batch Tuning Simulator

Adjust the incoming traffic and batch settings to see the impact on cost (invocations) and latency.

Incoming Traffic (Records/sec)

100 RPS

Batch Size (Max Records)

100 Items

Batch Window (Seconds)

0 sec

Invocations per Minute

6000

Avg Latency Added

0 ms

*Simplified calculation assuming constant traffic distribution.

---

3. Handling Failures

If your Lambda function throws an error, the ESM stops processing that shard and retries the entire batch indefinitely until the records expire (24h). This is the “poison pill” problem.

Strategies

1. BisectBatchOnFunctionError: If a batch fails, AWS splits it into two halves and retries each recursively. This isolates the bad record.
2. MaximumRecordAgeInSeconds: Skip records that are too old (e.g., > 60s).
3. MaximumRetryAttempts: Give up after N retries.
4. On-Failure Destination: Send failed records to an SQS queue (DLQ) for manual inspection.

Partial Batch Failure (Best Practice)

Instead of failing the whole batch, catch exceptions in your loop and return a list of itemIdentifiers that failed. AWS will only retry those specific items.

---

4. Code Implementation

Here is a robust handler pattern that implements idempotency and partial failure reporting.

Java

Go

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.DynamodbEvent;
import com.amazonaws.services.lambda.runtime.events.StreamsEventResponse;

import java.util.ArrayList;
import java.util.List;

public class StreamProcessor implements RequestHandler<DynamodbEvent, StreamsEventResponse> {

    @Override
    public StreamsEventResponse handleRequest(DynamodbEvent event, Context context) {
        List<StreamsEventResponse.BatchItemFailure> failures = new ArrayList<>();

        for (DynamodbEvent.DynamodbStreamRecord record : event.getRecords()) {
            try {
                processRecord(record);
            } catch (Exception e) {
                // Add the sequence number of the failed record to the list
                failures.add(new StreamsEventResponse.BatchItemFailure(
                    record.getDynamodb().getSequenceNumber()
                ));
            }
        }

        // Return list of failed items (if any)
        return new StreamsEventResponse(failures);
    }

    private void processRecord(DynamodbEvent.DynamodbStreamRecord record) {
        // 1. Idempotency Check (e.g., check Redis if EventID was processed)
        // 2. Business Logic
        if (record.getEventName().equals("INSERT")) {
             System.out.println("New Item: " + record.getDynamodb().getKeys());
        }
    }
}

package main

import (
	"context"
	"fmt"
	"github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-lambda-go/lambda"
)

func handleRequest(ctx context.Context, event events.DynamoDBEvent) (events.DynamoDBEventResponse, error) {
	var failures []events.DynamoDBBatchItemFailure

	for _, record := range event.Records {
		err := processRecord(record)
		if err != nil {
			// Mark this specific record as failed
			failures = append(failures, events.DynamoDBBatchItemFailure{
				ItemIdentifier: record.Change.SequenceNumber,
			})
		}
	}

	// Return the list of failures (if any) so AWS retries only those
	return events.DynamoDBEventResponse{
		BatchItemFailures: failures,
	}, nil
}

func processRecord(record events.DynamoDBEventRecord) error {
	// Idempotency & Logic
	if record.EventName == "INSERT" {
		fmt.Printf("Processing %s\n", record.Change.Keys["PK"].String())
	}
	return nil
}

func main() {
	lambda.Start(handleRequest)
}

[!TIP] Idempotency is Mandatory: Because of retries (network issues, timeouts), your function will process the same record twice eventually. Ensure your logic handles this (e.g., UPSERT instead of INSERT in downstream systems).

---

5. Summary

• ESM manages the polling and invocation loop.
• Batch Window allows you to trade latency for cost savings (fewer invocations).
• Partial Failures should be handled by returning BatchItemFailures to avoid reprocessing successful records.
• Poison Pills must be handled with Bisect/DLQ settings to prevent stream blockage.

Next, we will look at how to scale beyond Lambda using Kinesis Data Streams.