A kafka-topic event lets AWS Lambda poll one Kafka topic and invoke your function with batches of records. The
simplest source is a Stacktape Kafka cluster: Stacktape configures IAM access,
networking, and the Lambda event-source mapping.
Kafka is an expensive choice if you only need background jobs. Prefer an SQS queue or Kinesis stream unless Kafka protocol compatibility is a requirement.
Stacktape Kafka cluster
startFrom is deliberately required:
latestprocesses records that arrive after the mapping becomes active.earlieststarts from the oldest retained records and can cause a large first processing burst.
Stacktape creates shared Lambda and STS interface VPC endpoints for an on-demand Lambda poller attached to a native
cluster. Those endpoints have their own hourly and data-processing charges in every availability zone. The Lambda
function itself does not need joinDefaultVpc merely to consume the trigger; the poller has separate networking.
The endpoints use private DNS for the shared VPC and their default endpoint policy, so ordinary workload calls remain
governed by workload IAM rather than being accidentally blocked at the endpoint.
The topic must already exist. MSK Serverless disables automatic topic creation and CloudFormation does not support declarative topics for Serverless clusters. See Create topics.
Existing Amazon MSK cluster
Use mskClusterArn for an existing cluster in the same AWS account and region:
This v1 contract assumes the cluster uses IAM authentication. Stacktape scopes the function role to the exact cluster, topic, and consumer group. You remain responsible for the existing cluster's security groups, subnets, routes, and on-demand poller connectivity. A public subnet and internet gateway alone are not enough: the event-source poller needs NAT or private Lambda and STS endpoints. Secret-based SCRAM and mTLS for an existing MSK ARN are not exposed in this contract; use the self-managed form when those credentials are required.
Cross-account MSK event sources require an AWS-managed VPC connection and are not supported by this v1 event.
A native Stacktape cluster in a reused VPC can be used by directly connected workloads, but cannot be a Lambda Kafka
event source in v1. Lambda and STS private-DNS endpoints are VPC-global and Stacktape will not collide with endpoints
owned by another project. Use an existing mskClusterArn after providing its networking yourself, or use a
Stacktape-owned VPC outside us-east-1 for the native cluster-and-trigger path.
Self-managed Kafka
For Kafka outside Amazon MSK, provide bootstrap brokers and a Secrets Manager authentication secret:
The vpc block configures the event-source poller's network interfaces. Setting joinDefaultVpc on the function
does not make the poller able to reach private brokers. Provide broker-reachable subnets and security groups; multi-AZ
placement is recommended. The poller's subnets also need NAT or Lambda and STS interface endpoints; add a Secrets Manager endpoint when
authentication secrets cannot be reached through NAT.
Supported authentication types are BASIC_AUTH, SASL_SCRAM_256_AUTH, SASL_SCRAM_512_AUTH, and MTLS. For SASL,
the secret contains username and password. For mTLS, configure clientCertificate and optionally
serverRootCaCertificate secret ARNs. Stacktape grants secret reads. If a secret is encrypted with a customer-managed
KMS key, add kms:Decrypt for that exact key with iamRoleStatements; a key ARN cannot be inferred from the secret ARN.
Batching and consumer groups
batchSize accepts 1–10,000 records. Omit maxBatchWindowSeconds to keep Lambda's 500 ms Kafka default. Unlike SQS,
Kinesis, and DynamoDB mappings, Kafka can use a batch larger than 10 while retaining that sub-second default.
Omit consumerGroupId for the safe default: AWS assigns a unique group to the mapping. The advanced explicit value is
for retaining an existing group identity or controlled migrations. It is immutable, must be unique across mappings,
and an existing group's committed offsets take precedence over startFrom. Sharing an ID with another active consumer
splits partitions instead of duplicating messages.
Changing a source, topic, start position, or explicit group replaces the mapping. AWS can reject create-before-delete replacement when the old mapping still owns the same explicit group. In that case, remove the mapping in one deployment and add it back in the next.
Kafka event functions can have a maximum timeout of 840 seconds (14 minutes). Records are delivered at least once; make handlers idempotent and decode Kafka record keys and values from base64.
KafkaTopicIntegrationSee AWS Lambda Kafka event-source documentation for the runtime event shape and current service constraints.