Batch Jobs



A batch job is a compute resource designed to run a containerized task until it completes. The execution is triggered by an event, such as an HTTP request, a message in a queue, or an object uploaded to a bucket.

A key feature of batch jobs is the ability to use spot instances, which can reduce compute costs by up to 90%.

Like other Stacktape compute resources, batch jobs are serverless, meaning you don't need to manage the underlying infrastructure. Stacktape handles server provisioning, scaling, and security for you. You can also equip your batch job's environment with a GPU in addition to CPU and RAM.

Under the hood

Stacktape uses a combination of AWS services to provide a seamless experience for running containerized jobs:

  • AWS Batch: Provisions the virtual machines where your job runs and manages the execution.
  • AWS Step Functions: Manages the job's lifecycle, including retries and timeouts, using a serverless state machine.
  • AWS Lambda: A trigger function that connects the event source to the batch job and starts its execution.

The execution flow is as follows:

  1. An event from an integration (like an API Gateway) invokes the trigger function.
  2. The trigger function starts the batch job state machine.
  3. The state machine queues the job in AWS Batch.
  4. AWS Batch provisions the necessary resources (like a VM) and runs your containerized job.

When to use

Batch jobs are ideal for long-running, resource-intensive tasks like data processing, ETL pipelines, or machine learning model training.

If you're unsure which compute resource to use, this table provides a comparison of container-based resources in Stacktape:

Resource typeDescriptionUse-cases
web-servicecontinuously running container with public endpoint and URLpublic APIs, websites
private-servicecontinuously running container with private endpointprivate APIs, services
worker-servicecontinuously running container not accessible from outsidecontinuous processing
multi-container-workloadcustom multi container workload - you can customize accessibility for each containermore complex use-cases requiring customization
batch-jobsimple container job - container is destroyed after job is doneone-off/scheduled processing jobs

Advantages

  • Pay-per-use: You only pay for the compute time your job consumes.
  • Resource flexibility: The environment automatically scales to provide the CPU, memory, and GPU your job needs.
  • Time flexibility: Batch jobs can run for as long as needed.
  • Secure by default: The underlying environment is securely managed by AWS.
  • Easy integration: Can be triggered by a wide variety of event sources.

Disadvantages

  • Slow start time: After a job is triggered, it's placed in a queue and can take anywhere from a few seconds to a few minutes to start.

Basic usage

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: schedule
          properties:
            scheduleRate: cron(0 14 * * ? *) # every day at 14:00 UTC
(async () => {
  const event = JSON.parse(process.env.STP_TRIGGER_EVENT_DATA);

  // process the event
})();

Container

Your code for a batch job runs inside a Docker container. You can configure its properties:

BatchJobContainerAPI referenceParent:BatchJob
packagingRequired

How to build or specify the container image for this job.

Type: BatchJobContainerPackaging

environment

Environment variables injected into the container at runtime.

Type: Array of EnvironmentVar

Use $ResourceParam() or $Secret() to inject database URLs, API keys, etc.


Image

A Docker container is a running instance of a Docker image. You can provide an image in four ways:

Environment variables

A list of environment variables to pass to the script or command.

Values can be:

environment:
  - name: STATIC_ENV_VAR
    value: my-env-var
  - name: DYNAMICALLY_SET_ENV_VAR
    value: $MyCustomDirective('input-for-my-directive')
  - name: DB_HOST
    value: $ResourceParam('myDatabase', 'host')
  - name: DB_PASSWORD
    value: $Secret('dbSecret.password')

Pre-set environment variables

Stacktape pre-sets the following environment variables for your job:

NameValue
STP_TRIGGER_EVENT_DATAContains JSON stringified event from an event integration that triggered this batch job.
STP_MAXIMUM_ATTEMPTSThe total number of attempts for this job before it is marked as failed.
STP_CURRENT_ATTEMPTThe current attempt number.

Logging

Any output from your code to stdout or stderr is captured and stored in an AWS CloudWatch log group.

You can view logs in two ways:

  • AWS CloudWatch Console: Get a direct link from the Stacktape Console or by using the stacktape stack-info command.
  • Stacktape CLI: Use the stacktape logs command to stream logs directly in your terminal.

Log storage can incur costs, so you can configure retentionDays to automatically delete old logs.

BatchJobLoggingAPI referenceParent:BatchJob
disabled

Disable logging to CloudWatch.

Type: boolean

retentionDaysDefault: 90

How many days to keep logs.

Type: number ENUM

Possible values: 13571430609012015018036540054573118273653

logForwarding

Forward logs to an external service (Datadog, Highlight.io, or any HTTP endpoint).

Type: HttpEndpointLogForwarding or HighlightLogForwarding or DatadogLogForwarding

Uses Kinesis Data Firehose (~$0.03/GB). Failed deliveries go to a backup S3 bucket.

Forwarding logs

You can forward logs to third-party services. See Log Forwarding for more details.

Computing resources

You can specify the amount of CPU, memory, and GPU for your batch job. AWS Batch selects the most cost-effective instance type that fits your job's requirements. To learn more about GPU instances, refer to the AWS Docs.

AWS reserves some memory for system processes. Requesting exactly 8192 MB (8 GB) may provision a larger instance than needed. Use 7680 MB instead to fit on a standard 8 GB instance.

Omit for CPU-only workloads.

BatchJobResourcesAPI referenceParent:BatchJob
cpuRequired

Number of vCPUs for the job (e.g., 1, 2, 4).

Type: number

memoryRequired

Memory in MB. Use slightly less than powers of 2 for efficient instance sizing.

Type: number

AWS reserves some memory for system processes. Requesting exactly 8192 MB (8 GB) may provision a larger instance than needed. Use 7680 MB instead to fit on a standard 8 GB instance.

gpu

Number of GPUs. The job will run on a GPU instance (NVIDIA A100, A10G, etc.).

Type: number

Omit for CPU-only workloads.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: batch-jobs/js-batch-job.js
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: schedule
          properties:
            scheduleRate: 'cron(0 14 * * ? *)' # every day at 14:00 UTC

Spot instances

Use this when: Your job can safely be restarted from the beginning (e.g., data imports, image processing, ML training with checkpoints). Combine with retryConfig to auto-retry on interruption.

Don't use when: Your job has side effects that can't be repeated (e.g., sending emails, charging payments) or must finish within a strict deadline.

If interrupted, your container gets a SIGTERM and 120 seconds to shut down gracefully.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      useSpotInstances: true

Retries

Auto-retry on failure, timeout, or Spot interruption.

BatchJobRetryConfigurationAPI referenceParent:BatchJob
attemptsDefault: 1

Max retry attempts before the job is marked as failed.

Type: number

retryIntervalSeconds

Seconds to wait between retries.

Type: number

retryIntervalMultiplierDefault: 1

Multiply wait time by this factor after each retry (exponential backoff).

Type: number

E.g., with retryIntervalSeconds: 5 and retryIntervalMultiplier: 2, waits are 5s, 10s, 20s, etc.

Timeout

Max run time in seconds. The job is killed if it exceeds this, then retried if retryConfig is set.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      timeout: 1200

Storage

Each batch job has its own ephemeral storage with a fixed size of 20GB. This storage is temporary and is deleted after the job completes or fails. To store data permanently, use Buckets.

Trigger events

Batch jobs are invoked in response to events from various integrations. A single job can have multiple triggers. The data payload from the trigger is available in the STP_TRIGGER_EVENT_DATA environment variable as a JSON string.

Be cautious when configuring event integrations. A high volume of events can trigger a large number of batch jobs, leading to unexpected costs. For example, 1000 HTTP requests to a connected API Gateway will result in 1000 job invocations.


HTTP Api event

Triggers the job in response to a request to a specified HTTP API Gateway. Routes are matched based on the most specific path. For more details, see the AWS Docs.

resources:
  myHttpApi:
    type: http-api-gateway

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: http-api-gateway
          properties:
            httpApiGatewayName: myHttpApi
            path: /hello
            method: GET

Lambda function connected to an HTTP API Gateway "myHttpApi"

HttpApiIntegrationAPI referenceParent:BatchJob
typeRequired
No description

Type: string "http-api-gateway"

properties.httpApiGatewayNameRequired

The name of the HTTP API Gateway.

Type: string

properties.methodRequired

The HTTP method that will trigger this integration.

Type: string ENUM

Possible values: *DELETEGETHEADOPTIONSPATCHPOSTPUT

You can specify an exact method (e.g., GET) or use * to match any method.

properties.pathRequired

The URL path that will trigger this integration.

Type: string

  • Exact path: /users
  • Path with parameter: /users/{id}. The id will be available in event.pathParameters.id.
  • Greedy path: /files/{proxy+}. This will match any path starting with /files/.
properties.authorizer

An authorizer to protect this route.

Type: StpAuthorizer

Unauthorized requests will be rejected with a 401 Unauthorized response.

properties.payloadFormatDefault: '1.0'

The payload format version for the Lambda integration.

Type: string ENUM

Possible values: 1.02.0

For details on the differences between formats, see the AWS documentation.

Cognito authorizer

Restricts access to users authenticated with a User Pool. The request must include an access token. If authorized, the job receives user claims in its payload.

resources:
  myGateway:
    type: http-api-gateway

  myUserPool:
    type: user-auth-pool
    properties:
      userVerificationType: email-code

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: http-api-gateway
          properties:
            httpApiGatewayName: myGateway
            path: /some-path
            method: '*'
            authorizer:
              type: cognito
              properties:
                userPoolName: myUserPool

Example cognito authorizer

import { CognitoIdentityProvider } from '@aws-sdk/client-cognito-identity-provider';

const cognito = new CognitoIdentityProvider({});

(async () => {
  const event = JSON.parse(process.env.STP_TRIGGER_EVENT_DATA);
  const userData = await cognito.getUser({ AccessToken: event.headers.authorization });
  // do something with your user data
})();

Example lambda batch job that fetches user data from Cognito

CognitoAuthorizerAPI reference
typeRequired

Cognito JWT authorizer

Type: string "cognito"

Configures an HTTP API authorizer that validates JSON Web Tokens (JWTs) issued by a Cognito user pool. This is the simplest way to protect routes when your users sign in via user-auth-pool.

Stacktape turns this into an API Gateway v2 authorizer of type JWT that checks the token's issuer and audience.

properties.userPoolNameRequired

Name of the user pool to protect the API

Type: string

The Stacktape name of the user-auth-pool resource whose tokens should be accepted by this HTTP API authorizer. Stacktape uses this to:

  • Set the expected audience to the user pool client ID.
  • Build the expected issuer URL based on the user pool and AWS region.

In practice this means only JWTs issued by this pool (and its client) will be considered valid.

properties.identitySources

Where to read the JWT from in the request

Type: Array of string

A list of identity sources that tell API Gateway where to look for the bearer token, using the $request.* syntax from API Gateway (for example '$request.header.Authorization').

If you omit this, Stacktape defaults to reading the token from the Authorization HTTP header, using a JWT authorizer as described in the API Gateway v2 authorizer docs (AWS::ApiGatewayV2::Authorizer).


Lambda authorizer

Uses a dedicated Lambda function to decide if a request is authorized. The authorizer function returns a policy document or a simple boolean response. You can configure identitySources to specify which parts of the request are used for authorization. To learn more, see the AWS Docs.

LambdaAuthorizerAPI reference
typeRequired

Lambda-based HTTP API authorizer

Type: string "lambda"

Configures an API Gateway request authorizer that runs a Lambda function to decide whether a request is allowed. This is useful when your authorization logic can't be expressed as simple JWT validation – for example when you check API keys, look up permissions in a database, or integrate with a non-JWT identity system.

Stacktape creates an AWS::ApiGatewayV2::Authorizer of type REQUEST and wires it up to your Lambda.

properties.functionNameRequired

Name of the authorizer function

Type: string

The Stacktape name of a function resource that should run for each authorized request. API Gateway calls this Lambda, passes request details, and uses its response to allow or deny access.

properties.iamResponse

Use IAM-style (v1) authorizer responses

Type: boolean

  • If true, your Lambda must return a full IAM policy document (the "v1" format).
  • If false or omitted, Stacktape enables simple responses (the HTTP API v2 payload format) so your Lambda can return a small JSON object with an isAuthorized flag and optional context.

This flag is wired to EnableSimpleResponses on the underlying AWS::ApiGatewayV2::Authorizer.

properties.identitySources

Where to read identity data from

Type: Array of string

A list of request fields API Gateway should pass into your Lambda authorizer (for example headers, query parameters, or stage variables) using the $request.* syntax.

When left empty, no specific identity sources are configured and your Lambda must inspect the incoming event directly.

properties.cacheResultSeconds

Cache authorizer results

Type: number

Number of seconds API Gateway should cache the result of the Lambda authorizer for a given identity. While cached, repeated requests skip calling your authorizer function and reuse the previous result.

This value is applied to AuthorizerResultTtlInSeconds. If omitted, Stacktape sets it to 0 (no caching).


Schedule event

Triggers the job on a defined schedule using either a fixed rate (e.g., every 5 minutes) or a cron expression.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        # invoke function every two hours
        - type: schedule
          properties:
            scheduleRate: rate(2 hours)
        # invoke function at 10:00 UTC every day
        - type: schedule
          properties:
            scheduleRate: cron(0 10 * * ? *)
ScheduleIntegrationAPI referenceParent:BatchJob
typeRequired
No description

Type: string "schedule"

properties.scheduleRateRequired

The schedule rate or cron expression.

Type: string

Examples: rate(2 hours), cron(0 10 * * ? *)

properties.input
No description

Type: UNSPECIFIED

properties.inputPath

A JSONPath expression to extract a portion of the event to pass to the target.

Type: string

This is useful for forwarding only a specific part of the event payload. You can only use one of input, inputPath, or inputTransformer.

Example:

inputPath: '$.detail'
properties.inputTransformer

Customizes the event payload sent to the target.

Type: EventInputTransformer

This allows you to extract values from the original event and use them to construct a new payload. You can only use one of input, inputPath, or inputTransformer.

Example:

inputTransformer:
  inputPathsMap:
    eventTime: '$.time'
  inputTemplate:
    message: 'This event occurred at <eventTime>.'

Event Bus event

Triggers the job when a matching event is received by a specified event bus. You can use the default AWS event bus or a custom event bus.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: event-bus
          properties:
            useDefaultBus: true
            eventPattern:
              source:
                - 'aws.autoscaling'
              region:
                - 'us-west-2'

Batch job connected to the default event bus

resources:
  myEventBus:
    type: event-bus

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: event-bus
          properties:
            eventBusName: myEventBus
            eventPattern:
              source:
                - 'mycustomsource'

Batch job connected to a custom event bus

EventBusIntegrationAPI referenceParent:BatchJob
typeRequired
No description

Type: string "event-bus"

properties.eventPatternRequired

A pattern to filter events from the event bus.

Type: EventBusIntegrationPattern

Only events that match this pattern will trigger the target. For details on the syntax, see the AWS EventBridge documentation on event patterns.

properties.eventBusArn

The ARN of an existing event bus.

Type: string

Use this to subscribe to an event bus that is not managed by your stack. You must specify only one of eventBusArn, eventBusName, or useDefaultBus.

properties.eventBusName

The name of an event bus defined in your stack's resources.

Type: string

You must specify only one of eventBusArn, eventBusName, or useDefaultBus.

properties.useDefaultBus

Uses the default AWS event bus.

Type: boolean

You must specify only one of eventBusArn, eventBusName, or useDefaultBus.

properties.onDeliveryFailure

A destination for events that fail to be delivered to the target.

Type: EventBusOnDeliveryFailure

In rare cases, an event might fail to be delivered. This property specifies an SQS queue where failed events will be sent.

properties.input
No description

Type: UNSPECIFIED

properties.inputPath

A JSONPath expression to extract a portion of the event to pass to the target.

Type: string

This is useful for forwarding only a specific part of the event payload. You can only use one of input, inputPath, or inputTransformer.

Example:

inputPath: '$.detail'
properties.inputTransformer

Customizes the event payload sent to the target.

Type: EventInputTransformer

This allows you to extract values from the original event and use them to construct a new payload. You can only use one of input, inputPath, or inputTransformer.

Example:

inputTransformer:
  inputPathsMap:
    instanceId: '$.detail.instance-id'
    instanceState: '$.detail.state'
  inputTemplate:
    message: 'Instance <instanceId> is now in state <instanceState>.'

SNS event

Triggers the job when a message is published to an SNS topic.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: sns
          properties:
            topicName: mySnsTopic

  mySnsTopic:
    type: sns-topic
SnsIntegrationAPI referenceParent:BatchJob
typeRequired
No description

Type: string "sns"

properties.snsTopicName

The name of an SNS topic defined in your stack's resources.

Type: string

You must specify either snsTopicName or snsTopicArn.

properties.snsTopicArn

The ARN of an existing SNS topic.

Type: string

Use this to subscribe to a topic that is not managed by your stack. You must specify either snsTopicName or snsTopicArn.

properties.filterPolicy
No description

Type: UNSPECIFIED

properties.onDeliveryFailure

A destination for messages that fail to be delivered to the target.

Type: SnsOnDeliveryFailure

In rare cases (e.g., if the target function cannot scale fast enough), a message might fail to be delivered. This property specifies an SQS queue where failed messages will be sent.


SQS event

Triggers the job when messages are available in an SQS queue. Messages are processed in batches. If the job fails to start, messages return to the queue after the visibility timeout. If the job starts but then fails, the messages are considered processed.

A single queue should be consumed by a single compute resource. If you need a fan-out pattern, consider using an SNS or EventBus integration.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: sqs
          properties:
            sqsQueueName: mySqsQueue

  mySqsQueue:
    type: sqs-queue
SqsIntegrationAPI referenceParent:BatchJob
typeRequired
No description

Type: string "sqs"

properties.sqsQueueName

The name of an SQS queue defined in your stack's resources.

Type: string

You must specify either sqsQueueName or sqsQueueArn.

properties.sqsQueueArn

The ARN of an existing SQS queue.

Type: string

Use this to consume messages from a queue that is not managed by your stack. You must specify either sqsQueueName or sqsQueueArn.

properties.batchSizeDefault: 10

The maximum number of records to process in a single batch.

Type: number

Maximum is 10,000.

properties.maxBatchWindowSeconds

The maximum time (in seconds) to wait before invoking the function with a batch of records.

Type: number

Maximum is 300 seconds. If not set, the function is invoked as soon as messages are available.


Kinesis event

Triggers the job when records are available in a Kinesis Data Stream. It's similar to SQS but designed for real-time data streaming. You can add a Kinesis stream using CloudFormation resources.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: kinesis-stream
          properties:
            autoCreateConsumer: true
            maxBatchWindowSeconds: 30
            batchSize: 200
            streamArn: $CfResourceParam('myKinesisStream', 'Arn')
            onFailure:
              arn: $CfResourceParam('myOnFailureSqsQueue', 'Arn')
              type: sqs

cloudformationResources:
  myKinesisStream:
    Type: AWS::Kinesis::Stream
    Properties:
      ShardCount: 1

  myOnFailureSqsQueue:
    Type: AWS::SQS::Queue
KinesisIntegrationAPI referenceParent:BatchJob
typeRequired
No description

Type: string "kinesis-stream"

properties.kinesisStreamName

The name of a Kinesis stream defined in your stack's resources.

Type: string

You must specify either kinesisStreamName or streamArn.

properties.streamArn

The ARN of an existing Kinesis stream to consume records from.

Type: string

Use this to consume from a stream that is not managed by your stack. You must specify either kinesisStreamName or streamArn.

properties.consumerArn

The ARN of a specific stream consumer to use.

Type: string

This cannot be used with autoCreateConsumer.

properties.autoCreateConsumer

Automatically creates a dedicated stream consumer for this integration.

Type: boolean

This is recommended for minimizing latency and maximizing throughput. For more details, see the AWS documentation on stream consumers. This cannot be used with consumerArn.

properties.maxBatchWindowSeconds

The maximum time (in seconds) to wait before invoking the function with a batch of records.

Type: number

Maximum is 300 seconds.

properties.batchSizeDefault: 10

The maximum number of records to process in a single batch.

Type: number

Maximum is 10,000.

properties.startingPositionDefault: TRIM_HORIZON

The position in the stream from which to start reading records.

Type: string ENUM

Possible values: LATESTTRIM_HORIZON

  • LATEST: Read only new records.
  • TRIM_HORIZON: Read all available records from the beginning of the stream.
properties.maximumRetryAttempts

The number of times to retry a failed batch of records.

Type: number

Important: If an error occurs, the entire batch is retried, including records that were processed successfully. Your function should be idempotent to handle this.

properties.onFailure

A destination (SQS queue or SNS topic) for batches that fail after all retry attempts.

Type: DestinationOnFailure

properties.parallelizationFactor

The number of batches to process concurrently from the same shard.

Type: number

properties.bisectBatchOnFunctionError

Splits a failed batch in two before retrying.

Type: boolean

This can be useful if a failure is caused by a batch being too large.


DynamoDB event

Triggers the job in response to item-level changes in a DynamoDB table. You must enable DynamoDB Streams on your table.

resources:
  myDynamoDbTable:
    type: dynamo-db-table
    properties:
      primaryKey:
        partitionKey:
          name: id
          type: string
      streamType: NEW_AND_OLD_IMAGES

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: dynamo-db-stream
          properties:
            streamArn: $ResourceParam('myDynamoDbTable', 'streamArn')
            batchSize: 200
DynamoDbIntegrationAPI referenceParent:BatchJob
typeRequired
No description

Type: string "dynamo-db-stream"

properties.streamArnRequired

The ARN of the DynamoDB table stream.

Type: string

properties.maxBatchWindowSeconds

The maximum time (in seconds) to wait before invoking the function with a batch of records.

Type: number

Maximum is 300 seconds.

properties.batchSizeDefault: 100

The maximum number of records to process in a single batch.

Type: number

Maximum is 1,000.

properties.startingPositionDefault: TRIM_HORIZON

The position in the stream from which to start reading records.

Type: string

  • LATEST: Read only new records.
  • TRIM_HORIZON: Read all available records from the beginning of the stream.
properties.maximumRetryAttempts

The number of times to retry a failed batch of records.

Type: number

Important: If an error occurs, the entire batch is retried, including records that were processed successfully. Your function should be idempotent to handle this.

properties.onFailure

A destination (SQS queue or SNS topic) for batches that fail after all retry attempts.

Type: DestinationOnFailure

properties.parallelizationFactor

The number of batches to process concurrently from the same shard.

Type: number

properties.bisectBatchOnFunctionError

Splits a failed batch in two before retrying.

Type: boolean

This can be useful if a failure is caused by a batch being too large.


S3 event

Triggers the job when a specific event (like object created) occurs in an S3 bucket.

resources:
  myBucket:
    type: bucket

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: s3
          properties:
            bucketArn: $ResourceParam('myBucket', 'arn')
            s3EventType: 's3:ObjectCreated:*'
            filterRule:
              prefix: order-
              suffix: .jpg
S3IntegrationAPI referenceParent:BatchJob
typeRequired
No description

Type: string "s3"

properties.bucketArnRequired

The ARN of the S3 bucket to monitor for events.

Type: string

properties.s3EventTypeRequired

The type of S3 event that will trigger the function.

Type: string ENUM

Possible values: s3:ObjectCreated:*s3:ObjectCreated:CompleteMultipartUploads3:ObjectCreated:Copys3:ObjectCreated:Posts3:ObjectCreated:Puts3:ObjectRemoved:*s3:ObjectRemoved:Deletes3:ObjectRemoved:DeleteMarkerCreateds3:ObjectRestore:*s3:ObjectRestore:Completeds3:ObjectRestore:Posts3:ReducedRedundancyLostObjects3:Replication:*s3:Replication:OperationFailedReplications3:Replication:OperationMissedThresholds3:Replication:OperationNotTrackeds3:Replication:OperationReplicatedAfterThreshold

properties.filterRule

A filter to apply to objects, so the function is only triggered for relevant objects.

Type: S3FilterRule

S3FilterRuleAPI referenceParent:S3Integration
prefix

The prefix that an object's key must have to trigger the function.

Type: string

suffix

The suffix that an object's key must have to trigger the function.

Type: string


Cloudwatch Log event

Triggers the job when a log record is added to a specified CloudWatch log group. The event payload is BASE64 encoded and GZIP compressed.

resources:
  myLogProducingLambda:
    type: function
    properties:
      packaging:
        type: stacktape-lambda-buildpack
        properties:
          entryfilePath: lambdas/log-producer.ts

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: cloudwatch-log
          properties:
            logGroupArn: $ResourceParam('myLogProducingLambda', 'arn')
CloudwatchLogIntegrationAPI referenceParent:BatchJob
typeRequired
No description

Type: string "cloudwatch-log"

properties.logGroupArnRequired

The ARN of the log group to watch for new records.

Type: string

properties.filter

A filter pattern to apply to the log records.

Type: string

Only logs that match this pattern will trigger the function. For details on the syntax, see the AWS documentation on filter and pattern syntax.


Application Load Balancer event

Triggers the job when an Application Load Balancer receives an HTTP request matching specified conditions (e.g., path, headers, method).

resources:
  # load balancer which routes traffic to the function
  myLoadBalancer:
    type: application-load-balancer

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: application-load-balancer
          properties:
            # referencing load balancer defined above
            loadBalancerName: myLoadBalancer
            priority: 1
            paths:
              - /invoke-my-job
              - /another-path
ApplicationLoadBalancerIntegrationAPI referenceParent:BatchJob
typeRequired

Triggers a function when an Application Load Balancer receives a matching HTTP request.

Type: string "application-load-balancer"

You can route requests based on HTTP method, path, headers, query parameters, and source IP address.

properties.loadBalancerNameRequired

The name of the Application Load Balancer.

Type: string

This must reference a load balancer defined in your Stacktape configuration.

properties.priorityRequired

The priority of this integration rule.

Type: number

Load balancer rules are evaluated in order from the lowest priority to the highest. The first rule that matches an incoming request will handle it.

properties.listenerPort

The port of the load balancer listener to attach to.

Type: number

You only need to specify this if the load balancer uses custom listeners.

properties.paths

A list of URL paths that will trigger this integration.

Type: Array of string

The request will be routed if its path matches any of the paths in this list. The comparison is case-sensitive and supports * and ? wildcards.

Example: /users, /articles/*

properties.methods

A list of HTTP methods that will trigger this integration.

Type: Array of string

Example: GET, POST, DELETE

properties.hosts

A list of hostnames that will trigger this integration.

Type: Array of string

The hostname is parsed from the Host header of the request. Wildcards (* and ?) are supported.

Example: api.example.com, *.myapp.com

properties.headers

A list of header conditions that the request must match.

Type: Array of LbHeaderCondition

All header conditions must be met for the request to be routed.

properties.queryParams

A list of query parameter conditions that the request must match.

Type: Array of LbQueryParamCondition

All query parameter conditions must be met for the request to be routed.

properties.sourceIps

A list of source IP addresses (in CIDR format) that are allowed to trigger this integration.

Type: Array of string

Note: If the client is behind a proxy, this will be the IP address of the proxy.


Accessing other resources

By default, AWS resources cannot communicate with each other. Access must be granted explicitly using IAM permissions. Stacktape handles most of this automatically, but for resource-to-resource communication, you need to configure permissions.

Relational Databases are an exception, as they use their own connection-string-based access control.

There are two ways to grant permissions:

Using connectTo

The connectTo property is a simplified way to grant basic access to other Stacktape-managed resources. It automatically configures the necessary IAM permissions and injects environment variables with connection details into your batch job.

resources:
  photosBucket:
    type: bucket

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      connectTo:
        # access to the bucket
        - photosBucket
        # access to AWS SES
        - aws:ses

List the names of resources this workload needs to communicate with. Stacktape automatically:

  • Grants IAM permissions (e.g., S3 read/write, SQS send/receive)
  • Opens network access (security group rules for databases, Redis)
  • Injects environment variables with connection details: STP_[RESOURCE_NAME]_[PARAM]

Example: connectTo: ["myDatabase", "myBucket"] gives this workload full access to both resources and injects STP_MY_DATABASE_CONNECTION_STRING, STP_MY_BUCKET_NAME, etc.

Using iamRoleStatements

For fine-grained control, you can provide raw IAM role statements. This allows you to define custom permissions to any AWS resource.

resources:
  myBatchJob:
    type: batch-job
    properties:
      resources:
        cpu: 2
        memory: 1800
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      iamRoleStatements:
        - Resource:
            - $CfResourceParam('NotificationTopic', 'Arn')
          Effect: Allow
          Action:
            - 'sns:Publish'

cloudformationResources:
  NotificationTopic:
    Type: AWS::SNS::Topic

Default VPC connection

Certain resources, like Relational Databases, must be placed within a VPC. If your stack contains such resources, Stacktape automatically creates a default VPC and connects them to it.

Batch jobs are connected to this VPC by default, allowing them to communicate with other VPC-enabled resources without extra configuration. To learn more, see our guide on VPCs.

Referenceable parameters

The following parameters can be easily referenced using $ResourceParam directive directive.

To learn more about referencing parameters, refer to referencing parameters.

jobDefinitionArn
  • Arn of the job definition resource

  • Usage: $ResourceParam('<<resource-name>>', 'jobDefinitionArn')
stateMachineArn
  • Arn of the state machine controlling the execution flow of the batch job

  • Usage: $ResourceParam('<<resource-name>>', 'stateMachineArn')
logGroupArn
  • Arn of the log group aggregating logs from the batch job

  • Usage: $ResourceParam('<<resource-name>>', 'logGroupArn')

Pricing

You are charged for:

  • The compute instances running your batch jobs.
  • A negligible amount for the Lambda functions and Step Functions that manage the job's execution.

Pricing depends on the instance type and region. You can significantly reduce costs (by up to 90%) by using spot instances.

API reference

BatchJobAPI reference
typeRequired
No description

Type: string "batch-job"

properties.containerRequired

Docker container image and environment for the job.

Type: BatchJobContainer

properties.resourcesRequired

CPU, memory, and GPU requirements. AWS auto-provisions a matching instance.

Type: BatchJobResources

properties.timeout

Max run time in seconds. The job is killed if it exceeds this, then retried if retryConfig is set.

Type: number

properties.useSpotInstances

Use discounted spare AWS capacity. Saves up to 90%, but jobs can be interrupted.

Type: boolean

Use this when: Your job can safely be restarted from the beginning (e.g., data imports, image processing, ML training with checkpoints). Combine with retryConfig to auto-retry on interruption.

Don't use when: Your job has side effects that can't be repeated (e.g., sending emails, charging payments) or must finish within a strict deadline.

If interrupted, your container gets a SIGTERM and 120 seconds to shut down gracefully.

properties.logging

Container logging (stdout/stderr). Sent to CloudWatch, viewable with stacktape logs.

Type: BatchJobLogging

properties.retryConfig

Auto-retry on failure, timeout, or Spot interruption.

Type: BatchJobRetryConfiguration

properties.events

Events that trigger this job (schedules, HTTP requests, S3 uploads, SQS messages, etc.).

Type: Array of ApplicationLoadBalancerIntegration or SnsIntegration or SqsIntegration or KinesisIntegration or DynamoDbIntegration or S3Integration or ScheduleIntegration or CloudwatchLogIntegration or HttpApiIntegration or EventBusIntegration

properties.connectTo

Give this resource access to other resources in your stack.

Type: Array of string

List the names of resources this workload needs to communicate with. Stacktape automatically:

  • Grants IAM permissions (e.g., S3 read/write, SQS send/receive)
  • Opens network access (security group rules for databases, Redis)
  • Injects environment variables with connection details: STP_[RESOURCE_NAME]_[PARAM]

Example: connectTo: ["myDatabase", "myBucket"] gives this workload full access to both resources and injects STP_MY_DATABASE_CONNECTION_STRING, STP_MY_BUCKET_NAME, etc.

properties.iamRoleStatements

Raw IAM policy statements for permissions not covered by connectTo.

Type: Array of StpIamRoleStatement

Added as a separate policy alongside auto-generated permissions. Use this for accessing AWS services directly (e.g., Rekognition, Textract, Bedrock).

overrides

Escape hatch to modify the underlying CloudFormation resources Stacktape creates.

Type: Object

Use dot-notation paths to override specific properties on any child resource. Find resource logical IDs with stacktape stack-info --detailed.

overrides:
  MyDbInstance:
    Properties.StorageEncrypted: true
CognitoAuthorizerAPI reference
typeRequired

Cognito JWT authorizer

Type: string "cognito"

Configures an HTTP API authorizer that validates JSON Web Tokens (JWTs) issued by a Cognito user pool. This is the simplest way to protect routes when your users sign in via user-auth-pool.

Stacktape turns this into an API Gateway v2 authorizer of type JWT that checks the token's issuer and audience.

properties.userPoolNameRequired

Name of the user pool to protect the API

Type: string

The Stacktape name of the user-auth-pool resource whose tokens should be accepted by this HTTP API authorizer. Stacktape uses this to:

  • Set the expected audience to the user pool client ID.
  • Build the expected issuer URL based on the user pool and AWS region.

In practice this means only JWTs issued by this pool (and its client) will be considered valid.

properties.identitySources

Where to read the JWT from in the request

Type: Array of string

A list of identity sources that tell API Gateway where to look for the bearer token, using the $request.* syntax from API Gateway (for example '$request.header.Authorization').

If you omit this, Stacktape defaults to reading the token from the Authorization HTTP header, using a JWT authorizer as described in the API Gateway v2 authorizer docs (AWS::ApiGatewayV2::Authorizer).

LambdaAuthorizerAPI reference
typeRequired

Lambda-based HTTP API authorizer

Type: string "lambda"

Configures an API Gateway request authorizer that runs a Lambda function to decide whether a request is allowed. This is useful when your authorization logic can't be expressed as simple JWT validation – for example when you check API keys, look up permissions in a database, or integrate with a non-JWT identity system.

Stacktape creates an AWS::ApiGatewayV2::Authorizer of type REQUEST and wires it up to your Lambda.

properties.functionNameRequired

Name of the authorizer function

Type: string

The Stacktape name of a function resource that should run for each authorized request. API Gateway calls this Lambda, passes request details, and uses its response to allow or deny access.

properties.iamResponse

Use IAM-style (v1) authorizer responses

Type: boolean

  • If true, your Lambda must return a full IAM policy document (the "v1" format).
  • If false or omitted, Stacktape enables simple responses (the HTTP API v2 payload format) so your Lambda can return a small JSON object with an isAuthorized flag and optional context.

This flag is wired to EnableSimpleResponses on the underlying AWS::ApiGatewayV2::Authorizer.

properties.identitySources

Where to read identity data from

Type: Array of string

A list of request fields API Gateway should pass into your Lambda authorizer (for example headers, query parameters, or stage variables) using the $request.* syntax.

When left empty, no specific identity sources are configured and your Lambda must inspect the incoming event directly.

properties.cacheResultSeconds

Cache authorizer results

Type: number

Number of seconds API Gateway should cache the result of the Lambda authorizer for a given identity. While cached, repeated requests skip calling your authorizer function and reuse the previous result.

This value is applied to AuthorizerResultTtlInSeconds. If omitted, Stacktape sets it to 0 (no caching).

EventInputTransformerAPI referenceParent:EventBusIntegration or ScheduleIntegration
inputTemplateRequired
No description

Type: UNSPECIFIED

inputPathsMap
No description

Type: UNSPECIFIED

EventBusIntegrationPatternAPI referenceParent:EventBusIntegration
version
No description

Type: UNSPECIFIED

detail-type
No description

Type: UNSPECIFIED

source
No description

Type: UNSPECIFIED

account
No description

Type: UNSPECIFIED

region
No description

Type: UNSPECIFIED

resources
No description

Type: UNSPECIFIED

detail
No description

Type: UNSPECIFIED

replay-name
No description

Type: UNSPECIFIED

SnsOnDeliveryFailureAPI referenceParent:SnsIntegration
sqsQueueArn

The ARN of the SQS queue for failed messages.

Type: string

sqsQueueName

The name of an SQS queue (defined in your Stacktape configuration) for failed messages.

Type: string

DestinationOnFailureAPI referenceParent:DynamoDbIntegration or KinesisIntegration
arnRequired

The ARN of the SNS topic or SQS queue for failed batches.

Type: string

typeRequired

The type of the destination.

Type: string ENUM

Possible values: snssqs

S3FilterRuleAPI referenceParent:S3Integration
prefix

The prefix that an object's key must have to trigger the function.

Type: string

suffix

The suffix that an object's key must have to trigger the function.

Type: string

LbHeaderConditionAPI referenceParent:ApplicationLoadBalancerIntegration
headerNameRequired

The name of the HTTP header.

Type: string

valuesRequired

A list of allowed values for the header.

Type: Array of string

The condition is met if the header's value in the incoming request matches any of the values in this list. The comparison is case-insensitive.

LbQueryParamConditionAPI referenceParent:ApplicationLoadBalancerIntegration
paramNameRequired

The name of the query parameter.

Type: string

valuesRequired

A list of allowed values for the query parameter.

Type: Array of string

The condition is met if the query parameter's value in the incoming request matches any of the values in this list. The comparison is case-insensitive.

KinesisIntegrationAPI referenceParent:BatchJob
typeRequired
No description

Type: string "kinesis-stream"

properties.kinesisStreamName

The name of a Kinesis stream defined in your stack's resources.

Type: string

You must specify either kinesisStreamName or streamArn.

properties.streamArn

The ARN of an existing Kinesis stream to consume records from.

Type: string

Use this to consume from a stream that is not managed by your stack. You must specify either kinesisStreamName or streamArn.

properties.consumerArn

The ARN of a specific stream consumer to use.

Type: string

This cannot be used with autoCreateConsumer.

properties.autoCreateConsumer

Automatically creates a dedicated stream consumer for this integration.

Type: boolean

This is recommended for minimizing latency and maximizing throughput. For more details, see the AWS documentation on stream consumers. This cannot be used with consumerArn.

properties.maxBatchWindowSeconds

The maximum time (in seconds) to wait before invoking the function with a batch of records.

Type: number

Maximum is 300 seconds.

properties.batchSizeDefault: 10

The maximum number of records to process in a single batch.

Type: number

Maximum is 10,000.

properties.startingPositionDefault: TRIM_HORIZON

The position in the stream from which to start reading records.

Type: string ENUM

Possible values: LATESTTRIM_HORIZON

  • LATEST: Read only new records.
  • TRIM_HORIZON: Read all available records from the beginning of the stream.
properties.maximumRetryAttempts

The number of times to retry a failed batch of records.

Type: number

Important: If an error occurs, the entire batch is retried, including records that were processed successfully. Your function should be idempotent to handle this.

properties.onFailure

A destination (SQS queue or SNS topic) for batches that fail after all retry attempts.

Type: DestinationOnFailure

properties.parallelizationFactor

The number of batches to process concurrently from the same shard.

Type: number

properties.bisectBatchOnFunctionError

Splits a failed batch in two before retrying.

Type: boolean

This can be useful if a failure is caused by a batch being too large.

EnvironmentVarAPI referenceParent:BatchJobContainer
nameRequired

Environment variable name (e.g., DATABASE_URL, API_KEY).

Type: string

valueRequired

Environment variable value. Numbers and booleans are converted to strings.

Type: string or number or boolean

StpIamRoleStatementAPI referenceParent:BatchJob
ResourceRequired

ARNs of the AWS resources this statement applies to. Use "*" for all resources.

Type: Array of string

Sid

Optional identifier for this statement (for readability).

Type: string

EffectDefault: Allow

Whether to allow or deny the specified actions.

Type: string

Action

AWS actions to allow or deny (e.g., s3:GetObject, sqs:SendMessage).

Type: Array of string

Condition
No description

Type: UNSPECIFIED