A batch job is a compute resource designed to run a containerized task until it completes. The execution is triggered by an event, such as an HTTP request, a message in a queue, or an object uploaded to a bucket.

A key feature of batch jobs is the ability to use spot instances, which can reduce compute costs by up to 90%.

Like other Stacktape compute resources, batch jobs are serverless, meaning you don't need to manage the underlying infrastructure. Stacktape handles server provisioning, scaling, and security for you. You can also equip your batch job's environment with a GPU in addition to CPU and RAM.

Under the hood

Stacktape uses a combination of AWS services to provide a seamless experience for running containerized jobs:

• AWS Batch: Provisions the virtual machines where your job runs and manages the execution.
• AWS Step Functions: Manages the job's lifecycle, including retries and timeouts, using a serverless state machine.
• AWS Lambda: A trigger function that connects the event source to the batch job and starts its execution.

The execution flow is as follows:

1. An event from an integration (like an API Gateway) invokes the trigger function.
2. The trigger function starts the batch job state machine.
3. The state machine queues the job in AWS Batch.
4. AWS Batch provisions the necessary resources (like a VM) and runs your containerized job.

When to use

Batch jobs are ideal for long-running, resource-intensive tasks like data processing, ETL pipelines, or machine learning model training.

If you're unsure which compute resource to use, this table provides a comparison of container-based resources in Stacktape:

Resource type	Description	Use-cases
web-service	continuously running container with public endpoint and URL	public APIs, websites
private-service	continuously running container with private endpoint	private APIs, services
worker-service	continuously running container not accessible from outside	continuous processing
multi-container-workload	custom multi container workload - you can customize accessibility for each container	more complex use-cases requiring customization
batch-job	simple container job - container is destroyed after job is done	one-off/scheduled processing jobs

Advantages

• Pay-per-use: You only pay for the compute time your job consumes.
• Resource flexibility: The environment automatically scales to provide the CPU, memory, and GPU your job needs.
• Time flexibility: Batch jobs can run for as long as needed.
• Secure by default: The underlying environment is securely managed by AWS.
• Easy integration: Can be triggered by a wide variety of event sources.

Disadvantages

• Slow start time: After a job is triggered, it's placed in a queue and can take anywhere from a few seconds to a few minutes to start.

Basic usage

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: schedule
          properties:
            scheduleRate: cron(0 14 * * ? *) # every day at 14:00 UTC

(async () => {
  const event = JSON.parse(process.env.STP_TRIGGER_EVENT_DATA);

  // process the event
})();

Container

Your code for a batch job runs inside a Docker container. You can configure its properties:

Image

A Docker container is a running instance of a Docker image. You can provide an image in four ways:

• Images built using stacktape-image-buildpack
• Images built using external-buildpack
• Images built from a custom-dockerfile
• prebuilt-images

Environment variables

environment:
  - name: STATIC_ENV_VAR
    value: my-env-var
  - name: DYNAMICALLY_SET_ENV_VAR
    value: $MyCustomDirective('input-for-my-directive')
  - name: DB_HOST
    value: $ResourceParam('myDatabase', 'host')
  - name: DB_PASSWORD
    value: $Secret('dbSecret.password')

Pre-set environment variables

Stacktape pre-sets the following environment variables for your job:

Name	Value
STP_TRIGGER_EVENT_DATA	Contains JSON stringified event from an event integration that triggered this batch job.
STP_MAXIMUM_ATTEMPTS	The total number of attempts for this job before it is marked as failed.
STP_CURRENT_ATTEMPT	The current attempt number.

Logging

Any output from your code to stdout or stderr is captured and stored in an AWS CloudWatch log group.

You can view logs in two ways:

• AWS CloudWatch Console: Get a direct link from the Stacktape Console or by using the stacktape stack-info command.
• Stacktape CLI: Use the stacktape logs command to stream logs directly in your terminal.

Log storage can incur costs, so you can configure retentionDays to automatically delete old logs.

Forwarding logs

You can forward logs to third-party services. See Log Forwarding for more details.

Computing resources

You can specify the amount of CPU, memory, and GPU for your batch job. AWS Batch selects the most cost-effective instance type that fits your job's requirements. To learn more about GPU instances, refer to the AWS Docs.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: batch-jobs/js-batch-job.js
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: schedule
          properties:
            scheduleRate: 'cron(0 14 * * ? *)' # every day at 14:00 UTC

Spot instances

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      useSpotInstances: true

Retries

Timeout

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      timeout: 1200

Storage

Each batch job has its own ephemeral storage with a fixed size of 20GB. This storage is temporary and is deleted after the job completes or fails. To store data permanently, use Buckets.

Trigger events

Batch jobs are invoked in response to events from various integrations. A single job can have multiple triggers. The data payload from the trigger is available in the STP_TRIGGER_EVENT_DATA environment variable as a JSON string.

HTTP Api event

Triggers the job in response to a request to a specified HTTP API Gateway. Routes are matched based on the most specific path. For more details, see the AWS Docs.

resources:
  myHttpApi:
    type: http-api-gateway

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: http-api-gateway
          properties:
            httpApiGatewayName: myHttpApi
            path: /hello
            method: GET

Cognito authorizer

Restricts access to users authenticated with a User Pool. The request must include an access token. If authorized, the job receives user claims in its payload.

resources:
  myGateway:
    type: http-api-gateway

  myUserPool:
    type: user-auth-pool
    properties:
      userVerificationType: email-code

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: http-api-gateway
          properties:
            httpApiGatewayName: myGateway
            path: /some-path
            method: '*'
            authorizer:
              type: cognito
              properties:
                userPoolName: myUserPool

import { CognitoIdentityProvider } from '@aws-sdk/client-cognito-identity-provider';

const cognito = new CognitoIdentityProvider({});

(async () => {
  const event = JSON.parse(process.env.STP_TRIGGER_EVENT_DATA);
  const userData = await cognito.getUser({ AccessToken: event.headers.authorization });
  // do something with your user data
})();

Lambda authorizer

Uses a dedicated Lambda function to decide if a request is authorized. The authorizer function returns a policy document or a simple boolean response. You can configure identitySources to specify which parts of the request are used for authorization. To learn more, see the AWS Docs.

Schedule event

Triggers the job on a defined schedule using either a fixed rate (e.g., every 5 minutes) or a cron expression.

• Learn more about rate expressions
• Learn more about Cron expressions

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        # invoke function every two hours
        - type: schedule
          properties:
            scheduleRate: rate(2 hours)
        # invoke function at 10:00 UTC every day
        - type: schedule
          properties:
            scheduleRate: cron(0 10 * * ? *)

Event Bus event

Triggers the job when a matching event is received by a specified event bus. You can use the default AWS event bus or a custom event bus.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: event-bus
          properties:
            useDefaultBus: true
            eventPattern:
              source:
                - 'aws.autoscaling'
              region:
                - 'us-west-2'

resources:
  myEventBus:
    type: event-bus

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: event-bus
          properties:
            eventBusName: myEventBus
            eventPattern:
              source:
                - 'mycustomsource'

SNS event

Triggers the job when a message is published to an SNS topic.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: sns
          properties:
            topicName: mySnsTopic

  mySnsTopic:
    type: sns-topic

SQS event

Triggers the job when messages are available in an SQS queue. Messages are processed in batches. If the job fails to start, messages return to the queue after the visibility timeout. If the job starts but then fails, the messages are considered processed.

A single queue should be consumed by a single compute resource. If you need a fan-out pattern, consider using an SNS or EventBus integration.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: sqs
          properties:
            sqsQueueName: mySqsQueue

  mySqsQueue:
    type: sqs-queue

Kinesis event

Triggers the job when records are available in a Kinesis Data Stream. It's similar to SQS but designed for real-time data streaming. You can add a Kinesis stream using CloudFormation resources.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: kinesis-stream
          properties:
            autoCreateConsumer: true
            maxBatchWindowSeconds: 30
            batchSize: 200
            streamArn: $CfResourceParam('myKinesisStream', 'Arn')
            onFailure:
              arn: $CfResourceParam('myOnFailureSqsQueue', 'Arn')
              type: sqs

cloudformationResources:
  myKinesisStream:
    Type: AWS::Kinesis::Stream
    Properties:
      ShardCount: 1

  myOnFailureSqsQueue:
    Type: AWS::SQS::Queue

DynamoDB event

Triggers the job in response to item-level changes in a DynamoDB table. You must enable DynamoDB Streams on your table.

resources:
  myDynamoDbTable:
    type: dynamo-db-table
    properties:
      primaryKey:
        partitionKey:
          name: id
          type: string
      streamType: NEW_AND_OLD_IMAGES

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: dynamo-db-stream
          properties:
            streamArn: $ResourceParam('myDynamoDbTable', 'streamArn')
            batchSize: 200

S3 event

Triggers the job when a specific event (like object created) occurs in an S3 bucket.

resources:
  myBucket:
    type: bucket

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: s3
          properties:
            bucketArn: $ResourceParam('myBucket', 'arn')
            s3EventType: 's3:ObjectCreated:*'
            filterRule:
              prefix: order-
              suffix: .jpg

Cloudwatch Log event

Triggers the job when a log record is added to a specified CloudWatch log group. The event payload is BASE64 encoded and GZIP compressed.

resources:
  myLogProducingLambda:
    type: function
    properties:
      packaging:
        type: stacktape-lambda-buildpack
        properties:
          entryfilePath: lambdas/log-producer.ts

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: cloudwatch-log
          properties:
            logGroupArn: $ResourceParam('myLogProducingLambda', 'arn')

Application Load Balancer event

Triggers the job when an Application Load Balancer receives an HTTP request matching specified conditions (e.g., path, headers, method).

resources:
  # load balancer which routes traffic to the function
  myLoadBalancer:
    type: application-load-balancer

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: application-load-balancer
          properties:
            # referencing load balancer defined above
            loadBalancerName: myLoadBalancer
            priority: 1
            paths:
              - /invoke-my-job
              - /another-path

Accessing other resources

By default, AWS resources cannot communicate with each other. Access must be granted explicitly using IAM permissions. Stacktape handles most of this automatically, but for resource-to-resource communication, you need to configure permissions.

Relational Databases are an exception, as they use their own connection-string-based access control.

There are two ways to grant permissions:

Using connectTo

The connectTo property is a simplified way to grant basic access to other Stacktape-managed resources. It automatically configures the necessary IAM permissions and injects environment variables with connection details into your batch job.

resources:
  photosBucket:
    type: bucket

  myBatchJob:
    type: batch-job
    properties:
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      connectTo:
        # access to the bucket
        - photosBucket
        # access to AWS SES
        - aws:ses

Using iamRoleStatements

For fine-grained control, you can provide raw IAM role statements. This allows you to define custom permissions to any AWS resource.

resources:
  myBatchJob:
    type: batch-job
    properties:
      resources:
        cpu: 2
        memory: 1800
      container:
        packaging:
          type: stacktape-image-buildpack
          properties:
            entryfilePath: path/to/my/batch-job.ts
      iamRoleStatements:
        - Resource:
            - $CfResourceParam('NotificationTopic', 'Arn')
          Effect: Allow
          Action:
            - 'sns:Publish'

cloudformationResources:
  NotificationTopic:
    Type: AWS::SNS::Topic

Default VPC connection

Certain resources, like Relational Databases, must be placed within a VPC. If your stack contains such resources, Stacktape automatically creates a default VPC and connects them to it.

Batch jobs are connected to this VPC by default, allowing them to communicate with other VPC-enabled resources without extra configuration. To learn more, see our guide on VPCs.

Referenceable parameters

Pricing

You are charged for:

• The compute instances running your batch jobs.
• A negligible amount for the Lambda functions and Step Functions that manage the job's execution.

Pricing depends on the instance type and region. You can significantly reduce costs (by up to 90%) by using spot instances.

API reference