Batch Job

A Stacktape batch job runs a containerized task to completion for work such as data processing, ML training, video encoding, and imports. Use a batch job when the work has a clear finish point, needs explicit CPU, memory, or GPU requirements, and should be triggered by schedules, HTTP requests, storage events, queues, streams, or event buses.

Pricing context: Batch jobs run on EC2 instances that AWS Batch starts for queued jobs and terminates when the queue is empty, so you pay for the instances while jobs run. Spot capacity can save up to 90% when interruptions are acceptable.

When to use

A batch job is the right choice when the workload should start, do a defined unit of work, and exit. Common examples include nightly exports, object-processing pipelines, one-off video transcodes, queue-driven processing bursts, and ML jobs that can run in a container with declared CPU, memory, and optional GPU requirements.

  • Long-running jobs — use a batch job for work that does not fit naturally into a short request/response function.
  • Compute-heavy tasks — specify cpu, memory, and optionally gpu; Stacktape lets AWS select matching compute from the declared requirements.
  • Event-triggered processing — trigger jobs from schedules, HTTP API Gateway, Application Load Balancer, S3, SNS, SQS, Kinesis, DynamoDB streams, CloudWatch Logs, or EventBridge event buses.
  • Interruptible work — enable Spot capacity for jobs that can restart safely, especially with retryConfig.

When NOT to use

  • Always-on HTTP services — use a web-service for containers that must serve public HTTP traffic continuously.
  • Internal always-on services — use a private-service for long-lived containers reachable only within your stack or VPC.
  • Continuous background workers — use a worker-service for containers that run continuously rather than starting per job.
  • Short event handlers — use a Lambda function for short-lived, event-driven tasks where per-invocation billing is preferable.
  • Non-idempotent side effects on Spot — keep useSpotInstances disabled for jobs that charge payments, send emails, or perform work that cannot safely be repeated.

Basic example

This example runs a scheduled TypeScript container every night. js-bundle packaging builds the container from ./src/jobs/nightly-report.ts; resources requests 2 vCPU and 4096 MB memory; timeout stops the job if it runs longer than one hour.

export default defineConfig(() => {
  const nightlyReport = new BatchJob({
    container: {
      packaging: new JsBundleImagePackaging({
        entryfilePath: './src/jobs/nightly-report.ts'
      }),
      environment: { REPORT_BUCKET_PREFIX: 'reports/nightly' }
    },
    resources: {
      cpu: 2,
      memory: 4096
    },
    timeout: 3600,
    events: [
      {
        type: 'schedule',
        properties: {
          scheduleRate: 'cron(0 2 * * ? *)'
        }
      }
    ]
  });

  return { resources: { nightlyReport } };
});

The schedule expression uses AWS cron syntax and runs in UTC. Use rate(...) for simple intervals such as rate(2 hours), and use cron(...) when the job must run at a specific time or day. See schedule triggers for trigger details.

Compute resources

A batch job declares the CPU and memory required by one job run. Stacktape uses those requirements to let AWS provision matching compute. Add gpu only for workloads that require GPU acceleration, such as model training or GPU-backed rendering; omit gpu for normal CPU-only processing.

resources: {
  cpu: 8,
  memory: 30720,
  gpu: 1
}

cpu is the number of vCPUs, a whole number, because AWS Batch assigns whole vCPUs on EC2 instances. memory is memory in MB. The source recommends using slightly less than exact powers of two for memory-sensitive sizing because AWS reserves memory for system processes; for example, 7680 MB may fit better than exactly 8192 MB on an 8 GB instance class.

Jobs run on x86 instances with the Amazon ECS-optimized Amazon Linux 2023 image. Jobs with gpu use its NVIDIA variant on p4d and g5 instances. The instances require IMDSv2 and use cgroup v2. Job credentials come from the job role, not instance metadata, but software inside the container that reads instance metadata with IMDSv1, or that reads memory limits only from cgroup v1 (such as Java before 8u372), needs a current version.

Packaging

A batch job supports four container image packaging types. Start with js-bundle for JavaScript or TypeScript and buildpack for other languages, then switch only when the image needs precise runtime control.

TypeWhen to use
js-bundleA JavaScript or TypeScript job with one entry file. Stacktape bundles it into a small image.
buildpackBuild from source without a Dockerfile. Detects Python, Go, Rust, Java, PHP, Ruby, .NET, Node.js and more.
dockerfileControl the Dockerfile, build context, build args and startup command
prebuilt-imageRun an image that already exists in a registry and optionally override its command

Use container.environment for runtime settings that belong to this job, such as feature flags or fixed paths. Use $Secret() or $ResourceParam() when a value should come from Stacktape directives instead of being hard-coded; see directives for the supported directive model.

Dev mode

Batch jobs run in AWS Batch when triggered; they are not local processes. --resources and --skip-resources cannot select or skip them. Their images are built and pushed during dev startup, so restart dev mode after changing Batch code. Use a dev-specific configuration to omit a job or disable its triggers. A deployed job can reach local databases through the dev network, with its AWS costs, permissions and network round trip.

Triggers

A batch job can start from several event sources: HTTP API Gateway, Application Load Balancer, S3, SNS, SQS, Kinesis, DynamoDB streams, CloudWatch Logs, schedules, and EventBridge event buses. Pick the trigger based on where the unit of work originates, and keep the job handler idempotent when the source can retry delivery.

Trigger typeUse it for
scheduleRecurring jobs such as reports, cleanup, imports, and periodic syncs
s3Object-created, object-removed, restore, or replication events from a bucket
sqsQueue-backed jobs where each message or batch represents work to process
snsPub/sub fanout where a topic delivery should start the job
kinesis-streamStream records processed in batches
dynamo-db-streamTable change records after DynamoDB streams are enabled
http-api-gatewayHTTP request starts a job through an HTTP API Gateway
application-load-balancerHTTP request starts a job through an Application Load Balancer rule
cloudwatch-logMatching CloudWatch log records start the job
event-busEventBridge events matching a pattern start the job

The API reference carries the full property shape for each integration. For a broader decision guide across trigger families, see triggers overview.

Reading trigger data

The container receives the serialized trigger event in STP_TRIGGER_EVENT_DATA. Parse it in your process entrypoint. STP_CURRENT_ATTEMPT and STP_MAXIMUM_ATTEMPTS are string environment variables describing this execution's attempt; this is a container process, not a Lambda handler(event) invocation.

const event = JSON.parse(process.env.STP_TRIGGER_EVENT_DATA ?? '{}');
const attempt = Number(process.env.STP_CURRENT_ATTEMPT ?? '1');
const maximumAttempts = Number(process.env.STP_MAXIMUM_ATTEMPTS ?? '1');
await processJob(event, { attempt, maximumAttempts });

Retries and timeout

A batch job can stop runaway work with timeout and retry failed attempts with retryConfig. Use a timeout whenever the job processes external input, calls remote APIs, or might hang on corrupted data. Use retries when failures are transient or the job can safely restart from the beginning.

timeout: 7200,
retryConfig: {
  attempts: 3,
  retryIntervalSeconds: 30,
  retryIntervalMultiplier: 2
}

timeout is measured in seconds; when a job exceeds it, the job is killed and retried if retryConfig is set. attempts defaults to 1, retryIntervalSeconds defaults to 0, and retryIntervalMultiplier defaults to 1. attempts includes the initial attempt. With the example's three attempts, interval 30 and multiplier 2, the current implementation waits 60 seconds before attempt 2 and 120 seconds before attempt 3. Waits follow interval × failed-attempt number × multiplier; there is no wait after the final attempt.

Retries repeat the whole job attempt. Make job code idempotent before enabling multiple attempts for writes, imports, notifications, or any operation that can create duplicated side effects.

Spot capacity

useSpotInstances lets a batch job use discounted spare AWS capacity. The source documents savings up to 90%, but Spot jobs can be interrupted. Use Spot for restartable data imports, image processing, or ML training with checkpoints; keep on-demand capacity for strict deadlines or non-repeatable side effects.

useSpotInstances: true,
retryConfig: {
  attempts: 5,
  retryIntervalSeconds: 60
}

When AWS interrupts a Spot-backed batch job, the container receives SIGTERM and has 120 seconds to shut down gracefully. Store progress externally if partial work matters, and combine Spot with retries so interrupted jobs are resubmitted instead of ending as failed work.

Connecting to resources

Use connectTo when a batch job needs access to resources in the same stack. Stacktape can grant IAM permissions, open network access for supported databases and Redis, and inject connection environment variables into the job container using the STP_[RESOURCE_NAME]_[PARAM] naming pattern.

connectTo: [mainDatabase, reports],

With the resource names above, the job receives database variables such as STP_MAIN_DATABASE_CONNECTION_STRING, STP_MAIN_DATABASE_HOST, and STP_MAIN_DATABASE_PORT, plus bucket variables such as STP_REPORTS_NAME and STP_REPORTS_ARN. For permissions outside connectTo, use iamRoleStatements.

Logging

A batch job can send container stdout and stderr to CloudWatch Logs. Logging is enabled unless logging.disabled is set, and the documented default retention period is 90 days. Use logs for progress markers, failure causes, and job input identifiers that help you trace one run after a retry.

logging: {
  retentionDays: 30
}

retentionDays accepts CloudWatch retention values exposed in the API reference, including 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, and 3653. View runtime logs with stacktape logs.

Batch Job FAQ

When should I use a batch-job vs a Lambda function or worker-service?

Use a Lambda function for short-lived, event-driven tasks billed per invocation. Use a worker-service when the container should stay running continuously. Use a batch job when the work is long-running, container-native, compute-heavy, possibly GPU-backed, and has a clear end state where the process starts, does a unit of work, and exits.

Should I use Spot instances for batch jobs?

Use useSpotInstances: true when the job can restart from the beginning or resume from checkpoints. Spot capacity can reduce cost up to 90%, but AWS can interrupt the job: the container receives SIGTERM and has 120 seconds to shut down gracefully. Combine Spot with retryConfig so interrupted jobs are resubmitted, and avoid Spot for payment, email, or other non-idempotent, deadline-sensitive work.

Why do my batch jobs run more than once or create duplicate side effects?

Retries repeat the entire job attempt, and several trigger sources (SQS, SNS, Kinesis, streams) can also redeliver. attempts defaults to 1, retryIntervalSeconds to 0, and retryIntervalMultiplier to 1, so duplicates only appear once you raise the attempt count or use a retrying trigger. Make job code idempotent before enabling multiple attempts for writes, imports, or notifications.

How should I size cpu and memory for a batch job?

cpu is a whole number of vCPUs and memory is in MB. Because AWS reserves some memory for system processes, request slightly less than an exact power of two for memory-sensitive sizing: for example 7680 MB often fits better than exactly 8192 MB on an 8 GB instance class. Add resources.gpu only for workloads that need GPU acceleration such as model training or rendering.

How much does a batch job cost?

These jobs use managed EC2 or EC2 Spot capacity. Estimate cost from the selected EC2 instances and how long capacity stays running, including startup, drain and idle time, plus storage, networking and supporting services. Fargate per-vCPU/per-GB prices do not apply to this implementation. See optimization tips for broader guidance.

API Reference

BatchJobProps
Click a property in the tree to see its description, default value, and full type.

Referenceable parameters

The following parameters can be easily referenced using $ResourceParam directive directive.

To learn more about referencing parameters, refer to referencing parameters.

jobDefinitionArn
  • Arn of the job definition resource

  • Usage: $ResourceParam('<<resource-name>>', 'jobDefinitionArn')
stateMachineArn
  • Arn of the state machine controlling the execution flow of the batch job

  • Usage: $ResourceParam('<<resource-name>>', 'stateMachineArn')
logGroupArn
  • Arn of the log group aggregating logs from the batch job

  • Usage: $ResourceParam('<<resource-name>>', 'logGroupArn')