Debugging Lambda Functions

Stacktape dev mode runs Lambda functions on real AWS infrastructure while you iterate on code locally. Stacktape deploys a dev stack to run selected functions on AWS. The dev TUI shows logs and supports on-demand rebuilds — giving you a fast edit-test loop without running a full stack deployment.

How it works

Lambda functions are dev-compatible workloads that deploy to the AWS-backed dev stack and execute on AWS Lambda. Unlike container workloads that run locally in Docker, Lambda functions run in their real cloud environment. This surfaces issues that local emulation would miss — you test against AWS Lambda rather than a local approximation.

Stacktape deploys a dev stack when one does not already exist for the target stage. Your selection in the resource picker controls which workloads dev mode starts and renders in the TUI. Unchanged starts reuse the existing dev stack. Restarting after cloud configuration or cloud-image changes, or with missing required outputs, can update it. The dev TUI then launches with log output, keyboard controls for rebuilding functions, and status for any local databases you selected.

  1. Load config — Stacktape reads your configuration and discovers dev-compatible resources (functions, containers, databases)
  2. Select resources — Choose which resources to run via the interactive picker or --resources / --skip-resources flags
  3. Deploy dev stack — A dev stack is deployed automatically on first run
  4. Start local databases — Selected databases run as Docker containers on your machine
  5. Connect the dev box — when deployed functions use local databases, dev mode starts the dev box if it is stopped and opens one tunnel per database
  6. Run workloads and stream logs — Functions are deployed, logs appear in the dev TUI, and you can rebuild on demand

Starting dev mode

Run stacktape dev to start dev mode. Lambda functions appear as selectable resources in the interactive picker, or you can specify them directly with CLI flags.

Run all dev-compatible resources in your config:

stacktape dev --stage dev --resources all

Run only specific Lambda functions by name (comma-separated):

stacktape dev --stage dev --resources myApi,myWorker

Run everything except specific resources:

stacktape dev --stage dev --skip-resources myWorker

Without --resources or --skip-resources, Stacktape displays an interactive multi-select picker listing all dev-compatible resources. All resources are pre-selected by default — deselect any you don't need and confirm to start.

If you specify a resource name that doesn't exist in your config, Stacktape suggests similar names using fuzzy matching. The --resources and --skip-resources flags accept comma-separated values.

Here is a minimal config with a Lambda function. This function is dev-compatible and will appear in the resource picker when you run stacktape dev:

export default defineConfig(() => {
  const api = new LambdaFunction({
    packaging: new JsBundleLambdaPackaging({
      entryfilePath: './src/handler.ts'
    }),
    memory: 512,
    timeout: 30
  });

  return {
    resources: { api }
  };
});

The memory property sets the function's memory allocation in MB — AWS Lambda allocates proportional CPU. The timeout property sets the maximum execution time in seconds. Both are tunable; see Lambda functions for supported ranges and defaults.

First-time setup

The first run deploys a dev stack. Dev mode never updates or deletes a stack it did not create, regardless of its name or status. Choose another stage, or explicitly delete an unwanted stack. A dev stack in ROLLBACK_COMPLETE cannot be updated: delete it explicitly before starting again. A dev stack in UPDATE_ROLLBACK_COMPLETE can restart normally. A stack still rolling back or in UPDATE_ROLLBACK_FAILED must finish or recover before an update can proceed.

Use a dedicated stage name for dev mode — dev, dev-local, or a personal stage like dev-jane.

Dev mode and full deployments share the same stage namespace. Running stacktape dev --stage production against an existing production deployment will be rejected. Always use a separate stage name for dev mode.

If your config includes beforeDev hooks with a scriptName, dev mode registers those hook names in the dev TUI during startup.

Rebuilds

After changing your function code, trigger a rebuild from the dev TUI. Stacktape runs the rebuild handler for the workload without exiting the dev session.

Rebuild all running workloads by typing rs and pressing Enter:

rs

Rebuild a specific function by name:

rs myApi

If the workload name doesn't match any running resource, the dev TUI prints a warning. This helps catch typos — use the exact resource name from your config.

The dev TUI supports these keyboard commands:

CommandAction
rsRebuild all workloads
rs <name>Rebuild a specific workload by name
c or clearClear the log output
q or quitStop dev mode and exit
When iterating on a single function, use rs <functionName> instead of rs to skip rebuilding other workloads. This keeps the feedback loop tight when you're focused on one function.

Viewing logs

Stacktape provides two ways to view Lambda function logs: real-time streaming in the dev TUI during active development, and historical log fetching with stacktape logs for deployed stages.

Dev TUI log streaming

The Stacktape dev TUI shows logs for running workloads during the dev session, giving you immediate feedback on each invocation. Logs appear within the TUI alongside status information for all running workloads and local resources. This is the primary debugging interface during active development — you see function output, errors, and stack traces without leaving your terminal.

Structure your Lambda function output as JSON objects. JSON logs are easier to search and filter — both in the dev TUI during development and later with stacktape logs and CloudWatch Logs Insights when investigating production issues.

Historical logs with logs

Outside of dev mode, use stacktape logs to fetch logs for a Stacktape resource in a deployed stack. For Lambda debugging, pass the Lambda function resource name with --resource-name.

View logs for a specific function:

stacktape logs myApi --stage production

By default, stacktape logs uses a one-hour start time and returns up to 100 events.

View the last 30 minutes of logs:

stacktape logs myApi --stage production --since 30m

Filter logs containing a specific pattern:

stacktape logs myApi --stage production --filter "ERROR"

Run a CloudWatch Logs Insights query for structured searching:

stacktape logs myApi --stage production --query "fields @timestamp, @message | filter @message like /error/ | sort @timestamp desc | limit 20"

The --since flag accepts relative values (1h, 30m, 1d, 5s) and absolute ISO timestamps. Use --limit to control how many log events are returned. --until sets the upper time boundary for --query Logs Insights queries. --raw switches the standard log-fetching output to machine-readable JSON; the --query path prints formatted results unless agent mode is active.

CloudWatch Logs Insights queries (via --query) are billed by the amount of data scanned. For high-volume functions, use narrow time ranges and specific filters to control cost.

Connecting to local databases

Deployed dev functions reach the local databases on your machine through the dev box. A function that uses a local database, through connectTo or $ResourceParam, gets the database's usual STP_* variables pointing at the box, and the box forwards each connection to your machine.

dev function (dev network, private subnet) ──► dev box ──► Systems Manager session ──► your machine ──► local database

Nothing is published on the internet. Your machine opens the connection to the box, through an AWS Systems Manager session authenticated by your AWS credentials and encrypted. On the box, the relay port of each database admits only dev functions.

The AWS account is the trust boundary. Every dev function in the account's dev network can reach every developer's relay ports, not only the functions of that developer's stage. Local DynamoDB and OpenSearch have no password, and local Redis has one only when its defaultUserPassword is set; Postgres and MySQL always have one. The passwords of local databases, including the ones Stacktape generates, are written into the dev functions' environment variables, so they appear in the dev stack's Lambda configuration and CloudFormation template. They are valid only for the databases on your machine. Use a separate AWS account for people who must not reach each other's local databases.

A database's relay port stays open for 2 minutes after its session ends, so a session that restarts finds it open. A function that connects while no session holds the port, for example a schedule that fires during a restart, waits up to 30 seconds for the next session of the stage and then reaches its database. When no session comes back in time, the box closes the connection, and the database client reports a closed connection. A connection is refused when no session has opened the port since the box started, or when the last session ended more than 2 minutes before.

A query takes your machine's round trip to the region plus a few milliseconds; about 45 ms from Central Europe to eu-west-1. Bulk transfers are limited to about 0.7 MB/s per developer by Systems Manager, so a query that returns 5 MB takes about 7 s. Seeding or exporting large amounts of data is faster from your machine directly.

The dev network

The first stacktape dev that needs it creates the dev network of the AWS account and region, once; every later session and stage reuses it. Creating it takes about 4 minutes, most of it Lambda creating the network interfaces of dev functions. The dev stack deploys meanwhile, so a first start that creates both takes about 5 minutes. The network has:

  • a VPC with private subnets, where dev functions that use a local database run, and every dev function with joinDefaultVpc: true. All of them share one security group, so new dev stages create their functions in seconds;
  • the dev box, a t4g.nano EC2 instance. It runs the relay for local databases and gives VPC functions internet access (it is their NAT).

A dev stage no longer creates a VPC or NAT of its own. VPC functions work in dev mode and reach the internet through the box. The exception is a dev stage that still deploys VPC resources itself, such as a database marked dev.remote: its VPC functions stay in that stage's VPC and cannot reach local databases.

The dev box costs about $0.01 per hour while a dev session runs. It stops itself 30 minutes after the last session ends, and the next stacktape dev starts it again (the TUI shows "Starting the dev box…", which takes about 30 seconds and overlaps the rest of startup). A stopped box costs about $0.75 per month for its disk and the SSM parameter that holds its relay. While it is stopped, dev functions in the network have no internet access: schedules and queue consumers of a dev stage that fire while nobody develops fail until the next session.

See the network with stacktape dev network and delete it with stacktape dev network delete.

Several developers

Each developer runs dev mode against their own stage, and every stage gets its own relay ports. A stage's deployed functions can reach only one session's databases, so one dev session at a time holds a stage. This also applies to one developer running the same stage in two worktrees or on two machines.

A second stacktape dev for a stage that another session holds is refused before it deploys anything. The error names the holder: the user, the host and the project directory. Run it again with --take-over to move all of the stage's ports to the new session at once. The first session's open database connections, such as a warm function's connection pool, are closed. The first session stops itself with a message that names the new one, so it cannot redeploy functions over the new session's. The deployed functions then reach only the new session's databases.

AWS permissions

The AWS identity that runs stacktape dev needs, besides the usual deployment permissions: cloudformation:DescribeStacks on the dev network stack, ec2:DescribeInstances, ec2:StartInstances on the box, ssm:DescribeInstanceInformation, ssm:StartSession on the dev box and on the AWS-StartPortForwardingSession document, ssm:TerminateSession, and ssm:GetParameters, ssm:GetParametersByPath, ssm:PutParameter, ssm:DeleteParameter and ssm:DeleteParameters on /stacktape/dev-network/*, where the relay port leases, their port reservations and the record of which session runs each stage are kept. Without the read, a start elsewhere is refused only after the dev network step; without the delete, starts in another directory or on another machine are refused for up to 3 minutes after a session stops.

When a newer Stacktape version brings a new relay, the first session that finds the box running an older one restarts the relay with SSM Run Command, without stopping the box. Other sessions on the box lose their connections for a few seconds and reconnect on their own. This needs ssm:SendCommand on the dev box and on the AWS-RunShellScript document, and ssm:GetCommandInvocation. Without them, dev mode warns and runs with the older relay, and the next session tries again.

When dev mode generates a missing database password for a $Secret, it tags the secret with each dev stage that uses it. Creating it needs secretsmanager:CreateSecret and secretsmanager:TagResource. stacktape delete of a dev stage never deletes the secret: it removes the stage's tag (secretsmanager:ListSecrets, secretsmanager:UntagResource) and prints Kept generated secret <name>. Delete it with stacktape secret delete when no stage needs it. A deployment that is not a dev session removes the dev tags, and the secret is then the deployment's. Stacktape 4.0.0-preview.11 and earlier scheduled a deleted stage's generated secrets for deletion; a deployment or dev session that uses a secret scheduled this way restores it (secretsmanager:RestoreSecret).

Container workloads that run on your machine use the AWS permissions of their role in the dev stack: dev mode assumes the role with your credentials, so they need sts:AssumeRole on it. When your IAM permissions don't allow that, the container's start fails with AWS_ASSUME_ROLE_NOT_PERMITTED and the policy statement to add, instead of waiting for AWS. The container roles of the dev stack trust your AWS account only for your own IAM user, or the IAM role of your session (an aws:PrincipalArn condition), and no other principal. Credentials of a federated user cannot be named this way; for them, dev mode adds the trust after the deploy, and a container's first start on a new stage can then wait several minutes until AWS accepts it.

The first session in an account and region creates the dev network with CloudFormation, using the credentials of whoever runs it. That identity also needs ssm:GetParameter on the public Amazon Linux parameter /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64, cloudformation:ListStackResources on the dev network stack, and the rights to create the network's VPC, subnets, route tables, internet gateway, security groups, launch template, EC2 instance, IAM role and instance profile (with iam:PassRole), and the keeper Lambda function. The box starts before CloudFormation has finished its instance profile, which takes about 2 minutes, and the session attaches the profile once it exists. That needs ec2:AssociateIamInstanceProfile on the box and iam:PassRole on its role (stacktape-dev-network-…-BoxRole-…). When the first session is interrupted before it attaches the profile, the next session in the account and region attaches it, so every developer whose session may come next needs these two rights too. stacktape dev network and stacktape dev network delete also need lambda:ListFunctions, and the deletion needs the rights to delete those resources. While the box runs, the deletion asks it for live sessions over Systems Manager, with the same ssm:DescribeInstanceInformation, ssm:StartSession and ssm:TerminateSession rights as a session.

Skipping the tunnels

--no-tunnel keeps deployed functions away from your local databases for one session; they get no database variables. VPC functions still run in the dev network. To connect functions to deployed databases instead, use --remote-resources.

Using deployed databases

Dev mode uses an isolated dev stack and emulates supported databases locally. When a function needs a real deployed database or Redis resource, pass its name through --remote-resources:

stacktape dev --stage dev --remote-resources myDatabase

This keeps the dev stack isolated while wiring the selected resource to its deployed AWS counterpart.

Environment cleanup

Stacktape registers cleanup hooks when dev mode starts. When you press Ctrl+C or the process receives SIGTERM, Stacktape runs the registered cleanup hooks. Dev mode registers hooks for:

  • Credentials — temporary credentials used during the dev session
  • Local resources — Docker containers for local databases
  • Health monitor — the local resource health monitor
  • Dev server — the dev server process
  • Tunnels — the tunnels to the dev box and its Systems Manager session
  • Named proxy — the named proxy process

The dev TUI prints a cleanup summary showing which workloads, local resources, and tunnels are being stopped.

Dev mode does not change deployed functions at session start or exit: they are deployed with the dev box address, which stays valid. If dev mode exits unexpectedly (process kill, power loss), the next start of the same stage ends the Systems Manager session the old one left open.

Agent mode

For AI-assisted debugging workflows, dev mode supports an --agent flag that starts a background daemon with an HTTP API. Agent mode registers rebuild handlers and keeps workload and local-resource state updated for the agent server. See agent mode in dev for the supported API surface.

stacktape dev --stage dev --agent

The agent daemon starts on port 7331 when available. If you do not pass --agent-port, Stacktape finds an available port starting at 7331. Use --agent-port to request a specific agent port. Stop a running agent with stacktape dev stop:

stacktape dev stop --agent-port 7331

When agent mode is enabled, dev mode updates agent state with workload and local-resource details and registers rebuild handlers. For endpoint-level details, API documentation, integration with coding assistants, and common workflows, see agent mode in dev.

Debugging strategies

Lambda functions in dev mode run on AWS infrastructure rather than locally, so the documented debugging path is log streaming in the dev TUI combined with targeted rebuilds. The strategies below help you get the most out of this workflow.

Structured logging

Lambda functions run on AWS during dev mode, so the primary debugging approach is structured logging rather than local debugger attachment. The tradeoff is high-fidelity testing against the real Lambda runtime. Structured logging is the most effective debugging approach — add JSON log statements at key points in your function. The dev TUI shows them during the session, and you can query them later with stacktape logs.

export default async (event: any) => {
  console.info(JSON.stringify({
    action: 'request_received',
    path: event.path,
    method: event.httpMethod
  }));

  try {
    const result = await processRequest(event);
    console.info(JSON.stringify({
      action: 'request_processed',
      statusCode: result.statusCode
    }));
    return result;
  } catch (error) {
    console.error(JSON.stringify({
      action: 'request_failed',
      error: error.message,
      stack: error.stack
    }));
    throw error;
  }
};

Use descriptive action fields so you can later filter with --filter "request_failed" or query with --query "fields @timestamp, @message | filter @message like /request_failed/".

Testing HTTP-triggered functions

For Lambda functions with HTTP API Gateway triggers, invoke the function's endpoint directly with curl, a browser, or any HTTP client while watching logs stream in the dev TUI. For HTTP-triggered Lambda functions, use the URL shown by dev mode when one is available, or inspect the deployed stack outputs for the endpoint.

Narrowing the feedback loop

When debugging a specific function, use --resources <name> to select it for session rebuilds, watching and logs. Unselected functions are not built: they keep the code they run until you select them again. Combine with rs <name> in the TUI for targeted rebuilds. Selected Lambda functions are deployed by Stacktape during dev mode, so exercise their configured triggers as you normally would — invoke HTTP endpoints, send messages into queues like SQS, or wait for schedules — and watch the function process events live in the dev TUI.

Dev mode vs deployed-stack debugging

Stacktape offers two distinct workflows for investigating Lambda function behavior: interactive dev mode for active development, and stacktape logs for historical log analysis on deployed stages. Choose based on whether you are iterating on code or investigating an issue in an existing deployment.

AspectDev mode (stacktape dev)Deployed stack (logs)
Log deliveryStreamed in the dev TUIFollows CloudWatch logs in a terminal by default; --follow=false takes a snapshot
Code iterationRebuild from TUI with rsRequires full stacktape deploy
Database accessLocal databases through the dev box, or deployed ones via --remote-resourcesAWS-deployed databases
Interaction modelInteractive TUI with keyboard commandsFollows until stopped, or exits after a snapshot
InfrastructureDev stackFull production stack
Local resourcesDocker-based databases on your machineNot applicable
Best forActive development and rapid iterationInvestigating issues in deployed stages

Use dev mode for active development — the rebuild-and-test loop from the TUI is significantly faster than deploying each change. Use stacktape logs for investigating issues in deployed production or staging stacks where you need historical logs without disrupting the running environment. You can also view logs in the Stacktape Console.

Debugging Lambda Functions FAQ

Can I set breakpoints in Lambda functions during dev mode?

Lambda functions in dev mode run on AWS Lambda infrastructure, not on your local machine. The debugging path documented here is log streaming in the dev TUI combined with targeted rebuilds using rs <name>. The tradeoff is high-fidelity testing against the real Lambda runtime on AWS. Add targeted console.info(JSON.stringify(...)) statements at key points and rebuild quickly after changes.

Do my Lambda function's event sources work in dev mode?

Lambda functions are included as dev-compatible workloads, and dev mode deploys a dev stack that includes selected functions. To verify a trigger during dev mode, invoke the function's configured entry point — call the HTTP endpoint directly, send a message to a connected queue, or wait for a scheduled trigger — and watch the dev TUI logs for output. See triggers overview for the available event source types.

Can I debug only some functions while keeping others untouched?

Use --resources myApi,myWorker to select functions, or --skip-resources myWorker to exclude one. Selection limits which functions the running session rebuilds and watches. Updating the dev stack does not build unselected functions, so a broken function you skip does not fail the start: it keeps running the code it ran before, and the session's status lists it with when that code was deployed. Changes to its source are deployed only once you select it again. A function is built anyway when the dev stack has no code for it yet (a new dev stack or function) or when its configuration changed, such as its environment, memory or permissions: its old code never ran with the new configuration. The start output says why. Unselected functions stay in the dev stack with their configured triggers removed. Use a separate stage when another session's triggers need to keep running. Batch jobs cannot be selected or skipped; use a dev-specific configuration to omit them or disable their triggers, and restart dev mode after changing Batch code.

Why does dev mode refuse to start on my stage?

Dev mode refuses any stack it did not create, regardless of name or status. Use a dedicated stage such as dev-jane. A dev stack in ROLLBACK_COMPLETE requires explicit deletion before restarting; see first-time setup.

How do my local databases connect to functions running on AWS?

Through the dev box of your account's dev network: functions connect to the box's private address, and the box forwards each connection to your machine over an encrypted Systems Manager session. See Connecting to local databases.

How much does running dev mode cost?

Dev mode deploys a dev stack on real AWS infrastructure, so standard AWS charges apply for the resources it runs (Lambda invocations, API Gateway endpoints, and other deployed resources). The dev box costs about $0.01 per hour while a dev session runs and about $0.75 per month while it is stopped. Selected local databases run on your machine as Docker containers, so no AWS database service charges apply for those during the dev session. Note that stacktape logs CloudWatch Logs Insights queries (via --query) are billed by the amount of data scanned — use narrow time ranges and specific filters on high-volume functions.

How do I view logs from a Lambda function that's already deployed?

Use stacktape logs with --resource-name and --stage to fetch CloudWatch logs from any deployed stage. In a terminal, it shows the last hour of logs (up to 100 events), then follows new lines. Use --follow=false for a snapshot; piped, redirected and agent runs take snapshots unless --follow is requested. Add --filter "ERROR" to narrow results, or use --query for CloudWatch Logs Insights queries that can search structured JSON fields across log streams. You can also view logs in the Stacktape Console.