logoStacktape docs


BucketsWork in progress
Work in progress

Overview and basic concepts

  • Buckets allow you to persistently store files.
  • A Bucket is a collection of "objects". An object is a file and any metadata (headers, tags,...) that describes that file.
  • Buckets are easy to use, reliable, durable and highly available. They are powered by AWS S3 object storage service.
  • Buckets are serverless. They scale automatically to meet your needs and you are paying only for the files saved (not unused capacity).
  • Buckets have a flat structure instead of a hierarchy like you would find in a file system. However, you can simulate a "folder hierarchy" by using a common prefix. For example, all objects stored with a name that starts with photos/ prefix will be shown in the same "folder" in the AWS console.

When to use

  • Buckets can be used to host websites, store user-generated content, store data for Big data analytics, serve as data lakes, backup and restore, archives, etc.

  • Because of performance reasons, object storage in general is not optimal for use-cases that require very low read/write latency.

Advantages

  • Easy to use - AWS S3 has a simple, HTTP-based API. You can also easily manipulate your objects using an AWS SDK.

  • Serverless - Bucket automatically scales to meet your needs. You don't have to worry about scaling your storage size and are paying only for the files saved.

  • Reliabile & highly available - S3 Buckets are designed for 99.999999999% (11 9’s) of data durability. Files (objects) are stored across multiple physical locations (Availability zones).

  • Storage flexibility - You can store your files in multiple storage classes. Different storage classes have different latencies, pricing and durability.

  • Access control - You can easily control who can access your bucket.

  • Supports encryption - Supports server-side encryption.

  • Integrations - You can easily trigger a function or a batch-job in a reaction to a bucket event.

Disadvantages

  • Performance - Compared to using a block storage (physical disk attached to a machine), reading/write operations are significantly slower.

Basic usage

Copy

resources:
myBucket:
type: bucket
myFunction:
type: function
properties:
packaging:
type: stacktape-lambda-buildpack
properties:
entryfilePath: path/to/my/lambda.ts
environment:
- name: BUCKET_NAME
value: $ResourceParam('myBucket', 'arn')
accessControl:
allowAccessTo:
- myBucket

Lambda function connected to a bucket

Copy

import { S3 } from '@aws-sdk/client-s3';
const s3 = new S3({});
// getObject returns a readable stream, so we need to transform it to string
const streamToString = (stream) => {
const chunks = [];
return new Promise((resolve, reject) => {
stream.on('data', (chunk) => chunks.push(Buffer.from(chunk)));
stream.on('error', (err) => reject(err));
stream.on('end', () => resolve(Buffer.concat(chunks).toString('utf8')));
});
};
const handler = async (event, context) => {
await s3.putObject({
Bucket: process.env.BUCKET_NAME,
Key: 'my-file.json',
Body: JSON.stringify({ message: 'hello' }) // or fs.createReadStream('my-source-file.json')
});
const res = await s3.getObject({
Bucket: process.env.BUCKET_NAME,
Key: 'my-file.json'
});
const body = await streamToString(res.Body);
};
export default handler;

Lambda function that uploads and downloads a file from a bucket

Directory upload

  • Allows you to upload a specified directory to the bucket on every deployment
  • After the upload is finished, your bucket will contain the contents of the local folder.
  • Files are uploaded using parallel, multipart uploads.

Existing contents of the bucket will be deleted and replaced with the contents of the local directory. You should not use directoryUpload for buckets with application-generated or user-generated content.

Copy

resources:
myBucket:
type: bucket
properties:
directoryUpload:
directoryPath: ../public

Uploads the public folder to the bucket on every deployment

directoryPath
Required

Path to the directory that should be uploaded to the bucket

Type: string

  • After the sync is finished, your bucket will contain the contents of the local folder.
  • Path is relative to current working directory.

Existing contents of the bucket will be deleted and replaced with the contents of the local directory. You should not use bucket with autoSync enabled for application-generated or user-generated content.

filters

Filters allow you to set properties of files (objects) that match the filter pattern

Type: Array of DirectoryUploadFilter

skipFiles

Configures which files should be skipped from upload (glob patterns).

Type: Array of string

  • Relative to the directoryPath
headersPreset

Configures HTTP headers of uploaded files to be optimized for selected preset

Type: string ENUM

Possible values: gatsby-static-websitesingle-page-appstatic-website

Available presets:

  • static-website:
    • sets cache-control header value for all uploaded files to public, max-age=0, s-maxage=31536000, must-revalidate.
    • this setup caches all the content on the CDN but never caches files in the browser.
  • gatsby-static-website:
  • single-page-app:
    • optimized for a Single page application
    • html files are never cached (so that when you re-deploy your website, your users always get the latest content)
    • All other assets (.js, .css, etc.) are cached indefinitely. You should ALWAYS add a content hash to the filename, so that your users always get a new version of your website after you deploy it. To learn how to add a content hash to your files, refer to the docs of your bundler, for example webpack docs.
  • You can override these presets using custom filters.

    When headersPreset is used, cdn.invalidateAfterDeploy must also be configured.

disableS3TransferAcceleration

Disables S3 transfer acceleration.

Type: boolean

  • S3 transfer acceleration improves upload times of your directory contents.
  • Objects are uploaded to the nearest AWS edge location, and routed to the bucket from there using AWS backbone network.
  • Used to improve upload times and security.
  • S3 transfer acceleration includes (insignificant) additional costs.

Adding metadata

  • You can add metadata to uploaded files. You can configure headers and tags.
  • Filters allow you to configure properties of files (objects) that match the filter pattern.
  • Configurable properties:
    • headers: when a client makes a request for an object, these headers will be added to the HTTP response. If you are using a CDN, all headers will be forwarded. If you are using the bucket to host a website, this can be for example used to add cache-control headers on a per-file basis (different cache behavior for different file types).
    • tags: can be used to filter specific objects, for example in Lifecycle rules.
DirectoryUploadFilter  API reference
Parent API reference: DirectoryUpload
includePattern
Required

Configures which files should be handled by this filter (glob pattern)

Type: string

  • Relative to the directoryPath
excludePattern

Configures which files should be excluded from this filter, even if they match the include pattern (glob pattern)

Type: string

  • Relative to the directoryPath
headers

Configures HTTP headers for files (objects) that match this filter

Type: Array of KeyValuePair

  • If you are using a CDN, these headers will be forwarded to the client.
  • This can be used to implement HTTP caching for your static content.
tags

Tags to apply to the files that match this filter

Type: Array of KeyValuePair

  • Tags help you to categorize your objects. They can also be used to filter objects when using lifecycleRules
  • To learn more about object tagging, refer to AWS Docs

A single file (object) can have only one tag with the same key.

Encryption

  • If enabled, all objects uploaded to the bucket will be server-side encrypted using the AES256 algorithm.

Copy

resources:
myBucket:
type: bucket
properties:
encryption: true

Cors

  • Web browsers use CORS (Cross-Origin Resource Sharing) to block the website from making requests to a different origin (server) than the one the website is served from. This means that if you make a request from a website served from https://my-website.s3.eu-west-1.amazonaws.com/ to https://my-api.my-domain.com, the request will fail.

  • If you enable CORS and do not specify any cors rules, the default rule with following configuration is used:

    • AllowedMethods: GET, PUT, HEAD, POST, DELETE
    • AllowedOrigins: '*'
    • AllowedHeaders: Authorization, Content-Length, Content-Type, Content-MD5, Date, Expect, Host, x-amz-content-sha256, x-amz-date, x-amz-security-token
  • When the bucket receives a preflight request from a browser, it evaluates the CORS configuration for the bucket and uses the first CORS rule that matches the incoming browser request to enable a cross-origin request. For a rule to match, the following conditions must be met:

    • The request's Origin header must match one of allowedOrigins element.
    • The request method (for example, GET or PUT) or the Access-Control-Request-Method header in case of a preflight OPTIONS request must be one of the allowedMethods.
    • Every header listed in the request's Access-Control-Request-Headers header on the preflight request must match one of headers allowedHeaders.

Copy

resources:
myBucket:
type: bucket
properties:
cors:
enabled: true

BucketCorsConfig  API reference
Parent API reference: Bucket
enabled
Required

Enables CORS (Cross-Origin Resource Sharing) HTTP headers for the bucket

Type: boolean

  • If you enable CORS and do not specify any cors rules, default rule with following configuration is used:
    • AllowedMethods: GET, PUT, HEAD, POST, DELETE
    • AllowedOrigins: '*'
    • AllowedHeaders: Authorization, Content-Length, Content-Type, Content-MD5, Date, Expect, Host, x-amz-content-sha256, x-amz-date, x-amz-security-token
corsRules

List of CORS rules

Type: Array of BucketCorsRule

  • Web browsers use CORS (Cross-Origin Resource Sharing) to block the website from making requests to a different origin (server) than the one the website is served from. This means that if you make a request from a website served from https://my-website.s3.eu-west-1.amazonaws.com/ to https://my-api.my-domain.com, the request will fail.

  • If you enable CORS and do not specify any cors rules, the default rule with following configuration is used:

    • AllowedMethods: GET, PUT, HEAD, POST, DELETE
    • AllowedOrigins: '*'
    • AllowedHeaders: Authorization, Content-Length, Content-Type, Content-MD5, Date, Expect, Host, x-amz-content-sha256, x-amz-date, x-amz-security-token
  • When the bucket receives a preflight request from a browser, it evaluates the CORS configuration for the bucket and uses the first CORS rule that matches the incoming browser request to enable a cross-origin request. For a rule to match, the following conditions must be met:

    • The request's Origin header must match one of allowedOrigins element.
    • The request method (for example, GET or PUT) or the Access-Control-Request-Method header in case of a preflight OPTIONS request must be one of the allowedMethods.
    • Every header listed in the request's Access-Control-Request-Headers header on the preflight request must match one of headers allowedHeaders.
BucketCorsRule  API reference
Parent API reference: BucketCorsConfig
allowedOrigins

Origins to accepts cross-domain requests from

Type: Array of string

  • Origin is a combination of scheme (protocol), hostname (domain), and port of the URL
  • Examples of same origin: http://example.com/app1:80 http://example.com/app2
  • Examples of a different origin: http://example.com/app1 https://example.com/app2
allowedHeaders

Allowed HTTP headers

Type: Array of string

  • Each header name in the Access-Control-Request-Headers header of a preflight request must match a corresponding entry in the rule.
allowedMethods

Allowed HTTP methods

Type: Array of string ENUM

Possible values: *DELETEGETHEADPATCHPOSTPUT

exposedResponseHeaders

Response headers that should be made available to scripts running in the browser, in response to a cross-origin request

Type: Array of string

maxAge

Time in seconds that browser can cache the response for a preflight request

Type: number

Versioning

  • If enabled, bucket keeps multiple variants of an object.
  • This can help you to recover objects from an accidental deletion/overwrite, or to store multiple objects with the same name.

Copy

resources:
myBucket:
type: bucket
properties:
versioning: true

CDN

You can use CDN with bucket.

By putting CDN in front of a bucket, you can distribute content of the bucket to hundreds of edge locations all around the world.


This is useful in cases if you want to for example serve a static website from your bucket.


Stacktape can:

  • automatically upload your content (e.g. static website) to the bucket
  • configure CDN and deliver your content with minimal latency across the world

For information about using CDN refer to our CDN docs.

Copy

resources:
myBucket:
type: bucket
properties:
directoryUpload:
directoryPath: my-web/build
headersPreset: static-website
cdn:
enabled: true

Bucket with CDN and directory upload enabled

Object lifecycle rules

  • Lifecycle rules allow you to configure what happens to your objects after a configured period of time.
  • They can be deleted, transitioned to another storage class, etc.
  • These rules can be applied to only a subset of objects in the bucket using path prefix and object tags.

Storage class transition

  • By default, all objects are in the standard (general purpose) class.
  • Depending on your access patterns, you can transition your objects to a different storage class to save costs.
  • To better understand differences between storage classes, refer to AWS Docs
  • To learn more about storage class transitions, refer to AWS Docs

Copy

resources:
myBucket:
type: bucket
properties:
lifecycleRules:
- type: class-transition
properties:
daysAfterUpload: 90
storageClass: 'GLACIER'

Bucket configured to transfer all objects to GLACIER storage class 90 days after being uploaded

ClassTransition  API reference
Parent API reference: Bucket
type
Required

No description

Type: string "class-transition"

properties.daysAfterUpload
Required

Number of days after the object is transitioned to another storage class

Type: number

  • Relative to the date uploaded
  • Depending on how often you need to access your objects, transitioning them to another storage can lead to a significant price reduction.
properties.storageClass
Required

Storage class to transition to

Type: string ENUM

Possible values: DEEP_ARCHIVEGLACIERINTELLIGENT_TIERINGONEZONE_IASTANDARD_IA

  • By default, all objects are in the standard (general purpose) class.
  • Depending on your access patterns, you can transition your objects to a different storage class to save costs.
  • To better understand differences between storage classes, refer to AWS Docs
  • To learn more about storage class transitions, refer to AWS Docs
properties.prefix

Prefix of the objects to which the lifecycle rule is applied

Type: string

properties.tags

Tags of the objects to which the lifecycle rule is applied

Type: Array of KeyValuePair

  • To learn more about tagging objects, refer to AWS Docs

Expiration

Allows you to delete objects from the bucket after the specified amount of days after upload.

Expiration  API reference
Parent API reference: Bucket
type
Required

No description

Type: string "expiration"

properties.daysAfterUpload
Required

Number of days after the object is considered expired

Type: number

  • Relative to the date uploaded
properties.prefix

Prefix of the objects to which the lifecycle rule is applied

Type: string

properties.tags

Tags of the objects to which the lifecycle rule is applied

Type: Array of KeyValuePair

  • To learn more about tagging objects, refer to AWS Docs

This can be useful in cases if objects become irrelevant for you after some time. These objects can be deleted, thus saving you storage costs.

The following example shows:

  • All uploaded objects are transferred to GLACIER storage class 90 days after upload.
  • After 365 days, objects are completely deleted.

Copy

resources:
myBucket:
type: bucket
properties:
lifecycleRules:
- type: class-transition
properties:
daysAfterUpload: 90
storageClass: 'GLACIER'
- type: expiration
properties:
daysAfterUpload: 365

Non-current version class transition

Allows you to transition versioned objects into a different storage class.

NonCurrentVersionClassTransition  API reference
Parent API reference: Bucket
type
Required

No description

Type: string "non-current-version-class-transition"

properties.daysAfterVersioned
Required

Number of days after the non-current object is transitioned to another storage class

Type: number

  • Relative to date it became non-current (old)
  • Depending on how often you need to access your objects, transitioning them to another storage can lead to a significant price reduction.
properties.storageClass
Required

Storage class to transition to

Type: string ENUM

Possible values: DEEP_ARCHIVEGLACIERINTELLIGENT_TIERINGONEZONE_IASTANDARD_IA

  • To learn more about storage classes and transitions, refer to AWS Docs
properties.prefix

Prefix of the objects to which the lifecycle rule is applied

Type: string

properties.tags

Tags of the objects to which the lifecycle rule is applied

Type: Array of KeyValuePair

  • To learn more about tagging objects, refer to AWS Docs

Same as class transition rule but applied to old versions of objects. This can be useful when you want to archive old versions of an object after some time to save you costs.

The following example shows:

  • all versioned objects are transferred to GLACIER storage class 10 days after they are versioned (become non-current version).

Copy

resources:
myBucket:
type: bucket
properties:
versioning: true
lifecycleRules:
- type: non-current-version-class-transition
properties:
daysAfterVersioned: 10
storageClass: 'DEEP_ARCHIVE'

Non-current version expiration

Allows you to delete old versions of objects from the bucket after the specified amount of days after they are versioned.

NonCurrentVersionExpiration  API reference
Parent API reference: Bucket
type
Required

No description

Type: string "non-current-version-expiration"

properties.daysAfterVersioned
Required

Number of days after the non-current object becomes expired.

Type: number

  • Relative to date it became non-current (old)
  • This rule is effective only if the bucket has versioning enabled.
properties.prefix

Prefix of the objects to which the lifecycle rule is applied

Type: string

properties.tags

Tags of the objects to which the lifecycle rule is applied

Type: Array of KeyValuePair

  • To learn more about tagging objects, refer to AWS Docs

This can be useful if you only need to keep older versions of objects for some time and then they can be deleted thus saving you storage costs.

The following example shows:

  • all versioned objects are deleted ten days after being version (become non-current version).

Copy

resources:
myBucket:
type: bucket
properties:
versioning: true
lifecycleRules:
- type: non-current-version-expiration
properties:
daysAfterVersioned: 10

Abort incomplete multipart upload

Allows you to stop multipart uploads that do not complete within a specified number of days after being initiated.

AbortIncompleteMultipartUpload  API reference
Parent API reference: Bucket
type
Required

No description

Type: string "abort-incomplete-multipart-upload"

properties.daysAfterInitiation
Required

Number of days after the in-complete upload is aborted, and it's parts deleted

Type: number

  • Relative to the start of multipart upload
properties.prefix

Prefix of the objects to which the lifecycle rule is applied

Type: string

properties.tags

Tags of the objects to which the lifecycle rule is applied

Type: Array of KeyValuePair

  • To learn more about tagging objects, refer to AWS Docs

When a multipart upload is not completed within the timeframe, it becomes eligible for an abort operation, and Amazon S3 stops the multipart upload (and deletes the parts associated with the multipart upload, thus saving you costs).

The following example shows:

  • all incomplete uploads are stopped (and parts deleted) 5 days after initiation of multipart upload.

Copy

resources:
myBucket:
type: bucket
properties:
lifecycleRules:
- type: abort-incomplete-multipart-upload
properties:
daysAfterInitiation: 5

Accessibility

  • Configures who can access the bucket.
BucketAccessibility  API reference
Parent API reference: Bucket
accessibilityMode
Default: privateRequired

Configures pre-defined accessibility modes for the bucket

Type: string ENUM

Possible values: privatepublic-readpublic-read-write

  • Allows you to easily configure the most commonly used access patterns.
  • Available modes:
    • public-read-write - Everyone can read from and write to the bucket.
    • public-read - Everyone can read from the bucket. Only workloads and entities with sufficient IAM permissions can write to the bucket.
    • private - (default) Only workloads and entities with sufficient IAM permissions can read from or write to the bucket.
  • For functions, batch jobs and container workloads, you can grant required IAM permissions to read/write from the bucket using allowsAccessTo or iamRoleStatements in their configuration.
accessPolicyStatements

Advanced access configuration that leverages IAM policy statements

Type: Array of BucketPolicyIamRoleStatement

  • Gives fined-grained access control to the bucket

Accessibility modes

  • Allows you to easily configure the most commonly used access patterns.
  • Available modes:
    • public-read-write - Everyone can read from and write to the bucket.
    • public-read - Everyone can read from the bucket. Only workloads and entities with sufficient IAM permissions can write to the bucket.
    • private - (default) Only workloads and entities with sufficient IAM permissions can read from or write to the bucket.
  • For functions, batch jobs and container workloads, you can grant required IAM permissions to read/write from the bucket using allowsAccessTo or iamRoleStatements in their configuration.

Access policy statements

Copy

resources:
myBucket:
type: bucket
properties:
accessibility:
accessibilityMode: private
accessPolicyStatements:
- Resource:
- $ResourceParam('myBucket', 'arn')
Action:
- 's3:ListBucket'
Principal: '*'

BucketPolicyIamRoleStatement  API reference
Parent API reference: BucketAccessibility
Principal
Required

No description

Type: UNSPECIFIED

Resource
Required

List of resources we want to access

Type: Array of string

  • See AWS reference here.
Sid

Statement identifier.

Type: string

  • See AWS reference here.
Effect

Effect of the statement

Type: string

  • See AWS reference here.
Action

List of actions allowed/denied by the statement

Type: Array of string

see AWS reference here.

Condition

No description

Type: UNSPECIFIED

Referenceable parameters

The following parameters can be easily referenced using $ResourceParam directive directive.

To learn more about referencing parameters, refer to referencing parameters.

name
  • AWS (physical) name of the bucket

  • Usage: $ResourceParam('<<resource-name>>', 'name')
arn
  • Arn of the bucket

  • Usage: $ResourceParam('<<resource-name>>', 'arn')
cdnDomain
  • Default domain of the CDN distribution (only available if you DO NOT configure custom domain names for the CDN).

  • Usage: $ResourceParam('<<resource-name>>', 'cdnDomain')
cdnUrl
  • Default url of the CDN distribution (only available if you DO NOT configure custom domain names for the CDN).

  • Usage: $ResourceParam('<<resource-name>>', 'cdnUrl')
cdnCustomDomains
  • Comma-separated list of custom domain names assigned to the CDN (only available if you configure custom domain names for the CDN).

  • Usage: $ResourceParam('<<resource-name>>', 'cdnCustomDomains')
cdnCustomDomainUrls
  • Comma-separated list of custom domain name URLs of the CDN (only available if you configure custom domain names for the CDN).

  • Usage: $ResourceParam('<<resource-name>>', 'cdnCustomDomainUrls')

API reference

KeyValuePair  API reference
Parent API reference: (AbortIncompleteMultipartUpload or NonCurrentVersionClassTransition or ClassTransition or NonCurrentVersionExpiration or Expiration or DirectoryUploadFilter)
key
Required

Key

Type: string

value
Required

Value

Type: string

Bucket  API reference
type
Required

No description

Type: string "bucket"

properties.directoryUpload

Allows you to upload a specified directory to the bucket on every deployment

Type: DirectoryUpload

  • After the upload is finished, your bucket will contain the contents of the local folder.
  • Files are uploaded using parallel, multipart uploads.

Existing contents of the bucket will be deleted and replaced with the contents of the local directory. You should not use directoryUpload for buckets with application-generated or user-generated content.

properties.accessibility

Configures accessibility of the bucket.

Type: BucketAccessibility

properties.cors

Configures CORS (Cross-Origin Resource Sharing) HTTP headers for the bucket.

Type: BucketCorsConfig

  • Web browsers use CORS (Cross-Origin Resource Sharing) to block the website from making requests to a different origin (server) than the one the website is served from. This means that if you make a request from a website served from https://my-website.s3.eu-west-1.amazonaws.com/ to https://my-api.my-domain.com, the request will fail.
properties.versioning

Enables versioning of objects in the bucket

Type: boolean

  • If enabled, bucket keeps multiple variants of an object.
  • This can help you to recover objects from an accidental deletion/overwrite, or to store multiple objects with the same name.
properties.encryption

Enables encryption of the objects stored in this bucket

Type: boolean

  • Objects are encrypted using the AES256 algorithm.
properties.lifecycleRules

Configures the way objects are stored throughout their lifecycle.

Type: Array of (Expiration or NonCurrentVersionExpiration or ClassTransition or NonCurrentVersionClassTransition or AbortIncompleteMultipartUpload)

  • Lifecycle rules are used to transition objects to different storage class, delete old objects, etc.
properties.cdn

Configures AWS Cloudfront CDN (Content Delivery Network) to be in front of your Bucket

Type: BucketCdnConfiguration

  • CDN is a globally distributed network that can cache responses from your bucket at the edge - close to your users.
  • AWS Cloudfront has 205 edge locations on 6 continents.
  • The CDN is used to:
    • reduce latency & improve load times
    • reduce bandwidth costs
    • reduce the amount of traffic coming to the origin
    • improve security
  • The "origin" is the resource(bucket) attached to the CDN. CDN caches responses from the origin at the edge for specified amount of time.
  • If the content requested by the client is in the CDN cache, the CDN immediately returns it to the client without making a request to the origin.
  • If the content is NOT in the cache, the CDN makes a request to the Origin. The response from the origin is then forwarded to the client, and cached at the edge.
overrides

Overrides one or more properties of the specified child resource.

Type: Object

  • Child resources are specified using their cloudformation logical id (e.g. MyBucketBucket).
  • To see all configurable child resources for given Stacktape resource, use stacktape stack-info --detailed command.
  • To see the list of properties that can be overridden, refer to AWS Cloudformation docs.
Need help? Ask a question on SlackDiscord or info@stacktape.com.