logoStacktape docs




Redis Clusters

Overview and basic concepts

  • Redis clusters are fully managed (powered by Amazon ElastiCache for Redis). They are easy to set up, operate and scale. You don't have to worry about capacity scaling, hardware & VM provisioning, cluster setup, patching or backups.

  • Redis is a fast, NoSQL in-memory data store that can provide sub-millisecond latency.

  • Amazon ElastiCache redis clusters are built on open-source Redis and compatible with the Redis APIs. They work with Redis clients and use the open Redis data format to store your data.

When to use

Redis is used in cases when the performance is important. Example use-cases include caching, chat apps, messaging, queues, real-time analytics, etc.

Advantages

  • Performance - Redis is an in-memory data store and can deliver sub-millisecond response times. In comparison to disk-based databases (where most operations need a roundtrip to a disk), in-memory data stores can be an order of magnitude faster.
  • Fully managed - You don't have to worry about capacity scaling, hardware & VM provisioning, cluster setup, sharding process, patching, backups, etc.
  • High availability - Redis supports both clustered and non-clustered modes and provides high availability via support for automatic failovers with minimal impact.

Disdvantages

  • Data persistence - Redis offers 2 options for persistency: regular snapshotting and append-only files. None of them is as secure as a real transactional server providing undo/redo logging, point-in-time recovery, etc.
  • Hard to store large amounts of data - All your data must fit in memory.
  • Requires topology-aware clients - Clients connecting to the Redis cluster should be aware of the cluster topology. This requires more configuration on the client.

Basic usage

  • Following example shows basic usage of redis-cluster resource. Only required properties are nodeInstanceSize and defaultUserPassword.
  • The example shows a single node redis-cluster.

resources:
myRedisCluster:
type: redis-cluster
properties:
nodeInstanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')
myFunction:
type: function
properties:
packageConfig:
filePath: path/to/my/lambda.ts
environment:
- name: REDIS_HOST
value: $Param('redisCluster', 'RedisCluster::PrimaryEndPoint.Address')
- name: REDIS_PORT
value: $Param('redisCluster', 'RedisCluster::PrimaryEndPoint.Port')
- name: REDIS_PASSWORD
value: $This().resources.myRedisCluster.properties.defaultUserPassword

Lambda function connected to a single-node redis cluster

import { Cluster } from 'ioredis';
const redisClient = new Redis({
host: process.env.REDIS_HOST,
port: Number(process.env.REDIS_PORT),
password: process.env.REDIS_PASSWORD
});
const handler = async (event, context) => {
await redisClient.set('currentTime', `${Date.now()}`);
const value = await redisClient.get('currentTime');
return { result: value };
};
export default handler;

Lambda function storing and reading data from a single-node redis cluster

Node instance size

No description

resources:
myRedisCluster:
type: redis-cluster
properties:
nodeInstanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')

Default user password

  • Redis clusters are password protected. To communicate with the cluster, you must configure a password for the default user.
  • Communication with the cluster is encrypted in-transit.
  • You should not input the password directly. The recommended way is using a .env file together with File directive or even better, using secrets.
  • Password constraints:
    • Must be only printable ASCII characters.
    • Must be at least 16 characters and no more than 128 characters in length.
    • Cannot contain any of the following characters: '/', '"', or '@'.

resources:
myRedisCluster:
type: redis-cluster
properties:
nodeInstanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')

Cluster topology

  • Cluster topology is based on the amount of replica nodes and the amount of shards.
  • Performance and high-availability of the cluster are dependent on the topology.

Single node (non-sharded) cluster

  • Single node, non-sharded cluster has only one node. It's not highly available.

resources:
myRedisCluster:
type: redis-cluster
properties:
nodeInstanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')

Single node (non-sharded) cluster topology
Single node (non-sharded) cluster topology

Multi node (non-sharded) cluster

  • Adding replica nodes (read replicas) to a cluster has 2 benefits:
    • increases the read throughput
    • increases availability. Replica node can become the primary node in case of primary node failure.
  • Load balancing between replicas is automatically handeled by AWS.
  • When there are multiple shards, this number specifies the number of replicas for each shard cluster.

Changing numReplicaNodes when sharding is enabled is not possible and will result in error.

resources:
myRedisCluster:
type: redis-cluster
properties:
nodeInstanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')
numReplicaNodes: 2

Multi node (non-sharded) cluster topology
Multi node (non-sharded) cluster topology

Sharded cluster

  • This property is only effective when enableSharding is set to true.

resources:
myRedisCluster:
type: redis-cluster
properties:
nodeInstanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')
numShards: 2
numReplicaNodes: 2

Sharded cluster topology
Sharded cluster topology

import { Cluster } from 'ioredis';
const redisClusterClient = new Cluster(
[
{
host: process.env.REDIS_HOST,
port: Number(process.env.REDIS_PORT)
}
],
{
redisOptions: { tls: {}, password: process.env.REDIS_PASSWORD },
dnsLookup: (address, callback) => callback(null, address)
}
);
const handler = async (event, context) => {
await redisClusterClient.set('currentTime', `${Date.now()}`);
const value = await redisClusterClient.get('currentTime');
return { result: value };
};
export default handler;

Using a sharded cluster from a lambda function

Logging

  • When logging is enabled, slow-log will be sent to a pre-created Cloudwatch log group.

resources:
myRedisCluster:
type: redis-cluster
properties:
nodeInstanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')
logging:
enabled: true

enabled

Enables sending slow-log to a Cloudwatch log group.

Type: boolean

format
Default: json

Configures the log format

Type: string ENUM

Possible values: jsontext

retentionDays
Default: 90

Amount of days the logs will be retained in the log group

Type: number ENUM

Possible values: 13571430609012015018036540054573118273653

Accessibility

  • You can configure which resources and hosts can access your cluster.
  • Redis clusters do not support public IP addresses. Therefore, only 2 access modes are supported.
RedisAccessibility  API reference
Parent API reference: RedisCluster
accessibilityMode
Default: 'vpc'Required

Configures the accessibility mode for this cluster

Type: string ENUM

Possible values: scoping-workloads-in-vpcvpc

The following modes are supported:

  • vpc - The cluster can be accessed only from resources within your VPC. This means any function (provided it has joinDefaultVpc set to true), batch job or container workload within your stack can access the cluster.
  • scoping-workloads-in-vpc - similar to vpc mode, but even more restrictive. In addition to being in the same VPC, the resources and hosts accessing your redis cluster must also have sufficient IAM permissions (for funtions, batch jobs and container workloads, these permissions can be granted using allowsAccessTo or iamRoleStatements in their configuration).

Redis clusters do not support public IP addresses. Therefore, you cannot connect to them from your own computer. To avoid this shortcoming, you can use a bastion server. Native support for bastion servers in Stacktape will be available soon.

To learn more about VPCs, refer to VPC Docs.

VPC mode

  • The cluster can be accessed only from resources within the default VPC.
  • Any function (provided it has joinDefaultVpc set to true), batch job or container workload within your stack can access the cluster.
  • Additionally, IP addresses configured in whitelistedIps can also access the database (even from the internet). To disable this behavior, and enforce database isolation ONLY to the VPC, you can set the forceDisablePublicIp property.

resources:
myRedisCluster:
type: redis-cluster
properties:
nodeInstanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')
accessibility:
accessibilityMode: vpc

Scoping workloads in VPC mode

  • Similar to vpc mode, but even more restrictive. In addition to resource being in the VPC, any host or resource trying to access your cluster must also have the sufficient IAM (Identity and Access management) permissions. You can grant these permission to your workloads in 2 ways:
  • Additionally, IP addresses configured in whitelistedIps can also access the cluster (even from the internet). To disable this behavior, and enforce cluster isolation, you can set the forceDisablePublicIp property.

resources:
myRedisCluster:
type: redis-cluster
properties:
nodeInstanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')
accessibility:
accessibilityMode: scoping-workloads-in-vpc

Pricing

You are charged for:

  • Computing nodes, shards and replica nodes:
    • Price depends on the instance size, number of shards and number of replica nodes.
    • Can be calculated as number of shards * (number of replica nodes + 1 (there is always primary)) * price for node instance
    • EXAMPLE 1: Cheapest non-sharded cluster using cache.t3.micro instance with 0 replicaNodes: 1 * (0+1) * $0.017 = $0.017/hour (~$12.50/month)
    • EXAMPLE 2: Sharded cluster with 2 shards using cache.t3.micro and 1 replicaNode: 2 * (1+1) * $0.017 = $0.068 per hour (~$49/month)
  • Data transfer:
    • In most cases there should be no additional cost.
  • Backups:
    • If retained for at most 1 day: free
    • $0.085/GB/month after that

Free Tier (eligible for first 12 months)

  • Includes 750 hours of t2.micro instances (t3.micro for the regions in which t2.micro is not available) each month.

To learn more about pricing, refer to AWS pricing page

API Reference

RedisCluster  API reference
type
Required

No description

Type: string "redis-cluster"

properties.instanceSize
Required

Instance size for every node in the cluster (both primary node and replica nodes, if configured)

Type: string ENUM

Possible values: cache.m4.10xlargecache.m4.2xlargecache.m4.4xlargecache.m4.largecache.m4.xlargecache.m5.12xlargecache.m5.24xlargecache.m5.2xlargecache.m5.4xlargecache.m5.largecache.m5.xlargecache.m6g.12xlargecache.m6g.24xlargecache.m6g.2xlargecache.m6g.4xlargecache.m6g.largecache.m6g.xlargecache.r4.16xlargecache.r4.2xlargecache.r4.4xlargecache.r4.8xlargecache.r4.largecache.r4.xlargecache.r5.12xlargecache.r5.24xlargecache.r5.2xlargecache.r5.4xlargecache.r5.largecache.r5.xlargecache.r6g.12xlargecache.r6g.24xlargecache.r6g.2xlargecache.r6g.4xlargecache.r6g.largecache.r6g.xlargecache.t2.mediumcache.t2.microcache.t2.smallcache.t3.mediumcache.t3.microcache.t3.small

  • Different instance sizes have different memory and network capabilities.
  • Instance size can be changed without interupting the cluster or losing the data.
  • To learn more, and see a detailed list of available instances, refer to AWS pricing page

properties.defaultUserPassword
Required

Password for the default cluster user.

Type: string

  • Redis clusters are password protected. To communicate with the cluster, you must configure a password for the default user.
  • Communication with the cluster is encrypted in-transit.
  • You should not input the password directly. The recommended way is using a .env file together with File directive or even better, using secrets.
  • Password constraints:
    • Must be only printable ASCII characters.
    • Must be at least 16 characters and no more than 128 characters in length.
    • Cannot contain any of the following characters: '/', '"', or '@'.

properties.enableSharding

Enables sharding for the cluster

Type: boolean

  • Sharding enables you to scale redis cluster horizontally, by splitting the data into shards.
  • Each shard is then served by a shard cluster, which is responsible for given shard. Each shard cluster has a primary instance and numReplicaNodes amount of replicas.
  • You can update (increase or decrease) number of shards without interupting the cluster.
  • You cannot update number of replicas per shard cluster on sharded cluster.
  • Routing to correct shard and shard rebalancing is automatically handeled by AWS.
  • To learn more about sharding, refer to AWS Docs.

Sharding comes with multiple limitations:

  • Sharding can only be enabled during redis cluster initial creation. After that the cluster cannot be updated from non-sharded to sharded or vice-versa.
  • Changing numReplicaNodes when sharding is enabled is not possible. In this case it is necessary to delete and recreate the cluster.
  • numReplicaNodes must be set to at least 1 when sharding is enabled.

properties.numShards
Default: 1

Configures number of shards for the cluster

Type: number

  • This property is only effective when enableSharding is set to true.

properties.numReplicaNodes
Default: 0

Number of replica nodes in the cluster

Type: number

  • Adding replica nodes (read replicas) to a cluster has 2 benefits:
    • increases the read throughput
    • increases availability. Replica node can become the primary node in case of primary node failure.
  • Load balancing between replicas is automatically handeled by AWS.
  • When there are multiple shards, this number specifies the number of replicas for each shard cluster.

Changing numReplicaNodes when sharding is enabled is not possible and will result in error.

properties.enableAutomaticFailover

Enables automatic failover of cluster's primary instance

Type: boolean

  • When automatic failover is enabled and primary instance fails, replica takes over and serves as primary instance.
  • To use automatic failover numReplicaNodes must be at least 1.
  • On sharded clusters automatic failover is enabled by default and cannot be disabled.

When you are deploying(updating) your cluster and you wish to enable automatic failover at least one replica must already exist. If replicas do not exist for the cluster and you wish to enable automatic failover, you need to run stacktape deploy twice:

  1. first deploy: add replicas to the cluster.
  2. second deploy: enable automatic failover. Similarly if you wish to disable automatic failover and remove replicas, it is 2 deploy process:
  3. first deploy: disable automatic failover
  4. second deploy: remove all replicas

properties.logging

Configures logging behavior and logging format

Type: RedisLogging

  • Slow log is sent to a pre-created Cloudwatch log group.
  • By default, logs are retained for 180 days.
  • You can browse your logs in 2 ways:
    • go to the log group page in the AWS CloudWatch console. You can use stacktape stack-info command to get a direct link.
    • use stacktape logs command to print logs to the console

properties.automatedBackupRetentionDays
Default: 0

Configures the retention period for automated backups

Type: number

  • When set to 0, automatic backups are disabled.
  • You can also take manual backup snapshots (in the AWS console or using the API). The retention period is not applied to manual backup snapshots.

properties.port
Default: 6379

Port on which the redis-cluster should listen for connections

Type: number

properties.accessibility

Configures which resources and hosts can access your cluster

Type: RedisAccessibility

overrides

Overrides one or more properties of the specified child resource.

Type: Object

  • Child resouces are specified using their descriptive name (e.g. DbInstance or Events.0.HttpApiRoute).
  • To see all configurable child resources for given Stacktape resource, use stacktape stack-info --detailed command.
  • To see the list of properties that can be overriden, refer to AWS Cloudformation docs.