Stacktape


Redis Clusters

Overview and basic concepts

  • Redis clusters are fully managed (powered by Amazon ElastiCache for Redis). They are easy to set up, operate and scale. You don't have to worry about capacity scaling, hardware & VM provisioning, cluster setup, patching or backups.

  • Redis is a fast, NoSQL in-memory data store that can provide sub-millisecond latency.

  • Amazon ElastiCache redis clusters are built on open-source Redis and compatible with the Redis APIs. They work with Redis clients and use the open Redis data format to store your data.

When to use

Redis is used in cases when the performance is important. Example use-cases include caching, chat apps, messaging, queues, real-time analytics, etc.

Advantages

  • Performance - Redis is an in-memory data store and can deliver sub-millisecond response times. In comparison to disk-based databases (where most operations need a roundtrip to a disk), in-memory data stores can be an order of magnitude faster.
  • Fully managed - You don't have to worry about capacity scaling, hardware & VM provisioning, cluster setup, sharding process, patching, backups, etc.
  • High availability - Redis supports both clustered and non-clustered modes and provides high availability via support for automatic failovers with minimal impact.

Disdvantages

  • Data persistence - Redis offers 2 options for persistency: regular snapshotting and append-only files. None of them is as secure as a real transactional server providing undo/redo logging, point-in-time recovery, etc.
  • Hard to store large amounts of data - All your data must fit in memory.
  • Requires topology-aware clients - Clients connecting to the Redis cluster should be aware of the cluster topology. This requires more configuration on the client.

Basic usage

  • Following example shows basic usage of redis-cluster resource. Only required properties are instanceSize and defaultUserPassword.
  • The example shows a single node redis-cluster.

Copy

resources:
myRedisCluster:
type: redis-cluster
properties:
instanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')
myFunction:
type: function
properties:
packaging:
type: stacktape-lambda-buildpack
properties:
entryfilePath: path/to/my/lambda.ts
environment:
- name: REDIS_HOST
value: $ResourceParam('redisCluster', 'host')
- name: REDIS_PORT
value: $ResourceParam('redisCluster', 'port')
- name: REDIS_PASSWORD
value: $This().resources.myRedisCluster.properties.defaultUserPassword

Lambda function connected to a single-node redis cluster

Copy

import { Cluster } from 'ioredis';
const redisClient = new Redis({
host: process.env.REDIS_HOST,
port: Number(process.env.REDIS_PORT),
password: process.env.REDIS_PASSWORD
});
const handler = async (event, context) => {
await redisClient.set('currentTime', `${Date.now()}`);
const value = await redisClient.get('currentTime');
return { result: value };
};
export default handler;

Lambda function storing and reading data from a single-node redis cluster

Node instance size

  • Different instance sizes have different memory and network capabilities.
  • Instance size can be changed without interrupting the cluster or losing the data.
  • To learn more, and see a detailed list of available instances, refer to AWS pricing page

Copy

resources:
myRedisCluster:
type: redis-cluster
properties:
instanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')

Default user password

  • Redis clusters are password protected. To communicate with the cluster, you must configure a password for the default user.
  • Communication with the cluster is encrypted in-transit.
  • You should not input the password directly. The recommended way is using secrets.
  • Password constraints:
    • Must be only printable ASCII characters.
    • Must be at least 16 characters and no more than 128 characters in length.
    • Cannot contain any of the following characters: '/', '"', or '@'.

Copy

resources:
myRedisCluster:
type: redis-cluster
properties:
instanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')

Cluster topology

  • Cluster topology is based on the amount of replica nodes and the amount of shards.
  • Performance and high-availability of the cluster are dependent on the topology.

Single node (non-sharded) cluster

  • Single node, non-sharded cluster has only one node. It's not highly available.

Copy

resources:
myRedisCluster:
type: redis-cluster
properties:
instanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')

Single node (non-sharded) cluster topology
Single node (non-sharded) cluster topology

Multi node (non-sharded) cluster

  • Adding replica nodes (read replicas) to a cluster has 2 benefits:
    • increases the read throughput
    • increases availability. Replica node can become the primary node in case of primary node failure.
  • Load balancing between replicas is automatically handled by AWS.
  • When there are multiple shards, this number specifies the number of replicas for each shard cluster.

Changing numReplicaNodes when sharding is enabled is not possible and will result in error.

Copy

resources:
myRedisCluster:
type: redis-cluster
properties:
instanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')
numReplicaNodes: 2

Multi node (non-sharded) cluster topology
Multi node (non-sharded) cluster topology

Sharded cluster

  • Sharding enables you to scale redis cluster horizontally, by splitting the data into shards.
  • Each shard is then served by a shard cluster, which is responsible for given shard. Each shard cluster has a primary instance and numReplicaNodes amount of replicas.
  • You can update (increase or decrease) number of shards without interrupting the cluster.
  • You cannot update number of replicas per shard cluster on sharded cluster.
  • Routing to correct shard and shard re-balancing is automatically handled by AWS.
  • To learn more about sharding, refer to AWS Docs.

Sharding comes with multiple limitations:

  • Sharding can only be enabled during redis cluster initial creation. After that the cluster cannot be updated from non-sharded to sharded or vice-versa.
  • Changing numReplicaNodes when sharding is enabled is not possible. In this case it is necessary to delete and recreate the cluster.
  • numReplicaNodes must be set to at least 1 when sharding is enabled.

Copy

resources:
myRedisCluster:
type: redis-cluster
properties:
instanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')
enableSharding: true
numShards: 2
numReplicaNodes: 2

Sharded cluster topology
Sharded cluster topology

Copy

import { Cluster } from 'ioredis';
const redisClusterClient = new Cluster(
[
{
host: process.env.REDIS_HOST,
port: Number(process.env.REDIS_PORT)
}
],
{
redisOptions: { tls: {}, password: process.env.REDIS_PASSWORD },
dnsLookup: (address, callback) => callback(null, address)
}
);
const handler = async (event, context) => {
await redisClusterClient.set('currentTime', `${Date.now()}`);
const value = await redisClusterClient.get('currentTime');
return { result: value };
};
export default handler;

Using a sharded cluster from a lambda function

Logging

  • When logging is enabled, slow-log will be sent to a pre-created Cloudwatch log group.

Copy

resources:
myRedisCluster:
type: redis-cluster
properties:
instanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')
logging:
format: json

disabled
format
Default: json
retentionDays
Default: 90

Accessibility

  • You can configure which resources and hosts can access your cluster.
  • Redis clusters do not support public IP addresses. Therefore, only 2 access modes are supported.
RedisAccessibility  API reference
Parent API reference: RedisCluster
accessibilityMode
Default: vpcRequired

VPC mode

  • The cluster can be accessed only from resources within the default VPC.
  • Any function (provided it has joinDefaultVpc set to true), batch job or container workload within your stack can access the cluster.
  • Additionally, IP addresses configured in whitelistedIps can also access the database (even from the internet). To disable this behavior, and enforce database isolation ONLY to the VPC, you can set the forceDisablePublicIp property.

Copy

resources:
myRedisCluster:
type: redis-cluster
properties:
instanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')
accessibility:
accessibilityMode: vpc

Scoping workloads in VPC mode

  • Similar to vpc mode, but even more restrictive. In addition to resource being in the VPC, any host or resource trying to access your cluster must also have the sufficient IAM (Identity and Access management) permissions. You can grant these permission to your workloads in 2 ways:
  • Additionally, IP addresses configured in whitelistedIps can also access the cluster (even from the internet). To disable this behavior, and enforce cluster isolation, you can set the forceDisablePublicIp property.

Copy

resources:
myRedisCluster:
type: redis-cluster
properties:
instanceSize: cache.t3.micro
defaultUserPassword: $Secret('redis.password')
accessibility:
accessibilityMode: scoping-workloads-in-vpc

Referenceable parameters

The following parameters can be easily referenced using $ResourceParam directive directive.

To learn more about referencing parameters, refer to referencing parameters.

host
  • Hostname (address) of the primary instance that can be used for both reads and writes.

  • Usage: $ResourceParam('<<resource-name>>', 'host')
readerHost
  • Hostname (address) that can be used for reads only. (only available for NON-sharded clusters). If you use multiple replicas, it is advised to use readerHost for read operations to offload the primary host. ReaderHost automatically balances requests between available read replicas.

  • Usage: $ResourceParam('<<resource-name>>', 'readerHost')
port
  • Port of the cluster.

  • Usage: $ResourceParam('<<resource-name>>', 'port')
sharding
  • Indicates whether cluster is sharded. Available values: enabled or disabled.

  • Usage: $ResourceParam('<<resource-name>>', 'sharding')

Pricing

You are charged for:

  • Computing nodes, shards and replica nodes:
    • Price depends on the instance size, number of shards and number of replica nodes.
    • Can be calculated as number of shards * (number of replica nodes + 1 (there is always primary)) * price for node instance
    • EXAMPLE 1: Cheapest non-sharded cluster using cache.t3.micro instance with 0 replicaNodes: 1 * (0+1) * $0.017 = $0.017/hour (~$12.50/month)
    • EXAMPLE 2: Sharded cluster with 2 shards using cache.t3.micro and 1 replicaNode: 2 * (1+1) * $0.017 = $0.068 per hour (~$49/month)
  • Data transfer:
    • In most cases there should be no additional cost.
  • Backups:
    • If retained for at most 1 day: free
    • $0.085/GB/month after that

Free Tier (eligible for first 12 months)

  • Includes 750 hours of t2.micro instances (t3.micro for the regions in which t2.micro is not available) each month.

To learn more about pricing, refer to AWS pricing page

API Reference

RedisCluster  API reference
type
Required
properties.instanceSize
Required
properties.defaultUserPassword
Required
properties.enableSharding
properties.numShards
Default: 1
properties.numReplicaNodes
properties.enableAutomaticFailover
properties.logging
properties.automatedBackupRetentionDays
properties.port
Default: 6379
properties.accessibility
overrides
Need help? Ask a question on SlackDiscord or info@stacktape.com.