Close

Atlas Mongo Clusters

Overview

By using atlasMongoClusters resource you are able to deploy MongoDB clusters managed by one of the most innovative cloud database service provider Atlas MongoDB. Moreover, with Stacktape you can seamlessly integrate Atlas MongoDB clusters with the rest of your stack resources, while keeping your database isolated and secured. Your self-healing clusters are made up of distributed database instances to ensure no single point of failure.

When using atlasMongoClusters resources in your stack, new Atlas MongoDB project is created, for each different stage deployed. This ensures absolute isolation between multiple stages (copies) of your environment.

Usage

➡️ Setting Atlas stack properties

When using atlasMongoClusters resources, you need to setup atlasMongo section in stackConfig section of you configuration file. In this section you need to provide:

  • Atlas MongoDB API keys - API keys are used to interact with Atlas MongoDB provider when deploying clusters. You can get API Keys for you organization by following Atlas MongoDB docs tutorial. We strongly advise you to store the API keys (namely privateKey) as a secret. --- @todo this will be broken soon
  • Organization Id - Id of your Atlas MongoDB organization in which projects will be created as a part of stack stage deployment.
  • Connectivity settings - Connectivity settings of the Atlas MongoDB project (created as a part of stack stage deployment) are set using allowConnectionsFrom property. All atlasMongoClusters in your stack deployment will share this settings.

stackConfig:
atlasMongo:
# as Atlas MongoDB is a third service provider it is neccessary to provide API keys with sufficient rights
apiCredentials:
privateKey: "$GetSecret('mongo-private-api-key')"
publicKey: 'xxxxxxx'
# "allowConnectionsFrom" option is shared between "atlasMongoClusters" of your stack as these are set per Atlas project NOT cluster
allowConnectionsFrom: 'vpc'
# organization id, identifying your Atlas MongoDb organization
organizationId: 'xxxxxxxxxxx07a593cbe63dd'


➡️ Basic usage

Following template shows basic atlasMongoCluster use. Only required parameter is instanceSize.

stackConfig:
atlasMongo:
apiCredentials:
privateKey: "$GetSecret('mongo-private-api-key')"
publicKey: 'xxxxxxx'
allowConnectionsFrom: 'internet'
organizationId: 'xxxxxxxxxxx07a593cbe63dd'
resources:
atlasMongoClusters:
myMongoCluster:
instanceSize: 'M2'


➡️ Restricting access to clusters

Access to the atlasMongoClusters can be restricted using allowConnectionsFrom property in atlasMongo section of stackConfig. All atlasMongoClusters in your stack deployment share these settings.

The allowConnectionsFrom property can be set to one of the following values:

  • "internet" - least restrictive setting. When using this option atlasMongoClusters can be accessed from any IP address on the internet. However even when using this option, access to cluster can be finely controlled. See example. Also see section Accessing clusters from workloads for deeper insights on accessing clusters.

  • "vpc" - atlasMongoClusters can be accessed only from resources in your vpc. This means any function(provided it has joinVpc set to true), batchJob or containerWorkload within your stack can access atlasMongoClusters. (You can additionally use property additionalWhitelistedIps to whitelist specific ip addresses from the outer internet.)

  • "scoping-workloads-in-vpc" - similar to vpc setting but even more restrictive. In addition to resource being in your vpc, it also has to scope the atlasMongoCluster in its allowAccessTo list. See example below. (You can additionally use property additionalWhitelistedIps to whitelist specific ip addresses from the outer internet.)

  • "whitelisted-ips-only" - atlasMongoClusters can only be accessed from ip addresses and cidr blocks listed in the additionalWhitelistedIps list.

When using Atlas MongoDB shared tier clusters (M2 and M5) allowConnectionsFrom property should be set to "internet". This is because shared clusters are not using same level of network isolation as dedicated clusters(M10+). and therefore do not allow for same level of network inter-connection.

Nevertheless, even when using allowConnectionsFrom set to "internet", clusters are still tightly protected as Stacktape utilizes strict identity access management between your stack workloads and atlas mongo clusters. See section Accessing clusters from workloads

Example "internet"

stackConfig:
atlasMongo:
apiCredentials:
privateKey: "$GetSecret('mongo-private-api-key')"
publicKey: 'xxxxxxx'
allowConnectionsFrom: 'internet'
organizationId: 'xxxxxxxxxxx07a593cbe63dd'
resources:
atlasMongoClusters:
myMongoCluster:
instanceSize: 'M2'
functions:
myMongoFunction:
filePath: 'lambdas/mongo-lambda.ts'
memory: 512
# by allowing access to cluster, lambda receives permissions for reading and writing into cluster databases
allowAccessTo:
- 'myMongoCluster'
environment:
# injecting the connection string as environment variable
MONGODB_CONNECTION_STRING: "$GetParam('myMongoCluster', 'AtlasMongoCluster::SrvConnectionString')"

Example "scoping-workloads-in-vpc"

stackConfig:
atlasMongo:
apiCredentials:
privateKey: "$GetSecret('mongo-private-api-key')"
publicKey: 'xxxxxxx'
allowConnectionsFrom: 'scoping-workloads-in-vpc'
organizationId: 'xxxxxxxxxxx07a593cbe63dd'
resources:
functions:
# functionOne does NOT have access to cluster eventhough it is joined in vpc
functionOne:
filePath: 'path/to/my-lambda.ts'
joinVpc: true
# functionTwo does have access to cluster, because it is scoping the database in allowAccessTo list
functionTwo:
filePath: 'path/to/my-lambda-2.ts'
joinVpc: true
allowAccessTo:
- 'myMongoCluster'
atlasMongoClusters:
myMongoCluster:
instanceSize: 'M10'


➡️ Accessing clusters from workloads

Following example demonstrates how to grant a lambda function myMongoFunction a permission to access your atlasMongoCluster myMongoCluster.

By listing myMongoCluster in allowAccessTo of myMongoFunction, function is injected with credentials needed for accessing the cluster.

stackConfig:
atlasMongo:
apiCredentials:
privateKey: "$GetSecret('mongo-private-api-key')"
publicKey: 'xxxxxxx'
allowConnectionsFrom: 'internet'
organizationId: 'xxxxxxxxxxx07a593cbe63dd'
resources:
atlasMongoClusters:
myMongoCluster:
instanceSize: 'M2'
functions:
myMongoFunction:
filePath: 'lambdas/mongo-lambda.ts'
memory: 512
# by allowing access to cluster, lambda receives permissions for reading and writing into cluster databases
allowAccessTo:
- 'myMongoCluster'
environment:
# injecting the connection string as environment variable
MONGODB_CONNECTION_STRING: "$GetParam('myMongoCluster', 'AtlasMongoCluster::SrvConnectionString')"

Code example

Following snippet shows how we connect to myMongoCluster from the myMongoFunction using the popular mongoose library.

When using auth mechanism MONGODB_AWS credentials for authentication are automatically loaded from function's environment variables. As mentioned above, function (or any other type of workload) receives the permissions by having the myMongoCluster listed in its allowAccessTo list.

import mongoose from 'mongoose';
let connection;
export default async (event, context) => {
// lambda handler code
// we are using the injected connection string to create connection
// ...
connection =
connection ||
(await mongoose.connect(process.env.MONGODB_CONNECTION_STRING, {
authMechanism: 'MONGODB-AWS',
authSource: '$external',
useNewUrlParser: true,
dbName: 'my-test-database'
}));
// do whatever you wish with the connection
// rest of the lambda code
// ...
};


➡️ Setting disk size

You can set disk size with parameter diskSizeGB. By default all M10+ clusters autoscale the storage without any users manual intervention. You can disable this behaviour by setting autoScaling.disableDiskScaling property to true.

Setting disk size (and disk autoscaling) is NOT available for shared clusters (M2 and M5). For M10+ clusters default disk size depends on instance size (see Atlas MongoDB docs).

stackConfig:
atlasMongo:
apiCredentials:
privateKey: "$GetSecret('mongo-private-api-key')"
publicKey: 'xxxxxxx'
allowConnectionsFrom: 'internet'
organizationId: 'xxxxxxxxxxx07a593cbe63dd'
resources:
atlasMongoClusters:
myMongoCluster:
instanceSize: 'M10'
diskSizeGB: 20


➡️ Autoscaling

You can easily setup autoscaling of the instanceSize by using autoScaling section.

In this section you can configure your Atlas cluster to automatically scale its cluster tier, storage capacity, or both in response to cluster usage. Cluster auto-scaling removes the need to write scripts or use consulting services to make scaling decisions. To help control costs, you can specify a range of cluster tiers (instanceSizes) to which your cluster can automatically scale.

Autoscaling is NOT available for shared tier clusters (instanceSize M2 and M5).

stackConfig:
atlasMongo:
apiCredentials:
privateKey: "$GetSecret('mongo-private-api-key')"
publicKey: 'xxxxxxx'
allowConnectionsFrom: 'internet'
organizationId: 'xxxxxxxxxxx07a593cbe63dd'
resources:
atlasMongoClusters:
myMongoCluster:
instanceSize: 'M10'
autoScaling:
# OPTIONAL minimal instance size, cluster can scale DOWN to
minInstanceSize: 'M10'
# OPTIONAL maximal instance size, cluster can scale UP to
maxInstanceSize: 'M30'
# OPTIONAL set disableDiskScaling to "true" to avoid automatic disk scaling when disk is close to full
disableDiskScaling: true # default is false, i.e diskScaling is ENABLED
# OPTIONAL by disabling scale down you are ensuring
# that the cluster can only scale UP to bigger instance but not back DOWN to smaller one
disableScaleDown: true # default is false, i.e scaleDown is ENABLED

How autoscaling works

InstanceSize Scale-UP

Atlas scales the cluster up to the next tier(instanceSize) if one of the following is true for any node in the cluster:

  • Average CPU Utilization has exceeded 75% for the past hour, or
  • Memory Utilization has exceeded 75% for the past hour.

InsanceSize Scale-DOWN

Atlas scales the cluster down to the next lowest tier(instanceSize) if both of the following are true for all nodes in the cluster:

  • The average CPU Utilization and Memory Utilization over the past 24 hours is below 50%, and
  • The cluster has not been scaled down (manually or automatically) in the past 24 hours.

Disk scaling

When disk scaling is enabled, Atlas automatically increases cluster storage when disk space used reaches 90%.

  • Atlas increases cluster storage capacity to achieve 70% disk utilization.
  • Atlas never automatically scales cluster storage down.

➡️ Creating master user

Optionally you can create a master database user with atlasAdmin priviliges over your cluster.

In many cases you do NOT need to create the master database user, as the atlas mongo clusters can be accessed from your workloads (functions, containers, batchjobs) without creating explicit master user. This is thanks to identity access management Stacktape utilizes between your stack workloads (functions, batchjobs, containerWorkloads) and atlas mongo clusters (explained in section Accessing clusters from workloads).

However, in cases when you need o access clusters from outside of your stack, master user can be useful.

stackConfig:
atlasMongo:
apiCredentials:
privateKey: "$GetSecret('mongo-private-api-key')"
publicKey: 'xxxxxxx'
allowConnectionsFrom: 'internet'
organizationId: 'xxxxxxxxxxx07a593cbe63dd'
resources:
atlasMongoClusters:
myMongoCluster:
instanceSize: 'M2'
# name of the database user
masterUserName: 'my-master-user'
# password of the database user (we recommend storing passwords using secrets)
masterUserPassword: "$GetSecret('mongo-master-password')"


➡️ Automatic backups

Atlas Cloud Backups provide localized backup storage using the native snapshot functionality of the underlying AWS provider. You can enable cloud backups by using cloudBackupEnabled property.


Additionally to cloudBackupEnabled property, you can also use continousCloudBackup property as well. Continuous Cloud Backups replay the oplog to restore a cluster from a particular point in time within a window specified in the Backup Policy.

stackConfig:
atlasMongo:
apiCredentials:
privateKey: "$GetSecret('mongo-private-api-key')"
publicKey: 'xxxxxxx'
allowConnectionsFrom: 'internet'
organizationId: 'xxxxxxxxxxx07a593cbe63dd'
resources:
atlasMongoClusters:
myMongoCluster:
instanceSize: 'M10'
# enable cloud backup
cloudBackupEnabled: true
# enable continous cloud backup
continousCloudBackup: true

Enabling backups and continuous cloud backups increases the monthly cost of your cluster. To learn more about cost implications read:

The default backup policy is shown in the following table. The default snapshot time is 18:00 UTC. Notice that the behaviour for Continous Cloud Backup (if enabled) is different for different tiers of clusters:

Policy Type
Tier
Continuous Cloud Backup
Snapshot Taken
Snapshot Retained
Hourly
NVMe
Enabled
Every 12 hours
7 days
Hourly
non-NVMe
Enabled
Every 6 hours
7 days
Daily
All
Either
Every day
7 days
Weekly
All
Either
Every Saturday
4 weeks
Monthly
All
Either
Last day of the month
12 months
Properties cloudBackupEnabled and continuousCloudBackup are not available for shared tier clusters (instanceSize M2 and M5). However, Atlas takes daily snapshots of your M2 and M5 clusters which you can restore to clusters tiers M2 or greater.

API Reference

Property in Stacktape configAllowed types
stackConfig.atlasMongo.additionalWhitelistedIps[]string
stackConfig.atlasMongo.allowConnectionsFromenumRequired
stackConfig.atlasMongo.apiCredentialsApiCredentialsRequired
stackConfig.atlasMongo.organizationIdstringRequired
resources.atlasMongoClusters.{name}.autoScalingAutoScaling
resources.atlasMongoClusters.{name}.biConnectorBiConnector
resources.atlasMongoClusters.{name}.cloudBackupEnabledboolean
resources.atlasMongoClusters.{name}.clusterTypeenum
resources.atlasMongoClusters.{name}.continuousCloudBackupboolean
resources.atlasMongoClusters.{name}.diskSizeGBnumber
resources.atlasMongoClusters.{name}.instanceSizeenumRequired
resources.atlasMongoClusters.{name}.masterUserNamestring
resources.atlasMongoClusters.{name}.masterUserPasswordstring
resources.atlasMongoClusters.{name}.numShardsnumber
resources.atlasMongoClusters.{name}.replicationSpecsReplicationSpecs
🗄️ Resources — Previous
Databases
Next — 🗄️ Resources
Buckets