OpenSearch (Elastic)
OpenSearch is a scalable, open-source search and analytics suite. It's used for various purposes, including full-text search, log analytics, and real-time application monitoring. As a fully managed service, Stacktape makes it easy to deploy, operate, and scale OpenSearch clusters in the AWS Cloud.
Under the hood, Stacktape uses AWS OpenSearch Service, which was previously known as Amazon Elasticsearch Service.
Basic configuration
To set up a basic OpenSearch domain, you only need to specify the resource type and the desired OpenSearch version.
- Cluster config determines size, number and type of instances used in your OpenSearch domain cluster.
- If you do not specify
clusterConfig
, cluster will only contain singlem4.large.search
node.
resources:myOpenSearch:type: open-search-domainproperties:version: "2.17"
Connecting to an OpenSearch domain
You can connect to your OpenSearch domain either from anywhere on the internet or exclusively from resources within your stack's VPC, depending on your accessibility configuration.
In either case, you need valid IAM credentials. You can grant these to your stack's resources using the connectTo
property.
resources:myOpenSearch:type: open-search-domainproperties:accessibility:accessibilityMode: vpcwebService:type: web-serviceproperties:# ...connectTo:- myOpenSearch
A web service connected to an OpenSearch domain.
import { Client, Connection } from "@opensearch-project/opensearch";import { defaultProvider } from "@aws-sdk/credential-provider-node";import aws4 from "aws4";// The OpenSearch domain endpoint is automatically injected as an environment variableconst host = `https://${process.env.STP_MY_OPEN_SEARCH_DOMAIN_ENDPOINT}`;const createAwsConnector = (credentials, region) => {class AmazonConnection extends Connection {buildRequestObject(params) {const request = super.buildRequestObject(params) as any;request.service = "es";request.region = region;request.headers = request.headers || {};request.headers["host"] = request.hostname;return aws4.sign(request, credentials);}}return {Connection: AmazonConnection};};const getClient = async () => {const credentials = await defaultProvider()();return new Client({...createAwsConnector(credentials, "eu-west-1"),node: host});};async function search() {const client = await getClient();await client.indices.create({index: "test-index"});}
Example code for connecting to an OpenSearch domain.
Data nodes
You can configure the instance type and number of data nodes in your cluster.
- Specify
instanceCount
together withinstanceType
to configure data nodes in your cluster. - Data nodes handle data storage, indexing, and query processing in the cluster.
- For production setups, it's best to pair data nodes with dedicated master nodes to improve cluster stability.
- Check AWS Docs for the list of supported instance types.
resources:myOpenSearch:type: open-search-domainproperties:clusterConfig:instanceType: r6g.large.searchinstanceCount: 3
OpenSearch version
You can specify the version of OpenSearch you want to use.
- Explicitly specify open-search-domain version to lock it and prevent potential problems when default version changes.
- Currently, default version is
2.17
but this can change with later Stacktape releases.
resources:myOpenSearch:type: open-search-domainproperties:version: "2.17"
Dedicated master nodes
For production environments, it's recommended to use dedicated master nodes to improve cluster stability.
- Specify
dedicatedMasterCount
together withdedicatedMasterType
if you wish to use dedicated master nodes in your cluster. - Dedicated master nodes manage cluster state and coordinate node activities but do not store data or serve queries.
- Dedicated masters are recommended for clusters with 3+ nodes to improve stability and prevent split-brain scenarios.
- Master instance type must be appropriately sized based on the number of nodes and shards in the cluster (see AWS guidelines).
- Number of masters should always be odd (3, 5, or 7) for quorum-based fault tolerance.
- Adds to cluster cost but improves reliability for larger or mission-critical setups.
- To check recommendations and best practices for dedicated master nodes, see AWS Docs.
resources:myOpenSearch:type: open-search-domainproperties:version: "2.17"clusterConfig:instanceType: r6g.large.searchinstanceCount: 3dedicatedMasterType: r6g.large.searchdedicatedMasterCount: 3
Warm nodes
Warm nodes provide a cost-effective way to store large amounts of read-only data.
- Specify
warmType
andwarmCount
properties if you wish to use UltraWarm storage instances for your cluster. - Warm nodes store infrequently accessed or older data, optimizing costs for time-series or log data.
- Data on warm nodes remains searchable but with higher query latency compared to hot nodes.
- Use Case: Ideal for retaining historical data without impacting the performance of frequently accessed data.
- To learn more about UltraWarm storage and check prerequisites and best practices for dedicated master nodes, see AWS Docs.
resources:myOpenSearch:type: open-search-domainproperties:version: "2.17"clusterConfig:instanceType: r6g.large.searchinstanceCount: 3warmType: ultrawarm1.medium.searchwarmCount: 2
Multi-AZ with standby
You can enable a multi-AZ deployment with a standby Availability Zone for high availability.
- When enabled, this option ensures high availability and consistent performance by enforcing several best practices such as:
- The nodes and data copies are distributed across three AZs with one of the AZs serving only as a backup. This ensures that during failure, the standby AZ can jump in without causing re-balancing and availability issues.
- OpenSearch version 1.3 or above
- Auto-Tune enabled on the domain
- Three dedicated master nodes and data nodes
- Only GP3 or SSD-backed instances and a subset of instance types are currently supported.
- To learn more about Multi-AZ with Standby, see AWS Docs.
resources:myOpenSearch:type: open-search-domainproperties:version: "2.17"clusterConfig:instanceType: r6g.large.searchinstanceCount: 3dedicatedMasterType: r6g.large.searchdedicatedMasterCount: 3standbyEnabled: true
Restrict access
You can restrict access to your OpenSearch domain to a specific VPC.
- Use
accessibility
andaccessibilityMode
properties if you wish to restrict access to the domain. - By default, the domain is accessible from anywhere on the internet (but still protected using IAM).
- You can further restrict this access by making domain available only to resources in your stack's VPC.
Note that If you launch a new domain with accessibility mode set to vpc
or scoping-workloads-in-vpc
, you can't later switch it to mode internet
.
The reverse is also true: If you create a domain with accessibility mode internet
,
you can't later place it within a VPC. Instead, you must create a new domain and migrate your data.
The following modes are supported:
- internet - Least restrictive mode. The domain can be accessed from anywhere on the internet.
- vpc - The domain can be accessed only from resources within your VPC. This
means any function (provided it has
joinDefaultVpc
set to true), batch job, container workload or container service within your stack can access the cluster (if it also has required credentials). - scoping-workloads-in-vpc - similar to vpc mode, but even more restrictive. In addition to being in the same VPC, the resources of your stack
accessing the cluster must also have sufficient security group permissions (for functions, batch jobs and container services, these permissions
can be granted together with IAM permissions using
connectTo
in their configuration).
To learn more about VPCs, refer to VPC Docs.
resources:myOpenSearch:type: open-search-domainproperties:accessibility:accessibilityMode: vpc
Logging
You can enable and configure logging for your OpenSearch domain.
- Stacktape will automatically create required log groups and policies for you.
- You can set custom retention period for the individual log types or disable log collection completely.
- To learn more about log collection, see AWS Docs.
resources:myOpenSearch:type: open-search-domainproperties:logging:errorLogs:retentionDays: 30searchSlowLogs:retentionDays: 14indexSlowLogs:disabled: true
Storage
You can configure the storage for your OpenSearch domain using EBS volumes.
- Storage configuration only supported for instances supporting EBS storage (not with dedicated storage).
- Setting
iops
andthroughput
is only allowed for instances withgp3
storage type (most new generation instances aregp3
).
resources:myOpenSearch:type: open-search-domainproperties:clusterConfig:instanceType: r6g.large.searchinstanceCount: 3storage:size: 100iops: 3000throughput: 200
Disable Multi-AZ awareness
You can disable Multi-AZ awareness if you don't need the high availability features.
- By default Multi AZ awareness is enabled for any cluster with more than one node. This means that OpenSearch Service allocates the nodes and replica index shards across multiple AZs to prevent data loss and minimize downtime in the event of node or data center failure.
- Multi AZ awareness only takes effect for clusters with more than one data node.
- Setting
multiAzDisabled
totrue
and disabling zone awareness for your cluster is not recommended. - To learn more about multi AZ cluster, see AWS Docs.
resources:myOpenSearch:type: open-search-domainproperties:version: "2.17"clusterConfig:instanceType: r6g.large.searchinstanceCount: 3multiAzDisabled: true