Overview and basic concepts

Amazon Simple Queue Service (SQS) lets you send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.

Amazon SQS provides a simple and reliable way for customers to decouple and connect components (microservices) together using queues.

Under the hood

SQS queue is an abstraction for Amazon Simple Queue Service (SQS).

When to use

Stacktape offers 3 types of AWS managed resources intended for messaging/communication(all use "pay per use" model and scale automatically):

1. AWS SNS topic
2. AWS SQS queue
3. AWS EventBridge bus

Depending on you use case you might choose to use:

Use SNS topic when

• You want to publish messages to MANY different subscribers with a single action.
• Require high throughput and reliability for publishing and delivery to consumers.
• Have many subscribers.

Use SQS queue when

• You are looking for reliable 1:1 asynchronous communication to decouple your applications from one another
• You want to rate limit your consumption of messages (in case of SQS consumer fetches the messages so you can control the rate)

Use Event bus when

• You want to publish messages to many subscribers, and use the event data itself to match targets interested certain patterns (with even bus, you can filter which messages should be delivered to the subscriber based on message content)
• You want integration with other SaaS providers such as Shopify, Datadog, Pagerduty, or others.

Basic usage

By default sqs-queue does not need to have any properties defined.

Copy
resources:
  mainQueue:
    type: sqs-queue

Integrating with workloads

You can invoke function or a batch-job every time messages are in sqs-queue. This can be done by creating event integration:

• Messages are processed in batches
• If the SQS queue contains multiple messages, the function/batch-job is invoked with multiple messages contained in an event
• If function fails while processing messages or batch-job fails to start messages will not be considered processed and will appear in the queue again after visibility timeout

Copy
resources:
  myLambda:
    type: function
    properties:
      packaging:
        type: stacktape-lambda-buildpack
        properties:
          entryfilePath: path/to/my-lambda.ts
      events:
        - type: sqs
          properties:
            sqsQueueName: mySqsQueue

  mySqsQueue:
    type: sqs-queue

Alternatively to event integrations, you can also receive messages using AWS SDK. See our worker-service example

Worker example

Copy
resources:
  myQueue:
    type: sqs-queue
    properties:
      fifoEnabled: true

  processing:
    type: worker-service
    properties:
      packaging:
        type: stacktape-image-buildpack
        properties:
          entryfilePath: processing.ts
      connectTo:
        - myQueue
      resources:
        cpu: 0.25
        memory: 512

Copy
import { DeleteMessageCommand, ReceiveMessageCommand, SQSClient } from '@aws-sdk/client-sqs';

const sqsClient = new SQSClient({});

const interval = setInterval(async () => {
  try {
    const { Messages } = await sqsClient.send(new ReceiveMessageCommand({ QueueUrl: process.env.STP_MY_QUEUE_URL }));
    if (Messages?.length) {
      // do some message processing
      // .....

      // After processing, delete messages from queue.
      // If you do NOT delete messages, they will become visible after visibility timeout elapses.
      // This can lead to same message being processed twice.
      await Promise.all(
        Messages.map(({ ReceiptHandle }) =>
          sqsClient.send(new DeleteMessageCommand({ QueueUrl: process.env.STP_MY_QUEUE_URL, ReceiptHandle }))
        )
      );
    }
    console.info('Nothing to process');
  } catch (err) {
    // error handling
    console.error(err);
    clearInterval(interval);
  }
}, 10000);

Sending messages

You can use AWS SDK to send messages to the queue.

Copy
import { SendMessageCommand, SQSClient } from "@aws-sdk/client-sqs";

const sqsClient = new SQSClient({});

sqsClient
  .send(new SendMessageCommand({ MessageBody: "Hello from the other side", QueueUrl: process.env.STP_MY_QUEUE_URL }))
  .catch((err) => {
    console.info(`Sending message failed: ${err}`);
  });

Enabling FIFO

You can enable FIFO(First in First Out) for your SQS queue, ensuring ordering of events.

Copy
resources:
  mainQueue:
    type: sqs-queue
    properties:
      fifoEnabled: true

Content based deduplication

Visibility timeout

You can set visibility timeout using visibilityTimeoutSeconds property.

Copy
resources:
  mainQueue:
    type: sqs-queue
    properties:
      visibilityTimeoutSeconds: 300