Log Forwarding

Introduction

Log forwarding is the process of transmitting log data generated by various applications, systems, and services to a centralized destination or third-party log management provider. In cloud environments, like Amazon Web Services (AWS), the primary log management service is Amazon CloudWatch. However, people often choose to forward logs from CloudWatch to third-party providers for several reasons:

third-party log management platforms offer advanced features and functionalities beyond what CloudWatch provides, such as real-time log analysis, machine learning-powered insights, and customizable dashboards
some organizations have specific compliance requirements or industry standards that necessitate the use of dedicated log management providers.

How it works

Stacktape allows for easy log forwarding to endpoint of your choice.

Log forwarding is done using Amazon Kinesis Data Firehose delivery stream.
When using log forwarding, you will incur costs based on the amount of data being transferred to the destination (~$0.03 per transferred GB). Refer to AWS Kinesis Firehose Pricing page to see details.
Currently supported destinations for logs:
- http-endpoint
  - delivers logs to any HTTP endpoint.
  - The endpoint must follow Firehose request and response specifications. (Many of the third party vendors are compliant with this specifications out of the box.)
- datadog
  - delivers logs to Datadog.
- highlight
  - delivers logs to Highlight.io project.

Refer to our docs for more information.

Logs that fail to be delivered to the destination even after multiple retries (time spend on retries can be configured) are put into bucket with name {stackName}-{resourceName}-logs-{generatedHash}

HTTP endpoint forwarding

You can forward logs to any HTTP endpoint that follows Firehose request and response specifications.

Many of the third party vendors are compliant with this specifications out of the box.


Copy
resources:
  myFunction:
    type: function
    properties:
      logging:
        logForwarding:
          type: http-endpoint
          properties:
            endpointUrl: https://my-endpoint.com
      packaging:
        type: stacktape-lambda-buildpack
        properties:
          entryfilePath: lambdas/js-lambda.js

HttpEndpointLogForwarding API reference

type

Required

properties.endpointUrl

Required

properties.gzipEncodingEnabled

properties.parameters

properties.retryDuration

properties.accessKey

Datadog forwarding

Delivers logs to your Datadog site.


Copy
resources:
  myFunction:
    type: function
    properties:
      logging:
        logForwarding:
          type: datadog
          properties:
            apiKey: your_datadog_api_key
      packaging:
        type: stacktape-lambda-buildpack
        properties:
          entryfilePath: lambdas/js-lambda.js

DatadogLogForwarding API reference

type

Required

properties.apiKey

Required

properties.endpointUrl

Default: https://aws-kinesis-http-intake.logs.datadoghq.com/v1/input

Highlight forwarding

Delivers logs to your Highlight project.


Copy
resources:
  myFunction:
    type: function
    properties:
      logging:
        logForwarding:
          type: datadog
          properties:
            apiKey: your_datadog_api_key
      packaging:
        type: stacktape-lambda-buildpack
        properties:
          entryfilePath: lambdas/js-lambda.js

HighlightLogForwarding API reference

type

Required

properties.projectId

Required

properties.endpointUrl

Default: https://pub.highlight.io/v1/logs/firehose