Splunk® Data Stream Processor

Getting Data In

On April 3, 2023, Splunk Data Stream Processor will reach its end of sale, and will reach its end of life on February 28, 2025. If you are an existing DSP customer, please reach out to your account team for more information.
This documentation does not apply to the most recent version of Splunk® Data Stream Processor. For documentation on the most recent version, go to the latest release.

Use the Amazon CloudWatch Metrics Connector with Splunk DSP

Use the Amazon CloudWatch Metrics Connector to collect performance data from Amazon CloudWatch. The Amazon CloudWatch Metrics Connector uses namespaces and regions to provide system-wide visibility into resource utilization, application performance, and operational health. Search for "Using Amazon CloudWatch Metrics" in the Amazon Web Services (AWS) documentation for more information about setting up and using Amazon CloudWatch Metrics.

To use the Amazon CloudWatch Metrics Connector, start by creating a connection that allows it to access data from Amazon CloudWatch. Then, add the Amazon CloudWatch Metrics Connector to the start of your data pipeline and configure it to use the connection that you created.

Behavior of the Amazon CloudWatch Metrics Connector

The first time a scheduled job runs, the connector collects metrics for only one time interval as defined by the Period parameter. For all following scheduled jobs, the connector collects metrics from the last execution up to the current time.

Create a connection using the Amazon CloudWatch Metrics Connector

Create a connection so that the Amazon CloudWatch Metrics Connector can access data from Amazon CloudWatch and send the data into a DSP pipeline.

If you are editing a connection that's being used by an active pipeline, you must reactivate that pipeline after making your changes.

Prerequisites

Before you can use the Amazon CloudWatch Metrics Connector, you must have an AWS account. If you don't have an AWS account, ask your AWS administrator to create an account and provide the access key ID and secret access key. Search for "Access Keys (Access Key ID and Secret Access Key)" in the AWS documentation for more information about access key credentials.

Make sure that your AWS account has the following permissions for Amazon CloudWatch Metrics:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:GetMetricData",
                "cloudwatch:GetMetricStatistics",
                "cloudwatch:ListMetrics"
            ],
            "Resource": "*"
        }
    ]
}

You have the option of filtering empty metrics from the AWS/EC2 namespace out of the results. Filtering out these empty metrics allows the connector to collect other metrics more quickly while making fewer AWS API calls. If you want to filter out empty metrics, make sure that your AWS account also has the following optional permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "1",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "autoscaling:DescribeAutoScalingGroups"
            ],
            "Resource": "*"
        }
    ]
}

Steps

  1. From the Data Management page, click the Connections tab.
  2. Click Create New Connection.
  3. Select Amazon CloudWatch Metrics Connector and then click Next.
  4. Complete the following fields:
    Field Description
    Connection Name A unique name for your Amazon CloudWatch connection.
    Access Key ID Your AWS access key ID.
    Secret Access Key Your AWS secret access key.
    Metrics A list of metric queries, which determine the namespaces and regions that the connector collects data from. For each metric query that you want to include, click Add Group and select the appropriate values from the following drop-down lists:
    • Namespaces: A list of namespaces that you want to collect data from.
    • Regions: A list of regions that you want to collect data from.
    • Statistics: A list of statistics that you want to collect. Each statistic represents an aggregation of metrics data collected over a specified period of time.
    Period The length of time (in seconds) associated with each Amazon CloudWatch statistic.
    Delay (Optional) The number of seconds to delay the metrics data collection, to compensate for latency in the availability of Amazon CloudWatch Metrics data points. The default is 300 seconds.
    Scheduled This parameter is on by default, indicating that jobs run automatically. Toggle this parameter off to stop the scheduled job from automatically running. Jobs that are currently running are not affected.
    Schedule The time-based job schedule that determines when the connector executes jobs for collecting data. Select a predefined value or write a custom CRON schedule. All CRON schedules are based on UTC.
    Workers The number of workers you want to use to collect data.

    If your data fails to get into DSP, check the fields again to make sure you have the correct name, AWS access key ID, and AWS secret access key for your Amazon CloudWatch Metrics connection. DSP doesn't run a check to see if you enter the valid credentials.

  5. Click Save.

You can now use your connection in a data pipeline.

Last modified on 23 October, 2020
Get data in with the Collect service and a pull-based connector   Use the Amazon Metadata Connector with Splunk DSP

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.1.0


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters