Splunk® Supported Add-ons

Splunk Add-on for AWS

Download manual as PDF

Download topic as PDF

Sizing, performance, and cost considerations for the Splunk Add-on for AWS

Before you configure this add-on, review these sizing, performance, and cost considerations.

General

The following table provides a general guidance on the recommended maximum daily indexing volume for each typical AWS source type on a clustered indexer to achieve acceptable dashboard reporting performance. Use this as a rough guideline to plan for the number of indexers to deploy in your clustered enviornment. Adding more indexers to a cluster improves indexing and search retrival performance, but since this also incurs some additional within-cluster data replication traffic, adjust the number of indexers in your cluster based on your actual system performance.

Source Type Daily Indexing Volume per Indexer (GB)
aws:cloudwatchlogs:vpcflow 25 - 30
aws:s3:accesslogs 80 - 120
aws:cloudtrail 150 - 200
aws:billing 50 - 100

The sizing recommendations assume the following hardware configurations for the Splunk platform. You can also use the system requirements for use of Splunk Enterprise on-premises in the Splunk Enterprise Installation Manual as a reference.

Splunk Platform Type CPU Cores RAM EC2 Instance Type
Search Head 8 16 GB c4.xlarge
Indexer 16 64 GB m4.4xlarge

Input configuration screens require data transfer from AWS to populate the services, queues, and buckets available to your accounts. If your network to AWS is slow, these may take some time to load. If you encounter timeout issues, you can manually type in resource names.

For more information, see Performance Reference for the Splunk Add-on for AWS data inputs.

CloudTrail

Consideration Special notes
Sizing and Performance None
AWS Cost Using CloudTrail itself does not incur charges, but standard S3, SNS, and SQS charges apply.
See http://aws.amazon.com/pricing/services/.

Config

Consideration Special notes
Sizing and Performance None
AWS Cost Using Config incurs charges from AWS. See http://aws.amazon.com/config/pricing/.
In addition, standard S3, SNS, and SQS charges apply. See http://aws.amazon.com/pricing/services/.

Config Rules

Consideration Special notes
Sizing and Performance None
AWS Cost None

CloudWatch

Consideration Special notes
Sizing and Performance The smaller the granularity you configure, the more events you collect.
Best practice: Create separate inputs that match your needs for different regions, services, and metrics. For each input, configure a granularity that matches the precision that you require, setting a larger granularity value in cases where indexing fewer, less-granular events is acceptable. You can increase granularity temporarily when a problem is detected.

AWS rate-limits the number of free API calls against the CloudWatch API. In testing with a period of 300 and a polling interval or 1800, Splunk determined that collecting data for 2 million metrics does not, by itself, exceed the current default rate limit, but that collecting 4 million metrics does exceed it. If you have millions of metrics to collect in your environment, consider paying to have your API limit raised, or remove less-essential metrics from your input and configure larger granularities in order to make fewer API calls.

AWS Cost Using CloudWatch and making requests against the CloudWatch API incurs charges from AWS.
See http://aws.amazon.com/cloudwatch/pricing/

CloudWatch Logs (VPC Flow Logs)

Consideration Special notes
Sizing and Performance AWS limits each account to 10 requests per second, each of which will return no more than 1 MB of data. This means the data ingestion and indexing rate will be no more than 10MB/s. The add-on modular input can process up to 4K events per second in a single log stream.
Best practices:
  • If volume is a concern, configure the only_after parameter to limit the amount of historical data you collect.
  • If you have high volume VPC Flow Logs, configure one or more Kinesis inputs to collect them instead of using the CloudWatch Logs input.
AWS Cost Using CloudWatch Logs incurs charges from AWS. See http://aws.amazon.com/cloudwatch/pricing/
Transferring data out of CloudWatch Logs incurs charges from AWS. See http://aws.amazon.com/ec2/pricing/

Inspector

Consideration Special notes
Sizing and Performance None
AWS Cost Using Amazon Inspector incurs charges from AWS. See http://aws.amazon.com/inspector/pricing/.

Kinesis

Consideration Special notes
Sizing and Performance See Performance reference for the Kinesis input in the Splunk Add-on for AWS
AWS Cost Using Amazon Kinesis incurs charges from AWS. See https://aws.amazon.com/kinesis/streams/pricing/.

S3

Consideration Special notes
Sizing and Performance AWS throttles S3 data collection at the bucket level, so expect some delay before all data arrives in your Splunk platform.
You can configure multiple S3 inputs for a single S3 bucket to improve performance. The Splunk platform dedicates one process for each data input, so provided that your system has sufficient processing power, performance will improve with multiple inputs. See Performance reference for the S3 input in the Splunk Add-on for AWS.
AWS Cost Using S3 incurs charges from AWS. See http://aws.amazon.com/s3/pricing/.

Billing

Consideration Special notes
Sizing and Performance Detailed billing reports can be very large in size, depending on your environment. If you configure the add-on to collect detailed reports, it collects all historical reports available in the bucket by default. In addition, for each newly finalized monthly and detailed report, the add-on will collect new copies of the same report once per interval until the etag is unchanged.
Best practice: Configure separate inputs for each billing report type that you want to collect. Use the regex and interval parameters in the add-on's input configuration to limit the number of reports that you collect with each input.
AWS Cost Billing reports themselves do not incur charges, but standard S3 charges apply.
See http://aws.amazon.com/s3/pricing/.

SQS

Consideration Special notes
Sizing and Performance None
AWS Cost Using SQS incurs charges from AWS. https://aws.amazon.com/sqs/pricing/

SNS

Consideration Special notes
Sizing and Performance None
AWS Cost Using SNS incurs charges from AWS. https://aws.amazon.com/sns/pricing/
PREVIOUS
Hardware and software requirements for the Splunk Add-on for AWS
  NEXT
Configure AWS services for the Splunk Add-on for AWS

This documentation applies to the following versions of Splunk® Supported Add-ons: released


Comments

Hi Rvn,
The Splunk software uses the GetMetricStatistics API to collect Cloudwatch metrics.

Thanks,
Mike

Mglauser splunk, Splunker
November 12, 2018

For Cloudwatch metrics, does Splunk uses GetMetricData or GetMetricStatistics to collect Cloudwatch metrics?

Rvn
October 23, 2018

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters