Splunk® App for AWS (Legacy)

Installation and Configuration Manual

Acrobat logo Download manual as PDF


On July 15, 2022, the Splunk App for AWS will reach its end of life (EOL). After this date, Splunk will no longer maintain or develop this product. Splunk App for AWS is used for both IT monitoring and security use cases because it provides dashboards for both ITOps and security teams. The IT monitoring functionality in Splunk App for AWS is migrating to a content pack in Data Integrations called the Content Pack for Amazon Web Services Dashboards and Reports. The security use case functionality in Splunk App for AWS is migrating to the new Splunk App for AWS Security Dashboards. For more about migration options, see this community post.
This documentation does not apply to the most recent version of Splunk® App for AWS (Legacy). For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

Sizing, performance, and cost considerations for the Splunk App for AWS

Before you set up the Splunk App for AWS and start ingesting data, review these guidelines for each input.

General

The following table provides a general guidance on the recommended maximum daily indexing volume for each typical AWS source type on a clustered indexer to achieve acceptable dashboard reporting performance. Use this as a rough guideline to plan for the number of indexers to deploy in your clustered enviornment. Adding more indexers to a cluster improves indexing and search retrival performance, but since this also incurs some additional within-cluster data replication traffic, adjust the number of indexers in your cluster based on your actual system performance.

Source Type Daily Indexing Volume per Indexer (GB)
aws:cloudwatchlogs:vpcflow 25 - 30
aws:s3:accesslogs 80 - 120
aws:cloudtrail 150 - 200
aws:billing 50 - 100

The sizing recommendations assume the following hardware configurations for the Splunk platform. You can also use the system requirements for use of Splunk Enterprise on-premises in the Splunk Enterprise Installation Manual as a reference.

Splunk Platform Type CPU Cores RAM EC2 Instance Type
Search Head 8 16 GB c4.xlarge
Indexer 16 64 GB m4.4xlarge

Input configuration screens require data transfer from AWS to populate the services, queues, and buckets available to your accounts. If your network to AWS is slow, these may take some time to load.

Refer to the release notes for a list of performance known issues.

CloudTrail

Consideration Special notes
Sizing and Performance None
AWS Cost Using CloudTrail itself does not incur charges, but standard S3, SNS, and SQS charges apply.
See http://aws.amazon.com/pricing/services/.

Config

Consideration Special notes
Sizing and Performance None
AWS Cost Using Config incurs charges from AWS. See http://aws.amazon.com/config/pricing/.
In addition, standard S3, SNS, and SQS charges apply. See http://aws.amazon.com/pricing/services/.

Config Rules

Consideration Special notes
Sizing and Performance None
AWS Cost None

CloudWatch

Consideration Special notes
Sizing and Performance The smaller the granularity you configure, the more events you collect.

Best practice: Configure a granularity that matches the precision that you require, setting a larger granularity value in cases when indexing fewer, less-granular events is acceptable. You can increase granularity temporarily when a problem is detected.

AWS rate-limits the number of free API calls against the CloudWatch API. In testing with a period of 300 and a polling interval or 1800, Splunk determined that collecting data for 2 million metrics does not, by itself, exceed the current default rate limit, but that collecting 4 million metrics does exceed it. If you have millions of metrics to collect in your environment, consider paying to have your API limit raised, or remove less-essential metrics from your input and configure larger granularities in order to make fewer API calls.
AWS Cost Using CloudWatch and making requests against the CloudWatch API incurs charges from AWS.
See http://aws.amazon.com/cloudwatch/pricing/

CloudWatch Logs

Consideration Special notes
Sizing and Performance AWS limits each account to 10 requests per second, each of which will return no more than 1 MB of data. This means the data ingestion and indexing rate will be no more than 10MB/s. The add-on modular input can process up to 4K events per second in a single log stream.
Best practices:
  • If volume is a concern, configure the only_after parameter, available in the Splunk Add-on for AWS's CloudWatch Logs input, to limit the amount of historical data you collect.
  • If you have high volume VPC Flow Logs, configure one or more Kinesis inputs to collect them instead of using the CloudWatch Logs input. You can configure Kinesis inputs directly in the Splunk Add-on for AWS. See Add a Kinesis input for the Splunk Add-on for AWS.
    For high volume VPC Flow Logs, it is also recommended that you deploy at least two search heads in a cluster. Note that you may want to reduce the summary range of the VPC Flow data model to accommodate for the higher data volume in your dashboards.
AWS Cost Using CloudWatch Logs incurs charges from AWS. See http://aws.amazon.com/cloudwatch/pricing/
Transferring data out of CloudWatch Logs incurs charges from AWS. See http://aws.amazon.com/ec2/pricing/

Amazon Inspector

Consideration Special notes
Sizing and Performance None
AWS Cost Using Amazon Inspector incurs charges from AWS. See http://aws.amazon.com/inspector/pricing/.

S3

Consideration Special notes
Sizing and Performance AWS throttles S3 data collection at the bucket level, so expect some delay before all data arrives in your Splunk platform.
Recommended memory size per indexer for high-volume S3 data is 64 GB or larger.
AWS Cost Using S3 incurs charges from AWS. See http://aws.amazon.com/s3/pricing/.

Billing

Consideration Special notes
Sizing and Performance None
AWS Cost Billing reports themselves do not incur charges, but standard S3 charges apply.
See http://aws.amazon.com/s3/pricing/.
Last modified on 01 December, 2016
PREVIOUS
Plan your deployment of the Splunk App for AWS
  NEXT
Configure your AWS services for the Splunk App for AWS

This documentation applies to the following versions of Splunk® App for AWS (Legacy): 4.2.0, 4.2.1, 5.0.0, 5.0.1, 5.0.2


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters