Splunk® Supported Add-ons

Splunk Add-on for AWS

Download manual as PDF

Download topic as PDF

Configure inputs for the Splunk Add-on for AWS

Input configuration overview

You can use the Splunk Add-on for AWS to collect many types of useful data from AWS to gain valuable insight of your cloud infrastructure. For each supported data type, one or more input types are provided for data collection.

Follow these steps to plan and perform your AWS input configuration:

  1. Read the Supported data types and corresponding AWS input types section to view a list of supported data types and use the matrix to determine which input type to configure for your data collection.
  2. View the details of the input type you plan to configure under AWS input types. From there, click the input type link to go to the input configuration details.
  3. Follow the steps described in the input configuration details to complete the configuration.

Supported data types and corresponding AWS input types

The following matrix lists all the data types that can be collected using the Splunk Add-on for AWS and the corresponding input types that you can configure to collect this data.

For some data types, the Splunk Add-on for AWS provides you with the flexibility to choose from multiple input types based on specific requirements (for example, collect historical logs as opposed to only collect newly created logs). Whenever applicable, SQS-based S3 is the recommended input type to use for all of its collectible data types.

S = Supported, R = Recommended (when there are multiple supported input types)

Data Type Source type SQS-based S3 Generic S3 Incremental S3 AWS Config Config Rules Inspector Cloud-
Trail
Cloud-
Watch
Cloud-
Watch Logs
Description Billing Kinesis SQS
Billing aws:
billing
S
Cloud-
Watch
aws:
cloudwatch
S
CloudFront Access Logs aws:
cloudfront:
accesslogs
R S S
Config aws:
config, aws:
config:
notification
R S
Config Rules aws:
config:
rule
S
Description aws:
description
S
ELB Access Logs aws:elb:
accesslogs
R S S
Inspector aws:
inspector
S
Cloud-
Trail
aws:
cloudtrail
R S S S
S3 Access Logs aws:s3:
accesslogs
R S S
VPC Flow Logs aws:
cloudwatchlogs:
vpcflow
S R
SQS aws:sqs S
Others custom source types S S S S S

AWS input types

The Splunk Add-on for AWS provides two categories of input types to gather useful data from your AWS environment:

  • Dedicated (single-purpose) input types: designed to ingest one specific data type
  • Multi-purpose input types: collect multiple data types from the S3 bucket

Some data types can be ingested using either a dedicated input type or a multi-purpose input type. For example, CloudTrail logs can be collected using any of the following input types: CloudTrail, S3, or SQS-based S3. The SQS-based S3 input type is the recommended option because it is more scalable and provides higher ingestion performance.

Dedicated input types

To ingest a specific type of logs, configure the corresponding dedicated input designed to collect the log type. Click the input type name in the table below for instructions on how to configure it.

Input Description
AWS Config Configuration snapshots, historical configuration data, and change notifications from the AWS Config service.
Config Rules Compliance details, compliance summary, and evaluation status of your AWS Config Rules.
Inspector Assessment Runs and Findings data from the Amazon Inspector service.
CloudTrail AWS API call history from the AWS CloudTrail service.
CloudWatch Logs Logs from the CloudWatch Logs service, including VPC Flow Logs. VPC Flow Logs allow you to capture IP traffic flow data for the network interfaces in your resources.
CloudWatch Performance and billing metrics from the AWS CloudWatch service.
Description Metadata about your AWS environment.
Billing Billing data from the billing reports that you collect in the Billing & Cost Management console.
Kinesis Data from your Kinesis streams.
Note: It is a best practice to collect VPC flow logs and CloudWatch logs through Kinesis streams. However, the AWS Kinesis input currently has the following limitations:
  • Multiple inputs collecting data from a single stream cause duplicate events in Splunk.
  • Does not support monitoring of dynamic shards repartition, which means when there is a shard split or merge, the add-on cannot automatically discover and collect data in the new shards until it is restarted. After you repartition shards, you must restart Splunk to collect data from the partitions.
SQS Data from your AWS SQS.

Multi-purpose input types

Configure multi-purpose inputs to ingest supported log types.

Splunk recommends that you use the SQS-based input type to collect its supported log types. If you are already collecting logs using generic S3 inputs, you can still create SQS-based inputs and migrate your existing generic S3 inputs to the new inputs. For detailed migration steps, see Migrate from the S3 input to the SQS-based input in this manual.

If the log types you want to collect are not supported by the SQS-based input type, use the generic S3 input type instead.

Read the multi-purpose input types comparison table to view the differences between the multi-purpose S3 collection input types.

Click the input type name in the table below for instructions on how to configure it.

Input Description
SQS-based S3 (recommended) A more scalable and higher-performing alternative to the generic and incremental S3 inputs, the SQS-based S3 input polls messages from SQS that subscribes to SNS notification events from AWS services and collects the corresponding log files - generic log data, CloudTrail API call history, Config logs, and access logs - from your S3 buckets in real time.
Unlike the other S3 input types, the SQS-based S3 input type takes advantage of the SQS visibility timeout setting and enables you to configure multiple inputs to scale out data collection from the same folder in an S3 bucket without ingesting duplicate data. Also, the SQS-based S3 input automatically switches to multipart, in-parallel transfers when a file is over a specific size threshold, thus preventing timeout errors caused by large file size.
Generic S3 General-purpose input type that can collect any log type from S3 buckets: CloudTrail API call history, access logs, and even custom non-AWS logs.
The generic S3 input lists all the objects in the bucket and examines each file's modified date every time it runs to pull uncollected data from an S3 bucket. When the number of objects in a bucket is large, this can be a very time-consuming process with low throughput.
Incremental S3 The incremental S3 input type collects four AWS service log types.
There are four types of logs you can collect using the Incremental S3 input:
  • CloudTrail Logs: This add-on will search for the cloudtrail logs under <bucket_name>/<log_file_prefix>/AWSLogs/<Account ID>/CloudTrail/<Region ID>/<YYYY/MM/DD>/<file_name>.json.gz.
  • ELB Access Logs: This add-on will search the elb access logs under <bucket_name>/<log_file_prefix>/AWSLogs/<Account ID>/elasticloadbalancing/<Region ID>/<YYYY/MM/DD>/<file_name>.log.gz.
  • S3 Access Logs: This add-on will search the S3 access logs under <bucket_name>/<log_file_prefix><YYYY-mm-DD-HH-MM-SS><UniqueString>.
  • CloudFront Access Logs: This add-on will search the cloudfront access logs under <bucket_name>/<log_file_prefix><distributionID><YYYY/MM/DD>.<UniqueID>.gz

The incremental S3 input only lists and retrieves objects that have not been ingested from a bucket by comparing datetime information included in filenames against checkpoint record, which significantly improves ingestion performance.

Multi-purpose input types comparison table

Generic S3 Incremental S3 SQS-based S3 (recommended)
Supported log types Any log type, including non-AWS custom logs 4 AWS services log types: CloudTrail logs, S3 access logs, CloudFront access logs, ELB access logs 5 AWS services log types (Config logs, CloudTrail logs, S3 access logs, CloudFront access logs, ELB access logs), as well as non-AWS custom logs
Data collection method Lists all objects in the bucket and compares modified date against the checkpoint Directly retrieves AWS log files whose filenames are distinguished by datetime Decodes SQS messages and ingests corresponding logs from the S3 bucket
Ingestion performance Low High High
Can ingest historical logs (logs generated in the past)? Yes Yes No
Scalable? No No Yes
You can scale out data collection by configuring multiple inputs to ingest logs from the same S3 bucket without creating duplicate data
Fault-tolerant? No
Each generic S3 input is a single point of failure.
No
Each incremental S3 input is a single point of failure.
Yes
Takes advantage of the SQS visibility timeout setting. Any SQS message not successfully processed in time by the SQS-based S3 input will reappear in the queue and will be retrieved and processed again.
In addition, data collection can be horizontally scaled out so that if one SQS-based S3 input fails, other inputs can still continue to pick up messages from the SQS queue and ingest corresponding data from the S3 bucket.
PREVIOUS
Manage accounts for the Splunk Add-on for AWS
  NEXT
Configure Config inputs for the Splunk Add-on for AWS

This documentation applies to the following versions of Splunk® Supported Add-ons: released


Comments

Hi Benstewartuk,
You might be running into this issue because you are collecting data around the application load balancer, which version 4.3.0 of the Splunk Add-on for AWS does not support yet and the current app does not support panels for this type of load balancer. Howerver, the upcoming release of AWS add-on 4.4.0 and AWS app 5.1.0 will support both classic load balancer and application load balancer. Please stay tuned. Thanks!

Hunters splunk, Splunker
August 24, 2017

We're using v4.3.0 of the AWS add on, we have a data input configured for the 'Description' option which seems to work and pulls in data but we get the following error on most pages:
Some panels may not be displayed correctly because the following inputs have not been configured: Metadata - ELB
Is this a known issue or is there something we need to do on AWS to enable ELB descriptors?

Benstewartuk
August 24, 2017

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters