Splunk® Supported Add-ons

Splunk Add-on for AWS

Download manual as PDF

Download topic as PDF

Configure Kinesis inputs for the Splunk Add-on for AWS

Kinesis is the recommended input type for collecting VPC Flow Logs (source type: aws:cloudwatchlogs:vpcflow). This input type also supports the collection of custom data types through Kinesis streams.

Configure a Kinesis input for the Splunk Add-on for AWS on your data collection node through Splunk Web (best practice) or in local/aws_kinesis_tasks.conf. This data source is only available in a subset of AWS regions. See the AWS regional table for a full list of supported regions.

The Kinesis data input only supports gzip compression or plaintext data. It cannot ingest data with other encodings, nor can it ingest data with a mix of gzip and plaintext in the same input. Create separate Kinesis inputs for gzip data and plaintext data.

Configure a Kinesis input on the data collection node using one of the following ways:

Configure a Kinesis input using Splunk Web

To configure inputs in Splunk Web, click Splunk Add-on for AWS in the left navigation bar on Splunk Web home, then choose one of the following menu paths depending on which data type you want to collect:

  • Create New Input > VPC Flow Logs > Kinesis
  • Create New Input > Others > Kinesis

Fill out the fields as described in the following table:

Argument Corresponding Field in Splunk Web Description
account AWS Account The AWS account or EC2 IAM role the Splunk platform uses to access your Kinesis data. In Splunk Web, select an account from the drop-down list. In aws_kinesis_tasks.conf, enter the friendly name of one of the AWS accounts that you configured on the Configuration page or the name of the autodiscovered EC2 IAM role.
aws_iam_role Assume Role The IAM role to assume, see Manage IAM roles
region AWS Region The AWS region that contains the Kinesis streams. In aws_kinesis_tasks.conf, enter the region ID. See the AWS documentation for more information.
stream_names Stream Names Stream names in a comma-separated list. Each stream_name must be unique to each region. If a stream_name is the same for two different regions, it will fail. Leave blank to collect all streams.
encoding Encoding with The encoding of the stream data. Set to gzip or leave blank, which defaults to Base64. All stream data that you collect in a single input must have the same encoding. If you are collecting VPC Flow Logs data through this input, encoding is typically gzip.
init_stream_position Initial Stream Position LATEST or TRIM_HORIZON. LATEST starts data collection from the point the input is enabled. TRIM_HORIZON starts collecting with the oldest data record.
format Record Format CloudWatchLogs or none. If you choose CloudWatchLogs, this add-on will parse the data in CloudWatchLogs format.
sourcetype Source type A source type for the events.Enter aws:cloudwatchlogs:vpcflow if you are indexing VPC Flow Log data through Kinesis. Enter aws:kinesis if you are collecting any other Kinesis data.
index Index The index name where the Splunk platform puts the Kinesis data. The default is main.

Configure a Kinesis input using configuration file

To configure the input using the configuration files, create $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/aws_kinesis_tasks.conf using the following template.

account = <value>
region = <value>
stream_names = <value>
encoding = <value>
init_stream_position = <value>
format = <value> 
sourcetype = <value>
index = <value>

Here is an example stanza that collects Kinesis data for all streams available in the region.

account = splunkapp2
region = us-east-1
encoding =
init_stream_position = LATEST
index = aws
format = CloudWatchLogs
sourcetype = aws:kinesis
Last modified on 14 May, 2020
Configure Cost and Usage Report inputs for the Splunk Add-on for AWS
Configure SQS inputs for the Splunk Add-on for AWS

This documentation applies to the following versions of Splunk® Supported Add-ons: released, released

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters