Splunk® Supported Add-ons

Splunk Add-on for AWS

Download manual as PDF

Download topic as PDF

Configure Kinesis inputs for the Splunk Add-on for AWS

Kinesis is the recommended input type for collecting VPC Flow Logs (source type: aws:cloudwatchlogs:vpcflow). This input type also supports the collection of custom data types through Kinesis streams.

Configure a Kinesis input for the Splunk Add-on for AWS on your data collection node through Splunk Web (best practice) or in local/aws_kinesis_tasks.conf. This data source is only available in a subset of AWS regions. See the AWS documentation for a full list of supported regions: http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region.

The Kinesis data input only supports gzip compression or plaintext data. It cannot ingest data with other encodings, nor can it ingest data with a mix of gzip and plaintext in the same input. Create separate Kinesis inputs for gzip data and plaintext data.

Configure a Kinesis input on the data collection node using one of the following ways:

Configure a Kinesis input using Splunk Web

To configure inputs in Splunk Web, click Splunk Add-on for AWS in the left navigation bar on Splunk Web home, then choose one of the following menu paths depending on which data type you want to collect:

  • Create New Input > VPC Flow Logs > Kinesis
  • Create New Input > Others > Kinesis

Fill out the fields as described in the following table:

Argument Corresponding Field in Splunk Web Description
account AWS Account The AWS account or EC2 IAM role the Splunk platform uses to access your Kinesis data. In Splunk Web, select an account from the drop-down list. In aws_kinesis_tasks.conf, enter the friendly name of one of the AWS accounts that you configured on the Configuration page or the name of the autodiscovered EC2 IAM role.
aws_iam_role Assume Role The IAM role to assume, see Manage IAM roles
region AWS Region The AWS region that contains the Kinesis streams. In aws_kinesis_tasks.conf, enter the region ID. See the AWS documentation for more information.
stream_names Stream Names Stream names in a comma-separated list. Leave blank to collect all streams.
encoding Encoding with The encoding of the stream data. Set to gzip or leave blank, which defaults to Base64. All stream data that you collect in a single input must have the same encoding. If you are collecting VPC Flow Logs data through this input, encoding is typically gzip.
init_stream_position Initial Stream Position LATEST or TRIM_HORIZON. LATEST starts data collection from the point the input is enabled. TRIM_HORIZON starts collecting with the oldest data record.
format Record Format CloudWatchLogs or none. If you choose CloudWatchLogs, this add-on will parse the data in CloudWatchLogs format.
sourcetype Source type A source type for the events.Enter aws:cloudwatchlogs:vpcflow if you are indexing VPC Flow Log data through Kinesis. Enter aws:kinesis if you are collecting any other Kinesis data.
index Index The index name where the Splunk platform puts the Kinesis data. The default is main.

Configure a Kinesis input using configuration file

To configure the input using the configuration files, create $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/aws_kinesis_tasks.conf using the following template.

[<name>]
account = <value>
aws_iam_role=<value>
region = <value>
stream_names = <value>
encoding = <value>
init_stream_position = <value>
format = <value> 
sourcetype = <value>
index = <value>

Here is an example stanza that collects Kinesis data for all streams available in the region.

[splunkapp2:us-east-1]
account = splunkapp2
region = us-east-1
encoding =
init_stream_position = LATEST
index = aws
format = CloudWatchLogs
sourcetype = aws:kinesis
PREVIOUS
Configure Cost and Usage Report inputs for the Splunk Add-on for AWS
  NEXT
Configure SQS inputs for the Splunk Add-on for AWS

This documentation applies to the following versions of Splunk® Supported Add-ons: released


Comments

The soft limit to the number of Kinesis streams that can be displayed in the drop-down list in the UI is a known issue that will be resolved in the 4.4.0 release of the Splunk Add-on for AWS. For now, you can manually update the stream_names list in the aws_kinesis_tasks.conf to workaround this.

Hunters splunk, Splunker
August 22, 2017

Hi Vsingla1. The Kinesis stream names are held in the stream_names setting in aws_kinesis.tasks.conf.
stream_names = Kinesis stream names in a comma-separated list. Leave empty to collect all streams
You can update the list directly in the .conf file for your streams to be processes by the add-on. Hope it helps. Thanks!

Hunters splunk, Splunker
August 22, 2017

For Kinesis input, not all the stream names are visible in the "Stream Name" Dropdown.
The dropdown is sorted in Alphabetical order and only top 20 streams show up. e.g I have a kinesis stream starting with "X" but it does not show up. I have checked and the permissions are fine. I am sure there is a file that has a default limit of 20 set for the dropdown. Do you know which file is it and where to change this number?

Vsingla1
August 22, 2017

Is that correct on the 10 minutes question? Does it take 10 minutes for Splunk to send data to Kinesis? If not, what is the interval?

Bnelson5
February 8, 2017

Hi Csabatini, there is no interval setting for the Kinesis input.

Hunters splunk, Splunker
December 15, 2016

Is there an interval on the Kinesis collector? I thought I read 10 minutes somewhere? Is it configurable?

Csabatini
December 7, 2016

Hi Dave
For the Kinesis input, if you want to configure the aws account via conf file, the stanza is ‘account’, not ‘aws_account’.

Rwang splunk, Splunker
October 26, 2016

There is some inconsistency with the argument account/aws_account. In the first figure, that shows the stanza format, it uses account
[<name>]
account = <value>

In the argument table and the example stanza it uses aws_account

[splunkapp2:us-east-1]
aws_account = splunkapp2

Which is correct?

Dave

Davebrooking
October 20, 2016

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters