Splunk® Supported Add-ons

Splunk Add-on for AWS

Download manual as PDF

Download topic as PDF

Configure CloudWatch Log inputs for the Splunk Add-on for AWS

Splunk strongly recommends against using the CloudWatch Logs inputs to collect VPC Flow Logs data (source type: aws:cloudwatchlogs:vpcflow) since the input type will be deprecated in upcoming releases. Configure Kinesis inputs to collect VPC Flow Logs instead. The add-on includes index-time logic to perform the correct knowledge extraction for these events through the Kinesis input as well.

Configure a CloudWatch Logs input for the Splunk Add-on for Amazon Web Services on your data collection node through Splunk Web (recommended), or in local/aws_cloudwatch_logs_tasks.conf.

Configure a CloudWatch Logs input using Splunk Web

To configure inputs using Splunk Web, click Splunk Add-on for AWS in the left navigation bar on Splunk Web home, then choose one of the following menu paths depending on the data type you want to collect:

  • Create New Input > VPC Flow Logs > CloudWatch Logs.
  • Create New Input > Others > CloudWatch Logs.

Fill out the fields as described in the table:

Argument in configuration file Field in Splunk Web Description
account AWS Account The AWS account or EC2 IAM role the Splunk platform uses to access your CloudWatch Logs data. In Splunk Web, select an account from the drop-down list. In aws_cloudwatch_logs_tasks.conf, enter the friendly name of one of the AWS accounts that you configured on the Configuration page or the name of the autodiscovered EC2 IAM role.
region AWS Region The AWS region that contains the data. In aws_cloudwatch_logs_tasks.conf, enter the region ID.
groups Log group A comma-separated list of log group names.

Note: Do not use wildcards.

only_after Only After GMT time string in '%Y-%m-%dT%H:%M:%S' format. If set, only events after this time are queried and indexed. Defaults to 1970-01-01T00:00:00.
stream_matcher Stream Matching Regex REGEX to strictly match stream names. Defaults to .*
interval Interval The number of seconds to wait before the Splunk platform runs the command again. Default is 600 seconds.
sourcetype Source type A source type for the events. Enter aws:cloudwatchlogs:vpcflow if you are indexing VPC Flow Log data. Enter aws:cloudwatchlogs if you are collecting any other CloudWatch Logs data.
index Index The index name where the Splunk platform puts the CloudWatch Logs data. The default is main.

Configure a CloudWatch Logs input using configuration file

To configure the input using configuration file, create $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/aws_cloudwatch_logs_tasks.conf using the following template.

[<name>]
account = <value>
groups = <value>
index = <value>
interval = <value>
only_after = <value>
region = <value>
sourcetype = <value>
stream_matcher = <value>

Here is an example stanza that collects VPC Flow Log data from two log groups.

[splunkapp2:us-west-2]
account = splunkapp2
groups = SomeName/DefaultLogGroup, SomeOtherName/SomeOtherLogGroup
index = default
interval = 600
only_after = 1970-01-01T00:00:00
region = us-west-2
sourcetype = aws:cloudwatchlogs:vpcflow
stream_matcher = eni.*
PREVIOUS
Configure CloudTrail inputs for the Splunk Add-on for AWS
  NEXT
Configure CloudWatch inputs for the Splunk Add-on for AWS

This documentation applies to the following versions of Splunk® Supported Add-ons: released


Comments

Hi Sutthiphong2005,
In the Splunk Add-on for AWS, the "Assume Role" field in the Configuration menu lets you manage AWS IAM roles that can be assumed by IAM accounts for the Splunk Add-on for AWS to access the following AWS resources: Generic S3, Incremental S3, SQS-Based S3, Billing, Description, CloudWatch, Kinesis.
In terms of creating and configuring the IAM roles on the AWS side, see the "IAM Roles" section of the "AWS Identity and Access Management" manual in the AWS documentation. Hopefully this helps, and please feel free to reach out via the "Was this topic useful" tab of this manual.

Mglauser splunk, Splunker
October 22, 2018

how can I 'assume role' in menu Cloudwatch Log Input?. If not, how can I pull cloudwatch log if I would like to use 'Assume Role'.

Sutthiphong2005
October 21, 2018

When will Kinesis Inputs be support for the AWS GovCloud Region (us-gov-west-1)? I currently get an error that it is not supported when I attempt to add the input.

Keithrhea
January 23, 2018

Hi Badrinath,
We plan to deprecate the CloudWatch Logs input because AWS doesn't intend that interface for high throughput, and it throttles quickly.
Thanks!

Hunters splunk, Splunker
April 9, 2017

Hi Team

We are consuming data from many custom cloudwatch log groups created for our applications, could you please let us know why this feature would not be supported in future releases and what is the alternate option we have apart from Kinesis Stream.

Badrinath dash
April 2, 2017

It’s the CloudWatch Logs input that will be deprecated in future releases. The doc has been updated to avoid any ambiguity regarding this.

Hunters splunk, Splunker
March 8, 2017

Hi,
I want to confirm this.. There are 2 inputs in this add-on - 'Cloudwatch' and 'Cloudwatch Logs'.
Which of these 2 input will be deprecated in future releases specifically? Will it be 'Cloudwatch' or 'Cloudwatch Logs'?

Vsingla1
March 3, 2017

Dear Vineet, the Delay parameter is not exposed on the configuration UI, but you can view its description in this file: $SPLUNK_HOME/etc/apps/Splunk_TA_aws/README/aws_cloudwatch_logs_tasks.conf.spec
Again, it is not advisable to use the CloudWatch Log input since Splunk will no longer support it in future AWS releases.
Thanks!

Hunters splunk, Splunker
December 22, 2016

Hi Hunter,
Thanks for the information.
I have another question. What is the purpose for "delay" field? There is not description about this parameter so any information will be useful.

Vsingla1
December 21, 2016

Dear Vineet, please note the following:
Warning: Splunk strongly recommends against using this modular input since it will be deprecated in upcoming releases.
Currently, log stream wildcard is not supported, but wildcard is supported in Cloud Watch metrics and dimensions.
Thanks!

Hunters splunk, Splunker
December 20, 2016

I see new version 4.1.2 has been released for this app on Nov 19. I upgraded my installation from 4.1.1 to 4.1.2. But still the wildcard for log groups is not accepted. It is real inconvenience to manually add log groups, especially when we have lot of groups to capture data from. Do you have any idea when will it be fixed?

Vsingla1
December 20, 2016

Hi there, yes you are right that the delay field cannot be changed via gui. it can be changed by editing the config file manually. My problem is I cannot find any information on what the field actually does. I am having trouble with cloudwatch logs data turning up late and I was looking at the delay field as a possible cause.

Williamholder
November 27, 2016

Thanks for your feedback, Bill! However, seems there is no Delay field in the corresponding UI. Could you please recheck? Thanks!

Hunters splunk, Splunker
November 25, 2016

Any information on what the delay field is for, and how can I change it?

Williamholder
November 13, 2016

Hi Michael
Wildcards for log group names are not supported in this release (4.1.1), but we plan to add this feature for the next release. I recommend that you can follow this add-on in Splunkbase https://splunkbase.splunk.com/app/1876/. You will get notice as soon as the new version released.
Actually, if you have large number of VPC flow logs, I recommend you configure them through the Kinesis input, Kinesis has better performance than Cloudwatch logs.

Rwang splunk, Splunker
October 25, 2016

Hello, we recently experienced an issue where enabling the Cloudwatch Logs input caused our daily indexing to jump from ~2G a day to nearly 30G a day.

The issue appears to be with VPC Flow Logs the default expiration of data is 6 months. Changing this to 1 day fixed our out of control daily indexing issue.

Oktaops
October 19, 2016

Currently, Wild cards for log group names are not accepted. I have tried regex as well as "*". This makes it extremely inconvenient since we have many log group names that appear and disappear a lot. Allowing wild cards would be extremely helpful.

Michael.kushma
October 12, 2016

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters