Splunk® Supported Add-ons

Splunk Add-on for AWS

Download manual as PDF

Download topic as PDF

Configure CloudTrail inputs for the Splunk Add-on for AWS

The CloudTrail input type supports the collection of CloudTrail data (source type: aws:cloudtrail). However, it is highly recommended that you configure SQS-based S3 inputs to collect this type of data.

Before you begin configuring your CloudTrail inputs, be aware of the following behaviors:

  • Create a single enabled CloudTrail modular input for each unique SQS > SNS > S3 bucket path. Multiple enabled modular inputs can cause conflicts when trying to delete SQS messages or S3 records that another modular input is attempting to access and parse. Be sure to disable or delete testing configs before going to production.
  • If you have multiple AWS regions from which you want to gather CloudTrail data, the Amazon Web Services best practice is that you can configure a trail that applies to all regions in the AWS partition in which you are working. You then set up one CloudTrail input to collect data from the centralized S3 bucket where log files from all the regions are stored.

Configure a CloudTrail input on the data collection node using one of the following ways:

Configure a CloudTrail input using Splunk Web

To configure inputs in Splunk Web, click on Splunk Add-on for AWS in the left navigation bar on Splunk Web home, then click Create New Input > CloudTrail. Fill out the fields as described in the table:

Argument in configuration file Field in Splunk Web Description
aws_account AWS Account The AWS account or EC2 IAM role the Splunk platform uses to access your CloudTrail data. In Splunk Web, select an account from the drop-down list. In inputs.conf, enter the friendly name of one of the AWS accounts that you configured on the Configuration page or the name of the autodiscovered EC2 IAM role.
aws_region AWS Region The AWS region that contains the log notification SQS queue. In inputs.conf, enter the region ID. See the AWS documentation for more information.
sqs_queue SQS queue name The name of the queue to which AWS sends new CloudTrail log notifications. In Splunk Web, you can select a queue from the drop-down list, if your account permissions allow you to list queues, or enter the queue name manually. The queue name is the final segment of the full queue URL. For example, if your SQS queue URL is http://sqs.us-east-1.amazonaws.com/123456789012/testQueue, then your SQS queue name is testQueue.
remove_files_when_done Remove logs when done A boolean value indicating whether the Splunk platform should delete log files from the S3 bucket after indexing is complete. Default is false.
exclude_describe_events Exclude events A boolean value indicating whether or not to exclude certain events, such as read-only events that can produce a high volume of data. Default is true.
blacklist Deny list for exclusion A PCRE regular expression that specifies event names to if exclude_describe_events is set to true. Leave blank to use the default regex, ^(?:Describe|List|Get).
excluded_events_index Excluded events index The name of the index in which the Splunk platform should put excluded events. Default is empty, which discards the events.
interval Interval The number of seconds to wait before the Splunk platform runs the command again. Default is 30 seconds.
log_partitions n/a Configure partitions of a log file to be ingested. This add-on will search the log files for <Region ID> and <Account ID>. For example, log_partitions = AWSLogs/<Account ID>/CloudTrail/<Region>.
sourcetype Source type A source type for the events. Enter a value only if you want to override the default of aws:cloudtrail. Event extraction relies on the default value of source type. If you change the default value, you must update props.conf as well.
index Index The index name where the Splunk platform puts the CloudTrail data. The default is main.

Configure a CloudTrail input using configuration file

To configure inputs manually in inputs.conf, create a stanza using the following template and add it to $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/inputs.conf. If the file or path does not exist, create it.

aws_account = <value>
aws_region = <value>
sqs_queue = <value>
exclude_describe_events = <value>
remove_files_when_done = <value>
blacklist = <value>
excluded_events_index = <value>
interval = <value>
sourcetype = <value>
index = <value>

Some of these settings have default values that can be found in $SPLUNK_HOME/etc/apps/Splunk_TA_aws/default/inputs.conf:

aws_account =
sourcetype = aws:cloudtrail
exclude_describe_events = true
remove_files_when_done = false
queueSize = 128KB
persistentQueueSize = 24MB
interval = 30

The values in default/inputs.conf correspond to the default values in Splunk Web as well as some internal values that are not exposed in Splunk Web for configuration. If you choose to copy this stanza to /local and use it as a starting point to configure your inputs.conf manually, change the stanza title from aws_cloudtrail to aws_cloudtrail://<name>.

Switch from a CloudTrail input to an SQS-based S3 input

The SQS-based S3 input is a more fault-tolerant and higher-performing alternative to the CloudTrail input for collecting CloudTrail data. If you are already collecting CloudTrail data using a CloudTrail input, you can configure an SQS-based S3 input and seamlessly switch to the new input for CloudTrail data collection with little disruption.

  1. Disable the CloudTrail input you are using to collect CloudTrail data.
  2. Set up a dead-letter queue and the SQS visibility timeout setting for the SQS queue from which you are collecting CloudTrail data. See Configure SQS.
  3. Create an SQS-based S3 input, pointing to the SQS Queue you configured in the last step. Refer to Configure SQS-based S3 inputs for the Splunk Add-on for AWS for the detailed configuration steps.

Once configured, the new SQS-based S3 input will replace the old CloudTrail input to collect CloudTrail data from the same SQS queue.

Last modified on 07 July, 2020
Configure Inspector inputs for the Splunk Add-on for AWS
Configure CloudWatch Log inputs for the Splunk Add-on for AWS

This documentation applies to the following versions of Splunk® Supported Add-ons: released

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters