Splunk® Supported Add-ons

Splunk Add-on for AWS

Download manual as PDF

Download topic as PDF

Configure CloudTrail inputs for the Splunk Add-on for AWS

Complete the steps to configure CloudTrail inputs for the Splunk Add-on for Amazon Web Services (AWS):

  1. You must manage accounts for the add-on as a prerequisite. See Manage accounts for the Splunk Add-on for AWS.
  2. Configure AWS services for the CloudTrail input.
  3. Configure AWS permissions for the CloudTrail input. You can skip this step and configure AWS permissions at once, if you prefer. See Configure AWS permissions for all Splunk Add-on for AWS inputs at once.
  4. Configure CloudTrail inputs either through Splunk Web or configuration files.

The CloudTrail input type supports the collection of CloudTrail data (source type: aws:cloudtrail). However, you might want to configure SQS-based S3 inputs to collect this type of data. See [http://docs.splunk.com/Documentation/AddOns/released/AWS/SQS-basedS3 Configure SQS-based S3 inputs for the Splunk Add-on for AWS]

Before you begin configuring your CloudTrail inputs, be aware of the following behaviors:

  • Create a single enabled CloudTrail modular input for each unique Simple Queue Service (SQS) > Simple Notification Service (SNS) > S3 bucket path. Multiple enabled modular inputs can cause conflicts when trying to delete SQS messages or S3 records that another modular input is attempting to access and parse. Be sure to disable or delete testing configurations before going to production.
  • If you have multiple AWS regions from which you want to gather CloudTrail data, the Amazon Web Services best practice is that you configure a trail that applies to all regions in the AWS partition in which you are working. You can then set up one CloudTrail input to collect data from the centralized S3 bucket where log files from all the regions are stored.

Configure AWS services for the CloudTrail input

The Splunk Add-on for AWS collects events from an SQS that subscribes to the SNS notification events from CloudTrail. Configure CloudTrail to produce these notifications, then create an SQS in each region for the add-on to access them.

Although AWS offers global trails, or one CloudTrail configuration in one region to collect trail data from all regions, SQS messages do not arrive as expected in this case. Either configure separate CloudTrail S3 > SNS > SQS paths for each region to ensure that you capture all your data or, if you want to configure a global CloudTrail, skip steps 3 through 6 in the following steps and instead configure the add-on to collect data from that S3 bucket directly.

  1. Enable CloudTrail. Follow the instructions in the AWS documentation. See http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html.
  2. Create an S3 bucket in which to store the CloudTrail events. Follow the AWS documentation to ensure the permissions for this bucket are correct. See http://docs.aws.amazon.com/awscloudtrail/latest/userguide/create-s3-bucket-policy-for-cloudtrail.html.
  3. Enable SNS notifications. See: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/getting_notifications_top_level.html.
  4. Create a new SQS.
  5. If you are in the China region, explicitly grant DeleteMessage and SendMessage permissions to the SQS that you just created. This step is not necessary in commercial regions.
  6. Subscribe the SQS to the SNS notifications that you enabled in step 3.
  7. Grant IAM permissions to access the AWS account that the add-on uses to connect to your AWS environment. See Manage accounts for the Splunk Add-on for AWS for details.

Configure AWS permissions for the CloudTrail input

Required permissions for the S3 bucket that collects your CloudTrail logs:

  • Get*
  • List*
  • Delete*

Granting the delete permission is required to support the option to remove log files when done collecting them with the add-on. If you set this parameter to false, you do not need to grant delete permissions.

Required permissions for the SQS subscribed to the S3 bucket that collects CloudTrail logs:

  • GetQueueAttributes
  • ListQueues
  • ReceiveMessage
  • GetQueueUrl
  • DeleteMessage

In the Resource section of the policy, specify the ARNs of the S3 buckets and SQS queues from which you want to collect data.

See the following sample inline policy to configure CloudTrail input permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sqs:GetQueueAttributes",
                "sqs:ListQueues",
                "sqs:ReceiveMessage",
                "sqs:GetQueueUrl",
                "sqs:DeleteMessage",
                "s3:Get*",
                "s3:List*",
                "s3:Delete*"
            ],
            "Resource": [
               "*"
            ]
        }
    ]
}

For more information and sample policies, see these resources in the AWS documentation:


Configure a CloudTrail input using Splunk Web

To configure inputs in Splunk Web:

  1. Click on Splunk Add-on for AWS in the navigation bar on Splunk Web home.
  2. Click Create New Input > CloudTrail.
  3. Use the following table to complete the fields for the new input in the .conf file or in Splunk Web:
Argument in configuration file Field in Splunk Web Description
aws_account AWS Account The AWS account or EC2 IAM role the Splunk platform uses to access your CloudTrail data. In Splunk Web, select an account from the drop-down list. In inputs.conf, enter the friendly name of one of the AWS accounts that you configured on the Configuration page or the name of the automatically discovered EC2 IAM role.
aws_region AWS Region The AWS region that contains the log notification SQS queue. In inputs.conf, enter the region ID. See http://docs.aws.amazon.com/general/latest/gr/rande.html#d0e371.
sqs_queue SQS queue name The name of the queue to which AWS sends new CloudTrail log notifications. In Splunk Web, you can select a queue from the drop-down list, if your account permissions allow you to list queues, or enter the queue name manually. The queue name is the final segment of the full queue URL. For example, if your SQS queue URL is http://sqs.us-east-1.amazonaws.com/123456789012/testQueue, then your SQS queue name is testQueue.
remove_files_when_done Remove logs when done A Boolean value indicating whether the Splunk platform should delete log files from the S3 bucket after indexing is complete. The default is false.
exclude_describe_events Exclude events A Boolean value indicating whether or not to exclude certain events, such as read-only events that can produce a high volume of data. The default is true.
blacklist Deny list for exclusion A PCRE regular expression that specifies event names to exclude if exclude_describe_events is set to true. Leave blank to use the default regex ^(?:Describe|List|Get).
excluded_events_index Excluded events index The name of the index in which the Splunk platform puts excluded events. The default is empty, which discards the events.
interval Interval The number of seconds to wait before the Splunk platform runs the command again. The default is 30 seconds.
log_partitions n/a Configure partitions of a log file to be ingested. This add-on searches the log files for <Region ID> and <Account ID>. For example, log_partitions = AWSLogs/<Account ID>/CloudTrail/<Region>.
sourcetype Source type A source type for the events. Enter a value only if you want to override the default of aws:cloudtrail. Event extraction relies on the default value of source type. If you change the default value, you must update props.conf as well.
index Index The index name where the Splunk platform puts the CloudTrail data. The default is main.

Configure a CloudTrail input using configuration files

To configure inputs manually in inputs.conf, create a stanza using the following template and add it to $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/inputs.conf. If the file or path does not exist, create it.

[aws_cloudtrail://<name>]
aws_account = <value>
aws_region = <value>
sqs_queue = <value>
exclude_describe_events = <value>
remove_files_when_done = <value>
blacklist = <value>
excluded_events_index = <value>
interval = <value>
sourcetype = <value>
index = <value>

Some of these settings have default values that can be found in $SPLUNK_HOME/etc/apps/Splunk_TA_aws/default/inputs.conf:

[aws_cloudtrail]
aws_account =
sourcetype = aws:cloudtrail
exclude_describe_events = true
remove_files_when_done = false
queueSize = 128KB
persistentQueueSize = 24MB
interval = 30

The values in default/inputs.conf correspond to the default values in Splunk Web as well as some internal values that are not exposed in Splunk Web for configuration. If you choose to copy this stanza to /local and use it as a starting point to configure your inputs.conf manually, change the stanza title from aws_cloudtrail to aws_cloudtrail://<name>.

Switch from a CloudTrail input to an SQS-based S3 input

The SQS-based S3 input is a more fault-tolerant and higher-performing alternative to the CloudTrail input for collecting CloudTrail data. If you are already collecting CloudTrail data using a CloudTrail input, you can configure an SQS-based S3 input and seamlessly switch to the new input for CloudTrail data collection with little disruption.

  1. Disable the CloudTrail input you are using to collect CloudTrail data.
  2. Set up a Dead-Letter Queue (DLQ) and the SQS visibility timeout setting for the SQS queue from which you are collecting CloudTrail data. See Configure SQS-based S3 inputs for the Splunk Add-on for AWS.
  3. Create an SQS-based S3 input, pointing to the SQS queue you configured in the last step. Configure SQS-based S3 inputs for the Splunk Add-on for AWS for the detailed configuration steps.

Once configured, the new SQS-based S3 input replaces the old CloudTrail input to collect CloudTrail data from the same SQS queue.

Last modified on 28 August, 2020
PREVIOUS
Configure Config Rules inputs for the Splunk Add-on for AWS
  NEXT
Configure CloudWatch inputs for the Splunk Add-on for AWS

This documentation applies to the following versions of Splunk® Supported Add-ons: released


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters