Splunk® Supported Add-ons

Splunk Add-on for AWS

Download manual as PDF

Download topic as PDF

Configure AWS services for the Splunk Add-on for AWS

To collect data from Amazon Web Services, you must first enable or configure the AWS services that produce the data.

As a last step for each service, you need to configure the appropriate permissions for accounts or EC2 IAM roles that the Splunk Add-on for AWS uses to connect to your AWS environment, so that the add-on can access the data from the services you have configured. See Configure AWS permissions for details.

Note: If your account is in the AWS China region or the AWS GovCloud region, not all AWS services are available to you.

Prerequisite: Performing all the steps below requires administrator access to your AWS account. If you do not have the required permissions to perform all the actions yourself, work with an AWS admin to complete all steps, including creating the accounts or EC2 IAM roles with the permissions that the Splunk Add-on for AWS uses to connect.

Configure AWS Config

The Splunk Add-on for AWS collects events from a Simple Queue Service (SQS) that subscribes to the Simple Notification Service (SNS) notification events from AWS Config. Configure AWS Config to produce these notifications, then create the SQS for the add-on to access them.

1. Enable Config by following the AWS Config setup guide: http://docs.aws.amazon.com/config/latest/developerguide/setting-up.html.

2. Follow the AWS Config Getting Started guide (http://docs.aws.amazon.com/config/latest/developerguide/getting-started.html) to specify an S3 bucket to save the data and an SNS topic to stream Config notifications to. Do not use an existing bucket or SNS. Following the AWS Config setup allows AWS to automatically create the IAM role for AWS config so that it has the necessary permissions for the bucket and SNS.

3. Finish the setup steps in the AWS Config Getting Started guide and verify that you have successfully completed the setup process. If you used the AWS console, you should see the Resource Lookup page. If you use the CLI, you can follow this verification guide: http://docs.aws.amazon.com/config/latest/developerguide/gs-cli-verify-subscribe.html.

4. Create a new SQS.

5. Subscribe the SQS exclusively to the the SNS Topic that you created in Step 2.

6. Grant IAM permissions to access the S3 bucket and SQS to the AWS account that the add-on uses to connect to your AWS environment. See Configure AWS permissions for details.

Configure AWS Config Rules

AWS Config Rules requires no additional configuration beyond that described in the AWS documentation.

1. Enable AWS Config for all regions for which you want to collect data in the add-on. Follow the AWS Config setup guide: http://docs.aws.amazon.com/config/latest/developerguide/setting-up.html.

2. Set up AWS Config Rules by following the instructions in the AWS Config documentation: http://docs.aws.amazon.com/config/latest/developerguide/evaluate-config_set-up.html

3. Grant the necessary permissions to the AWS account used for this input. See Configure AWS permissions for details.

Configure CloudTrail

The Splunk Add-on for AWS collects events from a Simple Queue Service (SQS) that subscribes to the Simple Notification Service (SNS) notification events from CloudTrail. Configure CloudTrail to produce these notifications, then create an SQS in each region for the add-on to access them.

Note: Although AWS offers global trails, or one CloudTrail configuration in one region to collect trail data from all regions, SQS messages do not arrive as expected in this case. Either configure separate CloudTrail S3 > SNS > SQS paths for each region to ensure that you capture all your data or, if you want to configure a global CloudTrail, skip steps 3 through 6 below and instead configure the add-on to collect data from that S3 bucket directly.

1. Enable CloudTrail. Follow the instructions in the AWS documentation: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html.

2. Create an S3 Bucket in which to store the CloudTrail events. Follow the AWS documentation to ensure the permissions for this bucket are correct: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/create-s3-bucket-policy-for-cloudtrail.html

3. Enable SNS Notifications. See the AWS documentation for instructions: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/getting_notifications_top_level.html

4. Create a new SQS.

5. If you are in the China region, explicitly grant DeleteMessage and SendMessage permissions to the SQS that you just created. This step is not necessary in commercial regions.

6. Subscribe the SQS to the SNS Notifications that you enabled in step 3.

7. Grant IAM permissions to access the S3 bucket and SQS to the AWS account that the add-on uses to connect to your AWS environment. See Configure AWS permissions for details.

Configure CloudWatch

To enable AWS to produce billing metrics in CloudWatch, turn on Receive Billing Alerts in the Preferences section of the Billing and Cost Management console.

The CloudWatch service is automatically enabled to collect free metrics for your AWS services and requires no additional configuration for the Splunk Add-on for AWS. However, you do need to grant permissions to the AWS account(s) that the add-on uses to connect to the CloudWatch API. See Configure AWS permissions for details.

Configure CloudWatch Logs, including VPC Flow Logs

Ingesting data from CloudWatch Logs requires no additional configuration beyond that described in the AWS documentation. Similarly, VPC Flow Logs require no additional configuration for the Splunk Add-on for AWS, other than enabling them for your VPCs. However, you do need to grant permissions to the AWS account(s) that the add-on uses to connect to the VPC Flow Log groups and streams. See Configure AWS permissions for details.

See the AWS documentation for how to enable Flow Logs for your VPCs and configure an IAM role for them: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html.

Configure Inspector

Inspector requires no additional configuration for the Splunk Add-on for AWS. However, you do need to grant permissions to the AWS accounts or EC2 IAM roles that the add-on uses to connect to the Amazon Inspector API. See Configure AWS permissions for details.

Configure Kinesis

Kinesis requires no additional configuration for the Splunk Add-on for AWS. However, you do need to grant permissions to the AWS accounts or EC2 IAM roles that the add-on uses to connect to the Kinesis API. See Configure AWS permissions for details.

Configure S3

If you are collecting generic log files, S3 requires no additional configuration for the Splunk Add-on for AWS. However, you do need to grant permissions to the AWS account that the add-on uses to connect to your S3 buckets. See Configure AWS permissions for details.

If you are collecting access logs, you must configure logging in the AWS console to collect the logs in a dedicated S3 bucket. See the AWS documentation for more information on how to configure access logs:

Refer to the AWS S3 documentation for more information about how to configure S3 buckets and objects. http://docs.aws.amazon.com/gettingstarted/latest/swh/getting-started-create-bucket.html

Configure billing

The Splunk App for AWS collects billing metrics through CloudWatch and billing reports by collecting them from an S3 bucket.

To enable AWS to produce billing metrics in CloudWatch, turn on Receive Billing Alerts in the Preferences section of the Billing and Cost Management console.

To enable billing reports, turn on Receive Billing Reports in the Preferences section of the Billing and Cost Management console. Be sure to verify your S3 bucket in the billing and cost management console and select the report types that you want to collect.

There is no additional configuration required for the Splunk Add-on for AWS other than enabling the billing reports that you want to collect and setting up an S3 bucket to collect them. However, you do need to grant permissions to access the S3 bucket to the AWS account that the add-on uses to connect to your AWS environment. See Configure AWS permissions for details.

For more details on managing your AWS billing reports, see the AWS documentation: http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/detailed-billing-reports.html

For more information about accessing the billing report data in your searches, see Access Billing Report data.

Configure SNS

You need to grant permissions to the AWS accounts or EC2 IAM roles that the add-on uses to connect to the Amazon SNS API. See Configure AWS permissions for details.

If you plan to use the SQS-based S3 input, you must enable Amazon S3 bucket events to send notification messages to an SQS queue whenever the events occur. For instructions on setting up S3 bucket event notifications, see the AWS documentation:
https://docs.aws.amazon.com/AmazonS3/latest/UG/SettingBucketNotifications.html
http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html

Configure SQS

You need to grant permissions to the AWS accounts or EC2 IAM roles that the add-on uses to connect to the Amazon SQS API. See Configure AWS permissions for details.

If you plan to use the SQS-based S3 input, you must perform the following:

PREVIOUS
Sizing, performance, and cost considerations for the Splunk Add-on for AWS
  NEXT
Configure AWS permissions for the Splunk Add-on for AWS

This documentation applies to the following versions of Splunk® Supported Add-ons: released


Comments

The Splunk Add-on for AWS is very feature-rich, seamlessly supporting data collection from AWS services. You can also configure API Gateway on the AWS side to integrate Splunk into your AWS “serverless” infrastructure. For detailed information, see http://blogs.splunk.com/2017/01/06/splunk-and-aws-monitoring-metrics-in-a-serverless-world .

Hunters splunk, Splunker
January 9, 2017

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters