Configure your AWS services for the Splunk App for AWS

To collect data from Amazon Web Services, you must first enable or configure the AWS services that produce the data. Splunk recommends that you enable all services, otherwise some of the app dashboards will not be fully populated.

For each service, you must configure the appropriate IAM permissions for the accounts or EC2 IAM roles that the Splunk App for AWS uses to connect to your AWS environment, so that the app can access the data from the services you have configured. See Configure your AWS permissions for details.

Note: If your account is in the AWS China region or the AWS GovCloud region, not all AWS services are available to you.

• If you are in the AWS China region, the add-on only supports the services that AWS supports in that region. The China region does not support Config Rules, Inspector, CloudWatch Logs, or CloudFront services, nor does it offer CloudWatch metrics for ELB logs. For an up-to-date list of what products and services are supported in this region, see http://www.amazonaws.cn/en/products/.
• If you are in the AWS GovCloud region, the add-on only supports the services that AWS supports in that region. The GovCloud region does not support Config Rules, Inspector, or Kinesis at this time. For an up-to-date list of what services and endpoints are supported in this region, see the AWS documentation: http://docs.aws.amazon.com/govcloud-us/latest/UserGuide/using-services.html.

Prerequisite: Performing all the steps below requires administrator access to your AWS account. If you do not have the required permissions to perform all the actions yourself, work with an AWS admin to complete all steps, including creating the account(s) with the IAM permissions that the Splunk App for AWS uses to connect.

Configure AWS Config

The Splunk App for AWS collects events from a Simple Queue Service (SQS) that subscribes to the Simple Notification Service (SNS) notification events from AWS Config. Configure AWS Config to produce these notifications, then create the SQS for the app to access them.

1. Enable Config by following the AWS Config setup guide: http://docs.aws.amazon.com/config/latest/developerguide/setting-up.html. In the Resource types to record section, check the box to Include global resources. Enabling this option makes it possible to display IAM data (which is not specific to any one region) in your Topology dashboard.

2. Follow the AWS Config Getting Started guide (http://docs.aws.amazon.com/config/latest/developerguide/getting-started.html) to specify an S3 bucket to save the data and an SNS topic to stream Config notifications to. Do not use an existing bucket or SNS. Following the AWS Config setup allows AWS to automatically create the IAM role for AWS config so that it has the necessary permissions for the bucket and SNS.

3. Finish the setup steps in the AWS Config Getting Started guide and verify that you have successfully completed the setup process. If you used the AWS console, you should see the Resource Lookup page. If you use the CLI, you can follow this verification guide: http://docs.aws.amazon.com/config/latest/developerguide/gs-cli-verify-subscribe.html.

4. Create a new SQS.

5. Subscribe the SQS exclusively to the the SNS Topic that you created in Step 2.

6. Grant IAM permissions to access the S3 bucket and SQS to the AWS account that the app uses to connect to your AWS environment. See Configure your AWS permissions for details.

7. For best results, ensure that you have enabled CloudTrail in each region for which you have enabled Config. If you collect Config data with the app without enabling CloudTrail as well in the same region, some app dashboards may not be fully populated.

Configure AWS Config Rules

AWS Config Rules requires no additional configuration beyond that described in the AWS documentation.

1. Enable AWS Config for all regions for which you want to collect data in the add-on. Follow the AWS Config setup guide: http://docs.aws.amazon.com/config/latest/developerguide/setting-up.html.

2. Set up AWS Config Rules by following the instructions in the AWS Config documentation: http://docs.aws.amazon.com/config/latest/developerguide/evaluate-config_set-up.html

3. Grant the necessary permissions to the AWS account used for this input. See Configure your AWS permissions for details.

Configure CloudTrail

The Splunk App for AWS collects events from a Simple Queue Service (SQS) that subscribes to the Simple Notification Service (SNS) notification events from CloudTrail. Configure CloudTrail to produce these notifications, then create an SQS in each region for the app to access them.

Note: Although AWS offers global trails, or one CloudTrail configuration in one region to collect trail data from all regions, SQS messages do not arrive as expected in this case. Either configure separate CloudTrail S3 > SNS > SQS paths for each region to ensure that you capture all your data or, if you want to configure a global CloudTrail, skip steps 3 through 6 below and instead configure the app to collect data from that S3 bucket directly.

1. Enable CloudTrail. Follow the instructions in the AWS documentation: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html.

2. Create an S3 Bucket in which to store the CloudTrail events. Follow the AWS documentation to ensure the permissions for this bucket are correct: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/create-s3-bucket-policy-for-cloudtrail.html

3. Enable SNS Notifications. See the AWS documentation for instructions: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/getting_notifications_top_level.html

4. Create a new SQS.

5. If you are in the China region, explicitly grant DeleteMessage and SendMessage permissions to the SQS that you just created. This step is not necessary in commercial regions.

6. Subscribe the SQS to the SNS Notifications that you enabled in step 3.

7. Grant IAM permissions to access the S3 bucket and SQS to the AWS account that the app uses to connect to your AWS environment. See Configure your AWS permissions for details.

Configure CloudWatch

To enable AWS to produce billing metrics in CloudWatch, turn on Receive Billing Alerts in the Preferences section of the Billing and Cost Management console.

The CloudWatch service is automatically enabled to collect free metrics for your AWS services and requires no additional configuration for the Splunk App for AWS. However, you do need to grant permissions to the AWS account(s) that the app uses to connect to the CloudWatch API. See Configure your AWS permissions for details.

Configure CloudWatch Logs

Amazon CloudWatch Logs require no additional configuration for the Splunk App for AWS, other than enabling VPC Flow Logs, which are stored using CloudWatch Logs, for your VPCs. However, you do need to grant permissions to the AWS account(s) that the app uses to connect to the VPC Flow Log groups and streams. See Configure your AWS permissions for details.

See the AWS documentation for how to enable Flow Logs for your VPCs and configure an IAM role for them: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html.

Configure Inspector

Inspector requires no additional configuration for the Splunk App for AWS, other than setting up the Inspector service. However, you do need to grant permissions to the AWS accounts or EC2 IAM roles that the add-on uses to connect to the Amazon Inspector API. See Configure your AWS permissions for details.

Configure S3

If you are collecting generic log files, S3 requires no additional configuration for the Splunk App for AWS. However, you do need to grant permissions to the AWS account that the app uses to connect to your S3 buckets. See Configure your AWS permissions for details.

If you are collecting access logs, you must configure logging in the AWS console to collect the logs in a dedicated S3 bucket. See the AWS documentation for more information on how to configure access logs:

• Enable S3 access logs: http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html
• Enable ELB access logs: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-access-logs.html
• Enable CloudFront access logs: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html

Refer to the AWS S3 documentation for more information about how to configure S3 buckets and objects. http://docs.aws.amazon.com/gettingstarted/latest/swh/getting-started-create-bucket.html

Configure billing

The Splunk App for AWS collects billing metrics through CloudWatch and billing reports by collecting them from an S3 bucket.

To enable AWS to produce billing metrics in CloudWatch, turn on Receive Billing Alerts in the Preferences section of the Billing and Cost Management console.

To enable billing reports, turn on Receive Billing Reports in the Preferences section of the Billing and Cost Management console. The Splunk App for AWS can collect two kinds of reports from your AWS billing service: monthly cost allocation reports and detailed billing reports with resources and tags. Be sure to verify your S3 bucket in the billing and cost management console and select the report types that you want to collect.

There is no additional configuration required for the Splunk App for AWS to collect your billing reports. However, you do need to grant permissions to access the S3 bucket to the AWS account that the app uses to connect to your AWS environment. See Configure your AWS permissions for details.

For more details on managing your AWS billing reports, see the AWS documentation: http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/detailed-billing-reports.html

Configure Kinesis

Create Amazon Kinesis streams and configure data producers to continously put data into the stream for the Splunk App for AWS to consume.

1. Create an Amazon Kinesis stream through either Amazon Kinesis Management Console or Amazon Kinesis CreateStream API.

2. Configure your data producers to continuously put data into your Amazon Kinesis stream. Data producers can put data into Amazon Kinesis streams using the Amazon Kinesis Streams APIs, Amazon Kinesis Producer Library (KPL), or Amazon Kinesis Agent.

3. Control Access to Amazon Kinesis Streams Resources Using IAM. See http://docs.aws.amazon.com/streams/latest/dev/controlling-access.html .

For Amazon Kinesis Streams product details, refer to: https://aws.amazon.com/kinesis/streams/details