Splunk Cloud

Splunk Cloud Admin Manual

Download manual as PDF

Download topic as PDF

Get Amazon Web Services (AWS) data into Splunk Cloud

This section guides you through the steps to get AWS data into Splunk Cloud.

Before you begin

To get AWS data into Splunk Cloud, you need a high-level understanding of Splunk and AWS concepts.

Splunk Concepts

The following concepts relate to Splunk Cloud:

  • Indexes. The index is the repository for your data. When Splunk Cloud indexes raw data, it transforms the data into searchable events.
  • Inputs Data Manager. The Inputs Data Manager (IDM) is a component of your Splunk Cloud environment optimized for data ingestion. It is intended for use with Cloud data sources or when using add-ons that require inputs on the search tier.
  • Source types. A source type is one of the critical default fields that Splunk Cloud assigns to all incoming data. It tells Splunk Cloud what kind of data you have so that it can format the data intelligently during indexing.
  • Splunk apps and add-ons. In these configuration steps, you use an add-on to get data in and you use an app to visualize the data.

AWS Concepts

The following concept relates to AWS:
AWS CloudTrail. This is a service that enables governance, compliance, operational auditing, and risk auditing of your AWS account. CloudTrail provides event history of your AWS account activity, including actions taken through the AWS Management Console, AWS SDKs, command-line tools, and other AWS services.

Prerequisites in your Splunk Cloud environment

You must meet the following prerequisites before you can get AWS data into Splunk Cloud:

  • You must be assigned the sc_admin role.
  • Request Splunk Support to install the Splunk Add-on for AWS installed on your Inputs Data Manager and the Splunk App for AWS on your Splunk Cloud instance. Ensure you allow adequate time for these tasks to be completed before you attempt to get data in.
  • The Splunk Add-on for AWS makes REST API calls using HTTPS on port 443 so ensure this port is available.
  • Create a test index in your Splunk Cloud instance so that you can test your installation before going into production.

Prerequisites in your AWS environment

You must meet the following prerequisites before you can get AWS data into Splunk Cloud:

  • You need a valid AWS account with administrative permissions to configure the AWS services that provide your data. If you do not have permissions to perform all the actions yourself, work with an AWS administrator to complete all steps, including creating the accounts or EC2 IAM roles with the permissions that the Splunk Add-on for AWS uses to connect. If you need to review the AWS documentation, go to https://docs.aws.amazon.com/index.html .
  • You need permission to create IAM roles and users. This lets you set up accounts or EC2 IAM roles with the ability to collect data from your AWS services.

AWS region limitations

The Splunk Add-on for AWS supports all regions offered by AWS.

In the AWS China region and the AWS GovCloud regions, the add-on only supports the services that AWS supports in those regions. There are limitations in both of these regions, so ensure that you understand the limitations before you begin. For more information, see the following topics:

  • For an up-to-date list of what products and services are supported in this region, see amazonaws.cn/en/products/ or aws.amazon.com/about-aws/global-infrastructure/regional-product-services/.
  • For an up-to-date list of what services and endpoints are supported in the GovCloud region, see the AWS documentation: docs.aws.amazon.com/govcloud-us/latest/UserGuide/using-services.html.

Overview of getting your AWS CloudTrail data into Splunk Cloud

To get started, set up your Splunk Cloud Inputs Data Manager (IDM) to get data from AWS CloudTrail. There are many useful AWS services that you can configure later, but starting with CloudTrail provides a very comprehensive view of AWS activity.

To get AWS CloudTrail data into Splunk Cloud, complete the following high-level steps:

  1. Configure an access policy for Splunk Access in AWS.
  2. Create a Splunk Access user.
  3. Create a group for Splunk Access Users.
  4. Enable the AWS CloudTrail Service.
  5. Create an SQS subscription.
  6. Create an SQS subscription for the Dead Letter Queue.
  7. Configure the Splunk Add-on for AWS on your Splunk Cloud Inputs Data Manager instance.
  8. Configure CloudTrail inputs and view your dashboards.


The following graphic shows the configuration of AWS and Splunk Cloud that enables you to get AWS data into Splunk Cloud:
This graphic shows the workflow for getting AWS cloudtrail input into Splunk Cloud.
When you finish the configuration steps, AWS CloudTrail populates the following Splunk App for AWS dashboards:

  • Overview
  • Topology
  • Security Overview
  • IAM Activity
  • VPC Activity
  • Security Groups
  • Key Pairs Activity
  • Network ACLs
  • User Activity
  • Insights Overview
  • Security Anomaly Insights
  • Timeline

Step 1: Configure an access policy for Splunk Access in AWS

The graphic shows step 1 of the workflow
You need to set up an access account in AWS so that Splunk Cloud can communicate with AWS without having root access. The SplunkAccess account is used to collect data from AWS. To do this, complete the following steps:

  1. In AWS, open IAM > Policies > Create New Policy.
  2. Click Create New Policy.
  3. Select JSON.
  4. In the JSON visual editor, paste the following JSON code:
  5. {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "sqs:GetQueueAttributes",
           "sqs:ListQueues",
           "sqs:ReceiveMessage",
           "sqs:GetQueueUrl",
           "sqs:SendMessage",
           "sqs:DeleteMessage",
           "s3:ListBucket",
           "s3:GetObject",
           "s3:GetBucketLocation",
           "s3:ListAllMyBuckets",
           "s3:GetBucketTagging", 
           "s3:GetAccelerateConfiguration", 
           "s3:GetBucketLogging", 
           "s3:GetLifecycleConfiguration", 
           "s3:GetBucketCORS",
           "config:DeliverConfigSnapshot",
           "config:DescribeConfigRules",
           "config:DescribeConfigRuleEvaluationStatus",
           "config:GetComplianceDetailsByConfigRule",
           "config:GetComplianceSummaryByConfigRule",
           "iam:GetUser",
           "iam:ListUsers",
           "iam:GetAccountPasswordPolicy",
           "iam:ListAccessKeys",
           "iam:GetAccessKeyLastUsed", 
           "autoscaling:Describe*",
           "cloudwatch:Describe*",
           "cloudwatch:Get*",
           "cloudwatch:List*",
           "sns:Get*",
           "sns:List*",
           "sns:Publish",
           "logs:DescribeLogGroups",
           "logs:DescribeLogStreams",
           "logs:GetLogEvents",
           "ec2:DescribeInstances",
           "ec2:DescribeReservedInstances",
           "ec2:DescribeSnapshots",
           "ec2:DescribeRegions",
           "ec2:DescribeKeyPairs",
           "ec2:DescribeNetworkAcls",
           "ec2:DescribeSecurityGroups",
           "ec2:DescribeSubnets",
           "ec2:DescribeVolumes",
           "ec2:DescribeVpcs",
           "ec2:DescribeImages",
           "ec2:DescribeAddresses",
           "lambda:ListFunctions",
           "rds:DescribeDBInstances",
           "cloudfront:ListDistributions",
           "elasticloadbalancing:DescribeLoadBalancers",
           "elasticloadbalancing:DescribeInstanceHealth",
           "elasticloadbalancing:DescribeTags",
           "elasticloadbalancing:DescribeTargetGroups",
           "elasticloadbalancing:DescribeTargetHealth",
           "elasticloadbalancing:DescribeListeners",
           "inspector:Describe*",
           "inspector:List*",
           "kinesis:Get*",
           "kinesis:DescribeStream",
           "kinesis:ListStreams",
           "kms:Decrypt",
           "sts:AssumeRole"
         ],
         "Resource": [
           "*"
         ]
       }
     ]
    }
    
  6. Click Review Policy.
  7. Name the policy SplunkAccess.
  8. Give it a description.
  9. Click Create Policy.
  10. Return to the list of IAM policies and ensure that you see the new SplunkAccess policy in the list of policies. This graphic shows the creation of a SplunkAcess policy on AWS.

Step 2: Create a Splunk Access user

The graphic shows step 2 of the workflow
To create the Splunk Access user, complete the following steps:

  1. In AWS, from the IAM Users list, click Users.
  2. Click Add User.
  3. Add a user name. In this example, use the name Splunk_Access.
  4. For Select AWS Access type, select Programmatic access only. This safeguards your account. This graphic shows the selection of the Programmatic Access to ensure safe access.

  5. Click Next: Permissions to add the new user to a group.

Step 3: Create a group for Splunk Access users

The graphic shows step 3 of the workflow

After you create the user, you assign the user to a group:


  1. Click Create group:
    This graphic shows the creation of a user group in AWS

  2. In Group Name, enter SplunkAccessGroup.
  3. Apply the IAM policy you created. To do this, search for the IAM policy by entering splunk in the Filter Policies field. The IAM access policy you created displays.
  4. Select the checkbox next to the SplunkAccess IAM policy, and click Create Group.
  5. Ensure your new group is selected and click Next Review:
    This graphic shows the creation of a user group in AWS

  6. Now that you have created a user and associated that user with a group and an IAM policy, click Create User.
  7. After the user is created, Amazon provides access to a CSV file with the user credentials. Download this file and store it securely because you will need it later.

Step 4: Enable the AWS CloudTrail Service

The graphic shows step 4 of the workflow
Next, set up CloudTrail so that you can capture AWS data and send it to Splunk Cloud. AWS CloudTrail writes events to a Simple Notification Service (SNS) topic, and you can then create a Simple Queue Service (SQS) subscription. When SQS notifies Splunk Cloud of an event, Splunk Cloud can collect the events from the S3 bucket. To do this, complete the following steps:

  1. In AWS Management Console, click All Services, and select CloudTrail.
  2. Click Get Started Now.
  3. Name your trail. For this example, use the name 'cloudtrail.
  4. Apply the trail to all regions.
  5. In the Read/Write events field, select All.
  6. Create a storage location for CloudTrail. In this case, create a new S3 bucket. To do this, select Create a new S3 bucket. As a best practice, use the naming convention of cloudtrail-<AWSAccountID>.
  7. Click Advanced.
  8. In the Send SNS notification for every log file delivery field, clickYes.
  9. In the Create a new SNS topic field, click Yes.
  10. Enter a name for your SNS topic. For this example, name the topic cloudtrail.

Step 5: Create an SQS subscription

The graphic shows step 5 of the workflow
After the AWS CloudTrailService has been created, you need to set up an SQS to subscribe to the SNS topic created by AWS CloudTrail:

  1. From the AWS Management Console > Application Integration, select Simple Queue Service.
  2. Click Create New Queue.
  3. Enter a name for the queue. For example, call it cloudtrail.
  4. Select Standard Queue.
  5. Click Quick-Create Queue. Your new queue is displayed.
  6. Select your queue and select Queue actions.
  7. Select Subscribe Queue to SNS topic.
  8. In the Choose a Topic field, select the SNS topic you created in step 4:
    This graphic shows subscribing to a topic on AWS

  9. The Topic ARN auto-populates.
  10. Select Subscribe.

Step 6: Create an SQS subscription for the Dead Letter Queue

The graphic shows step 6 of the workflow
Next, you create a CloudTrail Dead Letter Queue. This queue is required for the new Splunk SQS-based S3 input. To keep the inputs stateless, the Dead Letter Queue notifies Splunk Cloud where the last input left off and where to continue collecting events from AWS. To do this, complete the following steps:

  1. From the AWS Management Console > Application Integration, select Simple Queue Service.
  2. Click Create New Queue.
  3. Enter a name for the queue. For this example, call it cloudtrail-dlq.
  4. Select Standard Queue.
  5. Click Quick-Create Queue. Your new queue is displayed.
  6. From the list of queues, select your cloudtrail queue, and click Queue Actions.
  7. Click Configure Queue.
  8. In the Dead Letter Queue Settings, select Use Redrive Policy.
  9. In the Dead Letter Queue field, enter cloudtrail-dlq.
  10. In the Maximum receives field, enter 3.
  11. Click Save Changes.

Step 7: Configure the Splunk Add-on for AWS on your Splunk Cloud Inputs Data Manager (IDM) instance

The graphic shows step 7 of the workflow
Now that you have configured your AWS settings, configure the Splunk Add-on for AWS to get data from your AWS account:

  1. From your Splunk Cloud IDM instance, open the Splunk Add-on for AWS from the list of available apps. You can log into your IDM at https://idm-<cloudname>.splunkcloud.com where <cloudname> represents your Splunk Cloud name.
  2. Click Configuration > Create an account > Add.
  3. In the Add Account field, enter a name for the account. For this example, name it splunk_access.
  4. Open the credentials file you previously downloaded. Enter the credentials in the Key ID and Secret Key fields.
  5. Leave the Region Category field as Global unless you are using GovCloud or AWS China.
  6. Click Add.

Step 8: Configure Cloudtrail inputs on your Splunk Cloud Inputs Data Manager (IDM) instance

The graphic shows step 8 of the workflow
Now that you have configure the Splunk Add-on for AWS, you need to configure Cloudtrail inputs on your IDM:

  1. In the Inputs tab, click Create New Input > Cloudtrail > SQS-Based S3.
  2. Enter a name for the CloudTrail input. For example, name it cloudtrail.
  3. From the drop-down list, select your AWS account.
  4. Select your AWS region.
  5. In the SQS queue field, select your cloudtrail queue.
  6. Use the default values for other settings, and click Save.

Now you can open your Splunk App for AWS dashboards and see that the dashboards for Cloudtrail are getting populated with data.

What's next?

Now that you have configured the Splunk AWS Add-on for your Splunk Cloud instance, you might want to set up your Splunk Cloud instance to get data from the following services:

  • Amazon CloudWatch is a monitoring service for AWS cloud resources and the applications you run on AWS. You can use Amazon CloudWatch to collect and track metrics, collect and monitor log files, set alarms, and automatically react to changes in your AWS resources. See the Splunk Add-on for AWS CloudWatch documentation.
  • AWS Config is a service that enables you to assess, audit, and evaluate the configurations of your AWS resources. Config continuously monitors and records your AWS resource configurations and allows you to automate the evaluation of recorded configurations against desired configurations. See the Splunk Add-on for AWS Config documentation.
  • AWS Inspector is an automated security assessment scanner that can evaluate security loopholes and deviation from the best practices for applications hosted on AWS. AWS Inspector communicates with EC2 instances using agents installed on it and generates reports. See the Splunk Add-on for AWS Inspector documentation.
Last modified on 11 May, 2020
PREVIOUS
Introduction to Getting Data In
  NEXT
Get Microsoft Azure data into Splunk Cloud

This documentation applies to the following versions of Splunk Cloud: 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 8.0.2001, 8.0.2003, 8.0.2004


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters