AWS Kinesis Data Firehose prerequisites for Data Manager

An AWS Admin completes prerequisites ahead of time so that a Splunk Admin can use Data Manager for onboarding. Alternatively, an AWS Admin can complete the entire process. Data Manager contains optional steps to guide you through this choice.

Version 1.3.2 or higher of the Splunk Add-on for Amazon Kinesis Firehose must be installed on your Splunk Cloud search heads in order to perform searches using the Splunk Common Information Model (CIM) model.

Single account prerequisites

Single account onboarding is when you ingest data from a single AWS account. Choose one account for use as both the management account and as the target account from which to ingest data. This one account allows you to create, update, and delete stack sets across multiple regions.

More details follow for additional assistance with onboarding.

Create a SplunkDMReadOnly role

This role allows Splunk Cloud to read metadata from your AWS events, and logs.

Configure through the console

Complete the following steps in the AWS console.

1. Log into your AWS account.
2. Navigate to IAM > Roles.
3. Click Create role.
4. Click Another AWS account.
5. In the Role Name field, type exactly the name of SplunkDMReadOnly and click Create role.
6. Click SplunkDMReadOnly.
7. Under the Permissions tab, click Add inline policy.
   1. Click the JSON tab.
   2. Overwrite the JSON text by copying and pasting the Role Policy from the Data Manager UI.
   3. Click Review Policy.
   4. In the Name field, type any name of your choice, such as SplunkDMReadOnlyPolicy.
   5. Click Create policy.
8. Under the Trust relationships tab, click Edit trust relationship.
   1. Overwrite the JSON text by copying and pasting the Trust Relationship from the Data Manager UI.
   2. Replace the <DATA_ACCOUNT_ID> variables with your account ID.
   3. Click Update Trust Policy.

Configure through the CLI

Prepare the terminal to use the AWS credentials that allow you to run the following CLI command against your AWS account. See AWS CLI Prerequisites.

1. Create the SplunkDMReadOnly role, replacing the <EXTERNAL_ID> variable from the Trust Relationship in the Data Manager UI:
   aws iam create-role --role-name SplunkDMReadOnly --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"sts:AssumeRole","Principal":{"AWS":"*"},"Condition":{"StringEquals":{"sts:ExternalId":"<EXTERNAL_ID>"}}}]}'
2. Create the inline policy for SplunkDMReadOnlyPolicy and attach it to the role, replacing the <DATA_ACCOUNT_ID> variables with your AWS account ID from the Role Policy in the Data Manager UI:
   aws iam put-role-policy --policy-name SplunkDMReadOnlyPolicy --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["iam:GetRole","iam:PassRole","iam:GetRolePolicy"],"Resource":"arn:aws:iam::<DATA_ACCOUNT_ID>:role/SplunkDM*"},{"Effect":"Allow","Action":"guardduty:GetMasterAccount","Resource":"arn:aws:guardduty:*:<DATA_ACCOUNT_ID>:detector/*"},{"Effect":"Allow","Action":["securityhub:GetEnabledStandards","securityhub:GetMasterAccount"],"Resource":"arn:aws:securityhub:*:<DATA_ACCOUNT_ID>:hub/default"},{"Effect":"Allow","Action":"cloudformation:GetTemplate","Resource":"arn:aws:cloudformation:*:<DATA_ACCOUNT_ID>:stack/SplunkDM*/*"},{"Effect":"Allow","Action":["cloudtrail:DescribeTrails","cloudformation:DescribeStacks","guardduty:ListDetectors","access-analyzer:ListAnalyzers"],"Resource":"*"}]}' --role-name SplunkDMReadOnly

(Optional) Create an onboarding user

If you are the AWS admin and will be completing the AWS data onboarding, then you can use your admin privileges to complete the data onboarding steps. If you want a different user to continue with the onboarding, then create a user in the AWS account with the following permissions. The user can be created as an IAM user, IAM role, SAML user, or any of your company's AWS user creation policies. Make sure that this user has both AWS CLI and console access.

Configure through the console

As one example, consider the scenario of creating an IAM user to complete the data onboarding. To create an IAM user, complete the following steps in the AWS console.

1. Log into your AWS account.
2. Navigate to IAM > Users.
3. Click Add user.
   1. In the User name field, type any name of your choice, such as OnboardingUser.
   2. For the Access type check box, select AWS Management Console access.
   3. For the Console password radio button, select the option of your choice.
   4. For the Required password reset check box, select User must create a new password at next sign-in.
   5. Click Next: Permissions.
4. For Set permissions complete the following steps:
   1. Click Attach existing policies directly.
   2. Click Create policy.
   3. In the new browser window that opens, click the JSON tab.
   4. Overwrite the JSON text by copying and pasting the Permissions from the Data Manager UI.
   5. Replace the <DATA_ACCOUNT_ID> variables with your account ID.
   6. Click Next: Tags > Next: Review.
   7. In the Name field, type any name of your choice, such as OnboardingUserPolicy.
   8. Click Create policy.
5. Go back to the previous tab, so that you see the set permissions section.
6. Click the refresh icon.
7. In the Filter policies field, search for your policy name.
8. Select the check box for your policy.
9. Click Next: Tags > Next: Review.
10. Click Create user.

Multiple account prerequisites

Multiple account onboarding is when you ingest data from multiple AWS accounts.

Choose one AWS account as a control account. The control account is an AWS account ID that you designate as the management account. It allows you to create, update, and delete stack sets across multiple accounts and regions. It is a separate account from the data accounts that you plan to monitor.

Choose multiple AWS accounts as your data accounts. The data accounts are AWS account IDs that you designate as the target accounts from which to ingest data. Data accounts are managed by the control account. The same data account cannot be used in multiple data inputs, managed by different control accounts, in Data Manager.

More details follow for additional assistance with onboarding.

Choose a control account

The AWS admin must choose a control account for this data input. The control account is an AWS account where you will drive StackSet operations. It allows you to create, update, and delete StackSets to manage resources across multiple data accounts and regions.

(Optional) Create an onboarding user

If you are the AWS admin and will be completing the AWS data onboarding, then you can use your admin privileges to complete the data onboarding steps. If you want a different user to continue with the onboarding, then create a user in the AWS account with the following permissions. The user can be created as an IAM user, IAM role, SAML user, or any of your company's AWS user creation policies. Make sure that this user has both AWS CLI and console access.

Configure through the console

As one example, consider the scenario of creating an IAM user to complete the data onboarding. To create an IAM user, complete the following steps in the AWS console.

1. Log into your AWS account.
2. Navigate to IAM > Users.
3. Click Add user.
   1. In the User name field, type any name of your choice, such as OnboardingUser.
   2. For the Access type check box, select AWS Management Console access.
   3. For the Console password radio button, select the option of your choice.
   4. For the Required password reset check box, select User must create a new password at next sign-in.
   5. Click Next: Permissions.
4. For Set permissions complete the following steps:
   1. Click Attach existing policies directly.
   2. Click Create policy.
   3. In the new browser window that opens, click the JSON tab.
   4. Overwrite the JSON text by copying and pasting the Permissions from the Data Manager UI.
   5. Replace the <CONTROL_ACCOUNT_ID> variables with your account ID.
   6. Click Next: Tags > Next: Review.
   7. In the Name field, type any name of your choice, such as OnboardingUserPolicy.
   8. Click Create policy.
5. Go back to the previous tab, so that you see the set permissions section.
6. Click the refresh icon.
7. In the Filter policies field, search for your policy name.
8. Select the check box for your policy.
9. Click Next: Tags > Next: Review.
10. Click Create user.

Create the AWSCloudFormationStackSetAdministrationRole in the control account

If this role already exists in the control account, the AWS admin can skip this step. This role allows you to manage StackSet operations from the control account.

Configure through the console

Complete the following steps in the AWS console.

1. Log into your control account.
2. Navigate to IAM > Roles.
3. Click Create role.
4. From Choose a use case, click CloudFormation as the service.
5. Click Next: Permissions > Next: Tags > Next: Review.
6. In the Role Name field, type exactly the name of AWSCloudFormationStackSetAdministrationRole and click Create role.
7. Click AWSCloudFormationStackSetAdministrationRole.
8. Under the Permissions tab, click Add inline policy.
   1. Click the JSON tab.
   2. Copy and paste the Role Policy from the Data Manager UI, making sure to overwrite the text in the json text box.
   3. Click Review Policy.
   4. In the Name field, type any name of your choice, such as AWSCloudFormationStackSetAdministrationRolePolicy.
   5. Click Create policy.

Configure through the CLI

Prepare the terminal to use the AWS credentials that allow you to run the following CLI command against your control account. See AWS CLI Prerequisites.

1. Create the AWSCloudFormationStackSetAdministrationRole:
   aws iam create-role --role-name AWSCloudFormationStackSetAdministrationRole --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"cloudformation.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
2. Create the inline policy for AWSCloudFormationStackSetAdministrationRolePolicy and attach it to the role:
   aws iam put-role-policy --policy-name AWSCloudFormationStackSetAdministrationRolePolicy --policy-document '{"Version":"2012-10-17","Statement":[{"Action":["sts:AssumeRole"],"Resource":["arn:*:iam::*:role/AWSCloudFormationStackSetExecutionRole"],"Effect":"Allow"}]}' --role-name AWSCloudFormationStackSetAdministrationRole

Create the AWSCloudFormationStackSetExecutionRole in the data accounts

If this role already exists in each data account that you'll be using for this configuration, the AWS admin can skip this step. If the role doesn't exist, then the AWS admin creates the AWSCloudFormationStackSetExecutionRole in each data account. This role allows the control account to create stack instances in your data accounts. The stack instances create resources that include IAM roles, CloudWatch log subscription filters, CloudWatch event bridge rules, and Kinesis Data Firehose delivery streams.

Configure through the console

Complete the following steps in the AWS console.

1. Log into your data account.
2. Navigate to IAM > Roles.
3. Click Create role.
4. Click Another AWS account.
5. In the Account ID field, type your control account ID.
6. Click Next: Permissions > Next: Tags > Next: Review.
7. In the Role Name field, type exactly the name of Create the AWSCloudFormationStackSetExecutionRole and click Create role.
8. Click AWSCloudFormationStackSetExecutionRole. Click refresh if it is not available.
9. Under the Permissions tab, click Add inline policy.
   1. Click the JSON tab.
   2. Copy and paste the Role Policy from the Data Manager UI, making sure to overwrite the text in the json text box.
   3. The security warning is normal. No action is needed.
   4. Click Review Policy.
   5. In the Name field, type any name of your choice, such as AWSCloudFormationStackSetExecutionRolePolicy.
      The summary notice is normal. No action is needed.
   6. Click Create policy.
10. Repeat for each data account that you want added to this configuration.

Configure through the CLI

Prepare the terminal to use the AWS credentials that allow you to run the following CLI command against each data account. See AWS CLI Prerequisites.

1. Create the AWSCloudFormationStackSetExecutionRole, replacing the <CONTROL_ACCOUNT_ID> variable with your control account ID:
   aws iam create-role --role-name AWSCloudFormationStackSetExecutionRole --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::<CONTROL_ACCOUNT_ID>:root"},"Action":"sts:AssumeRole","Condition":{}}]}'
2. Create the inline policy for AWSCloudFormationStackSetExecutionRolePolicy and attach it to the role:
   aws iam put-role-policy --policy-name AWSCloudFormationStackSetExecutionRolePolicy --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"*","Resource":"*"}]}' --role-name AWSCloudFormationStackSetExecutionRole

Create the SplunkDMReadOnly role in the control account

If this role already exists in the control account, the AWS admin can skip this step. If the role doesn't exist, then the AWS admin creates the SplunkDMReadOnly role in the control account. This role is needed in the control account for reading IAM user and CloudFormation StackSet status. Make sure that the AWS administrator replaces the account identifiers in the policy.

Configure through the console

Complete the following steps in the AWS console.

1. Log into your control account.
2. Navigate to IAM > Roles.
3. Click Create role.
4. Click Another AWS account.
5. In the Account ID field, copy and paste the Splunk Cloud account ID from the Trust Relationship in the Data Manager UI.
   For example, copy 123456789012 from the principal object:
   "Principal" : {"arn:aws:iam::123456789012:role/cfgh-d12345-12345"}.
6. Click the Options check box for Require external ID.
   1. In the External ID field, copy and paste the sts:Externald from the Trust Relationship in the Data Manager UI.
      For example, copy ffcbd123-1a234-123b-12c3-1234567890b from the conditions object:
      "Conditions": {"StringEquals": {"sts:ExternalID": "ffcbd123-1a234-123b-12c3-1234567890b"}}.
7. Click Next: Permissions > Next: Tags > Next: Review.
8. In the Role Name field, type exactly the name of SplunkDMReadOnly and click Create role.
9. Click SplunkDMReadOnly.
10. Under the Permissions tab, click Add inline policy.
    1. Click theJSON tab.
    2. Overwrite the JSON text by copying and pasting the Role Policy from the Data Manager UI.
    3. Replace the <CONTROL_ACCOUNT_ID> variables with your control account ID.
    4. Click Review Policy.
    5. In the Name field, type any name of your choice, such as SplunkDMReadOnlyPolicy.
    6. Click Create policy.
11. Under the Trust relationships tab, click Edit trust relationship.
    1. Update the AWS principal, such as "arn:aws:iam::123456789012:role/cfgh-d12345-12345" from the Data Manager UI.
    2. Click Update Trust Policy.

Configure through theCLI

Prepare the terminal to use the AWS credentials that allow you to run the following CLI command against your control account. See AWS CLI Prerequisites.

1. Create the SplunkDMReadOnly role, replacing the <CONTROL_ACCOUNT_ID> variables with your control account ID and replacing the <EXTERNAL_ID> variable from the Trust Relationship in the Data Manager UI:
   aws iam create-role --role-name SplunkDMReadOnly --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam::<CONTROL_ACCOUNT_ID>:role/cfgh-d12345-12345"]},"Action":"sts:AssumeRole","Condition":{"StringEquals":{"sts:ExternalId":"<EXTERNAL_ID>"}}}]}'
2. Create the inline policy for SplunkDMReadOnlyPolicy and attach it to the role, replacing the <CONTROL_ACCOUNT_ID> variables with your control account ID:
   aws iam put-role-policy --policy-name SplunkDMReadOnlyPolicy --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["iam:GetRole","iam:GetRolePolicy","iam:ListRolePolicies","iam:ListAttachedRolePolicies","iam:GetPolicy","iam:GetPolicyVersion","cloudformation:DescribeStackSet","cloudformation:DescribeStacks","cloudformation:ListStackInstances","cloudformation:ListStackSetOperations"],"Resource":["arn:aws:cloudformation:*:<CONTROL_ACCOUNT_ID>:stack/SplunkDM*/*","arn:aws:cloudformation:*:<CONTROL_ACCOUNT_ID>:stackset/SplunkDM*:*","arn:aws:iam::<CONTROL_ACCOUNT_ID>:role/AWSCloudFormationStackSetAdministrationRole","arn:aws:iam::<CONTROL_ACCOUNT_ID>:policy/*"]}]}' --role-name SplunkDMReadOnly

Create a StackSet in your control account to push the SplunkDMReadOnly role to each of your data accounts

Download the CloudFormation Template that you will use in your control account to create a StackSet that will create this role in each data account. Select only one region for deployment, preferably US East (Virginia), but the region is your choice for the prerequisites. Do not deploy this template in more than one region. This role allows Splunk Cloud to read metadata from CloudTrail, Security Hub, GuardDuty, CloudFormation, Firehose, S3, lambda, events, and logs.

Configure through the console

Complete the following steps in the AWS console.

1. Download the template from the Data Manager UI.
2. Log into your control account.
3. Navigate to CloudFormation > StackSets.
4. Click Create StackSet from any region, preferably US East (Virginia).
5. Under Permissions, select the following:
   • IAM role name: AWSCloudFormationStackSetAdministrationRole
   • IAM execution role name: AWSCloudFormationStackSetExecutionRole
6. Click Template is ready.
7. Click Upload a template file and choose the file you downloaded. File name cannot contain parenthesis.
8. Click Next.
9. Name the StackSet such as SplunkDMReadOnly, and click Next.
10. Click Next.
11. Under Account numbers, provide a comma-separated list of all your data account IDs.
12. Under Specify regions, specify any region, preferably US East (Virginia).
13. Under Deployment options, set the Maximum concurrent accounts to the number of data accounts that you're using.
14. Click Next.
15. Check the check box for I acknowledge that AWS CloudFormation might create IAM resources with custom names.
16. Click Submit.

AWS data source prerequisites

Some data sources only need to be selected during onboarding, but others need to be configured ahead of time.

Configure CloudTrail

If you select CloudTrail as a data source, you need to make sure that your AWS CloudTrail is configured to send its data to a CloudWatch log group for the accounts and regions that you select. See Sending Events to CloudWatch Logs.

Configure CloudFormation Access logs

If you select CloudFormation Access logs as a data source, you need to make sure that youConfigure S3 to send notifications to SQS. This lets S3 notify the add-on that new events were written to the S3 bucket for the accounts and regions that you select.

Configure IAM Access Analyzer

If you select IAM Access Analyzer, it needs to be enabled in every region where you want to monitor access to your resources. See Enabling Access Analyzer.

Configure Load Balancer Access logs

If you select Load Balancer Access logs as a data source, you need to make sure that your AWS Load Balancer Access logs are configured to send its data to a CloudWatch log group for the accounts and regions that you select.

Configure Security Hub or GuardDuty

If you select Security Hub or GuardDuty, you need to make sure that your AWS Security Hub or GuardDuty is enabled for the accounts and regions that you select. See Enabling Security Hub and Enable Amazon GuardDuty.

AWS CLI Prerequisites

You need AWS CLI version 2 to run the commands, such as the following:

$ aws --version
aws-cli/2.0.4 Python/3.8.2 Darwin/19.6.0 botocore/2.0.0dev8

The aws2 dev version is not supported.

There are numerous ways to prepare your terminal to use the credentials for your data account. Use the AWS documentation for details about configuring your CLI terminal with credentials to run AWS commands. See Configuring the AWS CLI.