Configure Custom Logs in Data Manager

In your Amazon Web Services (AWS) deployment, use Amazon CloudWatch Logs to store, access and monitor logs from custom log sources. In Data Manager, use the Amazon CloudWatch Logs Custom Logs data source to ingest AWS custom logs into your Splunk Cloud platform instance.

For more information see the Enabling logging from certain AWS services topic in the AWS documentation.

Configure custom source types in Data Manager

A custom source type is a default field that identifies the data structure of an event. A source type determines how the Splunk platform formats the data during the indexing process.

Your custom source type serves as the source type for events ingested through this input. Custom source types are only supported by the Custom Logs data source. The aws:cloudwatchlogs: prefix is added to the beginning of your custom source type by default.

Configure log groups in Data Manager

Onboard log groups by specific log group names, or bulk ingest all log groups by region, or by selected common log group prefixes. Log groups cannot be onboarded more than once.

Create a log group in CloudWatch Logs

A log group is created when you install a CloudWatch Logs agent on an Amazon EC2 instance process. Log groups can also be created in the CloudWatch console.

CloudWatch Logs automatically receive log events from some AWS services. Users can also send log events to CloudWatch Logs.

For more information, see the Working with log groups and log streams topic in the Amazon CloudWatch Logs user guide.

Configure Custom Logs in Data Manager

Perform the following steps to configure custom logs in Data Manager

Click Expand to review the steps to configure custom logs in Data Manager

1. On the Data Management page, click New Data Input.
   
   
2. On the Choose Cloud Data Platform page, select Amazon Web Services, and click Next.
   
   
3. On the AWS Data Onboarding page, select Amazon CloudWatch Logs - Custom Logs, and click Next.
4. On the Prerequisites for Onboarding Amazon CloudWatch Logs - Custom Logs page,
   
   
   1. Navigate to the Create a Role step.
      1. Click View Role Policy, and review the sample role policy.

      
      
      
      

      2. Click View Trust Relationship and review the sample trust relationship.
5. Click Next.
6. On the Input Amazon CloudWatch Logs Data Information - Custom Logs page,
   1. Enter a Data Input Name.
   2. Enter an AWS Data Account ID.
   3. In the Selected Data Sources section, select a data destination for your Custom Logs from the dropdown menu.
   4. In the Select Regions section, select the us-east-1 region.
   5. In the Enter a Custom Source Type section, enter a custom source type name. The aws:cloudwatchlogs: prefix is added to the beginning of your custom source type by default.
   6. In the Onboard log groups section, click Add groups.
      1. On the Onboard log groups page, select the log groups that you want to onboard from the dropdown menu for each available region.
         
         
      2. Click Save.
   7. Click Review Data Input
7. On the Review Data Input page, review your data input selections, and click Next.
8. On the Setup Data Ingestion page,
   1. Navigate to the Download the CloudFormation Stack Template section, and click the Data Ingestion Template button to download the CloudFormation Stack Template that you will run in every region in your AWS deployment to establish resources for sending data from that region.
   2. In the Choose a Method to Run the Template on Your Accounts and Regions section, select either the AWS CLI or the AWS Console method, and perform the listed steps in order to run the template on your AWS account and regions.
      AWS CLI steps
      
      
      AWS Console steps
      
      
      If you choose the AWS Console method, navigate to step four, and copy the listed Stack Name, which will be used when you navigate to your AWS deployment to create your CloudFormation stack.
   3. Once you have created your CloudFormation stack, and have run the CloudFormation template on your accounts and regions, click Review Finish Setup and Monitor Data Input.
9. On the Data Management page, you can see the status of your data input.