Troubleshoot AWS CloudTrail data ingestion

Troubleshoot the AWS CloudTrail data ingestion process.

CloudTrail logs are not enabled by default.

CloudTrail log data cannot be found

AWS CloudTrail log data cannot be found.

Cause

AWS CloudTrail is not enabled or is not configured correctly, or Splunk HEC is not configured correctly.

Solution

1. In AWS, navigate to CloudTrail in the AWS region selected when the data input was created in Data Manager.
2. If CloudWatch logging is not enabled for that CloudTrail, enable sending CloudTrail logs to CloudWatch. See the Sending events to CloudWatch Logs topic in the AWS documentation.
3. Navigate to Data Management. Click the Data Input Details tab, and go to the Account Establishment Details section. If a stack is in FAILED state, refer to Deployment Status: Failed for more troubleshooting steps.
4. Verify the Splunk HTTP Event Collector (HEC) configuration. See the HTTP Event Collector (HEC) configuration reference topic in this manual to troubleshoot Splunk software-side HEC configurations.
5. In AWS, navigate to Data Ingestion through Lambda functions to troubleshoot the Lambda function.
6. If the HEC token is present and enabled in the Splunk software, in AWS, navigate to Kinesis > Delivery streams.
7. Select SplunkDMCloudTrailDeliveryStream and verify that the status is active.
8. Click on the Configuration section and verify the source record transformation is enabled with SplunkDMCloudWatchLogProcessor as the Lambda function.
9. Navigate to CloudTrail, select the trail and make a note of the CloudWatch log group.
10. Navigate to the CloudWatch log group noted in the previous step.
11. Under Subscription filters, the Destination ARN should target to the Kinesis firehose delivery stream.
12. If any AWS resource is missing or misconfigured, delete the data input in Data Manager and recreate it.
13. If the configuration is correct and your data still cannot be found, Contact Splunk Support.