Troubleshoot AWS IAM User data ingestion

Troubleshoot AWS IAM User data ingestion process.

AWS IAM User data cannot be found

AWS IAM User data cannot be found.

Cause

AWS IAM User is not configured correctly and data is not being ingested from AWS IAM Users.

Solution

1. Make sure there are IAM Users in the account and region you are working with, and you have waited at least 3 hours after you created the input.
2. Navigate to Data Management. Click the Data Input Details tab, and go to the Account Establishment Details section.
3. If a stack is in FAILED state, refer to Deployment Status: Failed for more troubleshooting steps.
4. Verify that the Splunk HTTP Event Collector (HEC) configuration is correct. Refer to Troubleshoot the HEC Configuration for more troubleshooting steps. Make sure the indexer acknowledgement is disabled for the HEC token of the input you are troubleshooting.
5. Verify that the data ingestion pipeline has been setup correctly in the account in the region us-east-1. There is one EventBridge rule you must check, the schedule rule SplunkDMMetadataIAMUsersScheduleRule which is created in the us-east-1 region even if you are onboarding other regions. This rule is triggered periodically every 3 hours to trigger the Lambda function that fetches the existing IAM Users data.
   1. In AWS console for us-east-1, navigate to Rules > Amazon EventBridge Service and verify that exists. T
   2. Verify that the target for this rule is set to the SplunkDMMetadataIAMUsers Lambda function and the status is Enabled.
   3. Verify that the Event Schedule for SplunkDMMetadataEC2NetworkAclScheduleRule is correct.
6. If the EventBridge Rule or Lambda Function does not exist, delete the Data Input on Splunk and recreate it.
7. If the data ingestion pipeline is setup correctly, click on Metrics for the rule and check when the event rule was last triggered.
8. Navigate to the Lambda console in the region and select SplunkDMMetadataIAMUsers. Verify that the Environment variables on the Lambda function match the Input ID and the HEC token configuration for that input.
9. If there is any discrepancy with this configuration, delete the Data Input and recreate it.
10. If the configuration is correct and your data cannot be found, debug the SplunkDMMetadataIAMUsers Lambda function.
    1. Select Monitor and verify that the Lambda function was invoked by looking at Invocations metrics. Make sure to select the appropriate time range.
    2. If the lambda was invoked in that time interval, then check the Throttles and Error count metrics. If any of the Throttles and Error count metrics is non-zero, check the logs of the Lambda function by clicking on View logs in CloudWatch.
11. If the configuration is correct and your data still cannot be found, Contact Splunk Support.