Troubleshoot AWS EC2 Instance data ingestion

Troubleshoot AWS EC2 Instance data ingestion process.

AWS EC2 Instance data cannot be found

AWS EC2 Instance data cannot be found.

Cause

AWS EC2 is not configured correctly and data is not being ingested from AWS EC2 instances.

Solution

1. Make sure that there are EC2 Instances in the account and region you are working with, and you have waited at least 3 hours after you created the input.
2. Navigate to Data Management. Click the Data Input Details tab, and go to the Account Establishment Details section.
3. If a stack is in FAILED state, refer to Deployment Status: Failed for more troubleshooting steps.
4. Verify that the Splunk HTTP Event Collector (HEC) configuration is correct. Refer to Troubleshoot the HEC Configuration for more troubleshooting steps. Make sure the indexer acknowledgement is disabled for the HEC token of the input you are troubleshooting.
5. Verify that the data ingestion pipeline has been setup correctly in the account and region. There are two EventBridge rules you must check, the pattern rule SplunkDMMetadataEC2InstPatternRule which is triggered only when an EC2 instance changes state to RUNNING, and the schedule rule SplunkDMMetadataEC2InstScheduleRule which is triggered periodically every 3 hours to trigger the Lambda function that fetches the existing EC2 instances data.
   1. Navigate to Amazon EventBridge console in the account and region, and under Rules verify that SplunkDMMetadataEC2InstPatternRule and SplunkDMMetadataEC2InstScheduleRule exist.
   2. Verify that the target for both rules is set to the SplunkDMMetadataEC2Inst Lambda function and the status is Enabled.
   3. Verify that the Event Pattern for SplunkDMMetadataEC2InstPatternRule and the Event Schedule for SplunkDMMetadataEC2InstScheduleRule are correct.
   4. If either of the EventBridge Rules or Lambda Functions do not exist, delete the Data Input and recreate it.
6. If the data ingestion pipeline is setup correctly, click on Metrics for the rule and check when the event rule was last triggered.
7. Navigate to the Lambda console in the region and select SplunkDMMetadataEC2Inst. Verify that the Environment variables on the Lambda function match the Input ID and the HEC token configuration for that input.
8. If there is any discrepancy with this configuration, delete the Data Input and recreate it.
9. If the configuration is correct and your data still cannot be found, debug the SplunkDMMetadataEC2Inst Lambda function.
   1. Select Monitor and verify that the Lambda function was invoked by looking at Invocations metrics. Make sure to select the appropriate time range.
   2. If the Lambda function was invoked in that time interval, then check the Throttles and Error count metrics. If any of the Throttles and Error count metrics is non-zero, check the logs of the Lambda function by clicking on View logs in CloudWatch.
10. If the configuration is correct and your data still cannot be found, Contact Splunk Support.