Troubleshoot AWS EC2 Security Group data ingestion

Troubleshoot AWS EC2 Security Group data ingestion process.

AWS EC2 Security Group data cannot be found

AWS EC2 Security Group data cannot be found

Cause

AWS EC2 is not configured correctly and data is not being ingested from AWS EC2 Security Group.

Solution

1. Make sure that there are EC2 Security Groups in the account and region you are working with, and you have waited at least 3 hours after you created the input.
2. Navigate to Data Management. Click the Data Input Details tab, and go to the Account Establishment Details section.
3. If a stack is in FAILED state, refer to Deployment Status: Failed for more troubleshooting steps.
4. Verify that the Splunk HTTP Event Collector (HEC) configuration is correct. Refer to Troubleshoot the HEC Configuration for more troubleshooting steps. Make sure the indexer acknowledgement is disabled for the HEC token of the input you are troubleshooting.
5. Verify that the data ingestion pipeline has been setup correctly in the account and region. There are two EventBridge rules you must check, the pattern rule SplunkDMMetadataEC2SGPatternRule which is triggered when a new EC2 Security Group is created, and the schedule rule SplunkDMMetadataEC2SGScheduleRule which is triggered periodically every 3 hours to trigger the Lambda function that fetches the existing EC2 Security Groups data.
   1. Navigate to Amazon EventBridge console in the account and region and under Rules verify that SplunkDMMetadataEC2SGPatternRule and SplunkDMMetadataEC2SGScheduleRule exist.
   2. Verify that the target for both rules is set to SplunkDMMetadataEC2SG Lambda function and the status is Enabled.
   3. Verify that the Event pattern for SplunkDMMetadataEC2SGPatternRule and Event Schedule for SplunkDMMetadataEC2SGScheduleRule are correct.
6. If any of the above EventBridge Rules or Lambda Function do not exist, delete the Data Input and recreate it.
7. If the data ingestion pipeline is setup correctly, click on Metrics for the rule and check when the event rule was last triggered.
8. Navigate to the Lambda console in the region and select SplunkDMMetadataEC2SG. Verify that the Environment variables on the Lambda function matches the Input ID and the HEC token configuration for that input.
9. If there is any discrepancy with this configuration, delete the Data Input and recreate it.
10. If the configuration is correct and your data still cannot be found, debug the SplunkDMMetadataEC2SG Lambda function.
    1. Select Monitor and verify if the Lambda function was invoked by looking at Invocations metrics. Make sure to select the appropriate time range.
    2. If the Lambda function was invoked in that time interval, then check the Throttles and Error count metrics. If any of the Throttles and Error count metrics is non-zero, check the logs of the Lambda function by clicking on View logs in CloudWatch.
11. If the configuration is correct and your data still cannot be found, Contact Splunk Support.