Troubleshoot the Splunk App for AWS

Isolating the component with the problem

The Splunk App for AWS relies on the Splunk Add-on for Amazon Web Services for input collection and knowledge management. When troubleshooting, determine whether the issue you are experiencing is relevant to the app or to the add-on. In general, if your AWS data is successfully reaching your Splunk indexes, the issue is with the app. If data is not reaching your Splunk indexes, then you should check the following:

• Configuration problems with the accounts and inputs handled by the Splunk Add-on for Amazon Web Services. See Troubleshoot the Splunk Add-on for AWS for troubleshooting specific to the add-on.
• Insufficient IAM permissions in your Amazon Web Services account or EC2 IAM role. See Configure your AWS permissions for the Splunk App for AWS.
• Conflicts between the app and add-on configurations, such as advanced settings shown only in the add-on.
• (On-premises customers only) Problems with the remote target configuration between your search head and forwarder, if used. See Install the Splunk App for AWS on Splunk Enterprise.

Advanced settings

Depending on where you have deployed the app, you may have access to advanced settings.

If you are using the Splunk App for AWS on an on-premises instance of Splunk Enterprise or on a Splunk Cloud instance with a URL of the pattern https://prd-*.cloud.splunk.com, you can access more settings by editing the Splunk Add-on for Amazon Web Services data inputs directly. For details, see Configure inputs for the Splunk Add-on for AWS in the Splunk Add-on for AWS manual, part of the Supported Add-ons documentation.

If you are using the Splunk App for AWS on a Splunk Cloud instance with a URL of the pattern https://*.splunkcloud.com, you cannot access these settings.

Dashboards not showing data from custom indexes

If you configure custom indexes using the Splunk Add-on for AWS rather than using the Splunk App for AWS, you need to manually update your local/macros.conf file to specify which indexes the app dashboards should search. If you use the Configure screen in the app, the app will update the macros automatically.

Topology dashboard shows no data

If your Topology dashboard shows no data, first verify that you are using an account that has access to AWS Config service. This dashboard relies on AWS Config data, so if your account is in the China or GovCloud region, you cannot use this dashboard because the required service is not available to you in AWS at this time.

Next, check that the required saved searches are enabled. The topology dashboard requires data from a set of saved searches that you can find in the app under Search > Reports. These searches runs every hour and help populate your Topology dashboard. If you configure your inputs through the app, the saved search is automatically enabled and scheduled. If, however, you configure your inputs through the add-on instead, you need to manually enable and schedule the saved searches.

See Saved searches for the Splunk App for AWS for more information.

Accessing logs

You can access internal log data for help with troubleshooting by searching by source type. See Troubleshoot the Splunk Add-on for AWS for information about accessing add-on logs.

Billing metric not available for CloudWatch

If you do not see the Billing namespace listed on the input configuration page for CloudWatch, check that you have turned on Receive Billing Alerts in the Preferences section of the AWS Billing and Cost Management console.

VPC Flow data model performance or bundle size issues

If you have high volume VPC flow logs that you are ingesting through the Splunk Add-on for AWS's Kinesis input, you may find that the data model acceleration takes too long and/or the replication bundle becomes too large. To mitigate this, you can decrease the VPC Flow data model summary range to one day instead of the seven day default.

S3 input performance issues

You can configure multiple S3 inputs for a single S3 bucket to improve performance. The Splunk platform dedicates one process for each data input, so provided that your system has sufficient processing power, performance will improve with multiple inputs.

Note: Be sure that multiple inputs do not collect the same S3 folder and file data, to prevent indexing duplicate data.