Troubleshoot the Splunk App for AWS

Isolating the component with the problem

The Splunk App for AWS relies on the Splunk Add-on for Amazon Web Services for input collection and knowledge management. When troubleshooting, determine whether the issue you are experiencing is relevant to the app or to the add-on. In general, if your AWS data is successfully reaching your Splunk indexes, the issue is with the app. If data is not reaching your Splunk indexes, then you should check the following:

• Configuration problems with the accounts and inputs handled by the Splunk Add-on for Amazon Web Services. See Troubleshoot the Splunk Add-on for AWS for troubleshooting specific to the add-on.
• Insufficient IAM permissions in your Amazon Web Services account or EC2 IAM role. See Configure your AWS permissions for the Splunk App for AWS.
• Conflicts between the app and add-on configurations, such as advanced settings shown only in the add-on.
• (On-premises customers only) Problems with the remote target configuration between your search head and forwarder, if used. See Install the Splunk App for AWS on Splunk Enterprise.

Some dashboards temporarily fail to display data after upgrade

After you upgrade the Splunk App for AWS to version 5.0.0, data is temporarily unavailable for these dashboards that were previously well-displayed: Topology, VPC Flow Log, and CloudFront Access Log. This is because these dashboards are powered by some saved searches that are introduced in the new release and are not scheduled to run by default. This problem will automatically go away when Splunk runs the scheduled saved search App Upgrader that comes with the Splunk App for AWS 5.0.0. You can also manually run the App Upgrader saved search to resolve the dashboard display problem right away. Running App Upgrader updates the Detailed Billing data model and schedules the execution of the saved searches powering these dashboards.

Custom dashboards fail to display data properly after upgrade

After you upgrade the Splunk App for AWS to a newer release, custom dashboards you modified and saved to local in the previous version override the dashboards that come with the new version and may use out-of-date macros and not display data correctly. To resolve this issue, delete local copies of the affected dashboards.

Advanced settings

Depending on where you have deployed the app, you may have access to advanced settings.

If you are using the Splunk App for AWS on an on-premises instance of Splunk Enterprise or on a Splunk Cloud instance with a URL of the pattern https://prd-*.cloud.splunk.com, you can access more settings by editing the Splunk Add-on for Amazon Web Services data inputs directly. For details, see Configure inputs for the Splunk Add-on for AWS in the Splunk Add-on for AWS manual, part of the Supported Add-ons documentation.

If you are using the Splunk App for AWS on a Splunk Cloud instance with a URL of the pattern https://*.splunkcloud.com, you cannot access these settings.

Dashboards not showing data from custom indexes

If you configure input using custom indexes on the Splunk Add-on for AWS rather than Splunk App for AWS, you need to enable saved search "Update Macros". It will automatically update the macro searches used in dashboards based on the custom indexes configured in the inputs.

Or, you can manually update your local/macros.conf file to specify which indexes the app dashboards should search.

If you use the Configure screen in the app, the app will update the macros automatically.

See Macros for the Splunk App for AWS for more information.

Topology dashboard shows no data

If your Topology dashboard shows no data, first verify that you are using an account that has access to AWS Config service.

If you use a clustered distributed Splunk deployment, you need to perform some additional steps:

• Configure the search head tier to directly forward data to the indexer tier.
• Distribute the summary index configuration bundle across clustered indexers.

For detailed instructions, see Install in a clustered distributed environment.

If you have previous AWS Config data before upgrade, you need to manually run saved search "Config: Topology History Generator", which will migrate previous AWS Config data before update to summary index.

Then, check that the required saved searches are enabled. The topology dashboard requires data from a set of saved searches that you can find in the app under Search > Reports. These searches runs every hour and help populate your Topology dashboard. If you configure your inputs through the app, the saved search is automatically enabled and scheduled. If, however, you configure your inputs through the add-on instead, you need to manually enable and schedule the saved searches.

See Saved searches for the Splunk App for AWS for more information.

Accessing logs

You can access internal log data for help with troubleshooting by searching by source type. See Troubleshoot the Splunk Add-on for AWS for information about accessing add-on logs.

Billing metric not available for CloudWatch

If you do not see the Billing namespace listed on the input configuration page for CloudWatch, check that you have turned on Receive Billing Alerts in the Preferences section of the AWS Billing and Cost Management console.

VPC Flow data model performance or bundle size issues

If you have high volume VPC flow logs that you are ingesting through the Splunk Add-on for AWS's Kinesis input, you may find that the data model acceleration takes too long and/or the replication bundle becomes too large. To mitigate this, you can decrease the VPC Flow data model summary range to one day instead of the seven day default.

S3 input performance issues

You can configure multiple S3 inputs for a single S3 bucket to improve performance. The Splunk platform dedicates one process for each data input, so provided that your system has sufficient processing power, performance will improve with multiple inputs.

Note: Be sure that multiple inputs do not collect the same S3 folder and file data, to prevent indexing duplicate data.

Unexpected termination of S3 dashboard saved searches

Some saved searches powering S3 dashboards (Data Events and Traffic Analysis) terminate unexpectedly due to insufficient memory caused by too many concurrent searches. To resolve this issue, consider the following:

• Increase RAM on the indexer (better performance)
• If the indexer runs Linux, increase the swap size on the indexer (more cost-efficient)