Troubleshoot the Splunk App for AWS

Isolating the component with the problem

The Splunk App for AWS relies on the Splunk Add-on for Amazon Web Services for input collection and knowledge management. When troubleshooting, determine whether the issue you are experiencing is relevant to the app or to the add-on. In general, if your AWS data is successfully reaching your Splunk indexes, the issue is with the app. If data is not reaching your Splunk indexes, then you should check for configuration problems with the accounts and inputs handled by the Splunk Add-on for Amazon Web Services. See Troubleshoot the Splunk Add-on for AWS for troubleshooting specific to the add-on.

Some dashboards temporarily fail to display data after upgrade

After you upgrade the Splunk App for AWS to version 5.2.0, data is temporarily unavailable for these dashboards that were previously well-displayed: Topology, VPC Flow Log, and CloudFront Access Log. This is because these dashboards are powered by some saved searches that are introduced in the new release and are not scheduled to run by default. This problem will automatically go away when Splunk runs the scheduled saved search App Upgrader that comes with the Splunk App for AWS 5.1.0. You can also manually run the App Upgrader saved search to resolve the dashboard display problem right away. Running App Upgrader updates the Detailed Billing data model and schedules the execution of the saved searches powering these dashboards.

Custom dashboards fail to display data properly after upgrade

After you upgrade the Splunk App for AWS to a newer release, custom dashboards you modified and saved to local in the previous version override the dashboards that come with the new version. The dashboards may use out-of-date macros and not display data correctly, and you may see this error message: The search specifies a macro "<macro_name>" that cannot be found. To resolve this issue, delete local copies of the affected dashboards.

Dashboards not showing data from custom indexes

If you configure inputs using custom indexes, macros that support dashboard performance must be updated to include the custom indexes. By default, the Splunk App for AWS runs a saved search called Addon Synchronization every hour that automatically updates the macros to include custom indexes you specified when configuring inputs.

You can also manually run the Addon Synchronization saved search to immediately update the macros.

See Saved searches for the Splunk App for AWS for more information.

Alternatively, you can update your local/macros.conf file to specify which indexes the app dashboards should search.

See Macros for the Splunk App for AWS for more information.

Topology dashboard shows no data

If your Topology dashboard shows no data, first verify that you are using an account that has access to AWS Config service.

If you use a clustered distributed Splunk deployment, you need to perform some additional steps:

• Configure the search head tier to directly forward data to the indexer tier.
• Distribute the summary index configuration bundle across clustered indexers.

For detailed instructions, see Install in a clustered distributed environment.

If you have previous AWS Config data before upgrade, you need to manually run saved search "Config: Topology History Generator", which will migrate previous AWS Config data before update to summary index.

Then, check that the required saved searches are enabled. The topology dashboard requires data from a set of saved searches that you can find in the app under Search > Reports. These searches runs every hour and help populate your Topology dashboard. If you configure your inputs through the app, the saved search is automatically enabled and scheduled. If, however, you configure your inputs through the add-on instead, you need to manually enable and schedule the saved searches. For more information, see Saved searches for the Splunk App for AWS.

If your Topology dashboard shows no data after upgrading, verify that you're storing data in the aws_topology_daily_snapshot index. You can run a search like this for the last 30 days:

index=aws_topology_daily_snapshot

If you aren't seeing any data in this index, check indexes.conf on your indexer tier to confirm whether the index exists. If it doesn't, create it now. For more information, see Create indexes and schedule saved searches for the Splunk App for AWS.

Accessing logs

You can access internal log data for help with troubleshooting by searching by source type. See Troubleshoot the Splunk Add-on for AWS for information about accessing add-on logs.

Billing metric not available for CloudWatch

If you do not see the Billing namespace listed on the input configuration page for CloudWatch, check that you have turned on Receive Billing Alerts in the Preferences section of the AWS Billing and Cost Management console.

S3 input performance issues

You can configure multiple S3 inputs for a single S3 bucket to improve performance. The Splunk platform dedicates one process for each data input, so provided that your system has sufficient processing power, performance will improve with multiple inputs.

Note: Be sure that multiple inputs do not collect the same S3 folder and file data, to prevent indexing duplicate data.

Unexpected termination of S3 dashboard saved searches

Some saved searches powering S3 dashboards (Data Events and Traffic Analysis) terminate unexpectedly due to insufficient memory caused by too many concurrent searches. To resolve this issue, consider the following:

• Increase RAM on the indexer (better performance)
• If the indexer runs Linux, increase the swap size on the indexer (more cost-efficient)

Billing dashboards fail to load

If billing dashboards are not populating with billing data, check to make sure data models are working properly and you are ingesting an invoice from the AWS Billing Management Console.

Billing data is indexed but dashboards are not loading

When billing data has been indexed but the Billing dashboards fail to load, check your billing data models (Detailed Billing, Detailed Billing CUR, Instance Hour, and Instance Hour CUR). If your billing data model keeps showing the Building status for a long time but never reaches 100%, try removing all the billing tags under the Configure menu. If this does not resolve the problem, try removing the data model definition from the local folder. Do not select more than eight billing tags. Too many billing tags may cause performance issues.

Cost and Usage Report data is not populating dashboards

If you select Billing (Cost and Usage Report) for the billing report type and are unable to see data in billing dashboards, run this search with the time range picker set to All time:

`aws-billing-sourcetype-cur`| search InvoiceId=* | stats count by source

When the search completes, you should see at least one row of results. If you do not see any results, the AWS Billing Management Console has not generated an invoice yet. The AWS Billing Management Console typically generates an invoice on the 5th day of each month. For more information, see When will my AWS bill be ready? on the Amazon website.

If CUR data is not being populated in billing dashboards but the AWS Billing Management generated an invoice already, rebuild the Detailed Billing CUR and Instance hour CUR data models and check the dashboards again.

Performance degraded on Billing dashboards after upgrading

If you notice performance degradation after upgrading the Splunk App for AWS, data model acceleration may be disabled and the aws-data-model-acceleration search macro definition may not specify to search only data that's already been accelerated. To enable data model acceleration and configure the app to search only data that's already been accelerated, see Do these things after you upgrade.

EC2 insights or recommendations fail to load

If you can't see the EC2 insights or recommendations on the EC2 Insights, Topology, or Insights Overview dashboards, make sure you're using Python for Scientific Computing version 1.1 or 1.2. If you're using version 2.0, follow the steps in Python for Scientific Computing to repopulate the dashboards.

The EC2 Recommendation Service feature is available for on-premises customers only and is not available on Splunk Cloud.

Unable to select a billing report type option in configure menu

If you don't follow the correct upgrade steps from version 6.0.1 to 6.0.2, the options to select your billing report type are disabled in the Configure dashboard. To remedy this issue, run the KV Store Synchronization saved search after upgrade in order for the billing report type checkboxes to be enabled. For instructions, see Upgrade the Splunk App for AWS.

Unable to see EC2 insights dashboard in an on-premises environment

After upgrading from version 6.0.1 to 6.0.2 of the Splunk App for AWS, if you can't see the EC2 Insights dashboard under the Insights menu, perform the following steps:

1. Within the Splunk App for AWS, click Search > Search.
2. Run the following query with a time range of All time:

   | outputlookup instance_type
   

3. Then run the following query with a time range of All time:

    | checkinstancetype
   

4. Reload your browser tab and confirm you can see EC2 Insights in the Insights menu.