Troubleshoot the Content Pack for Amazon Web Services Dashboards and Reports

The Content Pack for Amazon Web Services Dashboards and Reports relies on the Splunk Add-on for AWS for input collection and knowledge management. When troubleshooting, determine whether the issue you are experiencing is relevant to the content pack or to the add-on.

In general, if your AWS data successfully reaches your Splunk platform indexes, the content pack requires troubleshooting. If data is not reaching your Splunk platform indexes, then check for configuration problems with the accounts and inputs handled by the Splunk Add-on for AWS.

Here are some common issues in the Content Pack for Amazon Web Services Dashboards and Reports and how to resolve them.

Access logs

You can access internal log data for help with troubleshooting by searching by source type. See Troubleshoot the Splunk Add-on for AWS for information about accessing add-on logs.

The content pack isn't working as expected

Problem

The content pack might not work as expected in some of the following ways:

• Configurations aren't reflected
• The tag filter doesn't populate after configuring the tags
• Two data models have the same name

Cause

There is a conflict with the knowledge objects in the content pack and the Splunk App for AWS.

Solution

Check if the Splunk App for AWS is enabled on the same instance and disable it. If the Splunk App for AWS is not disabled, the Content Pack for Amazon Web Services Dashboards and Reports uses the configurations and the knowledge object definitions from the app. Enabling the app results in a knowledge object conflict.

The Reserved Instance Planner dashboard isn't loading

Problem

The dashboard takes a long time to load.

Cause

Logging on as a normal user causes the dashboard to reach the limit for concurrent searches.

Solution

1. In Splunk Web, log in as an admin user.
2. Go to Settings > Users.
3. Edit the specified user and add the power role from the Assign Roles box.
4. Click Save.

Notable CloudTrail Activity by Origin Map panel doesn't populate

Problem

In the Overview Dashboard, the Notable CloudTrail Activity by Origin Map panel doesn't display data.

Cause

If the Content Pack for Amazon Web Services Dashboards and Reports is installed with the Splunk App for Infrastructure (SAI) on the same instance, this causes a conflict with the [aws:cloudtrail] stanza in the props.conf file for SAI.

Solution

Update the [aws:cloudtrail] stanza in the props.conf file:

1. Navigate to splunk_app_infrastructure/local.
2. Add the following stanza to the props.conf file:

   [aws:cloudtrail]
   KV_MODE = json
   

3. Save the file.

Splunk App for AWS and the Content Pack for Amazon Web Services Dashboards and Reports do not work on the same instance

On January 5, 2022, the Splunk App for AWS will reach its end of life. After this date, Splunk will no longer maintain or develop this product. The functionality in this app is migrating to the Content Pack for Amazon Web Services Dashboards and Reports.

Problem

If the Splunk App for AWS and the Content Pack for Amazon Web Services Dashboards and Reports are installed on the same search head cluster or standalone instance, you might see the following issues occur:

• The Configure dashboard won't work on the same instance that the content pack is installed.
• The Billing tag configuration in the content pack conflicts with the app.

Cause

There is a conflict with the knowledge objects in the content pack and the Splunk App for AWS.

Solution

Remove or disable the Splunk App for AWS from the search head. As a best practice, do not use the Splunk App for AWS and the Content Pack for Amazon Web Services Dashboards and Reports on the same instance.

To learn how to migrate from an implementation of the app to the content pack, see Migrate from the Splunk App for AWS to the Content Pack for Amazon Web Services Dashboards and Reports.

Dashboards not showing data from custom indexes

Problem

The content pack dashboards do not populate with data from custom indexes.

Cause

If you configure inputs using custom indexes, macros that support dashboard performance must be updated to include the custom indexes. By default, the content pack runs a saved search called Addon Synchronization every hour that automatically updates the macros to include custom indexes you specified when configuring inputs. If you don't see data in your dashboards, the search might not have run yet.

Solution

You can manually run the Addon Synchronization saved search to immediately update the macros. Alternatively, you can update the macros.conf file to specify the indexes that the app dashboards search:

• To learn about editing .conf files, see How to edit a configuration file.
• For more information on the available macros, see Macros for the Splunk App for AWS.

Billing metric not available for Amazon CloudWatch

Problem

The Billing namespace is not listed on the input configuration page for Amazon CloudWatch.

Cause

Billing alerts might not be enabled.

Solution

Check that you enabled Receive Billing Alerts in the Preferences section of the AWS Billing and Cost Management console.

S3 input performance issues

You can configure multiple S3 inputs for a single S3 bucket to improve performance. The Splunk platform dedicates one process for each data input, so if your system has sufficient processing power, performance improves with multiple inputs.

Be sure that multiple inputs do not collect the same S3 folder and file data to prevent indexing duplicate data.

Billing dashboards are not loading data

Problem

The Billing dashboards don't load data, and tags are not getting populated in the Billing Tags dashboards. .

Cause

Saved searches need to run in order to populate tags in the Billing Tags dashboards.

Solution

1. In Splunk Web, go to Settings > Searches, reports, and alerts.
2. To find the saved searches, select AWS Dashboards and Reports in the App: drop-down list.
3. Run the saved searches called Billing: Billing Reports S3Key Generator and Billing CUR: Billing Reports AssemblyId Generator
4. Enable Schedule Report.
5. Specify a schedule to run each saved search.
6. When you're done, click Save and exit the saved search configuration.

Dashboards for Cost and Usage Report data are not populating

Problem

The Cost and Usage Report (CUR) data is not loading in the Billing dashboards.

Cause

The AWS Billing Management Console did generate an invoice yet.

Solution

If you are unable to see data, run this search with the time range picker set to All time:

`aws-billing-sourcetype-cur`| search InvoiceId=* | stats count by source

When the search completes successfully, you see at least one row of results. If you do not see any results, the AWS Billing Management Console did not generate an invoice yet. The AWS Billing Management Console typically generates an invoice on the fifth day of each month. For more information, see When will my AWS bill be ready?.

Confirm the following in your AWS console:

• The AWS CUR you use to monitor billing data in the content pack must have a Time granularity of Hourly or Daily and Report Versioning set to Create new version report. For more information about the time granularity of AWS CUR, see Creating an AWS Cost and Usage Report.
• You should have a CUR with at least one month of available data and a generated invoice. To learn more about this billing report type, see Configure Cost and Usage Report inputs for the Splunk Add-on for AWS.

If CUR data is not populated in billing dashboards but the AWS Billing Management already generated an invoice, rebuild the Detailed Billing CUR and Instance Hour CUR data models and check the dashboards again.

Performance of Billing dashboards is degraded

Problem

The Billing dashboards are loading slowly.

Cause

You need to enable data model acceleration. By default, data model acceleration is disabled when you install the content pack. For more information about accelerating data models, see Accelerate data models in the Splunk Enterprise Knowledge Manager Manual.

Solution

First, enable data model acceleration:

1. In Splunk Web, go to Settings > Data Models. Enable acceleration for every data model that belongs to the Content Pack for Amazon Web Services Dashboards and Reports.
2. Click Edit and select Edit Acceleration.
3. Select Accelerate and save the data model configuration.

Then, update the definition for the aws-data-model-acceleration search macro. This configures the search macro to search only data that is already accelerated, speeding up searches for dashboards and reports:

1. In Splunk Web, go to Settings > Advanced Search.
2. Select Search macros.
3. Select the aws-data-model-acceleration search macro.
4. Change the definition from summariesonly=f to summariesonly=t.
5. Save the search macro.

Finally, schedule the Addon Synchronization searches:

1. In Splunk Web, go to Settings > Searches, reports, and alerts.
2. To find the saved searches, select AWS Dashboards and Reports from the App: drop-down list.
3. Run the Addon Synchronization saved searches.
4. To configure schedules for the Addon Synchronization saved searches, click Edit under the Actions column and select Edit Schedule.
5. Enable Schedule Report.
6. Specify a regular schedule to run each saved search.
7. When you're done, Save and exit the saved search configuration.

The number of EC2 instances is different in the Topology dashboard

Problem

The number of EC2 instances might be different in other dashboards compared to the Topology dashboard.

Cause

There are a few reasons this might occur:

• The Topology - AWS dashboard uses the aws:config sourcetype to populate EC2 instances. The Overview dashboards use the aws:metadata sourcetype. The aws:config sourcetype is region-specific, and provides events only when certain actions are performed on an EC2 instance. The aws:metadata sourcetype is region-independent, and provides the current status of an EC2 instance.
• Dashboards that use the aws:metadata sourcetype track the status of EC2 instances, but it might not match EC2 instances in the Topology dashboard if the config service or config input in the Splunk Add-on for Amazon Web Services was disabled for a period of time.
• If you start using the Content Pack for Amazon Web Services Dashboards and Reports , you don't receive events for previously-created EC2 instances with the aws:config sourcetype unless you perform any action in an EC2 instance from the AWS Management Console, whereas the aws:metadata sourcetype provides all the current EC2 instances you are running.

Solution

There is no recommended solution for this problem.