Splunk® Supported Add-ons

Splunk Add-on for AWS

Download manual as PDF

Download topic as PDF

Access billing data for the Splunk Add-on for AWS

The Splunk Add-on for Amazon Web Services supports extracting generic data from your S3 buckets. One example of data that you might want to extract from your S3 buckets is your AWS billing report data. Use the billing input to collect your AWS billing reports, then extract useful information from them using pre-built reports included with this add-on. The add-on's pre-built reports are based on the AWS report formats to make it easier for you to access and work with this data. You can use these reports as examples of how to use the Splunk platform to explore your other S3 data.

The Billing input does not collect billing reports for your AWS Marketplace charges.

Billing report collection behavior

Amazon Web Services offers four distinct billing reports, which you can read more about in the AWS documentation: http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/detailed-billing-reports.html#d0e2817. You can collect any of the four types of billing reports using the Splunk Add-on for AWS.

The Amazon Billing service updates all four of these reports continuously over the month until they are finalized a few days after the last day of the month. Until the reports are finalized, the daily updates add new billing information for the most recent day's activity and update previously reported line items. For example, the billing service might recalculate pricing discounts for events that already passed.

Because any portion of the data in the report might be updated, the add-on pulls the entire billing report every time it pulls any of the four report types. The Monthly report and Monthly cost allocation report are quite small, so the add-on retrieves these daily by default, resulting in many copies of the same (small) report. Detailed reports, however, are very large, so the add-on collects these reports only for months prior to the current month.

By default, the add-on collects all available reports that match the collection criteria for all available months. For example, assume that you install the Splunk Add-on for AWS on February 15, 2016 and configure your inputs to collect both Monthly cost allocation reports and Detailed billing reports with resources and tags. On the first day, the add-on collects the Monthly report for February 2016 current up to February 16, 2015 and the monthly reports for all previous months available for your account. The add-on also collects the Detailed report for January 2016, as well as the Detailed report for all previous months available for your account. On every subsequent day, the add-on downloads a new version of the Monthly report, current up to the latest day. It also checks the etag of the most recently completed month's (in this case, January's) Detailed report against the latest January Detailed report available from the Amazon Billing service. If they are the same, it does not download a Detailed report again until the month ends. On March 1, the add-on begins downloading Detailed reports for the month of February every day until the etag for that report is unchanged.

See the guidelines for configuring billing inputs for details of how to use cron schedules and regex to control the number of reports that you collect.

When you want to access any of the report data in the Splunk platform, use the reports included with the add-on to view the latest snapshots. You cannot adjust the reports.

Billing report types

Monthly report

The Monthly report lists AWS usage for each product dimension used by an account and its IAM users in monthly line items. You can download this report from the Bills page of the Billing and Cost Management console.

File Name Format: <AWS account number>-aws-billing-csv-yyyy-mm.csv

This report is small in size, so the add-on pulls the entire report once daily to get the latest snapshot.

Monthly cost allocation report

The Monthly cost allocation report contains the same data as the monthly report, but it also includes any cost allocation tags that you have created. You must obtain this report from the Amazon S3 bucket that you specify. Standard AWS storage rates apply.

File Name Format: <AWS account number>-aws-cost-allocation-yyyy-mm.csv

This report is small in size, so the add-on pulls the entire report once daily to get the latest snapshot.

Detailed billing report

The Detailed billing report lists AWS usage for each product dimension used by an account and its IAM users in hourly line items. You must obtain this report from the Amazon S3 bucket that you specify. Standard AWS storage rates apply.

File Name Format: <AWS account number>-aws-billing-detailed-line-items-yyyy-mm.csv.zip

This report can grow very large, so the add-on collects the report only after the month has ended. The add-on continues to collect the report once per day until it is finalized by Amazon billing services.

Detailed billing report with resources and tags

The Detailed billing report with resources and tags contains the same data as the detailed billing report, but also includes any cost allocation tags you have created and ResourceIDs for the AWS resources used by your account. You must obtain this report from the Amazon S3 bucket that you specify. Standard AWS storage rates apply.

File Name Format: <AWS account number>-aws-billing-detailed-line-items-with-resources-and-tags-yyyy-mm.csv.zip

This report can grow very large, so the add-on collects the report only after the month has ended. The add-on continues to collect the report once per day until it is finalized by Amazon billing services.

Event types for billing data

The Splunk platform indexes two types of billing data using the sourcetype aws:billing:

  • Monthly reports, which have the event type aws_billing_monthly_report
  • Detailed reports, which have the event type aws_billing_detail_report

Access the add-on's preconfigured reports

The Splunk Add-on for Amazon Web Services includes several reports based on the indexed billing report data. You can find these saved reports in Splunk Web by clicking Home > Reports and looking for items with the prefix AWS Bill - . Some of the saved searches return a table. Others return just a single value, such as AWS Bill - Total Cost till Now.

The Splunk platform typically indexes multiple monthly report snapshots, because AWS places multiple snapshots of the current month's billing report into the S3 bucket. To obtain the most recent monthly report snapshot click Home > Reports and open the saved report called AWS Bill - Monthly Latest Snapshot. Or, search for it using the search string: | savedsearch "AWS Bill - Monthly Latest Snapshot"

You can obtain the most recent detailed report by clicking Home > Reports and opening the saved report called AWS Bill - Daily Cost. Or, search for it using the search string | savedsearch "AWS Bill - Daily Cost". Searching against detailed reports can be slow due to the volume of data in the report. Splunk recommends accelerating the searches against detailed reports.

Report sources

These saved reports are based on AWS Billing Reports instead of the billing metric data in CloudWatch. By default, Total or Monthly reports are based on data indexed from the AWS Monthly Reports (*-aws-billing-csv-yyyy-mm.csv or *-aws-cost-allocation-yyyy-mm.csv) on the S3 bucket, while Daily reports are based on AWS Detail Reports (*-aws-billing-detailed-line-items-yyyy-mm.csv.zip or *-aws-billing-detailed-line-items-with-resources-and-tags-yyyy-mm.csv.zip).

Default index behavior

By default, reports look for data in the default index, main. If you changed the default index when you configured the data input, the reports will not work unless you include the index in the default search indexes list or change the two reports so they filter to the custom index.

To include a custom index in the default search indexes list, perform the following tasks:

  1. Click Settings > Users and authentication > Access controls > Roles > [Role that uses the saved searches] > Indexes searched by default.
  2. Include the custom index in default search indexes list.
  3. Repeat for each role that uses the saved searches.

To change the saved searches to filter to a custom index:

  1. Open the saved search AWS Bill - Monthly Latest Snapshot.
  2. Add a filter to specify the index you configured. For example, index=new_index.
  3. Save your changes to the saved search.
  4. Repeat these steps for the other saved search, AWS Bill - Detailed Cost.
PREVIOUS
Troubleshoot the Splunk Add-on for AWS
  NEXT
Lookups for the Splunk Add-on for AWS

This documentation applies to the following versions of Splunk® Supported Add-ons: released


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters