Splunk Cloud

Splunk Cloud User Manual

Download manual as PDF

Download topic as PDF

Manage Splunk Cloud indexes

Splunk Cloud administrators create indexes to organize data, apply role-based access permissions to indexes that contain relevant user data, fine-tune data, specify how long to retain data in indexes, and so on.

Indexes store the data you have sent to your Splunk Cloud deployment. To manage indexes, Splunk Cloud administrators can perform these tasks:

  • Create, update, delete, and view properties of indexes.
  • Monitor the size of data in the indexes to remain within the limits of a data plan or to identify a need to increase the data plan.
  • Modify data retention settings for individual indexes to control when Splunk Cloud automatically deletes data or moves it to storage.
  • Optimize search performance by managing the number of indexes and the data sources that are stored in specific indexes.
  • Delete indexes. Caution: This function deletes all data from an index and removes the index. The operation is final and can't be reversed.
  • Move expired data from indexes to self storage or a Splunk-supported archive (Dynamic Data Active Archive). Data from the index is not deleted until it is successfully moved to the storage location. Archived data can be restored to Splunk Cloud for searching. Data from a self storage location can no longer be searched from Splunk Cloud. However, it can be restored to a Splunk Enterprise instance for searching if necessary.

Best practices for creating indexes

Consider these best practices when creating indexes:

  • Create separate indexes for long-term and short-term data. For example, you might need to keep security logs for one year but web access logs for only one month. Using separate indexes, you can set different data retention times for each type of data.
  • Apply logical or role-based boundaries for indexes. For example, create separate indexes for different departments.
  • Devise a naming convention to easily track, navigate, and organize indexes.
  • To configure your data retention settings, see the best practice listed here: Manage Data Retention Settings.

The Indexes page

To view the Indexes page, select Settings > Indexes. The Indexes page lists the indexes in a Splunk Cloud deployment and lets administrators to create, update, delete, and modify the properties of indexes. To modify settings for an index, click its name.

From this page you can:

  • Create an index.
  • View index details such as the following.
    • Index name: The name specified when the index was created.
    • Current size: The approximate amount of data currently stored in the index.
    • Max size: The maximum amount of raw data (in TB, GB, or MB) retained in the index.
    • Event count: The number of events in the index.
    • Earliest event: The earliest event found in the index.
    • Latest event: The most recent event found in the index.
    • Searchable Retention: The maximum age of events retained in the index.
    • Status: Enabled or disabled. Data in a disabled index is ignored in searches.
    • Storage Type: The storage settings for expired data from a given index. Can be self storage, archive, or no additional storage.
    • Delete an index. Caution: Deletes all data from an index and removes the index. The operation is final and can't be reversed.

Create a Splunk Cloud index

To create an index:

  1. Select Settings > Indexes.
  2. Click New.
  3. In the Index Name field, specify a unique name for the index. Names must begin with a lowercase letter or a number and can include uppercase letters, hyphens, and underscores.
  4. In the Max raw data size field, specify the maximum amount of raw data allowed before data is removed from the index. Set this value to zero to specify an unlimited maximum raw data size.
  5. In the Retention (Days) field, specify the number of days before an event is removed from an index.
  6. In the Dynamic Data Storage field, select Splunk Archive to send data to the Splunk Dynamic Data Active Archive, or choose Self Storage to move expired data to your own self-storage area. If you don't want to maintain expired Splunk data, leave No additional storage selected.
  7. If you enabled data self storage, select a location for data self storage. Or, click Edit self storage locations to add a new self storage location. For more information about data self storage and instructions for configuring a data self storage location, see Store expired Splunk Cloud data.
  8. If you enabled Dynamic Data Active Archive, configure retention settings for the archive. For more information, see Archive expired Splunk Cloud data.
  9. Click Save.

The index appears after you refresh the page. Retention settings are applied to individual indexes, and data retention policy settings apply to all of the data that is stored in your Splunk Cloud deployment. Monitor and verify that the data retention settings for all indexes does not meet or exceed the values set in the data retention policy. For more information, see Splunk Cloud data policies.

Manage data retention settings

Each index uses two settings to determine when to delete data:

  • The maximum size of the raw index data (MB, GB, or TB, specified in the Max raw data size field)
  • The maximum age of events in the index (specified in the Retention (days) field)

When the index reaches the specified maximum size or events reach the specified maximum age, the oldest data is deleted or is moved to your self-storage location (depending on your configuration).

For example, you ingest data from a particular datasource at a rate of 10 GB per day, and you want to retain and search against the last 90 days worth of data. Given your search and data retention requirements, you should set the values so that the Retention (days) value is reached before the Max raw data size threshold is reached. Given the above parameters, you might configure the retention settings to the following.

  • Max raw data size set to 1800 GB
  • Retention (days) set to 90

The graphic shows data retention settings for an index. It is intended to orient the user.

These values together account for both your ingestion rate and the time you want to retain the data. You will need to consider these factors for each index that you create.

Finally, it's a good idea to check your data retention in the Cloud Monitoring Console to ensure you estimated your ingestion rate correctly and your storage consumption is within your entitlement. If you did not correctly estimate your ingestion rate, you might have a shorter retention period than expected.

For more information about data self storage and instructions for configuring a data self storage location, see Store expired Splunk Cloud data.

For more information about archiving data, see Archive expired Splunk Cloud data.

Splunk Cloud administrators can specify the settings that determine when data is removed from a specific index as follows.

For more information about data self storage and instructions for configuring a data self storage location, see Store expired Splunk Cloud data.

The new data retention settings appear after you refresh the page.

Delete index data and the index from Splunk Cloud

Splunk Cloud administrators can delete an index.

Caution: This function deletes all data from an index and removes the index. The operation is final and can't be reversed.

  1. Select Settings > Indexes.
  2. Identify the index and click Delete from the Action column.
  3. Click OK to confirm that you want to delete the data and index from Splunk Cloud.

The data and index are deleted from Splunk Cloud and can't be restored. Note: You can't delete default indexes and third-party indexes from the Indexes page.

Last modified on 02 May, 2020
Monitor Splunk Cloud deployment health
Set limits for concurrent scheduled searches

This documentation applies to the following versions of Splunk Cloud: 8.0.2001, 8.0.2003, 8.0.2004

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters