Troubleshoot performance issues due to large KV Store collections

Search heads might slow down or crash if you exceed the storage capacity for KV Stores. Based on Splunk service limits, each KV Store collection can have a maximum size of 25 GB, and an entire KV Store can have a maximum size of 100 GB. See Service limits and constraints. Following are some tips to avoid exceeding the size limitations for KV Store collections:

• Do not write an excessive amount of data to your KV Store collections using scheduled saved searches.
• Do not retain an excessive amount of old data in your KV Store collections.

Follow these steps if your search heads crash or slow down due to large KV Store collections:

• Identify KV Store collections that exceed size limitations
• Identify data sources that write to KV Store collections
• Deactivate / Turn off saved searches that use specific KV Store collections
• Delete data in a KV Store collection
• Manage large KV Stores such as Access Tracker
• Add a retention policy to KV Store collections
• Increase storage space for KV Store collections
• Customize growth monitoring for KV Store collections

Identify KV Store collections that exceed size limitations

Use the following SPL search examples to identify and analyze the largest KV Store collections in your security operations center (SOC).

List KV Store collections by size (in bytes)

You can list the KV Store collections by size (in bytes) using the following search:

| rest splunk_server=local /services/server/introspection/kvstore/collectionstats | mvexpand data | spath input=data | rex field=ns "(?<app_name>.*)\\.(?<collection_name>.*)" | rename size as collection_size_in_bytes | eval time = now() | fields app_name, collection_name, collection_size_in_bytes, time | where collection_size_in_bytes > 0 | sort -collection_size_in_bytes

List KV Store collections by largest weekly average growth rate

You can list the KV Store collections by largest weekly average growth rate using the following search:

| inputlookup es_avg_rate_collection_size_growth | where avg_rate_of_size_increase > 0 | sort -avg_rate_of_size_increase

Visualize changes to the largest KV Store collections

You can visualize changes in the size of the largest KV Store collections over time using the following search:

When you run the following search, click the '''Visualization''' tab and select '''Line Chart''' as the visualization type.

| inputlookup es_daily_collection_size | chart first(collection_size_in_bytes) AS collection_size over time BY collection_name limit=<number of collections to display at once>

Visualize changes in a single KV Store collection type over time

You can visualize changes in a single KV Store collection size over time using the following search:

When you run the following search, click the '''Visualization''' tab and select '''Line Chart''' as the visualization type.

| inputlookup es_daily_collection_size | where collection_name=<collection-name> and app_name=<app-name> | chart first(collection_size_in_bytes) AS collection_size over time

Identify data sources for large KV Store collections

If specific KV Store collections display memory errors, identify the saved searches that might be writing data to the KV Store collections. Turn off the scheduled saved searches that are not useful and remove unnecessary data from the KV Store collection. Alternatively, you can also modify the scheduled saved searches to reduce the frequency with which data is written to the KV Store collections. You can modify the searches by reducing the time range of the data, removing a data source, and decreasing the size or number of fields.

Identify saved searches that use specific KV Store collections

Following is an example SPL search to identify the saved searches that use a specific KV Store collection:

| rest "services/saved/searches" | search search="*<your_collection_name>*" | fields title, description, search, disabled

This SPL search checks whether the KV Store collection name appears in the search. You can also customize this search by checking for built-in and custom macros in the macros.conf file that uses the KV Store collection.

Turn off saved searches that use specific KV Store collections

You can turn off saved searches that are not required using one of the following two methods:

• Use Splunk Web
• Edit the savedsearches.conf configuration file

Follow these steps to turn off the search using Splunk Web:

1. In the Splunk app, navigate to Settings > Searches, Reports and Alerts.
2. In the Name column, find your saved search.
3. Click Edit.
4. Click Deactivate / Turn off.

If the searches that write to a specific KV Store collection are not required and can be turned off, set the disabled flag to True or 1 in the savedsearches.conf configuration file.

You can also turn off the saved search by adding the search to your local savedsearches.conf configuration file using the following CURL command:

curl –location –request POST 'https://<host>:<mPort>/servicesNS/nobody/{app}/saved/searches' \ -k -u <username>:<password> \ –header 'Content-Type: application/x-www-form-urlencoded' \ –data-urlencode 'name=My Saved Search' --data-urlencode 'disabled=1' \ --data-urlencode 'owner=nobody' \ --data-urlencode 'description=description text' \ --data-urlencode 'search="index=main"' \ --data-urlencode 'dispatch.index_earliest=-7d' --data-urlencode 'dispatch.index_latest=now'

Delete data in a KV Store collection

Manually delete bulk data that exceeds a specific time threshold using an SPL search.

Deleting data from a KV Store collection is a temporary solution, since the KV Store collection might get populated again with new data unless the saved searches that write to it are turned off or modified.

.

Use the following example SPL search to completely delete a KV Store collection:

| outputlookup <collection-name> append=F

Use the following example SPL search to to keep a subset of the KV Store collection based on a specific condition:

| inputlookup <collection-name> | where <condition for data to keep> | outputlookup <collection-name> append=F

For more information on using lookup command, see lookup.

Add a retention policy to KV Store collections

Retention policies in Splunk Enterprise Security can automatically clear the data from specific KV Store collections based on the time field. Retention policies can be added for Splunk Enterprise Security KV Store collections in the managed_configurations.conf configuration file.

Following is an example of a retention policy added to a KV Store collection within the managed_configurations.conf configuration file. In this example, the retention key represents the retention policy. The retention policy checks the value for the time field for each row and deletes any rows where the value of the time field is older than the value of earliestTime, which is 10 days.

[lookup:collection_X] endpoint = /services/data/transforms/lookups/collection_X label = Label for collection X description = Some description for collection X editable = 0 lookup_type = reserved retention = {\

  "disabled": 0,\
  "earliestTime": "-10d",\
  "timeField": "time",\
  "timeFormat": "%s"\

}

Use the following CURL command to customize the retention policy for your KV Store collection:

curl –location –request POST 'https://<host>:<mPort>/servicesNS/nobody/{app}/configs/conf-{file}/{lookup:collection-name}' \ -k -u <username>:<password> \ –header 'Content-Type: application/x-www-form-urlencoded' \ –data-urlencode 'retention={"disabled": 0, "earliestTime": "-7d", "timeField": "time, "timeFormat": "%s"} --data-urlencode 'label=My Collection Label' \ --data-urlencode 'lookup_type=reserved' \ --data-urlencode 'endpoint=/services/data/transforms/lookups/collection_endpoint' \ --data-urlencode 'editable=0' \ --data-urlencode 'description=My collection description'

You can see the changes to the KV Store collection after cnfiguring the retention policies in the $SPLUNK_HOME/etc/apps/{app}/local directory, which overwrites the default settings for the KV Store collection. You do not need to restart your Splunk instance because an auto-reload occurs by default when a POST configuration request is made.

Increase storage space for KV Store collections

If you are unable to reduce the size or growth of your impacted KV Store collections, you can increase the storage capacity of your KV Store. Contact your Splunk administrator to allocate more storage space for your KV Store collection.

Manage large KV Store collections

Follow these steps to manage default KV Store collections in Splunk Enterprise Security such as Access Tracker:

• Use the Access Tracker (a built-in KV Store collection for Splunk Enterprise Security), if new user or destination combinations result in KV Store collections growing by multiple GBs on each day.
• Adjust the retention policy for the Access Tracker.
  The retention period for the Access Tracker KV Store collection is one year by default. However, you can reduce the data retention period for the KV Store collection from the Content Management page in Splunk Enterprise Security. See Add a retention policy to KV Store collections
• Turn off saved searches that use Access Tracker and clear all data in Access Tracker.
  Multiple saved searches such as 'Access - Authentication Tracker - Lookup Gen' are turned on by default and use the Access Tracker KV Store collection. You can identify the saved searches that use Access Tracker using the following SPL search:

  | rest "services/saved/searches" | search search="*access_tracker*" | fields title, description, search, disabled

  You can append the SPL search to identify custom or built-in macros in the macros.conf file that uses Access Tracker.

  If the saved searches that use Access Tracker are not required, they can be turned off by setting the disabled flag to True.

If no saved searches exist that use Access Tracker, clear the data in Access Tracker by running the following command:

| outputlookup access_tracker append=F

Customize growth monitoring for KV Store collections

Use the following default saved search ESS - KV Size Growth Rate Alert - Base to trigger alerts and monitor the growth of KV Store collections on a daily basis. You can modify the default saved search ESS - KV Size Growth Rate Alert - Base configuration or set up more alerts by editing the default saved search in local/savedsearches.conf.

You can also use Splunk Web to set up alerts.

1. Click Settings > Searches, Reports, and Alerts.
2. Click Edit on ESS - KV Size Growth Rate Alert - Base and change the properties as required.

For more information on troubleshooting KVStore collections, see Troubleshoot KV Store.