Configure data models for Splunk Enterprise Security
Splunk Enterprise Security leverages data model acceleration to populate dashboards and views and provide correlation search results. The data models are defined and provided in the Common Information Model add-on (Splunk_SA_CIM), which is included in the Splunk Enterprise Security installation. Enterprise Security also installs unique data models that only apply to Splunk Enterprise Security content.
Data model acceleration search load
A data model is accelerated through a scheduled summarization search process initiated on the search head. The summarization search runs on the indexers, searching newly indexed data while using the data model as a filter. The resulting matches are saved to disk alongside the index bucket for quick access.
On Splunk platform 6.3 and later, up to two simultaneous summarization searches can run per data model, per indexer. For more information, see Parallel summarization in the Splunk Enterprise Capacity Planning Manual. To adjust parallel summarization settings on Splunk Cloud Platform, file a support ticket.
Constrain data model searches to specific indexes
The Splunk Common Information Add-on allows you to constrain the indexes searched by a data model for improved performance. See Set up the Splunk Common Information Model Add-on in the Splunk Common Information Model Add-on User manual.
Configure data model acceleration for CIM data models
The Splunk Common Information Add-on allows you to adjust your data model acceleration settings for each data model, including the backfill time, maximum concurrent searches, manual rebuilds, and scheduling priority. If you are using Splunk platform version 6.6.0, configure the tags whitelist setting to include any custom tags you use with CIM data models. See Accelerate CIM data models in the Splunk Common Information Model Add-on User manual.
Data model acceleration storage and retention
Data model acceleration uses the indexers for processing and storage, placing the accelerated data alongside each index. To calculate the additional storage needed on the indexers based on the total volume of data, use the formula:
Accelerated data model storage/year = Data volume per day * 3.4
This formula assumes that you are using the recommended retention rates for the accelerated data models.
For example, if you process 100GB/day of data volume for use with Enterprise Security, you need approximately 340GB of additional space available across all of the indexers to allow for up to one year of data model acceleration and source data retention.
Configuring storage volumes
By default, data model acceleration summaries reside in a predefined volume titled
_splunk_summaries at the following path:
$SPLUNK_DB/<index_name>/datamodel_summary/<bucket_id>/<search_head_or_pool_id>/DM_<datamodel_app>_<datamodel_name> . Data model acceleration storage volumes are managed in
indexes.conf using the
tstatsHomePath parameter. When configuring new storage volumes, the data model acceleration storage path defaults to the Splunk platform default index path of
$SPLUNK_HOME/var/lib/splunk unless explicitly configured otherwise. The storage used for data model acceleration is not added to index sizing calculations for maintenance tasks such as bucket rolling and free space checks.
To manage the data model acceleration storage independently of index settings, you must define a new storage path with
[volume:] stanzas. For an example of defining a volume and storing data model accelerations, see the Splunk platform documentation.
- For Splunk Enterprise, see Configure size-based retention for data models summaries in the Splunk Enterprise Knowledge Manager Manual.
- For Splunk Cloud Platform, see Configure size-based retention for data models summaries in the Splunk Cloud Platform Knowledge Manager Manual.
Data model default retention
The data model retention settings are contingent on the use case and data sources. A shorter retention uses less disk space and requires less processing time to maintain in exchange for limiting the time range of accelerated data.
|Data Model||Summary Range|
|Application State||1 month|
|Change Analysis||1 year|
|Compute Inventory||All Time|
|DLP (Data Loss Prevention)||1 year|
|Domain Analysis (ES)||1 year|
|Identity Management||All Time|
|Incident Management (ES)||0|
|Interprocess Messaging||1 year|
|Intrusion Detection||1 year|
|JVM (Java Virtual Machines)||All Time|
|Network Resolution (DNS)||3 months|
|Network Sessions||3 months|
|Network Traffic||3 months|
|Splunk Audit Logs||1 year|
|Threat Intelligence (ES)||All Time|
|Ticket Management||1 year|
You can use the following search to verify the current values:
| rest splunk_server=local count=0 /services/data/models | table title,acceleration.earliest_time
Use the CIM Setup page in the Splunk Common Information Model app to modify the retention setting for CIM data models. For more information, see Change the summary range for data model accelerations in the Splunk Common Information Model Add-on User manual. To change the summary range or other settings on a custom data model, manually edit the
datamodels.conf provided with the app or add-on.
- For instructions on how to edit these settings in Splunk Enterprise, see the datamodels.conf spec file in the Splunk Enterprise Admin Manual.
- If you are using Splunk Cloud Platform, file a support case to adjust these settings.
Data model acceleration rebuild behavior
In the Splunk platform, if the configuration of the data model structure changes, or the underlying search that creates the data model changes, a complete rebuild of the data model acceleration will initiate. Enterprise Security modifies the default behavior by applying data model configuration changes to the latest accelerations only, and prevents the removal of the prior accelerations. The indexers retain all existing accelerated data models with the prior configuration until the defined retention period is reached, or rolled with the index buckets. Prior configurations for all existing accelerated data models are retained for performance considerations. Rebuilding data models randomly can result in indexers being busy for days at a time. For best performance, do not change the manual rebuilds setting for any data models used by Splunk Enterprise Security.
- The rebuild configuration options are managed in the
For more information about acceleration and rebuild behavior, see the Splunk platform documentation.
- For Splunk Enterprise, see Advanced configurations for persistently accelerated data models in the Splunk Enterprise Knowledge Manager Manual.
- For Splunk Cloud Platform, see Advanced configurations for persistently accelerated data models in the Splunk Cloud Platform Knowledge Manager Manual.
- Use the Data Models management page to force a full rebuild. Navigate to Settings > Data Models, select a data model, use the left arrow to expand the row, and select the Rebuild link.
- To review the acceleration status for all data models, use the Data Model Audit dashboard.
Data model acceleration enforcement
Enterprise Security enforces data model acceleration through a modular input. To disable acceleration for a data model in ES:
- On the Splunk Enterprise toolbar, open Settings > Data inputs and select Data Model Acceleration Enforcement Settings.
- Select a data model.
- Uncheck the Acceleration Enforced option.
Data models used by Splunk Enterprise Security
For reference information about the data models used by Splunk Enterprise Security, see Data models used by ES in the Splunk developer portal.
Configure users and roles
Planning an upgrade of Splunk Enterprise Security
This documentation applies to the following versions of Splunk® Enterprise Security: 7.0.1, 7.0.2, 7.1.0
Feedback submitted, thanks!