Configure data models for Splunk Enterprise Security
Splunk Enterprise Security leverages data model acceleration to populate dashboards and views and provide correlation search results. The data models are defined and provided in the Common Information Model add-on (Splunk_SA_CIM), which is included in the Splunk Enterprise Security installation. Enterprise Security also installs unique data models that only apply to Splunk Enterprise Security content.
Data model acceleration search load
A data model is accelerated through a scheduled summarization search process initiated on the search head. The summarization search runs on the indexers, searching newly indexed data while using the data model as a filter. The resulting matches are saved to disk alongside the index bucket for quick access.
On Splunk platform 6.3 and later, up to two simultaneous summarization searches can run per data model, per indexer. For more information, see Parallel summarization in the Splunk Enterprise Capacity Planning Manual. To adjust parallel summarization settings on Splunk Cloud, file a support ticket.
Constrain data model searches to specific indexes
The Splunk Common Information Add-on allows you to constrain the indexes searched by a data model for improved performance. See Set up the Splunk Common Information Model Add-on in the Splunk Common Information Model Add-on User manual.
Configure data model acceleration for CIM data models
The Splunk Common Information Add-on allows you to adjust your data model acceleration settings for each data model, including the backfill time, maximum concurrent searches, manual rebuilds, and scheduling priority. If you are using Splunk platform version 6.6.0, configure the tags whitelist setting to include any custom tags you use with CIM data models. See Accelerate CIM data models in the Splunk Common Information Model Add-on User manual.
Data model acceleration storage and retention
Data model acceleration uses the indexers for processing and storage, placing the accelerated data alongside each index. To calculate the additional storage needed on the indexers based on the total volume of data, use the formula:
Accelerated data model storage/year = Data volume per day * 3.4
This formula assumes that you are using the recommended retention rates for the accelerated data models.
For example, if you process 100GB/day of data volume for use with Enterprise Security, you need approximately 340GB of additional space available across all of the indexers to allow for up to one year of data model acceleration and source data retention.
Configuring storage volumes
Data model acceleration storage volumes are managed in indexes.conf
using the tstatsHomePath
parameter. The data model acceleration storage path defaults to the Splunk platform default index path of $SPLUNK_HOME/var/lib/splunk
unless explicitly configured otherwise. The storage used for data model acceleration is not added to index sizing calculations for maintenance tasks such as bucket rolling and free space checks.
To manage the data model acceleration storage independently of index settings, you must define a new storage path with [volume:]
stanzas. For an example of defining a volume and storing data model accelerations, see the Splunk platform documentation.
- For Splunk Enterprise, see Configure size-based retention for data models summaries in the Splunk Enterprise Knowledge Manager Manual.
- For Splunk Cloud, see Configure size-based retention for data models summaries in the Splunk Cloud Knowledge Manager Manual.
Data model default retention
The data model retention settings are contingent on the use case and data sources. A shorter retention uses less disk space and requires less processing time to maintain in exchange for limiting the time range of accelerated data.
Data Model | Summary Range |
---|---|
Alerts | All Time |
Application State | 1 month |
Assets And Identities (ES) | None |
Authentication | 1 year |
Certificates | 1 year |
Change Analysis | 1 year |
Databases | None |
Data Loss Prevention | 1 year |
Domain Analysis (ES) | 1 year |
1 year | |
Incident Management (ES) | All Time |
Interprocess Messaging | 1 year |
Intrusion Detection | 1 year |
Inventory | None |
Java Virtual Machines | All Time |
Malware | 1 year |
Network Resolution (DNS) | 3 months |
Network Sessions | 3 months |
Network Traffic | 3 months |
Performance | 1 month |
Risk Analysis (ES) | All Time |
Splunk Audit Logs | 1 year |
Threat Intelligence (ES) | All Time |
Ticket Management | 1 year |
Updates | 1 year |
User and Entity Behavior Analytics (ES) | All Time |
Vulnerabilities | 1 year |
Web | 3 months |
Use the CIM Setup page in the Splunk Common Information Model app to modify the retention setting for CIM data models. For more information, see Change the summary range for data model accelerations in the Splunk Common Information Model Add-on User manual. To change the summary range or other settings on a custom data model, manually edit the datamodels.conf
provided with the app or add-on.
- For instructions on how to edit these settings in Splunk Enterprise, see the datamodels.conf spec file in the Splunk Enterprise Admin Manual.
- If you are using Splunk Cloud, file a support case to adjust these settings.
Data model acceleration rebuild behavior
In the Splunk platform, if the configuration of the data model structure changes, or the underlying search that creates the data model changes, a complete rebuild of the data model acceleration will initiate. Enterprise Security modifies the default behavior by applying data model configuration changes to the latest accelerations only, and prevents the removal of the prior accelerations. The indexers retain all existing accelerated data models with the prior configuration until the defined retention period is reached, or rolled with the index buckets. For best performance, do not change the manual rebuilds setting for any data models used by Splunk Enterprise Security.
- The rebuild configuration options are managed in the
datamodels.conf
file.
For more information about acceleration and rebuild behavior, see the Splunk platform documentation.
- For Splunk Enterprise, see Advanced configurations for persistently accelerated data models in the Splunk Enterprise Knowledge Manager Manual.
- For Splunk Cloud, see Advanced configurations for persistently accelerated data models in the Splunk Cloud Knowledge Manager Manual.
- Use the Data Models management page to force a full rebuild. Navigate to Settings > Data Models, select a data model, use the left arrow to expand the row, and select the Rebuild link.
- To review the acceleration status for all data models, use the Data Model Audit dashboard.
Data model acceleration enforcement
Enterprise Security enforces data model acceleration through a modular input. To disable acceleration for a data model in ES:
- On the Splunk Enterprise toolbar, open Settings > Data inputs and select Data Model Acceleration Enforcement Settings.
- Select a data model.
- Uncheck the Acceleration Enforced option.
- Save.
Data models used by Splunk Enterprise Security
For reference information about the data models used by Splunk Enterprise Security, see Data models used by ES in the Splunk developer portal.
Configure users and roles | Planning an upgrade of Splunk Enterprise Security |
This documentation applies to the following versions of Splunk® Enterprise Security: 5.0.0, 5.0.1, 5.1.0, 5.1.1, 5.2.0, 5.2.1, 5.2.2
Feedback submitted, thanks!