Data models in the Enterprise Security app

The Splunk App for Enterprise Security 3.0 and later make extensive use of accelerated data models to populate dashboards and views. Most data models are defined and provided in the Common Information Model app (Splunk_SA_CIM), which is included as part of the Enterprise Security app installation. Some data models are defined in the Enterprise Security app, and are unique. See Customized data models in Enterprise Security in this topic.

Data model acceleration rebuild behavior

Data model acceleration is designed to force an automatic rebuild any time the data model structure changes, or if the underlying search that creates the data model changes. In Splunk Enterprise 6.1 and later, a new rebuild option was created for data models. As implemented in the Enterprise Security app, a change to a data model will not cause an automatic rebuild of the data model accelerations. The changed data model values will apply to the latest data accelerated only. The legacy data model accelerations will be retained and rolled out with the buckets, or until the defined retention period is reached.

• The rebuild configuration options are managed in the datamodels.conf file.

• See "Advanced configurations for persistently accelerated data models" in the Knowledge Manager Manual.

• Use the Data Models management page to force a full rebuild. Navigate to Settings > Data Models, select a data model, use the left arrow to expand the row, and select the Rebuild link.

• Use the Data Model Audit dashboard to review the acceleration status for all data models.

Data model acceleration enforcement

Data model acceleration is enforced in Enterprise Security 3.0 and later through a modular input. There are 2 ways to disable data model acceleration:

1. Set the modular input to turn off Enforce Acceleration. To change the setting for a specific modular input, edit the input for the data model you are changing, uncheck the "Acceleration Enforced" setting and save.
2. Turn off our enforcement and manually edit all data model accelerations. Disable the input stanza for the data model, which will permit manual changes to a data model's acceleration settings to persist indefinitely.

Data model acceleration storage and retention

Data model acceleration storage volumes are managed in indexes.conf using the tstatsHomePath parameter, with the data model acceleration storage path defaulting to the Splunk Enterprise default index path $SPLUNK_HOME/var/lib/splunk unless explicitly configured. The storage used for data model acceleration is not added to index sizing calculations for maintenance tasks such as bucket rolling and free space checks.

To manage the data model acceleration storage independently of index settings, a new storage path must be defined with [volume:] stanzas. For an example of defining a volume and storing data model accelerations, see Configure size-based retention for data models summaries in the Knowledge Manager manual. Setting the retention of accelerated data models is managed in the datamodels.conf files.

Data model default retention

Data Model	Summary Range	Data Model	Summary Range
Alerts	All Time	Application State	1 month
Assets And Identites (ES)	All Time	Authentication	1 year
Certificates	1 year	Change Analysis	1 year
Databases	None	Domain Analysis (ES)	1 year
Email	1 year	Incident Management (ES)	All Time
Interprocess Messaging	1 year	Intrusion Detection	1 year
Inventory	None	Malware	1 year
Java Virtual Machines	All Time	Network Resolution (DNS)	3 months
Network Sessions	3 months	Network Traffic	3 months
Performance	1 month	Risk Analysis (ES)	All Time
Splunk Audit Logs	1 year	Threat Lists (ES)	All Time
Ticket Management	1 year	Updates	1 year
Vulnerabilities	1 year	Web	3 months

Common Information Model data models

For a list of the data models are included in the Splunk Common Information Model Add-on, see "What data models are included" in the Common Information Model Add-on Manual.

Customized data models in Enterprise Security

In addition to the data models available as part of the Common Information Model add-on, the Splunk App for Enterprise Security provides its own custom data models.

Assets And Identities

The fields in the Assets And Identities data model, and the Asset and Identity event categories, describe both asset inventory and individual account holders that should be made available across multiple Splunk application contexts.

Note: Any field in the All_Assets event category can be optionally pre-pended with dest_, dvc_, host_, orig_host_, or src_ for enrichment purposes. These fields are not required, but are often used in Apps alongside dest, dvc, host, orig_host, or src if they are available.

Tags are not applicable to the Asset And Identities data model and event category.

Fields for the Asset And Identities data model and event category

Object name(s)	Field name	Data type	Description	Expected values
All_Assets	asset_id	string	an identifier for the asset, such as an asset tag or serial number.
All_Assets	city	string	The city where the asset is located, such as San Francisco.
All_Assets	bunit	string	The business unit of the asset, such as Marketing.
All_Assets	category	MV string	The category of the asset, such as email_server or SOX-compliant.
All_Assets	country	string	The country where the asset is located, such as USA.
All_Assets	dns	MV string	A fully qualified domain name (FQDN) associated with the asset, such as server42.splunk.com.
All_Assets	ip	MV string	An IP address (either v4 or v6) associated with the asset, such as 192.168.4.2. Note: Please remove zero-padding on this field.
All_Assets	is_expected	boolean	A flag indicating whether the asset is expected to continually send data to Splunk. Note: Some apps may alert if is_expected is set to Y for an asset that is not sending data.	true, false
All_Assets	lat	string	The latitude of an asset's location.
All_Assets	location	string	The physical location of an asset.
All_Assets	long	string	The longitude of an asset's location.
All_Assets	mac	MV string	A MAC address associated with the asset, such as 06:10:9f:eb:8f:14. Note: Always force lower case on this field. Note: Always use colons instead of dashes, spaces, or no separator.
All_Assets	nt_host	string	The cross-platform short name or NetBIOS name of the asset, such as server42. Note: Always force lower case on this field.
All_Assets	owner	MV string	The owner of the asset, such as jdoe.
All_Assets	priority	string	The priority of the asset.	critical, high, medium, low, informational, unknown
All_Assets	requires_av	boolean	Flag that indicates whether the asset is expected to use a local antivirus or endpoint protection tool. Note that some apps may alert if requires_av is set to true for an asset that is not running an antivirus service and/or does not have event types properly configured for that service.	true, false
All_Assets	should_timesync	boolean	Flag that indicates whether the asset is expected to maintain time synchronization. Note that some apps may alert if should_timesync is set to true for an asset that is not running a time synchronization service and/or does not have event types properly configured for that service.	true, false
All_Assets	should_update	boolean	Flag that indicates whether the asset is expected to regularly apply patches. Note that some apps may alert if should_update is set to true for an asset that is not running a patching service and/or does not have event types properly configured for that service.	true, false
All_Identities	bunit	string	The business unit of the identity, such as Sales.
All_Identities	category	MV string	The category of the identity, such as sales or customer_facing.
All_Identities	city	string	The city where the identity is based, such as San Francisco.
All_Identities	country	string	The country where the identity is based, such as USA.
All_Identities	email	MV string	The email address (or addresses) associated with the identity is based. Note that this is a multivalue field.
All_Identities	end_date	timestamp	The end date of the identity, leave blank if not applicable. Note that presence of an end_date in the past may cause some Apps to create alerts from events involving this identity.
All_Identities	first	string	A first name for the identity, such as Jane.
All_Identities	identity	MV string	Account names and numbers associated with the identity. Note that this is a multivalue field.
All_Identities	last	string	A last name for the identity, such as Doe.
All_Identities	lat	string	The latitude of the identity's base location.
All_Identities	location	string	The base location for the identity, such as an office name.
All_Identities	long	string	The longitude of the identity's base location.
All_Identities	managed_by	MV string	The manager(s) of the identity such as jdoe. Note that this is a multivalue field and should use account names or numbers from the identity field.
All_Identities	nick	string	A nickname for the identity, such as Moerex.
All_Identities	phone	MV string	A phone number (or set of phone numbers) for the identity. Note that this is a multivalue field.
All_Identities	phone2	MV string	A phone number (or set of phone numbers) for the identity. Note that this is a multivalue field.
All_Identities	prefix	string	A prefix for the identity, such as Mr..
All_Identities	priority	string	The priority of the identity.	critical, high, medium, low, informational, unknown
All_Identities	start_date	timestamp	The start date of the identity.
All_Identities	suffix	string	A suffix for the identity, such as Jr.
All_Identities	watchlist	boolean	Flag if the identity is on a watchlist. Note that some apps may create alerts for events that involve this identity if this flag is set.	true, false

Domain Analysis

The Domain Analysis data model is available as part of the SA-NetworkProtection add-on, included with the Splunk App for Enterprise Security. Domain Analysis data model search searches for index=whois sourcetype=Whois:*.

The fields and tags in the Domain Analysis data model describe the domain information in your deployment.

Tags used with the Domain Analysis data model

Object name(s)	Tag name	Required?
All_Domains	index=whois sourcetype=Whois:*	YES

Fields for the Domain Analysis data model and event category

Object name(s)	Field name	Data type	Description	Expected values
All_Domains	domain	string	name of the domain
All_Domains	nameservers	string	name of the server associates with this domain
All_Domains	registrant	string
All_Domains	registrar	string
All_Domains	resolved_domain	string	resolved domain name

Incident Management

The Incident Management data model is available as part of the SA-ThreatIntelligence add-on, included with the Splunk App for Enterprise Security. This data model reads from index=notable.

The fields in the Incident Management event category describe events gathered by network monitoring devices and apps.

Tags used with the Incident Management event category

Object name(s)	Tag name or constraint	Required?
Notable_Events (Metatdata only)	index=notable	YES

Fields for the Incident Management data model

Object name(s)	Field name	Data type	Description	Possible values
Notable_Events_Meta	tag	string
Notable_Events_Meta	rule_id	string
Notable_Events_Meta	decoration	string
Correlation_Searches	control	string
Correlation_Searches	default_owner	string
Correlation_Searches	default_status	string
Correlation_Searches	description	string
Correlation_Searches	governance	string
Correlation_Searches	rule_name	string
Correlation_Searches	saved_search	string
Correlation_Searches	security_domain	string
Correlation_Searches	severity	string
Incident Review	comment	string
Incident Review	owner	string
Incident Review	reviewer	string
Incident Review	rule_id	string
Incident Review	security_domain	string
Incident Review	status_group	string
Incident Review	status_label	string
Incident Review	tag	string
Incident Review	urgency	string
Notable_Events	dest	string
Notable_Events	owner	string
Notable_Events	owner_realname	string
Notable_Events	rule_name	string
Notable_Events	security_domain	string
Notable_Events	source	string
Notable_Events	src	string
Notable_Events	status_label	string
Notable_Events	status_group	string
Notable_Events	tag	string
Notable_Events	urgency	string
Notable_Owners	owner	string
Notable_Owners	owner_realname	string
Review_Statuses	default	boolean
Review_Statuses	end	boolean
Review_Statuses	hidden	boolean
Review_Statuses	status	string
Review_Statuses	status_description	string
Review_Statuses	status_label	string
Security_Domains	is_enabled	boolean
Security_Domains	is_expected	boolean
Security_Domains	is_ignored	boolean
Security_Domains	security_domain_label	string
Suppression_Audit	action	string
Suppression_Audit	signature	string
Suppression_Audit	status	string
Suppression_Audit	suppression	string
Suppression_Audit	user	string
Suppression_Audit_Expired	suppression	string
Suppression_Eventtypes	description	string
Suppression_Eventtypes	disabled	boolean
Suppression_Eventtypes	end_time	timestamp
Suppression_Eventtypes	search	string
Suppression_Eventtypes	suppression	string
Suppression_Eventtypes	start_time	timestamp
Suppressed_Notable_Events	dest	string
Suppressed_Notable_Events	rule_name	string
Suppressed_Notable_Events	security_domain	string
Suppressed_Notable_Events	signature	string
Suppressed_Notable_Events	source	string
Suppressed_Notable_Events	suppression	string
Suppressed_Notable_Events	tag	string
Suppressed_Notable_Events	urgency	string
Urgencies	priority	string
Urgencies	severity	string
Urgencies	urgency	string

Risk Analysis

Object name(s)	Field name	Data type	Description	Expected values
All_Risk	description	string	A short description of the correlation search that generated the risk modifier.	calculated
All_Risk	risk_object	string	The value of the object this modifier applies to.	src,dest,etc.
All_Risk	risk_object_type	string	The object type this modifier applies to.	system,user,other,etc.
All_Risk	risk_score	integer	The amount of "points" to increase or decrease the risk_object's score by.

Threat Lists

The Threat Lists data model is available as part of the SA-ThreatIntelligence add-on, included with the Splunk App for Enterprise Security.

The fields and tags in the Threat Lists data model describe potential threats both inside and outside of your deployment.

See the Common Information Model Add-on Manual for more about data models.

Tags used with the Threat Lists data model

Object name(s)	Tag name or constraint	Required?
All_Threat_Lists	`threatlists`	YES

Fields for the Threat Lists data model and event category

Object name(s)	Field name	Data type	Description	Expected values
All_Threat_Lists	category	string	Category of the threat	proxy, spyware, network, malicious
All_Threat_Lists	description	string	Description of the threat, source, how it was detected, etc.
All_Threat_Lists	ip_count	string	Count of ip values associated with a specific threat	512, 32, 16, 256
All_Threat_Lists	ip	string	IP address associated with the threat	99.250.24.32
All_Threat_Lists	name	string	Name of the lookup that detected the threat	sans, iblocklist_tor
All_Threat_Lists	subnet	int	Subnet on which the threat was detected	23, 32, 24,27

High Performance Analytics Store namespaces

The Splunk App for Enterprise Security creates a High Performance Analytics Store namespace to store summary statistical data. This type of data model is stored on the search head.

Namespace details

This table shows a namespace and its attributes, including the searches that populate the namespace, the search macro used to identify the data that will be stored in the namespace, the fields of information stored, and the suggested retention period for the data.

sa_host_meta

Namespace	Location (SA-*)	Generating Search Search Schedule	Data Source (search macro)	Fields	Suggested Retention Period
sa_host_meta	SA-AuditAndDataProtection	Audit - Host Event Count over Time - TSIDX Gen 25 4,16 * * *	metasearch index=* sourcetype!=stash	_time, host*, tag, count	365 days

Legend

Fields with an (*) include a number of other fields. These fields include:

host* == host,host_bunit,host_category,host_pci_domain
dest* == dest,dest_bunit,dest_category,dest_pci_domain
dvc* == dvc,dvc_bunit,dvc_category,dvc_pci_domain
src* == src,src_bunit,src_category,src_pci_domain

src_user* == src_user,src_user_bunit,src_user_category
user* == user,user_bunit,user_category 

Namespace retention

The limit for the length of time that namespaces are retained ("namespace retention time") is specified in the $SPLUNK_HOME/etc/apps/<add-on>/default/tsidx_retention.conf file. The recommended retention times are not enforced in code out-of-the-box; they are commented out.

To apply the recommended settings, un-comment the retention settings in your local copy (local/tsidx_retention.conf) of the file and save it.

For example:

Out of the box, the $SPLUNK_HOME/etc/apps/SA-AuditAndDataProtection/default/tsidx_retention.conf file looks like this:

[sa_host_meta]
## 1 year
#retentionTimePeriodInSecs = 31556940 

Confgure the retention time in the "local" version - $SPLUNK_HOME/etc/apps/SA-AuditAndDataProtection/local/tsidx_retention.conf - like this:

[sa_host_meta]
# 90 days
retentionTimePeriodInSecs = 7776000