Manage global settings for assets and identities in Splunk Enterprise Security

Configure the global settings of the identity manager modular input to revise the way the identity manager works by default.

Prerequisites

Perform the following prerequisite tasks before starting on these settings:

1. Collect and extract asset and identity data in Splunk Enterprise Security.
2. Format the asset or identity list as a lookup in Splunk Enterprise Security.
3. Configure a new asset or identity list in Splunk Enterprise Security.

Enable merge for assets or identities

The merge process is enabled for assets and identities by default. However, in situations when you have a source file with duplication in the key fields, and you can't groom the file to make sure that the information belongs to the same asset or identity, then you have the option to disable the merge process.

Use the global settings to enable or disable merge as follows:

1. From the Splunk Enterprise Security menu bar, select Configure > Data Enrichment > Asset and Identity Management.
2. Click the Global Settings tab.
3. Scroll to the Enable Merge for Assets or Identities panel.
4. Use the toggle to enable or disable for Assets or Identities.

Using assets as an example, consider a source file with duplicates in the key field of nt_host, such as the following: ip,mac,nt_host,dns,owner,priority,lat,long,city,country,bunit,category,pci_domain,is_expected,should_timesync,should_update,requires_av
192.0.2.2,,host1,,,,,,,,,,,,,,
192.0.2.120,,host1,,,,,,,,,,,,,,
192.0.2.135,,host1,,,,,,,,,,,,,,
192.0.2.242,,host2,,,,,,,,,,,,,,
192.0.2.65,,host2,,,,,,,,,,,,,,


The default is to merge the three rows with nt_host of host1 into one asset, and merge the two rows with host2 into another asset.

asset	ip	nt_host	pci_domain
192.0.2.2 192.0.2.120 192.0.2.135 host1	192.0.2.2 192.0.2.120 192.0.2.135	host1	untrust
192.0.2.242 192.0.2.65 host2	192.0.2.242 192.0.2.65	host2	untrust

If you disable the merge, then the collection remains the same as the source file, and assets are not merged.

asset	ip	nt_host	pci_domain
192.0.2.2 host1	192.0.2.2	host1	untrust
192.0.2.120 host1	192.0.2.120	host1	untrust
192.0.2.135 host1	192.0.2.135	host1	untrust
192.0.2.242 host2	192.0.2.242	host2	untrust
192.0.2.65 host2	192.0.2.65	host2	untrust

When you do a lookup on an non-merged collection, there is no context for how to resolve the overlapping key field values. For example, the asset_lookup_by_str lookup in transforms.conf has max_matches = 1, so the first host it matches in the assets_by_str collection is the only one you'll see in your search results.

Enable entity zones for Assets or Identities

Entity zones are disabled for assets and identities by default. You can enable entity zones in situations when you have mergers or acquisitions with other companies, for example, and you have similar IP address spaces that you need to keep separate.

Enable entity zones in the global settings as follows:

1. From the Splunk Enterprise Security menu bar, select Configure > Data Enrichment > Asset and Identity Management.
2. Click the Global Settings tab.
3. Scroll to the Enable Zones for Assets or Identities panel.
4. Use the toggle to enable for Assets or Identities.
5. Type a lowercase word to use as a default zone name. This word auto-populates in the cim_entity_zone fields if you do not specify your own values when formatting an asset or identity list as a lookup.
6. (Optional) Click Configure Zones to build a clause and specify a condition.
   The conditions may be as follows:
   
   • Matches an event
   • Assigns a specified zone to the cim_entity_zone field
     

   In situations where you have a default value specified for your known entities, a default cim_entity_zone value is not assigned if a similar event occurs from an unknown entity.

   1. In the Condition field, type a boolean that returns true or false for an eval statement in SPL. See Eval in the Splunk Enterprise Search Reference.
   2. In the Zone field, type the name of a zone to assign when the match is made.
   3. Click +Add Clause to add additional clauses.
   4. Click x to delete clauses.
   5. Click Confirm to save the clauses.
   6. Click Save.

   Any events that do not have a specified cim_entity_zone, or do not match any clauses, are assigned the default zone.

Disable entity zones for Assets and Identities

Disable entity zones in the global settings as follows:

1. From the Splunk Enterprise Security menu bar, select Configure > Data Enrichment > Asset and Identity Management.
2. Click the Global Settings tab.
3. Scroll to the Enable Zones for Assets or Identities panel.
4. Use the toggle to disable for Assets or Identities. Any previously existing default zone is disabled, not deleted.
5. Click Save.

See Format an asset or identity list as a lookup in Splunk Enterprise Security.

Example

Using assets as an example, consider a default zone name of my_zone and a source file with the same ip of 10.0.2.109, nt_host of host1 and host2 in different zones, a cim_entity_zone defined as an asset lookup header, and one empty cim_entity_zone value such as the following: ip,mac,nt_host,dns,owner,priority,lat,long,city,country,bunit,category,pci_domain,is_expected,should_timesync,should_update,requires_av,cim_entity_zone
192.0.2.94,,host1,,,,,,,,,,,,,,,
192.0.2.155,,host1,,,,,,,,,,,,,,,zone2
192.0.2.90,,host2,,,,,,,,,,,,,,,zone1
192.0.2.39,,host2,,,,,,,,,,,,,,,zone1
10.0.2.109,,host2,,,,,,,,,,,,,,,zone1
10.0.2.109,,host3,,,,,,,,,,,,,,,zone3
10.0.2.109,,host4,,,,,,,,,,,,,,,zone3

If you enable entity zones, the behavior is to use the default zone name for the empty cim_entity_zone value and not to merge key fields such as ip and nt_host that are in different zones.

cim_entity_zone	asset	ip	nt_host	pci_domain
my_zone	192.0.2.94 host1	192.0.2.94	host1	untrust
zone2	192.0.2.155 host1	192.0.2.155	host1	untrust
zone1	192.0.2.90 192.0.2.39 10.0.2.109 host2	192.0.2.90 192.0.2.39 10.0.2.109	host2	untrust
zone3	10.0.2.109 host3 host4	10.0.2.109	host3 host4	untrust

If you disable entity zones, the behavior is to merge key fields such as ip and nt_host as usual.

asset	ip	nt_host	pci_domain
192.0.2.94 192.0.2.155 host1	192.0.2.94 192.0.2.155	host1	untrust
192.0.2.90 192.0.2.39 10.0.2.109 host2 host3 host4	192.0.2.90 192.0.2.39 10.0.2.109	host2 host3 host4	untrust

Ignored values for Assets or Identities

In situations when you want values to be ignored in your fields, you might want to use special words to represent null values. The default behavior is to merge rows of source data based on a match in any one of the key fields. In many cases your source data might have placeholder values that span multiple rows, which causes them to get merged into one large multivalue row. To avoid this, you can define the placeholder values, and clean them during the merge process, so that independent rows are still maintained in the final lookups.

Set null values

Use the global settings to set your null values as follows:

1. From the Splunk Enterprise Security menu bar, select Configure > Data Enrichment > Asset and Identity Management.
2. Click the Global Settings tab.
3. Scroll to the Asset Ignored Values tab or the Identity Ignored Values tab.
   The default values that are ignored are null, n/a, unknown, and undefined.
   1. For assets, in the Asset Ignored Values section, click Add Row.
   2. Type a word that you want ignored and not displayed in the merge results. This field is case-sensitive.
   3. For identities, in the Identity Ignored Values section, click Add Row.
   4. Type a lowercase word that you want ignored and not displayed in the merge results. This field is case-sensitive.
4. Click Save.

The ignored values setting applies to any type of field, such as multivalue field or single value field or key field or non-key field. The strings are saved as ignored_values in SplunkHome/etc/apps/SA-IdentityManagement/local/inputs.conf.

Remove null values

Use the global settings to remove your null values as follows:

1. From the Splunk Enterprise Security menu bar, select Configure > Data Enrichment > Asset and Identity Management.
2. Click the Global Settings tab.
   The default values that are ignored are null, n/a, unknown, and undefined.
3. Scroll to the Asset Ignored Values tab or the Identity Ignored Values tab.
4. Find the value and click the x to delete it.

Revise the enforcements used by the identity manager framework

Every five minutes when the identity manager runs, it automatically enforces configuration file settings used by the framework, including inputs.conf, props.conf, macros.conf, transforms.conf, and identityLookup.conf (deprecated).

With these enforcements enabled, if there are accidental changes made to your conf files, the settings are reverted back to the way they were. If you're doing manual testing or making changes on purpose to your conf files and you do not want the settings checked or reverted back, you can disable these enforcements.

Use the global settings to enable or disable enforcements as follows. For the majority of users who configure settings through the Splunk Web UI, there is no need to disable these settings:

1. From the Splunk Enterprise Security menu bar, select Configure > Data Enrichment > Asset and Identity Management.
2. Click the Global Settings tab.
3. Scroll to the Enforcements panel.
4. Use the toggle to enable or disable.

Using the example of Enforce props, you experience the following by default. If you add a custom field in Identity Settings, the field is automatically added to the props.conf file because the settings check occurs to sync and reload props to be consistent with the identity manager.

Using the example of Enforce props, you experience the following by disabling it. If you add a custom field in Identity Settings, then you have to add that custom field to the props.conf file manually because the settings check no longer occurs. With enforce props disabled, any manual identity settings changes made without using the Splunk Web UI are also ignored.

After upgrading to Enterprise Security 6.2.0, you need to enable the Enforce props setting if you want the identity manager to automatically enforce configuration file settings. On a fresh installation, Enterprise Security 6.2.0 has Enforce props set to enabled by default and the setting is enforced continuously. However, prior versions only enforce once and then switch the setting to false right away. If you're already using a previous version of ES with assets and identities, the /local/inputs.conf file already has enforce_props=false and it needs to be set back to true after you upgrade, if you want to ensure that settings are managed for you. The majority of users who configure settings through the Splunk Web UI will benefit from enabling the setting.

Revise the miscellaneous settings used by the identity manager framework

You can revise miscellaneous settings that are specific to the identity manager.

Revise how often the identity manager runs

The identity manager runs every 300 seconds (5 minutes) by default. For performance purposes, you can change this to a larger value so it does not run so frequently.

Use the global settings to change the time:

1. From the Splunk Enterprise Security menu bar, select Configure > Data Enrichment > Asset and Identity Management.
2. Click the Global Settings tab.
3. Scroll to the Miscellaneous Settings panel.
4. Type a number of seconds in the Time(s) field.

Revise the master host where the identity manager runs

The identity manager runs on the search head captain by default. If you want to separate search head responsibilities, or if the search head is experiencing performance issues due to resource consumption, then you can change the master host.

Use the global settings to change the master host if search head clustering is enabled:

1. From the Splunk Enterprise Security menu bar, select Configure > Data Enrichment > Asset and Identity Management.
2. Click the Global Settings tab.
3. Scroll to the Miscellaneous Settings panel.
4. Type a name in the Master host field that matches the name of a server in the cluster pool.

See System requirements and other deployment considerations for search head clusters.

Add additional context to string lookups based on CIDR blocks

By default, the asset_lookup_by_str lookup does not combine Classless Inter-Domain Routing (CIDR) enrichment in the output results. You can add additional enrichment to your asset and identity lookups based on CIDR blocks. This does not take away any functionality from your asset_lookup_by_cidr lookup.

Automatic lookups run in a certain order to populate enrichment data into empty fields. The order starts with asset_lookup_by_str first, and then asset_lookup_by_cidr is next. Once the string enrichment data is populated into a field, the field is no longer empty, so it does not get filled with CIDR data. Normally your CIDR data is only returned by asset_lookup_by_cidr, but sometimes that results in CIDR enrichment being lost because asset_lookup_by_str runs and matches first. With overlay CIDR enabled, your asset_lookup_by_str will include the CIDR data as well. For more information about automatic lookups and correlation setup, see Manage correlation setup in Splunk Enterprise Security.

To overlay CIDR enrichment into your string lookup results, use the global settings:

1. From the Splunk Enterprise Security menu bar, select Configure > Data Enrichment > Asset and Identity Management.
2. Click the Global Settings tab.
3. Scroll to the Miscellaneous Settings panel.
4. Toggle the Overlay CIDR setting to enable.

Using assets as an example, consider a source file with an ip address of 192.187.2.94, which is also a match for a CIDR range of 192.187.0.0/16 that has values in the owner field: ip,mac,nt_host,dns,owner,priority,lat,long,city,country,bunit,category,pci_domain,is_expected,should_timesync,should_update,requires_av
192.187.2.94,,,,owner1,,,,,,,,,,,,
192.187.0.0/16,,,,cidr_owner1,,,,,,,,,,,,
10.0.2.109,,,,owner2,,,,,,,,,,,,
10.0.2.0/24,,,,cidr_owner2,,,,,,,,,,,,


With overlay CIDR enabled, the behavior is to include CIDR field values within the string lookup's output results. When an event comes in that matches both an asset by string and also an asset by CIDR, you see the exact match data for the IP address and the most specific CIDR block data.

Using the search preview for asset_lookup_by_str returns results similar to the following:

asset	ip	owner	pci_domain
192.187.2.94	192.187.2.94	owner1 cidr_owner1	untrust
10.0.2.109	10.0.2.109	owner2 cidr_owner2	untrust

See Use the search preview to test the merge of asset and identity data in Splunk Enterprise Security.

With overlay CIDR disabled, the behavior is not to include any enrichment for CIDR field values in the string lookup's output results.

Using the search preview for asset_lookup_by_str returns results similar to the following:

asset	ip	owner	pci_domain
192.187.2.94	192.187.2.94	owner1	untrust
10.0.2.109	10.0.2.109	owner2	untrust

See Use the search preview to test the merge of asset and identity data in Splunk Enterprise Security.

The asset enrichment specific to CIDR fields is still available in the CIDR lookup's output results, just not in the string lookup's output results.

Using the search preview for asset_lookup_by_cidr returns results similar to the following:

asset	ip	owner	pci_domain
192.187.0.0/16	192.187.0.0/16	cidr_owner1	untrust
10.0.2.0/24	10.0.2.0/24	cidr_owner2	untrust

See Use the search preview to test the merge of asset and identity data in Splunk Enterprise Security.

The overlay_cidr setting is stored in the [identity_manager] stanza of the inputs.conf file.

Revise asset and identity lookups memory usage behavior

Prior to the release of Splunk Cloud 8.0.2004, KV Store backed lookups do not respect the max_memtable_bytes setting. This means that KV Store backed lookups are always stored in memory on the indexer.

With the release of Splunk Cloud 8.0.2004, KV Store backed lookups do respect the max_memtable_bytes setting. This means that a KV Store backed lookup is stored in memory until it exceeds the definition in the max_memtable_bytes setting.

You might experience the following behavior after upgrading. Using Splunk Enterprise 8.0 as an example, consider a KV Store lookup of 1 GB in size that is used as an automatic lookup, with max_memtable_bytes=25MB. If you upgrade to a Splunk Cloud version of 8.0.2004 or higher, the 1 GB size exceeds the max_memtable_bytes setting, so an index file is created and the lookup occurs on disk, which is slower.

The default setting in Splunk Cloud is max_memtable_bytes=100MB. Splunk Cloud customers need to contact technical support if necessary to revise this behavior.

To revise this behavior in an on-premises environment, increase your max_memtable_bytes in the $SPLUNK_HOME/etc/system/local/limits.conf file. See lookup of limits.conf in the Splunk Enterprise Admin Manual.

Reset your collections immediately

All the asset and identity source files that are enabled in the Asset and Identity Management page get merged into the following default collections in the collections.conf file: assets_by_str, assets_by_cidr, or identities_expanded.

If your collections get into an undesirable state, you can reset your collections at any time, rather than waiting for the automated process to clear out the KV store collection. It's similar to clearing cache manually.

1. From the Splunk Enterprise Security menu bar, select Configure > Data Enrichment > Asset and Identity Management.
2. Click Reset Collections. The button is globally available regardless if you are configuring in a particular tab.

When the identity manager runs again in 5 minutes, it rebuilds the collections based on which source files are enabled in the Asset Lookup Configuration or the Identity Lookup Configuration.