How Splunk Enterprise Security processes and merges asset and identity data

Splunk Enterprise Security takes the asset and identity data that you add as lookups and generates combined lookup files. Splunk Enterprise Security uses the generated lookup files to correlate asset and identity data with events using automatic lookups.

1. You collect asset and identity data from data sources using an add-on and a custom search or manually with a CSV file. See Collect and extract asset and identity data.
2. You format the data as a lookup, using a search or manually with a CSV file. See Format the asset or identity list as a lookup.
3. You configure the list as a lookup table, definition, and input. See Configure a new asset or identity list.
   1. Splunk Enterprise Security identity manager modular input detects changed content in the identity_manager://<input_name> and changes to stanzas in the input.
4. You configure any settings in the identity lookup configuration setup. See Define identity formats on the identity configuration page.
   1. Splunk Enterprise Security identity manager modular input updates settings in the transforms.conf stanza identity_lookup_expanded
5. Splunk Enterprise Security identity manager modular input updates the macros used to identify the input sources based on the currently enabled stanzas in inputs.conf. For example, update the `generate_identities` macro dynamically based on the conventions specified on the Identity Lookup Configuration page.
6. Splunk Enterprise Security identity manager modular input dispatches lookup generating saved searches if it identifies changes that require the asset and identity lists to be merged.
7. Splunk Enterprise Security merges all configured and enabled asset and identity lists using lookup generating saved searches.
   1. The primary saved searches concatenate the lookup tables referenced by the identity manager input, generate new fields, and output the concatenated asset and identity lists into target lookup table files.
   2. Secondary saved searches generate lookup tables for asset categories, identity categories, and asset PCI domains (in Splunk App for PCI Compliance).
8. You verify that the data looks as expected. See Verify that your asset or identity data was added to Splunk Enterprise Security.

The merging of identity and asset lookups does not validate or de-duplicate input. Errors from the identity manager modular input are logged in identity_manager.log. This log does not show data errors.

Merging assets and identities creates new lookup files

After the merging process completes, four lookups store your asset and identity data.

Function	Table name	Saved search	Lookup name
String-based asset correlation	assets_by_str.csv	Identity - Asset String Matches - Lookup Gen	LOOKUP-zu-asset_lookup_by_str-dest LOOKUP-zu-asset_lookup_by_str-dvc LOOKUP-zu-asset_lookup_by_str-src
CIDR subnet-based asset correlation	assets_by_cidr.csv	Identity - Asset CIDR Matches - Lookup Gen	LOOKUP-zv-asset_lookup_by_cidr-dest LOOKUP-zv-asset_lookup_by_cidr-dvc LOOKUP-zv-asset_lookup_by_cidr-src
String-based identity correlation	identities_expanded.csv	Identity - Identity Matches - Lookup Gen	LOOKUP-zy-identity_lookup_expanded-src_user LOOKUP-zy-identity_lookup_expanded-user
Default field correlation	identity_lookup_default_fields.csv asset_lookup_default_fields.csv		LOOKUP-zz-asset_identity_lookup_default_fields-dest LOOKUP-zz-asset_identity_lookup_default_fields-dvc LOOKUP-zz-asset_identity_lookup_default_fields-src LOOKUP-zz-asset_identity_lookup_default_fields-src_user LOOKUP-zz-asset_identity_lookup_default_fields-user

Asset fields after processing

Asset fields of the asset lookup after the saved searches perform the merge process.

Field	Action taken by ETL
bunit	unchanged
city	unchanged
country	unchanged
dns	Accepts pipe-delimited values and converts them to a multi-value field.
lat	unchanged
long	unchanged
mac	Accepts pipe-delimited values and converts them to a multi-value field.
nt_host	Accepts pipe-delimited values and converts them to a multi-value field.
owner	unchanged
priority	unchanged
asset_id	Generated from the values of dns, ip, mac, and nt_host fields.
asset_tag	Generated from the values of category, pci_domain, is_expected, should_timesync, should_update, requires_av, and bunit fields.
category	Appends "pci" if the value contains "cardholder". Accepts pipe-delimited values and converts them to a multi-value field.
ip	validates and splits the field into CIDR subnets as necessary. Accepts pipe-delimited values and converts them to a multi-value field.
pci_domain	Appends "trust" or "untrust" based on certain field values. Accepts pipe-delimited values and converts them to a multi-value field.
is_expected	Normalized to a boolean.
should_timesync	Normalized to a boolean.
should_update	Normalized to a boolean.
requires_av	Normalized to a boolean.
key	Generated by the ip, mac, nt_host, and dns fields after the original fields are transformed.

Identity fields after processing

Identity fields of the identity lookup after the saved searches perform the merge process.

Field	Action taken by ETL
bunit	unchanged
email	unchanged
endDate	unchanged
first	unchanged
last	unchanged
managedBy	unchanged
nick	unchanged
phone	unchanged
phone2	unchanged
prefix	unchanged
priority	unchanged
startDate	unchanged
suffix	unchanged
work_city	unchanged
work_country	unchanged
work_lat	unchanged
work_long	unchanged
watchlist	Normalized to a boolean.
category	Appends "pci" if the value contains "cardholder". Accepts pipe-delimited values and converts them to a multi-value field.
identity	Generated based on values in the input row and conventions specified in the Identity Lookup Configuration. Accepts pipe-delimited values and converts them to a multi-value field.
identity_id	Generated from the values of identity, first, last, and email.
identity_tag	Generated from the values of bunit, category, and watchlist.

Test the asset and identity merge process

Test the asset and identity merge process to confirm that the data produced by the merge process is expected and accurate. Run the saved searches that perform the merge process without outputting the data to the merged lookups to determine what the merge will do with your data without actually performing the merge.

Test the merge process without performing a merge and outputting the data to a lookup.

1. From the Splunk ES menu bar, select Configure > Content Management.
2. Locate the first of the three primary saved searches Identity - Asset CIDR Matches - Lookup Gen.
3. Click the search name to open it.
4. Copy the search from the Search field.
5. Open the Search page.
6. Paste the search and remove the `output_*` macro. For example, change | `asset_sources` | `make_assets_cidr` | `output_assets("SA-IdentityManagement", "assets_by_cidr.csv")` to | `asset_sources` | `make_assets_cidr`.
7. Run the search.
8. Repeat steps 2-7 for the other two searches, Identity - Asset String Matches - Lookup Gen and Identity - Identity Matches - Lookup Gen.

Force a merge

Run the primary saved searches directly to force a merge immediately without waiting the five minutes for the scheduled search to run.

1. Open the Search page.
2. Run the primary saved searches.
   

   | savedsearch "Identity - Asset String Matches - Lookup Gen"

   
   

   | savedsearch "Identity - Asset CIDR Matches - Lookup Gen"

   
   

   | savedsearch "Identity - Identity Matches - Lookup Gen"

Customize the asset and identity merge process

You can modify the saved searches that perform the asset and identity merge process to perform additional field transformations or data sanitization. Add any operations that you want to change in the merge process to the search before the `output_*` macro.

Caution: Certain modifications to the saved searches are unsupported and could break the merge process or asset and identity correlation. Do not perform the following actions.

• Add or delete fields from the output.
• Change the output location to a different lookup table or a KV store collection.
• Replace the `output_*` macros with the outputlookup command.