Splunk® Enterprise Security

Use Splunk Enterprise Security

Acrobat logo Download manual as PDF


This documentation does not apply to the most recent version of Splunk® Enterprise Security. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

How Splunk Enterprise Security processes and merges asset and identity data

Splunk Enterprise Security takes the asset and identity data that you add as lookups and generates combined lookup files. Splunk Enterprise Security uses the generated lookup files to correlate asset and identity data with events using automatic lookups.

  1. You collect asset and identity data from data sources using an add-on and a custom search or manually with a CSV file. See Collect and extract asset and identity data.
  2. You format the data as a lookup, using a search or manually with a CSV file. See Format the asset or identity list as a lookup.
  3. You configure the list as a lookup table, definition, and input. See Configure a new asset or identity list.
    1. Splunk Enterprise Security identity manager modular input detects changed content in the identity_manager://<input_name> and changes to stanzas in the input.
  4. You configure any settings in the identity lookup configuration setup. See Define identity formats on the identity configuration page.
    1. Splunk Enterprise Security identity manager modular input updates settings in the transforms.conf stanza identity_lookup_expanded
  5. Splunk Enterprise Security identity manager modular input updates the macros used to identify the input sources based on the currently enabled stanzas in inputs.conf. For example, update the `generate_identities` macro dynamically based on the conventions specified on the Identity Lookup Configuration page.
  6. Splunk Enterprise Security identity manager modular input dispatches lookup generating saved searches if it identifies changes that require the asset and identity lists to be merged.
  7. Splunk Enterprise Security merges all configured and enabled asset and identity lists using lookup generating saved searches.
    1. The primary saved searches concatenate the lookup tables referenced by the identity manager input, generate new fields, and output the concatenated asset and identity lists into target lookup table files.
    2. Secondary saved searches generate lookup tables for asset categories, identity categories, and asset PCI domains (in Splunk App for PCI Compliance).
  8. You verify that the data looks as expected. See Verify that your asset or identity data was added to Splunk Enterprise Security.

The merging of identity and asset lookups does not validate or de-duplicate input. Errors from the identity manager modular input are logged in identity_manager.log. This log does not show data errors.

Merging assets and identities creates new lookup files

After the merging process completes, four lookups store your asset and identity data.

Function Table name Saved search Lookup name
String-based asset correlation assets_by_str.csv Identity - Asset String Matches - Lookup Gen LOOKUP-zu-asset_lookup_by_str-dest
LOOKUP-zu-asset_lookup_by_str-dvc
LOOKUP-zu-asset_lookup_by_str-src
CIDR subnet-based asset correlation assets_by_cidr.csv Identity - Asset CIDR Matches - Lookup Gen LOOKUP-zv-asset_lookup_by_cidr-dest
LOOKUP-zv-asset_lookup_by_cidr-dvc
LOOKUP-zv-asset_lookup_by_cidr-src
String-based identity correlation identities_expanded.csv Identity - Identity Matches - Lookup Gen LOOKUP-zy-identity_lookup_expanded-src_user
LOOKUP-zy-identity_lookup_expanded-user
Default field correlation identity_lookup_default_fields.csv
asset_lookup_default_fields.csv
LOOKUP-zz-asset_identity_lookup_default_fields-dest
LOOKUP-zz-asset_identity_lookup_default_fields-dvc
LOOKUP-zz-asset_identity_lookup_default_fields-src
LOOKUP-zz-asset_identity_lookup_default_fields-src_user
LOOKUP-zz-asset_identity_lookup_default_fields-user

Asset fields after processing

Asset fields of the asset lookup after the saved searches perform the merge process.

Field Action taken by ETL
bunit unchanged
city unchanged
country unchanged
dns Accepts pipe-delimited values and converts them to a multi-value field.
lat unchanged
long unchanged
mac Accepts pipe-delimited values and converts them to a multi-value field.
nt_host Accepts pipe-delimited values and converts them to a multi-value field.
owner unchanged
priority unchanged
asset_id Generated from the values of dns, ip, mac, and nt_host fields.
asset_tag Generated from the values of category, pci_domain, is_expected, should_timesync, should_update, requires_av, and bunit fields.
category Appends "pci" if the value contains "cardholder". Accepts pipe-delimited values and converts them to a multi-value field.
ip validates and splits the field into CIDR subnets as necessary. Accepts pipe-delimited values and converts them to a multi-value field.
pci_domain Appends "trust" or "untrust" based on certain field values. Accepts pipe-delimited values and converts them to a multi-value field.
is_expected Normalized to a boolean.
should_timesync Normalized to a boolean.
should_update Normalized to a boolean.
requires_av Normalized to a boolean.
key Generated by the ip, mac, nt_host, and dns fields after the original fields are transformed.

Identity fields after processing

Identity fields of the identity lookup after the saved searches perform the merge process.

Field Action taken by ETL
bunit unchanged
email unchanged
endDate unchanged
first unchanged
last unchanged
managedBy unchanged
nick unchanged
phone unchanged
phone2 unchanged
prefix unchanged
priority unchanged
startDate unchanged
suffix unchanged
work_city unchanged
work_country unchanged
work_lat unchanged
work_long unchanged
watchlist Normalized to a boolean.
category Appends "pci" if the value contains "cardholder". Accepts pipe-delimited values and converts them to a multi-value field.
identity Generated based on values in the input row and conventions specified in the Identity Lookup Configuration. Accepts pipe-delimited values and converts them to a multi-value field.
identity_id Generated from the values of identity, first, last, and email.
identity_tag Generated from the values of bunit, category, and watchlist.

Test the asset and identity merge process

Test the asset and identity merge process to confirm that the data produced by the merge process is expected and accurate. Run the saved searches that perform the merge process without outputting the data to the merged lookups to determine what the merge will do with your data without actually performing the merge.

Test the merge process without performing a merge and outputting the data to a lookup.

  1. From the Splunk ES menu bar, select Configure > Content Management.
  2. Locate the first of the three primary saved searches Identity - Asset CIDR Matches - Lookup Gen.
  3. Click the search name to open it.
  4. Copy the search from the Search field.
  5. Open the Search page.
  6. Paste the search and remove the `output_*` macro. For example, change | `asset_sources` | `make_assets_cidr` | `output_assets("SA-IdentityManagement", "assets_by_cidr.csv")` to | `asset_sources` | `make_assets_cidr`.
  7. Run the search.
  8. Repeat steps 2-7 for the other two searches, Identity - Asset String Matches - Lookup Gen and Identity - Identity Matches - Lookup Gen.

Force a merge

Run the primary saved searches directly to force a merge immediately without waiting the five minutes for the scheduled search to run.

  1. Open the Search page.
  2. Run the primary saved searches.

    | savedsearch "Identity - Asset String Matches - Lookup Gen"


    | savedsearch "Identity - Asset CIDR Matches - Lookup Gen"


    | savedsearch "Identity - Identity Matches - Lookup Gen"

Customize the asset and identity merge process

You can modify the saved searches that perform the asset and identity merge process to perform additional field transformations or data sanitization. Add any operations that you want to change in the merge process to the search before the `output_*` macro.

Caution: Certain modifications to the saved searches are unsupported and could break the merge process or asset and identity correlation. Do not perform the following actions.

  • Add or delete fields from the output.
  • Change the output location to a different lookup table or a KV store collection.
  • Replace the `output_*` macros with the outputlookup command.
Last modified on 24 February, 2017
PREVIOUS
Configure asset and identity correlation in Splunk Enterprise Security
  NEXT
Asset and identity lookup header and field reference

This documentation applies to the following versions of Splunk® Enterprise Security: 4.5.0, 4.5.1, 4.5.2, 4.5.3, 4.6.0 Cloud only


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters