How Splunk Enterprise Security processes and merges asset and identity data
Splunk Enterprise Security takes the asset and identity data that you add as lookups and generates combined lookup files. Splunk Enterprise Security uses the generated lookup files to correlate asset and identity data with events using automatic lookups.
- You collect asset and identity data from data sources using an add-on and a custom search or manually with a CSV file. See Collect and extract asset and identity data.
- You format the data as a lookup, using a search or manually with a CSV file. See Format the asset or identity list as a lookup.
- You configure the list as a lookup table, definition, and input. See Configure a new asset or identity list.
- Splunk Enterprise Security identity manager modular input detects changed content in the
identity_manager://<input_name>
and changes to stanzas in the input.
- Splunk Enterprise Security identity manager modular input detects changed content in the
- You configure any settings in the identity lookup configuration setup. See Define identity formats on the identity configuration page.
- Splunk Enterprise Security identity manager modular input updates settings in the
transforms.conf
stanzaidentity_lookup_expanded
- Splunk Enterprise Security identity manager modular input updates settings in the
- Splunk Enterprise Security identity manager modular input updates the macros used to identify the input sources based on the currently enabled stanzas in
inputs.conf
. For example, update the`generate_identities`
macro dynamically based on the conventions specified on the Identity Lookup Configuration page. - Splunk Enterprise Security identity manager modular input dispatches lookup generating saved searches if it identifies changes that require the asset and identity lists to be merged.
- Splunk Enterprise Security merges all configured and enabled asset and identity lists using lookup generating saved searches.
- The primary saved searches concatenate the lookup tables referenced by the identity manager input, generate new fields, and output the concatenated asset and identity lists into target lookup table files.
- Secondary saved searches generate lookup tables for asset categories, identity categories, and asset PCI domains (in Splunk App for PCI Compliance).
- You verify that the data looks as expected. See Verify that your asset or identity data was added to Splunk Enterprise Security.
The merging of identity and asset lookups does not validate or de-duplicate input. Errors from the identity manager modular input are logged in identity_manager.log
. This log does not show data errors.
Merging assets and identities creates new lookup files
After the merging process completes, four lookups store your asset and identity data.
Function | Table name | Saved search | Lookup name |
---|---|---|---|
String-based asset correlation | assets_by_str.csv | Identity - Asset String Matches - Lookup Gen | LOOKUP-zu-asset_lookup_by_str-dest LOOKUP-zu-asset_lookup_by_str-dvc LOOKUP-zu-asset_lookup_by_str-src |
CIDR subnet-based asset correlation | assets_by_cidr.csv | Identity - Asset CIDR Matches - Lookup Gen | LOOKUP-zv-asset_lookup_by_cidr-dest LOOKUP-zv-asset_lookup_by_cidr-dvc LOOKUP-zv-asset_lookup_by_cidr-src |
String-based identity correlation | identities_expanded.csv | Identity - Identity Matches - Lookup Gen | LOOKUP-zy-identity_lookup_expanded-src_user LOOKUP-zy-identity_lookup_expanded-user |
Default field correlation | identity_lookup_default_fields.csv asset_lookup_default_fields.csv |
LOOKUP-zz-asset_identity_lookup_default_fields-dest LOOKUP-zz-asset_identity_lookup_default_fields-dvc LOOKUP-zz-asset_identity_lookup_default_fields-src LOOKUP-zz-asset_identity_lookup_default_fields-src_user LOOKUP-zz-asset_identity_lookup_default_fields-user |
Asset fields after processing
Asset fields of the asset lookup after the saved searches perform the merge process.
Field | Action taken by ETL |
---|---|
bunit | unchanged |
city | unchanged |
country | unchanged |
dns | Accepts pipe-delimited values and converts them to a multi-value field. |
lat | unchanged |
long | unchanged |
mac | Accepts pipe-delimited values and converts them to a multi-value field. |
nt_host | Accepts pipe-delimited values and converts them to a multi-value field. |
owner | unchanged |
priority | unchanged |
asset_id | Generated from the values of dns, ip, mac, and nt_host fields. |
asset_tag | Generated from the values of category, pci_domain, is_expected, should_timesync, should_update, requires_av, and bunit fields. |
category | Appends "pci" if the value contains "cardholder". Accepts pipe-delimited values and converts them to a multi-value field. |
ip | validates and splits the field into CIDR subnets as necessary. Accepts pipe-delimited values and converts them to a multi-value field. |
pci_domain | Appends "trust" or "untrust" based on certain field values. Accepts pipe-delimited values and converts them to a multi-value field. |
is_expected | Normalized to a boolean. |
should_timesync | Normalized to a boolean. |
should_update | Normalized to a boolean. |
requires_av | Normalized to a boolean. |
key | Generated by the ip, mac, nt_host, and dns fields after the original fields are transformed. |
Identity fields after processing
Identity fields of the identity lookup after the saved searches perform the merge process.
Field | Action taken by ETL |
---|---|
bunit | unchanged |
unchanged | |
endDate | unchanged |
first | unchanged |
last | unchanged |
managedBy | unchanged |
nick | unchanged |
phone | unchanged |
phone2 | unchanged |
prefix | unchanged |
priority | unchanged |
startDate | unchanged |
suffix | unchanged |
work_city | unchanged |
work_country | unchanged |
work_lat | unchanged |
work_long | unchanged |
watchlist | Normalized to a boolean. |
category | Appends "pci" if the value contains "cardholder". Accepts pipe-delimited values and converts them to a multi-value field. |
identity | Generated based on values in the input row and conventions specified in the Identity Lookup Configuration. Accepts pipe-delimited values and converts them to a multi-value field. |
identity_id | Generated from the values of identity, first, last, and email. |
identity_tag | Generated from the values of bunit, category, and watchlist. |
Test the asset and identity merge process
Test the asset and identity merge process to confirm that the data produced by the merge process is expected and accurate. Run the saved searches that perform the merge process without outputting the data to the merged lookups to determine what the merge will do with your data without actually performing the merge.
Test the merge process without performing a merge and outputting the data to a lookup.
- From the Splunk ES menu bar, select Configure > Content Management.
- Locate the first of the three primary saved searches Identity - Asset CIDR Matches - Lookup Gen.
- Click the search name to open it.
- Copy the search from the Search field.
- Open the Search page.
- Paste the search and remove the
`output_*`
macro. For example, change| `asset_sources` | `make_assets_cidr` | `output_assets("SA-IdentityManagement", "assets_by_cidr.csv")`
to| `asset_sources` | `make_assets_cidr`
. - Run the search.
- Repeat steps 2-7 for the other two searches, Identity - Asset String Matches - Lookup Gen and Identity - Identity Matches - Lookup Gen.
Force a merge
Run the primary saved searches directly to force a merge immediately without waiting the five minutes for the scheduled search to run.
- Open the Search page.
- Run the primary saved searches.
| savedsearch "Identity - Asset String Matches - Lookup Gen"
| savedsearch "Identity - Asset CIDR Matches - Lookup Gen"
| savedsearch "Identity - Identity Matches - Lookup Gen"
Customize the asset and identity merge process
You can modify the saved searches that perform the asset and identity merge process to perform additional field transformations or data sanitization. Add any operations that you want to change in the merge process to the search before the `output_*`
macro.
Caution: Certain modifications to the saved searches are unsupported and could break the merge process or asset and identity correlation. Do not perform the following actions.
- Add or delete fields from the output.
- Change the output location to a different lookup table or a KV store collection.
- Replace the
`output_*`
macros with theoutputlookup
command.
Configure asset and identity correlation in Splunk Enterprise Security | Asset and identity lookup header and field reference |
This documentation applies to the following versions of Splunk® Enterprise Security: 4.5.0, 4.5.1, 4.5.2, 4.5.3, 4.6.0 Cloud only
Feedback submitted, thanks!