Asset and Identity management
To effectively detect security intrusions, an organization must be able to correlate events in log data with specific assets and identities that may be responsible for, or affected by the intrusion. Splunk Enterprise Security uses an asset and identity system to correlate asset and identity information with events to enrich and provide context to your data. This system takes information form external data sources to populate lookups, which are then correlated with events at search time.
The Identity Management dashboard
The asset and identity lookups are managed through the Identity Management configuration dashboard. Browse to Configure > Data Enrichment and select Identity Management to review a list of configured assets and identities.
Field | Description |
---|---|
Name | A short descriptive name for the list. Selecting the link will open the Identity Manager Settings page and display the configuration of the chosen list. |
Category | A descriptive category name for the list. |
Description | A description of the contents. |
Type | Defines the lookup as an asset or an identity list. |
Source | The lookup definition name for the list. Selecting the link will open the Edit Lookup page with the contents of the chosen list. |
Status | Enabled or Disabled. Changing the status will initiate a merge at the next scheduled interval. |
Actions | Selecting Clone opens the Identity Manger Settings page with duplicated information from the selected list. |
Edit an existing asset or identity list
Edit an asset or identity list from Identity Management:
- In Enterprise Security, go to Configure > Data Enrichment and select Identity Management. A list of asset and identity files for Enterprise Security are displayed.
- Find the name of the asset or identity list you want to edit, and select Source. The list will open in an interactive editor.
- Use the scroll bars to view the columns and rows in the table. Double click a cell to add, change, or remove content.
- Click Save when you are finished.
You can also use the lookup editor in Lists and Lookups to edit an existing list.
- In Enterprise Security, go to Configure > Data Enrichment and click Lists and Lookups.
- Select the name of the list you want to edit. The list will open in an interactive editor.
- Use the scroll bars to view the columns and rows in the table. Double click a cell to add, change, or remove content.
- Click Save when you are finished.
Changes made to an asset or identity list will be reflected in search results after the next scheduled merge. For more information on the merging of asset and identity lists, see Merging the asset and identity lists in this topic.
Adding asset data
An asset represents any device or system in the environment that generates data. Asset correlation allows indexed events to be matched against a defined list of assets. When a match occurs, the original indexed event gains new fields through association with the asset, enriching the event with information on the asset's priority, location, or other details.
An asset list provides external information about the devices on your system, such as the asset priority, owner, and business unit; the geographic location of the asset; and the asset's DNS and Windows machine name. Some of these fields, such as latitude, longitude, and priority are used on dashboard charts. Other fields, such as business unit and category, are used by the filters at the top of the various domain dashboards. For an overview of asset fields with examples, see Asset correlation in this manual.
To add an asset source to Splunk Enterprise Security:
- Extract the asset data from a source
- Format the data as an asset lookup
- Configure an input for the asset list
- Merge the asset lists
Extract the asset data from a source
The preferred method of adding asset information into ES is through automated capture from an existing asset database. For a list of potential asset sources and collection methods, see Collection methods for assets and identities in this topic.
For an example of extracting asset data from events indexed in the Splunk platform, see Add asset information from indexed events in this topic.
To populate an asset list manually, see Static asset and identity information in this topic.
Format the data as an asset lookup
For a list of the fields and values in an asset list, see Asset lookup fields in this manual. The resulting file must be a plain text, csv formatted file with Unix line endings, and must include a .csv
filename extension.
For an example asset list, review the demo_assets.csv.default
file in SA-IdentityManagement/package/lookups
.
Defining multihomed hosts
When adding multihomed hosts or devices to an asset list, define each IP address as a unique record with an identical DNS name. The merging process does not support defining a multi-homed host as one record in an asset list.
Configure an input for the asset list
For instructions, see Configuring a new asset or identity list in this topic.
Merge the asset lists
For details, see Merging the asset and identity lists in this topic.
Adding identity data
An identity represents a user, credential, or a role used to grant access to a device or system. Identity correlation allows indexed events to be matched against a defined list of users or system accounts. When a match occurs, the original indexed event gains new fields through association with an identity, enriching the event with information on the identity's priority, role, or the functional area to which it belongs.
To add an identity source for use in Enterprise Security:
- Extract the identity data from a source
- Format the data as an identity lookup
- Configure an input for the identity list
- Merge the identity lists
The identity lists provides information about the users on your system, such as the screen or login name, first and last name, and email address. Some of these fields, such as priority
, watchlist
, and endDate
are used for dashboard charts and to calculate the urgency of notable events associated with identities. Other fields, such as business unit and category, are used by the filters at the top of the domain dashboards.
Extract the identity data from a source
The preferred method of adding identity information into ES is through automated capture from an existing identity database. For a list of potential sources and collection methods, see Collection methods for assets and identities in this topic.
To populate an identity list manually, see Static asset and identity information in this topic.
Format the data as an identity lookup
For a list of the fields and values in an identity list, see Identity lookup fields in this manual. The resulting file must be a plain text, csv formatted file with Unix line endings, and must include a .csv
filename extension.
For an example identity list, review the demo_identities.csv.default
file in SA-IdentityManagement/package/lookups
.
Configure an input for the identity list
For instructions, see Configuring a new asset or identity list in this topic.
Merge the identity lists
For details, see Merging the asset and identity lists in this topic.
Configuring a new asset or identity list
- Configure and upload the new lookup table file.
- Browse to Settings > Lookups > Lookup table files.
- Choose Add New.
- Select a Destination App of
SA-IdentityManagement
. - Select the lookup file to upload. The file must be a plain text csv format file with Unix line endings and include a
.csv
filename extension. Example:network_assets_from_CMDB.csv
- Provide the destination file name. Enter the name this lookup table file will have on the Splunk server. The name should include a
.csv
filename extension. For example,network_assets_from_CMDB.csv
- Save.
- Set the permissions on the lookup table file.
- In Lookup Table Files find the new lookup and select Permissions.
- Set Object should appear in to All apps.
- Set Read access for Everyone.
- Set Write access for
admin
or other roles. See Adding capabilities to a role in the Installation and Upgrade Manual. - Save.
- Add a new lookup definition.
- Browse to Settings > Lookups > Lookup definitions.
- Choose Add New.
- Select a Destination App of
SA-IdentityManagement
. - Provide a name for the lookup source. The name defined here must be the name used in the Identity Management input stanza definition. Example: network_assets_from_CMDB
- Select a Type of File based.
- Select the lookup table file created. Example:
network_assets_from_CMDB.csv
- Save.
- Set the permissions on the lookup definition.
- In Lookup definitions, find the new definition by its name and select Permissions.
- Set Object should appear in to All apps.
- Set Read access for Everyone.
- Set Write access for
admin
or other roles. See Adding capabilities to a role in the Installation and Upgrade Manual. - Save.
- Add a new source input stanza.
- Browse to Configure > Identity Management > Identity Manager.
- Select New.
- Add the information about the list to the fields in Identity Manager Settings. Fill out the required fields in the new source input.
- Define a Category, the new asset or identity list's short descriptive name. For example, CMDB_network_assets.
- Add a description of the contents.
- Set a Type of "asset" or "identity". For example, asset.
- Set Source to refer to the lookup definition name. For example,
lookup://network_assets_from_CMDB
. - Save.
- Verify the new lookup based source was imported. See Validate assets and identities are working in this topic.
Merging the asset and identity lists
The contents of all configured and enabled asset and identity lists in Identity Management are merged by a modular input scheduled to run every 5 minutes.
For asset correlation, the files are merged and then expanded, cross-referenced lookup files are created. When an event contains the fields: src
, dest
, host
, orig_host
, dvc
two comparisons are performed: one to check if the field value corresponds to a value in the asset table using a string match, and the other to check if the field value corresponds to a value in the asset table using a CIDR subnet match.
For identity correlation, the lookup files are merged and two expanded lookup files are created. When an event contains the fields: user
and src_user
a comparison is performed to check if the field value corresponds to a value in the identities table.
Function | Table Name | Lookup name |
---|---|---|
String-based asset correlation | assets_by_str.csv | LOOKUP-zu_asset_lookup_host_as_str_only LOOKUP-zu_asset_lookup_orig_host_as_str_only LOOKUP-zu_asset_lookup_src_as_str_only LOOKUP-zu_asset_lookup_dest_as_str_only LOOKUP-zu_asset_lookup_dvc_as_str_only |
CIDR subnet-based asset correlation | assets_by_cidr.csv | LOOKUP-zv_asset_lookup_host_as_cidr_only LOOKUP-zv_asset_lookup_orig_host_as_cidr_only LOOKUP-zv_asset_lookup_src_as_cidr_only LOOKUP-zv_asset_lookup_dest_as_cidr_only LOOKUP-zv_asset_lookup_dvc_as_cidr_only |
String-based identity correlation | identities_expanded.csv | LOOKUP-zy_identity_lookup_src_user_only LOOKUP-zy_identity_lookup_user_only |
Default field correlation | asset_identity_lookup_default_fields .csv | LOOKUP-zz-asset_identity_lookup_default_fields |
The automatic lookups that drive ES asset and identity correlation reside in the SA-IdentityManagement
app, and are defined in the SA-IdentityManagement/default/props.conf
file.
All asset and identities lookup files are stored in the path: $SPLUNK_HOME/etc/apps/SA-IdentityManagement/lookups/
The asset and identities lookups are applied to every search on the system, and not scoped to a source or sourcetype.
Force a merge
To perform an immediate check and merge of updates to the assets and identities lists, the modular input can be run from the CLI. Calling an input script from the CLI requires the $SPLUNK_HOME
environment variable to be set; in order to do this, run the following:
On *nix:
source /opt/splunk/bin/setSplunkEnv
On Windows:
splunk.exe envvars > setSplunkEnv.bat & setSplunkEnv.bat
Run merge:
$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config identity_manager | $SPLUNK_HOME/bin/python $SPLUNK_HOME/etc/apps/SA-IdentityManagement/bin/identity_manager.py --username=admin
Credentials are required. The user will be prompted to provide a password for the --username
defined.
When the identity manager input is triggered, it will evaluate all enabled lookups and check for changes. If no changes have been made to any lookups since the last run, the identity manager input will not regenerate the merged lookup files.
To force a merge of the assets or identities lists at the next interval, a file creation in the modinputs folder can be run from the CLI.
touch $SPLUNK_HOME/var/lib/splunk/modinputs/identity_manager/force_asset
touch $SPLUNK_HOME/var/lib/splunk/modinputs/identity_manager/force_identity
Enabling, disabling, or changing the configuration of an asset or identity list in Identity Management will begin a merge at the next scheduled interval.
Verify the merging process
To verify that the expansion process has completed, search the _internal
index.
To display the last time the merge occurred:
index=_internal source=*python_modular_input.log "Updated: target lookup table"
To display successive runs adding the events with no merging required:
index=_internal source=*python_modular_input.log "Updated: target lookup table" OR "No merging required"
The most common reason for failure is incorrect formatting or invalid data in the asset or identities lookup files.
Note: The merge process checks for new input every 5 minutes, but does not perform any work unless an asset or identity table has been modified. To force a merge of the contents in the merged asset and identity lists, update a lookup table with new information, or disable and enable an list in Identity Management.
Validate assets and identities are working
To test an asset lookup using a search, choose a record with data the ip
, mac
, nt_host
, or dns
fields from an asset list and search for it:
| stats count | eval src="1.2.3.4" | `get_asset(src)`
To view the available assets using a dashboard, browse to Security Domains > Identity > Asset Center. For more information, see Asset Center dashboard in this manual.
To test an identity lookup using a search, choose any record's identity
field from an identities list and search for it: | stats count | eval user="VanHelsing" | `get_identity4events(user)`
To view the available identities using a dashboard, browse to Security Domains > Identity > Identity Center. For more information, see Identity Center dashboard in this manual.
To view all available assets using the search command.
| `assets`
To view all available assets using the data model. |`datamodel("Identity_Management", "All_Assets")` |`drop_dm_object_name("All_Assets")`
Updating assets and identities
As an organization's asset and identities information changes frequently, it is best to update these lists automatically. This reduces the overhead and maintenance that manual updating requires, and improves data integrity. There are several ways to do this.
- Use DBConnect or another Splunk platform add-on to connect to an external database or repository.
- Use scripted inputs to import and format the lists.
- Use events indexed in the Splunk platform with a search to collect, sort, and export the data to a list.
Static asset and identity information
Edit the "static_assets" and "static_identities" lists to manually include new asset or identity information.
- In Enterprise Security, go to Configure > Data Enrichment and click Lists and Lookups.
- Select the "static_assets" or "static_identities" list. The list will open in an interactive editor.
- Use the scroll bars to view the columns and rows in the table. Double click in a cell to add, change, or remove content.
- Click Save when you are finished.
Collection methods for assets and identities
The preferred collection method for asset or identity information is through a Splunk platform add-on. Many add-ons can be used to automate connections to external systems for data collection. Use an add-on to connect, collect, and return data to Enterprise Security.
Suggested collection methods for assets and identiites.
Technology | Assets or Identities | Collection methods |
---|---|---|
Active Directory | Both | SA-ldapsearch and a custom search. For an example, see Add identity information from Active Directory in this topic. |
LDAP | Both | SA-ldapsearch and a custom search. |
CMDB | Assets | DB Connect and a custom search. |
ServiceNow | Both | Splunk Add-on for ServiceNow |
Asset Discovery | Assets | Asset Discovery App |
Bit9 | Assets | Splunk Add-on for Bit9 and a custom search. |
Cisco ISE | Both | Splunk Add-on for Cisco ISE and a custom search. |
Microsoft SCOM | Assets | Splunk Add-on for Microsoft SCOM and a custom search. |
Okta | Identities | Splunk Add-on for Okta and a custom search. |
Sophos | Assets | Splunk Add-on for Sophos and a custom search. |
Symantec Endpoint Protection | Assets | Splunk Add-on for Symantec Endpoint Protection and a custom search. |
Splunk platform | Assets | Add asset information from indexed events. |
Adding information from Active Directory
- Install and configure the Splunk Support for Active Directory app.
- Create and add a lookup file as a source of asset or identity information. See Loading a new asset or identities list for details. This lookup file configuration will become the target for a saved search to populate the lookup table file with information from Active Directory. When testing the AD integration, consider disabling the new lookup file configuration to prevent unnecessary merges by the Identity Manager modular input.
- Using the "
ldapsearch
" command provided with SA-ldapsearch, construct a search that polls Active Directory (AD) and places the results into a file. The exact syntax for this search will vary depending on the AD configuration.
Identity collection example:
|ldapsearch domain=<domain_name> search="(&(objectclass=user)(!(objectClass=computer)))" |makemv userAccountControl |search userAccountControl="NORMAL_ACCOUNT" |eval suffix="" |eval priority="medium" |eval category="normal" |eval watchlist="false" |eval endDate="" |table sAMAccountName,personalTitle,displayName,givenName,sn,suffix,mail,telephoneNumber,mobile,manager,priority,department,category,watchlist,whenCreated,endDate |rename sAMAccountName as identity, personalTitle as prefix, displayName as nick, givenName as first, sn as last, mail as email, telephoneNumber as phone, mobile as phone2, manager as managedBy, department as bunit, whenCreated as startDate |outputlookup my_identity_lookup
Note: This search assigns static values for "suffix
", "endDate
", "category
", "watchlist
", and "priority
". After a working search has been constructed and tested, you can replace static values with information from AD.
Asset collection example:
|ldapsearch domain=<domain name> search="(&(objectClass=computer))" |eval city="" |eval country="" |eval priority="medium" |eval category="normal" |eval dns=dNSHostName |eval owner=managedBy |rex field=sAMAccountName mode=sed "s/\$//g" |eval nt_host=sAMAccountName |makemv delim="," dn |rex field=dn "(OU|CN)\=(?<org>.+)" |table ip,mac,nt_host,dns,owner,priority,lat,long,city,country,bunit,category,pci_domain,is_expected,should_timesync,should_update,requires_av | outputlookup create_empty=false createinapp=true my_asset_lookup
Note: This search assigns static values for several fields. After a working search has been constructed and tested, you can replace static values with information from AD.
Add asset information from indexed events
Hosts communicating with the Splunk platform can be compared to the existing asset information using search commands. The table of unmatched hosts can be reviewed and exported as an asset list. Example:
| `host_eventcount` | search host_is_expected=false NOT host_asset_id=* | fields - firstTime,recentTime,lastTime,_time, host_owner_*,host_asset_tag,host_asset_id | sort -totalCount,dayDiff | table host,ip,mac,nt_host,dns,owner,priority,lat,long,city,country,bunit,category,pci_domain,is_expected,should_timesync,should_update,requires_av
User Activity Monitoring | Asset and Identity correlation |
This documentation applies to the following versions of Splunk® Enterprise Security: 4.1.0, 4.1.1, 4.1.2, 4.1.3, 4.1.4
Feedback submitted, thanks!