Splunk® Enterprise Security

Install and upgrade Splunk Enterprise Security

Acrobat logo Download manual as PDF


This documentation does not apply to the most recent version of Splunk® Enterprise Security. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

File conversion example

This topic covers how to convert the file configurations that have changed between the Splunk Enterprise Security Suite 1.1.x, the Splunk App for Enterprise Security 2.0.x, the Splunk App for Enterprise Security 2.4, and the Splunk App for Enterprise Security 3.0.

Note: Users of the Splunk Enterprise Security Suite 1.1.x need to perform a clean install or upgrade to 2.2.x using an older installer. The current installer does not support upgrades from Splunk Enterprise Security Suite 1.1.x. Contact Splunk Support for more information.

Overview

To upgrade from the Splunk App for Enterprise Security 2.2.x to version 2.4.x, review:

To manually upgrade from the Splunk Enterprise Security Suite 1.1.2 to the Splunk App for Enterprise Security 3.0 you will need to:

Additional information:

Upgrade from 2.x to 2.4

Convert correlation searches from 2.x to 2.4

The following sections apply to upgrades from Enterprise Security 2.x to Enterprise Security 2.4.x.

Note: These steps only apply to ES versions less than 2.0.* to 2.2. If the environment is 2.4 and above, then these file conversions are not necessary.

User-modified default correlation search

If there are no modifications to the default correlation searches, there is nothing to convert. The upgrade process will introduce the new correlation searches.

If there are modifications to the default correlation searches (for example if there is a local version of any given default correlation search), you will need to evaluate the differences between your local version and the new default version. In most cases, it is better to use the new default version, since there are usually enhancements or improvements to search logic.

To enable the new default correlation searches, simply delete your existing local modifications.

User-created custom correlation searches

Custom correlation searches need to be modified to work within the the Splunk App for Enterprise Security correlation search framework. There are new .conf files and settings that enhance the notable event framework.

Savedsearches.conf

Search String

The Splunk App for Enterprise Security 2.0.x used a custom search command (aggregate.py) to provide event aggregation to correlation search results, in order to generate of notable events:

For example: 

[Access - Insecure Or Cleartext Authentication - Rule]

In the Splunk App for Enterprise Security 2.0.x (old):

search = (tag=insecure OR tag=cleartext) NOT sourcetype=stash `authentication` | `get_event_id` | stats first(_raw) as orig_raw,first(event_id) as orig_event_id,count by app,dest | aggregate search="Access - Insecure Or Cleartext Authentication - Rule"

In the Splunk App for Enterprise Security 2.4.x and higher (new):

search = (tag=insecure OR tag=cleartext) NOT sourcetype=stash `authentication` | `get_event_id` | stats first(_raw) as orig_raw,first(event_id) as orig_event_id,count by app,dest

The Splunk App for Enterprise Security 2.0.x correlation searches had the following appended to the search string: 

  | aggregate search="Access - Insecure Or Cleartext Authentication - Rule"

The aggregate command associated metadata (for example, correlation search title, aggregation fields, and so on) and aggregated search results via settings specified to create a notable event.

This aggregate search command has been deprecated for the Splunk App for Enterprise Security 2.4.x and higher. The per-event-alerting feature in Splunk provides the aggregation capability directly via a savedsearch setting. Refer to "Define per-result alerts" in the Alerting Manual, part of the core Splunk product documentation, for more information.

Search settings

Correlation search (notable event) metadata settings used to reside as settings in savedsearches.conf. These settings are now located in a new correlationsearches.conf file, described below.

Aggregate.conf

The aggregate.conf file contained settings to configure aggregation rules for correlation search results.

The Splunk App for Enterprise Security leverages the Splunk per-event-alerting feature, deprecating aggregate.py. The aggregation rule settings have been moved into the correlation searches' saved search declaration. Refer to "Define per-result alerts" in the Alerting Manual, part of the core Splunk product documentation, for more information.

In the Splunk App for Enterprise Security 2.0.x (old) - aggregate.conf 

[Access - Insecure Or Cleartext Authentication - Rule]

index=notable
duration=86400
group_by=app,dest

This information has been moved to savedsearches.conf.

In the Splunk App for Enterprise Security 2.4.x and higher (new) - savedsearches.conf 

 [Access - Insecure Or Cleartext Authentication - Rule]

...
action.summary_index._name = notable
alert.suppress.fields = app,dest
alert.suppress.period = 86300
...

Note: The aggregate.conf file has been deprecated and should be removed after migrating the information it contains to the savedsearches.conf file. If the aggregate.conf file is left in place, Splunk may report that there are stanza errors at start up because the aggregate search command no longer exists.

Governance conversions

This table shows the changes in the location and format of the governance files in the Splunk App for Enterprise Security 2.0.x and the Splunk App for Enterprise Security 2.4 and higher.

Version Filename Domain App
Splunk App for Enterprise Security 2.0.x correlationsearches.conf SA or DA domain app
Splunk App for Enterprise Security 2.4.x or higher governance.conf SA or DA domain app

For upgrades from Splunk App for Enterprise Security 2.x to a newer version of the Splunk App for Enterprise Security, you will need to migrate governance information from correlationsearches.conf to governance.conf. The same settings syntax are used as before, but the settings have moved to a different .conf file: 

compliance.<n>.governance = <pci|hipaa|sox|nerc|fisma>
compliance.<n>.control = <string>

See "Configure correlation search governance" in this manual.

Upgrade from 1.1.2 to 3.0

Convert notable review statuses

Notable event review statuses in the Enterprise Security Suite 1.1.x were stored within a CSV file named review_statii.csv (SA-ThreatIntelligence/lookups/review_statii.csv).

In the Splunk App for Enterprise Security 3.0, review statuses are now stored within an equivalent configuration file named reviewstatuses.conf. The file review_statii.csv has been deprecated and is no longer used in the Splunk App for Enterprise Security 3.0.

The following are the out-of-box review statuses for Enterprise Security Suite 1.1.x contained in review_statii.csv:

  • unreviewed
  • reviewed
  • closed

The mapping of the out-of-the-box statuses from Enterprise Security 1.1.x to the Splunk App for Enterprise Security 3.0 is done automatically by the Enterprise Security Install App. The mapping is saved in a separate file (SA-ThreatIntelligence/default/reviewstatuses.conf) for any pre-existing incident review activity.

Manual conversion of statuses

These statuses have changed:

  • unreviewed > new
  • reviewed > resolved
  • closed > closed

In addition any custom 1.1.2 review statuses - statuses created in addition to the original three, need to be converted into new stanzas in the Splunk for Enterprise Security 3.0 local/reviewstatuses.conf format.

Important Note: In the Splunk App for Enterprise Security 3.0, each review status is represented as its own stanza that is identified by an incrementing integer.

The out-of-box review statuses for Splunk for Enterprise Security 3.0 contained in reviewstatuses.conf 

   ## reviewstatuses.conf
    [0]
    label = Unassigned
    
    [1]
    label = New
    
    [2]
    label = In Progress
    
    [3]
    label = Pending
    
    [4]
    label = Resolved
    
    [5]
    label = Closed

If there is a custom 1.1.x review status, that status will need a corresponding stanza created the next stanza increment.

An example would be - 1.1.x review_statii.csv.

  • unreviewed
  • reviewed
  • closed
  • custom_status_1
  • custom_status_2

The equivalent reviewstatuses.conf in the Splunk App for Enterprise Security 3.0: SA-ThreatIntelligence/local/reviewstatuses.conf 

    ## Enable status "custom_status_1"
    [6]
    disabled = 0 
    label = custom_status_1
    description = Some description for custom_status_1
    
    ## Enable status "custom_status_2"
    [7]
    disabled = 0 
    label = custom_status_2
    description = Some description for custom_status_2

Status Transition Authorization:

For custom review statuses, a transition authorization needs to be created. The transition authorizations allow specified users to transition from existing review statuses to the new/custom review status within the Incident Review dashboard.

To enable each new review status it must have transition capabilities declared and enabled for particular users. These are declared in SA-ThreatIntelligence/local/authorize.conf

First declare the transition capabilities. The convention here is all pre-existing review statuses need to be able to transition to the new custom status. For Splunk for Enterprise Security 2.x and higher there are six out-of-the-box review statuses. Ensure all default review statuses can transition to the custom statuses and vice versa.

In this example below, three custom statuses represented by 7,8, and 9: 

    [capability::transition_reviewstatus-0_to_7]
    disabled = 0
    
    [capability::transition_reviewstatus-0_to_8]
    disabled = 0
    
    [capability::transition_reviewstatus-0_to_9]
    disabled = 0
    
    [capability::transition_reviewstatus-1_to_7]
    disabled = 0
    
    [capability::transition_reviewstatus-1_to_8]
    disabled = 0
    
    [capability::transition_reviewstatus-1_to_9]
    disabled = 0
    
    [capability::transition_reviewstatus-2_to_7]
    disabled = 0
    
    [capability::transition_reviewstatus-2_to_8]
    disabled = 0
    
    [capability::transition_reviewstatus-2_to_9]
    disabled = 0
    
    ...
    ...
    
    [capability::transition_reviewstatus-6_to_7]
    disabled = 0
    
    [capability::transition_reviewstatus-6_to_8]
    disabled = 0
    
    [capability::transition_reviewstatus-6_to_9]
    disabled = 0

Once the capabilities have declared above, within the same authorize.conf these transitions need to be enabled for a particular Splunk Enterprise Security user role. For each transition created, a setting is created under the desired Splunk Enterprise Security user role.

By default, each transition should be at least enabled for admin and analyst roles. 

    [role_ess_admin]
    transition_reviewstatus-0_to_8 = enabled
    transition_reviewstatus-0_to_9 = enabled
    transition_reviewstatus-0_to_7 = enabled
    transition_reviewstatus-6_to_7 = enabled
    transition_reviewstatus-6_to_8 = enabled
    transition_reviewstatus-6_to_9 = enabled
    transition_reviewstatus-7_to_5 = enabled
    transition_reviewstatus-7_to_4 = enabled
    transition_reviewstatus-7_to_6 = enabled
    transition_reviewstatus-7_to_1 = enabled
    transition_reviewstatus-7_to_3 = enabled
    
    [role_ess_analyst]
    transition_reviewstatus-6_to_7 = enabled
    transition_reviewstatus-0_to_8 = enabled
    transition_reviewstatus-0_to_9 = enabled
    transition_reviewstatus-0_to_7 = enabled
    transition_reviewstatus-6_to_8 = enabled
    transition_reviewstatus-6_to_9 = enabled
    transition_reviewstatus-7_to_5 = enabled
    transition_reviewstatus-7_to_4 = enabled
    transition_reviewstatus-7_to_6 = enabled
    transition_reviewstatus-7_to_1 = enabled
    transition_reviewstatus-7_to_3 = enabled
    transition_reviewstatus-7_to_2 = enabled
    transition_reviewstatus-7_to_9 = enabled
    transition_reviewstatus-7_to_8 = enabled

Convert the incident Review Audit CSV file

For the Incident Review Audit dashboard, make the file compatible with new review statuses by updating the SA-ThreatIntelligence/lookups/incident_review.csv file.

In 1.1.x, the file had these headers: 

incident_review.csv > time,rule_id,status,comment,user

In 3.0, new headers were added, owner and urgency: 

incident_review.csv > time,rule_id,owner,urgency,status,comment,user

Example:

In 1.1.x (old):

time,rule_id,status,comment,user "11/08/2011 16:20:13 GMT","Access - Brute Force Access Behavior Detected - Rule11-08-2011PROD-POS01","reviewed","this is reviewed","admin"

"11/08/2011 16:20:38 GMT","High Number of Infections With Infection11-08-2011Trojan.Vundo","closed","this is closed","admin"

"11/08/2011 16:21:12 GMT","Access - Excessive Failed Logins -

Rule11-08-2011sshd91.194.85.54","SL3","assigning status SL3","admin"

Format conversions

In the Splunk App for Enterprise Security version 2.x and higher, two field/columns were introduced: Owner and Urgency.

Essentially, each row will need to have the two new field/columns. Since neither of these fields existed in the 1.1.x version, for existing rows, these fields can be left blank.

In 1.1.x version, the status column contains the verbose review status name. In 2.0 versions and higher, the review status column contains the integer representation of the review status that is used within default and local reviewstatuses.conf.

For any existing rows within 1.1.x manually update the values of the status column with their new equivalent integer representation as declared within the Splunk App for Enterprise Security 3.0 reviewstatuses.conf.

This must be done for the 1.1.x default statuses. 1.1.x default review statuses were:

  • unreviewed > 1 (new)
  • reviewed > 4 (resolved)
  • closed > 5 (closed)

In the example above, there was a custom status with name SL3. The SL3 status number was 9 in that environment. As such any existing row in 1.1.x incident_review.csv that had SL3 as status was replaced with the value 9 for review status.

Correlation search conversions from 1.1.2 to 3.0

To update correlation searches you need to break-out the correlation searches from SA-ThreatIntelligence to their respective SA (supporting add-on) domain apps. Although each correlation search is associated with a particular domain (Access, Audit, Network, and so on), all correlation searches used to live in the SA-ThreatIntelligence app.

In the Splunk App for Enterprise Security 3.0, these searches now live in their respective SA domain apps.

  • Move all correlation search declarations in savedsearches.conf to savedsearches.conf and correlationsearches.conf.

User-modified default correlation search

If there are no modifications to the default correlation searches, there is nothing to convert. The upgrade process will introduce the new correlation searches.

If there are modifications to the default correlation searches (for example if there is a local version of any given default correlation search), you will need to evaluate the differences between your local version and the new default version. In most cases, it is better to use the new default version, since there are usually enhancements or improvements to search logic.

To enable the new default correlation searches, simply delete your existing local modifications.

User-created custom correlation searches

Custom correlation searches need to be modified to work within the 3.0 correlation search framework. There are new .conf files and settings that enhance the notable event framework. See the information for your installed version for instructions to convert these correlation searches.

Savedsearches.conf

Search String:

The Splunk Enterprise Security Suite 1.1.2 used a combination of macros to store correlation search results as notable events:

For example: 

[Access - Insecure Or Cleartext Authentication - Rule]

In the Splunk Enterprise Security Suite 1.1.2 (old):

search = (tag=insecure OR tag=cleartext) `authentication` | `get_event_id` | stats first(_raw) as orig_raw,first(event_id) as orig_event_id,count by app,dest | `get_date(now())` | `get_rule_id(\"Access - Insecure Or Cleartext Authentication - Rule\".date.app.dest)` | eval _time=now()

The Splunk Enterprise Security Suite 1.1.2 correlation searches had the following appended to the search string:

| `get_date(now())` | `get_rule_id(\"Access - Insecure Or Cleartext Authentication - Rule\".date.app.dest)` | eval _time=now()

This macro construct collected and associated correlation search metadata (for example, correlation search title, aggregation fields, and so on) to the notable event. The macro construct has been deprecated for the Splunk App for Enterprise Security 2.x and higher.

Search String with per-event alerting:

The per-event-alerting feature in Splunk 6.0 provides the aggregation capability directly via a savedsearch setting.

In the Splunk App for Enterprise Security 3.0 (new) the search is:

search = (tag=insecure OR tag=cleartext) NOT sourcetype=stash `authentication` | `get_event_id` | stats first(_raw) as orig_raw,first(event_id) as orig_event_id,count by app,dest

These settings in the [Access - Insecure Or Cleartext Authentication - Rule], savedsearches.conf add per-event alerting to the correlation search: 

alert.digest_mode = 1
alert.suppress = 1
alert.suppress.fields = app,dest
alert.suppress.period = 86300
alert.track = false

The $SPUNK_HOME/etc/apps/SA-AccessProtection/default/savedsearches.conf settings look like this: 

....
[Access - Insecure Or Cleartext Authentication - Rule]
action.email.sendresults = 0
action.summary_index = 1
action.summary_index._name = notable
alert.digest_mode = 1
alert.suppress = 1
alert.suppress.fields = app,dest
alert.suppress.period = 86300
alert.track = false
cron_schedule = */5 * * * *
disabled = True
dispatch.earliest_time = rt-5m@m
dispatch.latest_time = rt+5m@m
dispatch.rt_backfill = 1
enableSched = 1
search = (tag=insecure OR tag=cleartext) NOT sourcetype=stash `authentication` | `get_event_id` | 
stats first(_raw) as orig_raw,first(event_id) as orig_event_id,count by app,dest
...

The per-event alert information is in the entries that begin alert. and the actual search for the rule is at the end. Refer to "Define per-result alerts" in the Alerting Manual, part of the core Splunk product documentation, for more information.

Search Settings:

Correlation search (notable event) metadata settings used to live as settings in savedsearches.conf. These settings are now in a new correlationsearches.conf file, described below.

Correlationsearches.conf

Correlationsearches.conf is a new file that maintains metadata for the notable events generated by a correlation search. The key-value fields formerly resided in the savedsearches.conf file. Now they are separated out into a separate correlationsearches.conf file.

The notable event metadata now lives in the Access - Insecure or Cleartext Authentication - Rule. 

[Access - Insecure Or Cleartext Authentication - Rule]
security_domain = access
severity = high
rule_name = Insecure Or Cleartext Authentication Detected
description = Detects authentication requests that transmit the password over the network as cleartext (unencrypted)
rule_title = 
rule_description = The device $dest$ is accepting insecure or cleartext $app$ authentication
drilldown_name = View insecure or cleartext authentication requests for $app$ on device $dest$
drilldown_search = (tag=insecure OR tag=cleartext) `authentication` | search dest=$dest$ app=$app$
default_status =
default_owner =

Governance conversions

This table shows the changes in the location and format of the governance files.

Version Filename Domain App
Splunk Enterprise Security Suite 1.1.2 savedsearches.conf SA-ThreatIntelligence
Splunk App for Enterprise Security 2.4 governance.conf SA or DA domain app
Splunk App for Enterprise Security 3.0 governance.conf SA or DA domain app

Governance used to be defined in savedsearches.conf. For upgrades from the Splunk Enterprise Security Suite 1.1.2 to the Splunk App for Enterprise Security 3.0 you will need to migrate governance definitions from savedsearches.conf to governance.conf.

Refer to "Configure correlation search governance" in this manual for more information.

In the solution add-on (SA) or domain add-on (DA) that contains the correlation search, use the following format to add governance to the local/governance.conf file.

Convert correlation searches from 1.1.x to 2.0.x

Note: These steps are only applicable if you are upgrading from the Splunk Enterprise Security Suite 1.1.x to the Splunk App for Enterprise Suite 2.0.x. If you are upgrading from the Splunk Enterprise Security Suite 1.1.x to the Splunk App for Enterprise Security 2.4, these steps do not apply.

Correlation Searches: savedsearches.conf > aggregate.conf, correlationsearches.conf, savedsearches.conf

  • Break-out of searches from SA-ThreatIntelligence > Access, Endpoint, Network, Audit

The Splunk Enterprise Security Suite 1.1.x correlation searches all lived in SA-ThreatIntelligence/default/savedsearches.conf.

They were identified as stanzas that have header with a specific naming convention: 

[<DOMAIN> - <RULE NAME> - RULE]
action.summary_index._name = notable
action.summary_index.security_domain = <DOMAIN>

Example: 

[Access - Brute Force Access Behavior Detected - Rule]
action.summary_index = 1 
action.summary_index._name = notable
action.summary_index.security_domain = access
...
...

In the Splunk App for Enterprise Security 2.0 and higher, correlation searches are defined in .conf files located in separate SAs. These include:

  • aggregate.conf
  • correlationsearches.conf
  • savedsearches.conf

Overall, the settings of a correlation search in the Splunk Enterprise Security Suite 1.1.x format still exist, but they are now split between the three new .conf files.

These .conf files exist in the SA folder for their respective domain.

The breakdown of the settings: 

 [<DOMAIN> - <RULE NAME> - RULE]

Splunk Enterprise Security Suite 1.1.x format:

action.email.sendresults = 0 
   action.summary_index = 1 
   action.summary_index._name = notable
   action.summary_index.security_domain = access
   action.summary_index.rule_description = The system $src$ has failed authentication $failure$ times and successfully authenticated $success$ times in the last hour
   #action.summary_index.rule_match = date,src
   action.summary_index.rule_name = Brute Force Access Behavior Detected
   action.summary_index.rule_title = Brute Force Access Behavior Detected From $src$
   action.summary_index.severity = high
   action.summary_index.drilldown_search = src=$src$ `authentication`
   action.summary_index.drilldown_name = View all login attempts by system $src$
   description = Detects excessive number of failed login attempts along with a successful attempt (this could indicate a successful brute force attack)
   cron_schedule = 5,20,35,50 * * * * 
   dispatch.earliest_time = -65m@m
   dispatch.latest_time = -5m@m
   enableSched = 1
search = `authentication` | chart count over src by action | search failure>6 success>0 | `get_date(now())` | `get_rule_id(\"Access - Brute Force Access Behavior Detected - Rule\".date.src)` | eval _time=now()

Mapping of settings from 1.1.x to 2.0.x correlation searches

Danny: Will this mapping work for 3.0? Or do they need to upgrade to 2.x and then re-map?

Setting location in the Splunk Enterprise Security Suite1.1.x > New setting location in the Splunk App for Enterprise Security 2.0.x 

--------------------------------------------------------
<DOMAIN>/local/aggregate.conf
<new field>  > i ndex=notable
<new field>  >  duration=86400
<new field>  >  group_by=

index: This setting is effectively the Splunk Enterprise Security Suite 1.1.x action.summary_index._name setting, but now with a different name.

duration: This setting is the duration that correlation search results should be aggregated. Default value should be 86400 seconds.

group_by: This new setting's value consists of a substring from the Splunk Enterprise Security Suite 1.1.x format's 'search' setting. Specifically:

Example:

search = `useraccounts` | search NOT password=* NOT password="" NOT password=x | `tag2field(interactive)` | `get_event_id` | stats count,first(_raw) as orig_raw,first(event_id) as orig_event_id by dest,user,password,is_interactive | `get_date(now())` | `get_rule_id(\"Access - Cleartext Password At Rest - Rule\".date.dest.user.password)` | eval _time=now()

The group_by field's value will take on a substring from the Splunk Enterprise Security Suite 1.1.x search above. They will be the field names that follow the `get_rule_id` macro following the ".date" string. It can be identified by this regular expression "\.date([\w\.]+)".

In this example, the group_by values are dest, user and password. 

  <DOMAIN>/local/correlationsearch.conf
    action.summary_index.security_domain  > security_domain
    action.summary_index.rule_description  > rule_description
    action.summary_index.rule_name   > rule_name
    action.summary_index.rule_title   > rule_title
    action.summary_index.severity   > severity
    action.summary_index.drilldown_search   > drilldown_search
    action.summary_index.drilldown_name  > drilldown_name
    description   > description
    <new field>   > default_status
    <new field>   > default_owner

The fields values here are directly mapped from the Splunk Enterprise Security Suite 1.1.x over the Splunk App for Enterprise Security 2.0.x. The default_status and default_owner fields are new for the Splunk App for Enterprise Security 2.0.x and do not have Splunk Enterprise Security Suite 1.1.x equivalent values. As such, keep their default values as empty string/null. 

    <DOMAIN>/local/savedsearches.conf
    action.email.sendresults 
    cron_schedule
    dispatch.earliest_time
    dispatch.latest_time
    enableSched
    search

Most field values are directly mapped over from the Splunk Enterprise Security Suite 1.1.x.

The search string value: In the Splunk Enterprise Security Suite 1.1.x (old):

search = | inputlookup append=T listeningports_tracker | `lower(transport)` | eval timeDiff=now()-firstTime | search timeDiff<86400 | stats dc(dest) as dest_count by transport,dest_port | search dest_count>10 | `get_date(now())` | `get_rule_id(\"Endpoint - Anomalous New Listening Port - Rule\".date.transport.dest_port)` | eval _time=now()

In the Splunk App for Enterprise Security 2.0.x (old):

search = | inputlookup append=T listeningports_tracker | `lower(transport)` | eval timeDiff=now()-firstTime | search timeDiff<86400 | stats dc(dest) as dest_count by transport,dest_port | search dest_count>10 | aggregate search="Endpoint - Anomalous New Listening Port - Rule"

Within the search string, replace everything following the get_date() macro with the new call to aggregate search command that takes the following format: 

aggregate search="<correlation_search_name>"
Last modified on 23 September, 2014
PREVIOUS
Manual upgrade of Enterprise Security 		  NEXT
Dashboards

This documentation applies to the following versions of Splunk® Enterprise Security: 3.0

Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters