Splunk® Enterprise Security

Use Splunk Enterprise Security

Acrobat logo Download manual as PDF


This documentation does not apply to the most recent version of Splunk® Enterprise Security. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

Notable events

Splunk Enterprise Security uses correlation searches to automate the review of events for security relevant incidents, and to detect patterns in your data. When a correlation search detects a suspicious pattern, it creates an alert called a notable event.

A notable event represents one or more anomalous incidents detected by a correlation search across data sources. For example, a notable event can represent:

  • The repeated occurrence of an abnormal spike in network usage over a period of time
  • A single occurrence of unauthorized access to a system
  • A host communicating with a server on a known threat list

The Incident Review dashboard displays all notable events, and categorizes them by potential severity so you can quickly triage, assign, and track issues as part of an incident response workflow. See Incident Review in this manual.

Configure notable events

Navigate to Configure > Incident Management in Enterprise Security to make various configuration changes to notable events.

Create a notable event

You can create a notable event from any indexed event using the Event Action menu or the Configure menu in ES. A notable event created in this way will include tracking fields such as Owner and Status, but will not include the unique fields or links created when a notable event is generated by a correlation search alert action.

By default, only administrators can manually create and edit new notable events. To grant other users, such as analysts, this capability, see "Configure user and roles" in the Installation and Upgrade Manual.

Note: Do not create a new notable event from an event shown on the Incident Review dashboard.

Create a notable event from an existing event

To create a new notable event from an indexed event:

  1. Use a drilldown search to view a log event.
  2. Expand the event and select Event Actions.
  3. Select Create notable event.
  4. On the New Notable Event page, enter a Title and Description for the event and change other fields as required.
  5. Select Save. The Incident Review dashboard will display with the new notable event.

Options for the notable event fields:

Field Description
Title Name for notable event
Domain Access, Audit, Endpoint, Identity, Network, or Threat
Urgency Level of severity: Informational, Low, Medium, High, Critical
Owner unassigned, Administrator, esadmin, esanalyst
Status Unassigned, New, In Progress, Pending, Resolved, Closed
Description Purpose of the notable event

Create a notable event from the Configure menu

  1. Browse to Configure > Incident Management > New Notable Event.
  2. On the Create Notable Event page, enter a Title and Description for the event and change other fields as required.
  3. Select Save. The Incident Review dashboard will display with the new notable event.

Notable Event Statuses

An analyst uses the notable event status to communicate the state of a notable event in the investigation workflow.

On the Splunk Enterprise Security toolbar, open Configure > Incident Management and select Notable Event Statuses.

Label Description Accepts edits
Unassigned An event has not been assigned. No
New (default) An event has not been reviewed. No
In Progress An Investigation or response is in progress. Yes
Pending An event closure is pending some action. Yes
Resolved An event has been resolved and awaits verification. Yes
Closed An event has been resolved and verified. Yes

The status name is used to map the stages of an investigation and is comparable to working with a ticket tracking system. The status name is also used to review and report on the progress of an investigation, as displayed on the Incident Review Audit dashboard.

By default, ES assigns every notable event a status of New when it is created. From the New status, an analyst can choose any other status for the notable event during their investigation. You can modify this pathway by defining status transitions.

Edit a notable event status

Use the Edit Notable Event Status page to change the status name, description, and update the status transition workflow for a notable event.

  1. On the Splunk Enterprise Security toolbar, open Configure > Incident Management and select Notable Event Statuses.
  2. Select a notable event status to open the Edit Notable Event Status page.

Note: The Unassigned and New statuses are defaults used when creating notable events, and cannot be edited.

Status transitions

The status names represent the steps required in investigating a notable event. Status transitions define the path of a notable event investigation.

An analyst assigned a notable event will change the status of the notable event as the investigation progresses. To change the current status on a notable event:

  • The analyst must be a member of a role that has permission to change a status. Notable event status transitions are available to the ess_analyst and ess_admin roles by default.
  • The follow-on status must be configured allow a transition from the current status. Every status can transition to any other status by default. For example, a notable event in a New status can transition directly to any other status including Closed.

Restricting status transitions

A notable event can be limited to a set of pre-defined paths to closure by editing the transitions available to each status. There are no restrictions on the transition paths by default.

The editing of status transitions requires the Edit Notable Event Status capability. The capability is assigned to the ess_analyst and ess_admin roles by default. For more information about user roles and Enterprise Security custom capabilities, see "Configure user and roles" in the Installation and Upgrade Manual.

  1. Define the status workflow for notable events.
    1. Decide which statuses are required, and if a sequence of statuses is needed before the workflow is complete.
    2. Decide if any roles will be allowed to bypass the full workflow.
  2. On the Splunk Enterprise Security toolbar, open Configure > Incident Management and select Notable Event Statuses.
  3. Select a notable event status to open the Edit Notable Event Status page.
  4. In Status Transitions, modify the To Status fields:
    1. To define which roles are allowed to transition a notable event to the selected status, choose the Authorization field and add or remove roles.
    2. To remove a transition an event to the selected status, choose Unselect All.
  5. Save the changes.
  6. Test the changes to the status workflow. If any transitions required adding or removing roles, test with credentials assigned to each role.

Add a new status

You can customize the available statuses and the workflow order.

  1. Define the status workflow for notable events.
    1. Decide where the new status is needed in the workflow.
    2. Decide if any roles (e.g. ess_admin) will be allowed to bypass the new status in the workflow.
  2. On the Splunk Enterprise Security toolbar, open Configure > Incident Management and select Notable Event Statuses.
  3. Select New.
  4. Add a label. This is the Status field value used on the Incident Review dashboard and for notable event status reporting. Example: Waiting on ITOps
  5. Add a description. The description is only referenced in the Notable Event Status page. Example: Waiting on another department.
  6. (Optional) Select Default status. Choose only if you are replacing the New status for notable events
  7. (Optional) Select End status. Choose when adding an additional Closed status for notable events.
  8. Define the Status Transitions by modifying the To Status fields:
    1. Review the status workflow and determine which statuses a notable event can transition to.
    2. Choose the Authorization field and add the roles allowed to transition a notable event to the selected status.
    3. Save the changes.
      Example: In our workflow, the "Waiting on ITOps" status occurs after "New" and "In Progress", but before "Pending." It is not a required status and can be skipped over to choose "Pending." Edit the Status Transitions in "Waiting on ITOps" for "Pending," "Resolved," and "Closed" and add the roles ess_admin and ess_analyst added under Authorization.
  9. Edit the statuses that will precede the new status in the workflow, and add the roles allowed to perform the transition.
    Example: In our workflow, a notable event can be given a status of "Waiting on ITOps" from a status of "New" and "In Progress." Edit the Status Transitions in both "New" and "In Progress" adding the ess_admin and ess_analyst roles under Authorization for "Waiting on ITOps".
  10. Test vigorously.

Create and manage notable event suppressions

Notable events can be suppressed by creating a search filter, or suppression, to hide any additional notable events that match the search conditions. A suppression filter is used to stop excessive or unwanted numbers of notable events from being displayed on the "Incident Review" dashboard.

Example: you may want to prevent certain types of notable events from appearing on the Incident Review dashboard or contributing to defined alert thresholds.

  • A suppression is applied to events that are stored in the notable index. A suppression filter hides notable events so they will not be seen.
  • Throttling is applied to events before they are added to the notable index. "Throttling" prevents notable events from being created.

There are two methods for creating suppression filters.

Create a suppression from Incident Review

ES40 IncidentRev NE Suppress.png

  1. Select a notable event on the Incident Review dashboard.
  2. From Actions menu select Suppress Notable Events. The Suppress Notable Events page opens.
  3. Type a Suppression Name.
  4. Provide a reason for the suppression using the Description field.
  5. Set a date range. After the time limit ends, the suppression filter is automatically disabled.
  6. Validate the Selected Fields are the fields you want to suppress notable events from.
  7. Optional: Use the change link to add or remove notable event fields.
  8. Save changes.

To review the suppression filter, browse to Configure > Incident Management > Notable Event Suppressions.

An event suppression only prevents new events from appearing on the Incident Review dashboard. Creating a notable event suppression will not change notable event counts on the Security Posture and auditing dashboards.

Create a suppression from Notable Event Suppressions

  1. Browse to Configure > Incident Management > Notable Event Suppressions.
  2. Click Create New Suppression to create a new notable event suppression.
  3. Set the Name and Description used for the suppression filter.
  4. Enter the search used to find events to be suppressed in the Search field.
  5. Set the Expiration Time. This defines a time limit for the suppression filter. If the time limit is met, the suppression filter is disabled.

Es-notable event suppression list.png

Edit notable event suppressions

  1. Browse to Configure > Incident Management > Notable Event Suppressions.
  2. Select a notable event suppression to open the Edit Notable Event Suppression page.
  3. Edit the Description and Search fields used for the suppression filter.

Disable notable event suppressions

  1. Browse to Configure > Incident Management > Notable Event Suppressions.
  2. Select Disable in the Status column for the notable event suppression.

Remove a notable event suppression

  1. Browse to Settings > Event types in the Splunk toolbar.
  2. Search for the the suppression event: notable_suppression-<suppression_name>.
  3. Select delete in the Actions column for the notable event suppression.

Audit notable event activity

Enterprise Security tracks all incident review activity for auditing on the Incident Review Audit dashboard.

Audit suppression activity

Enterprise Security tracks all suppression activity for auditing purposes on the Suppression Audit dashboard.

Notable Event Urgency assignment

Notable events are assigned an urgency level that is a combination of the severity of the correlation search and the priority assignment of the relevant asset or identity. You can use the Urgency field to prioritize the investigation of notable events.

ES40 Notable Urgency table.png

Note: When calculating the severity level, a notable event will display a default of "low" urgency when an asset or identity is categorized as "unknown." The "unknown" classification typically represents an object that has no match in the asset and identities system.

You can change the urgency calculation defaults in Enterprise Security.

  1. Browse to Configure > Data Enrichment > Lists/Lookups.
  2. Choose the Urgency Levels configuration. An editable, color coded table representing the urgency lookup file will be displayed.
  3. In any row where the priority or severity is listed as "unknown," review the assigned Urgency.
  4. If you want to, edit the table and change the Urgency.
  5. Save the changes

Notable event index

When a notable event is created, it is indexed on disk and stored in index=notable, much like other events indexed by the Splunk platform available in index=main. Notable events can be tracked, managed, and updated using the Incident Review dashboard, but can be searched directly using the notable index.

Splunk ES assigns a unique ID to each notable event at creation, and that ID is used to create and update the status and user assignment of a notable event.

Notable index fields

The rule_id field is a unique identifier for a notable event and is included for all correlation searches. The get_rule_id macro populates the rule_id field. If an event_id exists for a notable event, the macro will populate rule_id with the event_id for the notable event.

The event_id field identifies the original event that caused a rule to create an alert. Both notable and raw events can be associated with an event_id. A notable event created from a single original event has an event_id of orig_event_id.

You can use the event_id field to associate the original, raw event with the corresponding notable event in the notable index.

Not all correlation searches will include the event_id field. Correlation searches that examine an aggregate set of events, such as a brute force rule that triggers when a certain number of login failures occur, will not include the event_id field.

event_id orig_event_id rule_id
Purpose: Unique identifier for all events (notable or otherwise). Will also become "rule_id" if rule_id doesn't exist. Match a notable event to a particular original event Uniquely identify a notable event
Included when: When `get_event_id` macro is used to generate it. When a notable event is associated with a single original event For all correlation searches
Always included: Yes No Yes

Managing notable event history

Notable events are associated with users, statuses, and comments. Changes made to status names affect only the name of a status, not the status ID assigned to the notable event in the notable index.

If you change the name of a default notable event status, the name will change for both past and future notable events. For example, if you rename pending to waiting for customer, all notable events with a status of pending will then have a status of waiting for customer. The status ID assigned to the notable events will remain the same.

Working with notable events from search

Search existing notable events with the notable macro.

 `notable` 

The following fields are returned:

field description
_time local time when the notable event occurred
host the ES search head that recorded the notable event
source the correlation search that generated the notable event
sourcetype the notable event sourcetype (always stash)
src_user the user that created the correlation search, if applicable
dest the affected destination system, if applicable
tag::action the recorded action, if applicable (same as action)
tag::app the affected app, if applicable
tag::eventtype the type of event, if applicable
src the affected source system, if applicable
TaskCategory the task category, if applicable
vendor the affected vendor, if applicable
product the affected product, if applicable
dest_is_expected a flag indicating if the system is supposed to report to Splunk regularly
action the recorded action, if applicable (same as tag::action)
_raw the raw detail of the notable event

You can use these fields in the Splunk search pipeline to evaluate and report on generated notable events.

You can also use the `incident_review` macro to see incident review events. This macro will only include notable events that have been reviewed.

  | `incident_review`

The results show any incident review activity and returns the following fields:

field description
_time local time of the incident review event
comment the reviewer's comment on the notable event at the time of the incident review event
owner the assigned owner of the notable event at the time of the incident review event. This is the account name. To convert to a full name, use the `notable_owners` macro.
reviewer the user who performed the incident review event
rule_id the unique event identifier
rule_name the correlation search that generated the notable event
status the numeric status code of the notable event at the time of the incident review event
status_default true/false, whether the notable event is in its default status at the time of the incident review event
status_description the long form description of the notable event status at the time of the incident review event
status_end true/false, whether the notable event is in an end status at the time of the incident review event
status_group Open, New, or Closed
status_label the short form description of the notable event status at the time of the incident review event
time GMT time of the incident review event
urgency the urgency of the notable event at the time of the incident review event

You can use these fields in the Splunk search pipeline to evaluate and report on notable event incident review activity.

Last modified on 19 November, 2015
PREVIOUS
Incident Review
  NEXT
Security Posture dashboard

This documentation applies to the following versions of Splunk® Enterprise Security: 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.0.4, 4.0.5, 4.0.6


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters