Notable events

Splunk Enterprise Security uses correlation searches to automate the review of events for security relevant incidents, and to detect patterns in your data. When a correlation search detects a suspicious pattern, it creates an alert called a notable event.

A notable event represents one or more anomalous incidents detected by a correlation search across data sources. For example, a notable event can represent:

• The repeated occurrence of an abnormal spike in network usage over a period of time
• A single occurrence of unauthorized access to a system
• A host communicating with a server on a known threat list

The Incident Review dashboard displays all notable events, and categorizes them by potential severity so you can quickly triage, assign, and track issues as part of an incident response workflow. See Incident Review in this manual.

Configure notable events

Navigate to Configure > Incident Management in Enterprise Security to make various configuration changes to notable events.

• Choose to create a "New Notable Event".
• Review the default "Notable Event Statuses" and add, remove or change a status as desired.
• Review existing or create new "Notable Event Suppressions".
• Review the "Incident Review settings" to allow an urgency override, or enforce comment usage.

Create a notable event

You can create a notable event from any indexed event using the Event Action menu or the Configure menu in ES. A notable event created in this way will include tracking fields such as Owner and Status, but will not include the unique fields or links created when a notable event is generated by a correlation search alert action.

By default, only administrators can manually create and edit new notable events. To grant other users, such as analysts, this capability, see "Configure user and roles" in the Installation and Upgrade Manual.

Note: Do not create a new notable event from an event shown on the Incident Review dashboard.

Create a notable event from an existing event

To create a new notable event from an indexed event:

1. Use a drilldown search to view a log event.
2. Expand the event and select Event Actions.
3. Select Create notable event.
4. On the New Notable Event page, enter a Title and Description for the event and change other fields as required.
5. Select Save. The Incident Review dashboard will display with the new notable event.

Options for the notable event fields:

Field	Description
Title	Name for notable event
Domain	Access, Audit, Endpoint, Identity, Network, or Threat
Urgency	Level of severity: Informational, Low, Medium, High, Critical
Owner	unassigned, Administrator, esadmin, esanalyst
Status	Unassigned, New, In Progress, Pending, Resolved, Closed
Description	Purpose of the notable event

Create a notable event from the Configure menu

1. Browse to Configure > Incident Management > New Notable Event.
2. On the Create Notable Event page, enter a Title and Description for the event and change other fields as required.
3. Select Save. The Incident Review dashboard will display with the new notable event.

Notable Event Statuses

An analyst uses the notable event status to communicate the state of a notable event in the investigation workflow.

On the Splunk Enterprise Security toolbar, open Configure > Incident Management and select Notable Event Statuses.

Label	Description	Accepts edits
Unassigned	An event has not been assigned.	No
New (default)	An event has not been reviewed.	No
In Progress	An Investigation or response is in progress.	Yes
Pending	An event closure is pending some action.	Yes
Resolved	An event has been resolved and awaits verification.	Yes
Closed	An event has been resolved and verified.	Yes

The status name is used to map the stages of an investigation and is comparable to working with a ticket tracking system. The status name is also used to review and report on the progress of an investigation, as displayed on the Incident Review Audit dashboard.

By default, ES assigns every notable event a status of New when it is created. From the New status, an analyst can choose any other status for the notable event during their investigation. You can modify this pathway by defining status transitions.

Edit a notable event status

Use the Edit Notable Event Status page to change the status name, description, and update the status transition workflow for a notable event.

1. On the Splunk Enterprise Security toolbar, open Configure > Incident Management and select Notable Event Statuses.
2. Select a notable event status to open the Edit Notable Event Status page.

Note: The Unassigned and New statuses are defaults used when creating notable events, and cannot be edited.

Status transitions

The status names represent the steps required in investigating a notable event. Status transitions define the path of a notable event investigation.

An analyst assigned a notable event will change the status of the notable event as the investigation progresses. To change the current status on a notable event:

• The analyst must be a member of a role that has permission to change a status. Notable event status transitions are available to the ess_analyst and ess_admin roles by default.
• The follow-on status must be configured allow a transition from the current status. Every status can transition to any other status by default. For example, a notable event in a New status can transition directly to any other status including Closed.

Restricting status transitions

A notable event can be limited to a set of pre-defined paths to closure by editing the transitions available to each status. There are no restrictions on the transition paths by default.

The editing of status transitions requires the Edit Notable Event Status capability. The capability is assigned to the ess_analyst and ess_admin roles by default. For more information about user roles and Enterprise Security custom capabilities, see "Configure user and roles" in the Installation and Upgrade Manual.

1. Define the status workflow for notable events.
   1. Decide which statuses are required, and if a sequence of statuses is needed before the workflow is complete.
   2. Decide if any roles will be allowed to bypass the full workflow.
2. On the Splunk Enterprise Security toolbar, open Configure > Incident Management and select Notable Event Statuses.
3. Select a notable event status to open the Edit Notable Event Status page.
4. In Status Transitions, modify the To Status fields:
   1. To define which roles are allowed to transition a notable event to the selected status, choose the Authorization field and add or remove roles.
   2. To remove a transition an event to the selected status, choose Unselect All.
5. Save the changes.
6. Test the changes to the status workflow. If any transitions required adding or removing roles, test with credentials assigned to each role.

Add a new status

You can customize the available statuses and the workflow order.

1. Define the status workflow for notable events.
   1. Decide where the new status is needed in the workflow.
   2. Decide if any roles (e.g. ess_admin) will be allowed to bypass the new status in the workflow.
2. On the Splunk Enterprise Security toolbar, open Configure > Incident Management and select Notable Event Statuses.
3. Select New.
4. Add a label. This is the Status field value used on the Incident Review dashboard and for notable event status reporting. Example: Waiting on ITOps
5. Add a description. The description is only referenced in the Notable Event Status page. Example: Waiting on another department.
6. (Optional) Select Default status. Choose only if you are replacing the New status for notable events
7. (Optional) Select End status. Choose when adding an additional Closed status for notable events.
8. Define the Status Transitions by modifying the To Status fields:
   1. Review the status workflow and determine which statuses a notable event can transition to.
   2. Choose the Authorization field and add the roles allowed to transition a notable event to the selected status.
   3. Save the changes.
      Example: In our workflow, the "Waiting on ITOps" status occurs after "New" and "In Progress", but before "Pending." It is not a required status and can be skipped over to choose "Pending." Edit the Status Transitions in "Waiting on ITOps" for "Pending," "Resolved," and "Closed" and add the roles ess_admin and ess_analyst added under Authorization.
9. Edit the statuses that will precede the new status in the workflow, and add the roles allowed to perform the transition.
   Example: In our workflow, a notable event can be given a status of "Waiting on ITOps" from a status of "New" and "In Progress." Edit the Status Transitions in both "New" and "In Progress" adding the ess_admin and ess_analyst roles under Authorization for "Waiting on ITOps".
10. Test vigorously.

Create and manage notable event suppressions

Notable events can be suppressed by creating a search filter, or suppression, to hide any additional notable events that match the search conditions. A suppression filter is used to stop excessive or unwanted numbers of notable events from being displayed on the "Incident Review" dashboard.

Example: you may want to prevent certain types of notable events from appearing on the Incident Review dashboard or contributing to defined alert thresholds.

• A suppression is applied to events that are stored in the notable index. A suppression filter hides notable events so they will not be seen.

• Throttling is applied to events before they are added to the notable index. "Throttling" prevents notable events from being created.

There are two methods for creating suppression filters.

Create a suppression from Incident Review

1. Select a notable event on the Incident Review dashboard.
2. From Actions menu select Suppress Notable Events. The Suppress Notable Events page opens.
3. Type a Suppression Name.
4. Provide a reason for the suppression using the Description field.
5. Set a date range. After the time limit ends, the suppression filter is automatically disabled.
6. Validate the Selected Fields are the fields you want to suppress notable events from.
7. Optional: Use the change link to add or remove notable event fields.
8. Save changes.

To review the suppression filter, browse to Configure > Incident Management > Notable Event Suppressions.

An event suppression only prevents new events from appearing on the Incident Review dashboard. Creating a notable event suppression will not change notable event counts on the Security Posture and auditing dashboards.

Create a suppression from Notable Event Suppressions

1. Browse to Configure > Incident Management > Notable Event Suppressions.
2. Click Create New Suppression to create a new notable event suppression.
3. Set the Name and Description used for the suppression filter.
4. Enter the search used to find events to be suppressed in the Search field.
5. Set the Expiration Time. This defines a time limit for the suppression filter. If the time limit is met, the suppression filter is disabled.

Edit notable event suppressions

1. Browse to Configure > Incident Management > Notable Event Suppressions.
2. Select a notable event suppression to open the Edit Notable Event Suppression page.
3. Edit the Description and Search fields used for the suppression filter.

Disable notable event suppressions

1. Browse to Configure > Incident Management > Notable Event Suppressions.
2. Select Disable in the Status column for the notable event suppression.

Remove a notable event suppression

1. Browse to Settings > Event types in the Splunk toolbar.
2. Search for the the suppression event: notable_suppression-<suppression_name>.
3. Select delete in the Actions column for the notable event suppression.

Audit notable event activity

Enterprise Security tracks all incident review activity for auditing on the Incident Review Audit dashboard.

Audit suppression activity

Enterprise Security tracks all suppression activity for auditing purposes on the Suppression Audit dashboard.

Notable Event Urgency assignment

Notable events are assigned an urgency level that is a combination of the severity of the correlation search and the priority assignment of the relevant asset or identity. You can use the Urgency field to prioritize the investigation of notable events.

Note: When calculating the severity level, a notable event will display a default of "low" urgency when an asset or identity is categorized as "unknown." The "unknown" classification typically represents an object that has no match in the asset and identities system.

You can change the urgency calculation defaults in Enterprise Security.

1. Browse to Configure > Data Enrichment > Lists/Lookups.
2. Choose the Urgency Levels configuration. An editable, color coded table representing the urgency lookup file will be displayed.
3. In any row where the priority or severity is listed as "unknown," review the assigned Urgency.
4. If you want to, edit the table and change the Urgency.
5. Save the changes

Notable event index

When a notable event is created, it is indexed on disk and stored in index=notable, much like other events indexed by the Splunk platform available in index=main. Notable events can be tracked, managed, and updated using the Incident Review dashboard, but can be searched directly using the notable index.

Splunk ES assigns a unique ID to each notable event at creation, and that ID is used to create and update the status and user assignment of a notable event.

Notable index fields

The rule_id field is a unique identifier for a notable event and is included for all correlation searches. The get_rule_id macro populates the rule_id field. If an event_id exists for a notable event, the macro will populate rule_id with the event_id for the notable event.

The event_id field identifies the original event that caused a rule to create an alert. Both notable and raw events can be associated with an event_id. A notable event created from a single original event has an event_id of orig_event_id.

You can use the event_id field to associate the original, raw event with the corresponding notable event in the notable index.

Not all correlation searches will include the event_id field. Correlation searches that examine an aggregate set of events, such as a brute force rule that triggers when a certain number of login failures occur, will not include the event_id field.

	event_id	orig_event_id	rule_id
Purpose:	Unique identifier for all events (notable or otherwise). Will also become "rule_id" if rule_id doesn't exist.	Match a notable event to a particular original event	Uniquely identify a notable event
Included when:	When `get_event_id` macro is used to generate it.	When a notable event is associated with a single original event	For all correlation searches
Always included:	Yes	No	Yes

Managing notable event history

Notable events are associated with users, statuses, and comments. Changes made to status names affect only the name of a status, not the status ID assigned to the notable event in the notable index.

If you change the name of a default notable event status, the name will change for both past and future notable events. For example, if you rename pending to waiting for customer, all notable events with a status of pending will then have a status of waiting for customer. The status ID assigned to the notable events will remain the same.

Working with notable events from search

Search existing notable events with the notable macro.

 `notable` 

The following fields are returned:

field	description
_time	local time when the notable event occurred
host	the ES search head that recorded the notable event
source	the correlation search that generated the notable event
sourcetype	the notable event sourcetype (always stash)
src_user	the user that created the correlation search, if applicable
dest	the affected destination system, if applicable
tag::action	the recorded action, if applicable (same as action)
tag::app	the affected app, if applicable
tag::eventtype	the type of event, if applicable
src	the affected source system, if applicable
TaskCategory	the task category, if applicable
vendor	the affected vendor, if applicable
product	the affected product, if applicable
dest_is_expected	a flag indicating if the system is supposed to report to Splunk regularly
action	the recorded action, if applicable (same as tag::action)
_raw	the raw detail of the notable event

You can use these fields in the Splunk search pipeline to evaluate and report on generated notable events.

You can also use the `incident_review` macro to see incident review events. This macro will only include notable events that have been reviewed.

  | `incident_review`

The results show any incident review activity and returns the following fields:

field	description
_time	local time of the incident review event
comment	the reviewer's comment on the notable event at the time of the incident review event
owner	the assigned owner of the notable event at the time of the incident review event. This is the account name. To convert to a full name, use the `notable_owners` macro.
reviewer	the user who performed the incident review event
rule_id	the unique event identifier
rule_name	the correlation search that generated the notable event
status	the numeric status code of the notable event at the time of the incident review event
status_default	true/false, whether the notable event is in its default status at the time of the incident review event
status_description	the long form description of the notable event status at the time of the incident review event
status_end	true/false, whether the notable event is in an end status at the time of the incident review event
status_group	Open, New, or Closed
status_label	the short form description of the notable event status at the time of the incident review event
time	GMT time of the incident review event
urgency	the urgency of the notable event at the time of the incident review event

You can use these fields in the Splunk search pipeline to evaluate and report on notable event incident review activity.