Notable events

Splunk Enterprise Security detects patterns in your data and automatically reviews events for security-relevant incidents using correlation searches. When a correlation search detects a suspicious pattern, the correlation search creates an alert called a notable event.

A notable event represents one or more anomalous incidents detected by a correlation search across data sources. For example, a notable event can represent:

• The repeated occurrence of an abnormal spike in network usage over a period of time
• A single occurrence of unauthorized access to a system
• A host communicating with a server on a known threat list

Triage and manage notable events

The Incident Review dashboard displays all notable events, and categorizes them by potential severity so you can quickly triage, assign, and track issues as part of an incident response workflow. See Incident Review in this manual.

Manually create a notable event

You can manually create a notable event from an indexed event, or create one from scratch.

Note: By default, only administrators can manually create notable events. To grant other users this capability, see Configure user and roles in the Installation and Upgrade Manual.

Create a notable event from an existing event

You can create a notable event from any indexed event using the Event Actions menu. Do not create a notable event from notable events on the Incident Review dashboard.

1. From an event, view the event details and click Event Actions.
2. Select Create notable event.
3. Enter a Title for the event.
4. (Optional) Select a security Domain.
5. (Optional) Select an Urgency level.
6. (Optional) Select an Owner.
7. (Optional) Select a Status.
8. Enter a Description for the event that describes why you created the notable event or what needs to be investigated.
9. Save the new notable event. The Incident Review dashboard displays with your new notable event.

Note: A notable event created in this way includes tracking fields such as Owner and Status, but does not include the unique fields or links created when a notable event is generated by a correlation search alert action.

Create a notable event from scratch

Create a notable event based on observations, a finding from a security system outside Splunk, or something else.

1. Select Configure > Incident Management > New Notable Event.
2. Enter a Title for the event.
3. (Optional) Select a security Domain.
4. (Optional) Select an Urgency level.
5. (Optional) Select an Owner.
6. (Optional) Select a Status.
7. Enter a Description for the event that describes why you created the notable event or what needs to be investigated.
8. Save the new notable event. The Incident Review dashboard displays with your new notable event.

Change notable event settings

Navigate to Configure > Incident Management in Enterprise Security to make various configuration changes to notable events.

• Review the default Notable Event Statuses and add, remove or change a status as desired.
• Review existing or create new Notable Event Suppressions.
• Customize Incident Review to allow an urgency override or enforce comment usage.

Managing and monitoring notable event statuses

An analyst assigns a status to a notable event to communicate the state of the notable event in the investigation workflow. The status aligns with the stages of an investigation, and can be used to review and report on the progress of a notable event investigation on the Incident Review Audit dashboard.

To see the available statuses for notable events, select Configure > Incident Management > Notable Event Statuses

Label	Description	Can be edited
Unassigned	A notable event has not been assigned.	No
New (default)	A notable event has not been reviewed.	No
In Progress	An investigation or response to the notable event is in progress.	Yes
Pending	A notable event closure is pending some action.	Yes
Resolved	A notable event has been resolved and awaits verification.	Yes
Closed	A notable event has been resolved and verified.	Yes

Every notable event is assigned a status of New by default when it is created by a correlation search. You can customize notable event statuses to match an existing workflow at your organization.

Edit notable event statuses

Change the available statuses for notable events on the Edit Notable Event Status page.

1. On the Splunk Enterprise Security toolbar, select Configure > Incident Management > Notable Event Statuses.
2. Select a notable event status to open the Edit Notable Event Status page.
3. Change the Label or Description as desired.

Note: You cannot edit the Unassigned and New statuses because they are defaults used when creating notable events.

Manage notable event status history

Notable events are associated with users, statuses, and comments. Changes made to status names affect only the name of a status, not the status ID assigned to the notable event in the notable index.

If you change the name of a default notable event status, the name will change for both past and future notable events. For example, if you rename pending to waiting for customer, all notable events with a status of pending will then have a status of waiting for customer. The status ID assigned to the notable events will remain the same.

Notable event status transitions

The status names represent the steps required in investigating a notable event. Status transitions define the path of a notable event investigation.

An analyst assigned a notable event will change the status of the notable event as the investigation progresses. To change the current status on a notable event:

• The analyst must be a member of a role that has permission to change a status. Notable event status transitions are available to the ess_analyst and ess_admin roles by default.
• The follow-on status must allow a transition from the current status. Every status can transition to any other status by default. For example, a notable event in a New status can transition directly to any other status including Closed.

Restricting status transitions

You can define a transition workflow and limit which status can transition to another status, creating a predefined path for the notable event investigation workflow. By default, no transition path is defined or required and every status can transition to every other status.

Prerequisites

• In order to edit status transitions, you must have the ess_analyst role or your role must be assigned the Edit Notable Event Statuses capability. For more information about user roles and capabilities, see Configure user and roles in the Installation and Upgrade Manual.
• Define the status workflow for notable event investigations. Determine which statuses to require, and whether analysts should follow a specific sequence of statuses before completing the investigation workflow. Determine whether any roles can bypass the full workflow.

Restrict notable event status transitions

1. On the Splunk Enterprise Security toolbar, select Configure > Incident Management > Notable Event Statuses.
2. Select a notable event status to open the Edit Notable Event Status page.
3. In Status Transitions, modify the To Status fields.
   1. To define which roles are allowed to transition a notable event to the selected status, choose the Authorization field and add or remove roles.
   2. To remove a transition an event to the selected status, choose Unselect All.
4. Save the changes.
5. Test the changes to the status workflow. If any transitions required adding or removing roles, test with credentials assigned to each role.

Add a new status

Add a new status to the notable event investigation workflow. If you restrict status transitions, determine where this status fits in the workflow.

1. Define the status workflow for notable events.
   1. Determine where the new status is needed in the workflow.
   2. Determine whether any roles (e.g. ess_admin) will be allowed to bypass the new status in the workflow.
2. On the Splunk Enterprise Security toolbar, open Configure > Incident Management and select Notable Event Statuses.
3. Select New.
4. Add a label. This is the Status field value used on the Incident Review dashboard and for notable event status reporting. Example: Waiting on ITOps
5. Add a description. The description is only referenced in the Notable Event Status page. Example: Waiting on another department.
6. (Optional) Select Default status. Choose only if you are replacing the New status for notable events
7. (Optional) Select End status. Choose when adding an additional Closed status for notable events.
8. Define the Status Transitions by modifying the To Status fields.
   1. Review the status workflow and determine which statuses a notable event can transition to.
   2. Choose the Authorization field and add the roles allowed to transition a notable event to the selected status.
9. Save the changes.
   Example: In our workflow, the "Waiting on ITOps" status occurs after "New" and "In Progress", but before "Pending." It is not a required status and can be skipped over to choose "Pending." Edit the Status Transitions in "Waiting on ITOps" for "Pending," "Resolved," and "Closed" and add the roles ess_admin and ess_analyst added under Authorization.
10. Edit the statuses that will precede the new status in the workflow, and add the roles allowed to perform the transition.
    Example: In our workflow, a notable event can be given a status of "Waiting on ITOps" from a status of "New" and "In Progress." Edit the Status Transitions in both "New" and "In Progress" adding the ess_admin and ess_analyst roles under Authorization for "Waiting on ITOps".
11. Test to ensure the status can be assigned and that any status transitions involving it work.

Create and manage notable event suppressions

You can hide notable events from the Incident Review dashboard by creating a notable event suppression.

A suppression is a search filter that hides additional notable events from view, and is used to stop excessive or unwanted numbers of notable events from appearing on the Incident Review dashboard. Notable events that meet the search conditions are still created and added to the notable index. Suppressed notable events continue to contribute to notable event counts on the Security Posture and auditing dashboards.

To prevent notable events that meet certain conditions from being created, see Throttling.

You can create a suppression filter in two ways.

• Create a suppression from Incident Review. See Suppress a notable event.
• Create a suppression from the Configure menu. See Create a suppression from Notable Event Suppressions.

Create a suppression from Notable Event Suppressions

1. Select Configure > Incident Management > Notable Event Suppressions.
2. Click Create New Suppression.
3. Enter a Name and Description for the suppression filter.
4. Enter a Search to use to find notable events to be suppressed.
5. Set the Expiration Time. This defines a time limit for the suppression filter. If the time limit is met, the suppression filter is disabled.

Edit notable event suppressions

1. Select Configure > Incident Management > Notable Event Suppressions.
2. Select a notable event suppression to open the Edit Notable Event Suppression page.
3. Edit the Description and Search fields used for the suppression filter.

Disable notable event suppressions

1. Select Configure > Incident Management > Notable Event Suppressions.
2. Select Disable in the Status column for the notable event suppression.

Remove a notable event suppression

1. From the Splunk platform toolbar, select Settings > Event types.
2. Search for the the suppression event: notable_suppression-<suppression_name>.
3. Select delete in the Actions column for the notable event suppression.

Audit notable event activity

You can audit analyst incident review activity on the Incident Review Audit dashboard. Audit notable event suppressions with the Suppression Audit dashboard.

How urgency is assigned to notable events

Notable events are assigned an urgency level that is a combination of the severity of the correlation search and the priority assignment of the relevant asset or identity. You can use the Urgency field to prioritize the investigation of notable events.

You can change the urgency calculation defaults in Enterprise Security.

1. Browse to Configure > Data Enrichment > Lists/Lookups.
2. Choose the Urgency Levels configuration. An editable, color coded table representing the urgency lookup file will be displayed.
3. In any row where the priority or severity is listed as "unknown," review the assigned Urgency.
4. (Optional) Edit the table and change the Urgency.
5. Save the changes

Note: When calculating the severity level, a notable event displays a default of "low" urgency when an asset or identity is categorized as "unknown." The "unknown" classification typically represents an object that has no match in the asset and identities system.

Note: A notable event can be assigned an unknown urgency level if the severity value assigned by the correlation search or in a triggering event is not recognized by ES. This indicates an error in the severity value provided by the correlation search syntax. Verify that the correlation search severity is unknown, informational, low, medium, high, or critical.

Notable event index

When a notable event is created, it is indexed on disk and stored in index=notable, similar to how events indexed by the Splunk platform available in index=main. You can track, manage, and update notable events on the Incident Review dashboard, but you can search notable events directly using the notable index.

Splunk ES assigns a unique ID, called a rule_id to each notable event when it is created. That ID is used to create and update the status and user assignment of a notable event.

Notable index fields

Three index fields help you identify notable events.

Field	Description	Macro
rule_id	Unique identifier for a notable event. Included for all correlation searches.	`get_rule_id`
event_id	Identifies the original event that caused a correlation search to generate a notable event.	`get_event_id`
orig_event_id	Exists only when a notable event is created based on an event that already had an event_id field.	none.

You can use the macro `get_rule_id` to reference the rule_id for a notable event. For example, you can include it in a ticketing system integration or email.

Use the event_id to associate the original raw event with the corresponding notable event.

Not all correlation searches include an event_id. For example, searches that generate notable events based on an aggregate set of events will not include the event_id. An event_id exists when there is one single raw event creates a notable event.

Working with notable events from search

Search existing notable events with the notable macro: `notable`.

The following fields are returned:

Field	Description
_time	local time when the notable event occurred
host	the ES search head that recorded the notable event
source	the correlation search that generated the notable event
sourcetype	the notable event sourcetype (always stash)
src_user	the user that created the correlation search, if applicable
dest	the affected destination system, if applicable
tag::action	the recorded action, if applicable (same as action)
tag::app	the affected app, if applicable
tag::eventtype	the type of event, if applicable
src	the affected source system, if applicable
TaskCategory	the task category, if applicable
vendor	the affected vendor, if applicable
product	the affected product, if applicable
dest_is_expected	a flag indicating if the system is supposed to report to Splunk regularly
action	the recorded action, if applicable (same as tag::action)
_raw	the raw detail of the notable event

You can use these fields in the Splunk search pipeline to evaluate and report on generated notable events.

You can also use the `incident_review` macro to see incident review events. This macro will only include notable events that have been reviewed.

| `incident_review`

The results show any incident review activity and returns the following fields:

Field	Description
_time	local time of the incident review event
comment	the reviewer's comment on the notable event at the time of the incident review event
owner	the assigned owner of the notable event at the time of the incident review event. This is the account name. To convert to a full name, use the `notable_owners` macro.
reviewer	the user who performed the incident review event
rule_id	the unique event identifier
rule_name	the correlation search that generated the notable event
status	the numeric status code of the notable event at the time of the incident review event
status_default	true/false, whether the notable event is in its default status at the time of the incident review event
status_description	the long form description of the notable event status at the time of the incident review event
status_end	true/false, whether the notable event is in an end status at the time of the incident review event
status_group	Open, New, or Closed
status_label	the short form description of the notable event status at the time of the incident review event
time	GMT time of the incident review event
urgency	the urgency of the notable event at the time of the incident review event

You can use these fields in the Splunk search pipeline to evaluate and report on notable event incident review activity.