Notable events
Splunk Enterprise Security uses correlation searches to automate the review of events for security relevant incidents, and to detect patterns in your data. When a correlation search detects a suspicious pattern, it creates an alert called a notable event.
A notable event represents one or more anomalous incidents detected by a correlation search across data sources. For example, a notable event can represent:
- The repeated occurrence of an abnormal spike in network usage over a period of time
- A single occurrence of unauthorized access to a system
- A host communicating with a server on a known threat list
The Incident Review dashboard displays all notable events, and categorizes them by potential severity so you can quickly triage, assign, and track issues as part of an incident response workflow. See Incident Review in this manual.
Configure notable events
Navigate to Configure > Incident Management in Enterprise Security to make various configuration changes to notable events.
- Choose to create a "New Notable Event".
- Review the default "Notable Event Statuses" and add, remove or change a status as desired.
- Review existing or create new "Notable Event Suppressions".
- Review the "Incident Review settings" to allow an urgency override, or enforce comment usage.
Create a notable event
You can create a notable event from any indexed event using the Event Action menu or the Configure menu in ES. A notable event created in this way will include tracking fields such as Owner and Status, but will not include the unique fields or links created when a notable event is generated by a correlation search alert action.
By default, only administrators can manually create and edit new notable events. To grant other users, such as analysts, this capability, see "Configure user and roles" in the Installation and Upgrade Manual.
Note: Do not create a new notable event from an event shown on the Incident Review dashboard.
Create a notable event from an existing event
To create a new notable event from an indexed event:
- Use a drilldown search to view a log event.
- Expand the event and select Event Actions.
- Select Create notable event.
- On the New Notable Event page, enter a Title and Description for the event and change other fields as required.
- Select Save. The Incident Review dashboard will display with the new notable event.
Options for the notable event fields:
Field | Description |
---|---|
Title | Name for notable event |
Domain | Access, Audit, Endpoint, Identity, Network, or Threat |
Urgency | Level of severity: Informational, Low, Medium, High, Critical |
Owner | unassigned, Administrator, esadmin, esanalyst |
Status | Unassigned, New, In Progress, Pending, Resolved, Closed |
Description | Purpose of the notable event |
- Browse to Configure > Incident Management > New Notable Event.
- On the Create Notable Event page, enter a Title and Description for the event and change other fields as required.
- Select Save. The Incident Review dashboard will display with the new notable event.
Notable Event Statuses
An analyst uses the notable event status to communicate the state of a notable event in the investigation workflow.
On the Splunk Enterprise Security toolbar, open Configure > Incident Management and select Notable Event Statuses.
Label | Description | Accepts edits |
---|---|---|
Unassigned | An event has not been assigned. | No |
New (default) | An event has not been reviewed. | No |
In Progress | An Investigation or response is in progress. | Yes |
Pending | An event closure is pending some action. | Yes |
Resolved | An event has been resolved and awaits verification. | Yes |
Closed | An event has been resolved and verified. | Yes |
The status name is used to map the stages of an investigation and is comparable to working with a ticket tracking system. The status name is also used to review and report on the progress of an investigation, as displayed on the Incident Review Audit dashboard.
By default, ES assigns every notable event a status of New when it is created. From the New status, an analyst can choose any other status for the notable event during their investigation. You can modify this pathway by defining status transitions.
Edit a notable event status
Use the Edit Notable Event Status page to change the status name, description, and update the status transition workflow for a notable event.
- On the Splunk Enterprise Security toolbar, open Configure > Incident Management and select Notable Event Statuses.
- Select a notable event status to open the Edit Notable Event Status page.
Note: The Unassigned and New statuses are defaults used when creating notable events, and cannot be edited.
Status transitions
The status names represent the steps required in investigating a notable event. Status transitions define the path of a notable event investigation.
An analyst assigned a notable event will change the status of the notable event as the investigation progresses. To change the current status on a notable event:
- The analyst must be a member of a role that has permission to change a status. Notable event status transitions are available to the ess_analyst and ess_admin roles by default.
- The follow-on status must be configured allow a transition from the current status. Every status can transition to any other status by default. For example, a notable event in a New status can transition directly to any other status including Closed.
Restricting status transitions
A notable event can be limited to a set of pre-defined paths to closure by editing the transitions available to each status. There are no restrictions on the transition paths by default.
The editing of status transitions requires the Edit Notable Event Status capability. The capability is assigned to the ess_analyst and ess_admin roles by default. For more information about user roles and Enterprise Security custom capabilities, see "Configure user and roles" in the Installation and Upgrade Manual.
- Define the status workflow for notable events.
- Decide which statuses are required, and if a sequence of statuses is needed before the workflow is complete.
- Decide if any roles will be allowed to bypass the full workflow.
- On the Splunk Enterprise Security toolbar, open Configure > Incident Management and select Notable Event Statuses.
- Select a notable event status to open the Edit Notable Event Status page.
- In Status Transitions, modify the To Status fields:
- To define which roles are allowed to transition a notable event to the selected status, choose the Authorization field and add or remove roles.
- To remove a transition an event to the selected status, choose Unselect All.
- Save the changes.
- Test the changes to the status workflow. If any transitions required adding or removing roles, test with credentials assigned to each role.
Add a new status
You can customize the available statuses and the workflow order.
- Define the status workflow for notable events.
- Decide where the new status is needed in the workflow.
- Decide if any roles (e.g. ess_admin) will be allowed to bypass the new status in the workflow.
- On the Splunk Enterprise Security toolbar, open Configure > Incident Management and select Notable Event Statuses.
- Select New.
- Add a label. This is the Status field value used on the Incident Review dashboard and for notable event status reporting. Example: Waiting on ITOps
- Add a description. The description is only referenced in the Notable Event Status page. Example: Waiting on another department.
- (Optional) Select Default status. Choose only if you are replacing the New status for notable events
- (Optional) Select End status. Choose when adding an additional Closed status for notable events.
- Define the Status Transitions by modifying the To Status fields:
- Review the status workflow and determine which statuses a notable event can transition to.
- Choose the Authorization field and add the roles allowed to transition a notable event to the selected status.
- Save the changes.
Example: In our workflow, the "Waiting on ITOps" status occurs after "New" and "In Progress", but before "Pending." It is not a required status and can be skipped over to choose "Pending." Edit the Status Transitions in "Waiting on ITOps" for "Pending," "Resolved," and "Closed" and add the roles ess_admin and ess_analyst added under Authorization.
- Edit the statuses that will precede the new status in the workflow, and add the roles allowed to perform the transition.
Example: In our workflow, a notable event can be given a status of "Waiting on ITOps" from a status of "New" and "In Progress." Edit the Status Transitions in both "New" and "In Progress" adding the ess_admin and ess_analyst roles under Authorization for "Waiting on ITOps". - Test vigorously.
Create and manage notable event suppressions
Notable events can be suppressed by creating a search filter, or suppression, to hide any additional notable events that match the search conditions. A suppression filter is used to stop excessive or unwanted numbers of notable events from being displayed on the "Incident Review" dashboard.
Example: you may want to prevent certain types of notable events from appearing on the Incident Review dashboard or contributing to defined alert thresholds.
- A suppression is applied to events that are stored in the notable index. A suppression filter hides notable events so they will not be seen.
- Throttling is applied to events before they are added to the notable index. "Throttling" prevents notable events from being created.
There are two methods for creating suppression filters.
Create a suppression from Incident Review
- Select a notable event on the Incident Review dashboard.
- From Actions menu select Suppress Notable Events. The Suppress Notable Events page opens.
- Type a Suppression Name.
- Provide a reason for the suppression using the Description field.
- Set a date range. After the time limit ends, the suppression filter is automatically disabled.
- Validate the Selected Fields are the fields you want to suppress notable events from.
- Optional: Use the change link to add or remove notable event fields.
- Save changes.
To review the suppression filter, browse to Configure > Incident Management > Notable Event Suppressions.
An event suppression only prevents new events from appearing on the Incident Review dashboard. Creating a notable event suppression will not change notable event counts on the Security Posture and auditing dashboards.
Create a suppression from Notable Event Suppressions
- Browse to Configure > Incident Management > Notable Event Suppressions.
- Click Create New Suppression to create a new notable event suppression.
- Set the Name and Description used for the suppression filter.
- Enter the search used to find events to be suppressed in the Search field.
- Set the Expiration Time. This defines a time limit for the suppression filter. If the time limit is met, the suppression filter is disabled.
Edit notable event suppressions
- Browse to Configure > Incident Management > Notable Event Suppressions.
- Select a notable event suppression to open the Edit Notable Event Suppression page.
- Edit the Description and Search fields used for the suppression filter.
Disable notable event suppressions
- Browse to Configure > Incident Management > Notable Event Suppressions.
- Select Disable in the Status column for the notable event suppression.
Remove a notable event suppression
- Browse to Settings > Event types in the Splunk toolbar.
- Search for the the suppression event:
notable_suppression-<suppression_name>
. - Select delete in the Actions column for the notable event suppression.
Audit notable event activity
Enterprise Security tracks all incident review activity for auditing on the Incident Review Audit dashboard.
Audit suppression activity
Enterprise Security tracks all suppression activity for auditing purposes on the Suppression Audit dashboard.
Notable Event Urgency assignment
Notable events are assigned an urgency level that is a combination of the severity of the correlation search and the priority assignment of the relevant asset or identity. You can use the Urgency field to prioritize the investigation of notable events.
Note: When calculating the severity level, a notable event will display a default of "low" urgency when an asset or identity is categorized as "unknown." The "unknown" classification typically represents an object that has no match in the asset and identities system.
You can change the urgency calculation defaults in Enterprise Security.
- Browse to Configure > Data Enrichment > Lists/Lookups.
- Choose the Urgency Levels configuration. An editable, color coded table representing the urgency lookup file will be displayed.
- In any row where the priority or severity is listed as "unknown," review the assigned Urgency.
- If you want to, edit the table and change the Urgency.
- Save the changes
Notable event index
When a notable event is created, it is indexed on disk and stored in index=notable
, much like other events indexed by the Splunk platform available in index=main
. Notable events can be tracked, managed, and updated using the Incident Review dashboard, but can be searched directly using the notable index.
Splunk ES assigns a unique ID to each notable event at creation, and that ID is used to create and update the status and user assignment of a notable event.
Notable index fields
The rule_id field is a unique identifier for a notable event and is included for all correlation searches. The get_rule_id macro populates the rule_id field. If an event_id exists for a notable event, the macro will populate rule_id with the event_id for the notable event.
The event_id
field identifies the original event that caused a rule to create an alert. Both notable and raw events can be associated with an event_id. A notable event created from a single original event has an event_id of orig_event_id.
You can use the event_id field to associate the original, raw event with the corresponding notable event in the notable index.
Not all correlation searches will include the event_id field. Correlation searches that examine an aggregate set of events, such as a brute force rule that triggers when a certain number of login failures occur, will not include the event_id field.
event_id | orig_event_id | rule_id | |
---|---|---|---|
Purpose: | Unique identifier for all events (notable or otherwise). Will also become "rule_id" if rule_id doesn't exist. | Match a notable event to a particular original event | Uniquely identify a notable event |
Included when: | When `get_event_id` macro is used to generate it. | When a notable event is associated with a single original event | For all correlation searches |
Always included: | Yes | No | Yes |
Managing notable event history
Notable events are associated with users, statuses, and comments. Changes made to status names affect only the name of a status, not the status ID assigned to the notable event in the notable index.
If you change the name of a default notable event status, the name will change for both past and future notable events. For example, if you rename pending to waiting for customer, all notable events with a status of pending will then have a status of waiting for customer. The status ID assigned to the notable events will remain the same.
Working with notable events from search
Search existing notable events with the notable macro.
`notable`
The following fields are returned:
field | description |
---|---|
_time | local time when the notable event occurred |
host | the ES search head that recorded the notable event |
source | the correlation search that generated the notable event |
sourcetype | the notable event sourcetype (always stash) |
src_user | the user that created the correlation search, if applicable |
dest | the affected destination system, if applicable |
tag::action | the recorded action, if applicable (same as action) |
tag::app | the affected app, if applicable |
tag::eventtype | the type of event, if applicable |
src | the affected source system, if applicable |
TaskCategory | the task category, if applicable |
vendor | the affected vendor, if applicable |
product | the affected product, if applicable |
dest_is_expected | a flag indicating if the system is supposed to report to Splunk regularly |
action | the recorded action, if applicable (same as tag::action) |
_raw | the raw detail of the notable event |
You can use these fields in the Splunk search pipeline to evaluate and report on generated notable events.
You can also use the `incident_review`
macro to see incident review events. This macro will only include notable events that have been reviewed.
| `incident_review`
The results show any incident review activity and returns the following fields:
field | description |
---|---|
_time | local time of the incident review event |
comment | the reviewer's comment on the notable event at the time of the incident review event |
owner | the assigned owner of the notable event at the time of the incident review event. This is the account name. To convert to a full name, use the `notable_owners` macro. |
reviewer | the user who performed the incident review event |
rule_id | the unique event identifier |
rule_name | the correlation search that generated the notable event |
status | the numeric status code of the notable event at the time of the incident review event |
status_default | true/false, whether the notable event is in its default status at the time of the incident review event |
status_description | the long form description of the notable event status at the time of the incident review event |
status_end | true/false, whether the notable event is in an end status at the time of the incident review event |
status_group | Open, New, or Closed |
status_label | the short form description of the notable event status at the time of the incident review event |
time | GMT time of the incident review event |
urgency | the urgency of the notable event at the time of the incident review event |
You can use these fields in the Splunk search pipeline to evaluate and report on notable event incident review activity.
Incident Review | Security Posture dashboard |
This documentation applies to the following versions of Splunk® Enterprise Security: 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.0.4, 4.0.5, 4.0.6
Feedback submitted, thanks!