Splunk® IT Service Intelligence

REST API Reference

Splunk IT Service Intelligence (ITSI) version 4.12.x reached its End of Life on January 22, 2024. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see Before you upgrade IT Service Intelligence.
This documentation does not apply to the most recent version of Splunk® IT Service Intelligence. For documentation on the most recent version, go to the latest release.

ITSI REST API schema

The IT Service Intelligence (ITSI) REST API schema describes the JSON-based data structures of ITSI objects. Use this schema with the ITSI REST API to create API requests and interpret API responses. See ITSI REST API reference.

General details

ITSI backend store

ITSI stores its configuration in the KV store. KV store collections for ITSI are located here: 

https://<splunk_server>:8089/servicesNS/nobody/SA-ITOA/storage/collections/

Do not make any updates through the ITSI KV store collections endpoint above. Perform all operations using the REST endpoints documented in the ITSI REST API reference.

For more information about the KV store, see App Key Value Store on the Splunk developer portal.

ITSI object types

The ITSI REST API supports these object types:

  • entity
  • entity_type
  • service
  • base_service_template (service template)
  • deep_dive
  • glass_table
  • home_view (service analyzer)
  • kpi_template
  • kpi_threshold_template
  • kpi_base_search
  • event_management_state
  • notable_event
  • notable_event_group
  • notable_event_comment
  • notable_event_aggregation_policy
  • notable_event_email_template
  • correlation_search
  • maintenance_calendar
  • team

Note: The /SA-ITOA/<interface_category>/get_supported_object_types GET operation returns a list of currently supported object types. Note that although the entity_relationship and entity_relationship_rule objects are returned, these are not used at this time. For more information, see the ITSI REST API reference.

Common Attributes

The following attributes are common to all ITSI objects. These attributes are available in the model for each object type.

Field Type Description
object_type String Name of the object type.
create_by String The user who created this object.
create_source String The sourcetype initiating create. Has value manual for user-initiated creates. For internal use only.
create_time String Timestamp at the time of creation based on UTC time zone.
mod_source String Sourcetype initiating modification. Has value manual for user-initiated modifications. For internal use only.
mod_time String Timestamp of the last modification based on UTC time zone.
_owner String Splunk user nobody.
_user String User who performed the most recent operation on this object.
version String The version of the object. Currently the same as the ITSI app version.

Common attributes are omitted from the object schemas in this topic to make the documentation easier to read.

Entity

Description

An entity is a basic unit of configuration in an IT environment that meets a specific need for an IT service. Entities are usually servers, but can be other IT infrastructure components, such as network devices, storage subsystems, applications, and so on. Entities are optional.

The entity object contains field aliases and values that identify the entity in KPI searches.

Attributes

Field Type Description
_key String Auto-generated unique identifier for this entity. Can be any unique value.
title String Name of the entity. Can be any unique value.
description String User defined description of the entity.
object_type String entity
identifier Object values: Array of alias values that identify the entity
fields: Array of search fields that identify events for the entity.
informational Object values: Array of alias values that provide information/description for the entity.
fields: Array of search fields to extract information/description of the entity.
services Array Array of sub-objects with _key and title fields of services monitoring this entity via rules configured in services.
sec_grp String The team the object belongs to. The entity object can only belong to default_itsi_security_group (Global team).
sai_entity_key String This field exists in ITSI entities that have been merged with SAI entities. It symbolizes the original SAI entities's _key and is used for drilldowns to SAI.
entity_type_ids Array Array of _key values for each entity type associated with the entity.

For more information, see Overview of entity integrations in ITSI in the Entity Integrations manual.

Service

Description

An ITSI service is a representation of a real world IT service. You can configure an ITSI service to monitor various IT metrics using KPI searches, which reflect the health of a service. ITSI services can describe any real world IT service, such as a network service or email service.

The service object contains the service definition, including entities, KPIs, and dependent services.

Attributes

Field Type Description
_key String Auto-generated unique identifier for this service.
description String User defined description for the service.
title String Title of this service.
kpis Array Array of KPI descriptions for this service.
entity_rules Array Array of rules describing entities referenced by this service.
services_depends_on Array Array of service descriptions with KPIs in those services that this service depends on.
service_id String _key value of service that this service depends on.
kpis_depending_on Array Array of _key ids for each KPI in service identified by serviceid, which this service will depend on.
services_depending_on_me Array An array of service descriptions with KPIs in this service that those services depend on.
serviceid String _key value of service that depends on this service.
kpis_depending_on Array Array of of _key ids of each KPI in this service, which the service identified by serviceid will depend on.
Enabled Boolean If set to 1, service is enabled. If value is absent or not set to 1, service is disabled. On upgrade service is flagged as enabled.
sec_grp String The team the object belongs to.
base_service_template_id String The ID of the service template the service is linked to. Not required. If empty, the service is not linked to a service template. To create a service based on a service template, include this field.
service_tags Object The tags for the service. The service_tags object can have an array for tags and template_tags. tags are regular tags that are added manually and template_tags are tags that are populated from a service template.


Tags have to be strings and can't contain the following characters: / \ " ' ! @ ? . , ; $ ^

Example service_tags object: 

"service_tags": {
        "tags": [
            "unix",
            "seattle"
        ],
        "template_tags": [
            "cloud_systems",
            "us-west"
        ]
    },

For more information, see Overview of creating services in ITSI in the Service Insights manual.

Subordinate objects

Service Template

Description

ITSI service templates help you manage shared content for similar services. Services linked to a service template receive content from the service template, such as KPIs and entity rules. You must create a service template from an existing service.

The base_service_template object contains KPI definitions, entity rules, and any linked services.

Attributes

Field Type Description
_key String Auto-generated unique identifier for this service template.
description String User defined description for the service template.
title String Title of this service template.
kpis Array Array of KPI descriptions for this service template.
entity_rules Array Array of rules describing entities referenced by this service template.
service_id String _key value of the service this service template is generated from.
sec_grp String The team the service template belongs to. Service templates can only belong to default_itsi_security_group (Global team).
linked_services Array Array of services linked to this service template. if the user does not have access to all linked services, the linked_services field only contains the services they have read access to.
total_linked_services Integer The number of services linked to this service template.
last_sync_error String Error message if the last sync operation failed.
sync_status String Sync status of service template: "synced", "sync_scheduled", "syncing", "sync failed".
scheduled_time String The time to push service template changes to linked services if "sync later" is selected rather than "sync now".
scheduled_job Dict Sync job detail if "sync later" is selected rather than "sync now".
template_tags Array The service tags contained within the template.

Subordinate objects

Entity Rules

Description

entity_rules determine the specific entities that a KPI monitors in a service. This includes entities directly identified by title, and entities identified by regular expression-based rules.

Attributes

Entity rules are an array of rule groups separated by OR at the top level.

Field Type Description
rule_condition Boolean operator Uses the value AND indicating this rule appends all nested rules contained in the rule_items attribute.
rule_items Array Array of rules that are appended within a rule group.
field String The field in the entity definition to compare values to evaluate this rule.
rule_type String Takes values not or matches to indicate whether it's an inclusion or exclusion rule. Value can be matchesblank or doesnotmatchblank when used with service templates.
value String Values to evaluate in the rule. To specify multiple values, separate them with a comma. Values are not case sensitive.
field_type String Takes values alias or info specifying in which category of fields the field attribute is located.

Entity rules evaluation examples

The following examples show how ITSI evaluates entity_rules.

Match all entities that are a given title value like Foo 

"entity_rules": 
[
    {
       "rule_condition": "AND",
       "rule_items": 
        [
            {
                "field": "title",
                "rule_type": "matches",
                "value": "Foo",
                "field_type": "title"    
            }
        ]
    }
]

Match all entities that are a given alias field named category value pattern like *Foo* 

"entity_rules": 
[
    {
       "rule_condition": "AND",
       "rule_items": 
        [
            {
                "field": "category",
                "rule_type": "matches",
                "value": "*Foo*",
                "field_type": "alias"    
            }
        ]
    }
]

Exclude all entities that are a given info field named subcategory value like Foo 

"entity_rules":
[
    {
       "rule_condition": "AND",
       "rule_items":
        [
            {
                "field": "subcategory",
                "rule_type": "not",
                "value": "Foo",
                "field_type": "info"   
            }
        ]
    }
 
]

Exclude all entities that are a given title value pattern like *Foo* 

"entity_rules":
[
    {
       "rule_condition": "AND",
       "rule_items":
        [
            {
                "field": "title",
                "rule_type": "not",
                "value": "*Foo*",
                "field_type": "title"   
            }
        ]
    }
 
]

Match all entities that are a given info field named subcategory value like Foo AND also are a given alias field named category value pattern like "*Bar*"
OR
Match all entities that are one of the given title value and value pattern namely Title1, *Title2* or Title3 

"entity_rules":
[
    {
       "rule_condition": "AND",
       "rule_items":
        [
            {
                "field": "category",
                "rule_type": "matches",
                "value": "*Bar*",
                "field_type": "alias"   
            },
            {
                "field": "subcategory",
                "rule_type": "matches",
                "value": "Foo",
                "field_type": "info"   
            }
        ]
    },
    {
       "rule_condition": "AND",
       "rule_items":
        [
            {
                "field": "title",
                "rule_type": "matches",
                "value": "Title1,*title2*,Title3",
                "field_type": "title"   
            }
        ]
    }
]

Entity Type

Description

An entity_type defines how to classify a type of data source. For example, you can create a Linux, Windows, Unix/Linux add-on, VMware, or Kubernetes entity type. An entity type can include zero or more data drilldowns and zero or more dashboard drilldowns. You can use a single data drilldown or dashboard drilldown for multiple entity types.

Attributes

Field Type Description
title String The name of the entity type.
description String A description of the entity type.
dashboard_drilldowns Array An array of dashboard drilldown objects. Each dashboard drilldown defines an internal or external resource you specify with a URL and parameters that map to one of an entity fields. The parameters are passed to the resource when you open the URL. See Entity Type Dashboard Drilldown.
data_drilldown Array An array of data drilldown objects. Each data drilldown defines filters for raw data associated with entities that belong to the entity type. See Entity Type Data Drilldown.
vital_metrics Array An array of vital metric objects. Vital metrics are statistical calculations based on SPL searches that represent the overall health of entities of that type. See Entity Type Vital Metrics.

Subordinate objects

Entity Type Dashboard Drilldown

Description

A dashboard_drilldown lists the dashboards associated with an entity and its entity type.

Attributes

Field Type Description
title String The name of the dashboard.
id String A unique identifier for the dashboard.
base_url String An internal or external URL that points to the dashboard. This setting exists because for internal purposes, navigation suggestions are treated as dashboards. This setting is only required if is_splunk_dashboard is false.
is_splunk_dashboard Boolean true if the dashboard is a Splunk XML dashboard. If it's another dashboard type such as a JSON dashboard from the Splunk Dashboards app, or if it's a navigation link, this value is false.
dashboard_type String The type of dashboard being added. This element is required. The following options are available:
  • xml_dashboard - a Splunk XML dashboard. Any dashboards you add must be of this type.
  • navigation_link - a navigation URL. Should be used when base_url is specified.
params Object A set of parameters for the entity dashboard drilldown that provide a mapping of a URL parameter and its alias and static parameters.

Entity Type Data Drilldown

Description

A data_drilldown is a basic unit of configuration for an entity type. Entity data drilldown specifies filters that correlate raw data in Splunk indexes with an entity.

Attributes

Field Type Description
title String Name of the drilldown.
type String Type of raw data to associate with. Must be either metrics or events.
static_filter Object Filter down to a subset of raw data associated with the entity using static information like sourcetype.
entity_field_filter Array Further filter down to the raw data associated with the entity based on a set of selected entity alias or informational fields.

Entity Type Vital Metrics

Description

vital_metrics are a basic unit of configuration for an entity type. Vital metrics are statistical calculations based on SPL searches that represent the overall health of entities of that type.

Attributes

Field Type Description
metric_name String The title of the vital metric. When creating vital metrics, it's a best practice to include the aggregation method and the name of the metric being calculated. For example, Average CPU usage.
search String The search that computes the vital metric. The search must specify the following fields:
  • val for the value of the metric.
  • _time because the UI attempts to render changes over time. You can achieve this by adding span={time} to your search.
  • Fields as described in the split_by_fields configuration of this vital metric. For example, your search should be split by host,region if the split_by_fields configuration is [ "host", "region" ].
split_by_fields Array The fields that the search configuration is split on. Make sure the value matches the split by fields in the actual search.

For example:
search = "..... by host, region"
split_by_fields = ["host", "region"]

matching_entity_fields Array Specifies the aliases of an entity to use to match with the fields specified by split_by_fields in the search result. The order of values should match the order of split_by_fields and the mapping is 1 to 1, so they must be of the same length.

For example:
split_by_fields = ["InstanceId", "region"]
matching_entity_fields = ["instance_id, zone"]

You can only use entity aliases for this field, not informational fields

is_key Boolean Indicates if the vital metric specified is a key metric. A key metric calculates the distribution of entities associated with the entity type to indicate the overall health of the entity type. The key metric is rendered as a histogram in the Infrastructure Overview. Only one vital metric can have is_key set to true.
unit String The unit of the vital metric. For example, KB/s.
alert_rule Object Displays vital metric alert threshold information. The following parameters are displayed:
  • suppress_time: suppress the alert until this time
  • cron_schedule: frequency of alert search
  • is_enabled: if alert is enabled
  • critical_threshold: range of values that indicate critical severity level
  • warning_threshold: range of values that indicate warning severity level
  • info_threshold: range of values that indicate info severity level
  • entity_filter: filter entities based on the field dimensions

For example:
entity_filter: [{"field":"os", "value":"Ubuntu", "field_type":"info"}]

Service KPI

Description

KPI is the data structure that drives the monitoring of service metrics. Each KPI object contains specific information, including a user-configured base search, from which ITSI generates the search that monitors a metric. KPI objects also contain information on how to apply thresholds that determine the metric severity level.

KPI objects (kpis) are defined and contained within the service object type data structure.

Attributes

Field Type Description
_key String Auto-generated unique ID for this KPI.
title String User-defined name for the KPI
description String User-defined description for the KPI.
type String kpi_primary
kpi_template_kpi_id String User-defined ID for the KPI. Used to refer to KPIs within a KPI template in modules. This uniquely identifies a KPI template in ITSI.
isadhoc Boolean If true the search is split on entities and thresholds are computed for both entity and aggregate.
is_service_entity_filter Boolean If true a filter is used on the search based on the entities included in the service.
datamode String The data model to use for search generation if this is a data model type search.
datamodel_filter Array ITSI generated clauses for user-defined filters on top of the data model fields. Used in the KPI search to filter events required by this KPI.
threshold_field String User-specified field on which statistical operations are performed and whose value determines KPI health.
entity_statop String Statistical operation (avg, max, mean, and so on) used to combine data for alert_values on a per entity basis (used if entity_breakdown is true).
aggregate_statop String Statistical operation (avg, max, median, stdev, and so on) used to combine data for the aggregate alert_value (used for all KPI).
urgency Integer User-assigned importance value for this KPI.
unit String User-defined units for the values in threshold field.
entity_id_fields String Fields from this KPI's search events that will be mapped to the alias fields defined in entities for the service containing this KPI. This field enables the KPI search to tie the aliases of entities to the fields from the KPI events in identifying entities at search time.
entity_alias_filtering_fields String Subset of aliases from all entities included in the service containing this KPI, to restrict this KPI to only the subset of entities matching via the subset of aliases. Helps filter entities for this KPI among the ones selected in the service containing this KPI.
cron_schedule String The cron schedule that determines the frequency of this KPI search.
base_search String KPI search defined by user for this KPI. All generated searches for the KPI are based on this search.
kpi_base_search String A basic search generated for the KPI search.
search String Generated search for this KPI for base statistics on the threshold field.
search_entities String Generated search for this KPI for base statistics on the threshold field to use for "Per Entity" threshold type.
search_aggregate String Generated search for this KPI for base statistics on the threshold field to use for "Aggregate" or "Both" threshold type.
search_time_series String Generated search used primarily to show preview information in the KPI configuration page.
search_time_series_entities String Generated search used primarily to show preview information for "Per Entity" threshold type in the KPI configuration page
search_time_series_aggregate String Generated search used primarily to show preview information for "Aggregate" or "Both" threshold type in the KPI configuration page.
search_time_compare String Generated search used specifically by glass table.
search_alert String Generated search used for alerting based on KPI threshold. This is the search that runs on schedule via the saved search for this KPI.
search_alert_entities String Generated search to use for alerting based on KPI threshold for "Per Entity" threshold type.
search_alert_entities String Generated search to use for alerting based on KPI threshold for "Aggregate" or "Both" threshold type.
alert_on String Specified if the threshold type for this KPI is "Per Entity" or "Aggregate" or "Both". Possible values: aggregate, entities, both.
alert_period String User specified interval to run the KPI search in minutes.
alert_lag Integer Contains the number of seconds of lag to apply to the alert search. The maximum value is 30 minutes (1800 seconds).
search_alert_earliest String Earliest time to look for events every time KPI search runs. This determines how far back each time window is during KPI search runs.
tz_offset String ISO time zone offset. Note: Do not change this value.
time_variate_thresholds Boolean If true, thresholds for alerts are pulled from time_variate_thresholds_specification.
time_variate_thresholds_specification Object Data structure for time variate threshold specs.
backfill_enabled Boolean Indicates if backfill has been enabled for this KPI
backfill_earliest_time String Requested earliest time for backfill (relative time offset). Should be in the format -Xd, where 'd' means the time is in days, 'X' is number of days to backfill, and '-' means the date is in the past.
adaptive_thresholds_is_enabled Boolean Determines if adaptive threshold is enabled for this KPI.
adaptive_thresholding_training_window String Earliest time for the Adaptive Threshold training algorithm to run over (latest time is always 'now') (e.g. '-7d')
anomaly_detection_is_enabled Boolean Determines if trending anomaly detection is enabled.
cohesive_anomaly_detection_is_enabled Boolean Determines if cohesive anomaly detection is enabled.
anomaly_detection_alerting_enabled Boolean Determines if anomaly detection will alert for anomalies.
anomaly_detection_training_window String Earliest time for the training algorithm to run over (latest time is always 'now') (e.g. '-7d').
trending_ad Object Data structure for trending anomaly detection algorithm settings. See Anomaly Detection Algorithm Settings.
cohesive_ad Object Data structure for cohesive anomaly detection algorithm settings. See Anomaly Detection Algorithm Settings.
gap_severity String Severity level assigned for data gaps (info, normal, low, medium, high, critical, or unknown)
gap_severity_color String Severity color assigned for data gaps.
gap_severity_color_light String Severity color assigned for data gaps.
gap_severity_value String Severity value assigned for data gaps.
entity thresholds Object User-defined thresholding levels for "Per Entity" threshold type. For more information, see KPI Threshold Setting.
aggregate_thresholds Object User-defined thresholding levels for "Aggregate" threshold type. For more information, see KPI Threshold Setting.
Enabled Boolean If set to 1, KPI is enabled. If absent or not set to 1, KPI is disabled. On upgrade KPI is flagged as enabled. Field is read-only.
base_service_template_id String The key of service template object if the KPI is inherited from a service template.
entity_breakdown_id_field String KPI search events are split by the alias field defined in entities for the service containing this KPI.

Service Template KPI

Description

KPI is the data structure that drives the monitoring of service metrics. KPI objects for service templates differ slightly from KPI objects for services. For example, service template KPIs can only use base searches, not ad hoc searches or searches based on data models. You can't enable anomaly detection for service template KPIs.

KPI objects, kpis, for service templates are defined and contained within the base_service_template object type data structure.

Attributes

Field Type Description
_key String Auto-generated unique ID for this KPI.
title String User-defined name for the KPI
description String User-defined description for the KPI.
type String kpi_primary.
kpi_template_kpi_id String User-defined ID for the KPI. Used to refer to KPIs within a KPI template in ITSI modules. This uniquely identifies a KPI template in ITSI.
is_service_entity_filter Boolean If true a filter is used on the search based on the entities included in the service.
datamode String The data model to use for search generation if this is a data model type search.
datamodel_filter Array ITSI generated clauses for user-defined filters on top of the data model fields. Used in the KPI search to filter events required by this KPI.
threshold_field String User-specified field on which statistical operations are performed and whose value determines KPI health.
entity_statop String Statistical operation (avg, max, mean, and so on) used to combine data for alert_values on a per entity basis (used if entity_breakdown is true).
aggregate_statop String Statistical operation (avg, max, median, stdev, and so on) used to combine data for the aggregate alert_value (used for all KPI).
urgency Integer User-assigned importance value for this KPI.
unit String User-defined units for the values in threshold field.
entity_id_fields String Fields from this KPI's search events that will be mapped to the alias fields defined in entities for the service containing this KPI. This field enables the KPI search to tie the aliases of entities to the fields from the KPI events in identifying entities at search time.
entity_alias_filtering_fields String Subset of aliases from all entities included in the service containing this KPI, to restrict this KPI to only the subset of entities matching via the subset of aliases. Helps filter entities for this KPI among the ones selected in the service containing this KPI.
cron_schedule String The cron schedule that determines the frequency of this KPI search.
base_search String KPI search defined by user for this KPI. All generated searches for the KPI are based on this search.
kpi_base_search String A basic search generated for the KPI search.
alert_on String Specified if the threshold type for this KPI is "Per Entity" or "Aggregate" or "Both". Possible values: aggregate, entities, both.
alert_period String User specified interval to run the KPI search in minutes.
alert_lag Integer Contains the number of seconds of lag to apply to the alert search. The maximum is 30 minutes (1800 seconds).
search_alert_earliest String Earliest time to look for events every time KPI search runs. This determines how far back each time window is during KPI search runs.
tz_offset String ISO time zone offset. Note: Do not change this value.
time_variate_thresholds Boolean If true, thresholds for alerts are pulled from time_variate_thresholds_specification.
time_variate_thresholds_specification Object Data structure for time variate threshold specs.
adaptive_thresholds_is_enabled Boolean Determines if adaptive threshold is enabled for this KPI.
adaptive_thresholding_training_window String Earliest time for the Adaptive Threshold training algorithm to run over (latest time is always 'now') (e.g. '-7d')
gap_severity String Severity level assigned for data gaps (info, normal, low, medium, high, critical, or unknown)
gap_severity_color String Severity color assigned for data gaps.
gap_severity_color_light String Severity color assigned for data gaps.
gap_severity_value String Severity value assigned for data gaps.
entity thresholds Object User-defined thresholding levels for "Per Entity" threshold type. For more information, see KPI Threshold Setting.
aggregate_thresholds String User-defined thresholding levels for "Aggregate" threshold type. For more information, see KPI Threshold Setting.
Enabled Boolean If set to 1, KPI is enabled. If absent or not set to 1, KPI is disabled. On upgrade KPI is flagged as enabled. Field is read-only.
entity_breakdown_id_field String KPI search events are split by the alias field defined in entities for the service containing this KPI.

Service Health KPI

The Service Health KPI tracks the health score of an entire service. Service Health KPIs have the same data structure as user defined KPIs.

Service Health KPIs have the following ID format: 

SHKPI-<_key id for the service>

Subordinate objects

KPI Threshold Settings

Description

KPI Threshold Settings define the thresholds that a KPI uses to compute health status information. KPI Threshold Settings also contain information for rendering KPI threshold graphs.

Attributes

Field Type Description
gaugeMin Integer Minimum value for the threshold gauge specified by user.
gaugeMax Integer Maximum value for the threshold gauge specified by user.
search String Generated search used to compute the thresholds for this KPI.
baseSeverityValue Integer Value for base threshold level.
baseSeverityColor String Severity color assigned for the base threshold level.
baseSeverityColorLight String Severity light color assigned for the base threshold level.
baseSeverityLabel String Severity label assigned for the base threshold level, including info, warning, critical, etc.
metricField String Thresholding field from the search.
renderBoundaryMin Integer Lower bound value to use to render the graph for the thresholds.
renderBoundaryMax Integer Upper bound value to use to render the graph for the thresholds.
isMaxStatic Boolean True when maximum threshold value is a static value, false otherwise.
isMinStatic Boolean True when min threshold value is a static value, false otherwise.

Subordinate data structures

KPI Threshold Levels

Description

KPI Threshold Levels determine how ITSI extracts health status information from KPI searches. Threshold levels are user-configured values that can be augmented further using adaptive thresholding.

Attributes

Field Type Description
thresholdValue Integer Value for the threshold field stats identifying this threshold level. This is the key value that defines the levels for values derived from the KPI search metrics.
severityColor String Severity color assigned for this threshold level.
severityColorLight String Severity light color assigned for this threshold level.
severityValue Integer Severity value assigned for this threshold level.
severityLabel String Severity label assigned for this threshold level like info, warning, critical, etc.
dynamicParam Integer Value of the dynamic parameter for adaptive thresholds.

KPI Threshold Templates

Description

A kpi_threshold_template is a set of predefined threshold values that you can apply to multiple KPIs.

Attributes

Field Type Description
title String Name of this template.
description String Description of this particular template.
adaptive_thresholding_training_window String Earliest time for the adaptive threshold training algorithm to run over. The latest time is always now.
time_variate_thresholds Boolean If true, thresholds for alerts are pulled from time_variate_thresholds_specification.
Time_variate_thresholds_specification Object Data structure for time variate threshold specification.
adaptive_thresholds_is_enabled Boolean If true, adaptive thresholding is enabled for this KPI.
sec_grp String The team the object belongs to. This object can only belong to default_itsi_security_group (Global team).

KPI Base Search

Description

Searches that can be aggregated together to reduce overall search load. KPI Base Searches include the core attributes of a KPI for search generation.

kpi_base_search objects are contained within the KPI (kpis) object data structure.

Attributes

Field Type Description
entity_alias_filtering_fields String The fields to filter on. See KPI definition.
_version String ITSI version number of this KPI base search.
description String General description for this KPI base search.
mod_source String Source of the last modification.
mod_time String The time of the last modification based on UTC time zone.
is_service_entity_filter Boolean If true a filter is used on the search based on the entities included in the service.
actions String
object_type String kpi_base_search
is_entity_breakdown String Determines if search breaks down by entities. See KPI definition.
_owner String KV store owner.
source_itsi_da String Source of DA used for this search. See KPI Threshold Templates.
metrics Array Set of statistical operations performed on threshold field.
aggregate_statop String Statistical operation (avg, max, median, stdev, and so on) used to combine data for the aggregate alert_value (used for all KPI).
unit String User-defined units for the values in threshold field.
title String Name of this metric
_key String Internal identifier.
threshold_field String The field on which the statistical operation runs.
entity_statop String Statistical operation (avg, max, mean, and so on) used to combine data for alert_values on a per entity basis (used if is_entity_breakdown is true).
search_alert_earliest String Earliest time to look for events every time KPI search runs. This determines how far back each time window is during KPI search runs.
alert_period String User specified interval to run the KPI search in minutes.
alert_lag String Contains the number of seconds of lag to apply to the alert search, max is 30 minutes (1800 seconds).
base_search String KPI search defined by user for this KPI. All generated searches for the KPI are based on this search.
entity_id_fields String Fields from this KPI's search events that will be mapped to the alias fields defined in entities for the service containing this KPI. This field enables the KPI search to tie the aliases of entities to the fields from the KPI events in identifying entities at search time.
identifying_name String Internal only
title String Name of this KPI base search.
mod_timestamp String Timestamp of last modification based on UTC time zone.
acl String Access control blob.
_user String Like owner, but different.
_key String Auto-generated unique ID for this KPI.
sec_grp String The team the object belongs to. This object can only belong to default_itsi_security_group (Global team).

For more information, see Create KPI base searches in ITSI in the Service Insights Manual.

Glass Table

Description

ITSI glass tables are custom visualizations that let you monitor KPI search results.

glass_table objects define all widgets and drawing elements that appear in the glass table.

Attributes

Field Type Description
_key String Unique identifier for this glass table.
title String Name of this glass table.
description String User-defined description for this glass table.
object_type String glass_table.
latest String Latest time for all of the widget searches on the glass table.
latest_label String Latest label displayed in the glass table instant picker. Matches latest attribute.
svg_coordinates String x and y viewbox offsets for the glass table.
content Array Array of JSON structures containing all attributes needed to draw the glass table. See Glass Table Widget Configuration.
is_epoch Boolean True when the glass table uses a custom (non-preset) time, false otherwise.
templateSelectedServiceId String The id of the service currently in focus if templatization is enabled.
templateSwappableServiceIds Array The array of services available to be swapped to for templatization.

Subordinate data structures

Glass Table Widget Configuration

Description

Glass Table Widget Configuration (content) is an array of JSON structures that contains all of the attributes needed to render the glass table. Each element of the array represents one glass table widget, and the attributes of the element are parsed into a glass table BaseWidgetViewManager object.

Attributes

Field Type Description
search String The search to power the widget.
labelVal String The text to show in the label located beneath the widget.
labelFlag Boolean True if labelVal is to be shown with the widget, false otherwise.
vizType Integer Numeric indication of which visualization type the widget is - SingleValue, Gauge, Sparkline, SVD from 0-3, respectively
threshold_field String Field in data to which thresholds apply.
threshold_comparator String Comparator used for threshold severity computation
threshold_values Array Array of values to indicate the bounds of the thresholds set for the widget.
threshold_labels Array Array of labels to match the threshold values set for the widget.
context_id String Id of service to which the widget's KPI belongs.
kpi_id String Id of KPI the widget represents
searchSource String Source of search for glass table widget - can be datamodel or ad hoc.
dataModelSpecification String Data model specification for the datamodel search.
dataModelStatOp String Datamodel stats operation for the datamodel search.
dataModelWhereClause String Datamodel where clause for the datamodel search.
threshold_eval String Threshold eval search clause for threshold severity evaluation.
aggregate_eval String Aggregate eval search clause for threshold severity evaluation.
base_search String Base search of the KPI the widget represents.
search_alert_earliest String Earliest time for the search that powers the widget.
entities String List of entities that the widget's KPI contains.
search_aggregate String Aggregate search of the KPI the widget represents.
search_time_series_aggregate String Time series search of the KPI the widget represents
search_time_compare String Compare time series search of the KPI the widget represents (for the SVD viz type).
search_type String Type of search the widget is powered by. Must match one of the search_* attributes of the widget.
relativeEarliest String Earliest time (in relative units) for the search that powers the widget.
defaultWidth Integer Initial width to use for the widget.
defaultHeight Integer Initial height to use for the widget.
existingKPI Boolean True if KPI exists in user's system, false otherwise.
alert_on String Threshold alert type (aggregate or entities) of the KPI the widget represents.
isThresholdEnabled Boolean True if thresholds should be applied to the widget's search results, false otherwise.
useKPISummary Boolean true if widget uses the kpi_summary_index to power its search, false otherwise.
unit String Unit string for widget to display.
gap_severity String Gap severity value of the KPI the widget represents.
gap_severity_color String Gap severity color of the KPI the widget represents.
drilldownSettingsModel String Model to hold properties required for generating URLs for custom drilldown.
useCustomDrilldown Boolean True if widget has custom drilldown turned on, false otherwise.

Glass Table Icon

Description

Contains SVG icon definitions and metadata for glass table icons.

Attributes

Field Type Description
_key String Auto-generated unique identifier for this icon.
title String Name of the icon.
category String Category of the icon.
default_width Integer Width of the icon.
default_height Integer Height of the icon.
svg_path String SVG path defining shape of the icon.
immutable Boolean Should the REST API allow editing of this icon. False for all icons imported from .conf files.
_time String Timestamp when the icon was added.
_owner String Name of the user that added this icon.

Deep Dive

Description

ITSI deep dives are investigative tools that help you identify and troubleshoot issues in your IT environment. You can use deep dives to view KPI search results over time, zoom-in on KPI metrics and log events, and visually correlate root cause. You can add different types of lanes to a deep dive view, including KPI lanes, which let you view KPI metrics in detail. You can also add lanes to view ad hoc and data model searches.

deep_dive objects contain all of the elements required to render deep dive lanes.

Attributes

Field Type Description
_key String Auto-generated unique identifier for this deep dive.
description String User-defined description for this deep dive.
title String Name of the deep dive.
object_type String deep_dive
earliest_time String Earliest time for all of the searches in this deep dive.
latest_time String Latest time for all of the searches in this deep dive.
focus_id String The service id of the service in focus.
topology_id String Define the service to be put in focus in the deep dive topology view. If none exists then the focus_id is set as the topology_id. view sidebar
lane_settings_collection Array <Array of lane settings specifying each lane's configuration. See Deep Dive Lane Settings.
is_named Boolean True when the deep dive is saved, false otherwise.

Subordinate data structures

Deep Dive Lane Setting

Description

Configuration settings that define what information a deep dive lane shows. Deep dive views use these settings for per lane configuration.

Attributes

Field Type Description
title String Name of the lane to display.
subtitle String The subtitle of the lane to display.
laneType String The type of lane to render. Possible values: event, kpi, metric (the default).
graphType String The type of graph to render
search String The search to use to get data for the lane.
searchSource String Represents how a search is generated. Possible values: datamodel, ad hoc search, or kpi search.
dataModelSpecification Object An object showing the selections that went into the generation of the search, null unless searchSource is data model. If defined, it is structured as {datamodel: <Data Model name> object: <Object Name>, field: <Field Info Data Structure>.
dataModelStatOp String Stats operation used in the data model search.
dataModelWhereClause String Where clause defined during data model search creation.
overwriteKpiTitle String Overwrite KPI title with user specified title.
overwriteEntityTitle String Overwrite Entity title with user specified title.
kpiTitle String The original title of the KPI as defined in the KPI model.
kpiServiceId String The id of the service associated with the selected KPI.
kpiUnit String The unit of the KPI driving this lane.
kpiAddToSummary String Add or remove from kpi summary based on user selection. [yes, no] Yes runs the search against kpi summary index and no runs raw search.
kpiStatsOp String Stats operation to calculate the KPI value, avg by default [avg, max, min, median].
entityAddToSummary String Shows the accelerated output for entity lanes. Always set to "yes."
thresholdIndicationEnabled String Enable/disable threshold indication. Disabled by default.
thresholdIndicationType String Type of threshold indication. [foreground/background] Foreground selected by default.
hideGraph String Only available with background threshold indications. If selected, hides the graph and only shows the top view with background thresholds [yes, no].
verticalAxisScale String Determines the scale of the y axis. It is linear or log.
verticalAxisBoundaryType String Determine the extent of the y axis. It is staticValue, value, or zeroValue.
verticalAxisStaticBounds Object If static, these are the bounds to use. Otherwise this is ignored. This is an object of the form[<min number>, <max number>].
dataGaps String null values in the data can be represented as gaps or connected through the graph.
graphColor String The color of the graph to render.
graphSeries String The field in the data which to plot as the range, if unspecified plots all.
excludeSeries String The series of data to omit from being displayed in graph. Series with a leading _ (indicating internal use) is always excluded.
laneOverlaySettingsModel Object Model to define the overlay lane settings.

Time Variate Thresholds Specification

Description

This data structure contains the threshold policy collection. A threshold policy includes information on which thresholds are to be applied (a threshold setting model), how those thresholds are generated, and the time periods to which the threshold policy applies. Each policy object includes a single time_blocks attribute that contains a list of time periods with which the policy is associated.

In the case of static thresholding there are no parameter attributes. In the case of dynamic thresholding, parameters are stored in a simple object within the policy.

Attributes

Field Type Description
title String The title of the threshold spec. Used when creating/modifying threshold spec templates.
description String User-defined description of the threshold specifications.
policies Object JSON object keyed by policy ID.
time_blocks Array Determines time periods with which the policy is associated.

Collection details

A threshold policy collection is accessed by the UUID key of the policy. There is no limit to the number of policies a collection can contain.

Threshold policies

{
  _key: <UUID>,
  title: <optional title>,
  aggregate_thresholds: <ThresholdSettingsModel>,
  entity_thresholds: <ThresholdSettingsModel>,
  policy_type: <ENUM of "static", "stdev", "quantile", or "range">
  time_blocks: <[] of time blocks>
}

Time blocks attribute

The time_blocks attribute uses a simplified cron expression format: 

[
  ['<minute> <hour> <*> <*> <day>', <duration in minutes>] 
]

<minute> values can only be 0, 15, 30, or 45. <hour> values use 24 hour day format. Unlike standard cron expressions, <day> values run from 0 (Monday) through 6 (Sunday).

For example: 

'time_blocks': [
  ['15 3 * * 3,4', 60]   // 1 hour time range, 3:15AM - 4:15AM on Thurs, Fri
]

The time block attribute must specify exactly one cron expression.

If your existing configuration doesn't match the UTC timestamp, use the kvstore_to_json.py script to correct the time zone discrepancy. See Time zone offset operations (mode 3) in the Administration Manual.

Usage

Generation of Time Varied Threshold lookup in custom search command

The main usage of the threshold policy structures is to determine which thresholds should be applied based on the time.

Configure adaptive threshold commands

Threshold policy structures are used to configure how the adaptive threshold commands work. They need access to the time blocks for a particular policy and the parameters in the policy. They access this information by reading the KPI and applying the information stored within.

Maintenance Calendar

Description

Use maintenance_calendar to put services and entities in maintenance mode at required intervals.

Attributes

Field Type Description
_key String Unique ID for the entry in the KV store.
title String Title of the maintenance window.
comment String Optional description of the maintenance window.
objects Array Array of dictionaries describing the objects put in maintenance by this maintenance window.
The schema for each object definition in the array:
_key: Unique if assigned to the object currently.
object_type: Type of object being identified. Currently only entity and service are supported.
start_time String Timestamp that marks the beginning of maintenance window. Based on UTC time.
end_time String Timestamp that marks the end of maintenance window. Based on UTC time.

If your existing configuration doesn't match the UTC timestamp, use the kvstore_to_json.py script to correct the time zone discrepancy. See kvstore_to_json.py operations in ITSI in the Administration Manual.

Event Management State

Description

The event_management_state object stores user settings for custom saved views of Episode Review. For instructions to save custom views through the UI, see Save a custom view of Episode Review in the User Manual.

Attributes

Field Type Description
_key String The unique ID for Episode Review view in the KV store.
title String A user-defined name for the custom episode review.
earliest String The earliest time for the main search in the Episode Review custom view.
latest String The latest time for the main search in the Episode Review custom view.
fetchLimit Integer The maximum number of notable events to fetch in a single request.
sortField String The field in the data (column in Episode Review) to sort notable events by.
sortDirection String Whether to sort notable events in ascending or descending order.
arbitrarySearch String The Splunk search string used to filter raw notable events.
filterCollection Array A set of filters that represent the Episode Review page filters.
viewingOption String Whether to display notable events as standard or prominent mode in Episode Review.
eventDeduplication Boolean If true, episode view is turned on. Otherwise individual notable events are displayed.
columnsShown Array A list of fields in the data (columns in Episode Review) to display.

Notable Event Group

Description

The notable_event_group contains information about an episode.

Attributes

Field Type Description
severity String The level of importance of the episode. Values must match an integer specified in the default version of itsi_notable_event_severity.conf (or the local version if you created one). Default values:
1 - Info
2 - Normal
3 - Low
4 - Medium
5 - High
6 - Critical
status String The triage status of the episode in Episode Review. Values must match an integer specified in the default version of itsi_notable_event_status.conf (or the local version if you created one). Default values:
0 - Unassigned
1 - New
2 - In Progress
3 - Pending
4 - Resolved
5 - Closed
owner String The Splunk user who is the owner of the episode.
_key String The episode ID that a change is associated with.

Notable Event Comment

Description

notable_event_comment contains comments associated with an episode.

Attributes

Field Type Description
comment String The text of the comment.
event_id String The episode ID that the comment is associated with.
is_group Boolean The episode ID that the comment is associated with.
filter_search String The search to retrieve all the comments for an episode.
earliest_time String The time, in UTC, of the first event in the episode.
latest_time String The time, in UTC, of the last event in the episode.

Notable Event Aggregation Policy

Description

notable_event_aggregation_policy contains the data for a notable event aggregation policy which aggregates notable events into episodes.

Attributes

Field Type Description
disabled Boolean 1 if the aggregation policy is disabled and 0 if enabled.
breaking_criteria Object A JSON blob of all the criteria used to break an episode.
filter_criteria Object A JSON blob of all the criteria used to filter events into an episode.
is_default Boolean Indicates if this is the default aggregation policy. 1 if it's the default policy and 0 if not.
description String The description of the notable event aggregation policy.
group_severity String The default severity of each episode created by the notable event aggregation policy.
group_status String The default status of each episode created by the notable event aggregation policy.
group_asignee String The default owner of each episode created by the notable event aggregation policy.
group_description String The default description of each episode created by the notable event aggregation policy.
title String The title of the notable event aggregation policy.
rules Array An array of all the rules and actions to be executed for the notable event aggregation policy.
split_by_field String A string containing all the fields to split episodes by.

Notable Event Email Template

Description

notable_event_email_template contains the data for email templates for episode actions. Once you create a template it's available for selection in all aggregation policies and is not policy-specific.

Attributes

Field Type Description
title String The name of the message template. This element is required.
message String The body of the email. Supports tokens such as $result.title$ and $result.description$. This element is required.

Correlation Search

Description

correlation_search contains the data for a correlation search. A correlation search is a recurring search that generates a notable event when search results meet specific conditions. A multi-KPI alert is a type of correlation search.

Attributes

Field Type Description
is_scheduled Integer Values: 1 means scheduled; 0 means not scheduled.
disabled Integer Values: 1 means disabled; 0 means enabled.
cron_schedule String Schedule searches to run periodically at fixed times, dates, or intervals using a cron expression. Default value is */5* * * * (every 5 minutes).
dispatch.earliest_time String Indicates the beginning of the time range for the search. The default value is -15m.
dispatch.latest_time String Indicates the end of the time range for the search. The default value is -now.
description String A description of the type of issue the search is intended to detect.
search String The Splunk search to run.
name String A name that describes the correlation search. For example, "cpu_load_percent".
action.itsi_event_generator.param.title String The title to use for the notable event in Episode Review. For example, mysql-01 server cpu Load %.
action.itsi_event_generator.param.description String A brief phrase to describe the notable event. For example, "This alert triggers when DB CPU load on the mysql-01 server reaches 80%."
action.itsi_event_generator.param.status String The triage status of the event in Episode Review. You can provide a token in the format %fieldname% to substitute the value of a third-party alert field. Values must match an integer specified in $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_notable_event_status.conf or /default/itsi_notable_event_status.conf if a local version does not exist. By default, these values are 0-5.
action.itsi_event_generator.param.owner Array The ITSI role to which the notable event is assigned in Episode Review.
action.itsi_event_generator.param.severity String The level of importance of the event. You can provide a token in the format %fieldname% to substitute the value of a third-party alert field. Values must match an integer specified in $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_notable_event_severity.conf or /default/itsi_notable_event_severity.conf if a local version does not exist. By default, these values are 1-6.
action.itsi_event_generator.param.drilldown_search_title String The name of the drilldown search link. You can drill down to a specific Splunk search from an episode in Episode Review.
action.itsi_event_generator.param.drilldown_search_search String The Splunk search you drill down to.
action.itsi_event_generator.param.drilldown_search_latest_offset String Defines how far ahead from the time of the event to look for related events.
action.itsi_event_generator.param.drilldown_search_earliest_offset String Defines how far back from the time of the event to start looking for related events.
action.itsi_event_generator.param.drilldown_title String The name of the drilldown website link if you want to drill down to a specific website from the episode in Episode Review.
action.itsi_event_generator.param.drilldown_uri String The website you drill down to.
action.itsi_event_generator.param.event_identifier_fields String These identifier fields form the event hash field, which is added to every notable event to help identify unique alarm types.
action.itsi_event_generator.param.service_ids String One or more ITSI services to which this correlation search applies. You can only specify services that belong to teams for which you have read access.
action.itsi_event_generator.param.entity_lookup_field String The field in the data retrieved by the correlation search that is used to look up corresponding entities. For example, host.
action.itsi_event_generator.param.search_type String search_type = "basic", "composite_kpi_score_type", or composite_kpi_percentage_type
action.itsi_event_generator.param.meta_data Object One of two JSON object schemas, depending on whether it is a correlation search or a multi-KPI alert.
Correlation search object schema:
  • threshold_health_score - threshold score set by user
  • threshold_status - threshold status (default is undefined)
  • suppression criteria fields
  • alert_type - score or status
  • is_suppression - if suppression is enabled or not
  • is_consecutive - if count is consecutive or not
  • count - minimum number of times if this alert happens
  • suppression_period - suppression period in minute if it is non-consecutive
  • min_alert_period - minimum alert period of selected KPIs
  • run_every - frequency of search in minutes
  • score_based_kpis - list of KPIs which is added as part of a composite KPI. Each object in the list must have kpiid - <kpi id>, serviceid - <service id>, urgency - <urgency value>

Multi-KPI alert object schema:

  • time_label - time label for time range
  • percentage_based_kpis - list of KPIs and service IDs included. Each item should contain kpiid - <kpi id>, serviceid - <service id>, label_thresholds - <threshold and operation type for trigger>. label_thresholds data format is as follows:

{ operation : 'OR', // default for now thresholds : [ { severity: <severity name>, percentage: <percentage value>, percentage_operation: '>=', // default for now } ...... ] }

action.itsi_event_generator.param.editor String One of two values: advance_correlation_builder_editor or multi_kpi_alert_editor. It directs to the specific UI page to make edits based on the type of search, correlation search or multi-KPI alert.
action.itsi_event_generator Integer Value: 1
actions String Value: itsi_event_generator
alert.suppress Integer Enable suppression to minimize the number of duplicate notable events sent to Episode Review. Values: 1 (means enabled) or 0 (means disabled).
alert.suppress.fields String The fields to consider when determining if another event matches the current one.
alert.suppress.period String The number of seconds to ignore other events that have the same field values.
action.rss Integer Included in RSS feed. Values: 1 (means enabled) or 0 (means disabled).
action.email Integer Send an email when the alert is triggered. Values: 1 (means enabled) or 0 (means disabled).
action.email.to String The email addresses to send the email to.
action.email.subject String The subject of the email.
action.email.sendcsv Integer Send an email in CSV format. Values: 1 (means enabled) or 0 (means disabled).
action.email.sendpdf Integer Send an email with a PDF attachment. Values: 1 (means enabled) or 0 (means disabled).
action.email.inline Integer Send an email with the text inline. Values: 1 (means enabled) or 0 (means disabled).
action.email.format String Default value is pdf. Other values: html, csv.
action.email.sendresults String Include alert information as an email attachment. Values: 1 (means enabled) or 0 (means disabled).
action.script Integer Triggers a shell script if enabled. Values: 1 (means enabled) or 0 (means disabled).
action.script.filename String Provide the file name of the shell script to run when this alert is triggered.

Service Analyzer

Description

Service Analyzer is the ITSI UI home page. It displays service and KPI health scores that are trending at top severity levels. You can configure Service Analyzer to filter the display of services and KPIs relevant to the user.

The Service Analyzer object is called home_view.

Attributes

Field Type Description
_key String Unique ID for the entry in KV store.
object_type String home_view
-owner String User that creates the saved service analyzer.
title String User given title for the service analyzer.
earliest_time String Earliest time for the searches.
latest_time String Latest time for the searches.
serviceWhitelist String List of filtered services.
kpiWhitelist String List of filtered kpis.
isServiceFilterEnabled Boolean True if services are filtered, false by default.
isKpiFilterEnabled String True if kpis are filtered, false by default.
serviceTilesSettings Object SeverityTilesSettingModel with number of kpi tiles and filter.
view String Determines if service analyzer view is standard or full screen. Standard is default.
isDefault String True if it is the default (standard) service analyzer, false otherwise.
titleSize String medium|large], large by default.
searchType String maxseverity] [aggregate|maxseverity] aggregate shows the most recent service value and the max severity is service value unless there is an entity value with worst severity.

Team

Description

Teams are used to restrict service-level information in the following objects:

  • Glass tables
  • Service analyzers
  • Deep dives
  • Episode Review
  • Correlation searches
  • Multi-KPI alerts

The team object is called team.

Attributes

Field Type Description
identifying_name String The name of the team. Does not have to match title.
acl String Access control list for the team. Must include itoa_admin.
title String User provided name of the team. Does not have to match identifying_name.
description String User provided description of the team.
children List List of private teams created in ITSI. For private teams, this field will be an empty list.
parents List The parent of this team. Cannot be configured in current release.
_key String Unique ID for the entry in KV store.

Anomaly Detection Algorithm Settings

Attributes

Field Type Description
Sensitivity Integer Determines sensitivity of algorithm to variance in data. Note that acceptable values for both trending and cohesive algorithm sensitivity are between 0 and the sensitivity_max parameter value, as specified in the respective [trending:limits]and [cohesive:limits] stanzas, in mad.conf in the SA-ITSI-MetricAD namespace.
Last modified on 16 June, 2022
ITSI REST API reference  

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.11.0, 4.11.1, 4.11.2, 4.11.3, 4.11.4, 4.11.5, 4.11.6, 4.12.0 Cloud only, 4.12.1 Cloud only, 4.12.2 Cloud only, 4.13.1

Acrobat logo Download manual
Acrobat logo Download this page

Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters