Plan your ITSI deployment

Deploy Splunk IT Service Intelligence (ITSI) on a configured Splunk platform installation. Review the system and hardware requirements and the search head and indexer considerations before deploying IT Service Intelligence.

Preparation for deployment

Before you deploy IT Service Intelligence, perform the following steps:

1. Compile a list of services, KPIs, and glass table views that you want to create.
2. Compile a list of your entities. Entities are usually hosts, but can also be users, mobile devices, and so on. Entities for hosts must include, at a minimum, the IP address, host name, and designated role. For example, web, db, or app server.
3. Make sure your Splunk ITSI instance includes the default admin user. Deleting or renaming this user breaks ITSI installation and operation.
4. Verify your existing hardware performance using the following search query:
   

   index=_introspection sourcetype=splunk_resource_usage component=Hostwide earliest=-5m | timechart avg(data.cpu_user_pct) by host

   If the query takes more than 2-5 seconds to complete, check performance in the Job Inspector to investigate the issue. This slowness might indicate your current hardware is insufficient or badly configured, or you might have a high latency dispatch that requires architecture changes.
5. Confirm Splunk Enterprise version compatibility. See the Splunk products version compatibility matrix.

Available deployment architectures

You can deploy Splunk IT Service Intelligence in a single instance deployment or a distributed search deployment. Splunk IT Service Intelligence is also available in Splunk Cloud Platform. Before you deploy Splunk IT Service Intelligence on premises, familiarize yourself with the components of a Splunk platform deployment. See Components of a Splunk Enterprise deployment in the Capacity Planning Manual.

Single instance deployments

For a simple and small deployment, install ITSI on a single Splunk platform instance. A single instance functions as both a search head and an indexer. Use forwarders to collect your data and send it to the single instance for parsing, storing, and searching.

You can use a single instance deployment for a lab or test environment, or a small system with one or two users running concurrent searches. For instructions on installing ITSI on a single Splunk Enterprise instance, see Install Splunk IT Service Intelligence on a single instance.

Distributed deployments

You can deploy ITSI across any distributed architecture supported by Splunk Enterprise. This includes all types of deployment topologies, from small departmental deployments using a single instance for both indexer and search head, to large enterprise deployments using several search heads, dozens of indexers, and hundreds of forwarders. See Types of distributed deployments in the Splunk Enterprise Distributed Deployment Manual.

• For information about installing ITSI in a distributed environment, see Where to install IT Service Intelligence in a distributed environment.
• Improve search performance by using an index cluster and distributing the workload of searching data across multiple nodes. Using multiple indexers allows both the data collected by the forwarders and the workload of processing the data to be distributed across the indexers.
• Use forwarders to collect your data and send it to the indexers.

In a distributed search deployment, and to implement search head clustering, configure the search head to forward all data to the indexers. See Best practice: Forward search head data to the indexer layer in the Splunk Enterprise Distributed Search manual.

To properly scale your distributed search deployment with ITSI, see Introduction to capacity planning for Splunk Enterprise in the Capacity Planning Manual and Indexer and search head sizing examples.

Cloud deployments

Splunk IT Service Intelligence is available as a service in Splunk Cloud Platform. The Splunk Cloud Platform deployment architecture varies based on data and search load. Splunk Cloud Platform customers work with Splunk Support to set up, manage, and maintain their cloud infrastructure. For information on Splunk Cloud Platform deployments, see Splunk Cloud Platform deployment types in the Splunk Cloud Platform Admin Manual.

Splunk Enterprise system requirements

To determine the compatibility of the IT Service Intelligence versions and Splunk platform versions, see the Splunk products version compatibility matrix. To determine compatibility with Splunk Cloud Platform versions, see Splunk Cloud Platform system requirements.

Splunk IT Service Intelligence requires a 64-bit OS install on all search heads and indexers. For a list of supported operating systems, browsers, and file systems, see System requirements for use of Splunk Enterprise on-premises in the Splunk Enterprise Installation Manual.

Splunk Cloud Platform system requirements

To determine the compatibility of IT Service Intelligence versions and Splunk Cloud platform versions, see Current Splunk Cloud Platform and Premium App versions.

Hardware requirements

CPU core count and RAM are critical factors in indexer and search head performance. ITSI requires minimum hardware specifications that you increase according to your needs and usage of ITSI. These specifications also apply for a single instance deployment of ITSI.

Note that a search head in this case refers to a dedicated ITSI search head infrastructure. If ITSI shares a search head with other applications, additional resources are required beyond 16 cores and 12 GB of RAM.

Machine role	Minimum CPU	Minimum RAM	Minimum vCPU
Search head	16 cores required, 24+ recommended, or 32 vCPU at 2Ghz or greater speed per core	12 GB required, 16+ recommended	32 vCPU required, 48+ recommended
Indexer	16 cores	32 GB	32 vCPU required, 48+ recommended

Indexing is an I/O-intensive process. The indexers require sufficient disk I/O to ingest and parse data efficiently while responding to search requests. For the latest IOPS requirements to run Splunk Enterprise, see Reference Hardware: Indexer in the Splunk Enterprise Capacity Planning Manual.

You might need to increase the hardware specifications of your own ITSI deployment above the minimum hardware requirements depending on your environment. Depending on your system configuration, refer to the mid-range or high-performance specifications for Splunk platform reference hardware. See Mid-range specification and High-performance specification in the Splunk Enterprise Capacity Planning Manual.

If the number of indexer CPU cores in your deployment exceeds the minimum hardware specifications, you can implement one of the parallelization settings to improve the indexer performance for specific use cases. See Parallelization settings in the Capacity Planning Manual.

Operating system requirements

For a list of supported operating systems, browsers, and file systems, see System requirements for use of Splunk Enterprise on-premises in the Splunk Enterprise Installation Manual.

Ubuntu

When installing IT Service Intelligence on Ubuntu, use Bash shell. Do not use Dash shell as it can result in defunct processes.

ITSI license requirements

ITSI requires a separate ITSI license in addition to your Splunk Enterprise license. Install both the ITSI license and the Splunk Enterprise license on the license master. Your Splunk representative will provide you with an appropriate ITSI license at the time of purchase. For ITSI license installation instructions, see Install a license in the Splunk Enterprise Installation and Configuration Manual.

Splunk ITSI is a right-to-use (RTU) license. Splunk ITSI meters Splunk indexes for ingest-based license usage and capacity consumption calculations. These Splunk indexes are identified based on:

• Direct ingestion of data into Splunk and use of that data for ITSI use cases
• Indexes containing data used to populate ITSI KPIs

Once the indexes are identified, then license usage is measured on ingest calculation for data sources and indexes used by Splunk ITSI.

To calculate ITSI capacity consumption for ingest-based licenses, use the Splunk App for Chargeback. For more information, see Splunk App for Chargeback. Select Docs from the app's main dashboard to read additional documentation.

To calculate ITSI capacity consumption for workload-based licenses (for example, SVC), use the SVC Statistics dashboard in ITSI. For more information, see Use the ITSI SVC Statistics dashboard.

IT Service Intelligence ships with an internal license stack called the IT Service Intelligence Internals *DO NOT COPY* stack. Because ITSI Event Analytics functionality generates a large number of notable events, this internal stack ensures that you don't pay for these generated events. The sourcetypes used to track notable events and episodes are counted on this special stack with no impact on your Splunk Enterprise license. When calculating your daily license usage, disregard this stack, as it only counts internal ITSI usage.

However, if ITSI is installed on multiple environments with multiple license masters, and any indexer interacts with both environments, a duplicate licensing error occurs because both environments have the same auto-generated ITSI license stack. To remedy this issue, delete the internal license, install a secondary internal license, and disable the license_checker modular inputs:

1. Click Settings > Licensing and delete the IT Service Intelligence Internals *DO NOT COPY* stack.
2. Click Add license and upload the following license key file:

   Expand to see the license key file

   <license><signature>UrBfGVqTyzcpKMr7JDHsEFIVO1RLRWn9dZaKBm5BTRzdz6MIhE1XRf4FJ5JVpUxquLwf1LCkJ4QV78uq4kLClhvWuGPYNaMBmGe9w8MnWQi2TlMeTwwVHNoX6FB3TiNujQj3g+sQsP0IYTdtV98etDRNwOYIkuLFPap6dhvBIulBkFiBhGI+ELmHV7LRGcVgJYgBF8zpOjVXSQMh0RL+6MWbrVdkKZU5ducxDJcpUEjp0PR3QePtczXm5ZdETui42mtpyiZsMlcYMGmQWS9erKst5EX8R9BSyudZkbZL4uoSVdVv8691Ml1GLb2pbgQQLm/Qwb2XaCk5QNo4Odv4A==</signature>
   <payload>
   <type>fixed-sourcetype</type>
   <group_id>Enterprise</group_id>
   <quota>107374182400000</quota>
   <max_violations>5</max_violations>
   <window_period>30</window_period>
   <creation_time>1594969200</creation_time>
   <label>IT Service Intelligence Internals *DO NOT COPY*</label>
   <expiration_time>2163221999</expiration_time>
   <features>
   <feature>Auth</feature>
   <feature>FwdData</feature>
   <feature>LocalSearch</feature>
   <feature>ScheduledSearch</feature>
   <feature>Alerting</feature>
   <feature>SplunkWeb</feature>
   </features>
   <add_ons>
   <add_on name="itsi" type="app">
   <parameter key="size" value="1"/>
   </add_on>
   </add_ons>
   <sourcetypes>
   <sourcetype>itsi_*</sourcetype>
   </sourcetypes>
   <guid>2AEECCCF-EDBC-499E-862C-8C79844114D4</guid>
   </payload>
   </license> 
   

3. Click Settings > Data inputs > IT Service Intelligence license checker and disable both inputs.

Java requirements

IT Service Intelligence requires Java 8.x - 11.x to run anomaly detection and event management features. ITSI supports OpenJDK and Oracle JDK 8-11. Java installation is required on search heads only, not indexers or forwarders.

IT Service Intelligence search head considerations

IT Service Intelligence does not require a dedicated search head. However, ITSI is not supported on the same search head as Splunk Enterprise Security. For scalability beyond about 200 discrete KPIs, a search head cluster is a more stable option.

You can't disable real-time searches on either the indexer tier or the search head tier where ITSI is running, otherwise ITSI notable event grouping stops working.

Virtual machines

When running a search head on a virtual machine, make sure to allocate all available CPU and RAM to the search head.

Forward search head data to indexers

ITSI runs KPI searches on the search head and by default stores data in the local itsi_summary index. It is considered a best practice to forward all internal data from search heads to indexers. There are two basic search head configuration scenarios for forwarding data to indexers:

Search head type	Configuration
Non-clustered search heads	Configure search heads to forward data to indexers.
Clustered search heads	In this scenario, you must configure outputs.conf to forward data from search heads to indexers. Then use the deployer to push the configuration file to cluster members.

For detailed instructions on how to configure search heads to forward data to indexers, see Best practice: Forward search head data to the indexer layer in the Splunk Enterprise Distributed Search manual.

IT Service Intelligence and search head clustering

Search head clusters increase the search load on indexers. Add more indexers or allocate additional CPU cores to the indexers when implementing a search head cluster. See System requirements and other deployment considerations for search head clusters and Search head clustering architecture in the Distributed Search Manual.

Search head scaling considerations for Splunk IT Service Intelligence

Consider the following guidelines when implementing a search head cluster:

Factor	Increase this specification
A large number of concurrent searches	Increase CPU cores Increase RAM
A high number of real-time searches being run A large number of users logging in at the same time	Increase CPU cores
A large number of enabled correlation searches	Increase RAM

For instructions on deploying ITSI in a search head cluster environment, see Install IT Service Intelligence in a search head cluster environment.

If you plan to use ITSI's Event Analytics solution, including Episode Review, you must have a stable search head cluster environment. The search head cluster must be healthy, which means it is not skipping searches and can handle the current load in your production environment.

Indexer clustering support

IT Service Intelligence supports both single site and multisite indexer cluster architectures. See The basics of indexer cluster architecture and Multisite indexer cluster architecture in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.

By default, ITSI writes to a common set of indexes. See Configure indexes in ITSI in this manual. If you plan to run more than one instance of ITSI that searches and writes to the same indexing tier, follow the instructions at Configure multiple ITSI deployments to use the same indexing layer.

For a multisite indexer cluster architecture, do the following:

• Enable summary replication. See Replicated summaries in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.
• Set the ITSI search head to site0 to disable search affinity. See Disable search affinity in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.

If you use indexer clustering, the method you use to deploy apps and configuration files to indexer peers is different. See Manage common configurations across all cluster peers and Manage app deployment across all cluster peers in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.

If you use indexer clustering and also leverage ITSI's Event Analytics functionality, you need to configure your cluster masters and search heads so the Rules Engine can be resilient to indexer cluster rolling restarts and upgrades. For instructions to complete this one-time setup, see Configure the Rules Engine to handle indexer cluster rolling restarts and upgrades.

Real-time search requirements

You can't disable real-time searches on either the indexer tier or the search head tier where ITSI is running, otherwise ITSI notable event grouping stops working. Other ITSI features will also break, including Anomaly Detection and KPI alerting.

ITSI uses an indexed real-time search in place of the default real-time search. Indexed real-time searches allow your real-time searches to run after the events are indexed, which greatly improves indexing performance. You can change this default setting in indexes.conf with no negative effects. For instructions, see Indexed real-time search in the Splunk Enterprise Search Manual.

By default, only users with the admin role can run and save real-time searches. For more information on managing roles and assigning them to users, see Create and manage roles with Splunk Web in the Securing the Splunk Platform manual.

SSL requirements

An SSL configuration is required to run ITSI. SSL must be enabled on the splunkd port, port 8089, in order for certain utilities and scripts to function properly, including the following:

• kvstore_to_json.py
• command_check_for_kvstore_size.py
• disable_enable_itsi.py
• itsi_reset_default_team.py
• Migration and upgrade

A non-SSL environment is not supported. To secure your Splunk Enterprise deployment with SSL, see About securing Splunk Enterprise with SSL.

Performance considerations

ITSI works by way of KPI collection through searches against information stored within the Splunk Enterprise environment. ITSI production deployments might require additional hardware, depending on several factors, including the existing unused capacity of the environment, the number of concurrent KPI searches, the version of Splunk Enterprise in production, and other performance considerations specific to each deployment.

For more information, see Determine when to scale your Splunk Enterprise deployment in the Splunk Enterprise Capacity Planning Manual.

Planning your hardware requirements

ITSI performance depends on the ability to perform multiple fast, concurrent searches. Performance results depend on both search optimization and the capacity of your deployment to run multiple concurrent searches.

When planning your ITSI hardware requirements, consider these ITSI-specific factors that impact performance:

• Average KPI run time
• Frequency of KPIs (1, 5, or 15 minute)
• Number of entities that are being referenced per KPI

Also consider the following Splunk Enterprise factors that might impact performance:

• Average daily index volume. See How indexed data affects Splunk Enterprise performance in the Capacity Planning Manual.
• Number of concurrent users. See How concurrent users affect Splunk Enterprise performance in the Capacity Planning Manual.

ITSI capacity planning

ITSI capacity planning is governed by several variables. The three key variables in determining how many indexers and search heads you need are average KPI run time, the frequency of KPIs (1, 5, or 15 minute), and the number of entities being referenced per KPI. These variables can vary significantly in real-world deployments. Contact your Splunk sales representative for specific ITSI capacity planning recommendations based on your environment.

You must consider several other variables that impact the number of indexers and search heads you need, including the number of cores on those machines, the total amount of data being indexed, and total number of concurrent users.

Indexer and search head sizing examples

The following examples show roughly the number of indexers and search heads required to run the specified number of KPIs. These numbers are for example purposes only and vary based on your environment.

The following variables are fixed for each of the following examples:

• 5-minute KPIs
• 12 cores per search head and indexer
• Environment dedicated to ITSI alone
• Splunk Enterprise version 6.6 or later
• Use of "entity" refers to entities stored in the KV store and in the examples is a per-KPI measure, not the total number of entities in the system. If simple entity splits are done for KPIs and are not based on entities in a KV store, but extracted fields in Splunk searches, they need not be considered entities.
• 1 indexer required per 100 GB indexed

Example Set 1: Average run time per KPI = 10 seconds

Example A: 0 Entities per KPI, 100 GB indexed per day

KPIs	Indexers	Search heads
100	1 indexer	1 search head
500	2 indexers	1 search head
1,000	3 indexers	2 search heads

Rough capacity plan:

~ (Per 500 KPIs 1+ search head, 1+ indexer) + 1 Indexer.

Example B: 50 entities per KPI, 500 GB indexed per day

KPIs	Indexers	Search heads
100	5 indexers	1 search head
500	5 indexers	2 search heads
1,000	5 indexers	3 search heads

Rough capacity plan:

~ (Per 333 KPIs 1+ search head)

Example Set 2: Average run time per KPI = 5 seconds

Example A: 0 entities per KPI, 100 GB indexed per day

KPIs	Indexers	Search heads
100	1 indexer	1 search head
500	1 indexer	1 search head
1,000	2 indexers	2 search heads

Rough capacity plan:

~ (Per 950 KPIs 1+ search head), (Per 730 KPIs 1+ indexer)

Example B: 50 entities per KPI, 500 GB indexed per day

KPIs	Indexers	Search heads
100	5 indexer	1 search head
500	5 indexer	1 search head
1,000	5 indexers	3 search heads

Rough capacity plan:

~ (Per 333 KPIs 1+ search head)

It is important to distinguish between the number of KPIs and the number of KPI searches. When using KPI base searches, these requirements can be dramatically different. It's the number of actual search jobs that matters.

KV store size limits

Splunk IT Service Intelligence requires the KV store to store certain information on the search head. Also, in a dedicated search head environment, KPI data is stored locally. You need a minimum of 30 GB of free storage in $SPLUNK_HOME.

The limit of a single batch save to a KV store collection is 50 MB. As a result, if one KPI base search is in use by multiple services, and the total size of your services exceeds 50 MB, ITSI generates an error. Additionally, if the number of objects, such as services and KPIs, exceeds the KV store memory limits, services might be lost during a backup or migration. To avoid these issues, check the total amount of data that your services contain, and, if necessary, increase the KV store size limits in limits.conf.

1. Use the Backup/Restore UI or the kvstore_to_json.py script to create a backup of your system. For more information, see Back up and restore ITSI KV store data.
2. When the size of itsi_services___service___0.json exceeds max_size_per_batch_save_mb (default: 50 MB), increase the max_size_per_batch_save_mb to 1.5 times the value. Add the following stanza to $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf:

   [kvstore]
   max_size_per_batch_save_mb = [value]
   

3. Increase the value of max_size_per_batch_save_mb to a larger value.
4. Additionally, when the size of itsi_services___service___0.json exceeds max_size_per_result_mb (default: 500 MB), then increase the value of max_size_per_result_mb to 1.5 times the value. Add the following stanza to $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf:

   [kvstore]
   max_size_per_result_mb = [value]
   

5. Increase the value of max_size_per_result_mb to roughly 500 MB per 1,000 KPIs.

Search macros in ITSI

ITSI uses search macros to simplify and consolidate lengthy KPI searches. You can view a complete list of search macros used in ITSI, including macro definitions and usage details in macros.conf. For more information on search macros, see Use search macros in searches in the Splunk Enterprise Knowledge Manager Manual.

HTTP event collector

ITSI uses HTTP Event Collector (HEC) for event management. HEC runs as a separate app called splunk_httpinput and stores its input configuration in $SPLUNK_HOME/etc/apps/splunk_httpinput/local.

HEC requires that port 8088 be open for local traffic.

To ingest notable events, you need to turn on SSL for HEC. Follow these steps to turn on SSL:

1. Select Settings > Data Inputs.
2. Select HTTP Event Collector.
3. Select the Enable SSL checkbox.

For more information about HTTP Event Collector, see Set up and use HTTP Event Collector in Splunk Web in the Splunk Enterprise Getting Data In manual.

ITSI compatibility with other apps

Do not install ITSI and Splunk Enterprise Security on the same search head or search head cluster. With the exception of Enterprise Security, you can deploy ITSI on Splunk Enterprise instances with other Splunk apps.

For ITSI compatibility with all related apps and add-ons, see ITSI compatibility with other apps and add-ons.

For a comprehensive evaluation of your environment, consult Splunk Professional Services or your support representative.