ITSI deployment planning

Deploy Splunk IT Service Intelligence on a configured Splunk platform installation. Review the system and hardware requirements and the search head and indexer considerations before deploying IT Service Intelligence.

Preparation for deployment

The first step in planning your ITSI deployment is to evaluate your objectives. This includes determining the numbers and types of services, KPIs, and entities that you want your ITSI deployment to contain. It is also critical to confirm that you have sufficient underlying hardware capacity to support optimal ITSI performance. You must also confirm compatibility with the Splunk Enterprise version on which you plan to deploy ITSI.

In preparation for your ITSI deployment, make sure you have the following information:

1. A list of services, KPIs, and glass table views that you want to create.
2. A list of your entities. Entities are usually hosts, but can also be users, mobile devices, and so on. Entities for hosts should include at minimum IP address, host name, and designated role (for example, web, db, app server).
3. Verify existing hardware performance. Verify performance using this search query:
   

   index=_introspection sourcetype=splunk_resource_usage component=Hostwide earliest=-5m | timechart avg(data.cpu_user_pct) by host

   If it takes more than 2-5 seconds for the search query to complete, check performance in the Job Inspector to investigate the issue. This might indicate your current hardware is insufficient or badly configured. Or you might have a high latency dispatch requiring architecture changes.

4. Confirm Splunk Enterprise version compatibility.

Operating system requirement

For a list of supported operating systems, browsers, and file systems, see System Requirements in the Splunk Enterprise Installation Manual.

Splunk Enterprise system requirement

Splunk IT Service Intelligence requires a 64-bit OS install on all search heads and indexers. For the list of supported operating systems, browsers, and file systems, see System requirements for use of Splunk Enterprise on-premises in the Splunk Enterprise Installation Manual.

Use this table to determine the compatibility of the IT Service Intelligence 4.1.x versions and Splunk platform versions.

ITSI is incompatible with Splunk Enterprise versions 7.2.0 - 7.2.3.

Splunk IT Service Intelligence version	Splunk platform version
4.1.0, 4.1.1, 4.1.2, 4.1.5	7.0.x (except 7.0.5)
	7.0.5, 7.1.x, and 7.2.4 - 7.2.8 (Perform workaround below)

WORKAROUND:

To prevent ITSI Event Analytics from duplicating events on Splunk Enterprise versions 7.0.5, 7.1.x, and 7.2.4 - 7.2.8, create a limits.conf file on all search heads at $SPLUNK_HOME/etc/apps/SA-ITOA/local/ and add the following stanza:

[search]
phased_execution_mode = auto

If you do not plan on using Event Analytics, the workaround is not necessary.

ITSI license requirement

ITSI requires a separate ITSI license in addition to your Splunk Enterprise license. Your Splunk representative will provide you with an appropriate ITSI license at time of purchase. For ITSI license installation instructions, see Install a license in the Splunk Enterprise Installation and Configuration Manual.

Java requirements

ITSI 4.1.x requires Java 7 or Java 8 to run anomaly detection and event management features. Java 8 is required for Windows installations. ITSI supports OpenJDK and Oracle JDK 7 and 8. Java installation is required on search heads only, not indexers or forwarders.

Available deployment architectures

You can deploy Splunk IT Service Intelligence in a single instance deployment or a distributed search deployment. Splunk IT Service Intelligence is also available in Splunk Cloud. Before you deploy Splunk IT Service Intelligence on premises, familiarize yourself with the components of a Splunk platform deployment. See Components of a Splunk Enterprise deployment in the Capacity Planning Manual.

Single instance deployments

For a simple and small deployment, install ITSI on a single Splunk platform instance. A single instance functions as both a search head and an indexer. Use forwarders to collect your data and send it to the single instance for parsing, storing, and searching.

You can use a single instance deployment for a lab or test environment, or a small system with one or two users running concurrent searches. For instructions on installing ITSI on a single Splunk Enterprise instance, see Install ITSI on a single instance.

Distributed deployments

You can deploy ITSI across any distributed architecture supported by Splunk Enterprise. This includes all types of deployment topologies, from small departmental deployments using a single instance for both indexer and search head, to large enterprise deployments using several search heads, dozens of indexers, and hundreds of forwarders. See Types of distributed deployments in the Distributed Deployment Manual.

• For instructions on installing ITSI in a distributed environment, see Install ITSI in a distributed environment.
• Improve search performance by using an index cluster and distributing the workload of searching data across multiple nodes. Using multiple indexers allows both the data collected by the forwarders and the workload of processing the data to be distributed across the indexers.
• Use forwarders to collect your data and send it to the indexers.

In a distributed search deployment, and to implement search head clustering, configure the search head to forward all data to the indexers. See Forward search head data to the indexer layer in the Distributed Search manual.

To properly scale your distributed search deployment with ITSI, see Introduction to capacity planning for Splunk Enterprise in the Capacity Planning Manual and Indexer and search head sizing examples.

Cloud deployments

Splunk IT Service Intelligence is available as a service in Splunk Cloud. The Splunk Cloud deployment architecture varies based on data and search load. Splunk Cloud customers work with Splunk Support to set up, manage, and maintain their cloud infrastructure. For information on Splunk Cloud managed deployments, see the Types of Splunk Cloud deployment in the Splunk Cloud User Manual.

Search head considerations

IT Service Intelligence does not require a dedicated search head. However, note that ITSI is not supported on the same search head as Splunk Enterprise Security. For scalability beyond about 200 discrete KPIs, a search head cluster is recommended.

Do not disable real-time searches on the search head, otherwise notable event grouping in Episode Review will stop working.

Search head hardware requirements

CPU core count and RAM are critical factors in search head performance. ITSI requires minimum hardware specifications that you increase according to your needs and usage of ITSI. These specifications also apply for a single instance deployment of ITSI.

Hardware	Requirement
RAM	12GB required, 16+ recommended
CPU	12 cores required, 16+ recommended

Virtual machines

When running a search head on a virtual machine, make sure to allocate all available CPU and RAM to the search head.

Forward search head data to indexers

ITSI runs KPI searches on the search head and by default stores data in the local itsi_summary index. It is considered a best practice to forward all internal data from search heads to indexers. There are two basic search head configuration scenarios for forwarding data to indexers:

1. Non-clustered search heads: Configure search heads to forward data to indexers.
2. Clustered search heads: In this scenario, you must configure outputs.conf to forward data from search heads to indexers. Then use the deployer to push the configuration file to cluster members.

For detailed instructions on how to configure search heads to forward data to indexers, see Forward search head data to indexers in the Distributed Search manual.

Search head clustering

A search head cluster multiplies the maximum search concurrency of the Splunk environment by the number of search heads in the cluster. To manage the increase in search load when implementing a search head cluster, add additional indexers or allocate additional cores to indexers. For a complete list of requirements, see System requirements and other deployment considerations for search head clusters in the Splunk Enterprise Distributed Search Manual.

For instructions on deploying ITSI in a search head cluster environment, see Install ITSI on a search head cluster in this manual.

If you plan to use ITSI's event analytics solution, including Episode Review, you must have a stable search head cluster environment. The search head cluster must be healthy, which means it is not skipping searches and can handle the current load in your production environment.

Configure multiple ITSI deployments to use the same indexing layer

You can deploy separate non-clustered ITSI search heads for different purposes that forward data to the same indexers. For example, one search head could be used for production and a second search head could be used for testing. You can also deploy separate search head clusters that use the same indexer cluster. In each case, the search heads must be running the same version of ITSI and Splunk.

The data from each ITSI search head or search head cluster can be stored in separate indexes. For example, you could rename the itsi_summary index on the production ITSI search head instance to itsi_summary_prod and rename the itsi_summary index on the test ITSI search head instance to itsi_summary_test, and likewise for the other ITSI indexes. After completing the steps in this procedure, searches will point to the desired index for each separate deployment.

Rename the anomaly_detection index before enabling anomaly detection for KPIs. If the name for the anomaly_detection index is changed after anomaly detection has been enabled for any KPIs, the index will not be updated in anomaly detection searches. If anomaly detection was enabled prior to renaming the anomaly_detection index, disable both trending and cohesive anomaly detection for all the KPIs that use it, then enable it again.

To create custom ITSI indexes for multiple ITSI deployments, do the following:

1. On each search head, change the default ITSI index names to the new index names you want to use for the data from that search head in the following files:
   • itsi/local/alert_actions.conf
   • itsi/local/savedsearches.conf
   • SA-ITOA/local/macros.conf
   • SA-ITOA/local/alert_actions.conf
   • SA-ITOA/local/itsi_rules_engine.properties
2. Create a local version of commands.conf in SA-ITOA/local/commands.conf. Add the following lines to point to the local version of itsi_rules_engine.properties:
   
   

   [itsirulesengine]
   command.arg.3=-DitsiRulesEngine.configurationFile=../local/itsi_rules_engine.properties.
   

3. On each Splunk indexer, add a new index for each renamed ITSI index in Settings > indexes or in SA-IndexCreation/local/indexes.conf.
4. Restart Splunk software to put the changes into effect.
5. On each search head, do the following to check that searches are pointing to the correct indexes:
   • Navigate to Splunk > Data Inputs > HTTP Event Collector. You should see the renamed index names for the five ITSI event management tokens which have the following source types: itsi_notable:event, itsi_notable:archive, itsi_notable:audit, itsi_notable:group.
   • Check the Event Analytics Audit dashboard to make sure the searches run as expected as these searches use macros.
   • Try replacing macro searches with the name of the renamed index. For example:

     `itsi_event_management_index_with_close_events` | stats count AS events

     should return the same events as:

     index="<new name for itsi_tracked_alerts>" | stats count AS events

   • Make sure that the data is displaying as expected in service analyzers, deep dives, glass tables, and Episode Review.
   • Verify that ITSI users can access the new indexes.

Indexer considerations

In a large ITSI deployment, indexers must be able to process thousands of queries per minute. A proper ratio of search heads to indexers can help handle this load. For help determining your indexer requirements, consult your Splunk Professional Services or support representative.

Indexer hardware requirements

CPU core count and RAM are critical factors in indexer performance. ITSI requires minimum hardware specifications that you increase according to your needs and usage of ITSI. These specifications also apply for a single instance deployment of ITSI.

Hardware	Recommendation
RAM	32GB per indexer
CPU	16 cores per indexer with hyperthreading enabled, for a total of 32 threads (32vCPU).

Indexing is an I/O-intensive process. The indexers require sufficient disk I/O to ingest and parse data efficiently while responding to search requests. For the latest IOPS requirements to run Splunk Enterprise, see Reference Hardware: Indexer in the Capacity Planning Manual.

You might need to increase the hardware specifications of your own ITSI deployment above the minimum hardware requirements depending on your environment. Depending on your system configuration, refer to the mid-range or high-performance specifications for Splunk platform reference hardware. See Mid-range specification and High-performance specification in the Capacity Planning Manual.

If the number of indexer CPU cores in your deployment exceeds the minimum hardware specifications, you can implement one of the parallelization settings to improve the indexer performance for specific use cases. See Parallelization settings in the Capacity Planning Manual.

Indexer clustering

IT Service Intelligence supports both single site and multisite indexer cluster architectures. See The basics of indexer cluster architecture and Multisite cluster architecture in Managing Indexers and Clusters of Indexers.

A single site or multisite indexer cluster architecture can have one search head or one search head cluster with a running instance of ITSI. Additional single instance search heads cannot run ITSI unless specific configuration changes are made.

For a multisite indexer cluster architecture, Splunk recommends the following:

• Enable summary replication. See Replicated summaries in Managing Indexers and Clusters of Indexers.
• Set the ITSI search head to site0 to disable search affinity. See Disable search affinity in Managing Indexers and Clusters of Indexers.

If you use indexer clustering, the method you use to deploy apps and configuration files to indexer peers is different. See Manage common configurations across all cluster peers and Manage app deployment across all cluster peers in the Managing Indexers and Clusters of Indexers.

Performance considerations

ITSI works by way of KPI collection via searches against information stored within the Splunk Enterprise environment. ITSI production deployments might require additional hardware, depending on several factors, including the existing unused capacity of the environment, the number of concurrent KPI searches, the version of Splunk Enterprise in production, and other performance considerations specific to each deployment.

To determine when to scale your Splunk Enterprise deployment, see Performance Checklist in the Capacity Planning Manual.

Planning your hardware requirements

ITSI performance depends on the ability to perform multiple fast, concurrent searches. Performance results depend on both search optimization and the capacity of your deployment to run multiple concurrent searches.

When planning your ITSI hardware requirements, consider these ITSI-specific factors that impact performance:

• Average KPI run time
• Frequency of KPIs (1, 5 or 15 minute)
• Number of entities that are being referenced per KPI

Also consider the following Splunk Enterprise factors that might impact performance:

• Average daily index volume. See How indexed data affects Splunk Enterprise performance in the Capacity Planning Manual.
• Number of concurrent users. See How concurrent users affect Splunk Enterprise performance in the Capacity Planning Manual.

ITSI capacity planning

ITSI capacity planning is governed by several variables. The three key variables in determining how many indexers and search heads you need are average KPI run time, the frequency of KPIs (1, 5, or 15 minute), and the number of entities being referenced per KPI. These can vary significantly in real-world deployments and you should contact your Splunk sales representative for specific ITSI capacity planning recommendations based on your environment.

There are several other variables to consider that impact the number of indexers and search heads you need, including the number of cores on those machines, the total amount of data being indexed, the total number of concurrent users, and so on.

Indexer and search head sizing examples

The following examples show roughly the number of indexers and search heads required to run the specified number of KPIs. These numbers are for example purposes only and will vary based on your environment.

The following variables are fixed for each of the proceeding examples.

• Only 5 minute KPIs
• 12 cores per search head and indexer
• Environment dedicated to ITSI alone
• Splunk Enterprise version 6.6 or later
• Use of "entity" refers to entities stored in the KV store and in the examples is a per-KPI measure, not the total number of entities in the system. If simple entity splits are done for KPIs and are not based on entities in a KV store, but extracted fields in Splunk searches, they need not be considered entities.
• 1 indexer required per 100GB indexed

Example Set 1: Average run time per KPI = 10 seconds

Example A: 0 Entities per KPI, 100 GB indexed per day

KPIs	Indexers	Search Heads
100	1 indexer	1 search head
500	2 indexers	1 search head
1000	3 indexers	2 search heads

Rough capacity plan:

~ (Per 500 KPIs 1+ search head, 1+ indexer) + 1 Indexer.

Example B: 50 entities per KPI, 500 GB indexed per day

KPIs	Indexers	Search Heads
100	5 indexers	1 search head
500	5 indexers	2 search heads
1000	5 indexers	3 search heads

Rough capacity plan:

~ (Per 333 KPIs 1+ search head)

Example Set 2: Average run time per KPI = 5 seconds

Example A: 0 entities per KPI, 100 GB indexed per day

KPIs	Indexers	Search Heads
100	1 indexer	1 search head
500	1 indexer	1 search head
1000	2 indexers	2 search heads

Rough capacity plan:

~ (Per 950 KPIs 1+ search head), (Per 730 KPIs 1+ indexer)

Example B: 50 entities per KPI, 500 GB indexed per day

KPIs	Indexers	Search Heads
100	5 indexer	1 search head
500	5 indexer	1 search head
1000	5 indexers	3 search heads

Rough capacity plan:

~ (Per 333 KPIs 1+ search head)

It is important to distinguish between the number of KPIs and the number of KPI searches. When using KPI base searches, these two can be dramatically different, and it is the number of actual search jobs that matters.

KV store size limits

Splunk IT Service Intelligence requires the KV store to store certain information on the search head. Also, in a dedicated search head environment, KPI data is stored locally. You should have a minimum of 30 GB of free storage in $SPLUNK_HOME.

The limit of a single batch save to a KV store collection is 50MB. As a result, if one KPI base search is in use by multiple services, and the total size of your services exceeds 50MB, ITSI generates an error. Additionally, if the number of objects (services, KPIs, etc.), exceeds the KV store memory limits, services might be lost during a backup or migration. To avoid these issues, check the total amount of data that your services contain, and, if necessary, increase the KV store size limit in limits.conf, as shown below:

1. Use the Backup/Restore UI or the kvstore_to_json.py script to create a backup of your system. For more information, see Back up and restore ITSI KV store data.
2. If the size of itsi_services___service___0.json exceeds 50 MB, increase the KV store size limit, as shown in steps 3 and 4.
3. Add the following stanza to /local/limits.conf:

   [kvstore]
   
   # The maximum size, in megabytes, of a batch save query.
   max_size_per_batch_save_mb = 50
   

4. Increase the value of max_size_per_batch_save_mb to a higher value.
5. Additionally, if you have more than 1,000 kpi and services, add the following stanza to /local/limits.conf:

   [kvstore]
   
   # The maximum size, in megabytes, of the result that will be returned for a single query to a collection.
   max_size_per_result_mb = 100
   

6. Increase the value of max_size_per_result_mb to roughtly 50MB per 1,000 KPIs.

Search macros in ITSI

ITSI uses search macros to simplify and consolidate lengthy KPI searches. You can view a complete list of search macros used in ITSI, including macro definitions and usage details in macros.conf. For more information on search macros, see Use search macros in the Knowledge Manager Manual.

HTTP event collector

ITSI uses HTTP Event Collector (HEC) for event management. HEC runs as a separate app called splunk_httpinput and stores its input configuration in $SPLUNK_HOME/etc/apps/splunk_httpinput/local.

HEC requires port 8088 be open for local traffic. No additional HEC configuration is required.

For more information on HTTP Event Collector, see Set up and use HTTP Event Collector in the Splunk Enterprise Getting Data In manual.

ITSI compatibility with other apps

Do not install ITSI and Splunk Enterprise Security on the same search head or search head cluster. With the exception of Enterprise Security, ITSI can be deployed on Splunk Enterprise instances with other Splunk apps.

For a comprehensive evaluation of your environment, consult your Splunk Professional Services or support representative.