Splunk® IT Service Intelligence

Administer Splunk IT Service Intelligence

Download manual as PDF

This documentation does not apply to the most recent version of ITSI. Click here for the latest version.
Download topic as PDF

Deployment planning

Deploy Splunk IT Service Intelligence on a configured Splunk platform installation. Review the system and hardware requirements and the search head and indexer considerations before deploying IT Service Intelligence.

Preparation for deployment

The first step in planning your ITSI deployment is to evaluate your objectives. This includes determining the numbers and types of services, KPIs, and entities that you want your ITSI deployment to contain. It is also critical to confirm that you have sufficient underlying hardware capacity to support optimal ITSI performance. You must also confirm compatibility with the Splunk Enterprise version on which you plan to deploy ITSI.

In preparation for your ITSI deployment, make sure you have the following information:

  1. List of services, KPIs, and glass table views that you want to create.
  2. List of entities. Entities are usually hosts, but can also be users, mobile devices, and so on. Entities for hosts should include at minimum IP address, host name, and designated role (for example, web, db, app server).
  3. Verify existing hardware performance. Verify performance using this search query:

    index=_introspection sourcetype=splunk_resource_usage component=Hostwide earliest=-5m | timechart avg(data.cpu_user_pct) by host

    If it takes more than 2-5 seconds for the search query to complete, check performance in the Job inspector to investigate the issue. This might indicate your current hardware is insufficient or badly configured. Or you may have a high latency dispatch requiring architecture changes.

  4. Confirm Splunk Enterprise version compatibility.

Operating system requirement

For a list of supported operating systems, browsers, and file systems, see System Requirements in the Splunk Enterprise Installation Manual.

Splunk Enterprise version requirement

ITSI 3.1.x requires Splunk Enterprise version 6.6.x - 7.1.x on all search heads and indexers. For the latest information on Splunk Enterprise version requirements, see Splunk Enterprise version compatibility in the Distributed Search Manual.

ITSI license requirement

ITSI requires a separate ITSI license in addition to your Splunk Enterprise license. Your Splunk representative will provide you with an appropriate ITSI license at time of purchase. For ITSI license installation instructions, see Install a license in the Splunk Enterprise Installation and Configuration Manual.

Java requirements

ITSI 3.1.x requires Java 7 or Java 8 to run anomaly detection and event management features. Java 8 is required for Windows installations. ITSI supports OpenJDK and Oracle JDK 7 and 8. Java installation is required on search heads only, not indexers or forwarders.

If you are using Java 7 and running ITSI on Splunk Enterprise version 6.6.0 or later, you must install the Java Cryptography Extension (JCE) to run anomaly detection and event management features. JCE is required on search heads only.

For Java installation instructions, see Install Java requirements in this manual.

Deployment architectures

ITSI supports both cloud and on-premises deployment architectures.

Cloud

Splunk IT Service Intelligence is available as a service on Splunk Cloud. For more information on Splunk Cloud services, see the Splunk Cloud Products page.

Single-instance deployment

You can deploy ITSI on a single Splunk Enterprise instance. In this type of deployment, a single instance of Splunk Enterprise serves as both search head and indexer. A single instance deployment is ideal for a lab or test environment. For instructions on installing ITSI on a single Splunk Enterprise instance, see Install ITSI on a single instance.

Distributed deployment

You can deploy ITSI across any distributed architecture supported by Splunk Enterprise. This includes all types of deployment topologies, from small departmental deployments using a single instance for both indexer and search head, to large enterprise deployments using several search heads, dozens of indexers, and hundreds of forwarders. See Types of distributed deployments in the Distributed Deployment Manual.

For instructions on installing ITSI in a distributed environment, see Install ITSI in a distributed environment.

Clustering options

In a distributed deployment, ITSI supports both search head and indexer clustering options.

For details on search head clustering architecture, see Search head clustering architecture in the Distributed Search Manual. For details on indexer cluster architecture, see The basics of indexer cluster architecture in the Managing Indexers and Clusters of Indexers Manual.

Search head considerations

ITSI does not require a dedicated search head, however, note that ITSI is not supported on the same search head as Splunk Enterprise Security. For scalability beyond ~200 discrete KPIs, a search head cluster is recommended.

Note that real-time searches cannot be disabled on the search head, otherwise ITSI notable event grouping will stop working.

CPU and memory

CPU core count and RAM are critical factors in search head performance. Use the maximum number of CPU cores and RAM available for your system.

Virtual machines

When running a search head on a virtual machine, make sure to allocate all available CPU and RAM to the search head.

Forward search head data to indexers

ITSI runs KPI searches on the search head and by default stores data in the local itsi_summary index. It is considered a best practice to forward all internal data from search heads to indexers. There are two basic search head configuration scenarios for forwarding data to indexers:

  1. Non-clustered search heads: Configure search heads to forward data to indexers.
  2. Clustered search heads: In this scenario, you must configure outputs.conf to forward data from search heads to indexers. Then use the deployer to push the configuration file to cluster members.

For detailed instructions on how to configure search heads to forward data to indexers, see Forward search head data to indexers in the Distributed Search manual.

Non-clustered search heads

Do not deploy separate non-clustered ITSI search heads for different purposes (for example, one search head for production and another search head for testing) that forward data to the same indexer layer. This architecture is not supported as the data from each of the ITSI search heads is stored in the same indexes, which results in duplication of data in health scores, KPIs, and deep dives.

Search head clustering

A search head cluster multiplies the maximum search concurrency of the Splunk environment by the number of search heads in the cluster. To manage the increase in search load when implementing a search head cluster, add additional indexers or allocate additional cores to indexers. For a complete list of requirements, see System requirements and other deployment considerations for search head clusters in the Splunk Enterprise Distributed Search Manual.

For instructions on deploying ITSI in a search head cluster environment, see Install ITSI on a search head cluster in this manual.

Indexer considerations

In a large ITSI deployment, indexers must be able to process thousands of queries per minute. A proper ratio of search heads to indexers can help handle this load. For help determining your indexer requirements, consult your Splunk Professional Services or support representative.

CPU and memory

CPU core count and RAM are critical factors in indexer performance. Make sure to scale up indexer CPU cores as appropriate to handle the number of concurrent searches driven by ITSI. In addition, make sure to install as much RAM as possible on your indexer machines. We recommend 64GB or more of RAM per indexer.

Indexer clustering

ITSI supports both single and multi-site Indexer clusters. For more information, see Indexer cluster deployment overview in Managing Indexers and Clusters of Indexers.

Performance considerations

ITSI works by way of KPI collection via searches against information stored within the Splunk Enterprise environment. ITSI production deployments might require additional hardware, depending on several factors, including the existing unused capacity of the environment, the number of concurrent KPI searches, the version of Splunk Enterprise in production, and other performance considerations specific to each deployment.

To determine when to scale your Splunk Enterprise deployment, see Performance Checklist in the Capacity Planning Manual.

Planning your hardware requirements

ITSI performance depends on the ability to perform multiple fast, concurrent searches. Performance results depend on both search optimization and the capacity of your deployment to run multiple concurrent searches.

When planning your ITSI hardware requirements, consider these ITSI-specific factors that impact performance:

  • Average KPI run time.
  • Frequency of KPIs (1, 5 or 15 minute).
  • Number of entities that are being referenced per KPI

Also consider these Splunk Enterprise factors that might impact performance:

ITSI capacity planning

ITSI capacity planning is governed by several variables. The three key variables in determining how many indexers and search heads you need are average KPI run time, the frequency of KPIs (1, 5, or 15 minute), and the number of entities being referenced per KPI. These can vary significantly in real-world deployments and you should contact your Splunk sales representative for specific ITSI capacity planning recommendations based on your environment.

There are several other variables to consider that impact the number of indexers and search heads you need, including the number of cores on those machines, the total amount of data being indexed, the total number of concurrent users, and so on.

Indexer and search head sizing examples

The following examples show roughly the number of indexers and search heads required to run the specified number of KPIs. These numbers are for example purposes only and will vary based on your environment.

Fixed variables

These variables are fixed for each of the proceeding examples.

  • Only 5 minute KPIs
  • 12 Cores per search head and indexer
  • Environment dedicated to ITSI alone.
  • Splunk Enterprise version 6.6 or later.
  • Use of Entity refers to entities stored in the KV store and in the examples is a per KPI measure, not the total number of entities in the system. If simple entity split-bys are done for KPIs and are not based on entities in a KV store, but extracted fields in splunk searches, they need not be considered entities.
  • 1 Indexer required per 100GB indexed.

Example Set 1. Average Run Time per KPI = 10 seconds

Example A: 0 Entities per KPI, 100 GB indexed a day.

KPIs Indexers Search Heads
100 1 indexer 1 search head
500 2 indexers 1 search head
1000 3 indexers 2 search heads

Rough capacity plan:

~ (Per 500 KPIs 1+ search head, 1+ indexer) + 1 Indexer.


Example B: 50 Entities per KPI, 500 GB indexed a day.

KPIs Indexers Search Heads
100 5 indexers 1 search head
500 5 indexers 2 search heads
1000 5 indexers 3 search heads

Rough capacity plan:

~ (Per 333 KPIs 1+ search head)


Example Set 2. Average Run Time per KPI = 5 seconds

Example A: 0 Entities per KPI, 100 GB indexed per day.

KPIs Indexers Search Heads
100 1 indexer 1 search head
500 1 indexer 1 search head
1000 2 indexers 2 search heads

Rough capacity plan:

~ (Per 950 KPIs 1+ search head), (Per 730 KPIs 1+ indexer)


Example B: 50 Entities per KPI, 500 GB indexed per day.

KPIs Indexers Search Heads
100 5 indexer 1 search head
500 5 indexer 1 search head
1000 5 indexers 3 search heads

Rough capacity plan:

~ (Per 333 KPIs 1+ search head)


It is important to distinguish between the number of KPIs and the number of KPI searches. When using KPI base searches, these two can be dramatically different, and it is the number of actual search jobs that matters.

About ITSI modules

The following ITSI modules are installed as part of the Splunk IT Service Intelligence package:

Other modules are available for individual download from Splunkbase.

ITSI modules provide KPIs that are built on custom data models, which are optimized for ITSI use cases and for performance. For best results, we encourage using KPIs provided by ITSI modules. For more information on ITSI modules, see About ITSI modules in the Splunk IT Service Intelligence Modules manual.

All modules, whether included or downloaded and installed separately, require relevant data to be indexed before you can create services based on the KPIs included in the modules.

See the documentation for each module that you want to use for links to the supported add-ons that are relevant for the environment you are monitoring with your ITSI deployment. The ITSI admin provides data to ITSI by installing and configuring relevant Splunk add-ons.

The data models that come with ITSI modules are separate from the data models provided by SA-CIM. While you can safely use a small number of SA-CIM data models (20 or fewer) in a small POC test environment, in general, CIM-based data model performance with ITSI is insufficient and is not recommended in production environments. Data model acceleration does not improve CIM data model performance in ITSI and should not be used. For best performance use KPI base searches.

KV store

Splunk ITSI uses the KV Store. You can backup and restore ITSI KV store data using the Backup/Restore UI, or using the kvstore_to_json python script. For more information, see Backup and restore ITSI config in this manual.

KV store size limits

The limit of a single batch save to a KV store collection is 50MB. As a result, if one KPI base search is in use by multiple services, and the total size of your services exceeds 50MB, ITSI generates an error. Additionally, if the number of objects (services, KPIs, etc.), exceeds the KV store memory limits, services might be lost during a backup or migration. To avoid these issues, check the total amount of data that your services contain, and, if necessary, increase the KV store size limit in limits.conf, as shown below:

  1. Use the Backup/Restore UI or the kvstore_to_json.py script to create a backup of your system.
  2. If the size of itsi_services___service___0.json exceeds 50 MB, increase the KV store size limit, as shown in steps 3 and 4.
  3. Add the following stanza to /local/limits.conf:
    [kvstore]
    
    # The maximum size, in megabytes (MB), of a batch save query.
    max_size_per_batch_save_mb = 50
    
  4. Increase the value of max_size_per_batch_save_mb to a higher value.
  5. Additionally, if you have more than 1,000 kpi and services, add the following stanza to /local/limits.conf:
    [kvstore]
    
    # The maximum size, in megabytes (MB), of the result that will be returned for a single query to a collection.
    max_size_per_result_mb = 100
    
  6. Increase the value of max_size_per_result_mb to roughtly 50MB per 1,000 KPIs.

Time zone handling

ITSI version 2.5.0 and later honors the time zone setting of the logged-in user. When a user loads a page, that page appears in the time zone in which the user is physically located. For example, if you deploy ITSI on a server configured to Pacific Standard Time (PST), when the current time is 12:00PM PST, a user in Eastern Standard Time (EST) sees 3:00PM EST on all ITSI pages.

Time zone handling in ITSI is consistent with time zone handling in Splunk Enterprise. See Specify time zones for timestamps in the Splunk Enterprise Getting Data In manual.

In ITSI version 2.5.0 and later all time-sensitive configurations are normalized to UTC. Upon migration to version 2.5.0 or later, ITSI updates all time-specific fields in existing ITSI object configurations to UTC. See the ITSI REST API schema in the ITSI REST API Manual. For information on ITSI migration, see Upgrade ITSI in this manual.

ITSI provides a kvstore_to_json.py time zone offset option, which lets you correct for time zone discrepancies in ITSI maintenance_calendar and service objects. For detailed instructions, see Time_zone offset operations (mode 3) in this manual.

Search macros in ITSI

ITSI uses search macros to simplify and consolidate lengthy KPI searches. You can view a complete list of search macros used in ITSI, including macro definitions and usage details in $SPLUNK_HOME/etc/apps/SA-ITOA/default/macros.conf. For more information on search macros, see Use search macros in the Knowledge Manager Manual.

HTTP event collector

ITSI uses HTTP Event Collector (HEC) for event management. HEC runs as a separate app called splunk_httpinput and stores its input configuration in $SPLUNK_HOME/etc/apps/splunk_httpinput/local.

HEC requires port 8088 be open for local traffic. No additional HEC configuration is required.

For more information on HTTP Event Collector, see Set up and use HTTP Event Collector in the Splunk Enterprise Getting Data In manual.

ITSI compatibility with other apps

Do not install ITSI and Splunk Enterprise Security on the same search head or search head cluster. With the exception of Enterprise Security, ITSI can be deployed on Splunk Enterprise instances with other Splunk apps.

For a comprehensive evaluation of your environment, consult your Splunk Professional Services or support representative.

PREVIOUS
ITSI concepts and features
  NEXT
Install Splunk ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 3.1.0, 3.1.1, 3.1.2, 3.1.3, 3.1.4


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters