Install IT Service Intelligence in a search head cluster environment
Splunk IT Service Intelligence (ITSI) has specific requirements and processes for implementing search head clustering.
See the following pages for more information about search head clustering:
- For an overview of search head clustering, see Search head clustering architecture in the Splunk Enterprise Distributed Search manual.
- For a complete list of search head clustering requirements, see System requirements and other deployment considerations for search head clusters in the Splunk Enterprise Distributed Search manual.
What the search head cluster environment looks like
Here is a diagram of a small search head cluster, consisting of three members:
This diagram shows the key cluster-related components and interactions:
- One member serves as the captain, directing various activities within the cluster.
- The members communicate among themselves to schedule jobs, replicate artifacts, update configurations, and coordinate other activities within the cluster.
- The members communicate with search peers to fulfill search requests.
- Users can optionally access the search heads through a third-party load balancer.
- A deployer sits outside the cluster and distributes updates to the cluster members.
Note: This diagram is a highly simplified representation of a set of complex interactions between components. For example, each cluster member sends search requests directly to the set of search peers. On the other hand, only the captain sends the knowledge bundle to the search peers. Similarly, the diagram does not attempt to illustrate the messaging that occurs between cluster members. Read the text of this topic for the details of all these interactions.
Where to install ITSI and other dependencies
The following table describes the required locations for installing ITSI and other dependencies in your search head cluster environment.*Install the add-on on a heavy forwarder only when you deploy a heavy forwarder for AWS data collection.
|Component||Search heads||Indexers||Heavy forwarder||Description|
|Splunk IT Service Intelligence||Required||Required*||
You must install ITSI on each search head cluster node.
|Splunk Add-on for Amazon Web Services||Required||You must install the add-on if you are collecting data from AWS. Version 5.0.0 is supported.|
|HTTP Event Collector||Required||You must install the HTTP Event collector if you are collecting metrics from a *nix host. Collectd, which collects metrics data from *nix hosts, sends data to a HEC.|
|TCP input||Required||If you are collecting *nix and Windows logs and Windows metrics, configure a TCP input. You need to configure a port to receive data from a universal forwarder.|
Prerequisites for installing ITSI in a search head cluster environment
ITSI supports installation on Linux-based search head clusters only. ITSI does not support installation on Windows search head clusters.
Before installing ITSI in a search head cluster environment, verify that you have the following:
- One deployer
- The same version of Splunk Enterprise on the deployer and search head cluster nodes
- The same app versions, not including ITSI, on the deployer and search head cluster nodes
- The backup of
etc/shcluster/appson the deployer before installing ITSI
- The backup of
etc/appsfrom one of the search head cluster nodes
- The backup of the KV store from one of search head cluster nodes
Follow these steps to set up ITSI in a search head cluster environment.
If you are installing ITSI in an existing search head cluster environment that might have other apps deployed already, you must follow all of the steps in this section. Be careful to not delete or remove any existing content in the
1. Install ITSI in a search head cluster environment
To install ITSI on a search head cluster, perform the following steps:
- Log in to splunk.com with your credentials.
- Download the latest version of IT Service Intelligence from Splunkbase. See the Splunk IT Service Intelligence product page.
- On the deployer, extract the ITSI installation package into
$SPLUNK_HOME/etc/shcluster/apps. For example:
tar -xvf splunk-it-service-intelligence_<latest_version>.spl -C $SPLUNK_HOME/etc/shcluster/apps
- From the deployer, run the following command to deploy ITSI to the cluster members:
splunk apply shcluster-bundle
- At the end of the bundle push, a rolling restart occurs if necessary. During a rolling restart, approximately 10% of the members restart at a time, until all have restarted. See Restart the search head cluster.
Note: During a rolling restart, all members, including the captain, restart. Restart of the captain triggers the election process, which can result in a new captain. After the final member restarts, the cluster requires approximately 60 seconds to stabilize. During this interval, error messages might appear. You can ignore these messages. They should desist after 60 seconds.
Control the restart process
You should usually let the cluster automatically trigger any rolling restart, as necessary. However, if you need to maintain control over the restart process, you can run a version of
splunk apply shcluster-bundle that stops short of the restart. If you do so, you must later initiate the restart yourself. The configuration bundle changes will not take effect until the members restart.
splunk apply shcluster-bundle without triggering a restart, use this version of the command:
splunk apply shcluster-bundle -action stage && splunk apply shcluster-bundle -action send
The members will receive the bundle, but they will not restart. Splunk Web will display the message "Splunk must be restarted for changes to take effect."
To initiate a rolling restart later, invoke the
splunk rolling-restart command from the captain:
splunk rolling-restart shcluster-members
2. Install required Java components
Using 32-bit JRE/JDK on ITSI versions 4.3.x or later might cause the Rules Engine to fail with unclear errors in the search.log. If this occurs, perform the workaround described in ITSI-4663.
IT Service Intelligence requires Java 8.x - 11.x to run anomaly detection and notable event management features. You can install Java prior to or after installing ITSI but before you start running ITSI.
Install Java on all search heads running ITSI. On RHEL and Ubuntu Linux, you can install the vendor packages
java-1.8.0-openjdk on RHEL Linux and
openjdk-8-jdk on Ubuntu Linux. Alternatively, you can download and install the latest version of Oracle Java 8-11 (JRE or JDK).
JAVA_HOME environment variable is set correctly to the base of the Java installation, or the
java executable (or
java.exe in Windows) can be found using the
PATH environment variable, no additional action is required. This is typically the case if you install the vendor Java packages in Linux or OS X.
If you install Java to a custom location, for example, when you install Oracle Java directly from Oracle's website, and neither
JAVA_HOME is set to the Java installation, you must add the bin bath of the JDK in
$HOME/.bashrc. Perform the following steps:
- Change to your home directory.
- Open the .bashrc file.
- Add the following line to the file. Replace the JDK directory with the name of your java installation directory.
export PATH=/usr/java/<JDK Directory>/bin:$PATH
- Save the file and exit.
- Use the source command to force Linux to reload the .bashrc file which normally is read only when you log in each time.
If you want to set the
PATH for all users, you need to log in as root in the bash shell and perform the above steps on the .profile file in the etc directory and not the .bashrc file in the home directory.
3. Configure indexers and license masters
The ITSI installation package places all ITSI directories in
$SPLUNK_HOME/etc/apps. Perform the following steps to set up indexers and license masters:
$SPLUNK_HOME/etc/apps/on all individual indexers in your environment.
SA-UserAccesson all license masters in your cluster. If a search head in your environment is also a license master, the license master components are installed when you install ITSI on the search heads.
4. Configure search heads and cluster members to forward data to indexers
In a search head cluster environment, configure search heads to forward data. ITSI runs KPI searches on search heads and, by default, stores data in the local
itsi_summary index. For more information, see Best practice: Forward search head data to the indexer layer in the Splunk Enterprise Distributed Search manual.
5. Restart the license master
After deploying ITSI, if this is the first ITSI license that you are installing on the license master, you must restart your instance.
6. (Optional) Migrate an existing search head to a search head cluster
You cannot add a standalone ITSI search head or search head pool member to a search head cluster. To migrate ITSI configurations to a search head cluster, perform the following steps:
- Identify any custom configurations and modifications in the prior ITSI installation. Check to make sure there is no local copy of itsi_settings.conf that might conflict with the default file when you deploy ITSI to the cluster.
- Configure and start a search head cluster. For more information, see Deploy a search head cluster in the Splunk Enterprise Distributed Search manual.
- Deploy the latest version of ITSI on the search head cluster.
- Review and migrate the customized configurations to the search head cluster deployer for replication to the cluster members.
- Shut down the old ITSI search head.
For more information, see the topic Migrate settings from a standalone search head to a search head cluster in the Splunk Enterprise Distributed Search manual.
For assistance in planning a Splunk ITSI deployment migration, contact Splunk Services.
7. Configure data collection
You can collect data from Linux, Mac OS X, and Windows hosts, Kubernetes and OpenShift clusters, Docker containers, and VMware vCenter Servers. If you installed and configured the Splunk Add-on for Amazon Web Services on a heavy forwarder, you can also collect data from your AWS accounts. For more information, see Overview of entity integrations in ITSI.
Where to install IT Service Intelligence in a distributed environment
Configure indexes in ITSI
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.9.0, 4.9.1, 4.9.2, 4.9.3, 4.10.0 Cloud only, 4.10.1 Cloud only, 4.10.2