How to upgrade a distributed Splunk Enterprise environment

Distributed Splunk Enterprise environments vary widely. Some have multiple indexers or search heads, and others have indexer- and search-head clusters. These types of environments present challenges over upgrading single-instance installations.

Determine the upgrade procedure to follow for your type of environment

Depending on the kind of distributed environment you have, you might have to follow separate instructions to complete the upgrade. This topic provides guidance on how to upgrade distributed environments that do not have any clustered elements like index- or search-head clusters. Environments with clustered elements, such as indexer clusters and search head clusters, have different upgrade procedures in different topics. Search head pooling has been removed in version 8.0 of Splunk Enterprise, so there are no upgrade instructions for that type of distributed deployment.

• To upgrade a distributed environment that does not have any clustered elements, follow the procedures in this topic.
• To upgrade an environment with index clusters, see Upgrade an indexer cluster in Managing Indexers and Clusters of Indexers.
• To upgrade an environment with search head clusters, see Upgrade a search head cluster in Distributed Search.
• If you have additional questions about upgrading your distributed Splunk Enterprise environment, log a case at the Splunk Support Portal.

Cross-version compatibility between distributed components

While there is some range in compatibility between various Splunk software components, they work best when they are all at a specific version. If you have to upgrade one or more components of a distributed deployment, you should confirm that the components you upgrade remain compatible with the components that you don't.

• For information on compatibility between differerent versions of search heads and search peers (indexers), see System requirements and other deployment considerations for distributed search in Distributed Search.
• For information on compatibility between indexers and forwarders, see Compatibility between forwarders and indexers in Forwarding Data.

Test apps prior to the upgrade

Before you upgrade a distributed environment, confirm that Splunk apps work on the version of Splunk Enterprise that you want to upgrade to.

1. On a reference machine, install the full version of Splunk Enterprise that you currently run.
2. Install the apps on this instance.
3. Access the apps to confirm that they work as you expect.
4. Upgrade the instance.
5. Access the apps again to confirm that they still work.

If the apps work as you expect, move them to $SPLUNK_HOME/etc/apps on each search head during the search head upgrade process.

Upgrade a distributed environment with multiple indexers and non-pooled search heads

This procedure upgrades the search head tier, then the indexing tier, to maintain availability.

Prepare the upgrade

1. Confirm that any apps that the non-pooled search heads use will work on the upgraded version of Splunk, as described in "Test your apps prior to the upgrade" in this topic.
2. (Optional) If you use a deployment server in your environment, disable it temporarily. This prevents the server from distributing invalid configurations to your other components.
3. (Optional) Upgrade the deployment server, but do not restart it.

Upgrade the search heads

1. Stop Splunk Enterprise services on one of the search heads.
2. Upgrade the search head. Do not let it restart.
3. After you upgrade the search head, place the confirmed working apps into the $SPLUNK_HOME/etc/apps directory of the search head.
4. Re-enable and restart the search head.
5. Test apps on the search head for operation and functionality.
6. If there are no problems with the search head, then disable and upgrade the remaining search heads, one by one. Repeat this step until you have reached the last search head in your environment.
7. (Optional) Test each search head for operation and functionality after you bring it up.
8. After you upgrade the last search head, test all of the search heads for operation and functionality.

Upgrade the indexers

1. Stop Splunk Enterprise services, and upgrade the indexers, one by one. You can restart the indexers immediately after you upgrade them.
2. Test search heads to ensure that they find data across all indexers.
3. After you upgrade all indexers, restart your deployment server.

Upgrade forwarders

After your distributed environment upgrade, review the forwarder versions used in your environment and check for feature compatibility and support. See Compatibility between forwarders and Splunk Enterprise indexers in the Universal Forwarder Manual.

To upgrade universal forwarders, see the following topics in the Forwarder Manual:

• Upgrade the Windows universal forwarder
• Upgrade the universal forwarder for *nix systems