Splunk® Enterprise

Distributed Deployment Manual

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Upgrade your distributed environment

This topic discusses the process of upgrading components of a distributed Splunk deployment.

Upgrading a distributed Splunk environment presents challenges over upgrading an indexer-only Splunk installation. For the purposes of reducing downtime and ensuring no data is lost, we strongly recommend that you upgrade your Splunk components in a specific order. This order is described in the instructions below.

Note: This is a high-level guidance on upgrading Splunk in a distributed environment. We realize that distributed Splunk environments differ, and therefore do not offer detailed step-by-step procedures. If you have additional questions about upgrading your distributed Splunk environment after reading this topic, you can log a case via the Splunk Support Portal.

Cross-version compatibility between distributed components

For information on compatibility between differerent versions of search heads and search peers (indexers), see "Cross-version compatibility for search heads".

For information on compatibility between indexers and forwarders, see "Indexer and universal forwarder compatibility".

Test your apps prior to the upgrade

Before upgrading your distributed environment, make sure that all of your Splunk apps work on the version of Splunk that you plan to upgrade to.

Important: This procedure is required if you are upgrading a distributed environment with a search head pool, because pooled search heads use shared storage space for apps and configurations.

To ensure that your apps work on the desired upgraded version of Splunk:

1. On a reference machine, install the full version of Splunk that you currently run.

Note: You can also use an existing Splunk instance, provided that it is not indexing relevant data and is at the same version level as the other instances in your environment.

2. Install the apps on this Splunk instance.

3. Confirm that the apps work as expected.

4. Upgrade the Splunk instance to the desired version.

5. Test the apps again to make sure they work as desired in the new version.

If the apps work as expected, you can move them to the appropriate location during the upgrade of your distributed Splunk environment:

  • If you use non-pooled search heads, move the apps to $SPLUNK_HOME/etc/apps on each search head during the search head upgrade process.
  • If you use pooled search heads, move the apps to the shared storage location where the pooled search heads expect to find the apps.

Caution: The migration utility warns you of apps that need to be copied to shared storage for pooled search heads when you upgrade them. It does not, however, copy them for you. You must manually copy all updated apps - including apps that ship with Splunk (such as the Search app and the data preview feature, which is implemented as an app) - to shared storage during the upgrade process. Failure to do so can cause problems with Splunk's user interface after the upgrade is complete.

Upgrade a distributed environment with multiple indexers and non-pooled search heads

To maintain availability, Splunk recommends that, when upgrading your distributed Splunk environment with multiple indexers and non-pooled search heads, that you upgrade the search heads first, then upgrade the indexing infrastructure that supports the search heads. If you have deployment servers in the environment, be sure to disable those prior to upgrading your search heads.

To upgrade a distributed Splunk environment with multiple indexers and non-pooled search heads:

Prepare the upgrade

1. Confirm that any apps that the pooled search heads use will work on the upgraded version of Splunk, as described in "Test your apps prior to the upgrade" in this topic.

2. If you use a deployment server in your environment, disable it temporarily. This prevents the server from distributing invalid configurations to your other Splunk components.

3. Upgrade your deployment server, but do not restart it.

Upgrade the search heads

4. Disable and upgrade one of the search heads. Do not allow it to restart.

5. After you upgrade the search head, place the confirmed working apps into the $SPLUNK_HOME/etc/apps directory of the search head.

6. Restart this search head and test for operation and functionality.

7. If there are no problems with the search head, then disable and upgrade the remaining search heads, one by one. Repeat this step until you have reached the last search head in your environment. Optionally, you can test each search head for operation and functionality after you bring it up.

8. Once you have upgraded the last search head, test all of the search heads for operation and functionality.

Upgrade the indexers

9. Disable and upgrade your indexers, one by one. You can restart the indexers immediately after you upgrade them.

10. Test your search heads to ensure that they find data across all your indexers.

11. After all indexers have been upgraded, restart your deployment server.

Upgrade a distributed environment with multiple indexers and pooled search heads

If your distributed Splunk environment has pooled search heads, the process to upgrade the environment becomes significantly more complex. If your organization has restrictions on downtime, this type of upgrade is best done within a maintenance window.

The key concepts to understand about upgrading this kind of environment are:

  • Pooled search heads must be enabled and disabled as a group.
  • The version of Splunk on all pooled search heads must be the same.
  • Apps and configurations that the search heads use must be tested prior to upgrading the search head pool.

If you have additional concerns about the guidance shown here, you can log a case via the Splunk Support Portal.

To upgrade a distributed Splunk environment with multiple indexers and pooled search heads:

Prepare the upgrade

1. Confirm that any apps that the pooled search heads use will work on the upgraded version of Splunk, as described in "Test your apps prior to the upgrade" in this topic.

2. If you use a deployment server in your environment, disable it temporarily. This prevents the server from distributing invalid configurations to your other Splunk components.

3. Upgrade your deployment server, but do not restart it.

Upgrade the search head pool

4. Designate a search head (Search Head #1) in your search head pool to upgrade as a test for functionality and operation.

Note: Search heads must be removed from the search head pool temporarily before you upgrade them. This must be done for several reasons:

  • To prevent changes to the apps and/or user objects hosted on the search head pool shared storage.
  • To stop the inadvertent migration of local apps and system settings to shared storage during the upgrade.
  • To ensure that you have a valid local configuration to use as a fallback, should a problem occur during the upgrade.

If problems occur as a result of the upgrade, search heads can be temporarily used in a non-pooled configuration as a backup.

5. Bring down all of the search heads in your environment.

Note: Search capability will be unavailable at this time, and will remain unavailable until you restart all of the search heads after upgrading.

6. Place the confirmed working apps (as tested in Step 1) in the search head pool shared storage area.

7. Remove Search Head #1 from the search head pool.

Note: Review "Configure search head pooling" for instructions on how to enable and disable search head pooling on each search head.

8. Upgrade Search Head #1.

9. Restart Search Head #1 and test for operation and functionality.

Note: In this case, 'operation and functionality' means that the Splunk instance starts and that you can log into it. It does not mean that you can use apps or objects hosted on shared storage. It also does not mean distributed searches will run correctly.

10. If the upgraded Search Head #1 functions as desired, bring it down and add it back to the search head pool.

11. Upgrade the remaining search heads in the pool, one by one, following Steps 7 through 10 above.

Caution: Remove each search head from the search head pool before you upgrade, and add them back to the pool after you upgrade. While it is not necessary to confirm operation and functionality of each search head, only one search head at a time can be up during the upgrade phase. Do not start any of the other search heads until you have upgraded all of them.

12. Once you have upgraded the last search head in the pool, then restart all of them.

13. Test all search heads for operation and functionality across all of the apps and user objects that are hosted on the search head pool.

14. Test distributed search across all of your indexers.

Upgrade the indexers

15. Once you have confirmed that your search heads are functioning as desired, choose an indexer to keep the environment running (Indexer #1), and another to upgrade initially (Indexer #2).

Note: If you do not have downtime concerns, you do not need to perform this step.

16. Bring down all of the indexers except Indexer #1.

Note: If you do not have downtime concerns, you can bring down all of the indexers.

17. Upgrade Indexer #2.

18. Bring up Indexer #2 and test for operation and functionality.

Note: Search heads running the latest version of Splunk can communicate with indexers running earlier versions of Splunk.

19. Once you have confirmed proper operation on Indexer #2, bring down Indexer #1.

20. Upgrade Indexer #1 and all of the remaining indexers, one by one. You can restart the indexers immediately after you upgrade them.

21. Confirm operation and functionality across all of your indexers.

22. Restart your deployment server, and confirm its operation and functionality.

Upgrade forwarders

When upgrading your distributed Splunk environment, you can also upgrade any universal forwarders in that environment. This is not required, however, and you might want to consider whether or not you need to. Forwarders are always compatible with later version indexers, so you do not need to upgrade them just because you've upgraded the indexers they're sending data to.

To upgrade universal forwarders, review the following topics in this manual:

PREVIOUS
Deploy in multi-tenant environments
  NEXT
Upgrade the Windows universal forwarder

This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7, 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters