Distributed Deployment Manual

 


Upgrade your distributed environment

NOTE - Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.

Upgrade your distributed environment

This topic discusses the concepts of upgrading components of a distributed Splunk deployment.

Upgrading a distributed Splunk environment presents challenges over upgrading an indexer-only Splunk installation. For the purposes of reducing downtime and ensuring no data is lost, we strongly recommend that you upgrade your Splunk components in a specific order. This order is depicted in the instructions below.

Note: This is a high-level guidance on upgrading Splunk in a distributed environment. We realize that every distributed Splunk environment is different, and therefore do not offer detailed step-by-step procedures. If you have additional questions about upgrading your distributed Splunk environment after reading this topic, you can log a case via the Splunk Support Portal.

Cross-version compatibility between distributed components

For information on compatibility between differerent versions of search heads and search peers (indexers), see "Cross-version compatibility for search heads".

For information on compatibility between indexers and forwarders, see "Indexer and universal forwarder compatibility".

Test your apps prior to the upgrade

Before upgrading your distributed environment, make sure that all of your Splunk apps work on the version of Splunk that you plan to upgrade to.

Important: This procedure is required if you are upgrading a distributed environment with a search head pool, because pooled search heads use shared storage space for apps and configurations.

To ensure that your apps work on the desired upgraded version of Splunk:

1. On a reference machine, install the full version of Splunk that you currently run.

Note: You can also use an existing Splunk instance, provided that it is not indexing relevant data and is at the same version level as the other instances in your environment.

2. Install the apps on this Splunk instance.

3. Confirm that the apps work as expected.

4. Upgrade the Splunk instance to the desired version.

5. Test the apps again to make sure they work as desired in the new version.

If the apps work as expected, you can move them to the appropriate location during the upgrade of your distributed Splunk environment:

  • If you use non-pooled search heads, move the apps to $SPLUNK_HOME/etc/apps on each search head during the search head upgrade process.
  • If you use pooled search heads, move the apps to the shared storage location where the pooled search heads expect to find the apps.

Upgrade a distributed environment with multiple indexers and non-pooled search heads

To maintain availability, Splunk recommends that, when upgrading your distributed Splunk environment with multiple indexers and non-pooled search heads, that you upgrade the search heads first, then upgrade the indexing infrastructure that supports the search heads. If you have deployment servers in the environment, be sure to disable those prior to upgrading your search heads.

To upgrade a distributed Splunk environment with multiple indexers and non-pooled search heads:

Prepare the upgrade

1. Confirm that any apps that the pooled search heads use will work on the upgraded version of Splunk, as described in "Test your apps prior to the upgrade" in this topic.

2. If you use a deployment server in your environment, disable it temporarily. This prevents the server from distributing invalid configurations to your other Splunk components.

3. Upgrade your deployment server, but do not restart it.

Upgrade the search heads

4. Disable and upgrade one of the search heads. Do not allow it to restart.

5. After you upgrade the search head, place the confirmed working apps into the $SPLUNK_HOME/etc/apps directory of the search head.

6. Restart this search head and test for operation and functionality.

7. If there are no problems with the search head, then disable and upgrade the remaining search heads, one by one. Repeat this step until you have reached the last search head in your environment. Optionally, you can test each search head for operation and functionality after you bring it up.

8. Once you have upgraded the last search head, test all of the search heads for operation and functionality.

Upgrade the indexers

9. Disable and upgrade your indexers, one by one. You can restart the indexers immediately after you upgrade them.

10. Test your search heads to ensure that they find data across all your indexers.

11. After all indexers have been upgraded, restart your deployment server.

Upgrade a distributed environment with multiple indexers and pooled search heads

If your distributed Splunk environment has pooled search heads, the process to upgrade the environment becomes significantly more complex. If your organization has restrictions on downtime, this type of upgrade is best done within a maintenance window.

The key concepts to understand about upgrading this kind of environment are:

  • Pooled search heads must be enabled and disabled as a group.
  • The version of Splunk on all pooled search heads must be the same.
  • Apps and configurations that the search heads use must be tested prior to upgrading the search head pool.

If you have additional concerns about the guidance shown here, you can log a case via the Splunk Support Portal.

To upgrade a distributed Splunk environment with multiple indexers and pooled search heads:

Prepare the upgrade

1. Confirm that any apps that the pooled search heads use will work on the upgraded version of Splunk, as described in "Test your apps prior to the upgrade" in this topic.

2. If you use a deployment server in your environment, disable it temporarily. This prevents the server from distributing invalid configurations to your other Splunk components.

3. Upgrade your deployment server, but do not restart it.

Upgrade the search head pool

4. Designate a search head (Search Head #1) in your search head pool to upgrade as a test for functionality and operation.

Note: Search heads must be removed from the pool temporarily to prevent changes to the search head pool shared storage and to trigger migration of local apps and system settings during the upgrade. If problems occur as a result of the upgrade, the search head can be temporarily used in a non-pooled configuration as a backup.

5. Bring down all of the search heads in your environment.

Note: Search capability will be unavailable at this time, and will remain unavailable until you restart all of the search heads after upgrading.

6. Place the confirmed working apps in the search head pool shared storage area.

7. Remove Search Head #1 from the search head pool.

Note: Review "Configure search head pooling" for instructions on how to enable and disable search head pooling on each search head.

8. Upgrade Search Head #1.

9. Restart Search Head #1 and test for operation and functionality.

10. If the upgraded Search Head #1 functions as desired, bring it down and add it back to the search head pool.

11. Upgrade the remaining search heads in the pool, one by one.

Caution: Remove each search head from the search head pool before you upgrade, and add them back to the pool after you upgrade. Do not allow the search heads to restart until you have upgraded them all.

12. Once you have upgraded the last search head in the pool, then restart all of them.

13. Test all search heads for operation and functionality across all of your indexers.

Upgrade the indexers

14. Once you have confirmed that your search heads are functioning as desired, choose an indexer to keep the environment running (Indexer #1), and another to upgrade initially (Indexer #2).

Note: If you do not have downtime concerns, you do not need to perform this step.

15. Bring down all of the indexers except Indexer #1.

Note: If you do not have downtime concerns, you can bring down all of the indexers.

16. Upgrade Indexer #2.

17. Bring up Indexer #2 and test for operation and functionality.

Note: Search heads running the latest version of Splunk can communicate with indexers running earlier versions of Splunk.

18. Once you have confirmed proper operation on Indexer #2, bring down Indexer #1.

19. Upgrade Indexer #1 and all of the remaining indexers, one by one. You can restart the indexers immediately after you upgrade them.

20. Confirm operation and functionality across all of your indexers.

21. Restart your deployment server, and confirm its operation and functionality.

Upgrade forwarders

When upgrading your distributed Splunk environment, you can also upgrade any universal forwarders in that environment. This is not required, however, and you might want to consider whether or not you need to. Forwarders are always compatible with later version indexers, so you do not need to upgrade them just because you've upgraded the indexers they're sending data to.

To upgrade universal forwarders, review the following topics in this manual:

This documentation applies to the following versions of Splunk: 4.2.1 , 4.2.2 , 4.2.3 , 4.2.4 , 4.2.5 View the Article History for its revisions.


You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!