About upgrading to 9.2 READ THIS FIRST

This topic describes changes in behavior to note for Splunk Enterprise and Splunk Universal Forwarder when you upgrade to version 9.2.

Lower or higher versions of this topic might present different information than the information for your target version. Use the Version drop-down list to choose the product version to which you're upgrading.

Administrators and advanced users can review changes to the .conf and .conf.spec files by using the Compare feature on different branches of the files in the jewnix/splunk-spec-files GitHub repository. Splunk thanks David Twersky (jewnix) for providing and maintaining this repository, and links to its contents with his permission.

Splunk App and Add-on Compatibility

Not all Splunk apps and add-ons are compatible with Splunk Enterprise version 9.2.

• See the Splunk products version compatibility matrix for information about which versions of Splunk IT Service Intelligence and Splunk Enterprise Security are compatible with this version of Splunk Enterprise.
• You can visit Splunkbase to confirm that your apps and add-ons are compatible with this version.

If your app or add-on is not compatible with version 9.2, consider delaying your upgrade until a compatible version is available.

Key points for upgrading to version 9.2

The following is a list of important items that you must consider before you upgrade Splunk Enterprise and its components. The sections that follow provide supporting details for these key points. Read through all sections in the topic before you begin your upgrade activities.

• In Splunk Enterprise 9.2, usage of older jQuery libraries is restricted by default. Admins can still enable the use of older jQuery libraries in a self-serviceable manner. See Older jQuery libraries are restricted by default.
• Use the Splunk Products version compatibility matrix to ensure that any premium Splunk apps and add-ons you run are compatible with version 9.2. If they are not, do not upgrade until compatible versions become available.
• Splunk supports a direct upgrade to Splunk Enterprise 9.2 from versions 9.0.x and higher only. Splunk supports a direct upgrade of a universal forwarder to version 9.2 from versions 8.2.x and higher of the universal forwarder only.
• To upgrade search head or indexer clusters, see Follow specific instructions to upgrade clusters later in this topic.
• Back up your App Key Value Store (KV Store) databases prior to starting an upgrade.
• If you run Linux machines that use the second extended (ext2) file system, upgrade that file system to third extended (ext3) prior to starting an upgrade.

Changes that can potentially break Splunk Enterprise installations

There is no supported method for rolling back an installation to a prior release. Follow the guidance for these critical items to avoid breaking your existing installation during an upgrade.

The Windows Event Log data input has changes in how it expects incoming events from Windows Event Collector subscriptions

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.1

The Windows Event Log input now has a setting that lets you specify the format of incoming events from a Windows Event Collector (WEC) subscription. Previously, the input chose the event format based on the name of the destination log for the WEC subscription. Now, you can specify the format that the Splunk platform is to expect when it receives events through the subscription.

The name of the destination log for the WEC subscription determines the default format that the Splunk platform expects for incoming events from a subscription. For subscriptions whose destination log names begin with the string ForwardedEvents, the event format is rendered_event, or "RenderedText" in Microsoft parlance. For all other destination log names, the event format is raw_event, or "Events" in MS parlance.

If you have WEC subscriptions whose destination log names do not begin with ForwardedEvents but want to use the RenderedText format for that subscription, set wec_event_format to a value of rendered_text in the inputs.conf configuration file stanza for the subscription whose event format you want to change. See the following example:

[WinEventLog://WEC-Channel1]
wec_event_format = rendered_event

For more information about the setting, see Monitor Windows Event Log data in the Getting Data In Manual.

Splunk Enterprise 9.1 fixes a critical vulnerability in deployment server but might introduce problems for older deployment clients

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

Splunk introduced a fix for a vulnerability in deployment server in version 9.0 of Splunk Enterprise, and this fix is also available in version 9.1. If you run a deployment server, upgrade that server to version 9.1 of Splunk Enterprise as soon as possible. Before the upgrade, carefully review your deployment server setup and the current versions of the deployment clients in your Splunk Enterprise network. Depending on the setup of your deployment server and whether that component shares a computer with other Splunk Enterprise components, you might need to do the following to ensure your deployment server and clients communicate without problems:

• Isolate deployment server from other components on a machine. Isolating your deployment server means you only have to upgrade that component. The sole exception for isolation is if you run a deployment server and a license manager on the same machine.
• Confirm that all deployment clients in your network run version 7.0.0 or higher of Splunk Enterprise or the universal forwarder. You don't have to upgrade deployment clients to version 9.0.0, but they must be at version 7.0.0 or higher to communicate with version 9.0.0 deployment servers.

See the following topics for additional information:

• SVD-2022-0608 for more about the vulnerability.
• Security updates in the Securing Splunk Enterprise Manual for more about the security updates that come with Splunk Enterprise 9.0 and higher.
• Client version compatibility in the Updating Splunk Enterprise Instances Manual for more about which versions of deployment client that the version 9.0.0 deployment server supports.

The Linux universal forwarder installer creates a new non-root operating system user called 'splunkfwd'

Applicable components: Universal forwarder
Applicable OSes: Linux
Version introduced: 9.0

If you install version 9.1 or higher of the Linux universal forwarder, the installer now creates the 'splunkfwd' operating system user as a non-root user for managing your universal forwarder. In previous versions, the default user is 'splunk'. The 'splunkfwd' user is a new "least privileged" user that provides only the capabilities necessary to run the universal forwarder, which improves security. To learn more about how to work with the 'splunkfwd' user, see Manage a Linux least-privileged user in the Universal Forwarder manual.

Follow specific instructions to upgrade clusters

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 7.3

To upgrade indexer or search head clusters, follow the upgrade procedure for the type of deployment you have.

• If your deployment has indexer clusters, follow the index cluster upgrade instructions.
• If your deployment has search head clusters, follow the search head cluster upgrade instructions.

On Splunk Enterprise indexer cluster manager nodes, enable maintenance mode before you upgrade

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.1

Because of changes to bucket data with the release of the latest version of Metrics Store, indexers that run versions of Splunk Enterprise lower than 8.1 cannot handle bucket replications from versions that run 8.1 and higher. Before you upgrade a cluster manager node, ensure it is in maintenance mode first. This is a standard part of the indexer cluster update process.

See Use maintenance mode in Managing Indexers and Clusters of Indexers.

Back up App Key Value Store prior to starting an upgrade

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 7.3

Back up the app key value store (KV store) before any maintenance like an upgrade.

Confirm that you have accounted for this downtime in your upgrade planning. See Back up and restore KV store for more information.

The indexer cluster slave-apps directory is renamed to peer-apps

Applicable components: Splunk Enterprise
Applicable OSes: all

During upgrade of a Splunk Enterprise indexer cluster, the installer renames the slave-apps directory on an indexer cluster node to peer-apps. To accommodate this renaming, you must manually change any external hard-coded references to slave-apps, such as in external scripts, TLS certificates, and so on, to peer-apps. See Update common peer configurations and apps in Managing Indexers and Clusters of Indexers.

The setting master_uri is renamed to manager_uri

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

In the [License] stanza of the server.conf configuration file, the master_uri setting has been renamed to manager_uri. The old setting name is deprecated but still usable in 9.2. During an upgrade to 9.x, if you have a mixed deployment of 9.x and 8.x instances, do not change master_uri to manager_uri in the 8.x instances. Version 8.x instances do not recognize the manager_uri setting name.

Occurrences that appear to be problems but are not

You might see things happen during or immediately after an upgrade that appear to indicate that the upgrade is not working. In nearly all cases, the occurrences that happen here can be expected.

If the following things occur during the upgrade, let the upgrade continue and do not interrupt it. If they occur immediately afterward, then perform benchmark tests on the deployment and compare to any benchmarks that you set up as part of the first phase of upgrading. Consider involving Splunk Support only if those benchmarks differ by a significant margin.

• On indexers, memory and CPU usage increases due to the following:
  • New data ingestion pipelines
• Permissions on the Splunk Enterprise introspection directory might change. Confirm that the user that runs Splunk Enterprise has write permission to the $SPLUNK_HOME/var/log/introspection directory.
• Deployment servers might push updates to all deployment clients due to app bundle hash recalculations.

Changes to deployment server architecture

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 9.2

See Upgrade pre-9.2 deployment servers in Updating Splunk Enterprise Instances for important details related to these changes.

Significant aspects of the deployment server have been redesigned in Splunk Enterprise version 9.2 to improve performance and manageability. Because of these architectural improvements, deployment server upgrades that span the version 9.2 release automatically undergo changes to implement these improvements.

Considerations for changed or removed features

The following major features have been changed or removed from this version. If you use features that have been removed, and have not yet migrated off them, consider delaying your upgrade until you have.

Changes to login_content setting in web.conf

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.1.5, 9.2.2, 9.3.0

For security purposes, the login_content setting in web.conf no longer executes scripts.

A new capability has been added that lets you edit passwords stored within an app

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.1

A new capability has been added that lets you edit credentials that have been stored within a Splunk app. This capability, edit_storage_passwords, can be assigned to a role to grant the ability to make changes to passwords stored within the context of an app. Previously, you needed the admin_all_objects capability to make these kinds of changes. While the admin_all_objects capability still lets you modify stored passwords in an app, it is no longer a requirement and lets you further secure your environment by reducing the number of roles that need the superuser-level capability.

After you upgrade, review any cloned roles that include the admin_all_objectscapability in the context of creating and updating passwords in an app. Consider replacing that capability with the new edit_storage_passwords capability instead.for these roles. You can also use the REST API to modify the access control list of each app to apply role restrictions to the apps' password storage configuration, including but not limited to restricting the ability to change passwords within an app to a specific role.

After you perform these adjustments, users who hold roles that contain the edit_storage_passwords capability can only modify credentials in apps to which their roles specifically grant them access. For example, if a user holds a role that grants them access to writing passwords in app1, they can't write passwords in app2 unless they hold a role that contains either or both of the edit_storage_passwords or admin_all_objects capabilities. The admin_all_objects capability overrides the edit_storage_passwords capability and still gives users the ability to see and change storage passwords on any Splunk app.

For more information about app access control endpoints (ACLs) and the REST API, see Access endpoint desecriptions in the REST API Reference Manual.

Older jQuery libraries are restricted by default

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.1

Usage of older jQuery libraries is restricted by default, but admins can still enable the use of older jQuery libraries in a self-serviceable manner. Select Settings, then Server Settings. Then, from the Internal Library Settings page, toggle the jQuery Libraries older than 3.5 section.

To learn more about enabling and restricting access to older jQuery libraries, see Control access to jQuery and other internal libraries.

Index buckets use zstd compression by default

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The indexes.conf configuration file setting journalCompression has a default of "zstd" beginning with this release.

The default tsidxWritingLevel for indexes has changed

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The default tsidxWritingLevel in indexes.conf has changed from 2 to 3, beginning with this release.

Splunk Enterprise 7.2 introduced a new file format and optimizations for tsidx files that resulted in improved search performance through decreased I/O, lowered storage usage, and improved utilization of SmartStore caches. These optimizations are encapsulated in levels, with new levels added in higher releases of Splunk Enterprise. Changing the default tsidxWritingLevel changes the optimizations used by both the index tsidx files and data model accelerations. See The tsidx writing level in the Managing Indexers and Clusters of Indexers manual.

The IOWait status check is delayed on startup

Applicable components: Splunk Enterprise
Applicable OSes: Linux
Version introduced: 9.0

The IOWait Health Report incorporates a 120 second delay after the Splunk Enterprise service starts to prevent false alerts.

The Bucket Health alert threshold is more aggressive

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The threshold values for the Health Report "Bucket" feature are set to alert more aggressively when an indexer rolls a large number of small buckets.

The web.conf setting 'simplexml_dashboard_create_version' is deprecated

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The simplexml_dashboard_create_version setting in the web.conf configuration file has been deprecated. Changing the default value might introduce security risks. Do not change the value without consulting Splunk Support.

Considerations for new features

Splunk has introduced the following new features in this version of Splunk Enterprise. You might need to perform some configuration after an upgrade to enable and take advantage of these features.

Macros now replicate by default to search peers

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.1

Macros used in apps are now replicated by default to search peers as part of the knowledge bundle in Splunk deployments. As a result of this change, searches that previously failed now run successfully, which could affect downstream performance.

If you don't want to replicate macros for your apps, you can suppress replication by setting replicate.macros = false in the [replicationSettings:refineConf] stanza in the distsearch.conf file. Be aware that disabling distribution of macros might negatively impact your search results.

Learn about known upgrade issues

To learn about any additional upgrade issues for Splunk Enterprise, see the Known Issues - Upgrade Issues page in the Release Notes.