About upgrading to 9.1 READ THIS FIRST

This topic describes changes in behavior to note for Splunk Enterprise and Splunk Universal Forwarder when you upgrade to version 9.1. Use the Version drop-down list to choose the product version to which you're upgrading. Lower or higher versions of this topic might present different information than the information for your target version.

Administrators and advanced users can review changes to the .conf and .conf.spec files by using the Compare feature on different branches of the files in the jewnix/splunk-spec-files GitHub repository. Splunk thanks David Twersky (jewnix) for providing and maintaining this repository, and links to its contents with his permission.

Splunk App and Add-on Compatibility

Not all Splunk apps and add-ons are compatible with Splunk Enterprise version 9.1.

• See the Splunk products version compatibility matrix for information about which versions of Splunk IT Service Intelligence and Splunk Enterprise Security are compatible with this version of Splunk Enterprise.
• You can visit Splunkbase to confirm that your apps and add-ons are compatible with this version.

If your app or add-on is not compatible with version 9.1, consider delaying your upgrade until a compatible version is available.

Key points for upgrading to version 9.1

The following is a list of important elements that you must consider before upgrading Splunk Enterprise and its components. The sections that follow provide supporting details for these key points. Read through all sections in the topic before you begin your upgrade activities.

• Splunk Enterprise 9.0 introduced security changes and enhancements that you can enable after you upgrade. Splunk Enterprise 9.1 also includes these changes. Be aware of the changes to your deployment if you upgrade to version 9.1 from a version that is lower than 9.0. In a later release, Splunk might enable these enhancements for you. See Security updates in the Securing Manual to learn about the changes and how they might affect you.
• Splunk Enterprise 9.0 introduced fixes for a critical vulnerability in the deployment server. Depending on the set up of your Splunk Enterprise deployment, you might need to isolate your deployment server from other Splunk Enterprise components and perform upgrades of deployment clients that are currently lower than version 7.0.0 of Splunk Enterprise or the universal forwarder to at least version 7.0.0. You do not need to upgrade your deployment clients to version 9.0.0 or higher at this time. See Changes that can potentially break Splunk Enterprise installations later in this topic for specific details.
• Migrate your App Key Value Store storage engine from the Memory Mapped (MMAP) storage engine to the WiredTiger storage engine, and update your MongoDB version from 3.6 to 4.2. These updates are required in Splunk Enterprise 9.0 and higher. See Migrate the KV store storage engine in the Admin manual to plan your migration.
• Adjust your scripts and templates to use Python 3-compatible syntax before you upgrade. Splunk Enterprise 9.0 and higher remove support for Python 2. See Python 3 migration for more information.
• In Splunk Enterprise 9.1, usage of older jQuery libraries is restricted by default. Admins can still enable the use of older jQuery libraries in a self-serviceable manner. See Older jQuery libraries are restricted by default.
• Use the Splunk Products version compatibility matrix to ensure that any premium Splunk apps and add-ons you run are compatible with version 9.1. If they are not, do not upgrade until compatible versions become available.
• Data Fabric Search (DFS) and all associated server-side components are removed. Upgrading to Splunk Enterprise 9.0 or higher removes DFS functionality.
• Splunk supports a direct upgrade to Splunk Enterprise 9.1 from versions 8.2.x and higher only. Splunk supports a direct upgrade of a universal forwarder to version 9.1 from versions 8.2.x and higher of the universal forwarder only.
• To upgrade search head or indexer clusters, see Follow specific instructions to upgrade clusters later in this topic.
• Back up your App Key Value Store (KV Store) databases prior to starting an upgrade. If you run version 7.1 and lower of Splunk Enterprise, you must stop Splunk Enterprise instances first.
• If you run Linux machines that use the second extended (ext2) file system, upgrade that file system to third extended (ext3) prior to starting an upgrade.

Changes that can potentially break Splunk Enterprise installations

There is no supported method for rolling back an installation to a prior release. Follow the guidance for these critical items to avoid breaking your existing installation during an upgrade.

Upgrades of Splunk Enterprise to version 9.1 can fail on machines that run an operating system that does not use UTF-8 for character set encoding

If you run an operating system that does not use Unicode Transformation Format - 8 bit (UTF-8) for character set encoding, upgrades from an earlier version of Splunk Enterprise that supports a direct upgrade to version 9.1 can fail. Previously, this notice catalogued problems that occurred on Windows Server using the Japanese system locale, but this problem can occur on any operating system that does not use UTF-8 for character set encoding.

The failure presents in one of two ways:

• The installation fails. On Windows machines, the installer rolls back the installation.
• The installation appears to succeed, but Splunk Web then displays a blank page when you attempt to access the instance.

The failure occurs because of a problem with the operating system and the Splunk Enterprise installation package. Non-UTF8 character sets contain multiple-byte characters which the installer does not process correctly. The problem occurs only when you attempt to upgrade an existing version of Splunk Enterprise.

To work around the problem, prior to an upgrade, perform the following procedure:

1. On the instance that you want to upgrade, use a text editor to edit the $SPLUNK_HOME/etc/splunk-launch.conf file, or %SPLUNK_HOME%\etc\splunk-launch.conf on Windows.
2. In the file, add the following line:
   

   PYTHONUTF8=1

3. Save the file and exit the text editor.
4. Perform the upgrade.

If you have not performed the workaround prior to the upgrade, and the upgrade appears to succeed but then Splunk Web displays a blank page when you connect to it, you can perform the workaround, then restart the Splunk Enterprise instance after making the modification to the splunk-launch.conf file. You don't need to perform the upgrade again in this scenario.

The Windows Event Log data input has changes in how it expects incoming events from Windows Event Collector subscriptions

The Windows Event Log input now has a setting that lets you specify the format of incoming events from a Windows Event Collector (WEC) subscription. Previously, the input chose the event format based on the name of the destination log for the WEC subscription. Now, you can specify the format that the Splunk platform is to expect when it receives events through the subscription.

The name of the destination log for the WEC subscription determines the default format that the Splunk platform expects for incoming events from a subscription. For subscriptions whose destination log names begin with the string ForwardedEvents, the event format is rendered_event, or "RenderedText" in Microsoft parlance. For all other destination log names, the event format is raw_event, or "Events" in MS parlance.

If you have WEC subscriptions whose destination log names do not begin with ForwardedEvents but want to use the RenderedText format for that subscription, set wec_event_format to a value of rendered_text in the inputs.conf configuration file stanza for the subscription whose event format you want to change. See the following example:

[WinEventLog://WEC-Channel1]
wec_event_format = rendered_event

For more information about the setting, see Monitor Windows Event Log data in the Getting Data In Manual.

The default value for the 'reverse_lookup_honor_case_sensitive_match' setting in the transforms.conf configuration file has changed

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.1
The default value of the reverse_lookup_honor_case_sensitive_match setting in the transforms.conf configuration file has changed to false from true in version 9.1.0. This change was made to reduce the potential for inconsistent behavior in case sensitivity for reverse lookups. After you upgrade, reverse lookups no longer automatically honor case sensitivity.

Splunk Enterprise 9.1 fixes a critical vulnerability in deployment server but might introduce problems for older deployment clients

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

Splunk introduced a fix for a vulnerability in deployment server in version 9.0 of Splunk Enterprise, and this fix is also available in version 9.1. If you run a deployment server, upgrade that server to version 9.1 of Splunk Enterprise as soon as possible. Before the upgrade, carefully review your deployment server setup and the current versions of the deployment clients in your Splunk Enterprise network. Depending on the setup of your deployment server and whether that component shares a computer with other Splunk Enterprise components, you might need to do the following to ensure your deployment server and clients communicate without problems:

• Isolate deployment server from other components on a machine. Isolating your deployment server means you only have to upgrade that component. The sole exception for isolation is if you run a deployment server and a license manager on the same machine.
• Confirm that all deployment clients in your network run version 7.0.0 or higher of Splunk Enterprise or the universal forwarder. You don't have to upgrade deployment clients to version 9.0.0, but they must be at version 7.0.0 or higher to communicate with version 9.0.0 deployment servers.

See the following topics for additional information:

• SVD-2022-0608 for more about the vulnerability.
• Security updates in the Securing Splunk Enterprise Manual for more about the security updates that come with Splunk Enterprise 9.0 and higher.
• Client version compatibility in the Updating Splunk Enterprise Instances Manual for more about which versions of deployment client that the version 9.0.0 deployment server supports.

REST endpoint access authorization will be more secure in environments with distributed search

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

Beginning with Splunk Enterprise version 9.0, REST endpoints that distributed search nodes use to dispatch search jobs to indexers will use their own user-level tokens during that dispatching, rather than using a system-level token. You can override that default behavior by changing a setting in the limits.conf configuration file, but Splunk might remove that capability in a future release.

For more information about this change, see Security Updates in the Securing the Splunk Platform Manual.

Communication between deployment clients and deployment servers will be more secure

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0 Beginning in Splunk Enterprise version 9.0, the communication between deployment clients and deployment servers will receive security improvements.

Channel subscription logic, which is the logic that deployment clients use to establish a subscription to deployment servers, will change. Clients must subscribe to deployment server channels using an exact text match for the channel name. They must authenticate using a pass4symmKey that is identical on both the client and deployment server to download content from those servers. Clients will verify that they are actually communicating with a deployment server before attempting to download client bundles.

For more information about this change, see Security Updates in the Securing the Splunk Platform Manual.

New capability requirements for some risky search commands can potentially break existing user- and app-related searches

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

Beginning with version 9.0 of Splunk Enterprise, Splunk has created new capabilities that a user must possess through a role to run various risky search commands.

The following table lists the commands and the new capabilities that users need to run the commands:

Search command	New required capability
sendalert	run_sendalert
dump	run_dump
Any custom command	run_custom_command

By default, the user and power roles receive these new capabilities. If a user that does not hold these roles runs these commands, that user will no longer be able to do so after an upgrade unless you either add the capabilities to at least one role that the user holds, or inherit either the user or power roles to a role that the user holds. Webhook alert actions and any alert action integrations that send webhooks may also be affected by this change.

For more information about the Splunk platform approach towards managing risky commands see Safeguards for risky commands in the Securing the Splunk Platform manual.

The Linux universal forwarder installer creates a new non-root operating system user called 'splunkfwd'

Applicable components: Universal forwarder
Applicable OSes: Linux
Version introduced: 9.0

If you install version 9.1 or higher of the Linux universal forwarder, the installer now creates the 'splunkfwd' operating system user as a non-root user for managing your universal forwarder. In previous versions, the default user is 'splunk'. The 'splunkfwd' user is a new "least privileged" user that provides only the capabilities necessary to run the universal forwarder, which improves security. To learn more about how to work with the 'splunkfwd' user, see Manage a Linux least-privileged user in the Universal Forwarder manual.

TLS certificate validation is now available for all Splunk Enterprise instance types

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

If you use TLS to secure your Splunk Enterprise deployment, you can now enable TLS host name validation across the instances in that deployment. A new setting in the server.conf configuration file lets you control whether or not this validation happens on machines where you require certificates for secure communications. Some Splunk Enterprise instance types use certificates automatically for this type of connection. This version of Splunk Enterprise will provide notice when it cannot validate TLS certificates. Later versions of Splunk Enterprise might begin enforcing this type of certificate validation, which can cause connectivity problems between indexers, search heads, deployment servers, cluster nodes, and App Key Value Store.

For more information about this change, including how to turn on enforcement of certificate validation, see Security Updates in the Securing the Splunk Platform Manual.

Transport Layer Security (TLS) certificate validation is available for Python 3 modules

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

Splunk Inc. ships Python 3 as part of the Splunk Enterprise software package. Some Splunk-provided Python 3 modules, including but not limited to the httplib Python module, help developers and users alike establish secure connections to other Splunk Enterprise instances and APIs. The modules make these secure connections as part of their normal operations without validating the TLS certificates that the modules use to make those secure connections. Beginning with version 9.0 of Splunk Enterprise, you can use a setting in the server.conf configuration file to force TLS certificate validation checks. Later versions of Splunk Enterprise might enable these checks by default. Apps, APIs, and instances that use Python 3 modules which do not check TLS certificates might break when TLS certificate validation is on.

For more information about this change, see Security Updates in the Securing the Splunk Platform Manual.

Splunk Enterprise indexers and App Key Value Store nodes can now verify TLS certificates that they use to connect to each other

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

Splunk Enterprise indexers and App Key Value Store nodes use TLS to communicate securely with one another. Beginning with version 9.0 of Splunk Enterprise, you can use a setting in the server.conf configuration file to force the nodes to verify the certificates that they use to make secure connections. Later versions of Splunk Enterprise might enable these checks by default. When this happens, indexers and App Key Value Store nodes will not be able to communicate with one another any longer over TLS.

For more information about this change, see Security Updates in the Securing the Splunk Platform Manual.

Splunk Enterprise search peers will can now verify TLS certificates that they use to connect to each other

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

Splunk Enterprise search peers use TLS to communicate securely with one another. Beginning with version 9.0 of Splunk Enterprise, you can use a setting in the server.conf configuration file to force the nodes to verify the certificates that they use to make secure connections. Later versions of Splunk Enterprise might enable these checks by default. When this happens, search peers will not be able to communicate with one another any longer over TLS.

For more information about this change, see Security Updates in the Securing the Splunk Platform Manual.

The Splunk Command Line Interface can now verify TLS certificates that it uses to connect to external Splunk Enterprise nodes

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The Splunk Enterprise CLI uses TLS to communicate securely with Splunk Enterprise instances on other machines. Beginning with version 9.0 of Splunk Enterprise, you can use an argument in the CLI to force it to verify the certificates that it use to make secure connections. Later versions of Splunk Enterprise will enable these checks in the CLI by default. When this happens, the CLI will not be able to connect to other Splunk Enterprise nodes using TLS if it cannot validate the certificates.

For more information about this change, visit our Security Updates page in the Securing the Splunk Platform Manual.

Splunk Enterprise and Universal Forwarders must be version 8.2 or higher to upgrade to version 9.1

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 9.1

Splunk supports upgrades of Splunk Enterprise and the universal forwarder to version 9.1 from version 8.2 and higher only. If you run a version of Splunk Enterprise or universal forwarder that is lower than 8.2, you must upgrade to version 8.2 or higher as an intermediate step before attempting the upgrade to version 9.1. See Upgrade paths to version 9.1 for more information.

Follow specific instructions to upgrade clusters

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 7.3

To upgrade indexer or search head clusters, follow the upgrade procedure for the type of deployment you have.

• If your deployment has indexer clusters, follow the index cluster upgrade instructions.
• If your deployment has search head clusters, follow the search head cluster upgrade instructions.

On Splunk Enterprise indexer cluster manager nodes, enable maintenance mode before you upgrade

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.1

Because of changes to bucket data with the release of the latest version of Metrics Store, indexers that run versions of Splunk Enterprise lower than 8.1 cannot handle bucket replications from versions that run 8.1 and higher. Before you upgrade a cluster manager node, ensure it is in maintenance mode first. This is a standard part of the indexer cluster update process.

See Use maintenance mode in Managing Indexers and Clusters of Indexers.

Back up App Key Value Store prior to starting an upgrade

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 7.3

Back up the app key value store (KV store) before any maintenance like an upgrade.

If you are upgrading from version 7.1 or lower, you must stop Splunk Enterprise before you can back up the KV Store databases. Confirm that you have accounted for this downtime in your upgrade planning. See Back up and restore KV store for more information.

The indexer cluster slave-apps directory is renamed to peer-apps

During the upgrade, the indexer cluster slave-apps directory is automatically renamed to peer-apps. To accommodate this renaming, you must manually change any external hardcoded references to slave-apps, such as external scripts, SSL certificates, and so on, to peer-apps. See Update common peer configurations and apps in Managing Indexers and Clusters of Indexers.

The setting master_uri is renamed to manager_uri

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

In the [License] stanza of server.conf, master_uri is renamed to manager_uri. The old setting name is deprecated but still usable in 9.1. During upgrade to 9.x, if you have a mixed deployment of 9.x and 8.x instances, do not change master_uri to manager_uri in the 8.x instances. Version 8.x instances do not recognize the setting name manager_uri.

Occurrences that appear to be problems but are not

You might see things happen during or immediately after an upgrade that appear to indicate that the upgrade is not working. In nearly all cases, the occurrences that happen here can be expected.

If the following things occur during the upgrade, let the upgrade continue and do not interrupt it. If they occur immediately afterward, then perform benchmark tests on the deployment and compare to any benchmarks that you set up as part of the first phase of upgrading. Consider involving Splunk Support only if those benchmarks differ by a significant margin.

• On indexers, memory and CPU usage increases due to the following:
  • New data ingestion pipelines
• Permissions on the Splunk Enterprise introspection directory might change. Confirm that the user that runs Splunk Enterprise has write permission to the $SPLUNK_HOME/var/log/introspection directory.
• Deployment servers might push updates to all deployment clients due to app bundle hash recalculations.

Upgrading your Splunk Forwarder to version 9.1 on Linux using the .rpm and .deb packages creates superfluous splunkfwd user

When you upgrade your Linux universal forwarder to 9.1, a user called "splunkfwd" is created. This user creation is a bug and does not impact your process. This issue is resolved in 9.1.1. You can ignore or remove this user or upgrade to 9.1.1 instead. See the 9.1 release notes for more information.

Considerations for changed or removed features

The following major features have been changed or removed from this version. If you use features that have been removed, and have not yet migrated off them, consider delaying your upgrade until you have.

A new capability has been added that lets you edit passwords stored within an app

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.1
A new capability has been added that lets you edit credentials that have been stored within a Splunk app. This capability, edit_storage_passwords, can be assigned to a role to grant the ability to make changes to passwords stored within the context of an app. Previously, you needed the admin_all_objects capability to make these kinds of changes. While the admin_all_objects capability still lets you modify stored passwords in an app, it is no longer a requirement and lets you further secure your environment by reducing the number of roles that need the superuser-level capability.

After you upgrade, review any cloned roles that include the admin_all_objectscapability in the context of creating and updating passwords in an app. Consider replacing that capability with the new edit_storage_passwords capability instead.for these roles. You can also use the REST API to modify the access control list of each app to apply role restrictions to the apps' password storage configuration, including but not limited to restricting the ability to change passwords within an app to a specific role.

After you perform these adjustments, users who hold roles that contain the edit_storage_passwords capability can only modify credentials in apps to which their roles specifically grant them access. For example, if a user holds a role that grants them access to writing passwords in app1, they can't write passwords in app2 unless they hold a role that contains either or both of the edit_storage_passwords or admin_all_objects capabilities. The admin_all_objects capability overrides the edit_storage_passwords capability and still gives users the ability to see and change storage passwords on any Splunk app.

For more information about app access control endpoints (ACLs) and the REST API, see Access endpoint desecriptions in the REST API Reference Manual.

Older jQuery libraries are restricted by default

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.1

Usage of older jQuery libraries is restricted by default, but admins can still enable the use of older jQuery libraries in a self-serviceable manner. Select Settings, then Server Settings. Then, from the Internal Library Settings page, toggle the jQuery Libraries older than 3.5 section.

To learn more about enabling and restricting access to older jQuery libraries, see Control access to jQuery and other internal libraries.

Index buckets use zstd compression by default

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The indexes.conf setting journalCompression defaults to using zstd compression, beginning with this release.

The default tsidxWritingLevel for indexes has changed

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The default tsidxWritingLevel in indexes.conf has changed from 2 to 3, beginning with this release.

Splunk Enterprise 7.2 introduced a new file format and optimizations for tsidx files that resulted in improved search performance through decreased I/O, lowered storage usage, and improved utilization of SmartStore caches. These optimizations are encapsulated in levels, with new levels added in higher releases of Splunk Enterprise. Changing the default tsidxWritingLevel changes the optimizations used by both the index tsidx files and data model accelerations. See The tsidx writing level in the Managing Indexers and Clusters of Indexers manual.

The IOWait status check is delayed on startup

Applicable components: Splunk Enterprise
Applicable OSes: Linux
Version introduced: 9.0

The IOWait Health Report incorporates a 120 second delay after the Splunk Enterprise service starts to prevent false alerts.

The Bucket Health alert threshold is more aggressive

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The threshold values for the Health Report "Bucket" feature are set to alert more aggressively when an indexer rolls a large number of small buckets.

The web.conf setting 'simplexml_dashboard_create_version' is deprecated

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The simplexml_dashboard_create_version setting in the web.conf configuration file has been deprecated. Changing the default value might introduce security risks. Do not change the value without consulting Splunk Support.

The use of SSL compression is replaced with HTTP compression except when forwarding

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.2

In an effort to improve scalability and efficiency, the use of SSL compression is no longer enabled by default for SSL communications between Splunk Enterprise instances. The SSL compression is replaced with HTTP compression by default, except when forwarding data over SSL. This change does not prevent the use of SSL compression when negotiating communications over SSL with earlier Splunk Enterprise instances. For example, an earlier client negotiating with an 8.2 instance will still use SSL compression, and an 8.2 client negotiating with earlier servers will use HTTP compression.

This change effects the existing settings:

Setting	New default with 8.2	Prior default
server.conf useHTTPClientCompression	true	false
server.conf useClientSSLCompression	false	true
outputs.conf useClientSSLCompression	true	Referenced the server.conf useClientSSLCompression setting (default: true)

If you have modified the useHTTPClientCompression or useClientSSLCompression settings in the past, check your environment to determine if the new defaults will impact your compression settings after an upgrade. Use btool to verify how compression is set in your environment based upon the settings above, and where those changes are made.

If you have explicitly enabled the server.conf setting useClientSSLCompression using a custom app or a local .conf file, after the upgrade both the SSL and HTTP compression will be enabled simultaneously. After the upgrade is complete, verify the useClientSSLCompression setting, and make sure the SSL compression is disabled.

The splunkd.log events now include thread names

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.2

The default logging level for splunkd.log events now includes thread names to improve troubleshooting.

An additional option is available when performing a searchable rolling restart for multisite clusters

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.2

The new server.conf setting searchable_rolling_site_down_policy improves searchable rolling restart performance when the prerequisites are met.

See How the manager determines the number of multisite peers to restart in each round in the Managing Indexers and Clusters of Indexers manual.

A Splunk Enterprise or universal forwarder installation will no longer create an inputs.conf with the hostname

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.1

There's a change in the default installation behavior for Splunk Enterprise and universal forwarder instances. When installing earlier releases, the installation process would determine the hostname and set it in a newly created $SPLUNK_HOME/etc/system/local/inputs.conf file. On 8.1 and higher releases, splunkd no longer creates the inputs.conf file during installation. The service checks and sets the hostname as part of the inputs.conf setting host = $decideOnStartup when the service starts.

Some logging categories for the authentication system have changed

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.1

As part of overall improvements to support of the Security Assertion Markup Language (SAML) authentication scheme, various logging channels for the Splunk platform authentication system have changed. The logging category names that previously started with AuthenticationManager now begin with AuthenticationProvider. Following an upgrade, review the log.cfg in $SPLUNK_HOME/etc/ to confirm that the names have been changed.

Splunk Enterprise license enforcement change

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.1

For license stack volumes of less than 100GB of data per day, Splunk Enterprise will disable search when license limits are violated, after 45 warnings within a 60-day rolling window. For more information, see the License Enforcement FAQ on splunk.com.

The default logging level for audit logs has changed

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.1

The default logging level for auditing of certain administrative events, information that goes to the _audit index, has been lowered from the INFO level to the DEBUG level. Additionally, you now gain the ability to control the logging level that auditing events happens.

While this is a net positive for most customers due to the lower default verbosity of logging, it also means that some audit events, such as capability checks, are no longer logged by default due to the lower log level. To restore the old behavior, you can configure the Splunk platform logging utility by editing the $SPLUNK_HOME/etc/log-local.cfg file and raising the category.AuditLogger entry from DEBUG back to INFO.

Splunk data collection practices have changed

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0

In an effort to provide better support to its customers and improve its products and services, Splunk has changed its data collection practices. After you upgrade, Splunk automatically opts you in to sharing telemetry data. To learn what data Splunk collects, see What data Splunk collects in the Admin Manual.

The change happens when you first log into an instance with an administrator user after the upgrade. You receive a pop-up message that indicates the policy change, and the instance overwrites any existing data-sharing configurations after you acknowledge the pop-up.

You can still opt out of sharing telemetry data at any time. To learn how, see How to opt out in the "Share data in Splunk Enterprise" topic of the Admin Manual.

The "access controls" Splunk Web menu options have changed

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.0

The "Access controls" menu item under the "Users and Authentication" header in the Settings menu in Splunk Web has been replaced with individual links to user, role, password, and authentication scheme management.

The default memory resource allocation for workload categories has changed

Applicable components: Splunk Enterprise
Applicable OSes: Linux
Version introduced: 8.0

On upgrade to version 8.x, the default memory resource allocation for the ingest workload category changes from 20% to 100%. This might cause an increase in memory usage in the ingest category after upgrade. The default memory resource allocation for the search and misc. categories remains the same, at 70% and 10%, respectively.

For more information on workload categories, see Configure workload categories in the Workload Management manual.

The "failure to localize search" Splunk Web error message has been replaced with a "partial results" message

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.0

When you upgrade to version 8.x and run a search that encounters a localization error, the message that appears in Splunk Web has changed. Instead of displaying a "failure to localize" message, Splunk Web notifies users that the search process might have returned partial results.

Roles with the "install_apps" capability also need the "list_settings" capability to access certain REST endpoints

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.0

After you upgrade to version 8.x of Splunk Enterprise, any roles that hold the install_apps capability must also hold the list_settings capability for role users to be able to manage app installations through REST (using the manager/appinstall/<app> endpoint, for example.

Splunk Enterprise forwarders at version 8.x cannot forward metrics data to indexers that run lower than version 8.x

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0

There is no support for the forwarding of metrics data from forwarders that run version 8.x to indexers that run version 7.3 or lower. If you need to forward metrics data from a version 8.x forwarder, confirm that all indexers that receive this kind of data also run version 8.x.

Upgraded search head nodes will complain of missing metrics indexes in lower-version indexer nodes

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0

If you upgrade your distributed search environment to version 8.x of Splunk Enterprise but keep your indexer nodes at a version below 8.0, your search head nodes will generate alerts about a missing _metrics index when they try to send metrics.log events to those indexers. You cannot work around this problem by adding an _metrics index to the indexer cluster, as the format of the metrics data is different on indexers that run a lower version. Upgrade all Splunk Enterprise components to the latest version where possible.

After an upgrade, configure limits.conf settings on all search head cluster nodes that handle metrics data

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0

After you upgrade your search head and indexer clusters to version 8.x of Splunk Enterprise, edit limits.conf on each search head cluster and set the always_use_single_value_output setting under the [mcollect] stanza to false. This lets these nodes use the "multiple measures per metric data point" schema when you convert logs to metrics with the mcollect command or use metrics rollups. This new schema increases your data storage capacity and improves metrics search performance.

In metric indexes only, by default the Splunk software removes rawdata journal files from buckets when they roll from hot to warm

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.1

Rawdata journal files are not used by metric searches. However, after the optimizations introduced in 8.0, rawdata journal files take up a large amount of the volume of a metrics index bucket. In version 8.1 and later, the Splunk software removes these files from metric index buckets when they roll from hot to warm by default. The software removes the file from /rawdata directory in the bucket and replaces it with a dummy journal file that has no usable data other than the journal header. This reduces the amount of space that metric indexes take up on disk.

This does not apply to metric indexes that have replication enabled (repFactor = auto in limits.conf) for indexer clustering. For more information, or to learn how to disable rawdata journal removal for a specific index, see the description of the metric.stubOutRawdataJournal setting in limits.conf.spec.

Metric data points containing metric names that are empty or composed entirely of white spaces cannot be indexed or searched upon

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.1

In version 8.1 and later, Splunk Enterprise cannot index metric data points containing metric names that are empty or composed entirely of white spaces. This can happen with ingested metrics data as well as event data that is converted to metrics data through the log-to-metrics sourcetype or some other method.

After the upgrade, Splunk Enterprise searches cannot return metric data points containing empty or white-spaced metric names, if those metric data points were indexed in a previous version of Splunk.

StatsD metric data inputs now produce single-measurement metric data points by default

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0.3

For ease of use, by default Splunk Enterprise version 8.0.3 and higher converts StatsD metric data into single-measurement metric data points that have one key-value pair for the metric name and one key-value pair for the measurement value. If, prior to 8.0.3, you used StatsD inputs that converted their data to metric data points that could carry multiple measurements, you need to add STATSD_EMIT_SINGLE_MEASUREMENT_FORMAT=false to a stanza for the metric source type in props.conf.

Streaming metric alerts are not available on indexer clusters that run version 7.2 and lower

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0

Splunk Enterprise version 8.0 introduced streaming metric alerts, which are evaluated on a continuous basis, and which can reduce the load on your system by enabling similar alerts to share the same search process. There is no support for streaming metric alerts on indexer clusters that run version 7.2 or lower. If you have an indexer cluster and a search head cluster, and you upgrade the search head cluster to version 8.x, you lose the ability to trigger streaming metric alerts for that search head cluster. If you need to retain this ability, upgrade the indexer cluster to version 8.x first.

You can no longer use wildcards in real-time searches that use the mstats command

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.0

Beginning with Splunk Enterprise version 8.0, you can no longer use wildcards in real-time searches that contain the mstats command. If you have these kinds of searches, change them so that they no longer use wildcards of any kind, either in aggregation fields or metric names, prior to upgrading

Metrics indexing and search is now case-sensitive

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0

The indexing and search schema for metrics data is case-sensitive as of Splunk Enterprise 8.0. Instances of Splunk Enterprise that are lower than 8.0 might encounter problems retrieving results when searching instances that run version 8.0 or higher.

Older, deprecated configuration file settings for knowledge bundle replication have been removed

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.0

Many older, previously deprecated settings under the [replicationSettings] stanza in distsearch.conf have been removed. If your configuration uses these settings, consider changing them before you start an upgrade.

Configuration file settings for specifying the mounted bundle option for knowledge bundle replication have changed

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.0

The distsearch.conf setting for specifying the mounted bundle option for knowledge bundle replication has changed. Previously, the setting was shareBundles=false. The new setting is replicationPolicy=mounted. If the upgrade process finds the old setting in your configuration files, it converts it automatically to the new setting, so no manual intervention is required on your part.

The Django framework and all its associated components have been removed from the Splunk platform. Apps and dashboards that use this framework will not function anymore. Before you upgrade, confirm that apps and dashboards you regularly use no longer use Django, as an upgrade will render them inoperable. -->

When you upgrade to a version of Splunk Enterprise that is 7.2 or higher, the behavior of certain field alias configurations changes. This behavior affects events where the alias field is already present before field alias processing takes place.

• If both the source field and the alias field are present in the event, the search head overwrites the value of the alias field to match the value of the source field.
• If the alias field is present in the event but the source field is missing or has a null value, the search head removes the alias field from the event.

This is how field aliasing should work in these situations. However, you might have searches that rely on the old field alias behavior. A new ASNEW syntax has been added to enable you to provide the pre-7.2 behavior to your field alias configurations.

The pre-7.2 field alias behavior enabled users to create sets of field alias configurations that matched multiple source fields with one alias field. You can use the ASNEW syntax to continue doing this. A better practice would be to use a calculated field that uses the coalesce function to create a new field that takes the value of one or more existing fields. This method lets you be explicit about ordering of input field values in the case of null fields. For example: EVAL-ip = coalesce(clientip,ipaddress).

See Field alias behavior change in the Release Notes for further details. -->

The tstats search command, which performs statistical queries on indexed fields in tsidx files, has updated behavior with regard to the number of events it finds in a bucket.

Previously, the command reported the scan count of events in a bucket. After an upgrade, the command now reports the actual number of matched events in a bucket as the scan count. If you use tstats for searches, you might see scan counts fall significantly. This does not indicate incorrect results. -->

Considerations for new features

Splunk has introduced the following new features in this version of Splunk Enterprise. You might need to perform some configuration after an upgrade to enable and take advantage of these features.

Macros now replicate by default to search peers

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.1
Macros used in apps are now replicated by default to search peers as part of the knowledge bundle in Splunk deployments. As a result of this change, searches that previously failed now run successfully, which could affect downstream performance.

If you don't want to replicate macros for your apps, you can suppress replication by setting replicate.macros = false in the [replicationSettings:refineConf] stanza in the distsearch.conf file. Be aware that disabling distribution of macros might negatively impact your search results.

New configuration settings for importing structured data files

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0

Splunk Enterprise 8.0.0 and higher has added some new settings for configuring the indexing of structured data. Previously, the structured data processor converted characters that it encountered in header field names that were neither alphanumeric nor a space into underscores. With the new HEADER_FIELD_ACCEPTABLE_SPECIAL_CHARACTERS setting in props.conf, you can control which characters the processor accepts as valid in header field names. Some characters might cause the processor to malfunction, so if you import lots of structured data and want to use the feature, use a test index to confirm that the processor imports the data in the way you want.

Learn about known upgrade issues

To learn about any additional upgrade issues for Splunk Enterprise, see the Known Issues - Upgrade Issues page in the Release Notes.