About upgrading to 9.0 READ THIS FIRST

This topic describes changes in behavior to note for Splunk Enterprise and Splunk Universal Forwarder when you upgrade to version 9.0. Use the Version drop-down list to choose the product version to which you're upgrading. Lower or higher versions of this topic might present different information than the information for your target version.

Administrators and advanced users can review changes to the .conf and .conf.spec files by using the Compare feature on different branches of the files in this GitHub repository: https://github.com/jewnix/splunk-spec-files. Splunk thanks David Twersky (jewnix) for providing and maintaining this repository, and we link to its contents with his permission.

Splunk App and Add-on Compatibility

Not all Splunk apps and add-ons are compatible with Splunk Enterprise version 9.0.

• See the Splunk products version compatibility matrix for information about which versions of Splunk IT Service Intelligence and Splunk Enterprise Security are compatible with this version of Splunk Enterprise.
• You can visit Splunkbase to confirm that your apps and add-ons are compatible with this version.

If your app or add-on is not compatible with version 9.0, consider delaying your upgrade until a compatible version is available.

Key points for upgrading to version 9.0

The following is a list of important elements that you must consider before upgrading Splunk Enterprise and its components. The sections that follow provide supporting details for these key points. Read through all sections in the topic before you begin your upgrade activities.

• Splunk Enterprise 9.0 has some security changes and enhancements that you can enable after you upgrade. In a later release, Splunk might enable these enhancements for you. See Security updates in the Securing Manual to learn about the changes and how they might affect you.
• Splunk Enterprise 9.0 introduces fixes for a critical vulnerability in the deployment server. Depending on the set up of your Splunk Enterprise deployment, you might need to isolate your deployment server from other Splunk Enterprise components and perform upgrades of deployment clients that are currently lower than version 7.0.0 of Splunk Enterprise or the universal forwarder to at least version 7.0.0. You do not need to upgrade your deployment clients to version 9.0.0 at this time. See Changes that can potentially break Splunk Enterprise installations later in this topic for specific details.
• Customers who have alerts configured that use the trigger condition "rises by" or "drops by" might see a significant increase in skipped or deferred searches after they upgrade to Splunk Enterprise version 9.0.0 through 9.0.5. See Increased skipped search rate after upgrade to 9.0 in the Release Notes.
• Migrate your App Key Value Store storage engine from the Memory Mapped (MMAP) storage engine to the WiredTiger storage engine, and update your MongoDB version from 3.6 to 4.2. These updates are required in Splunk Enterprise 9.0. See Migrate the KV store storage engine in the Admin manual to plan your migration.
• Adjust your scripts and templates to use Python 3-compatible syntax before you upgrade. Splunk Enterprise 9.0 removes support for Python 2. See Python 3 migration for more information.
• Use the Splunk Products version compatibility matrix to ensure that any premium Splunk apps and add-ons you run are compatible with version 9.0. If they are not, do not upgrade until compatible versions become available.
• Data Fabric Search (DFS) and all associated server-side components are removed. Upgrading to Splunk Enterprise 9.0 removes DFS functionality.
• Upgrading Splunk Enterprise directly to version 9.0 is only supported from versions 8.1.x and higher. Upgrading a universal forwarder directly to version 9.0 is supported from versions 8.1.x and higher.
• To upgrade search head or indexer clusters, see Follow specific instructions to upgrade clusters later in this topic.
• Back up your App Key Value Store (KV Store) databases prior to starting an upgrade. If you run version 7.1 and lower of Splunk Enterprise, you must stop Splunk Enterprise instances first.
• If you run Linux machines that use the second extended (ext2) file system, upgrade that file system to third extended (ext3) prior to starting an upgrade.

Changes that can potentially break Splunk Enterprise installations

There is no supported method for rolling back an installation to a prior release. Follow the guidance for these critical items to avoid breaking your existing installation during an upgrade.

Splunk Enterprise 9.0 fixes a critical vulnerability in deployment server but might introduce problems for older deployment clients

If you run a deployment server, upgrade that server to version 9.0 of Splunk Enterprise as soon as possible. Before the upgrade, carefully review your deployment server setup and the current versions of the deployment clients in your Splunk Enterprise network. Depending on the setup of your deployment server and whether that component shares a computer with other Splunk Enterprise components, you might need to do the following to ensure your deployment server and clients communicate without problems:

• Isolate deployment server from other components on a machine. Isolating your deployment server means you only have to upgrade that component. The sole exception for isolation is if you run a deployment server and a license manager on the same machine.
• Confirm that all deployment clients in your network run version 7.0.0 or higher of Splunk Enterprise or the universal forwarder. You don't have to upgrade deployment clients to version 9.0.0, but they must be at version 7.0.0 or higher to communicate with version 9.0.0 deployment servers.

See the following topics for additional information:

• SVD-2022-0608 for more about the vulnerability.
• Security updates in the Securing Splunk Enterprise Manual for more about the security updates that come with Splunk Enterprise 9.0.
• Client version compatibility in the Updating Splunk Enterprise Instances Manual for more about which versions of deployment client that the version 9.0.0 deployment server supports.

REST endpoint access authorization will be more secure in environments with distributed search

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

Beginning with Splunk Enterprise version 9.0, REST endpoints that distributed search nodes use to dispatch search jobs to indexers will use their own user-level tokens during that dispatching, rather than using a system-level token. You can override that default behavior by changing a setting in the limits.conf configuration file, but Splunk might remove that capability in a future release.

For more information about this change, see Security Updates in the Securing the Splunk Platform Manual.

Communication between deployment clients and deployment servers will be more secure after a configuration change

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0 Beginning in Splunk Enterprise version 9.0, the communication between deployment clients and deployment servers will receive security improvements.

Channel subscription logic, which is the logic that deployment clients use to establish a subscription to deployment servers, will change. Clients must subscribe to deployment server channels using an exact text match for the channel name. They must authenticate using a pass4symmKey that is identical on both the client and deployment server to download content from those servers. Clients will verify that they are actually communicating with a deployment server before attempting to download client bundles.

This change requires you to make a configuration update to your deployment clients and servers. For more information about the change, see Security Updates in the Securing the Splunk Platform Manual.

New capability requirements for some risky search commands can potentially break existing user- and app-related searches

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

Beginning with version 9.0 of Splunk Enterprise, Splunk has created new capabilities that a user must possess through a role to run various risky search commands.

The following table lists the commands and the new capabilities that users need to run the commands:

Search command	New required capability
sendalert	run_sendalert
dump	run_dump
Any custom command	run_custom_command

By default, the user and power roles receive these new capabilities. If a user that does not hold these roles runs these commands, that user will no longer be able to do so after an upgrade unless you either add the capabilities to at least one role that the user holds, or inherit either the user or power roles to a role that the user holds.

For more information about the Splunk platform approach towards managing risky commands see Safeguards for risky commands in the Securing the Splunk Platform manual.

TLS certificate validation is now available for all Splunk Enterprise instance types

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

If you use TLS to secure your Splunk Enterprise deployment, you can now enable TLS host name validation across the instances in that deployment. A new setting in the server.conf configuration file lets you control whether or not this validation happens on machines where you require certificates for secure communications. Some Splunk Enterprise instance types use certificates automatically for this type of connection. This version of Splunk Enterprise will provide notice when it cannot validate TLS certificates. Later versions of Splunk Enterprise might begin enforcing this type of certificate validation, which can cause connectivity problems between indexers, search heads, deployment servers, cluster nodes, and App Key Value Store.

For more information about this change, including how to turn on enforcement of certificate validation, see Security Updates in the Securing the Splunk Platform Manual.

Transport Layer Security (TLS) certificate validation is available for Python 3 modules

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

Splunk Inc. ships Python 3 as part of the Splunk Enterprise software package. Some Splunk-provided Python 3 modules, including but not limited to the httplib Python module, help developers and users alike establish secure connections to other Splunk Enterprise instances and APIs. The modules make these secure connections as part of their normal operations without validating the TLS certificates that the modules use to make those secure connections. Beginning with version 9.0 of Splunk Enterprise, you can use a setting in the server.conf configuration file to force TLS certificate validation checks. Later versions of Splunk Enterprise might enable these checks by default. Apps, APIs, and instances that use Python 3 modules which do not check TLS certificates might break when TLS certificate validation is on.

For more information about this change, see Security Updates in the Securing the Splunk Platform Manual.

Splunk Enterprise indexers and App Key Value Store nodes can now verify TLS certificates that they use to connect to each other

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

Splunk Enterprise indexers and App Key Value Store nodes use TLS to communicate securely with one another. Beginning with version 9.0 of Splunk Enterprise, you can use a setting in the server.conf configuration file to force the nodes to verify the certificates that they use to make secure connections. Later versions of Splunk Enterprise might enable these checks by default. When this happens, indexers and App Key Value Store nodes will not be able to communicate with one another any longer over TLS.

For more information about this change, see Security Updates in the Securing the Splunk Platform Manual.

Splunk Enterprise search peers will can now verify TLS certificates that they use to connect to each other

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

Splunk Enterprise search peers use TLS to communicate securely with one another. Beginning with version 9.0 of Splunk Enterprise, you can use a setting in the server.conf configuration file to force the nodes to verify the certificates that they use to make secure connections. Later versions of Splunk Enterprise might enable these checks by default. When this happens, search peers will not be able to communicate with one another any longer over TLS.

For more information about this change, see Security Updates in the Securing the Splunk Platform Manual.

The Splunk Command Line Interface can now verify TLS certificates that it uses to connect to external Splunk Enterprise nodes

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The Splunk Enterprise CLI uses TLS to communicate securely with Splunk Enterprise instances on other machines. Beginning with version 9.0 of Splunk Enterprise, you can use an argument in the CLI to force it to verify the certificates that it use to make secure connections. Later versions of Splunk Enterprise will enable these checks in the CLI by default. When this happens, the CLI will not be able to connect to other Splunk Enterprise nodes using TLS if it cannot validate the certificates.

For more information about this change, visit our Security Updates page in the Securing the Splunk Platform Manual.

The indexer cluster slave-apps directory is renamed to peer-apps

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

During the upgrade, the indexer cluster slave-apps directory is automatically renamed to peer-apps. To accommodate this renaming, you must manually change any external hardcoded references to slave-apps, such as external scripts, SSL certificates, and so on, to peer-apps. See Update common peer configurations and apps in Managing Indexers and Clusters of Indexers.

The setting master_uri is renamed to manager_uri

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

In the [License] stanza of server.conf, master_uri is renamed to manager_uri. The old setting name is deprecated but still usable in 9.0. During upgrade to 9.x, if you have a mixed deployment of 9.x and 8.x instances, do not change master_uri to manager_uri in the 8.x instances. Version 8.x instances do not recognize the setting name manager_uri.

Plan your upgrade to work with the Python 3 migration

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

Beginning with version 9.0, Splunk Enterprise uses the Python 3 interpreter only. See Python 3 migration for more information.

Splunk Enterprise and Universal Forwarders must be version 8.1 or higher to upgrade to version 9.0

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 9.0

Splunk supports upgrades of Splunk Enterprise and the universal forwarder to version 9.0 from version 8.1 and higher only. If you run a version of Splunk Enterprise or universal forwarder that is lower than 8.1, you must upgrade to version 8.1 or higher as an intermediate step before attempting the upgrade to version 9.0. See Upgrade paths to version 9.0 for more information.

Follow specific instructions to upgrade clusters

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 7.3

To upgrade indexer or search head clusters, follow the upgrade procedure for the type of deployment you have.

• If your deployment has indexer clusters, follow the index cluster upgrade instructions.
• If your deployment has search head clusters, follow the search head cluster upgrade instructions.

On Splunk Enterprise indexer cluster manager nodes, enable maintenance mode before you upgrade

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.1

Because of changes to bucket data with the release of the latest version of Metrics Store, indexers that run versions of Splunk Enterprise lower than 8.1 cannot handle bucket replications from versions that run 8.1 and higher. Before you upgrade a cluster manager node, ensure it is in maintenance mode first. This is a standard part of the indexer cluster update process.

See Use maintenance mode in Managing Indexers and Clusters of Indexers.

Back up App Key Value Store prior to starting an upgrade

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 7.3

Back up the app key value store (KV store) before any maintenance like an upgrade.

If you are upgrading from version 7.1 or lower, you must stop Splunk Enterprise before you can back up the KV Store databases. Confirm that you have accounted for this downtime in your upgrade planning. See Back up and restore KV store for more information.

The Splunk Enterprise credential creation scheme might affect scripted upgrades

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 7.2

Splunk Enterprise 7.1 introduced an updated authentication scheme for users that requires that you create a password for the admin account. In version 7.2, the scheme was extended to let you customize administrator credential creation for Splunk Enterprise instances.

This scheme includes additional settings and configuration options, which can affect how you upgrade if you use scripts to automate the upgrade process. You might need to change your upgrade scripts before performing scripted upgrades. Specifically, confirm that you do not pass any illegal arguments to the Splunk CLI for starting or restarting Splunk Enterprise during the upgrade, as this could result in a situation where Splunk Enterprise does not start after the upgrade has finished.

Occurrences that appear to be problems but are not

You might see things happen during or immediately after an upgrade that appear to indicate that the upgrade is not working. In nearly all cases, the occurrences that happen here can be expected.

If the following things occur during the upgrade, let the upgrade continue and do not interrupt it. If they occur immediately afterward, then perform benchmark tests on the deployment and compare to any benchmarks that you set up as part of the first phase of upgrading. Consider involving Splunk Support only if those benchmarks differ by a significant margin.

• On indexers, memory and CPU usage increases due to the following:
  • New data ingestion pipelines
• Permissions on the Splunk Enterprise introspection directory might change. Confirm that the user that runs Splunk Enterprise has write permission to the $SPLUNK_HOME/var/log/introspection directory.
• Deployment servers might push updates to all deployment clients due to app bundle hash recalculations.

CPU and memory usage on indexers increases

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 7.3

By default, new data ingestion pipelines consume more resources when they are enabled.

Permissions on introspection directory might change

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 7.3

Confirm that the user who runs Splunk Enterprise has write permission to the $SPLUNK_HOME/var/log/introspection directory.

Deployment servers might push updates to all deployment clients due to app bundle hash recalculations.

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 7.3

Considerations for changed or removed features

The following major features have been changed or removed from this version. If you use features that have been removed, and have not yet migrated off them, consider delaying your upgrade until you have.

Index buckets use zstd compression by default

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The indexes.conf setting journalCompression defaults to using zstd compression, beginning with this release.

The default tsidxWritingLevel for indexes has changed

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The default tsidxWritingLevel in indexes.conf has changed from 2 to 3, beginning with this release.

Splunk Enterprise 7.2 introduced a new file format and optimizations for tsidx files that resulted in improved search performance through decreased I/O, lowered storage usage, and improved utilization of SmartStore caches. These optimizations are encapsulated in levels, with new levels added in higher releases of Splunk Enterprise. Changing the default tsidxWritingLevel changes the optimizations used by both the index tsidx files and data model accelerations. See The tsidx writing level in the Managing Indexers and Clusters of Indexers manual.

The IOWait status check is delayed on startup

Applicable components: Splunk Enterprise
Applicable OSes: Linux
Version introduced: 9.0

The IOWait Health Report incorporates a 120 second delay after the Splunk Enterprise service starts to prevent false alerts.

The Bucket Health alert threshold is more aggressive

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The threshold values for the Health Report "Bucket" feature are set to alert more aggressively when an indexer rolls a large number of small buckets.

The web.conf setting 'simplexml_dashboard_create_version' is deprecated

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The simplexml_dashboard_create_version setting in the web.conf configuration file has been deprecated. Changing the default value might introduce security risks. Do not change the value without consulting Splunk Support.

The use of SSL compression is replaced with HTTP compression except when forwarding

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.2

In an effort to improve scalability and efficiency, the use of SSL compression is no longer enabled by default for SSL communications between Splunk Enterprise instances. The SSL compression is replaced with HTTP compression by default, except when forwarding data over SSL. This change does not prevent the use of SSL compression when negotiating communications over SSL with earlier Splunk Enterprise instances. For example, an earlier client negotiating with an 8.2 instance will still use SSL compression, and an 8.2 client negotiating with earlier servers will use HTTP compression.

This change effects the existing settings:

Setting	New default with 8.2	Prior default
server.conf useHTTPClientCompression	true	false
server.conf useClientSSLCompression	false	true
outputs.conf useClientSSLCompression	true	Referenced the server.conf useClientSSLCompression setting (default: true)

If you have modified the useHTTPClientCompression or useClientSSLCompression settings in the past, check your environment to determine if the new defaults will impact your compression settings after an upgrade. Use btool to verify how compression is set in your environment based upon the settings above, and where those changes are made.

If you have explicitly enabled the server.conf setting useClientSSLCompression using a custom app or a local .conf file, after the upgrade both the SSL and HTTP compression will be enabled simultaneously. After the upgrade is complete, verify the useClientSSLCompression setting, and make sure the SSL compression is disabled.

The splunkd.log events now include thread names

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.2

The default logging level for splunkd.log events now includes thread names to improve troubleshooting.

An additional option is available when performing a searchable rolling restart for multisite clusters

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.2

The new server.conf setting searchable_rolling_site_down_policy improves searchable rolling restart performance when the prerequisites are met.

See How the manager determines the number of multisite peers to restart in each round in the Managing Indexers and Clusters of Indexers manual.

A Splunk Enterprise or universal forwarder installation will no longer create an inputs.conf with the hostname

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.1

There's a change in the default installation behavior for Splunk Enterprise and universal forwarder instances. When installing earlier releases, the installation process would determine the hostname and set it in a newly created $SPLUNK_HOME/etc/system/local/inputs.conf file. On 8.1 and higher releases, splunkd no longer creates the inputs.conf file during installation. The service checks and sets the hostname as part of the inputs.conf setting host = $decideOnStartup when the service starts.

Some logging categories for the authentication system have changed

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.1

As part of overall improvements to support of the Security Assertion Markup Language (SAML) authentication scheme, various logging channels for the Splunk platform authentication system have changed. The logging category names that previously started with AuthenticationManager now begin with AuthenticationProvider. Following an upgrade, review the log.cfg in $SPLUNK_HOME/etc/ to confirm that the names have been changed.

Splunk Enterprise license enforcement change

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.1

For license stack volumes of less than 100GB of data per day, Splunk Enterprise will disable search when license limits are violated, after 45 warnings within a 60-day rolling window. For more information, see the License Enforcement FAQ on splunk.com.

The default logging level for audit logs has changed

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.1

The default logging level for auditing of certain administrative events, information that goes to the _audit index, has been lowered from the INFO level to the DEBUG level. Additionally, you now gain the ability to control the logging level that auditing events happens.

While this is a net positive for most customers due to the lower default verbosity of logging, it also means that some audit events, such as capability checks, are no longer logged by default due to the lower log level. To restore the old behavior, you can configure the Splunk platform logging utility by editing the $SPLUNK_HOME/etc/log-local.cfg file and raising the category.AuditLogger entry from DEBUG back to INFO.

Splunk data collection practices have changed

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0

In an effort to provide better support to its customers and improve its products and services, Splunk has changed its data collection practices. After you upgrade, Splunk automatically opts you in to sharing telemetry data. To learn what data Splunk collects, see What data Splunk collects in the Admin Manual.

The change happens when you first log into an instance with an administrator user after the upgrade. You receive a pop-up message that indicates the policy change, and the instance overwrites any existing data-sharing configurations after you acknowledge the pop-up.

You can still opt out of sharing telemetry data at any time. To learn how, see How to opt out in the "Share data in Splunk Enterprise" topic of the Admin Manual.

The "access controls" Splunk Web menu options have changed

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.0

The "Access controls" menu item under the "Users and Authentication" header in the Settings menu in Splunk Web has been replaced with individual links to user, role, password, and authentication scheme management.

The default memory resource allocation for workload categories has changed

Applicable components: Splunk Enterprise
Applicable OSes: Linux
Version introduced: 8.0

On upgrade to version 8.x, the default memory resource allocation for the ingest workload category changes from 20% to 100%. This might cause an increase in memory usage in the ingest category after upgrade. The default memory resource allocation for the search and misc. categories remains the same, at 70% and 10%, respectively.

For more information on workload categories, see Configure workload categories in the Workload Management manual.

The "failure to localize search" Splunk Web error message has been replaced with a "partial results" message

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.0

When you upgrade to version 8.x and run a search that encounters a localization error, the message that appears in Splunk Web has changed. Instead of displaying a "failure to localize" message, Splunk Web notifies users that the search process might have returned partial results.

Roles with the "install_apps" capability also need the "list_settings" capability to access certain REST endpoints

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.0

After you upgrade to version 8.x of Splunk Enterprise, any roles that hold the install_apps capability must also hold the list_settings capability for role users to be able to manage app installations through REST (using the manager/appinstall/<app> endpoint, for example.

Splunk Enterprise forwarders at version 8.x cannot forward metrics data to indexers that run lower than version 8.x

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0

There is no support for the forwarding of metrics data from forwarders that run version 8.x to indexers that run version 7.3 or lower. If you need to forward metrics data from a version 8.x forwarder, confirm that all indexers that receive this kind of data also run version 8.x.

Upgraded search head nodes will complain of missing metrics indexes in lower-version indexer nodes

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0

If you upgrade your distributed search environment to version 8.x of Splunk Enterprise but keep your indexer nodes at a version below 8.0, your search head nodes will generate alerts about a missing _metrics index when they try to send metrics.log events to those indexers. You cannot work around this problem by adding an _metrics index to the indexer cluster, as the format of the metrics data is different on indexers that run a lower version. Upgrade all Splunk Enterprise components to the latest version where possible.

After an upgrade, configure limits.conf settings on all search head cluster nodes that handle metrics data

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0

After you upgrade your search head and indexer clusters to version 8.x of Splunk Enterprise, edit limits.conf on each search head cluster and set the always_use_single_value_output setting under the [mcollect] stanza to false. This lets these nodes use the "multiple measures per metric data point" schema when you convert logs to metrics with the mcollect command or use metrics rollups. This new schema increases your data storage capacity and improves metrics search performance.

In metric indexes only, by default the Splunk software removes rawdata journal files from buckets when they roll from hot to warm

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.1

Rawdata journal files are not used by metric searches. However, after the optimizations introduced in 8.0, rawdata journal files take up a large amount of the volume of a metrics index bucket. In version 8.1 and later, the Splunk software removes these files from metric index buckets when they roll from hot to warm by default. The software removes the file from /rawdata directory in the bucket and replaces it with a dummy journal file that has no usable data other than the journal header. This reduces the amount of space that metric indexes take up on disk.

This does not apply to metric indexes that have replication enabled (repFactor = auto in limits.conf) for indexer clustering. For more information, or to learn how to disable rawdata journal removal for a specific index, see the description of the metric.stubOutRawdataJournal setting in limits.conf.spec.

Metric data points containing metric names that are empty or composed entirely of white spaces cannot be indexed or searched upon

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.1

In version 8.1 and later, Splunk Enterprise cannot index metric data points containing metric names that are empty or composed entirely of white spaces. This can happen with ingested metrics data as well as event data that is converted to metrics data through the log-to-metrics sourcetype or some other method.

After the upgrade, Splunk Enterprise searches cannot return metric data points containing empty or white-spaced metric names, if those metric data points were indexed in a previous version of Splunk.

StatsD metric data inputs now produce single-measurement metric data points by default

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0.3

For ease of use, by default Splunk Enterprise version 8.0.3 and later converts StatsD metric data into single-measurement metric data points that have one key-value pair for the metric name and one key-value pair for the measurement value. If, prior to 8.0.3, you used StatsD inputs that converted their data to metric data points that could carry multiple measurements, you need to add STATSD_EMIT_SINGLE_MEASUREMENT_FORMAT=false to a stanza for the metric source type in props.conf.

Streaming metric alerts are not available on indexer clusters that run version 7.2 and lower

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0

Splunk Enterprise version 8.0 introduces streaming metric alerts, which are evaluated on a continuous basis, and which can reduce the load on your system by enabling similar alerts to share the same search process. There is no support for streaming metric alerts on indexer clusters that run version 7.2 or lower. If you have an indexer cluster and a search head cluster, and you upgrade the search head cluster to version 8.x, you lose the ability to trigger streaming metric alerts for that search head cluster. If you need to retain this ability, upgrade the indexer cluster to version 8.x first.

You can no longer use wildcards in real-time searches that use the mstats command

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.0

Beginning with Splunk Enterprise version 8.0, you can no longer use wildcards in real-time searches that contain the mstats command. If you have these kinds of searches, change them so that they no longer use wildcards of any kind, either in aggregation fields or metric names, prior to upgrading

Metrics indexing and search is now case-sensitive

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0

The indexing and search schema for metrics data is case-sensitive as of Splunk Enterprise 8.0. Instances of Splunk Enterprise that are lower than 8.0 might encounter problems retrieving results when searching instances that run version 8.0 or higher.

Older, deprecated configuration file settings for knowledge bundle replication have been removed

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.0

Many older, previously deprecated settings under the [replicationSettings] stanza in distsearch.conf have been removed. If your configuration uses these settings, consider changing them before you start an upgrade.

Configuration file settings for specifying the mounted bundle option for knowledge bundle replication have changed

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 8.0

The distsearch.conf setting for specifying the mounted bundle option for knowledge bundle replication has changed. Previously, the setting was shareBundles=false. The new setting is replicationPolicy=mounted. If the upgrade process finds the old setting in your configuration files, it converts it automatically to the new setting, so no manual intervention is required on your part.

The Django framework and all its associated components have been removed from the Splunk platform. Apps and dashboards that use this framework will not function anymore. Before you upgrade, confirm that apps and dashboards you regularly use no longer use Django, as an upgrade will render them inoperable.

Updated field alias behavior might result in null values

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 7.2

When you upgrade to a version of Splunk Enterprise that is 7.2 or higher, the behavior of certain field alias configurations changes. This behavior affects events where the alias field is already present before field alias processing takes place.

• If both the source field and the alias field are present in the event, the search head overwrites the value of the alias field to match the value of the source field.
• If the alias field is present in the event but the source field is missing or has a null value, the search head removes the alias field from the event.

This is how field aliasing should work in these situations. However, you might have searches that rely on the old field alias behavior. A new ASNEW syntax has been added to enable you to provide the pre-7.2 behavior to your field alias configurations.

The pre-7.2 field alias behavior enabled users to create sets of field alias configurations that matched multiple source fields with one alias field. You can use the ASNEW syntax to continue doing this. A better practice would be to use a calculated field that uses the coalesce function to create a new field that takes the value of one or more existing fields. This method lets you be explicit about ordering of input field values in the case of null fields. For example: EVAL-ip = coalesce(clientip,ipaddress).

See Field alias behavior change in the Release Notes for further details.

Splunk Enterprise meters metric data points under 150 bytes by volume on a scale that is similar to the scale used for event data

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 7.3

There is a change to the way that Splunk Enterprise meters metrics data in version 7.3. Prior to this release, all metric data points counted against the license as if they were a flat 150 bytes. As of 7.3, Splunk Enterprise meters each metric data point that measures less than 150 bytes by volume on a scale that is similar to the scale used for event data. This scale is capped at 150 bytes. Metric events with volumes over 150 bytes are metered as if they are only 150 bytes. As a result, license usage for such events might be lower than in prior release.

For additional information on licensing, see How Splunk Enterprise licensing works in the Admin Manual.

The tstats command now reports the actual number of matched events in an index bucket as the scan count of those events

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 7.3

The tstats search command, which performs statistical queries on indexed fields in tsidx files, has updated behavior with regard to the number of events it finds in a bucket.

Previously, the command reported the scan count of events in a bucket. After an upgrade, the command now reports the actual number of matched events in a bucket as the scan count. If you use tstats for searches, you might see scan counts fall significantly. This does not indicate incorrect results. -->

Considerations for new features

Splunk has introduced the following new features in this version of Splunk Enterprise. You might need to perform some configuration after an upgrade to enable and take advantage of these features.

New configuration settings for importing structured data files

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.0

Splunk Enterprise 8.0.0 and higher has added some new settings for configuring the indexing of structured data. Previously, the structured data processor converted characters that it encountered in header field names that were neither alphanumeric nor a space into underscores. With the new HEADER_FIELD_ACCEPTABLE_SPECIAL_CHARACTERS setting in props.conf, you can control which characters the processor accepts as valid in header field names. Some characters might cause the processor to malfunction, so if you import lots of structured data and want to use the feature, use a test index to confirm that the processor imports the data in the way you want.

Learn about known upgrade issues

To learn about any additional upgrade issues for Splunk Enterprise, see the Known Issues - Upgrade Issues page in the Release Notes.