Upgrade pre-9.2 deployment servers

Aspects of the deployment server have been significantly enhanced in Splunk Enterprise version 9.2 to improve performance and manageability. In addition, the improvements enable multiple deployment servers to coordinate their activities in a deployment server cluster, as described in Implement a deployment server cluster.

Because of these architectural improvements, deployment server upgrades that span the 9.2 release automatically undergo a number of changes to implement these improvements. For standalone deployment servers, no action is necessary on the part of the user beyond the normal process of upgrading a Splunk Enterprise instance, as the changes are implemented in an entirely automatic fashion. In addition, deployment clients, including pre-9.2 clients, continue to operate seamlessly with the updated deployment servers.

However, if you examine the standalone deployment server's directories, you will notice some differences. In particular, there is a new system-generated app, etc/apps/SplunkDeploymentServerConfig, which contains configuration files necessary to the proper functioning of the deployment server. Do not alter this directory or its files in any way. Note that this app is not a deployment app and so does not reside in etc/deployment-apps.

In addition, the system places new configurations in savedsearches.conf and macros.conf. Do not edit these system-generated configurations.

There are also some new logs generated by the deployment server in response to client phone home activities. These are placed in the client_events directory, new in version 9.2.

Possible issues with upgrade

Data not appearing in forwarder management UI

This problem can occur in Splunk Enterprise 9.2 or higher if your deployment server forwards its internal logs to a standalone indexer or to the peer nodes of an indexer cluster. This issue can occur after an upgrade or in a new installation of 9.2 or higher. To rectify, add these settings to outputs.conf on the deployment server:

[indexAndForward]
index = true
selectiveIndexing = true     

If you add these settings post-upgrade or post-installation, you might need to restart the deployment server.

Indexers require new internal deployment server indexes

The deployment server uses several internal indexes new in version 9.2. These indexes are included in all indexers at the 9.2 level and higher, but if you try to forward data from those indexes to a pre-9.2 indexer, problems can result.

If you forward data to your indexer tier, create these new internal deployment server indexes in indexes.conf on any pre-9.2 indexers in your environment:

[_dsphonehome]
[_dsclient]
[_dsappevent]

If the indexers are at version 9.2 or higher, they are already configured with those indexes.

If you add those indexes to peer nodes on an indexer cluster, be sure to set repFactor = auto, as you must for all peer node indexes.

Data does not appear when forwarded through an intermediate forwarder

This problem can occur if your deployment server forwards its internal index data through an intermediate forwarder to a standalone indexer or to the peer nodes of an indexer cluster. To rectify, add this setting to outputs.conf on the intermediate forwarder:

[tcpout] 
forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker|_dsclient|_dsphonehome|_dsappevent)

If you specify the configuration within a deployment app and use the deployment server to deploy the app to the affected intermediate forwarders, you can later uninstall the app when the intermediate forwarders are upgraded to a future release that incorporates the update.