Implement a deployment server cluster

You can scale your deployment server solution by implementing a deployment server cluster consisting of multiple deployment servers that share their configurations and activities.

Deployment server cluster architecture

A deployment server cluster consists of multiple deployment servers that share configurations and activities via a shared drive. The deployment clients connect to the pool of deployment servers through a load balancer or DNS mapping.

This diagram illustrates the basic architecture for a deployment server cluster that uses a load balancer to connect with the deployment clients. The DNS solution just substitutes DNS mapping in place of the load balancer.

The deployment clients point to the load balancer, rather than directly to a deployment server.

The shared drive contains the two directories shared by all deployment servers in the cluster: the deployment_apps directory and a log directory named client_events.

These directories must be mounted on each deployment server.

The deployment app bundle contains the usual set of user-defined deployment apps, as well as a system app, _splunk_ds_info. This app is new in Splunk Enterprise version 9.2. It includes the set of server class configuration files, which are shared across deployment servers.. Do not directly edit the contents of this directory.

The shared log file directory is named client_events. It is new in version 9.2, and it tracks the clients' phone home events through log entries.

Deploy a deployment server cluster

To deploy a deployment server cluster:

1. Set up the shared drive.
2. Install and configure the deployment servers.
3. Set up a load balancer or DNS mapping.

System requirements

Each deployment server in the cluster requires its own dedicated Splunk Enterprise instance, version 9.2 or higher.

The deployment clients connecting to the cluster can run a pre-9.2 version, since their interactions are the same whether connecting to a cluster or to a standalone deployment server. They simply need to connect to a load balancer or DNS, rather than directly to a deployment server.

To calculate the maximum number of deployment clients that the cluster can support, multiply the number of deployment servers by 25K. Therefore, a cluster of 3 deployment servers can service a maximum of 75K clients.

The maximum number of deployment servers in a cluster is limited to 3.

Requirements for the shared drive and load balancer or DNS are discussed in their respective sections.

New configuration files

If you examine the deployment server's directories, you will notice some differences compared to pre-9.2. In particular, there is an app, etc/apps/SplunkDeploymentServerConfig, which contains configuration files necessary to the proper functioning of the deployment server. Do not alter this directory or its files in any way. Note that this app is not a deployment app and so does not reside in etc/deployment-apps.

In addition, the system places new configurations in savedsearches.conf and macros.conf. Do not edit these system-generated configurations.

Set up a shared drive

The shared drive must be a network shared drive, such as an NFS host. The drive needs two high-level directories, one for deployment apps and another for log files. They must be readable/writeable by the splunk user on each deployment server.

The shared drive does not need to be dedicated to serving the deployment servers, but its use for the cluster requires sufficient space. Specifically, the log file directory needs approximately 100MB per deployment server for log files, possibly more if you choose to increase your logging beyond the defaults. The deployment app directory size depends on the number and size of deployment apps that your deployment servers need to manage.

When you create the two directories on the shared drive, you can name them as you wish. However, you must mount the directories on each deployment server with the following local paths and directory names:

• $SPLUNK_HOME/etc/deployment-apps
• $SPLUNK_HOME/var/log/client_events

You will initially be placing any existing deployment apps in the mounted deployment-apps directory on one of the deployment servers. Since this directory is mounted to the shared drive, the deployment apps will be available to all deployment servers.

When one of the deployment servers first updates its serverclass.conf file and then reloads, the system will create a _splunk_ds_info directory in the mounted deployment-apps directory and place the serverclass.conf file in it.

Do not manually edit the _splunk_ds_info directory.

Logs for client phone home events are placed in the mounted client_events directory.

Configure deployment servers

1. If you are planning to convert a standalone deployment server to a cluster member, back up its deployment-apps directory and serverclass.conf file before performing the upgrade.
2. Install Splunk Enterprise instances, 9.2 or higher, and configure them as deployment servers in the usual way. If you are incorporating an existing standalone deployment server, upgrade it to 9.2 or higher.
3. On each deployment server, add the configuration syncMode = sharedDir to the serverclass.conf file. This setting indicates that the deployment server is part of a cluster and will be sharing the app bundle and client_events directories, as well as the set of server classes.
4. On each deployment server, set up mounts to the directories on the shared drive.

   If incorporating an existing deployment server, first confirm that the $SPLUNK_HOME/var/log/client_events directory exists locally on the deployment server. If it doesn't, then create it before mounting the corresponding shared directory.

5. If incorporating an existing deployment server:
   • Move the backed-up deployment-apps directory to the deployment apps directory on the shared drive.
   • Run reload on the existing deployment server. This step is necessary in order to share its serverclass.conf file across all deployment servers.

Choose load balancer or DNS mapping

For efficient use of the deployment server cluster, insert a third-party load balancer or DNS record between the deployment clients and the deployment servers. A load balancer is preferred because it can allow you to configure sticky sessions. It is recommended that you choose a load balancer that allows for sticky sessions and supports the REST-based health check API, described in the REST API Reference Manual: cluster/manager/ha_active_status.

If you want the deployment clients to tap into the pool of deployment servers, rather than always connecting to the same deployment server, you must update the clients' configurations to point to the load balancer or DNS record instead of directly to a deployment server. You do so on each client, by updating its targetUri setting under the [target-broker:deploymentServer] stanza in deploymentclient.conf and restarting the client. See Specify the deployment server.

You can update the targetUri setting on the clients over time. In the meantime, each client will interact directly with its configured deployment server. Any client interactions will be recorded in the shared log directory, so that the information is available to all deployment servers.

Add server classes

You can add or edit server classes in the usual way, locally on any deployment server. After making your server class changes, you must run reload on the deployment server to make the changes available to all deployment servers.

If you use the forwarder management UI to make the server class changes, the UI automatically triggers the reload when you save your changes.

Following the reload, the updated serverclass.conf file will be uploaded to _splunk_ds_info in the shared drive's deployment apps directory. When other deployment servers poll for changes, they will find the updated file and use it to overwrite their local serverclass.conf files. It can take up to 60 seconds for the deployment servers to sync server class updates.