Migrate from a search head pool to a search head cluster

You can migrate the settings from a search head pool to a search head cluster. You cannot migrate the search head instances themselves, however. You must use new instances when enabling search head cluster members.

The migration procedure varies somewhat depending on whether you are migrating to a new cluster or to a cluster that is already running.

Types of objects to migrate

There are two types of objects to migrate:

• Custom app configurations. These originate under etc/apps on the search head pool shared storage.

• Private user configurations. These originate under etc/users on the search head pool shared storage.

In both cases, you copy the relevant directories from the search head pool shared storage to the search head cluster's deployer. You then use the deployer to propagate these directories to the cluster.

The deployer pushes the configurations to the cluster, using a different method for each type. Post-migration, the app configurations obey different rules from the user configurations.

For information on where deployed settings reside on the cluster members, see "Where deployed configurations live on the cluster members."

Custom app configurations

When it migrates an app's custom settings, the deployer places them in default directories on the cluster members. This includes any runtime changes that were made while the apps were running on the search head pool.

Because users cannot change settings in default directories, this means that users cannot perform certain runtime operations on these migrated entities:

• Delete. Users cannot delete any migrated entities.
• Move. Users cannot move these settings from one app to another.
• Change sharing level. Users cannot change sharing levels. For example, a user cannot change sharing from app-level to private.

Cluster users can override existing attributes by editing entities in place. Runtime changes get put in the local directories on the cluster members. Local directories override default directories, so the changes override the default settings.

Private user configurations

The deployer copies user configurations to the captain only. The captain then replicates the settings to all the cluster members through its normal method for replicating configurations, as described in "Configuration updates that the cluster replicates."

Unlike custom app configurations, the user configurations reside in the normal user locations on the cluster members and can later be deleted, moved, and so on. They behave just like any runtime settings created by cluster users through Splunk Web.

When you migrate user configurations to an existing search head cluster, the deployer respects attributes that already exist on the cluster. It does not overwrite any existing attributes within existing stanzas.

For example, say the cluster members have an existing file $SPLUNK_HOME/etc/users/admin/search/local/savedsearches.conf containing this stanza:

[my search]
search = index=_internal | head 1

and on the deployer, there's the file $SPLUNK_HOME/etc/shcluster/users/admin/search/local/savedsearches.conf with these stanzas:

[my search]
search = index=_internal | head 10
enableSched = 1

[my other search]
search = FOOBAR

This will result in a final merged configuration on the members:

[my search]
search = index=_internal | head 1
enableSched = 1

[my other search]
search = FOOBAR

The [my search] stanza, which already existed on the members, keeps the existing setting for its search attribute, but adds the migrated setting for the enableSched attribute, because that attribute did not already exist in the stanza. The [my other search] stanza, which did not already exist on the members, gets added to the file, along with its search attribute.

Note: Splunk does not support migration of per-user search history files.

Do not migrate default apps

When you migrate apps to the search head cluster, do not migrate any default apps, that is, apps that ship with Splunk Enterprise, such as the search app. If you push default apps to cluster members, you overwrite the version of those apps residing on the members, and you do not want to do this.

You can, however, migrate custom settings from a default app by moving them to a new app and exporting them globally.

Each of the migration procedures in this topic includes a step for migrating default app custom settings.

Migrate to a new search head cluster

To migrate settings from a search head pool to a new search head cluster:

1. Follow the procedure for deploying any new search head cluster. Specify the deployer location at the time that you initialize the cluster members. See "Deploy a search head cluster."

Caution: You must deploy new instances. You cannot reuse existing search heads.

2. Copy the etc/apps and etc/users directories on the shared storage location in the search head pool to the distribution directory on the deployer instance. The distribution directory is located at $SPLUNK_HOME/etc/shcluster.

For details on the distribution directory file structure, see "Where to place the configuration bundle on the deployer."

3. If you want to migrate custom settings from a default app, you can move them to a new app and export them globally. For example, to migrate settings from the search app :

a. Copy the .../search/local directory in the distribution directory to a new app directory, such as search_migration_app, in the distribution directory. Do not name this new app "search."

b. Export the settings globally to make them available to all apps, including the search app. To do this, create a .../search_migration_app/metadata/local.meta file and populate it with the following content:

   []
   export=system

See the default.meta specification file for details.

4. If $SPLUNK_HOME/etc/shcluster/apps contains any default apps, such as the search app, you must delete them now. Do not push them to the cluster members. If you do, they will overwrite the versions of those apps already on the members.

5. Run the splunk apply shcluster-bundle command on the deployer to push the configuration bundle to the cluster. See "Push the configuration bundle."

The deployer pushes etc/apps directly to the cluster members. It pushes etc/users to the captain, which asynchronously replicates the settings to the other cluster members.

Note: If you point the cluster members at the same set of search peers previously used by the search head pool, the cluster will need to rebuild any report acceleration summaries or data model summaries resident on the search peers. It does this automatically. It does not, however, automatically remove the old set of summaries.

Migrate to an existing search head cluster

To migrate settings from a search head pool to an existing search head cluster:

1. Copy the /etc/apps and etc/users directories on the shared storage location in the search head pool to a temporary directory where you can edit them.

2. If you want to migrate custom settings from a default app, you can move them to a new app and export them globally. For example, to migrate settings from the search app :

a. Copy the .../search/local directory in the temporary directory to a new app directory, such as search_migration_app, in the temporary directory. Do not name this new app "search."

b. Export the settings globally to make them available to all apps, including the search app. To do this, create a .../search_migration_app/metadata/local.meta file and populate it with the following content:

   []
   export=system

See the default.meta specification file for details.

3. In the temporary directory, delete these subdirectories:

• Any default apps, such as the search app. Do not push default apps to the cluster members. If you do, they will overwrite the versions of those apps already on the members.

• Any apps already existing in the deployer's distribution directory. Otherwise, the versions from the search head pool will overwrite the versions already on the members.

4. Copy the remaining subdirectories from the temporary location to the distribution directory on the deployer, located at $SPLUNK_HOME/etc/shcluster. Leave any subdirectories already in the distribution directory unchanged.

For details on the distribution directory file structure, see "Where to place the configuration bundle on the deployer."

5. Run the splunk apply shcluster-bundle command on the deployer to push the configuration bundle, including the migrated settings, to the cluster. See "Push the configuration bundle."

The deployer pushes etc/apps directly to the cluster members. It pushes etc/users to the captain, which asynchronously replicates the settings to the other cluster members.

Note: If you point the cluster members at the same set of search peers previously used by the search head pool, the cluster will need to rebuild any report acceleration summaries or data model summaries resident on the search peers. It does this automatically. It does not, however, automatically remove the old set of summaries.

Search head clustering and mounted bundles

For most types of deployments, including search head clustering, Splunk recommends that you use normal bundle replication, rather than mounted bundles with shared storage.

As a result of changes to bundle replication made in the 5.0 timeframe, such as the introduction of delta-based replication and improvements in streaming, the practical use case for mounted bundles is now extremely limited. In most cases, mounted bundles make little difference in the amount of network traffic or the speed at which bundle changes get distributed to the search peers. At the same time, they add significant management complexity, particularly when combined with shared storage. Because of delta-based replication, even if your configurations contain large files, normal bundle replication entails little ongoing replication cost, as long as those files rarely change.