Upgrade the universal forwarder

See the following instructions to upgrade the Windows universal forwarder or the *nix universal forwarder:

• Upgrade the Windows universal forwarder
• Upgrade the *nix universal forwarder

If you are planning to upgrade the universal forwarder to version 8.2, see About upgrading to 8.2 READ THIS FIRST in the ''Installation Manual'' first.

Upgrade the Windows universal forwarder

When you upgrade a universal forwarder, the installer updates the software without changing its configuration. You must make any necessary configuration changes after you complete the upgrade. A deployment server can assist in the configuration update process.

There are several forwarder upgrade scenarios:

• You can upgrade a single forwarder with the GUI installer
• You can upgrade a single forwarder with the command line installer
• You can perform a remote upgrade of a group of forwarders (good for deployments of any size)

As best practice when upgrading a Windows universal forwarder on Splunk Cloud Platform, run the most recent forwarder version, even if the forwarder is a higher version number than your Splunk Cloud Platform environment.

When you upgrade on Windows, make sure to stop Splunk. If Splunk is running during upgrade, the upgrade fails with error "ERROR: In order to migrate, Splunkd must not be running." "

Confirm that an upgrade is necessary

Begin by checking the forwarder compatibility. To determine if you need to upgrade your forwarder version to remain in support or use specific features, see the appropriate topic for your deployment:

• Splunk Cloud Platform: Supported forwarder versions in the Splunk Cloud Platform Service Description.
• Splunk Enterprise: Compatibility between forwarders and Splunk Enterprise indexers in the Splunk Products Version Compatibility Matrix.

If your forwarders are on the same major release of Splunk software as the indexers, they are compatible. However, you might need an upgrade to a different minor release due to a technical issue in a specific feature. Before upgrading forwarders, review the Known Issues and Fixed Issues.

You must perform any platform architecture changes manually

You cannot upgrade a 32-bit version of the universal forwarder with a 64-bit universal forwarder installer. To upgrade from 32-bit to 64-bit, follow these instructions:

1. Stop splunkd if it is running.
2. Back up your configurations, including any apps or add-ons (in %SPLUNK_HOME%\etc\apps). Also back up the checkpoint files located in %SPLUNK_HOME%\var\lib\splunk\modinputs.
3. Uninstall the existing 32-bit forwarder, as described in Uninstall the universal forwarder.
4. Install the 64-bit forwarder, as described in Install the universal forwarder from an installer.
5. Restore apps, configurations and checkpoints by copying them to the appropriate directories:

%SPLUNK_HOME%\etc\system\local for configuration files.

%SPLUNK_HOME%\etc\apps for apps and add-ons.

%SPLUNK_HOME%\var\lib\splunk\modinputs for checkpoint files.

Back your files up

Before you perform an upgrade, back up configuration files. See Back up configuration information in the Splunk Enterprise Admin manual.

There is no means of downgrading to a previous version. If you need to revert to an older forwarder release, uninstall the current version and reinstall the older release. Note that depending on the data that you are monitoring a complete uninstall/reinstall can cause re-ingestion of any historic logs being monitored.

Upgrade a single forwarder using the GUI installer

You can upgrade a single forwarder with the GUI installer. The installer stops the forwarder as part of the upgrade process.

1. Stop splunkd if it is running.
2. Download the new MSI file from the universal forwarder download page.
3. Double-click the MSI file. The installer displays the "Accept license agreement" panel.
4. Accept the license agreement and click "Install." The installer upgrades the forwarder, retains the existing configuration, and starts automatically when you complete the installation.

The installer puts a log of upgrade changes in the %TEMP% directory (This is usually the C:\TEMP directory but can be different based on your Windows machine configuration.) It also reports any errors in the Application Event Log.

Upgrade a single forwarder using the command line

You can upgrade a single forwarder by running the command line installer.

You cannot make configuration changes during an upgrade. The installer ignores any command line flags that you specify except for the AGREETOLICENSE flag.

1. Stop splunkd if it is running.
2. Download the new MSI file from the Splunk universal forwarder download page.
3. Run msiexec.exe to Install the universal forwarder from the command line.
   
   • For 32-bit platforms, use splunkuniversalforwarder-<...>-x86-release.msi.

         msiexec.exe /i splunkuniversalforwarder-<...>-x86-release.msi [AGREETOLICENSE=Yes /quiet]
   

   • For 64-bit platforms, use splunkuniversalforwarder-<...>-x64-release.msi.

         msiexec.exe /i splunkuniversalforwarder-<...>-x64-release.msi [AGREETOLICENSE=Yes /quiet]
   

   The value of <...> varies according to the particular release, for example, splunkuniversalforwarder-6.3.0-aa7d4b1ccb80-x64-release.msi.

4. Wait for the upgrade to complete. The forwarder starts automatically when you complete the installation.

The installer puts a log of upgrade changes in the %TEMP% directory. It also reports any errors in the Application Event Log.

Perform a remote upgrade of one or more forwarders

You can use a deployment tool such as Group Policy or System Center Configuration Manager to distribute the forwarder software among a group of forwarders in your environment. You might want to test the upgrade locally on one machine before performing a remote upgrade across all your forwarders.

The Splunk Enterprise deployment server cannot distribute the universal forwarder, only its apps and configurations. Do not attempt to use deployment server to distribute universal forwarders.

1. Download the new MSI file from the Splunk universal forwarder download page.
2. Load the MSI into your deployment tool. In the tool, specify the command line as follows.
   

      msiexec.exe /i splunkuniversalforwarder-<...>.msi AGREETOLICENSE=Yes /quiet
   

3. Start the deployment with your deployment tool.
4. Use the deployment monitor to verify that the universal forwarders function properly.

Upgrade the *nix universal forwarder

You have several scenarios for upgrading a *nix universal forwarder:

• Upgrade a single forwarder manually.
• Perform a remote upgrade of a group of forwarders. (Use this option for deployments of any size)

As best practice when upgrading a *nix universal forwarder on Splunk Cloud Platform, run the most recent forwarder version, even if the forwarder is a higher version number than your Splunk Cloud Platform environment.

Confirm that an upgrade is necessary

Begin by checking the forwarder compatibility. To determine if you need to upgrade your forwarder version to remain in support or use specific features, see the appropriate topic for your deployment:

• Splunk Cloud Platform: Supported forwarder versions in the Splunk Cloud Platform Service Description
• Splunk Enterprise: Compatibility between forwarders and Splunk Enterprise indexers in the Splunk Products Version Compatibility Matrix.

If your forwarders are on the same major release of Splunk software as the indexers, they are compatible. However, you might need an upgrade to a different minor release due to a technical issue in a specific feature. Before upgrading forwarders, review the Known Issues and Fixed Issues.

Back your files up

Before you perform the upgrade, back up your configuration files. See Back up configuration information in the Splunk Enterprise Admin Manual.

If you need to revert to an older forwarder release, uninstall the upgrade and reinstall the older release.

Confirm that you do not have scripts in place to auto-start forwarders. If you do, disable such scripts for now. You can re-enable them later, after the upgrade.

How upgrading works

After you perform the installation of the new forwarder, you must restart it for any changes to take effect. You can run the migration preview utility at that time to see what will change before the files are updated. If you choose to view the changes before proceeding, the forwarder writes the proposed changes to $SPLUNK_HOME/var/log/splunk/migration.log.<timestamp>

Upgrade a single forwarder

There are several packages that you can use to upgrade a universal forwarder. Tar files and pre-built package such as an .rpm, .deb, or .dmg file are available depending on the operating system.

If you use a .tar file to upgrade a forwarder, expand it into the same directory with the same ownership as the existing universal forwarder instance. This overwrites and replaces matching files but does not remove unique files.

If you use an RPM file, use the RPM package manager (rpm -U <splunk_package_name>.rpm) from a shell prompt to perform the upgrade.

If you use a .dmg file (on MacOS), double-click it and follow the instructions. After the installation starts, specify the same installation directory as your existing installation.

On hosts that run AIX, do not use the AIX version of tar to unarchive a tar file during an upgrade. Use the GNU version of tar instead. This version comes with the AIX Toolbox for Linux Applications package that comes with a base AIX installation. If your AIX does not come with this package installed, you can download it from IBM. See IBM AIX Toolbox download information.

1. Stop the forwarder.

     $SPLUNK_HOME/bin/splunk stop

2. Install the universal forwarder package directly over the existing deployment.

As best practice when upgrading a *nix universal forwarder on Splunk Cloud Platform, run the most recent forwarder version, even if the forwarder is a higher version number than your Splunk Cloud Platform environment.

3. Start the forwarder again.

     $SPLUNK_HOME/bin/splunk start

The forwarder displays the following:

This appears to be an upgrade of Splunk.
--------------------------------------------------------------------------------
Splunk has detected an older version of Splunk installed on this machine. To
finish upgrading to the new version, Splunk's installer will automatically
update and alter your current configuration files. Deprecated configuration
files will be renamed with a .deprecated extension.
You can choose to preview the changes that will be made to your configuration
files before proceeding with the migration and upgrade:
If you want to migrate and upgrade without previewing the changes that will be
made to your existing configuration files, choose 'y'.
If you want to see what changes will be made before you proceed with the
upgrade, choose 'n'.
Perform migration and upgrade without previewing configuration changes? [y/n]

4. Choose whether you want to run the migration preview script to see what changes will be made to your existing configuration files, or proceed with the migration and upgrade right away. If you choose to view the expected changes, the script provides a list of those changes.

5. Once you have reviewed these changes and are ready to proceed with migration and upgrade, run $SPLUNK_HOME/bin/splunk start again.

You can complete the last three steps in one line.

• To accept the license and view the expected changes (answer 'n') before continuing the upgrade:

      $SPLUNK_HOME/bin/splunk start --accept-license --answer-no

• For Linux upgrade you must use sudo to upgrade as the root user.

sudo $SPLUNK_HOME/bin/splunk start --accept-license --answer-no

• To accept the license and begin the upgrade without viewing the changes (answer 'y'):

      $SPLUNK_HOME/bin/splunk start --accept-license --answer-yes

Perform a remote upgrade

To perform a remote upgrade, first perform an upgrade on a test machine. Then, create a script to automate the upgrade on remote machines. You can use the following script, but you might need to modify the script to meet the needs of an upgrade.

1. Upgrade the universal forwarder on a test machine, as described in Install a nix universal forwarder.

2. Create a script wrapper for the upgrade commands.

3. Run the script on representative target machines to verify that it works with all required shells.

4. Execute the script against the desired set of hosts.

#!/bin/sh

# This script provides an example of how to deploy the universal forwarder
# to many remote hosts via ssh and common Unix commands.
#
# Note that this script will only work unattended if you have SSH host keys
# setup & unlocked.
# To learn more about this subject, do a web search for "openssh key management".


# ----------- Adjust the variables below -----------

# Populate this file with a list of hosts that this script should install to,
# with one host per line.  You may use hostnames or IP addresses, as
# applicable.  You can also specify a user to login as, for example, "foo@host".
#
# Example file contents:
# server1
# server2.foo.lan
# you@server3
# 10.2.3.4

HOSTS_FILE="/path/to/splunk.install.list"

# This is the path to the tar file that you wish to push out.  You may
# wish to make this a symlink to a versioned tar file, so as to minimize
# updates to this script in the future.

SPLUNK_FILE="/path/to/splunk-latest.tar.gz"

# This is where the tar file will be stored on the remote host during
# installation.  The file will be removed after installation.  You normally will
# not need to set this variable, as $NEW_PARENT will be used by default.
#
# SCRATCH_DIR="/home/your_dir/temp"

# The location in which to unpack the new tar file on the destination
# host.  This can be the same parent dir as for your existing 
# installation (if any).  This directory will be created at runtime, if it does
# not exist.

NEW_PARENT="/opt"

# After installation, the forwarder will become a deployment client of this
# host.  Specify the host and management (not web) port of the deployment server
# that will be managing these forwarder instances.  If you do not wish to use
# a deployment server, you may leave this unset.
#
# DEPLOY_SERV="splunkDeployMaster:8089"

# A directory on the current host in which the output of each installation
# attempt will be logged.  This directory need not exist, but the user running
# the script must be able to create it.  The output will be stored as
# $LOG_DIR/<[user@]destination host>.  If installation on a host fails, a
# corresponding file will also be created, as
# $LOG_DIR/<[user@]destination host>.failed.

LOG_DIR="/tmp/splunkua.install"

# For conversion from normal Splunk Enterprise installs to the universal forwarder:
# After installation, records of progress in indexing files (monitor)
# and filesystem change events (fschange) can be imported from an existing
# Splunk Enterprise (non-forwarder) installation.  Specify the path to that installation here.
# If there is no prior Splunk Enterprise instance, you may leave this variable empty ("").
#
# NOTE: THIS SCRIPT WILL STOP THE SPLUNK ENTERPRISE INSTANCE SPECIFIED HERE.
#
# OLD_SPLUNK="/opt/splunk"

# If you use a non-standard SSH port on the remote hosts, you must set this.
# SSH_PORT=1234

# You must remove this line, or the script will refuse to run.  This is to
# ensure that all of the above has been read and set. :)

UNCONFIGURED=1

# ----------- End of user adjustable settings -----------


# helpers.

faillog() {
  echo "$1" >&2
}

fail() {
  faillog "ERROR: $@"
  exit 1
}

# error checks.

test "$UNCONFIGURED" -eq 1 && \
  fail "This script has not been configured.  Please see the notes in the script."
test -z "$HOSTS_FILE" && \
  fail "No hosts configured!  Please populate HOSTS_FILE."
test -z "$NEW_PARENT" && \
  fail "No installation destination provided!  Please set NEW_PARENT."
test -z "$SPLUNK_FILE" && \
  fail "No splunk package path provided!  Please populate SPLUNK_FILE."
if [ ! -d "$LOG_DIR" ]; then
  mkdir -p "$LOG_DIR" || fail "Cannot create log dir at \"$LOG_DIR\"!"
fi

# some setup.

if [ -z "$SCRATCH_DIR" ]; then
  SCRATCH_DIR="$NEW_PARENT"
fi
if [ -n "$SSH_PORT" ]; then
  SSH_PORT_ARG="-p${SSH_PORT}"
  SCP_PORT_ARG="-P${SSH_PORT}"
fi

NEW_INSTANCE="$NEW_PARENT/splunkforwarder" # this would need to be edited for non-UA...
DEST_FILE="${SCRATCH_DIR}/splunk.tar.gz"

#
#
# create script to run remotely.
#
#

REMOTE_SCRIPT="
  fail() {
    echo ERROR: \"\$@\" >&2
    test -f \"$DEST_FILE\" && rm -f \"$DEST_FILE\"
    exit 1
  }
"

###   try untarring tar file.
REMOTE_SCRIPT="$REMOTE_SCRIPT
  (cd \"$NEW_PARENT\" && tar -zxf \"$DEST_FILE\") || fail \"could not untar /$DEST_FILE to $NEW_PARENT.\"
"

###   setup seed file to migrate input records from old instance, and stop old instance.
if [ -n "$OLD_SPLUNK" ]; then
  REMOTE_SCRIPT="$REMOTE_SCRIPT
    echo \"$OLD_SPLUNK\" > \"$NEW_INSTANCE/old_splunk.seed\" || fail \"could not create seed file.\"
    \"$OLD_SPLUNK/bin/splunk\" stop || fail \"could not stop existing splunk.\"
  "
fi

###   setup deployment client if requested.
if [ -n "$DEPLOY_SERV" ]; then
  REMOTE_SCRIPT="$REMOTE_SCRIPT
    \"$NEW_INSTANCE/bin/splunk\" set deploy-poll \"$DEPLOY_SERV\" --accept-license --answer-yes \
      --auto-ports --no-prompt || fail \"could not setup deployment client\"
  "
fi

###   start new instance.
REMOTE_SCRIPT="$REMOTE_SCRIPT
  \"$NEW_INSTANCE/bin/splunk\" start --accept-license --answer-yes --auto-ports --no-prompt || \
    fail \"could not start new splunk instance!\"
"

###   remove downloaded file.
REMOTE_SCRIPT="$REMOTE_SCRIPT
  rm -f "$DEST_FILE" || fail \"could not delete downloaded file $DEST_FILE!\"
"

#
#
# end of remote script.
#
#

exec 5>&1 # save stdout.
exec 6>&2 # save stderr.

echo "In 5 seconds, will copy install file and run the following script on each"
echo "remote host:"
echo
echo "===================="
echo "$REMOTE_SCRIPT"
echo "===================="
echo
echo "Press Ctrl-C to cancel..."
test -z "$MORE_FASTER" && sleep 5
echo "Starting."

# main loop.  install on each host.

for DST in `cat "$HOSTS_FILE"`; do
  if [ -z "$DST" ]; then
    continue;
  fi

  LOG="$LOG_DIR/$DST"
  FAILLOG="${LOG}.failed"
  echo "Installing on host $DST, logging to $LOG."

  # redirect stdout/stderr to logfile.
  exec 1> "$LOG"
  exec 2> "$LOG" 

  if ! ssh $SSH_PORT_ARG "$DST" \
      "if [ ! -d \"$NEW_PARENT\" ]; then mkdir -p \"$NEW_PARENT\"; fi"; then
    touch "$FAILLOG"
    # restore stdout/stderr.
    exec 1>&5 
    exec 2>&6
    continue
  fi

  # copy tar file to remote host.
  if ! scp $SCP_PORT_ARG "$SPLUNK_FILE" "${DST}:${DEST_FILE}"; then
    touch "$FAILLOG"
    # restore stdout/stderr.
    exec 1>&5 
    exec 2>&6
    continue
  fi
    
  # run script on remote host and log appropriately.
  if ! ssh $SSH_PORT_ARG "$DST" "$REMOTE_SCRIPT"; then
    touch "$FAILLOG" # remote script failed.
  else
    test -e "$FAILLOG" && rm -f "$FAILLOG" # cleanup any past attempt log.
  fi

  # restore stdout/stderr.
  exec 1>&5 
  exec 2>&6

  if [ -e "$FAILLOG" ]; then
    echo "    FAILED  "
  else
    echo "      SUCCEEDED"
  fi
done

FAIL_COUNT=`ls "${LOG_DIR}" | grep -c '\.failed$'`
if [ "$FAIL_COUNT" -gt 0 ]; then
  echo "There were $FAIL_COUNT remote installation failures."
  echo "  ( see ${LOG_DIR}/*.failed )"
else
  echo
  echo "Done."
fi

# Voila.