Splunk® Enterprise

Forwarding Data

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Remotely deploy a *nix universal forwarder with a static configuration

One of the main ways to deploy multiple universal forwarders remotely is through scripting. You can also use deployment management tools such as yum and Puppet. This topic focuses on script deployment.

For information on how to install and configure a single universal forwarder, see "Deploy a nix universal forwarder manually". That topic explains how to install onto a wide variety of *nix platforms from a package or a tar file and how to configure (and optionally migrate) using the CLI.

Steps to deployment

Once you have downloaded the universal forwarder and have planned your deployment, as described in "Deployment overview", perform these steps:

1. Install and configure the universal forwarder on a test machine, as described in "Deploy a nix universal forwarder manually".

2. Test and tune the configuration.

3. Create a script wrapper for the installation and configuration commands.

4. Run the script on representative target machines to verify that it works with all required shells.

5. Execute the script against the desired set of hosts.

6. Use the deployment monitor to verify that the universal forwarders are functioning properly.

Create and execute the script

Once you've validated your installation and configuration process by testing a fully configured universal forwarder, you're ready to incorporate the process into a script.

Script requirements

You need to place the installation package or tar file in a network location accessible by the target machines. You can set this up so that the script pushes the file over to each target host, or you can place the file in a generally accessible location, such as an NFS mount.

The script is responsible for error reporting. Logging to Splunk either directly or via a flat file is recommended.

Sample script

Here's a sample script you can use as a starting point. Note that this is only an example of the type of script you could create for your deployment. The comments in the script provide some guidance on how to modify it for your needs; however, the script will likely require further modification, beyond that indicated by the comments.

Among other things, the script:

  • Deploys the forwarder's tar file to a list of hosts specified in a file that the HOST_FILE variable points to. You will need to provide this file, in the format specified in the script comments.
  • Specifies the location on each destination host where the tar file will get unpacked.
  • Specifies a Splunk Enterprise instance to serve as a deployment server that can subsequently manage and update the forwarders. This is an optional configuration step.
  • Starts the forwarder executable on each host.

The script is well commented; be sure to study it carefully before modifying it for your environment.

Here's the sample deployment script:


# This script provides an example of how to deploy the universal forwarder
# to many remote hosts via ssh and common Unix commands.
# Note that this script will only work unattended if you have SSH host keys
# setup & unlocked.
# To learn more about this subject, do a web search for "openssh key management".

# ----------- Adjust the variables below -----------

# Populate this file with a list of hosts that this script should install to,
# with one host per line.  You may use hostnames or IP addresses, as
# applicable.  You can also specify a user to login as, for example, "foo@host".
# Example file contents:
# server1
# server2.foo.lan
# you@server3


# This is the path to the tar file that you wish to push out.  You may
# wish to make this a symlink to a versioned tar file, so as to minimize
# updates to this script in the future.


# This is where the tar file will be stored on the remote host during
# installation.  The file will be removed after installation.  You normally will
# not need to set this variable, as $NEW_PARENT will be used by default.
# SCRATCH_DIR="/home/your_dir/temp"

# The location in which to unpack the new tar file on the destination
# host.  This can be the same parent dir as for your existing 
# installation (if any).  This directory will be created at runtime, if it does
# not exist.


# After installation, the forwarder will become a deployment client of this
# host.  Specify the host and management (not web) port of the deployment server
# that will be managing these forwarder instances.  If you do not wish to use
# a deployment server, you may leave this unset.
# DEPLOY_SERV="splunkDeployMaster:8089"

# A directory on the current host in which the output of each installation
# attempt will be logged.  This directory need not exist, but the user running
# the script must be able to create it.  The output will be stored as
# $LOG_DIR/<[user@]destination host>.  If installation on a host fails, a
# corresponding file will also be created, as
# $LOG_DIR/<[user@]destination host>.failed.


# For conversion from normal Splunk Enterprise installs to the universal forwarder:
# After installation, records of progress in indexing files (monitor)
# and filesystem change events (fschange) can be imported from an existing
# Splunk Enterprise (non-forwarder) installation.  Specify the path to that installation here.
# If there is no prior Splunk Enterprise instance, you may leave this variable empty ("").
# OLD_SPLUNK="/opt/splunk"

# If you use a non-standard SSH port on the remote hosts, you must set this.
# SSH_PORT=1234

# You must remove this line, or the script will refuse to run.  This is to
# ensure that all of the above has been read and set. :)


# ----------- End of user adjustable settings -----------

# helpers.

faillog() {
  echo "$1" >&2

fail() {
  faillog "ERROR: $@"
  exit 1

# error checks.

test "$UNCONFIGURED" -eq 1 && \
  fail "This script has not been configured.  Please see the notes in the script."
test -z "$HOSTS_FILE" && \
  fail "No hosts configured!  Please populate HOSTS_FILE."
test -z "$NEW_PARENT" && \
  fail "No installation destination provided!  Please set NEW_PARENT."
test -z "$SPLUNK_FILE" && \
  fail "No splunk package path provided!  Please populate SPLUNK_FILE."
if [ ! -d "$LOG_DIR" ]; then
  mkdir -p "$LOG_DIR" || fail "Cannot create log dir at \"$LOG_DIR\"!"

# some setup.

if [ -z "$SCRATCH_DIR" ]; then
if [ -n "$SSH_PORT" ]; then

NEW_INSTANCE="$NEW_PARENT/splunkforwarder" # this would need to be edited for non-UA...

# create script to run remotely.

  fail() {
    echo ERROR: \"\$@\" >&2
    test -f \"$DEST_FILE\" && rm -f \"$DEST_FILE\"
    exit 1

###   try untarring tar file.
  (cd \"$NEW_PARENT\" && tar -zxf \"$DEST_FILE\") || fail \"could not untar /$DEST_FILE to $NEW_PARENT.\"

###   setup seed file to migrate input records from old instance, and stop old instance.
if [ -n "$OLD_SPLUNK" ]; then
    echo \"$OLD_SPLUNK\" > \"$NEW_INSTANCE/old_splunk.seed\" || fail \"could not create seed file.\"
    \"$OLD_SPLUNK/bin/splunk\" stop || fail \"could not stop existing splunk.\"

###   setup deployment client if requested.
if [ -n "$DEPLOY_SERV" ]; then
    \"$NEW_INSTANCE/bin/splunk\" set deploy-poll \"$DEPLOY_SERV\" --accept-license --answer-yes \
      --auto-ports --no-prompt || fail \"could not setup deployment client\"

###   start new instance.
  \"$NEW_INSTANCE/bin/splunk\" start --accept-license --answer-yes --auto-ports --no-prompt || \
    fail \"could not start new splunk instance!\"

###   remove downloaded file.
  rm -f "$DEST_FILE" || fail \"could not delete downloaded file $DEST_FILE!\"

# end of remote script.

exec 5>&1 # save stdout.
exec 6>&2 # save stderr.

echo "In 5 seconds, will copy install file and run the following script on each"
echo "remote host:"
echo "===================="
echo "===================="
echo "Press Ctrl-C to cancel..."
test -z "$MORE_FASTER" && sleep 5
echo "Starting."

# main loop.  install on each host.

for DST in `cat "$HOSTS_FILE"`; do
  if [ -z "$DST" ]; then

  echo "Installing on host $DST, logging to $LOG."

  # redirect stdout/stderr to logfile.
  exec 1> "$LOG"
  exec 2> "$LOG" 

  if ! ssh $SSH_PORT_ARG "$DST" \
      "if [ ! -d \"$NEW_PARENT\" ]; then mkdir -p \"$NEW_PARENT\"; fi"; then
    touch "$FAILLOG"
    # restore stdout/stderr.
    exec 1>&5 
    exec 2>&6

  # copy tar file to remote host.
  if ! scp $SCP_PORT_ARG "$SPLUNK_FILE" "${DST}:${DEST_FILE}"; then
    touch "$FAILLOG"
    # restore stdout/stderr.
    exec 1>&5 
    exec 2>&6
  # run script on remote host and log appropriately.
  if ! ssh $SSH_PORT_ARG "$DST" "$REMOTE_SCRIPT"; then
    touch "$FAILLOG" # remote script failed.
    test -e "$FAILLOG" && rm -f "$FAILLOG" # cleanup any past attempt log.

  # restore stdout/stderr.
  exec 1>&5 
  exec 2>&6

  if [ -e "$FAILLOG" ]; then
    echo "  -->   FAILED   <--"
    echo "      SUCCEEDED"

FAIL_COUNT=`ls "${LOG_DIR}" | grep -c '\.failed$'`
if [ "$FAIL_COUNT" -gt 0 ]; then
  echo "There were $FAIL_COUNT remote installation failures."
  echo "  ( see ${LOG_DIR}/*.failed )"
  echo "Done."

# Voila.

Execute the script

Once you've executed the script, check any log files generated by your installation script for errors. For example, the sample script saves to /tmp/splunkua.install/<destination hostname>.

Deploy a *nix universal forwarder manually
Make a universal forwarder part of a system image

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15


I too saw these errors. This is what I have done to fix it:

1.) Comment out the offending lines that cause the failure:
: '
fail() {
echo ERROR: \"\$@\" >&2
test -f \"$DEST_FILE\" && rm -f \"$DEST_FILE\"
exit 1

2.) The line that reads : if [ -e "$FAILLOG" ]; then
should be: if [ -s "$FAILLOG" ]; then


3.) This line is also messed up:
FAIL_COUNT=`ls "${LOG_DIR}" | grep -c '\.failed$'`

and should read:
FAIL_COUNT=`find "${LOG_DIR}"/*.failed -size +0k | grep -c '\.failed$'`

I also changed the last echo to be more helpful:
echo "Done. There were $FAIL_COUNT remote installation failures. See ${LOG_DIR} for details."

The issue with 1.) was that it was not necessary. 2.) & 3.) were issues because the script writes blank fail logs when there are no failures but was looking for any existence of them and shouldn't care if they are blank.

March 6, 2016

I was able to resolve most of the errors, which in the end were very minor. It still echos the Fail and test info, which it should not do, but it does work. I will try to find the original post on this script and add the updates there.

June 7, 2015

Script is not working as designed. Too many errors to describe. I am not a scripter, so I can not easily identify what is wrong. Running the sript through http://www.shellcheck.net has also noted a number of concerns.

June 7, 2015

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters