Splunk® Enterprise

Distributed Deployment Manual

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Extended example: deploy configurations to several forwarders

What we're aiming for

A common use for deployment servers is to manage configuration files for forwarders. In some distributed environments, forwarders can number into the thousands, and the deployment server greatly eases the work of configuring and updating them. This example shows how to use the deployment server to initially configure a set of dissimilar universal forwarders. A follow-up example, in "Example: add an input to forwarders", shows how to use the deployment server to update the forwarders' configurations with new inputs.

Before reading this example, you should already be familiar with how forwarders and receivers communicate, as described in "About forwarding and receiving" and the topics that follow it.

The example sets up the following distributed environment, in which a deployment server deploys configurations for three universal forwarders sending data to two indexers:

  • The deployment server Fflanda-SPLUNK3 (10.1.2.4) manages deployment for these universal forwarders:
    • Fflanda-WIN1
    • Fflanda-LINUX1
    • Fflanda-LINUX2
  • Fflanda-WIN1 forwards Windows event logs to the receiving indexer Fflanda-SPLUNK1 (10.1.2.2).
  • Fflanda-LINUX1 and Fflanda-LINUX2 forward Linux messages to the receiving indexer Fflanda-SPLUNK2 (10.1.2.3).

The forwarders are deployment clients, receiving their configuration files from the deployment server.

Here's the basic set up:

Deployment1.png

For information on forwarders (including the universal forwarder), see "About forwarding and receiving" in this manual.

For information on monitoring Windows event log data, see "Monitor Windows event log data".

For information on monitoring files, such as message logs, see "Monitor files and directories".

Overview of the set up

Here's an overview of the set up process (the detailed steps follow in the next section):

On the deployment server:

1. Create the set of server classes and apps for the deployment clients (forwarders) in serverclass.conf. You'll create two server classes to represent the two OS types (Windows, Linux). For each server class, you'll also create two deployment apps, for a total of four apps. The apps encapsulate:

  • The type of input -- the data that the universal forwarder will monitor (Windows event logs or Linux messages).
  • The type of output -- the indexer the forwarder will send data to (SPLUNK1 or SPLUNK2).

This configuration results in each universal forwarder belonging to a server class and receiving two apps: one for its inputs and one for its outputs.

2. Create directories to hold the deployment apps.

3. Create configuration files (outputs.conf and inputs.conf) to deploy to the forwarders. These files constitute the deployment apps and reside in the app directories.

4. Restart the deployment server.

On each Splunk indexer that will be receiving data from universal forwarders:

1. Enable receiving through the Splunk CLI.

2. Restart the indexer.

On each forwarder/deployment client:

1. Create a deploymentclient.conf file that points to the deployment server.

2. Restart the forwarder.

The rest is Splunk magic. After a short delay (while the forwarders receive and act upon their deployed content), Windows event logs begin flowing from Fflanda-WIN1 to Fflanda-SPLUNK1, and /var/log/messages begin flowing from Fflanda-LINUX1 and Fflanda-LINUX2 to Fflanda-SPLUNK2.

Detailed configuration steps

On the deployment server, Fflanda-SPLUNK3:

1. Install Splunk, if you haven't already done so.

2. Set up your server classes. Create $SPLUNK_HOME/etc/system/local/serverclass.conf with the following settings:

    # Global server class 
    [global]
    # Filter (whitelist) all clients
    whitelist.0=*

    # Server class for Windows
    [serverClass:Fflanda-WIN]
    # Filter (whitelist) all Windows clients
    whitelist.0=Fflanda-WIN*

    # App for inputting Windows event logs
    # This app is only for clients in the server class Fflanda-WIN
    [serverClass:Fflanda-WIN:app:winevt]
    #Enable the app and restart Splunk, after the client receives the app
    stateOnClient=enabled
    restartSplunkd=true

    # App for forwarding to SPLUNK1
    # This app is only for clients in the server class Fflanda-WIN
    [serverClass:Fflanda-WIN:app:fwd_to_splunk1]
    stateOnClient=enabled
    restartSplunkd=true
    
    # Server class for Linux
    [serverClass:Fflanda-LINUX]
    # Filter (whitelist) all Linux clients
    whitelist.0=Fflanda-LINUX*
    
    # App for inputting Linux messages
    # This app is only for clients in the server class Fflanda-LINUX
    [serverClass:Fflanda-LINUX:app:linmess]
    stateOnClient=enabled
    restartSplunkd=true

    # App for forwarding to SPLUNK2
    # This app is only for clients in the server class Fflanda-LINUX
    [serverClass:Fflanda-LINUX:app:fwd_to_splunk2]
    stateOnClient=enabled
    restartSplunkd=true

See "Define server classes" for details on how to configure this file.

3. Create the app deployment directories with the following commands:

    mkdir –p $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk1/default 
    mkdir –p $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk2/default 
    mkdir –p $SPLUNK_HOME/etc/deployment-apps/winevt/default 
    mkdir –p $SPLUNK_HOME/etc/deployment-apps/linmess/default

Each app gets its own directory, so that they can be deployed individually. In addition, the directory name determines the name of the app.

4. Create $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk1/default/outputs.conf with the following settings:

    [tcpout]
    defaultGroup=splunk1

    [tcpout:splunk1]
    # Specifies the server that receives data from the forwarder.
    server=10.1.2.2:9997

For information on outputs.conf, see "Configure forwarders with outputs.conf" in this manual.

5. Create $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk2/default/outputs.conf with the following settings:

    [tcpout]
    defaultGroup=splunk2

    [tcpout:splunk2]
    server=10.1.2.3:9997

6. Create $SPLUNK_HOME/etc/deployment-apps/winevt/default/inputs.conf with the following settings:

    [WinEventLog:Application]
    disabled=0

    [WinEventLog:Security]
    disabled=0

    [WinEventLog:System]
    disabled=0

For information on monitoring Windows event log data, see "Monitor Windows event log data" in the Getting Data In manual.

7. Create $SPLUNK_HOME/etc/deployment-apps/linmess/default/inputs.conf with the following settings:

    [monitor:///var/log/messages]
    disabled=false
    sourcetype=syslog

For information on monitoring files, such as message logs, see "Monitor files and directories" in the Getting Data In manual.

8. Restart Splunk on the deployment server.

Note: Because the deployment server in this example is newly configured, it requires a restart for its configuration to take effect. When clients poll the server for the first time, they'll get all the content designated for them. To deploy subsequent content, you generally do not need to restart the server. Instead, you just invoke the Splunk CLI reload command on the server, as described in "Deploy apps and configurations". By doing so, you ensure that the server will inform its clients of content changes. However, whenever you edit serverclass.conf, you must always restart the deployment server for the configuration changes to take effect.

To set up the receiving indexers, Fflanda-SPLUNK1 and Fflanda-SPLUNK2, perform these steps on the machines where you want them to reside:

1. Install Splunk, if you haven't already done so.

2. Run the following CLI command:

./splunk enable listen 9997 -auth <username>:<password>

This specifies that the indexer will serve as a receiver, listening for data on port 9997. With proper authorization, any forwarder can now send data to the receiver by designating its IP address and port number. You must enable the indexers as receivers before you enable the forwarders that will be sending data to them.

For more information on enabling receivers, see "Enable a receiver" in this manual.

3. Restart Splunk.

On each universal forwarder, Fflanda-WIN1, Fflanda-LINUX1, and Fflanda-LINUX2:

1. Install the forwarder, if you haven't already done so.

2. You can specify the deployment server as part of the installation process, or you can specify it in $SPLUNK_HOME/etc/system/local/deploymentclient.conf after the installation, using these settings:

   [deployment-client]

   [target-broker:deploymentServer]
   # Specify the deployment server that the client will poll.
   targetUri= 10.1.2.4:8089 

See "Configure deployment clients" for details on how to configure this file. To learn how to specify the deployment server at installation time, see "Universal forwarder deployment overview" and the topics that follow it.

3. Restart the universal forwarder.

Each forwarder will now poll the deployment server, download its configuration files, restart, and begin forwarding data to its receiving indexer.

For a follow-up example showing how to use the deployment server to update forwarder configurations, see "Example: Add an input to forwarders".

What the communication between the deployment server and its clients looks like

Using the above example, the communication from Fflanda-WIN1 to Fflanda-SPLUNK3 on port 8089 would look like this:

Fflanda-WIN1: Hello, I am Fflanda-WIN1.

Fflanda-SPLUNK3: Hello, Fflanda-WIN1. I have been expecting to hear from you. I have you down as a member of the Fflanda-WIN server class, and you should have the fwd_to_splunk1 (checksum=12345) and winevt (checksum=12378) apps.

Fflanda-WIN1: Hmmm, I don’t have those configs. Using this connection I just opened up to you, can I grab the configs from you?

Fflanda-SPLUNK3: Sure! I have them ready for you.

Fflanda-WIN1: Thanks! I am going to back off a random number of seconds between 1 and 60 (in case you have a lot of clients that are polling you at the moment) ... OK, now send me the files.

Fflanda-SPLUNK3: Done! You now have fwd_to_splunk1-timestamp.bundle and winevt-timestamp.bundle.

Fflanda-WIN1: Awesome! I am going to store them in my $SPLUNK_HOME/etc/apps directory. Now I am going to restart myself, and when I come back up I am going to read the configurations that you sent me directly out of the .bundle files, which I know are just tar balls with a different extension.

A couple of minutes go by....

Fflanda-WIN1: Hello, I am Fflanda-WIN1.

Fflanda-SPLUNK3: Hello, Fflanda-WIN1. I have been expecting to hear from you. I have you down as a member of the Fflanda-WIN server class, and you should have the fwd_to_splunk1 (checksum=12345) and winevt (checksum=12378) Apps.

Fflanda-WIN1: Hmmm, I already have both of those, but thanks anyway!

Later on, an admin modifies the winevt/inputs.conf file on Fflanda-SPLUNK3 to disable the collection of system event logs, and then runs the CLI command splunk reload deploy-server to force the deployment server to rescan serverclass.conf and the app directories. The next time Fflanda-WIN1 talks to Fflanda-SPLUNK3, it goes like this:

Fflanda-WIN1: Hello, I am Fflanda-WIN1.

Fflanda-SPLUNK3: Hello, Fflanda-WIN1. I have been expecting to hear from you. I have you down as a member of the Fflanda-WIN server class, and you should have the fwd_to_splunk1 (checksum=12345) and winevt (checksum=13299) Apps.

Fflanda-WIN1: Hmmm, I know I have those configs, but the checksum I have for the winevt configs is different than the one you just told me about. Using this connection I just opened up to you, can I grab the updated winevt config from you?

Fflanda-SPLUNK3: Sure! I have it ready for you.

Fflanda-WIN1: Thanks! I am going to back off a random number of seconds between 1 and 60 (in case you have a lot of clients that are polling you at the moment) ... Ok, now send me the updated config.

Fflanda-SPLUNK3: Done! You now have winevt-newer_timestamp.bundle.

Fflanda-WIN1: Awesome! I am going to store it my $SPLUNK_HOME/etc/apps directory and move the old winevt.bundle I had out of the way. Now I am going to restart myself, and when I come back up, I'll have the most up-to-date config.

PREVIOUS
Deploy apps and configurations
  NEXT
Example: add an input to forwarders

This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7, 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters