Extended example: Deploy configurations to several forwarders

What we're aiming for

A common use for deployment servers is to manage configuration files for forwarders. In some distributed environments, forwarders can number into the thousands, and the deployment server greatly eases the work of configuring and updating them. This example shows how to use the deployment server to initially configure a set of dissimilar universal forwarders. A follow-up example, in Example: Add inputs to forwarders, shows how to use the deployment server to update the forwarders' configurations with new inputs.

The example sets up the following distributed environment, in which a deployment server deploys configurations for three universal forwarders sending data to two indexers:

• The deployment server Fflanda-SPLUNK3 (10.1.2.4) manages deployment for these universal forwarders:
  • Fflanda-WIN1
  • Fflanda-LINUX1
  • Fflanda-LINUX2
• Fflanda-WIN1 forwards Windows event logs to the receiving indexer Fflanda-SPLUNK1 (10.1.2.2).
• Fflanda-LINUX1 and Fflanda-LINUX2 forward Linux messages to the receiving indexer Fflanda-SPLUNK2 (10.1.2.3).

The forwarders are deployment clients, receiving their configuration files from the deployment server.

Here's the basic set up:

For information on forwarders, see About forwarding and receiving in Forwarding Data. For more information about the universal forwarder, see About the universal forwarder in the Splunk Universal Forwarder product documentation.

For information on monitoring Windows event log data, see Monitor Windows event log data in Getting Data In.

For information on monitoring files, such as message logs, see Monitor files and directories in Getting Data In.

Configuration overview

Here's an overview of the set-up process (the detailed steps follow in the next section):

On each indexer that will be receiving data from the forwarders:

1. Enable receiving through the CLI.

2. Restart the indexer.

On each deployment client (forwarder):

1. Create a deploymentclient.conf file that points to the deployment server.

2. Restart the forwarder.

On the deployment server:

1. Create directories for the deployment apps.

2. Use forwarder management to create the set of server classes for the deployment clients (forwarders). You'll create two server classes to represent the two OS types (Windows, Linux). Each server class will map a set of clients to two separate apps, for a total of four apps. These apps, defined in a later step, encapsulate:

• The type of input -- the data that the universal forwarder will monitor (Windows event logs or Linux messages).
• The type of output -- the indexer the forwarder will send data to (SPLUNK1 or SPLUNK2).

This configuration results in each universal forwarder belonging to a server class and receiving two apps: one for its inputs and one for its outputs.

Note: You can also create the server classes directly in serverclass.conf, as described below in Configure server classes in serverclass.conf.

3. Populate the app directories with the contents of the apps. Each app consists of a configuration file, either outputs.conf or inputs.conf.

4. Deploy the apps by reloading the deployment server.

After a short delay (while the forwarders receive and act upon their deployed content), Windows event logs begin flowing from Fflanda-WIN1 to Fflanda-SPLUNK1, and /var/log/messages begin flowing from Fflanda-LINUX1 and Fflanda-LINUX2 to Fflanda-SPLUNK2.

Detailed configuration steps

For each receiving indexer (Fflanda-SPLUNK1 and Fflanda-SPLUNK2):

1. Install Splunk Enterprise on the machine where you want the indexer to reside.

2. Run the following CLI command:

splunk enable listen 9997 -auth <username>:<password>

This specifies that the indexer will serve as a receiver, listening for data on port 9997. With proper authorization, any forwarder can now send data to the receiver by designating its IP address and port number. You must enable the indexers as receivers before you configure the forwarders to send data to them.

For more information on enabling receivers, see Enable a receiver in the Forwarding Data manual.

3. Restart the indexer.

For each universal forwarder (Fflanda-WIN1, Fflanda-LINUX1, and Fflanda-LINUX2):

1. Install the forwarder on the machine where you want it to reside, if you haven't already done so.

2. Specify the deployment server that the forwarder will poll. You can do this as part of the forwarder installation process, or later, by editing $SPLUNK_HOME/etc/system/local/deploymentclient.conf, like this:

   [deployment-client]

   [target-broker:deploymentServer]
   # Specify the deployment server; for example, "10.1.2.4:8089".
   targetUri= <URI:port> 

See Configure deployment clients for details on how to configure this file.

3. Restart the forwarder.

On the deployment server (Fflanda-SPLUNK3):

A. Prepare the deployment server

1. Install Splunk Enterprise, if you haven't already done so.

2. Create these deployment app directories:

• $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk1
• $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk2
• $SPLUNK_HOME/etc/deployment-apps/winevt
• $SPLUNK_HOME/etc/deployment-apps/linmess

The directory name determines the name of the app: "fwd_to_splunk1", "fwd_to_splunk2", etc.

B. Configure the server classes for Windows forwarders

Note: This procedure and the one that follows both use forwarder management to configure the server classes. For an example of how to configure these server classes instead by direct editing of serverclass.conf, see Configure server classes in serverclass.conf later in this topic.

To configure the Fflanda-WIN server class for Windows forwarders, go to the forwarder management interface and perform these actions:

1. Select the Groups / Server Classes' tab.

2. Select the New Server Class button.

3. Enter Fflanda-WIN in the Server class name field of the pop-up window and click Save.

4. Select the three dots () in the last column of the server class that you just created.

'5. Select the Edit configurations button. This takes you to the Edit Applications Assigned page. On this page, there are two columns: Unselected Apps and Selected Apps, with a list of apps under each. For a new server class, all apps will initially be in the Unselected column.

6. In the Unselected Apps column, look for the apps winevt and fwd_to_splunk1. Click on each of these apps to move them to the Selected Apps column.

7. Click Save to save the apps in the server class.

C. Configure the server classes for Linux forwarders

To configure the Fflanda-LINUX server class for Linux forwarders, go to the forwarder management interface and perform these actions:

1. Select the Groups / Server Classes tab.

2. Select the New Server Class button.

3. Enter Fflanda-LINUX in the Server class name field of the pop-up window and click Save.

4. Select the three dots () in the last column of the server class. Select Edit configurations

5. In the Unselected Apps column, look for the apps linmess and fwd_to_splunk2. Click on each of these apps to move them to the Selected Apps column.

6. Click Save to save the apps in the server class. You'll now be prompted to add clients to the server class. Click the Add Clients button. This takes you to the Edit Clients page.

See Use forwarder management to define server classes for details on the forwarder management interface.

D. Populate the apps

1. Create $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk1/default/outputs.conf with the following settings:

    [tcpout]
    defaultGroup=splunk1

    [tcpout:splunk1]
    # Specifies the server that receives data from the forwarder.
    server=10.1.2.2:9997

For information on outputs.conf, see Configure forwarders with outputs.conf in the Forwarding Data manual.

2. Create $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk2/default/outputs.conf with the following settings:

    [tcpout]
    defaultGroup=splunk2

    [tcpout:splunk2]
    server=10.1.2.3:9997

3. Create $SPLUNK_HOME/etc/deployment-apps/winevt/default/inputs.conf with the following settings:

    [WinEventLog:Application]
    disabled=0

    [WinEventLog:Security]
    disabled=0

    [WinEventLog:System]
    disabled=0

For information on monitoring Windows event log data, see Monitor Windows event log data in the Getting Data In manual.

4. Create $SPLUNK_HOME/etc/deployment-apps/linmess/default/inputs.conf with the following settings:

    [monitor:///var/log/messages]
    disabled=false
    sourcetype=syslog

For information on monitoring files, such as message logs, see Monitor files and directories in the Getting Data In manual.

E. Deploy the apps

To deploy the apps, just reload the deployment server:

splunk reload deploy-server

Each forwarder will now poll the deployment server, download its configuration files, restart, and begin forwarding data to its receiving indexer.

Note: When you change the content of an app, you must always invoke the reload command, so that the clients know they need to download the new content.

For a follow-up example showing how to use the deployment server to update forwarder configurations, see Example: Add inputs to forwarders.

Configure server classes with serverclass.conf

Instead of using forwarder management to configure server classes, you can directly edit serverclass.conf. For most purposes, however, it is faster and easier to use forwarder management.

In the case of the example under discussion, you can fully define the required server classes with forwarder management, so it is not necessary to edit serverclass.conf. However, for those who prefer to work directly in configuration files, this subtopic shows how to create the example's set of server classes by editing serverclass.conf. This version takes advantage of some minor optimization only available through direct editing of serverclass.conf.

Create $SPLUNK_HOME/etc/system/local/serverclass.conf with the following settings:

    # Global server class 
    [global]
    # Filter (whitelist) all clients
    whitelist.0=*

    # Server class for Windows
    [serverClass:Fflanda-WIN]
    # Filter (whitelist) all Windows clients
    whitelist.0=Fflanda-WIN*

    # App for inputting Windows event logs
    # This app is only for clients in the server class Fflanda-WIN
    [serverClass:Fflanda-WIN:app:winevt]
    #Enable the app and restart Splunk, after the client receives the app
    stateOnClient=enabled
    restartSplunkd=true

    # App for forwarding to SPLUNK1
    # This app is only for clients in the server class Fflanda-WIN
    [serverClass:Fflanda-WIN:app:fwd_to_splunk1]
    stateOnClient=enabled
    restartSplunkd=true
    
    # Server class for Linux
    [serverClass:Fflanda-LINUX]
    # Filter (whitelist) all Linux clients
    whitelist.0=Fflanda-LINUX*
    
    # App for inputting Linux messages
    # This app is only for clients in the server class Fflanda-LINUX
    [serverClass:Fflanda-LINUX:app:linmess]
    stateOnClient=enabled
    restartSplunkd=true

    # App for forwarding to SPLUNK2
    # This app is only for clients in the server class Fflanda-LINUX
    [serverClass:Fflanda-LINUX:app:fwd_to_splunk2]
    stateOnClient=enabled
    restartSplunkd=true

See "Use serverclass.conf to define server classes" for details on how to configure this file.

What the communication between the deployment server and its clients looks like

Using the above example, the communication from Fflanda-WIN1 to Fflanda-SPLUNK3 on port 8089 would look like this:

Fflanda-WIN1: Hello, I am Fflanda-WIN1.

Fflanda-SPLUNK3: Hello, Fflanda-WIN1. I have been expecting to hear from you. I have you down as a member of the Fflanda-WIN server class, and you should have the fwd_to_splunk1 (checksum=12345) and winevt (checksum=12378) apps.

Fflanda-WIN1: Hmmm, I don't have those configs. Using this connection I just opened up to you, can I grab the configs from you?

Fflanda-SPLUNK3: Sure! I have them ready for you.

Fflanda-WIN1: Thanks! I am going to back off a random number of seconds between 1 and 60 (in case you have a lot of clients that are polling you at the moment) ... OK, now send me the files.

Fflanda-SPLUNK3: Done! You now have fwd_to_splunk1-timestamp.bundle and winevt-timestamp.bundle.

Fflanda-WIN1: Awesome! I am going to store them in my $SPLUNK_HOME/etc/apps directory. Now I am going to restart myself, and when I come back up I am going to read the configurations that you sent me directly out of the .bundle files, which I know are just tar balls with a different extension.

A couple of minutes go by....

Fflanda-WIN1: Hello, I am Fflanda-WIN1.

Fflanda-SPLUNK3: Hello, Fflanda-WIN1. I have been expecting to hear from you. I have you down as a member of the Fflanda-WIN server class, and you should have the fwd_to_splunk1 (checksum=12345) and winevt (checksum=12378) Apps.

Fflanda-WIN1: Hmmm, I already have both of those, but thanks anyway!

Later on, an admin modifies the winevt/inputs.conf file on Fflanda-SPLUNK3 to disable the collection of system event logs, and then runs the CLI command splunk reload deploy-server to force the deployment server to rescan serverclass.conf and the app directories. The next time Fflanda-WIN1 talks to Fflanda-SPLUNK3, it goes like this:

Fflanda-WIN1: Hello, I am Fflanda-WIN1.

Fflanda-SPLUNK3: Hello, Fflanda-WIN1. I have been expecting to hear from you. I have you down as a member of the Fflanda-WIN server class, and you should have the fwd_to_splunk1 (checksum=12345) and winevt (checksum=13299) Apps.

Fflanda-WIN1: Hmmm, I know I have those configs, but the checksum I have for the winevt configs is different than the one you just told me about. Using this connection I just opened up to you, can I grab the updated winevt config from you?

Fflanda-SPLUNK3: Sure! I have it ready for you.

Fflanda-WIN1: Thanks! I am going to back off a random number of seconds between 1 and 60 (in case you have a lot of clients that are polling you at the moment) ... Ok, now send me the updated config.

Fflanda-SPLUNK3: Done! You now have winevt-newer_timestamp.bundle.

Fflanda-WIN1: Awesome! I am going to store it in my $SPLUNK_HOME/etc/apps directory and move the old winevt.bundle I had out of the way. Now I am going to restart myself, and when I come back up, I'll have the most up-to-date config.