Extended example: deploy several standard forwarders

Extended example: deploy several standard forwarders

What we're aiming for

A common use for deployment servers is to manage forwarder configuration files. In some distributed environments, forwarders can number into the hundreds, and the deployment server greatly eases the work of configuring and updating them. This example shows how to use the deployment server to initially configure a set of dissimilar forwarders. A follow-up example, in "Example: add an input to forwarders", shows how to use the deployment server to update the forwarders' configurations with new inputs.

The example sets up the following distributed environment, in which a deployment server deploys configurations for three forwarders sending data to two indexers:

• The deployment server Fflanda-SPLUNK3 (10.1.2.4) manages deployment for these forwarders:
  • Fflanda-WIN1
  • Fflanda-LINUX1
  • Fflanda-LINUX2
• Fflanda-WIN1 forwards Windows event logs to the receiving indexer Fflanda-SPLUNK1 (10.1.2.2).
• Fflanda-LINUX1 and Fflanda-LINUX2 forward Linux messages to the receiving indexer Fflanda-SPLUNK2 (10.1.2.3).

The forwarders are deployment clients, receiving their configuration files from the deployment server.

Here's the basic set up:

For information on forwarders, see "About forwarding and receiving" in this manual.

For information on monitoring Windows event log data, see "Monitor Windows event log data" in this manual.

For information on monitoring files, such as message logs, see "Monitor files and directories" in this manual.

Overview of the set up

Here's an overview of the set up process (the detailed steps follow in the next section):

On the deployment server:

1. Create the set of server classes and apps for the deployment clients (forwarders) with serverclass.conf. You'll create two server classes to represent the two OS types (Windows, Linux). For each server class, you'll also create two deployment apps, for a total of four apps. The apps encapsulate:

• The type of input -- the data that the forwarder will monitor (Windows event logs or Linux messages).
• The type of output -- the indexer the forwarder will send data to (SPLUNK1 or SPLUNK2).

This configuration results in each forwarder belonging to a server class and receiving two apps: one for its inputs and one for its outputs.

2. Create directories to hold the deployment apps.

3. Create configuration files (outputs.conf and inputs.conf) to deploy to the forwarders. These files constitute the deployment apps and reside in the app directories.

4. Restart the deployment server.

On each Splunk indexer that will be receiving data from forwarders:

1. Enable receiving through the Splunk CLI.

2. Restart the receiver.

On each forwarder/deployment client:

1. Create a deploymentclient.conf file that points to the deployment server.

2. Restart the forwarder.

The rest is Splunk magic. After a short delay (while the forwarders receive and act upon their deployed content), Windows event logs begin flowing from Fflanda-WIN1 to Fflanda-SPLUNK1, and /var/log/messages begin flowing from Fflanda-LINUX1 and Fflanda-LINUX2 to Fflanda-SPLUNK2.

Detailed configuration steps

On the deployment server, Fflanda-SPLUNK3:

1. Install Splunk, if you haven't already done so.

2. Set up your server classes. Create $SPLUNK_HOME/etc/system/local/serverclass.conf with the following settings:

    # Global server class 
    [global]
    # Filter (whitelist) all clients
    whitelist.0=*

    # Server class for Windows
    [serverClass:Fflanda-WIN]
    # Filter (whitelist) all Windows clients
    whitelist.0=Fflanda-WIN*

    # App for inputting Windows event logs
    # This app is only for clients in the server class Fflanda-WIN
    [serverClass:Fflanda-WIN:app:winevt]
    #Enable the app and restart Splunk, after the client receives the app
    stateOnClient=enabled
    restartSplunkd=true

    # App for forwarding to SPLUNK1
    # This app is only for clients in the server class Fflanda-WIN
    [serverClass:Fflanda-WIN:app:fwd_to_splunk1]
    stateOnClient=enabled
    restartSplunkd=true
    
    # Server class for Linux
    [serverClass:Fflanda-LINUX]
    # Filter (whitelist) all Linux clients
    whitelist.0=Fflanda-LINUX*
    
    # App for inputting Linux messages
    # This app is only for clients in the server class Fflanda-LINUX
    [serverClass:Fflanda-LINUX:app:linmess]
    stateOnClient=enabled
    restartSplunkd=true

    # App for forwarding to SPLUNK2
    # This app is only for clients in the server class Fflanda-LINUX
    [serverClass:Fflanda-LINUX:app:fwd_to_splunk2]
    stateOnClient=enabled
    restartSplunkd=true

See "Define server classes" for details on how to configure this file.

3. Create the app deployment directories with the following commands:

    mkdir –p $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk1/default 
    mkdir –p $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk2/default 
    mkdir –p $SPLUNK_HOME/etc/deployment-apps/winevt/default 
    mkdir –p $SPLUNK_HOME/etc/deployment-apps/linmess/default

Each app gets its own directory, so that they can be deployed individually. In addition, the directory name determines the name of the app.

4. Create $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk1/default/outputs.conf with the following settings:

    [tcpout]
    defaultGroup=splunk1

    [tcpout:splunk1]
    # Specifies the server that receives data from the forwarder.
    server=10.1.2.2:9997

For information on outputs.conf, see "Configure forwarders with outputs.conf" in this manual.

5. Create $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk2/default/outputs.conf with the following settings:

    [tcpout]
    defaultGroup=splunk2

    [tcpout:splunk2]
    server=10.1.2.3:9997

6. Create $SPLUNK_HOME/etc/deployment-apps/winevt/default/inputs.conf with the following settings:

    [WinEventLog:Application]
    disabled=0

    [WinEventLog:Security]
    disabled=0

    [WinEventLog:System]
    disabled=0

For information on monitoring Windows event log data, see "Monitor Windows event log data" in this manual.

7. Create $SPLUNK_HOME/etc/deployment-apps/linmess/default/inputs.conf with the following settings:

    [monitor:///var/log/messages]
    disabled=false
    followTail=1
    sourcetype=syslog

For information on monitoring files, such as message logs, see "Monitor files and directories" in this manual.

8. Restart Splunk.

Note: Because the deployment server in this example is newly configured, it requires a restart for its configuration to take effect. When clients poll the server for the first time, they'll get all the content designated for them. To deploy subsequent content, you generally do not need to restart the server. Instead, you just invoke the Splunk CLI reload command on the server, as described in "Deploy apps and configurations". By doing so, you ensure that the server will inform its clients of content changes. However, whenever you edit serverclass.conf, you must always restart the deployment server for the configuration changes to take effect.

On each receiver, Fflanda-SPLUNK1 and Fflanda-SPLUNK2:

1. Install Splunk, if you haven't already done so.

2. Run the following CLI command:

./splunk enable listen 9997 -auth <username>:<password>

This specifies that the receiver will listen for data on port 9997. With proper authorization, any forwarder can now send data to the receiver by designating the receiver's IP address and port number. You must enable receivers before you enable the forwarders that will be sending data to them.

For information on enabling receivers, see "Enable forwarding and receiving" in this manual.

3. Restart Splunk.

On each forwarder, Fflanda-WIN1, Fflanda-LINUX1, and Fflanda-LINUX2:

1. Install Splunk, if you haven't already done so.

2. Create $SPLUNK_HOME/etc/system/local/deploymentclient.conf with the following settings:

   [deployment-client]

   [target-broker:deploymentServer]
   # Specify the deployment server that the client will poll.
   targetUri= 10.1.2.4:8089 

See "Configure deployment clients" for details on how to configure this file.

3. Restart Splunk.

Each forwarder will now poll the deployment server, download its configuration files, restart, and begin forwarding data to its receiving indexer.

For a follow-up example showing how to use the deployment server to update forwarder configurations, see "Example: Add an input to forwarders".

What the communication between the deployment server and its clients looks like

Using the above example, the communication from Fflanda-WIN1 to Fflanda-SPLUNK3 on port 8089 would look like this:

Fflanda-WIN1: Hello, I am Fflanda-WIN1.

Fflanda-SPLUNK3: Hello, Fflanda-WIN1. I have been expecting to hear from you. I have you down as a member of the Fflanda-WIN server class, and you should have the fwd_to_splunk1 (checksum=12345) and winevt (checksum=12378) apps.

Fflanda-WIN1: Hmmm, I don’t have those configs. Using this connection I just opened up to you, can I grab the configs from you?

Fflanda-SPLUNK3: Sure! I have them ready for you.

Fflanda-WIN1: Thanks! I am going to back off a random number of seconds between 1 and 60 (in case you have a lot of clients that are polling you at the moment) ... OK, now send me the files.

Fflanda-SPLUNK3: Done! You now have fwd_to_splunk1-timestamp.bundle and winevt-timestamp.bundle.

Fflanda-WIN1: Awesome! I am going to store them in my $SPLUNK_HOME/etc/apps directory. Now I am going to restart myself, and when I come back up I am going to read the configurations that you sent me directly out of the .bundle files, which I know are just tar balls with a different extension.

A couple of minutes go by....

Fflanda-WIN1: Hello, I am Fflanda-WIN1.

Fflanda-SPLUNK3: Hello, Fflanda-WIN1. I have been expecting to hear from you. I have you down as a member of the Fflanda-WIN server class, and you should have the fwd_to_splunk1 (checksum=12345) and winevt (checksum=12378) Apps.

Fflanda-WIN1: Hmmm, I already have both of those, but thanks anyway!

Later on, an admin modifies the winevt/inputs.conf file on Fflanda-SPLUNK3 to disable the collection of system event logs, and then runs the CLI command splunk reload deploy-server to force the deployment server to rescan serverclass.conf and the app directories. The next time Fflanda-WIN1 talks to Fflanda-SPLUNK3, it goes like this:

Fflanda-WIN1: Hello, I am Fflanda-WIN1.

Fflanda-SPLUNK3: Hello, Fflanda-WIN1. I have been expecting to hear from you. I have you down as a member of the Fflanda-WIN server class, and you should have the fwd_to_splunk1 (checksum=12345) and winevt (checksum=13299) Apps.

Fflanda-WIN1: Hmmm, I know I have those configs, but the checksum I have for the winevt configs is different than the one you just told me about. Using this connection I just opened up to you, can I grab the updated winevt config from you?

Fflanda-SPLUNK3: Sure! I have it ready for you.

Fflanda-WIN1: Thanks! I am going to back off a random number of seconds between 1 and 60 (in case you have a lot of clients that are polling you at the moment) ... Ok, now send me the updated config.

Fflanda-SPLUNK3: Done! You now have winevt-newer_timestamp.bundle.

Fflanda-WIN1: Awesome! I am going to store it my $SPLUNK_HOME/etc/apps directory and move the old winevt.bundle I had out of the way. Now I am going to restart myself, and when I come back up, I'll have the most up-to-date config.

• Was this topic useful?
• Post a comment