Manually configure metrics collection for *nix on Splunk App for Infrastructure

Admin privileges are required to configure metrics collection.

Install the collectd agent to collect system metrics on a *nix host instead of using the script when:

• You are installing the universal forwarder on a closed network
• You already have a universal forwarder on the host from which you want to collect data for the app
• You do not have trusted URLs from which you can download the universal forwarder package

For more information about collectd, see About using collectd.

If you manually configure metrics collection, you also need to manually configure log collection. For more information, see Manually configure log collection for *nix on Splunk App for Infrastructure.

Steps

Install and run the collectd agent as a non-root user.

1. Launch a shell or command prompt.
2. Install all dependencies.
   • For Redhat/Centos: yum install autoconf libtool bison byacc flex libcurl-devel wget
   • For Debian/Ubuntu: apt-get install autoconf libtool bison byacc flex libcurl4-openssl-dev wget pkg-config
3. Install collectd.

       cd ~
       git clone https://github.com/collectd/collectd.git
       cd collectd
       ./build.sh
       ./configure
       make
       make install
   

4. Copy the write_splunk plug-in.

       wget https://<Splunk App Instance>:8000/en-US/static/app/splunk_app_infrastructure/unix_agent/unix-agent.tgz
       tar xvfz unix-agent.tgz
       cp unix-agent/write_splunk.so /opt/collectd/lib/collectd/
   

5. Configure the collectd plug-in.

        cd /opt/collectd/etc/
        vi collectd.conf
   

   #
   # Config file for collectd(1).
   # Please read collectd.conf(5) for a list of options.
   # http://collectd.org/
   #
   
   ##############################################################################
   # Global                                                                     
   #
   #----------------------------------------------------------------------------#
   # Global settings for the daemon.                                            
   #
   ##############################################################################
   
   Hostname    "collectd.server.sample"
   #FQDNLookup   true
   #BaseDir     "/var/lib/collectd"
   #PIDFile     "/var/run/collectd.pid"
   #PluginDir   "/usr/lib64/collectd"
   #TypesDB     "/usr/share/collectd/types.db"
   
   #----------------------------------------------------------------------------#
   # When enabled, plugins are loaded automatically with the default options    #
   # when an appropriate <Plugin ...> block is encountered.                     
   #
   # Disabled by default.                                                       
   #
   #----------------------------------------------------------------------------#
   #AutoLoadPlugin false
   
   #----------------------------------------------------------------------------#
   # When enabled, internal statistics are collected, using "collectd" as the   #
   # plugin name.                                                               
   #
   # Disabled by default.                                                      
   #
   #----------------------------------------------------------------------------#
   #CollectInternalStats false
   
   #----------------------------------------------------------------------------#
   # Interval at which to query values. This may be overwritten on a per-plugin #
   # base by using the 'Interval' option of the LoadPlugin block:               
   #
   #   <LoadPlugin foo>                                                        
   #
   #       Interval 60                                                          
   #
   #   </LoadPlugin>                                                            
   #
   #----------------------------------------------------------------------------#
   Interval     60
   
   #MaxReadInterval 86400
   #Timeout         2
   #ReadThreads     5
   #WriteThreads    5
   
   # Limit the size of the write queue. Default is no limit. Setting up a limit is
   # recommended for servers handling a high volume of traffic.
   #WriteQueueLimitHigh 1000000
   #WriteQueueLimitLow   800000
   
   ##############################################################################
   # Logging                                                                    
   #
   #----------------------------------------------------------------------------#
   # Plugins which provide logging functions should be loaded first, so log     #
   # messages generated when loading or configuring other plugins can be        #
   # accessed.                                                                 
   #
   ##############################################################################
   
   LoadPlugin syslog
   LoadPlugin logfile
   <LoadPlugin "write_splunk">
           FlushInterval 10
   </LoadPlugin>
   
   ##############################################################################
   # LoadPlugin section                                                        
   #
   #----------------------------------------------------------------------------#
   # Lines beginning with a single `#' belong to plugins which have been built  #
   # but are disabled by default.                                               
   #
   #                                                                            
   #
   # Lines beginning with `##' belong to plugins which have not been built due  #
   # to missing dependencies or because they have been deactivated explicitly.  #
   ##############################################################################
   
   #LoadPlugin csv
   LoadPlugin cpu
   LoadPlugin memory
   LoadPlugin df
   LoadPlugin load
   LoadPlugin disk
   LoadPlugin interface
   
   ##############################################################################
   # Plugin configuration                                                       
   #
   #----------------------------------------------------------------------------#
   # In this section configuration stubs for each plugin are provided. A desc-  #
   # ription of those options is available in the collectd.conf(5) manual page. #
   ##############################################################################
   
   <Plugin logfile>
       LogLevel info
       File "/etc/collectd/collectd.log"
       Timestamp true
       PrintSeverity true
   </Plugin>
   
   <Plugin syslog>
       LogLevel info
   </Plugin>
   
   <Plugin cpu>
       ReportByCpu false
       ReportByState true
       ValuesPercentage true
   </Plugin>
   
   <Plugin memory>
       ValuesAbsolute false
       ValuesPercentage true
   </Plugin>
   
   <Plugin df>
       FSType "ext2"
       FSType "ext3"
       FSType "ext4"
       FSType "XFS"
       FSType "rootfs"
       FSType "overlay"
       FSType "hfs"
       FSType "apfs"
       FSType "zfs"
       FSType "ufs"
       ReportByDevice true
       ValuesAbsolute false
       ValuesPercentage true
       IgnoreSelected false
   </Plugin>
   
   <Plugin load>
       ReportRelative true
   </Plugin>
   
   <Plugin disk>
       Disk ""
       IgnoreSelected true
       UdevNameAttr "DEVNAME"
   </Plugin>
   
   <Plugin interface>
       IgnoreSelected true
   </Plugin>
   
   <Plugin write_splunk>
              server "<splunk app server>"
              port "8088"
              token "<HEC TOKEN>"
              ssl true
              verifyssl false
              key1:value1
   </Plugin>
   
   #Update Hostname, <HEC SERVER> & <splunk app server> in collectd.conf file above. Also, you can add dimensions as <Dimension "key:value">  to write_splunk plugin (optional)" 
   
   

6. Run collectd.

       /opt/collectd/sbin/collectd