Manually configure metrics and log collection for a Windows host for Splunk App for Infrastructure

To collect performance metrics and logs, you need to set up data collection using a universal forwarder. The universal forwarder collects data from a data source and sends the data to your Splunk deployment.

In the Splunk App for Infrastructure (SAI), use the Add Data page to set up a script that configures the universal forwarder for metrics and log collection. If you're already running a universal forwarder, you need to manually configure data inputs on it. Also configure data collection manually if you're on a closed network or do not have trusted URLs to download the universal forwarder package from.

Follow the steps in this topic to manually install and configure the universal forwarder, and configure data inputs to collect performance metrics and log collection.

For information about stopping or removing the universal forwarder for metrics and logs collection in SAI, see Uninstall the universal forwarder in the Splunk Universal Forwarder Forwarder Manual.

If you're already monitoring a Windows system and want to update the universal forwarder to collect more data with the Perfmon input to populate the process monitoring table in the Entity Overview, see Sample inputs.conf file for metrics and logs collection.

Steps

Follow these steps to manually configure data collection on a Windows system.

1. Install the universal forwarder on Windows

When installing the universal forwarder, confirm that you:

• Use the forwarder default options.
• During installation process, set the receiving indexer.
• Create the ${SPLUNK_HOME}\etc\apps\splunk_app_infra_uf_config directory
• Set up inputs.conf for the universal forwarder and place in this location: ${SPLUNK_HOME}\etc\apps\splunk_app_infra_uf_config\local\inputs.conf

To manually install and configure the universal forwarder on Windows, see Install a Windows universal forwarder from an installer in the Splunk Universal Forwarder Forwarder Manual.

2. Get available Windows performance data

To get Windows performance counters, use the typeperf command. The following list contains available performance counters for Windows performance monitoring (perfmon) inputs in SAI.

Typeperf  -q  displays all the counters
Typeperf -q <Object Name> to display counters for a particular perfmon object
> Typeperf -q Processor
\Processor(*)\% Processor Time
\Processor(*)\% User Time
\Processor(*)\% Privileged Time
\Processor(*)\Interrupts/sec
\Processor(*)\% DPC Time
\Processor(*)\% Interrupt Time
\Processor(*)\DPCs Queued/sec
\Processor(*)\DPC Rate
\Processor(*)\% Idle Time
\Processor(*)\% C1 Time
\Processor(*)\% C2 Time
\Processor(*)\% C3 Time
\Processor(*)\C1 Transitions/sec
\Processor(*)\C2 Transitions/sec
\Processor(*)\C3 Transitions/sec

3. Add perfmon stanzas to inputs.conf

For each perfmon object you want to collect, add a stanza in inputs.conf with the following settings:

• counters: List the counters you want to monitor for this object.
• instances: Use * to monitor all available instances. Or, specify single or multiple instances.
• object: Valid perfmon object name.
• mode: Enter single. SAI does not support the multikv mode.
• index: Enter em_metrics. If you use another index for metrics, replace em_metrics with the custom index.
• interval: How often, in seconds, to poll for new data.
• _meta: Add entity_type::Windows_Host and any custom dimensions to identify the system.
• useEnglishOnly: Enter true.
• sourcetype: Enter PerfmonMetrics:<metric>
• disabled: Enter 0 to enable the input. Enter 1 to disable the input.

For the Processor object, a valid perfmon stanza in inputs.conf might look like this:

[perfmon://CPU]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time
instances = *
interval = 30
mode = single
object = Processor
index = em_metrics
_meta =  os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:CPU
disabled = 0

The stanza uses default values for counters.

The default object names in props.conf are: CPU, Memory, PhysicalDisk, LogicalDisk, Network, System, Process. If you add a perfmon object to inputs.conf that is not collected by default, you also need to add the following stanza to props.conf in the Splunk add on for Infrastructure.

[PerfmonMetrics:<object name>]
TRANSFORMS-_value_for_perfmon_metrics_store_sai = value_for_perfmon_metrics_store_sai
TRANSFORMS-metric_name_for_perfmon_metrics_store_sai = metric_name_for_perfmon_metrics_store_sai
TRANSFORMS-object_for_perfmon_metrics_store_sai = object_for_perfmon_metrics_store_sai
TRANSFORMS-instance_for_perfmon_metrics_store_sai = instance_for_perfmon_metrics_store_sai
TRANSFORMS-collection_for_perfmon_metrics_store_sai = collection_for_perfmon_metrics_store_sai
EVAL-metric_type = "gauge"
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

For more information about perfmon stanzas, see Performance Monitor in the Splunk Enterprise Admin Manual.

4. Add Windows Event Log stanzas to inputs.conf

Add stanzas in inputs.conf for each Windows Event Log you want to collect, including Application, Security, System, Forwarded Events, and Setup. For Application Windows Event Logs, a valid stanza in inputs.conf might look like this.

[WinEventLog://Application]
checkpointInterval = 10
current_only = 0
disabled = 0
start_from = oldest

For more information, see Windows Event Log Log Monitor in the Splunk Enterprise Admin Manual.

5. Monitor Files for Log Collection

Add stanzas to in inputs.conf to monitor additional log files on the system. A monitor stanza for the Universal Forwarder log files might look like this.

[monitor://$SPLUNK_HOME\var\log\splunk\*.log*]
sourcetype = uf
disabled = false
index = _internal

For more information, see MONITOR: in the Splunk Enterprise Admin Manual.

Sample inputs.conf file for metrics and logs collection

This inputs.conf example includes all perfmon inputs and logs configurations you can configure with the easy install script. This example also collects data to populate all the default dashboards.

[perfmon://CPU]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 30
mode = single
object = Processor
index = em_metrics
_meta =  os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:CPU
disabled = 0

[perfmon://PhysicalDisk]
counters = % Disk Read Time;% Disk Write Time
instances = *
interval = 30
mode = single
object = PhysicalDisk
index = em_metrics
_meta =  os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:PhysicalDisk
disabled = 0

[perfmon://Network]
counters = Bytes Received/sec;Bytes Sent/sec;Packets Received/sec;Packets Sent/sec;Packets Received Errors;Packets Outbound Errors
instances = *
interval = 30
mode = single
object = Network Interface
index = em_metrics
_meta =  os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:Network
disabled = 0

[perfmon://Memory]
counters = Cache Bytes;% Committed Bytes In Use;Page Reads/sec;Pages Input/sec;Pages Output/sec;Committed Bytes;Available Bytes
interval = 30
mode = single
object = Memory
index = em_metrics
_meta =  os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:Memory
disabled = 0

[perfmon://System]
counters = Processor Queue Length;Threads;System Up Time
instances = *
interval = 30
mode = single
object = System
index = em_metrics
_meta =  os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:System
disabled = 0

[perfmon://Process]
counters = % Processor Time;% User Time;% Privileged Time;Elapsed Time;ID Process;Virtual Bytes;Working Set;Private Bytes;IO Read Bytes/sec;IO Write Bytes/sec
instances = *
interval = 30
mode = single
object = Process
index = em_metrics
_meta =  os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:Process
disabled = 0

[perfmon://LogicalDisk]
counters = Free Megabytes;% Free Space
instances = *
interval = 30
mode = single
object = LogicalDisk
index = em_metrics
_meta =  os::"Microsoft Windows Server 2012 R2 Standard" os_version::6.3.9600 entity_type::Windows_Host
useEnglishOnly = true
sourcetype = PerfmonMetrics:LogicalDisk
disabled = 0

[monitor://$SPLUNK_HOME\var\log\splunk\*.log*]
sourcetype = uf
disabled = false
index = _internal

[WinEventLog://Application]
checkpointInterval = 10
current_only = 0
disabled = 0
start_from = oldest

[WinEventLog://Security]
checkpointInterval = 10
current_only = 0
disabled = 0
start_from = oldest

[WinEventLog://System]
checkpointInterval = 10
current_only = 0
disabled = 0
start_from = oldest

[WinEventLog://Setup]
checkpointInterval = 10
current_only = 0
disabled = 0
start_from = oldest