Getting Data In

Monitor Windows performance in real time

Splunk Enterprise supports the monitoring of all Windows performance counters available to the system in real time, and includes support for both local and remote collection of performance data.

The Splunk Enterprise performance monitoring utility gives you the abilities of Performance Monitor in a web or command-line interface. Splunk uses the Performance Data Helper (PDH) API for performance counter queries on local machines.

The types of performance objects, counters and instances that are available to Splunk depend on the performance libraries installed on the system. Both Microsoft and third-party vendors provide libraries that contain performance counters. For additional information on performance monitoring, review "Performance Counters" (http://msdn.microsoft.com/en-us/library/aa373083%28v=VS.85%29.aspx) on MSDN.

Both full instances of Splunk and universal forwarders support local collection of performance metrics. Remote performance monitoring is available through WMI (Windows Management Instrumentation) and requires that Splunk runs as a user with appropriate Active Directory credentials.

The performance monitor input runs as a process called splunk-perfmon.exe. This process runs once for every input defined, at the interval specified in the input. You can configure performance monitoring using Splunk Web, or either inputs.conf (for local performance data) or wmi.conf (for performance data from a remote machine).

Why monitor performance metrics?

Performance monitoring is an important part of the Windows administrator's toolkit. Windows generates a lot of data about a system's health. Proper analysis of that data can make the difference between a healthy, well functioning system, and one that suffers downtime.

What's required to monitor performance counters?

The following table lists the explicit permissions needed to monitor performance counters in Windows. You might need additional permissions based on the performance objects or counters that you want to monitor.

For additional information on what's required to monitor performance metrics, read "Security and remote access considerations" later in this topic.

Activity:	Required permissions:
Monitor local performance metrics	* Splunk must run on Windows * Splunk must run as the Local System user
Monitor remote performance metrics on another computer over WMI	* Splunk must run on Windows * Splunk must run as a domain or remote user with at least read access to WMI on the target computer * Splunk must run as a domain or remote user with appropriate access to the Performance Data Helper libraries on the target computer

Security and remote access considerations

Splunk gets data from remote machines using either a forwarder or WMI. Splunk recommends using a universal forwarder to send performance data from remote machines to an indexer. Review "Introducing the universal forwarder" in the Forwarding Data Manual for information about how to install, configure and use the forwarder to collect performance metrics.

If you choose to install forwarders on your remote machines to collect performance data, then you can install the forwarder as the Local System user on those machines. The Local System user has access to all data on the local machine, but not to remote computers.

If you want Splunk to use WMI to get performance data from remote machines, then you must ensure that your network and Splunk instances are properly configured. You cannot install Splunk as the Local System user, and the user you install with determines the set of performance metrics Splunk sees. Review "Security and remote access considerations" in the "Monitor WMI Data" topic in this manual for additional information on the requirements you must satisfy in order for Splunk to collect remote data properly using WMI.

After you install Splunk with a valid user, add that user to the following groups before enabling local performance monitor inputs:

• Performance Monitor Users (domain group)
• Performance Log Users (domain group)

Enable local Windows performance monitoring

You can configure local performance monitoring either in Splunk Web, or by using configuration files.

Splunk Web is the preferred way to add performance monitoring data inputs. This is because you can make typos when using configuration files, and it's important to specify performance monitor objects exactly as the Performance Monitor API defines them. See "Important information about specifying performance monitor objects in inputs.conf" below for a full explanation.

Configure local Windows performance monitoring with Splunk Web

1. Click Settings in the upper right-hand corner of Splunk Web.

2. Under Data, click Data Inputs.

3. Click Local performance monitoring.

4. Click New to add an input.

5. Enter a unique, memorable name for this input.

6. Under Available objects, choose the performance object whose counters you wish to display.

Splunk loads the available performance counters for the selected object.

Note: You can only add one performance object per data input. This is due to how Microsoft handles performance monitor objects. Many objects enumerate classes that describe themselves dynamically upon selection. This can lead to confusion as to which performance counters and instances belong to which object, as defined in the input. If you need to monitor multiple objects, create additional data inputs for each object.

7. Under Counters, choose the counters in the Available counters list box that you want Splunk to monitor by clicking once on them.

The selected counter moves from the Available counters list box to the Selected counters list box.

8. Under Instances, select the instances you want Splunk to monitor by clicking on those instances in the Available instances list.

The selected instance moves from the Available instances list box to the Selected instances list box.

Note: The "_Total" instance is a special instance, and is present for many types of performance counters. This instance is defined as the average of any associated instances under the same counter. Data collected for this instance can be significantly different than for individual instances under the same counter.

For example, when monitoring performance data for the "Disk Bytes/Sec" performance counter under the "PhysicalDisk" object on a system with two disks installed, the available instances displayed include one for each physical disk - "0 C:" and "1 D:" - as well as the "_Total" instance. In this case, the "_Total" instance is the average of the two physical disk instances.

9. Specify an interval, in seconds, between polls.

10. Choose the destination index for this collection.

11. Click Save.

Splunk adds and enables the input.

Configure local Windows performance monitoring with configuration files

inputs.conf controls performance monitoring configurations. To set up performance monitoring using configuration files, create and/or edit inputs.conf in %SPLUNK_HOME%\etc\system\local. If you haven't worked with Splunk's configuration files before, be sure to read "About configuration files" before you begin.

the [perfmon://<name>] stanza defines performance monitoring inputs in inputs.conf. You specify one stanza per performance object that you wish to monitor.

In each stanza, you can specify the following attributes:

Attribute	Required?	Description
interval	Yes	How often, in seconds, to poll for new data. If this attribute is not present and defined, the input will not run, as there is no default.
object	Yes	The performance object(s) that you wish to capture. Specify either a string which exactly matches (including case) the name of an existing Performance Monitor object or use a regular expression to reference multiple objects. If this attribute is not present and defined, the input will not run, as there is no default.
counters	Yes	One or more valid performance counters that are associated with the object specified in object. Separate multiple counters with semicolons. You can also use an asterisk (*) to specify all available counters under a given object. If this attribute is not present and defined, the input will not run, as there is no default.
instances	No	One or more valid instances associated with the performance counter specified in counters. Multiple instances are separated by semicolons. You can specify all instances by using an asterisk (*), which is the default if you do not define the attribute in the stanza.
index	No	The desired index to route performance counter data to. If not present, the default index is used.
disabled	No	Whether or not to gather the performance data defined in this input. Set to 1 to disable this stanza, and 0 to enable it. If not present, it defaults to 0 (enabled).
samplingInterval	No	Advanced option. How often, in milliseconds, that Splunk should collect performance data. This attribute enables high-frequency performance sampling. When high-frequency performance sampling is enabled, Splunk collects performance data every interval and reports the average of the data as well as other statistics. It defaults to 100 ms, and must be less than what you specify with the interval attribute.
stats	No	Advanced option. A semicolon-separated list of statistic values which Splunk reports for high-frequency performance sampling. Allowed values are: average, min, max, dev, and count. The default is no setting (disabled).
mode	No	Advanced option. When you enable high-performance sampling, this attribute controls how Splunk outputs events. Allowed values are: single, multikv, multiMS, and multikvMS When you enable either multiMS or multikvMS, Splunk outputs two events for each performance metric collected. The first event is the average value, and the second is the statistics event. The statistics event has a special sourcetype depending on which output mode you use (perfmonMSStats for multiMS and perfmonMKMSStats for multikvMS) If you do not enable high-performance sampling, the multikvMS output mode is the same as the multikv output mode. The default is single.

The following example section of inputs.conf collects performance data from the local disk on the system and places it into the 'perfmon' index:

# Query the PhysicalDisk performance object and gather disk access data for
# all physical drives installed in the system. Store this data in the 
# "perfmon" index.
# Note: If the interval attribute is set to 0, Splunk resets the interval
# to 1.

[perfmon://LocalPhysicalDisk]
interval = 0
object = PhysicalDisk
counters = Disk Bytes/sec; % Disk Read Time; % Disk Write Time; % Disk Time
instances = *
disabled = 0
index = PerfMon

# Gather SQL statistics for all database instances on this SQL server.
# 'object' attribute uses a regular expression "\$.*" to specify SQL
# statistics for all available databases.
[perfmon://SQLServer_SQL_Statistics]
object = MSSQL\$.*:SQL Statistics
counters = *
instances = *

# Gather information on all counters under the "Process" and "Processor" 
# Perfmon objects.
# We use '.*' as a wild card to match the 'Process' and 'Processor' objects.
[perfmon://ProcessandProcessor]
object = Process.*
counters = *
instances = *

Important information about specifying performance monitor objects in inputs.conf

Use all lower case when specifying the perfmon keyword

When you create a performance monitor input in inputs.conf, you must use all lower case for the perfmon keyword, for example:

Correct	Incorrect
[perfmon://CPUTime]	[Perfmon://CPUTime] [PERFMON://CPUTime]

If you use capital or mixed-case letters for the keyword, Splunk warns of the problem on start-up, and the specified performance monitor input does not run.

Specify valid regular expressions to capture multiple performance monitor objects

When you need to specify multiple objects in a single performance monitor stanza, you must use a valid regular expression to capture those objects. For example, to specify a wildcard to match a string beyond a certain number of characters, do not use *, but rather .*. If the object contains a dollar sign or similar special character, you might need to escape it with a backslash (\).

Except in the above case, values must exactly match what is in the Performance Monitor API

When you specify values for the object, counters and instances attributes in [perfmon://] stanzas, be sure that those values exactly match those defined in the Performance Monitor API, including case, or the input might return incorrect data, or no data at all. If Splunk is unable to match a performance object, counter or instance value that you've specified, it logs that failure to splunkd.log. For example:

01-27-2011 21:04:48.681 -0800 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-perfmon.exe" -noui" splunk-perfmon - PerfmonHelper::enumObjectByNameEx: PdhEnumObjectItems failed for object - 'USB' with error (0xc0000bb8): The specified object is not found on the system.

The best way to ensure that you specify the correct objects, counters, and instances is to use Splunk Web to add performance monitor data inputs.

Enable remote Windows performance monitoring over WMI

You can configure remote performance monitoring either in Splunk Web or by using configuration files.

When collecting performance metrics over WMI, you must configure Splunk to run as an AD user with appropriate access for remote collection of performance metrics. You must do this before attempting to collect those metrics. Both the machine that runs Splunk and the machine(s) Splunk collects performance data from must reside in the same AD domain or forest.

Note: WMI self-throttles by design to prevent denial of service attacks. Splunk also reduces the number of WMI calls it makes over time as a precautionary measure if these calls return an error. Depending on the size, configuration, and security profile of your network, installing a local forwarder on the system from which you want to collect performance metrics might be a better choice. Consult "Considerations for deciding how to monitor remote Windows data" in this manual for additional information.

Important information regarding WMI-based performance metrics

When gathering remote performance metrics through WMI, you might notice that some metrics return zero values, or values that are not in line with values returned by Performance Monitor. This is because of a limitation in the implementation of WMI for performance monitor counters, and is not an issue with Splunk or how it retrieves WMI-based data..

WMI uses the Win32_PerfFormattedData_* classes to gather performance metrics. More info on the specific classes is available at "Win32 Classes" (http://msdn.microsoft.com/en-us/library/aa394084%28v=vs.85%29.aspx) on MSDN.

WMI defines the data structures within these classes as either 32- or 64-bit unsigned integers, depending on the version of Windows you are running. Performance Monitor objects, meanwhile, are defined as floating-point variables. This means that you might see WMI-based metrics that appear anomalous, due to rounding factors.

For example, if you collect data on the "Average Disk Queue Length" Performance Monitor counter at the same time you collect the Win32_PerfFormattedData_PerfDisk_PhysicalDisk\AvgDiskQueueLength metric through WMI, the WMI-based metric might return zero values even though the Performance Monitor metric returns values greater than zero (but less than 0.5). This is because WMI rounds the value down before displaying it.

If you require additional granularity in your performance metrics, it's better to configure the performance monitoring inputs on a universal forwarder on each machine from which you wish to collect performance data. You can then forward that data to an indexer. Data retrieved using this method is more reliable than data gathered remotely using WMI-based inputs.

Configure remote Windows performance monitoring with Splunk Web

1. Click System in the upper right-hand corner of Splunk Web.

2. Under Data, click Data Inputs.

3. Click Remote Performance monitoring.

4. Click New to add an input.

5. Enter a unique name for this collection.

6. Under Select target host, enter the name of a valid Windows host to query performance monitor objects from, then click "Query..."

Splunk connects to the host and gets the available performance objects.

7. In the "Available objects" drop-down, select the performance object whose counters you wish to display.

Splunk loads the available performance counters for the selected object.

Note: You can only add one performance object per data input. This is due to how Microsoft handles performance monitor objects. Many objects enumerate classes that describe themselves dynamically upon selection. This can lead to confusion as to which performance counters and instances belong to which object, as defined in the input. If you need to monitor multiple objects, create additional data inputs for each object.

8. Under Counters, choose the counters in the "Available counters" list box that you want Splunk to monitor by clicking once on them.

The selected counter moves from the "Available counters" list box to the "Selected counters" list box.

9. Next, under Instances, select the instances you want Splunk to monitor by clicking on those instances in the Available instances list.

The selected instance moves from the "Available instances" list box to the "Selected instances" list box.

Note: The "_Total" instance is a special instance, and is present for many types of performance counters. This instance is defined as the average of any associated instances under the same counter. Data collected for this instance can be - and oftentimes is - significantly different than for individual instances under the same counter.

For example, when monitoring performance data for the "Disk Bytes/Sec" performance counter under the "PhysicalDisk" object on a system with two disks installed, the available instances displayed include one for each physical disk - "0 C:" and "1 D:" - as well as the "_Total" instance. In this case, the "_Total" instance is the average of the two physical disk instances.

10. You can optionally tell Splunk to collect the same set of metrics from additional hosts by specifying those hosts, separated by commas, in the field provided.

11. Specify an interval, in seconds, between polls.

12. Optionally, choose the destination index for this collection.

By default, Splunk selects the "default" index.

13. Click Save.

Splunk adds and enables the input.

Note: Win32_PerfFormattedData_* classes do not show up as available objects in Splunk Web. If you wish to monitor Win32_PerfFormattedData_* classes, you must add them directly in wmi.conf.

Configure remote Windows performance monitoring with configuration files

Remote performance monitoring configurations are controlled by wmi.conf. To set up remote performance monitoring using configuration files, create and/or edit wmi.conf in %SPLUNK_HOME%\etc\system\local. If you haven't worked with Splunk's configuration files before, be sure to read "About configuration files" before you begin.

Caution: Splunk strongly recommends that you use Splunk Web to create remote performance monitor inputs. This is because the names of performance monitor objects, counters, and instances must exactly match what the Performance Monitor API defines, including case. Splunk Web uses WMI to get the properly-formatted names, eliminating the potential for typos.

wmi.conf contains one stanza for each remote performance monitor object that you wish to monitor. In each stanza, you specify:

Global settings

Attribute	Required?	Description	Default
initial_backoff	No	How long, in seconds, to wait before retrying a connection to a WMI provider when an error occurs. If Splunk continues to have problems connecting to the provider, then it will double the wait time between connection attempts until either it can connect, or until the wait time is greater than or equal to the integer specified in max_backoff.	5
max_backoff	No	The maximum amount of time, in seconds to attempt to reconnect to a WMI provider.	20
max_retries_at_max_backoff	No	How many times, after Splunk has reached max_backoff seconds between reconnection attempts with a WMI provider, to continue to attempt to reconnect to that provider.	2
checkpoint_sync_interval	No	How long, in seconds, to wait for state data to be flushed to disk.	2

Input-specific settings

Attribute	Required?	Description	Default
interval	Yes	How often, in seconds, to poll for new data. If this attribute is not present, the input will not run, as there is no default.	N/A
server	No	One or more valid servers against which you wish to monitor performance. Multiple entries are separated by commas.	The local machine
event_log_file	No	The names of one or more Windows event log channels to poll. This attribute tells Splunk that the incoming data is in event log format. Note: Do not use the event_log_file attribute in a stanza that already contains the wql attribute.	N/A
wql	No	A valid Windows Query Language (WQL) statement that specifies the performance object(s), counter(s), and instance(s) you wish to poll remotely. This attribute tells Splunk to expect data from a WMI provider. Note: Do not use the wql attribute in a stanza that already contains the event_log_file attribute.	N/A
namespace	No	The namespace in which the WMI provider you want to query resides. The value for this attribute can be either relative (Root\CIMV2) or absolute (\\SERVER\Root\CIMV2), but must be relative if you specify the server attribute. Note: Only use the namespace attribute in a stanza that contains the wql attribute.	Root\CIMV2
index	No	The desired index to route performance counter data to.	default
current_only	No	The characteristics and interaction of WMI-based event collections. if wql is defined, this attribute tells Splunk whether or not it should expect an event notification query. Set to 1 to tell Splunk to expect an event notification query, and 0 to tell it expect a standard query. See below for additional requirements on WQL and event notification queries. if event_log_file is defined, tells Splunk whether or not to only capture events that occur when Splunk is running. Set to 1 to tell Splunk to only capture events that occur when Splunk is running, and 0 to gather events from the last checkpoint or, if no checkpoint exists, the oldest events available.	N/A
disabled	No	Tells Splunk whether or not to gather the performance data defined in this input. Set this to 1 to disable performance monitoring for this stanza, and 0 to enable it.	0

The following example of wmi.conf gathers local disk and memory performance metrics and places them into the 'wmi_perfmon' index:

[settings]
initial_backoff = 5
max_backoff = 20
max_retries_at_max_backoff = 2
checkpoint_sync_interval = 2

# Gather disk and memory performance metrics from the local system every second.
# Store event in the "wmi_perfmon" Splunk index.

[WMI:LocalPhysicalDisk]
interval = 1
wql = select Name, DiskBytesPerSec, PercentDiskReadTime,PercentDiskWriteTime, PercentDiskTime from \
 Win32_PerfFormattedData_PerfDisk_PhysicalDisk
disabled = 0
index = wmi_perfmon

[WMI:LocalMainMemory]
interval = 10
wql = select CommittedBytes, AvailableBytes, PercentCommittedBytesInUse, Caption from \
 Win32_PerfFormattedData_PerfOS_Memory
disabled = 0
index = wmi_perfmon

Additional information on WQL query statements

WQL queries must be structurally and syntactically correct. If they are not, you might get undesirable results or no results at all when specifying them. In particular, when writing event notification queries (by specifying current_only=1 in the stanza in which a WQL query resides), your WQL statement must contain one of the clauses that specify such a query (WITHIN, GROUP, and/or HAVING). Review this MSDN article on Querying with WQL for additional information.

Splunk Web eliminates problems with WQL syntax by generating the appropriate WQL queries when you use it to create performance monitor inputs.

Caveats

Increased memory usage during collection of performance metrics

When collecting data on some performance objects, such as the "Thread" object and its associated counters, you might notice increased memory usage in Splunk. This is normal, as certain performance objects consume more memory than others during the collection process.

Processor Time counters do not return values of higher than 100

Due to how Microsoft tallies CPU usage with the Processor:% Processor Time and Process:% Processor Time counters, these counters do not return a value of more than 100 regardless of the number of CPUs or cores in the system. This is by design - these counters subtract the amount of time spent on the Idle process from 100%.