Getting Data In

 


Real-time Windows performance monitoring

NOTE - Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.

Real-time Windows performance monitoring

Performance monitoring is an important part of the Windows administrator's toolkit. Windows generates a lot of data about a system's health. Proper analysis of that data can make the difference between a healthy, well functioning system, and one that suffers many bouts of downtime.

Splunk supports the monitoring of all Windows performance counters available to the system in real time, and includes support for both local and remote collection of performance data.

Splunk's performance monitoring utility gives you the abilities of Performance Monitor in a web or command-line interface. Splunk uses the Performance Data Helper (PDH) API for performance counter queries on local machines.

The types of performance objects, counters and instances that are available to Splunk depend on the performance libraries installed on the system. Both Microsoft and third-party vendors provide libraries that contain performance counters. For additional information on performance monitoring, review "Performance Counters" (http://msdn.microsoft.com/en-us/library/aa373083%28v=VS.85%29.aspx) on MSDN.

Both full instances of Splunk and universal forwarders support local collection of performance metrics. Remote performance monitoring is available through WMI (Windows Management Instrumentation) and requires that Splunk runs as a user with appropriate Active Directory credentials.

The performance monitor input runs as a process called splunk-perfmon.exe. This process will run once for every input defined, at the interval specified in the input. You can configure performance monitoring using Splunk Web, or either perfmon.conf (for getting local performance data) or wmi.conf (for getting performance data from a remote machine).

Security and remote access considerations

Splunk gets data from remote machines using either WMI or a forwarder. Splunk recommends using a universal forwarder to send performance data from remote machines to an indexer. Review "Introducing the universal forwarder" in the Distributed Deployment Manual for information about how to install, configure and use the forwarder to collect performance metrics.

If you choose to install forwarders on your remote machines to collect performance data, then you can install the forwarder as the Local System user on those machines. The Local System user has access to all data on the local machine, but not to remote machines.

If you want Splunk to use WMI to get performance data from remote machines, then you must ensure that your network and Splunk instances are properly configured. You cannot install Splunk as the Local System user, and the user you install with determines the set of performance metrics Splunk will see. Review "Security and remote access considerations" in the "Monitor WMI Data" topic in this manual for additional information on the requirements you must satisfy in order for Splunk to collect remote data properly using WMI.

After you install Splunk with a valid user, add that user to the following groups before enabling local performance monitor inputs:

  • Performance Monitor Users (domain group)
  • Performance Log Users (domain group)

Enable local Windows performance monitoring

You can configure local performance monitoring either in Splunk Web, or by using configuration files.

Splunk Web is the preferred way to add performance monitoring data inputs. This is because you can make typos when using configuration files, and it's important to specify performance monitor objects exactly as they are defined in the Performance Monitor API. See "Important information about specifying performance monitor objects in perfmon.conf" below for a full explanation.

Configure local Windows performance monitoring with Splunk Web

1. Click Manager in the upper right-hand corner of Splunk Web.

2. Under Data, click Data Inputs.

3. Click Local performance monitoring.

4. Click New to add an input.

5. Enter a unique, memorable name for this input.

6. Under Available objects, choose the performance object whose counters you wish to display.

Splunk loads the available performance counters for the selected object.

Note: You can only add one performance object per data input. This is due to how Microsoft handles performance monitor objects. Many objects enumerate classes that describe themselves dynamically upon selection. This can lead to confusion as to which performance counters and instances belong to which object, as defined in the input. If you need to monitor multiple objects, create additional data inputs for each object.

7. Under Counters, choose the counters in the Available counters list box that you want Splunk to monitor by clicking once on them.

The selected counter moves from the Available counters list box to the Selected counters list box.

8. Under Instances, select the instances you want Splunk to monitor by clicking on those instances in the Available instances list.

The selected instance moves from the Available instances list box to the Selected instances list box.

Note: The "_Total" instance is a special instance, and is present for many types of performance counters. This instance is defined as the average of any associated instances under the same counter. Data collected for this instance can be significantly different than for individual instances under the same counter.

For example, when monitoring performance data for the "Disk Bytes/Sec" performance counter under the "PhysicalDisk" object on a system with two disks installed, the available instances displayed include one for each physical disk - "0 C:" and "1 D:" - as well as the "_Total" instance. In this case, the "_Total" instance is the average of the two physical disk instances.

9. Specify an interval, in seconds, between polls.

10. Choose the destination index for this collection.

11. Click Save.

Splunk adds and enables the input.

Configure local Windows performance monitoring with configuration files

Performance monitoring configurations are controlled by perfmon.conf. To set up performance monitoring using configuration files, create and/or edit perfmon.conf in %SPLUNK_HOME%\etc\system\local. If you haven't worked with Splunk's configuration files before, be sure to read "About configuration files" before you begin.

perfmon.conf contains one stanza, where you specify:

Attribute Required? Description
interval Yes How often, in seconds, to poll for new data. If this attribute is not present and defined, the input will not run, as there is no default.
object Yes The performance object that you wish to capture. If this attribute is not present and defined, the input will not run, as there is no default.
counters Yes One or more valid performance counters that are associated with the object specified in object. Multiple counters are separated by semicolons. You can also use an asterisk (*) to specify all available counters under a given object. If this attribute is not present and defined, the input will not run, as there is no default.
instances Yes, at least one One or more valid instances associated with the performance counter specified in counters. Multiple instances are separated by semicolons. You can specify all instances by using an asterisk (*).
index No The desired index to route performance counter data to. If not present, the default index is used.
disabled No Whether or not to gather the performance data defined in this input. Set to 1 to disable this stanza, and 0 to enable it. If not present, it defaults to 0 (enabled).

The following example of perfmon.conf collects performance data from the local disk on the system and places it into the 'perfmon' index:

# Query the PhysicalDisk performance object and gather disk access data for
# all physical drives installed in the system. Store this data in the 
# "perfmon" index.
# Note: If the interval attribute is set to 0, Splunk will reset the interval
# to 1.

[Perfmon:LocalPhysicalDisk]
interval = 0
object = PhysicalDisk
counters = Disk Bytes/sec; % Disk Read Time; % Disk Write Time; % Disk Time
instances = *
disabled = 0
index = PerfMon

Important information about specifying performance monitor objects in perfmon.conf

When specifying values for the object, counters and instances attributes in perfmon.conf stanzas, be sure that those values exactly match those defined in the Performance Monitor API, including case, or the input will return incorrect data, or no data at all. If Splunk is unable to match a performance object, counter or instance value that you've specified in perfmon.conf, it will log that failure to splunkd.log. For example:

01-27-2011 21:04:48.681 -0800 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-perfmon.exe" -noui" splunk-perfmon - PerfmonHelper::enumObjectByNameEx: PdhEnumObjectItems failed for object - 'USB' with error (0xc0000bb8): The specified object is not found on the system.

The best way to ensure that you specify the correct objects, counters, and instances is to use Splunk Web to add performance monitor data inputs.

Enable remote Windows performance monitoring over WMI

You can configure remote performance monitoring either in Splunk Web or by using configuration files.

When collecting performance metrics over WMI, you must configure Splunk to run as an AD user with appropriate access for remote collection of performance metrics. You must do this before attempting to collect those metrics. Both the machine running Splunk and the machine(s) Splunk collects performance data from must reside in the same AD domain or forest.

Note: WMI self-throttles by design to prevent denial of service attacks. Splunk will also throttle WMI calls it makes as an additional precautionary measure if these calls return an error. Depending on the size, configuration, and security profile of your network, installing a local forwarder on the system from which you want to collect performance metrics might be a better choice. Consult "Considerations for deciding how to monitor remote Windows data" in this manual for additional information.

Important information regarding WMI-based performance metrics

When gathering remote performance metrics through WMI, you might notice that some metrics return zero values, or values that are not in line with values returned by Performance Monitor. This is because of a limitation in the implementation of WMI for performance monitor counters, and is not an issue with Splunk or how it retrieves WMI-based data..

WMI uses the Win32_PerfFormattedData_* classes to gather performance metrics. More info on the specific classes is available at "Win32 Classes" (http://msdn.microsoft.com/en-us/library/aa394084%28v=vs.85%29.aspx) on MSDN.

The data structures within these classes are defined as either 32- or 64-bit unsigned integers, depending on the version of Windows you are running. Performance Monitor objects, meanwhile, are defined as floating-point variables. This means that you might see WMI-based metrics that appear anomalous, due to rounding factors.

For example, if you collect data on the "Average Disk Queue Length" Performance Monitor counter at the same time you collect the Win32_PerfFormattedData_PerfDisk_PhysicalDisk\AvgDiskQueueLength metric through WMI, the WMI based metric might return zero values even though the Performance Monitor metric is returning values greater than zero (but less than 0.5). This is because WMI rounds the value down before displaying it.

If you require additional granularity in your performance metrics, it's better to configure the performance monitoring inputs on a universal forwarder on each machine from which you wish to collect performance data. You can then forward that data to an indexer. Data retrieved using this method is more reliable than data gathered remotely using WMI-based inputs.

Configure remote Windows performance monitoring with Splunk Web

1. Click Manager in the upper right-hand corner of Splunk Web.

2. Under Data, click Data Inputs.

3. Click Remote Performance monitoring.

4. Click New to add an input.

5. Enter a unique name for this collection.

6. Under Select target host, enter the name of a valid Windows host to query performance monitor objects from, then click "Query..."

Splunk connects to the host and gets the available performance objects.

7. In the "Available objects" drop-down, select the performance object whose counters you wish to display.

Splunk loads the available performance counters for the selected object.

Note: You can only add one performance object per data input. This is due to how Microsoft handles performance monitor objects. Many objects enumerate classes that describe themselves dynamically upon selection. This can lead to confusion as to which performance counters and instances belong to which object, as defined in the input. If you need to monitor multiple objects, create additional data inputs for each object.

8. Under Counters, choose the counters in the "Available counters" list box that you want Splunk to monitor by clicking once on them.

The selected counter moves from the "Available counters" list box to the "Selected counters" list box.

9. Next, under Instances, select the instances you want Splunk to monitor by clicking on those instances in the Available instances list.

The selected instance moves from the "Available instances" list box to the "Selected instances" list box.

Note: The "_Total" instance is a special instance, and is present for many types of performance counters. This instance is defined as the average of any associated instances under the same counter. Data collected for this instance can be - and oftentimes is - significantly different than for individual instances under the same counter.

For example, when monitoring performance data for the "Disk Bytes/Sec" performance counter under the "PhysicalDisk" object on a system with two disks installed, the available instances displayed include one for each physical disk - "0 C:" and "1 D:" - as well as the "_Total" instance. In this case, the "_Total" instance is the average of the two physical disk instances.

10. You can optionally tell Splunk to collect the same set of metrics from additional hosts by specifying those hosts, separated by commas, in the field provided.

11. Specify an interval, in seconds, between polls.

12. Optionally, choose the destination index for this collection.

By default, the "default" index is selected.

13. Click Save.

The input is added and enabled.

Note: Win32_PerfFormattedData_* classes will not show up as available objects. If you wish to monitor Win32_PerfFormattedData_* it needs to be added directly in wmi.conf

Configure remote Windows performance monitoring with configuration files

Remote performance monitoring configurations are controlled by wmi.conf. To set up remote performance monitoring using configuration files, create and/or edit wmi.conf in %SPLUNK_HOME%\etc\system\local. If you haven't worked with Splunk's configuration files before, be sure to read "About configuration files" before you begin.

Caution: Splunk strongly recommends that you use Splunk Web to create remote performance monitor inputs. This is because the names of performance monitor objects, counters, and instances must exactly match what is defined in the Performance Monitor API, including case. Splunk Web uses WMI to get the properly-formatted names, eliminating this problem.

wmi.conf contains one stanza for each remote performance monitor object that you wish to monitor. In each stanza, you specify:

Global settings

Attribute Required? Description Default
initial_backoff No How long, in seconds, to wait before retrying a connection to a WMI provider when an error occurs. If Splunk continues to have problems connecting to the provider, then it will double the wait time between connection attempts until either it can connect, or until the wait time is greater than or equal to the integer specified in max_backoff. 5
max_backoff No The maximum amount of time, in seconds to attempt to reconnect to a WMI provider. 20
max_retries_at_max_backoff No How many times, after Splunk has reached max_backoff seconds between reconnection attempts with a WMI provider, to continue to attempt to reconnect to that provider. 2
checkpoint_sync_interval No How long, in seconds, to wait for state data to be flushed to disk. 2

Input-specific settings

Attribute Required? Description Default
interval Yes How often, in seconds, to poll for new data. If this attribute is not present, the input will not run, as there is no default. N/A
server No One or more valid servers against which you wish to monitor performance. Multiple entries are separated by commas. The local machine
event_log_file No The names of one or more Windows event log channels to poll. This attribute tells Splunk that the incoming data is in event log format.

Note: Do not use the event_log_file attribute in a stanza that already contains the wql attribute.

N/A
wql No A valid Windows Query Language (WQL) statement that specifies the performance object(s), counter(s), and instance(s) you wish to poll remotely. This attribute tells Splunk to expect data from a WMI provider.

Note: Do not use the wql attribute in a stanza that already contains the event_log_file attribute.

N/A
namespace No The namespace in which the WMI provider you want to query resides. The value for this attribute can be either relative (Root\CIMV2) or absolute (\\SERVER\Root\CIMV2), but must be relative if you specify the server attribute.

Note: Only use the namespace attribute in a stanza that contains the wql attribute.

Root\CIMV2
index No The desired index to route performance counter data to. default
current_only No The characteristics and interaction of WMI-based event collections.

  • if wql is defined, this attribute tells Splunk whether or not it should expect an event notification query. Set to 1 to tell Splunk to expect an event notification query, and 0 to tell it expect a standard query. See below for additional requirements on WQL and event notification queries.
  • if event_log_file is defined, tells Splunk whether or not to only capture events that occur when Splunk is running. Set to 1 to tell Splunk to only capture events that occur when Splunk is running, and 0 to gather events from the last checkpoint or, if no checkpoint exists, the oldest events available.
N/A
disabled No Tells Splunk whether or not to gather the performance data defined in this input. Set this to 1 to disable performance monitoring for this stanza, and 0 to enable it. 0

The following example of wmi.conf gathers local disk and memory performance metrics and places them into the 'wmi_perfmon' index:

[settings]
initial_backoff = 5
max_backoff = 20
max_retries_at_max_backoff = 2
checkpoint_sync_interval = 2

# Gather disk and memory performance metrics from the local system every second.
# Store event in the "wmi_perfmon" Splunk index.

[WMI:LocalPhysicalDisk]
interval = 1
wql = select Name, DiskBytesPerSec, PercentDiskReadTime, PercentDiskWriteTime, PercentDiskTime from Win32_PerfFormattedData_PerfDisk_PhysicalDisk
disabled = 0
index = wmi_perfmon

[WMI:LocalMainMemory]
interval = 10
wql = select CommittedBytes, AvailableBytes, PercentCommittedBytesInUse, Caption from Win32_PerfFormattedData_PerfOS_Memory
disabled = 0
index = wmi_perfmon

Additional information on WQL query statements

When building WQL queries, make sure that the queries are structurally and syntactically correct. If you don't, you might get undesirable results, or no results at all. In particular, when writing event notification queries (by specifying current_only=1 in the stanza in which a WQL query resides), your WQL statement must contain one of the clauses that specify such a query (WITHIN, GROUP, and/or HAVING). Review this MSDN article on Querying with WQL for additional information.

Splunk Web eliminates problems with WQL syntax by generating the appropriate WQL queries when it is used to create performance monitor inputs.

Increased memory usage during collection of performance metrics

When collecting data on some performance objects, such as the "Thread" object and its associated counters, you might notice increased memory usage in Splunk. This is normal, as certain performance objects consume more memory than others during the collection process.

This documentation applies to the following versions of Splunk: 4.2 , 4.2.1 , 4.2.2 , 4.2.3 , 4.2.4 , 4.2.5 , 4.3 , 4.3.1 , 4.3.2 , 4.3.3 , 4.3.4 , 4.3.5 , 4.3.6 , 4.3.7 View the Article History for its revisions.


You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!