Monitor WMI data

Monitor WMI data

Splunk supports WMI (Windows Management Instrumentation) data input for agentless access to Windows performance data and event logs. This means you can pull event logs from all the Windows servers and desktops in your environment without having to install anything on those machines.

The Splunk WMI data input can connect to multiple WMI providers and pull data from them. The WMI data input runs as a separate process (splunk-wmi.exe) on the Splunk server. It is configured as a scripted input in %SPLUNK_HOME%\etc\system\default\inputs.conf. Do not make changes to this file.

Note: This feature is only available on the Windows versions of Splunk.

Security and remote access considerations

Splunk requires privileged access to index many Windows data sources, including Event Logs, the Registry, and anything gathered remotely using WMI. This includes both the ability to connect to the computer you wish to poll, as well as permissions to read the appropriate data once connected.

To access WMI providers, Splunk must run as a user with permissions to perform remote WMI connections. This user must be a member of an Active Directory domain, and must have appropriate privileges to query WMI. Both the Splunk server making the query and the target servers being queried must be part of this Active Directory domain.

You'll need to have a domain account that is a member of the Domain Admins group to configure Splunk for this kind of access. If your account isn't a member of the Domain Admins group, you'll need to speak with someone who has that access, and have them perform these steps for you, or alternatively, can make your account a member of the Domain Admins group.

Note: If you installed Splunk as the Local System user, remote authentication over WMI will not work. The Local System user has no credentials to servers other than the machine on which the account resides. It is not possible to grant privileges to a machine's Local System account for access to another machine.

You can give the Splunk user access to WMI by doing one of the following:

• Adding it to the local Administrators group on each member server you wish to poll, or
• Adding it to the Domain Admins global group, or
• Assigning least-permissive rights as detailed below (recommended).

Important notice regarding group memberships and resource Access Control Lists (ACLs)

We strongly discourage either placing domain users into local groups on member servers, or assigning users directly to resource ACLs. Doing so is a security risk. To maintain security integrity, it's best to create a domain global group first, put the Splunk user in this group, and then assign permissions to the group, or place the group into the appropriate local group(s) on the member server.

Permissions Checklist

Following is a least-permissive checklist of requirements that must be satisfied before Splunk can access resources over WMI. For additional information and step-by-step instructions, read this Community Wiki topic on enabling access to WMI for non-administrator users.

While this option is more time consuming than the others, it maintains security integrity by allowing only the minimum permissions needed for Splunk to access WMI providers.

• Local Security Policy Permissions. The Splunk user needs the following Local Security Policy user rights assignments defined on each machine you poll:
  • Access this Computer from the Network
  • Act as part of the operating system
  • Log on as a batch job
  • Log on as a service
  • Profile System Performance
  • Replace a process level token

Note: If you want to deploy these user rights assignments domain-wide, you must use the Domain Security Policy (dompol.msc) MMC snap-in; they'll be inherited by any member servers on the network in the next AD replication cycle (though you will need to restart Splunk instances on those servers for the changes to take effect.) If you need to extend this access to domain controllers specifically, you'll need to assign the rights using the Domain Controller Security Policy (dcpol.msc) snap-in.

• Distributed Component Object Model (DCOM) configuration and permissions. DCOM needs to be enabled on every machine you wish to poll. In addition to this, the Splunk user needs to be assigned permissions to access DCOM. The quickest way to do this is to add the user to the "Distributed COM Users" local group on each member server. The better, more manageable way is to nest the "Distributed COM Users" domain global group into the "Distributed COM Users" local group on the server you want to poll, and then add the Splunk user to the "Distributed COM Users" global group. There are also a number of advanced ways to ensure that the Splunk user has access to DCOM, review this MSDN article on Securing a Remote WMI Connection for additional details.

• Performance Monitor configuration and permissions. In order for Splunk to be able to access remote performance objects over WMI, the user it runs as must be a member of the Performance Log Users local group on every member server you wish to poll. The best way to do this is to nest a domain global group (such as "Performance Log Users") into the "Performance Log Users" local group on each member server, then assign the Splunk user to the global group.

• WMI namespace security. The WMI namespace that Splunk accesses (most commonly Root\CIMV2) must have the proper permissions set. These permissions must be set on each server in your enterprise as there is no global WMI security. Use the WMI Security MMC snap-in (wmimgmt.msc) to enable the following permissions on the WMI tree for each host at the Root namespace for the Splunk user:
  • Execute Methods
  • Enable Account
  • Remote Enable
  • Read Security
  • These rights need to be assigned to the Root namespace, and all subnamespaces below Root.
  • See the Microsoft Knowledge Base article HOW TO: Set WMI Namespace Security in Windows Server 2003 for more information.

Note: At this time, there is no standard facility to deploy WMI security settings remotely to multiple machines at once using Group Policy. This MSDN blog, however, offers instructions on how to create a startup script that you can place inside a Group Policy Object (GPO), which will set the namespace security once the GPO is applied to the desired hosts. You can then deploy this GPO domain-wide, or to one or more Organizational Units (OUs).

• Firewall configuration. If you have a firewall enabled, you must configure it to allow access for WMI. If you are using the Windows Firewall, the exceptions list explicitly includes WMI. You must set this exception for both the originating and the remote machine. See the Microsoft topic on Connecting to WMI Remotely Starting with Vista for more details.

• User access control (UAC) configuration. If you're running Windows Vista, Windows 7 or Windows 2008, UAC is also an important topic to consider, as it also affects how permissions are assigned. Review this MSDN article on UAC and WMI for additional information on the ramifications of UAC and WMI remote polling.

Test access to WMI

Follow these steps to test the configuration of the Splunk server and the remote machine:

1. Log into the machine Splunk runs on as the user Splunk runs as.

Note: If attempting to log into a domain controller, you may need to change your domain controller security policy to assign the "Allow log on locally" policy for the designated account.

2. Open a command prompt window (click Start -> Run and type cmd).

3. Go to the bin sub-directory under your Splunk installation (for example, cd c:\Program Files\Splunk\bin).

4. Run the following command: splunk cmd splunk-wmi -wql "select * from win32_service" -namespace \\<server>\root\cimv2, replacing <server> with the name of the remote server.

5. If you see data streaming back and no error message, that means your were able to connect and query successfully. If there was an error, there would be a message with a reason on why it failed (look for the error="<msg>" string).

Configure WMI input

You can configure WMI input either in Splunk Web or by editing configuration files. More options are available when using configuration files.

Configure WMI with Splunk Web

1. Click Manager in the upper right-hand corner of Splunk Web.

2. Under System configurations, click Data Inputs.

3. Click WMI collections.

4. Click Add new to add an input.

5. Enter a unique name for this collection.

6. Enter a target host and click Query... to get a list of the available classes of properties to choose from.

Note: Only classes that are prefixed with Win32_PerfFormattedData_ are displayed in the list. If the class you wish to index does not start with the Win32_PerfFormattedData_ prefix, you must add it manually by editing wmi.conf.

7. Optionally, provide a comma-separated list of additional servers from which to pull data.

8. Specify an interval in seconds between polls.

9. Ensure the Enabled? radio button is set to Yes and click Save.

The input is added and enabled.

Disabling or deleting WMI inputs

To disable a WMI input, navigate to it and select the No radio button under Enabled?.

It's not possible to delete a WMI input from within Splunk Web. To delete a WMI input, use the information in the section below titled "Configure WMI with configuration files" to edit the wmi.conf file and remove the [WMI:<input_name>] stanza you'd like to delete.

Configure WMI with configuration files

Look at wmi.conf to see the default values for the WMI input. If you want to make changes to the default values, edit a copy of wmi.conf in %SPLUNK_HOME%\etc\system\local\. Only set values for the attributes you want to change for a given type of data input. Refer to About configuration files for more information about how Splunk uses configuration files.

[settings]
initial_backoff = 5
max_backoff = 20
max_retries_at_max_backoff = 2
checkpoint_sync_interval = 2

[WMI:AppAndSys]
server = foo, bar
interval = 10
event_log_file = Application, System, Directory Service
disabled = 0

[WMI:LocalSplunkWmiProcess]
interval = 5
wql = select * from Win32_PerfFormattedData_PerfProc_Process where Name = "splunk-wmi"
disabled = 0

# Listen from three event log channels, capturing log events that occur only
# while Splunk is running.  Gather data from three servers.
[WMI:TailApplicationLogs]
interval = 10
event_log_file = Application, Security, System
server = srv1, srv2, srv3
disabled = 0
current_only = 1

# Listen for process-creation events on a remote machine
[WMI:ProcessCreation]
interval = 1
server = remote-machine
wql = select * from __InstanceCreationEvent within 1 where TargetInstance isa 'Win32_Process'
disabled = 0
current_only = 1

# Receive events whenever someone plugs/unplugs a USB device to/from the computer
[WMI:USBChanges]
interval = 1
wql = select * from __InstanceOperationEvent within 1 where TargetInstance ISA 'Win32_PnPEntity' and TargetInstance.Description='USB Mass Storage Device'
disabled = 0
current_only = 1

The [settings] stanza specifies runtime parameters. The entire stanza and every parameter within it are optional. If the stanza is missing, Splunk assumes system defaults.

• The following attributes control how the agent reconnects to a given WMI provider when an error occurs. All times are in seconds:
  • initial_backoff: how much time to wait the first time after an error occurs before trying to reconnect. Thereafter, if errors keep occurring, the wait time doubles, until it reaches max_backoff.
  • max_backoff: the maximum amount of time to wait before invoking max_retries_at_max_backoff.
  • max_retries_at_max_backoff : if the wait time reaches max_backoff, try this many times at this wait time. If the error continues to occur, Splunk will not reconnect to the WMI provider in question until the Splunk services are restarted.
• checkpoint_sync_interval: minimum wait time for state data (event log checkpoint) to be written to disk. In seconds.

You can specify two types of data input: event log, and raw WQL (WMI query language) The event log input stanza contains the event_log_file parameter, and the WQL input stanza contains wql.

The common parameters for both types are:

• server: a comma-separated list of servers from which to pull data. If this parameter is missing, Splunk assumes the local machine.
• interval : how often to poll for new data, in seconds. Required.
• disabled: indicates whether this feature is enabled or disabled. Set this parameter to 1 to disable WMI input into Splunk.

WQL-specific parameters:

• namespace: specifies the path to the WMI provider. The local machine must be able to connect to the remote machine using delegated authentication. This attrbitue is optional. If you don't specify a path to a remote machine, Splunk will connect to the default local namespace (\root\cimv2), which is where most of the providers you are likely to query reside. Microsoft provides a list of namespaces for Windows XP and later versions of Windows.
• wql: provides the WQL query. The example above polls data about a running process named splunk-wmi every 5 seconds.

Event log-specific parameters:

• event_log_file: specify a comma-separated list of log files to poll in the event_log_file parameter. File names that include spaces are supported, as shown in the example.
• current_only: If set to 1, this option allows you to collect events that occur only while Splunk is running. It behaves like a tail to a file, displaying the last lines of the file as they are written.

You can also use the current_only parameter in raw WQL stanzas to collect WMI event notification data. Check the wmi.conf configuration file reference for examples.

Fields for WMI data

All events received from WMI have the source set to wmi.

• For event log data, the source type is set to WinEventLog:<name of log file> (for example WinEventLog:Application).
• For WQL data, the the source type is set to the name of the config stanza (for example, for a stanza named [WMI:LocalSplunkdProcess], the field is set to WMI:LocalSplunkdProcess).

The host is identified automatically from the data received.

Troubleshooting WMI logging

If you encounter problems receiving WMI events, or are not getting the results you expect, you can enable debugging in Splunk's logging engine in order to get additional data.

In order to enable debugging for WMI, you'll need to set two parameters.

1. Edit log.cfg in %SPLUNK_HOME\etc. Add the following parameter:

[splunkd]
 category.ExecProcessor=DEBUG 

2. Edit log-cmdline.cfg, also in %SPLUNK_HOME%\etc. Add the following parameter:

 category.WMI=DEBUG 

3. Restart Splunk:

C:\Program Files\Splunk\bin> splunk restart

4. Once Splunk has restarted, let it run for a few minutes until you see debug log information coming into Splunk.

Note: You can search Splunk's logfiles within Splunk by supplying index="_internal" as part of your search string. Review this topic on Splunk log files for additional information.

5. Once enough debug logging has been collected, send a diag to Splunk Support:

C:\Program Files\Splunk\bin> splunk diag

Note: Remember to revert back to the default settings below once you finish troubleshooting.

In log.cfg:

category.ExecProcessor=WARN

In log-cmdline.cfg:

category.WMI=ERROR

Note: Any changes made in log.cfg are overwritten when you upgrade Splunk. Create a log-local.cfg for your specific needs to avoid this problem.

For more additional information on troubleshooting WMI, see this Community Wiki topic.

• Was this topic useful?
• Post a comment