Monitor WMI-based data
Splunk Enterprise supports the use of Windows Management Instrumentation (WMI) providers for agentless access to Windows performance and event log data on remote machines. This means you can pull event logs from all the Windows servers and desktops in your environment without having to install anything on those machines.
Important: If it is possible for you to do so, Splunk recommends the use of a universal forwarder rather than WMI to collect data from remote machines. The resource load of WMI can exceed that of a Splunk universal forwarder in many cases. In particular, consider a forwarder if you are collecting multiple event logs or performance counters from each host, or from very busy hosts like domain controllers. Review "Considerations for deciding how to monitor remote Windows data" in this manual for additional information.
The WMI-based data inputs can connect to multiple WMI providers and get data from them. The WMI-based data input runs as a separate process on the Splunk Enterprise server called
splunk-wmi.exe. It is a scripted input.
What's required to monitor WMI-based data?
Here are the basic minimum requirements to monitor WMI-based data. You might need additional permissions based on the logs or performance counters you want to monitor.
For additional details on what's required to monitor WMI-based data, read "Security and remote access considerations" later in this topic.
|Monitor remote event logs over WMI||* Splunk Enterprise must run on Windows|
* Splunk Enterprise must run as a domain user with at least read access to WMI
* Splunk Enterprise must run as a domain user with appropriate access to the desired event logs
|Monitor remote performance monitor counters over WMI||* Splunk Enterprise must run on Windows|
* Splunk Enterprise must run as a domain user with at least read access to WMI
* Splunk Enterprise must run as a domain user with appropriate access to the Performance Data Helper libraries
Security and remote access considerations
Splunk Enterprise and your Windows network must be correctly configured for WMI data access. Review the following prerequisites before attempting to use Splunk Enterprise to get WMI data.
Before Splunk Enterprise can get WMI-based data:
- It must be installed with a user that has permissions to perform remote network connections.
- The user Splunk Enterprise runs as must be a member of an Active Directory (AD) domain or forest and must have appropriate privileges to query WMI providers.
- The Splunk user must also be a member of the local Administrators group on the computer that runs Splunk Enterprise.
- The computer that runs Splunk Enterprise must be able to connect to the remote machine and must have permissions to get the desired data from the remote machine once it has connected.
- Both the Splunk Enterprise server making the query and the target servers being queried must be part of the same AD domain or forest.
The Splunk user does not need to be a member of the Domain Admins group (and for security reasons, should not be). However, you must have domain administrator privileges in order to configure access for the Splunk user. If you don't have domain administrator access, then find someone who can either configure Splunk user access for you or give domain administrator rights to you.
Note: If you install Splunk Enterprise as the Local System user, remote authentication over WMI does not work. The Local System user has no credentials to any other machines on the network. It is not possible to grant privileges to a machine's Local System account for access to another machine.
You can give the Splunk user access to WMI providers by doing one of the following:
- Adding it to the local Administrators group on each member server you want to poll (not recommended for security reasons).
- Adding it to the Domain Admins global group (not recommended for security reasons).
- Assigning least-permissive rights as detailed below (recommended).
Important notice regarding group memberships and resource access control lists (ACLs)
To maintain security integrity, Splunk strongly recommends that you place Splunk users into a domain global group and assign permissions on servers and resource ACLs to that group, instead of assigning permissions directly to the user. Assigning permissions directly to users is a security risk, and can cause problems during security audits or future changes.
Configure WMI for least permissive access
If the user you've configured Splunk Enterprise to run as is not a domain administrator, you must configure WMI to provide access to this user. Splunk strongly recommends granting only least-permissive access to all Windows resources, including WMI. In order to grant this type of access, follow this checklist. For additional information and step-by-step instructions, refer to "How to enable WMI access for non-administrator domain users" in the Splunk Community Wiki.
There are several levels of access you must grant to the user Splunk Enterprise runs as in order for Splunk Enterprise to collect data over WMI using the least-permissive method:
1. Local Security Policy Permissions. The Splunk user needs the following Local Security Policy user rights assignments defined on each machine you poll for WMI-based data:
- Access this Computer from the Network
- Act as part of the operating system
- Log on as a batch job
- Log on as a service
- Profile System Performance
- Replace a process level token
Note: To deploy these user rights assignments domain-wide, use the Domain Security Policy (
dompol.msc) Microsoft Management Console (MMC) snap-in. Once deployed, those rights assignments will be inherited by any member servers on the network during the next AD replication cycle (though you must restart Splunk Enterprise instances on those servers for the changes to take effect.) To extend this access to domain controllers specifically, assign the rights using the Domain Controller Security Policy (
2. Distributed Component Object Model (DCOM) configuration and permissions. DCOM must be enabled on every machine you want to monitor. In addition, the Splunk user must be assigned permissions to access DCOM. There are many methods available to do this, but the best is to nest the "Distributed COM Users" domain global group into the "Distributed COM Users" local group on each server you want to monitor, then add the Splunk user to the "Distributed COM Users" domain global group. There are also a number of advanced ways to ensure that the Splunk user has access to DCOM. Review "Securing a Remote WMI Connection" (http://msdn.microsoft.com/en-us/library/aa393266(VS.85).aspx) on MSDN for additional details.
3. Performance Monitor configuration and permissions. In order for Splunk Enterprise to access remote performance objects over WMI, the user it runs as must be a member of the Performance Log Users local group on every member server you want to monitor. The best way to do this is to nest the "Performance Log Users" domain global group into the "Performance Log Users" local group on each member server and then assign the Splunk user to the global group.
4. WMI namespace security. The WMI namespace that Splunk Enterprise accesses (most commonly
Root\CIMV2) must have the proper permissions set. These permissions must be set on each server in your enterprise, as there is no global WMI security. Use the WMI Security MMC snap-in (
wmimgmt.msc) to enable the following permissions on the WMI tree for each host at the Root namespace for the Splunk user:
- Execute Methods
- Enable Account
- Remote Enable
- Read Security
These rights must be assigned to the Root namespace and all subnamespaces below it.
See the Microsoft Knowledge Base article "HOW TO: Set WMI Namespace Security in Windows Server 2003" (http://support.microsoft.com/kb/325353) in the Microsoft Knowledgebase for more information.
Note: There is no standard facility for deploying WMI security settings remotely to multiple machines at once using Group Policy. However, "Set WMI namespace security via GPO" (http://blogs.msdn.com/spatdsg/archive/2007/11/21/set-wmi-namespace-security-via-gpo-script.aspx) on MSDN Blogs offers instructions on how to create a startup script that you can place inside a Group Policy Object (GPO), which will set the namespace security once the GPO is applied to the desired hosts. You can then deploy this GPO domain-wide or to one or more Organizational Units (OUs).
5. Firewall configuration. If you have a firewall enabled, you must configure it to allow access for WMI. If you are using the Windows Firewall included with Windows XP, Windows Server 2003/2003 R2, Windows Vista, Windows 7, and WIndows Server 2008/2008 R2, the exceptions list explicitly includes WMI. You must set this exception for both the originating and the target servers. See "Connecting to WMI Remotely Starting with Vista" (http://msdn.microsoft.com/en-us/library/aa822854(VS.85).aspx) on MSDN for more details.
6. User access control (UAC) configuration. If you run Windows Vista, Windows 7, or Windows Server 2008/2008 R2, UAC affects how Windows assigns permissions. Review "User Account Control and WMI" (http://msdn.microsoft.com/en-us/library/aa826699(v=vs.85).aspx) on MSDN for additional information on the ramifications of UAC and WMI remote polling.
Test access to WMI providers
Once you've configured WMI and set up the Splunk user for access to your domain, you should test the configuration and access to the remote machine.
Important: This procedure includes steps to temporarily change the Splunk Enterprise data store directory (the location
SPLUNK_DB points to). You must do this before testing access to WMI. Failure to do so can result in missing WMI events. This is because the splunk-wmi.exe process updates the WMI checkpoint file every time it runs.
To test access to WMI providers:
1. Log into the machine Splunk Enterprise runs on as the Splunk user.
Note: If attempting to log into a domain controller, you might have to change your domain controller security policy to assign the "Allow log on locally" policy for the designated user.
2. Open a command prompt window (click Start -> Run and type
3. Go to the
bin subdirectory under your Splunk Enterprise installation (for example,
cd c:\Program Files\Splunk\bin).
4. Determine where Splunk Enterprise currently stores its data by running the following command:
> splunk show datastore-dir
Note: You must authenticate into your Splunk Enterprise instance in order to do this. Once you have, be sure to note where Splunk Enterprise stores its data. You'll need to remember it for later.
5. Run the following command to change where Splunk Enterprise stores its data temporarily:
> splunk set datastore-dir %TEMP%
Note: This example sets the data store directory to the current directory specified in the TEMP environment variable. If you want to set it to a different directory, you can do so, but the directory must already exist.
6. Restart Splunk Enterprise by running the following command:
> splunk restart
Note: It might take a while for Splunk Enterprise to restart. This is because it's creating a new data store at the area you specified in Step 5.
7. Once Splunk Enterprise has restarted, test access to WMI providers by running the following command, replacing
<server> with the name of the remote server:
> splunk cmd splunk-wmi -wql "select * from win32_service" -namespace \\<server>\root\cimv2
8. If you see data streaming back and no error messages, that means Splunk Enterprise was able to connect to the WMI provider and query successfully.
9. If there is an error, a message with a reason on what caused the error will appear. Look for the
error="<msg>" string in the output for clues on how to correct the problem.
After testing WMI access, be sure to point Splunk Enterprise back to the correct database directory by running the following command, and then restarting Splunk Enterprise:
> splunk set datastore-dir <directory shown from Step 4>
Configure WMI-based inputs
All remote data collection in Splunk Enterprise on Windows happens through either WMI providers or a forwarder. Review "Considerations for deciding how to monitor remote Windows data" in this manual for additional information about remote data collection.
You can configure WMI-based inputs either in Splunk Web or by editing configuration files. More options are available when using configuration files.
Configure WMI-based inputs with Splunk Web
To add WMI-based inputs, consult one of the appropriate topics in this manual:
Configure WMI-based inputs with configuration files
wmi.conf handles remote data collection configurations. Review this file to see the default values for WMI-based inputs. If you want to make changes to the default values, edit a copy of
%SPLUNK_HOME%\etc\system\local\. Only set values for the attributes you want to change for a given type of data input. Refer to About configuration files in the Admin Manual for more information about how Splunk Enterprise uses configuration files.
wmi.conf contains several stanzas:
[settings]stanza, which specifies global WMI parameters.
- One or more input-specific stanzas, which define how to connect to WMI providers to get data from the remote machine.
[settings] stanza specifies global WMI parameters. The entire stanza and every parameter within it are optional. If the stanza is missing, Splunk Enterprise assumes system defaults.
When Splunk Enterprise is not able to connect to a defined WMI provider, it generates an error in splunkd.log. For example:
05-12-2011 02:39:40.632 -0700 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - Unable to connect to WMI namespace "\\w2k3m1\root\cimv2" (attempt to connect took 42.06 seconds) (error="The RPC server is unavailable." HRESULT=800706BA)
The following attributes control how Splunk Enterprise reconnects to a given WMI provider when an error occurs:
||Tells Splunk Enterprise how long, in seconds, to wait the first time after an error occurs before trying to reconnect to the WMI provider. If connection errors continue to occur, Splunk Enterprise doubles the wait time until it reaches the value specified in
||Tells Splunk Enterprise the maximum amount of time, in seconds, that it should wait between connection attempts, before invoking
||If the wait time between connection attempts reaches
||Tells Splunk Enterprise how long, in seconds, to wait for state data (event log checkpoint) to be written to disk.||2|
Input-specific stanzas tell Splunk Enterprise how to connect to WMI providers on remote machines to get data. They are defined by one of two attributes that specify the type of data Splunk Enterprise should gather. The stanza name can be anything, but usually begins with
WMI:, for example:
When you configure WMI-based inputs in Splunk Web, Splunk Enterprise uses this naming convention for input-specific stanza headers.
You can specify one of two types of data inputs in an input-specific stanza:
- Event log. The
event_log_fileattribute tells Splunk Enterprise to expect event log data from the sources defined in the stanza.
- Windows Query Language (WQL). The
wqlattribute tells Splunk Enterprise to expect data from a WMI provider. When using this attribute, you must also specify a valid WQL statement. This attribute must be used when collecting performance data.
Caution: Do not define both of these attributes in one stanza. Use only one or the other. Otherwise, the input defined by the stanza will not run.
The common parameters for both types of inputs are:
||A comma-separated list of servers from which to get data. If this parameter is missing, Splunk Enterprise assumes that you want to connect to the local machine.||The local server|
||Tells Splunk Enterprise how often, in seconds, to poll for new data. If this attribute is not present and defined, the input that the stanza defines will not run.||N / A|
||Tells Splunk Enterprise whether this input is enabled or disabled. Set this parameter to 1 to disable the input, and 0 to enable it.||0 (enabled)|
The event log-specific parameters are:
||A comma-separated list of event log channels to monitor.||N / A|
||Tells Splunk Enterprise whether or not to collect events that occur only when it is running. If events are generated when Splunk Enterprise is stopped, it will not attempt to index those events when it is started again. Set to 1 to have Splunk Enterprise collect events that occur only when it is running, and 0 to have Splunk Enterprise collect all events.||0 (gather all events)|
||Tells Splunk Enterprise not to normalize the host name that is retrieved from a WMI event. By default, Splunk Enterprise normalizes host names - it produces a single name for the host by identifying various equivalent host names for the local system. Set this parameter to 1 to disable host name normalization in events, and 0 to normalize host names in events.||0 (normalize host names for WMI events)|
The WQL-specific parameters are:
||A valid WQL statement.||N / A|
||(Optional) Specifies the path to the WMI provider. The local machine must be able to connect to the remote machine using delegated authentication. If you don't specify a path to a remote machine, Splunk Enterprise connects to the default local namespace (
||Tells Splunk Enterprise whether or not an event notification query is expected. See "WQL query types: event notification versus standard" below for additional information. Set this attribute to 1 to tell Splunk Enterprise to expect an event notification query, and 0 to expect a standard query.||0 (expect a standard query)|
WQL query types: event notification versus standard
current_only attribute in WQL stanzas determines the type of query the stanza expects to use to collect WMI-based data. When you set the attribute to 1, the stanza expects event notification data. Event notification data is data that alerts you of an incoming event. To get event notification data, you must use an event notification query.
For example, if you want to find out when a remote server spawns processes, you must use an event notification query to get that information. Standard queries have no facilities for notifying you when an event has occurred, and can only return results on information that already exists.
Conversely, if you want to know what already-running processes on your system begin with the word "splunk", you must use a standard query. Event notification queries cannot tell you about static, pre-existing information.
Event notification queries require that the WQL statement defined for the stanza be structurally and syntactically correct. Improperly formatted WQL will cause the input defined by the stanza to not run. Review the wmi.conf configuration file reference for specific details and examples.
WQL query stanzas do not update the WMI checkpoint file
When you use a WQL query stanza to gather data through WMI, Splunk Enterprise does not update the WMI checkpoint file - the file that Splunk Enterprise uses to determine if WMI data has already been indexed. This is by design - a WQL query of any type returns dynamic data and thus a context for saving a checkpoint for the data produced cannot be built. This means that Splunk Enterprise indexes WMI data that it collects through WQL query stanzas as fresh data each time the stanza runs. This can result in the indexing of duplicate events and possibly impact license volume.
If you need to index data regularly, such as event logs, use the appropriate monitor on a universal forwarder. If you must use WMI, do not use a WQL query.
Examples of wmi.conf
The following is an example of a
[settings] initial_backoff = 5 max_backoff = 20 max_retries_at_max_backoff = 2 checkpoint_sync_interval = 2 [WMI:AppAndSys] server = foo, bar interval = 10 event_log_file = Application, System, Directory Service disabled = 0 [WMI:LocalSplunkWmiProcess] interval = 5 wql = select * from Win32_PerfFormattedData_PerfProc_Process where Name = "splunk-wmi" disabled = 0 # Listen from three event log channels, capturing log events that occur only # while Splunk Enterprise is running. Gather data from three servers. [WMI:TailApplicationLogs] interval = 10 event_log_file = Application, Security, System server = srv1, srv2, srv3 disabled = 0 current_only = 1 # Listen for process-creation events on a remote machine [WMI:ProcessCreation] interval = 1 server = remote-machine wql = select * from __InstanceCreationEvent within 1 where TargetInstance isa 'Win32_Process' disabled = 0 current_only = 1 # Receive events whenever someone plugs/unplugs a USB device to/from the computer [WMI:USBChanges] interval = 1 wql = select * from __InstanceOperationEvent within 1 where TargetInstance ISA 'Win32_PnPEntity' and TargetInstance.Description='USB Mass Storage Device' disabled = 0 current_only = 1
Fields for WMI data
- For event log data, Splunk Enterprise sets the source type to
WinEventLog:<name of log file>. For example,
- For WQL data, Splunk Enterprise sets the source type to the name of the stanza that defines the input. For example, for a stanza named
[WMI:LocalSplunkdProcess], Splunk sets the source type to
Splunk Enterprise automatically defines the originating host from the data received.
WMI and event transformations
WMI events are not available for transformation at index time. This means you cannot modify or extract WMI events as Splunk Enterprise indexes them. This is because WMI events arrive as a single source (a scripted input), which means they can only be matched as a single source.
You can, however, modify and extract WMI events at search time. You can also address WMI-based inputs at parse time by specifying the sourcetype
For more information on how to transform events as they arrive in Splunk Enterprise, read "About indexed field extraction" in this manual.
Troubleshooting WMI inputs
If you encounter problems receiving events through WMI providers or are not getting the results you expect, see "Common Issues with Splunk and WMI" in the Troubleshooting Manual.
Monitor file system changes on Windows
Monitor Windows Registry data
This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14