Monitor WMI-based data
Splunk supports the use of Windows Management Instrumentation (WMI) providers for agentless access to Windows performance and event log data on remote machines. This means you can pull event logs from all the Windows servers and desktops in your environment without having to install anything on those machines.
Important: If it is possible for you to do so, Splunk recommends the use of a universal forwarder rather than WMI to collect data from remote machines. The resource load of WMI can exceed that of a Splunk universal forwarder in many cases. In particular, consider a forwarder if you are collecting multiple event logs or performance counters from each host, or from very busy hosts like domain controllers. Review "Considerations for deciding how to monitor remote Windows data" in this manual for additional information.
The WMI-based data inputs can connect to multiple WMI providers and get data from them. The WMI-based data input runs as a separate process on the Splunk server called
splunk-wmi.exe. It is configured as a scripted input in
%SPLUNK_HOME%\etc\system\default\inputs.conf. Do not make changes to this file.
Note: This feature is only available on the Windows version of Splunk.
Security and remote access considerations
Before attempting to use Splunk to get data using WMI providers, you'll need to meet the following prerequisites:
- Splunk must be installed as a user with permissions to perform remote connections.
- The user Splunk runs as must be a member of an Active Directory (AD) domain or forest and must have appropriate privileges to query WMI providers.
- The Splunk user must also be a member of the local Administrators group on the computer that is running Splunk.
- The computer that is running Splunk must be able to connect to the remote machine and must have permissions to get the desired data from the remote machine once it has connected.
- Both the Splunk server making the query and the target servers being queried must be part of the same AD domain or forest.
The Splunk user does not need to be a member of the Domain Admins group (and for security reasons, should not be). However, you must have domain administrator privileges in order to configure access for your Splunk user. If you don't have domain administrator access, you must find someone who can either configure Splunk user access for you or give domain administrator rights to you.
Note: If you installed Splunk as the Local System user, remote authentication over WMI will not work. The Local System user has no credentials to any other machines on the network. It is not possible to grant privileges to a machine's Local System account for access to another machine.
You can give the Splunk user access to WMI providers by doing one of the following:
- Adding it to the local Administrators group on each member server you want to poll (not recommended for security reasons).
- Adding it to the Domain Admins global group (not recommended for security reasons).
- Assigning least-permissive rights as detailed below (recommended).
Important notice regarding group memberships and resource access control lists (ACLs)
To maintain security integrity, Splunk strongly recommends that you place Splunk users into a domain global group and assign permissions on servers and resource ACLs to that group, instead of assigning permissions directly to the user. Assigning permissions directly to users is a security risk, and can cause problems during audits or future changes.
Configure WMI for least permissive access
If the user you've configured Splunk to run as is not a domain administrator, you must configure WMI to provide access to this user. Splunk strongly recommends granting only least-permissive access to all Windows resources, including WMI. In order to grant this type of access, follow this checklist. For additional information and step-by-step instructions, refer to "How to enable WMI access for non-administrator domain users" in the Splunk Community Wiki.
There are several levels of access you must grant to the user Splunk runs as in order for Splunk to collect data over WMI using the least-permissive method:
1. Local Security Policy Permissions. The Splunk role account needs the following Local Security Policy user rights assignments defined on each machine you poll:
- Access this Computer from the Network
- Act as part of the operating system
- Log on as a batch job
- Log on as a service
- Profile System Performance
- Replace a process level token
Note: To deploy these user rights assignments domain-wide, use the Domain Security Policy (
dompol.msc) Microsoft Management Console (MMC) snap-in; those rights assignments will be inherited by any member servers on the network during the next AD replication cycle (though you must restart Splunk instances on those servers for the changes to take effect.) To extend this access to domain controllers specifically, assign the rights using the Domain Controller Security Policy (
2. Distributed Component Object Model (DCOM) configuration and permissions. DCOM must be enabled on every machine you want to poll. In addition, the Splunk user must be assigned permissions to access DCOM. There are many methods available to do this, but the best is to nest the "Distributed COM Users" domain global group into the "Distributed COM Users" local group on each server you want to poll, and then add the Splunk user to the "Distributed COM Users" domain global group. There are also a number of advanced ways to ensure that the Splunk user has access to DCOM. Review "Securing a Remote WMI Connection" (http://msdn.microsoft.com/en-us/library/aa393266(VS.85).aspx) on MSDN for additional details.
3. Performance Monitor configuration and permissions. In order for Splunk to access remote performance objects over WMI, the user it runs as must be a member of the Performance Log Users local group on every member server you want to poll. The best way to do this is to nest the "Performance Log Users" domain global group into the "Performance Log Users" local group on each member server and then assign the Splunk user to the global group.
4. WMI namespace security. The WMI namespace that Splunk accesses (most commonly
Root\CIMV2) must have the proper permissions set. These permissions must be set on each server in your enterprise, as there is no global WMI security. Use the WMI Security MMC snap-in (
wmimgmt.msc) to enable the following permissions on the WMI tree for each host at the Root namespace for the Splunk user:
- Execute Methods
- Enable Account
- Remote Enable
- Read Security
These rights must be assigned to the Root namespace and all subnamespaces below it.
See the Microsoft Knowledge Base article "HOW TO: Set WMI Namespace Security in Windows Server 2003" (http://support.microsoft.com/kb/325353) in the Microsoft Knowledgebase for more information.
Note: There is no standard facility for deploying WMI security settings remotely to multiple machines at once using Group Policy. However, "Set WMI namespace security via GPO" (http://blogs.msdn.com/spatdsg/archive/2007/11/21/set-wmi-namespace-security-via-gpo-script.aspx) on MSDN Blogs offers instructions on how to create a startup script that you can place inside a Group Policy Object (GPO), which will set the namespace security once the GPO is applied to the desired hosts. You can then deploy this GPO domain-wide or to one or more Organizational Units (OUs).
5. Firewall configuration. If you have a firewall enabled, you must configure it to allow access for WMI. If you are using the Windows Firewall included with Windows XP, Windows Server 2003/2003 R2, Windows Vista, Windows 7 and WIndows Server 2008/2008 R2, the exceptions list explicitly includes WMI. You must set this exception for both the originating and the target machines. See "Connecting to WMI Remotely Starting with Vista" (http://msdn.microsoft.com/en-us/library/aa822854(VS.85).aspx) on MSDN for more details.
6. User access control (UAC) configuration. If you're running Windows Vista, Windows 7 or Windows Server 20082008 R2, UAC affects how permissions are assigned. Review "User Account Control and WMI" (http://msdn.microsoft.com/en-us/library/aa826699(v=vs.85).aspx) on MSDN for additional information on the ramifications of UAC and WMI remote polling.
Test access to WMI providers
Once you've configured WMI and set up the Splunk user for access to your domain, you should test the configuration and access to the remote machine.
Important: This procedure includes steps to temporarily change Splunk's data store directory (the location
SPLUNK_DB points to). You must do this before testing access to WMI. Failure to do so can result in missing WMI events. This is because the splunk-wmi.exe process updates the WMI checkpoint file every time it is run.
To test access to WMI providers:
1. Log into the machine Splunk runs on as the Splunk user.
Note: If attempting to log into a domain controller, you may have to change your domain controller security policy to assign the "Allow log on locally" policy for the designated user.
2. Open a command prompt window (click Start -> Run and type
3. Go to the
bin subdirectory under your Splunk installation (for example,
cd c:\Program Files\Splunk\bin).
4. Determine where Splunk is storing its data by running the following command:
> splunk show datastore-dir
Note: You'll need to authenticate into your Splunk instance in order to do this. Once you have, be sure to note where Splunk is storing its data. You'll need to remember it for later.
5. Run the following command to change where Splunk stores its data temporarily:
> splunk set datastore-dir %TEMP%
Note: This example sets the data store directory to the current directory specified in the TEMP environment variable. If you want to set it to a different directory, you can do so, but the directory must already exist.
6. Restart Splunk by running the following command:
> splunk restart
Note: It may take a while for Splunk to restart. This is because it's creating a new data store at the area you specified in Step 5.
7. Once Splunk has restarted, test access to WMI providers by running the following command, replacing
<server> with the name of the remote server:
> splunk cmd splunk-wmi -wql "select * from win32_service" -namespace \\<server>\root\cimv2
8. If you see data streaming back and no error messages, that means Splunk was able to connect to the WMI provider and query successfully.
9. If there is an error, a message with a reason on what caused the error will appear. Look for the
error="<msg>" string in the output for clues on how to correct the problem.
After testing WMI access, be sure to point Splunk back to the correct Splunk database directory by running the following command, and then restarting Splunk:
> splunk set datastore-dir <directory shown from Step 4>
Configure WMI-based inputs
Note: Beginning with version 4.2, the procedure for adding WMI-based inputs has changed. There's no longer a WMI collections input available under "Data inputs" in Manager. You now access the WMI inputs through either the Remote event log collections or Remote performance monitoring data input types.
All remote data collection in Splunk on Windows is done through either WMI providers or a forwarder. Review "Considerations for deciding how to monitor remote Windows data" in this manual for additional information about remote data collection.
You can configure WMI-based inputs either in Splunk Web or by editing configuration files. More options are available when using configuration files.
Configure WMI-based inputs with Splunk Web
To add WMI-based inputs, consult one of the appropriate topics in this manual:
Configure WMI-based inputs with configuration files
Remote data collection configurations are controlled by wmi.conf. Review this file to see the default values for WMI-based inputs. If you want to make changes to the default values, edit a copy of
%SPLUNK_HOME%\etc\system\local\. Only set values for the attributes you want to change for a given type of data input. Refer to About configuration files in the Admin Manual for more information about how Splunk uses configuration files.
wmi.conf contains several stanzas:
[settings]stanza, which specifies global WMI parameters.
- One or more input-specific stanzas, which define how to connect to WMI providers to get data from the remote machine.
[settings] stanza specifies global WMI parameters. The entire stanza and every parameter within it are optional. If the stanza is missing, Splunk assumes system defaults.
When Splunk is not able to connect to a defined WMI provider, an error is generated in splunkd.log. For example:
05-12-2011 02:39:40.632 -0700 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - Unable to connect to WMI namespace "\\w2k3m1\root\cimv2" (attempt to connect took 42.06 seconds) (error="The RPC server is unavailable." HRESULT=800706BA)
The following attributes control how Splunk reconnects to a given WMI provider when an error occurs:
initial_backoff: Tells Splunk how long, in seconds, to wait the first time after an error occurs before trying to reconnect to the WMI provider. The default is 5. If connection errors continue to occur, Splunk doubles the wait time until it reaches the value specified in
max_backoff: Tells Splunk the maximum amount of time, in seconds, that it should wait between connection attempts, before invoking
max_retries_at_max_backoff. The default is 20.
max_retries_at_max_backoff: If the wait time between connection attempts reaches
max_backoff, tells Splunk to try to reconnect to the provider this many times, every
max_backoffseconds. The default is 2. If Splunk continues to encounter errors after it has made these attempts, it will give up, and won't attempt to connect to the problem provider again until it is restarted. However, it will continue to log errors such as the example shown above.
checkpoint_sync_interval: Tells Splunk how long, in seconds, to wait for state data (event log checkpoint) to be written to disk. The default is 2.
Input-specific stanzas tell Splunk how to connect to WMI providers on remote machines to get data. They are defined by one of two attributes that specify the type of data Splunk should gather. The stanza name can be anything, but usually begins with
WMI:, for example:
When you configure WMI-based inputs in Splunk Web, Splunk uses this naming convention for input-specific stanza headers.
You can specify one of two types of data inputs in an input-specific stanza:
- Event log. The
event_log_fileattribute tells Splunk to expect event log data from the sources defined in the stanza.
- Windows Query Language (WQL). The
wqlattribute tells Splunk to expect data from a WMI provider. When using this attribute, you must also specify a valid WQL statement. This attribute must be used when collecting performance data.
Caution: Do not define both of these attributes in one stanza. Use only one or the other. Otherwise, the input defined by the stanza will not run.
The common parameters for both types of inputs are:
server: A comma-separated list of servers from which to get data. If this parameter is missing, Splunk assumes that you want to connect to the local machine.
interval(required): Tells Splunk how often, in seconds, to poll for new data. If this attribute is not present and defined, the input that the stanza defines will not run. There is no default.
disabled: Tells Splunk whether this input is enabled or disabled. Set this parameter to 1 to disable the input, and 0 to enable it. The default is 0 (enabled).
The event log-specific parameters are:
event_log_file: A comma-separated list of event log channels to poll.
current_only: Tells Splunk whether or not to collect events that occur only when it is running. If events are generated when Splunk is stopped, Splunk will not attempt to index those events when it is started again. Set to 1 to have Splunk collect events that occur only when it is running, and 0 to have Splunk collect all events. The default is 0 (gather all available events.)
The WQL-specific parameters are:
wql: A valid WQL statement.
namespace(optional): Specifies the path to the WMI provider. The local machine must be able to connect to the remote machine using delegated authentication. If you don't specify a path to a remote machine, Splunk will connect to the default local namespace (
\Root\CIMV2). This default namespace is where most of the providers you are likely to query reside. Microsoft provides a list of namespaces for Windows XP and later versions of Windows (http://msdn.microsoft.com/en-us/library/aa394084(VS.85).aspx).
current_only: Tells Splunk whether or not an event notification query is expected. See "WQL query types: event notification versus standard" below for additional information. Set this attribute to 1 to tell Splunk to expect an event notification query, and 0 to expect a standard query. The default is 0 (standard query.)
WQL query types: event notification versus standard
current_only attribute in WQL stanzas determines the type of query the stanza expects to use to collect WMI-based data. When the attribute is set to 1, event notification data is expected. Event notification data is data that alerts you of an incoming event. To get event notification data, you must use an event notification query.
For example, if you want to find out when processes are spawned on a remote machine, you must use an event notification query to get that information. Standard queries have no facilities for notifying you when an event has occurred, and can only return results on information that already exists.
Conversely, if you want to know what already-running processes on your system begin with the word "splunk", you must use a standard query. Event notification queries cannot tell you about static, pre-existing information.
Event notification queries require that the WQL statement defined for the stanza be structurally and syntactically correct. Improperly formatted WQL will cause the input defined by the stanza to not run. Check the wmi.conf configuration file reference for specific details and examples.
Examples of wmi.conf
The following is an example of a
[settings] initial_backoff = 5 max_backoff = 20 max_retries_at_max_backoff = 2 checkpoint_sync_interval = 2 [WMI:AppAndSys] server = foo, bar interval = 10 event_log_file = Application, System, Directory Service disabled = 0 [WMI:LocalSplunkWmiProcess] interval = 5 wql = select * from Win32_PerfFormattedData_PerfProc_Process where Name = "splunk-wmi" disabled = 0 # Listen from three event log channels, capturing log events that occur only # while Splunk is running. Gather data from three servers. [WMI:TailApplicationLogs] interval = 10 event_log_file = Application, Security, System server = srv1, srv2, srv3 disabled = 0 current_only = 1 # Listen for process-creation events on a remote machine [WMI:ProcessCreation] interval = 1 server = remote-machine wql = select * from __InstanceCreationEvent within 1 where TargetInstance isa 'Win32_Process' disabled = 0 current_only = 1 # Receive events whenever someone plugs/unplugs a USB device to/from the computer [WMI:USBChanges] interval = 1 wql = select * from __InstanceOperationEvent within 1 where TargetInstance ISA 'Win32_PnPEntity' and TargetInstance.Description='USB Mass Storage Device' disabled = 0 current_only = 1
Fields for WMI data
- For event log data, Splunk sets the source type to
WinEventLog:<name of log file>. For example,
- For WQL data, Splunk sets the source type to the name of the stanza that defines the input. For example, for a stanza named
[WMI:LocalSplunkdProcess], Splunk sets the source type to
Splunk automatically defines the originating host from the data received.
Troubleshooting WMI logging
If you encounter problems receiving events through WMI providers or are not getting the results you expect, you can enable debugging in Splunk's logging engine in order to track down the problem.
To enable debugging for WMI-based inputs, you must set two parameters:
%SPLUNK_HOME\etc. Add the following parameter:
log-cmdline.cfg, also in
%SPLUNK_HOME%\etc. Add the following parameter:
Note: You can place this attribute/value pair anywhere in the file, as long as it is on its own line.
log-cmdline.cfg does not use stanzas.
3. Restart Splunk:
C:\Program Files\Splunk\bin> splunk restart
4. Once Splunk has restarted, let it run for a few minutes until you see debug log events coming into Splunk.
Note: You can search Splunk's logfiles within Splunk by supplying
index="_internal" as part of your search string. Review "What Splunk logs about itself" in the Admin Manual for additional information.
5. Once Splunk has collected enough debug log data, send a diag to Splunk Support:
C:\Program Files\Splunk\bin> splunk diag
Important: Once you finish troubleshooting, revert back to the default settings:
log.cfg, change the
category.ExecProcessor attribute to its default setting:
Note: You can also remove this entry from the file.
log-cmdline.cfg, change the
category.WMI attribute to its default setting:
Note: Any changes made to
log.cfg are overwritten when you upgrade Splunk. Create a
%SPLUNK_HOME%\etc to avoid this problem.
For more information on troubleshooting WMI, see "Troubleshooting WMI Issues" in the Community Wiki.
Monitor Windows event log data
Monitor Windows Registry data