Considerations for deciding how to monitor remote Windows data
This topic discusses the considerations you must take when using Splunk Enterprise to gather remote Windows data.
As with any software, getting the best out of Splunk Enterprise requires careful planning and execution. This is particularly important for remote operations - the goal is to get the most data using the least resources. This topic helps you decide where and how to deploy Splunk components for maximum efficiency when indexing data remotely from a large number of Windows hosts.
Remote Windows data overview
Splunk Enterprise collects remote Windows data for indexing in one of two ways:
- from Splunk forwarders
- via Windows Management Instrumentation (WMI)
Use a forwarder
Splunk can also collect remote Windows data remotely with a forwarder. There are several types of forwarders: light, heavy and universal. You can read more about the universal forwarder (and forwarding data in general) in "Introducing the universal forwarder" in the Forwarding Data Manual.
Splunk recommends using a universal forwarder to gather remote Windows data whenever possible. These are the advantages of using a universal forwarder:
- The universal forwarder is built to use minimal network and disk resources on the machines on which it's installed.
- If you install the universal forwarder as the Local System user, then it has administrative access to the server and requires no authentication to get data from there, as WMI does.
- It scales well in large environments and is easy to deploy. You can deploy it manually, with either a Microsoft deployment tool like System Center Configuration Manager (SCCM) or Systems Management Server (SMS), or a third party distribution solution such as BigFix/Tivoli.
Once you install a universal forwarder, it gathers information locally and sends it to a central Splunk Enterprise indexer. You tell the forwarder what data to gather either during the installation process or later, by distributing configuration updates manually or via the deployment server. You can also install add-ons into the universal forwarder.
There are some drawbacks to using the universal forwarder, depending on your network configuration and layout. More discussion of these trade-offs is available in the "Forwarders versus remote collection through WMI" section of this topic.
The Windows Management Instrumentation (WMI) framework allows Splunk Enterprise to attach to and collect virtually any kind of data from remote Windows machines. In this configuration, Splunk runs as a user that you specify at installation (or later on, in the Services control panel).
- gives Splunk as much access to the network as the specified account has for remote access
- lets Splunk collect data from remote Windows machines across the enterprise and place that data into a central repository
- is ideal for small to medium-sized networks with at least one Splunk indexer in each network segment
There are some caveats to this method of collection, however. See the "Forwarders versus WMI" section of this topic for specific information.
Note: While Active Directory (AD) monitoring does not use WMI, it has the same authentication considerations as data inputs that do use it. For more information on how Splunk monitors AD, see the topic "Monitor Active Directory" in this manual.
Getting data remotely over WMI? Be sure to read this
When collecting remote Windows data over WMI, you must consider the following important common issues:
Authentication for remote Windows data
Windows requires authentication for just about every kind of remote activity it performs. This is a particularly important concept to grasp when using Splunk in a Windows environment. Failure to understand how Splunk interacts with Windows over the network can lead to suboptimal search results, or no results at all. This section explains the various security-related factors surrounding getting remote Windows data into Splunk.
When you install Splunk Enterprise, you can specify the Local System user, or another user. More specific information about the installation ramifications of this choice can be found in the Installation Manual, but there are considerations for getting data in as well.
The user you tell Splunk to run as has a direct impact on the amount and available types of data Splunk can retrieve from remote machines. In order to get the data you want, you must give this user the permissions needed to properly access remote data sources.
The easiest way to do so is to make the user that Splunk runs as a member of the Administrators (or Domain Admins) groups. However, security concerns can make this choice problematic and, depending on how your Active Directory is configured, you might not even have permission to do this.
In most cases, you must configure the Splunk user account to have a minimum level of permissions needed to access the data source. This is called "least-permissive" access, and it entails:
- adding the user to various domain security groups
- making changes to the access control lists of various AD objects, depending on the data sources you need to access
If your AD domain security policy enforces password changes regularly, you must also:
- make sure that either the Splunk account's password is set to never expire, or that you manually change the password before it expires, as defined by the policy.
- restart Splunk services that run as that account on all servers in your network, once the password is changed.
Note: On recent versions of Windows Server, you can use managed service accounts (MSAs) to address issues with password expiry. Read "Managed service accounts on Windows Server 2008 and Windows 7" in the Installation Manual for additional information.
It's also a good idea to assign the Splunk account the "Deny log on locally" user rights assignment in Local Security Policy to prevent the user from logging in interactively.
While this method is initially more time-consuming, it gives you more control, and is more secure than handing out domain administrator access.
Individual topics in this manual that deal with remote access to Windows machines will have additional information and recommendations on how to configure the user Splunk runs as for least-permissive access. Review the Security and remote access considerations section on those pages for specifics.
Network and I/O usage considerations
Network bandwidth usage should be monitored closely, especially in networks with slow or thin WAN links. For this reason alone, in many cases, universal forwarders are a better choice for large remote collection operations.
Disk bandwidth is a concern as well. Anti-virus scanner drivers and drivers that intermediate between Splunk and the operating system should always be configured to ignore the Splunk directory and processes, regardless of the type of Splunk installation.
Splunk forwarders versus WMI
The recommended way to get data in from a remote Windows host is with a universal forwarder. This is because a universal forwarder offers the most types of data sources, provides more detailed data (for example, in performance monitoring metrics), minimizes network overhead, and reduces operational risk and complexity, particularly where authentication is concerned. It's also more scalable than WMI in many cases.
However, there are circumstances where remote collection is preferred, or even required. For example, if corporate or security policy restricts the code that can be run on production machines, or if there are performance or interoperability concerns with production software (such as a system that requires virus-scanning to be active). In these situations, Splunk supports using the native WMI interface to collect event logs and performance data.
These are the main areas of tradeoff between WMI and forwarders:
With respect to performance, a forwarder is a better choice when:
- You are collecting local event logs or flat files. This is because a forwarder requires less CPU and performs basic pre-compression of the data in an effort to reduce network overhead.
- You want to collect data from a machine without having to worry about authentication. When a forwarder is installed as the Local System user, it has administrative access to the machine on which it is installed. This allows you to collect any available data from that machine.
- You are collecting data from busy hosts such as domain controllers, AD operations masters, or machines that consistently experience periods of high utilization. This includes servers that run resource-intensive applications such as Exchange, SQL Server/Oracle, VMWare/VSphere/VCenter, Hyper-V, or SharePoint. This is because WMI might have problems keeping up with the amount of data these services generate. WMI polling is best-effort by design, and Splunk also throttles WMI calls to prevent unintentional denial-of-service attacks.
- You are concerned about CPU and network utilization on the machines from which you're collecting data. Forwarders are designed to use as little of these resources as possible, while WMI is more CPU intensive for the same set of data. WMI also uses more network resources to transfer that data to the collecting machine.
- You are concerned about scalability. Universal forwarders scale very well. Heavy forwarders do not scale as well as universal forwarders do, but both types of forwarder scale considerably better than WMI.
WMI is a better choice when:
- You are concerned about memory usage on a system with high memory utilization. Because forwarders have more polling options available, and reside on the local machine while collecting data, they use more memory than WMI does.
A forwarder is a better choice for deployment when:
- You have control of the base build of the OS, as is the case when you create system images.
- You have many data sources to collect, particularly if the data collected requires transformation of any kind.
Note: You cannot use a universal forwarder to process data in any way before it reaches the indexer. If you need to make any changes to your data before it is indexed, you must use a heavy forwarder.
WMI is a better choice when:
- You don't have control of the base OS build, or you don't have domain administrator access, or local administrator privileges on the machines from which you want to get data.
- You want or need only a limited set of data from a large number of hosts (for example, CPU data for usage billing).
A common deployment scenario is to first test using remote polling, then add successful or useful data inputs to your forwarder's configuration later, or at mass deployment time.
Both mechanisms offer logging and alerting to let you know if a host is coming on or offline or is no longer connected. However, to prevent an unintentional denial of service attack, the WMI polling service in Splunk begins to poll less frequently over time if it cannot contact a host for a period of time, and will eventually stop polling unreachable hosts altogether. As a result, Splunk does not advise remote polling over WMI for machines that are frequently offline, such as laptops or dynamically provisioned virtual machines.
The following table offers a list of data sources and indicates which data collection type(s) are appropriate for each data source.
|Data source||Local forwarder||WMI|
* For remote event log collection, you must know the name of the event log you wish to collect. On local forwarders, you have the option to collect all logs, regardless of name.
** Splunk Enterprise supports remote log file collection using the "\\SERVERNAME\SHARE" syntax; however, you must use CIFS (Common Internet File System, or Server Message Block) as your application layer file access protocol, and Splunk must have at least read access to both the share and the underlying file system.
Search Windows data on a non-Windows instance of Splunk
You can index and search your Windows data on a non-Windows instance of Splunk, but you must first use a Windows instance of Splunk to gather the Windows data. You can easily do this by installing a Splunk forwarder onto the Windows computer and configuring it to forward Windows data to the non-Windows instance of Splunk.
There are two main ways to proceed:
- Set up forwarders locally on each Windows machine from which you want data. These forwarders can send the Windows data to the non-Windows receiving instance of Splunk.
- Set up a forwarder on a separate Windows machine. The forwarder can use WMI to collect data from all the Windows machines in the environment and then forward the combined data to a non-Windows receiving instance of Splunk.
You must configure the non-Windows Splunk explicitly to handle the Windows data. For details, see "Searching data received from a forwarder running on a different operating system" in the Forwarding Data Manual.
For information on setting up forwarders, see "Set up forwarding and receiving" also in the Forwarding Data Manual.
How to get Windows data into Splunk Enterprise
Monitor Active Directory
This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14