Considerations for deciding how to monitor remote Windows data

Considerations for deciding how to monitor remote Windows data

Splunk is capable of powerful indexing, searching and reporting on your remote Windows data. However, as with any software, getting the best out of Splunk requires careful planning and execution. This is particularly important for remote operations - the goal is to get the most data using the fewest resources. This topic helps you decide where and how to deploy Splunk components for maximum efficiency when indexing data remotely from a large number of Windows hosts.

Remote Windows data overview

Splunk collects remote Windows data for indexing in one of two ways:

• via Windows Management Instrumentation (WMI)
• from Splunk forwarders

Use WMI

The Windows Management Instrumentation (WMI) framework allows Splunk to attach to and collect virtually any kind of data from remote Windows machines. In this configuration, Splunk runs as a user that you specify at installation (or later on, in the Services control panel).

This configuration:

• gives Splunk as much access to the network as the specified account has for remote access
• lets Splunk collect data from remote Windows machines across the enterprise and place that data into a central repository
• is ideal for small to medium-sized networks with at least one Splunk indexer in each network segment

There are some caveats to this method of collection, however. They are explained in the "Forwarders versus WMI" section of this topic.

Note: While Active Directory (AD) monitoring does not use WMI, it has the same authentication considerations as data inputs that do use it. For more information on how Splunk monitors AD, see the topic "Monitor Active Directory" in this manual.

Use a universal forwarder

Splunk can also collect remote Windows data remotely with a forwarder. There are several types of forwarders: light, heavy and universal. In nearly all cases, you'll want to use a universal forwarder. You can read more about the universal forwarder in "Introducing the universal forwarder" in the Distributed Deployment Manual.

Splunk recommends using a universal forwarder to gather remote Windows data whenever possible. These are the advantages of using a universal forwarder:

• The universal forwarder is built to use very little network or disk resources on the machines on which it's installed.
• If the universal forwarder is installed as the Local System user, then it has access to everything on the machine on which it's installed, and requires no authentication to get data, as WMI does.
• It scales very well in large environments, and is very easy to deploy. You can deploy it manually, by using either a Microsoft deployment tool such as System Center Configuration Manager (SCCM) or Systems Management Server (SMS), or a third party distribution solution such as BigFix/Tivoli.

Once a universal forwarder is installed, it gathers information locally and sends it to a central Splunk indexer. You tell the forwarder what data to gather either during the installation process or later, by distributing configuration updates via the deployment server.

There are some drawbacks to using the universal forwarder, depending on your network configuration and layout. More discussion on these trade-offs is available in the "Forwarders versus remote collection through WMI" section of this topic.

Polling data remotely over WMI? Be sure to read this

When collecting remote Windows data over WMI, you must consider some important common issues associated with doing so.

Authentication for remote Windows data

Windows requires authentication for just about every kind of remote activity it performs. While this is well understood by system administrators, it's a particularly important concept to grasp when using Splunk in a Windows environment. Failure to understand how Splunk interacts with Windows over the network can lead to suboptimal search results, or no results at all. This section explains the various security-related factors surrounding getting remote Windows data into Splunk.

When you install Splunk, you can specify the Local System user, or another user. More specific information about the installation ramifications of this choice can be found in the Installation Manual, but there are considerations for getting data in as well.

The user you tell Splunk to run as has a direct impact on the amount and available types of data Splunk can retrieve from remote machines. In order to get the data you want, you must give this user the permissions needed to properly access remote data sources.

The easiest way to do so is to make the user that Splunk runs as a member of the Administrators (or Domain Admins) groups. However, security concerns can make this choice problematic and, depending on how your Active Directory is configured, you might not even have permission to do this.

In most cases, you must configure the Splunk user account to have a minimum level of permissions needed to access the data source. This is called "least-permissive" access, and it entails:

• adding the user to various domain security groups
• making changes to the access control lists of various AD objects, depending on the data sources you need to access

If your AD domain security policy is configured to enforce password changes regularly, you must also:

• make sure that either the Splunk account's password is set to never expire, or that you manually change the password before it expires, as defined by the policy
• restart Splunk services that run as that account on all servers in your network, once the password is changed

It's also a good idea to assign the Splunk account the "Deny log on locally" user rights assignment in Local Security Policy to prevent the user from logging in interactively.

While this method is initially more time-consuming, it gives you more control, and is more secure than handing out domain administrator access.

Individual topics in this manual that deal with remote access to Windows machines will have additional information and recommendations on how to configure the user Splunk runs as for least-permissive access. Review the Security and remote access considerations section on those pages for specifics.

Bandwidth considerations

Network bandwidth usage should be monitored closely, especially in networks with slow or thin WAN links. For this reason alone, in many cases, universal forwarders are a better choice for large remote collection operations.

Disk bandwidth is a concern as well. Anti-virus scanner drivers and drivers that intermediate between Splunk and the operating system should always be configured to ignore the Splunk directory and processes, regardless of the type of Splunk installation.

Splunk forwarders versus WMI

The recommended way to get data in from a remote Windows host is with a universal forwarder. This is because a universal forwarder offers the most types of data sources, minimizes network overhead, and reduces operational risk and complexity, particularly where authentication is concerned. It's also more scalable than WMI in many cases.

However, there are circumstances where remote collection is preferred, or even required. For example, if corporate or security policy restricts the code that can be run on production machines, or if there are performance or interoperability concerns with production software (such as a system that requires virus-scanning to be active). In these situations, Splunk supports using the native WMI interface to collect event logs and performance data.

These are the main areas of tradeoff between WMI and forwarders:

• Performance
• Deployment
• Management

Performance

With respect to performance, a forwarder is a better choice when:

• You are collecting local event logs or flat files. This is because a forwarder requires less CPU and performs basic pre-compression of the data in an effort to reduce network overhead.
• You want to collect data from a machine without having to worry about authentication. When a forwarder is installed as the Local System user, it has administrative access to the machine on which it is installed. This allows you to collect any available data from that machine.
• You are collecting data from busy hosts such as domain controllers, AD operations masters, or machines that consistently experience periods of high resource utilization. This includes servers that run resource-intensive applications such as Exchange, SQL Server/Oracle, VMWare/VSphere/VCenter, Hyper-V, or SharePoint. This is because WMI might have problems keeping up with the amount of data that is generated. WMI polling is best-effort by design, and Splunk also throttles WMI calls to prevent unintentional denial-of-service attacks.
• You are concerned about CPU and network utilization on the machines from which you're collecting data. Forwarders are designed to use as little of these resources as possible, while WMI is more CPU intensive for the same set of data. WMI also uses more network resources to transfer that data to the collecting machine.
• You are concerned about scalability. Universal forwarders scale very well. Heavy forwarders do not scale as well as universal forwarders do, but both types of forwarder scale considerably better than WMI.

WMI is a better choice when:

• You are concerned about memory usage on a system with high memory utilization. Because forwarders have more polling options available, and reside on the local machine while collecting data, they use more memory than WMI does.

Deployment

A forwarder is a better choice for deployment when:

• You have control of the base build of the OS, as is the case when you create system images.
• You have many data sources to collect, particularly if the data collected requires transformation of any kind.

Note: You cannot use a universal forwarder to process data in any way before it reaches the indexer. If you need to make any changes to your data before it is indexed, you must use a heavy forwarder.

WMI is a better choice when:

• You don't have control of the base OS build, or you don't have domain administrator access, or local administrator privileges on the machines from which you want to get data.
• You want or need only a limited set of data from a large number of hosts (for example, CPU data for usage billing).

A common deployment scenario is to first test using remote polling, then add successful or useful data inputs to your forwarder's configuration later, or at mass deployment time.

Management

Both mechanisms offer logging and alerting to let you know if a host is coming on or offline or is no longer connected. However, to prevent an unintentional denial of service attack, the WMI polling service in Splunk will start to poll less frequently over time if it is unable to contact a host for a period of time, and will eventually stop polling unreachable hosts altogether. As a result, Splunk does not advise remote polling over WMI for machines that are frequently offline, such as laptops or dynamically provisioned virtual machines.

The following table offers a list of data sources and indicates which data collection type(s) are appropriate for each data source.

Data Sources and collection methods

Data Source	Local Forwarder	WMI
Event logs	Yes	Yes*
Performance	Yes	Yes
Registry	Yes	No
Active Directory	Yes	No
Log files	Yes	Yes**
Crawl	Yes	No

* For remote event log collection, you must know the name of the Event Log you wish to collect. On local forwarders, you have the option to collect all logs, regardless of name.

** Splunk supports remote log file collection using the "\\SERVERNAME\SHARE" syntax, however you must use CIFS (Common Internet File System, or Server Message Block) as your application layer file access protocol and Splunk must have at least read access to both the share and the underlying file system.

Search Windows data on a non-Windows instance of Splunk

You can index and search your Windows data on a non-Windows instance of Splunk, but you must first use a Windows instance of Splunk to gather the Windows data. You can easily do this by installing a Splunk forwarder onto the Windows machine and configuring it to forward Windows data to the non-Windows instance of Splunk.

There are two main ways to proceed:

• Set up forwarders locally on each Windows machine from which you want data. These forwarders can send the Windows data to the non-Windows receiving instance of Splunk.
• Set up a forwarder on a separate Windows machine. The forwarder can use WMI to collect data from all the Windows machines in the environment and then forward the combined data to a non-Windows receiving instance of Splunk.

You must configure the non-Windows Splunk explicitly to handle the Windows data. For details, see "Searching data received from a forwarder running on a different operating system" in the Distributed Deployment Manual.

For information on setting up forwarders, see "Set up forwarding and receiving" also in the Distributed Deployment Manual.

• Was this topic useful?
• Post a comment