Advanced help troubleshooting Splunk software for Windows

Review this topic if you're having trouble getting data into your Splunk for Windows instance, or if Splunk is having problems starting or running.

This topic provides solutions to common issues encountered when working with the Windows version of Splunk. It's divided into several subtopics:

• Generic issues
• Issues with WMI
• Issues with forwarders

General issues

This section contains solutions to common issues encountered when running Splunk on Windows.

Splunk fails to start

There are several factors that might prevent Splunk from starting properly. Whether it didn't start automatically, or you are having problems manually starting it, here are some solutions to try:

• Make sure that your system meets the Splunk system requirements. These requirements differ depending on the type of Splunk you're trying to run (full instance versus forwarder).

• Make sure that the Splunk services are enabled. Go into Control Panel and check that the splunkd and splunkweb services have their Startup type set to "Automatic."

• Check file and security permissions. When you install Splunk as a user other than Local System, Splunk does not have full permissions to run on the system by default. Try these solutions to get Splunk back up and running:
  • Make sure the Splunk user is in the local Administrators group on the machine.
  • Make sure that the Splunk user has Full Control permissions for the entire %SPLUNK_HOME% directory, and is also the owner of all files and subdirectories in %SPLUNK_HOME%. You must explicitly define this in the Security properties of the %SPLUNK_HOME% directory.
  • Be sure to read the "Considerations for deciding how to monitor remote Windows data" for additional information about permissions required to run Splunk as a domain user.

• Make sure Splunk isn't crashing. Check your %SPLUNK_HOME%\var\log\splunk directory. If Splunk crashes on startup, you'll see one or more dump files located in this directory. If Splunk is crashing, try the solutions listed above first. If those don't work, then uninstall and reinstall the program. If you still encounter problems, contact Splunk Support or visit the Answers page for additional guidance.

Note: Splunk Support requires an enterprise license in most cases.

No data is received

Splunk for Windows operates similarly to Splunk for other operating systems. If you're not getting data and it's not because of a permissions or network connectivity issue, then there is likely something happening within Splunk, such as an incorrectly configured input.

If you're having trouble collecting Windows event logs, review "Troubleshooting Windows event logs."

Try these solutions to figure out where your data is going:

• Make sure the clocks on all machines in your network are synchronized. If your Active Directory is set up correctly, this should already be done for you. Make sure the W32Time services on all your machines are running and properly syncing time with the appropriate domain controller.
• Make sure your inputs are correctly configured. In particular, the performance monitoring, Registry monitoring and Active Directory monitoring inputs must be properly configured if you want them to return data. When collecting performance logs remotely over WMI, for example, use Splunk Web instead of configuration files to create those inputs, as typos in configuration files will prevent Splunk from collecting the data you want.
• Make sure the type of input you are attempting to configure actually returns data. Some performance counters and Registry keys do not change throughout the course of a Windows session, for example. If there is no change, there is no data to collect.
• When using the Search app, make sure the time range for your search is correct. If you're searching for events that are outside the time range shown in the Search app, they won't appear there. Adjust the Search app's time range if needed.
• Make sure that the indexing or parsing pipelines on your indexer or forwarder are not blocked. For example, if Splunk can't send events from a forwarder to an indexer, due to a network issue, it may appear as though Splunk is not indexing the data, when data has not actually arrived. Check %SPLUNK_HOME%\var\log\splunk\metrics.log for information about the status of Splunk's processing queue. For more information on metrics.log, consult "Work with metrics.log" in this manual.

WMI issues

This section contains information about problems encountered when using WMI providers to gather data from remote machines.

No WMI-based events come into Splunk

When Splunk is unable to index WMI-based events, it is likely because of a permissions or security issue. Be sure to review the permissions checklist located in "Monitor WMI-based data" in the Getting Data In Manual. A summary of that checklist follows:

• Splunk must run as a user that is a member of the local Administrators group on the server doing the indexing.
• The Splunk user must be a member of the domain groups required to access the appropriate WMI resources.
• The Splunk user must be configured with specific local and domain security policy rights.
• WMI security must be correctly configured.
• Windows Firewall must be correctly configured.
• User Access Control must be considered.

You can also see additional information about Splunk's WMI operations by turning on debug logging. To turn on debug logging, follow the instructions in "Troubleshooting WMI Logging" in the Getting Data In Manual.

WMI-based events come in, but sometimes Splunk crashes

WMI can sometimes causes the Splunk WMI process (splunk-wmi.exe) to crash. If that happens, Splunk will start another WMI process immediately, but you might see crash files in your %SPLUNK_HOME%\var\log\splunk directory.

If Splunk is crashing, try the following solutions:

• Reduce the amount of WMI inputs on each Splunk instance. For best results, limit the number of WMI connections per instance to 120 or fewer on 32-bit Windows systems, or 240 or fewer for 64-bit systems. Note that each server monitored can use more than one WMI connection, depending on the amount of inputs configured for each server.
• Use a universal forwarder to get data. Splunk recommends that you use a universal forwarder to send data from remote machines to an indexer. Universal forwarders are more scalable and reliable than WMI in nearly all cases, and require far less security management than WMI does.

Splunk's WMI process runs slowly

Splunk makes what are known as semisynchronous calls to WMI providers. This means that when Splunk makes a call to WMI, it continues running while WMI deals with the request.

Semisynchronous mode offers the best balance of resource usage and security. It differs from the faster asynchronous mode, but is more secure due to the way that the system handles retrieval of the WMI objects. Both of these modes are faster than synchronous mode, which forces programs making that kind of WMI request to wait until WMI returns the data.

When WMI is dealing with a large number of requests, you might notice a slower response because memory usage on the system increases until the retrieved WMI objects are no longer needed by Splunk (after indexing).

More help

If you are still having issues, read "Troubleshooting common issues with Splunk and WMI".

Forwarder Issues

This section provides help for users who use Splunk's forwarding and receiving capabilities, including the new universal forwarder included with Version 4.2 and later.

Forwarder doesn't send any data

If you're using a forwarder to send data to a receiver and the receiver isn't getting any data, there are a number of things you can try to fix the problem:

• Make sure that there is network connectivity between the forwarder and the receiver.
  • On the machine running the forwarder, open a command prompt and telnet to the IP address and port of the receiver. For example, if your Splunk receiver is configured at IP address 192.168.1.10 port 9997, you would type:
    • > telnet 192.168.1.10 9997
  • Check the Windows Firewall on both the forwarder and the receiver. Windows Firewall must be configured to allow access in both directions for WMI. Either open ports to allow traffic, or disable Windows Firewall.

Note: On versions of Windows later than Windows XP or Windows Server 2003, the telnet client might not be installed. While the telnet client is not required for forwarding, you will not be able to use it to determine basic IP connectivity if it isn't installed. Follow the instructions located at "Install Telnet Client" (http://technet.microsoft.com/en-us/library/cc771275%28WS.10%29.aspx) on MS TechNet to install the telnet client.

• Make sure the configuration files on your forwarder are properly formatted.
  • Review your configuration files carefully, and check for spelling and syntax errors.
  • Stanza names must always be bracketed with square brackets ([ ]). Don't use curly braces or parentheses.
  • The syntax for remote performance monitoring differs significantly from local performance monitoring. Be sure to review "Monitor Windows performance" in the Getting Data In Manual for specific information.

• If the universal forwarder is running as a user other than Local System, confirm that security and access control are correctly configured.
  • Ensure that the Splunk user is a local Administrator on the machine.
  • Ensure that the Splunk user is valid (for example, it is neither locked out of the domain nor expired). Check that the user's password is also valid (not expired).
  • Ensure that the Splunk user has access to the desired resource(s).
  • Remember that special permissions are required to access some resources, such as the Security event log, Additionally, changing permissions for these resources sometimes requires special knowledge (such as the Security Description Definition Language).
  • Make sure that Active Directory is functioning correctly, and fix it if it is not.

Once you have confirmed any or all of these, restart the universal forwarder to ensure it gets a new authentication token from a domain controller.

Note: When assigning access, it's best practice to use the least permissive security paradigm. This entails denying all access to a resource initially, and only then granting access for specific users as necessary.

For more information

See the Admin Manual for information on getting started for Windows admins.

Have additional questions or need more help? Be sure to visit Splunk Answers and see what questions and answers the Splunk community has around troubleshooting Splunk on Windows.