Troubleshoot the Splunk App for Windows Infrastructure

This topic discusses how you can troubleshoot your Splunk App for Windows Infrastructure deployment if you aren't seeing the data that you expect.

Is the central Splunk App for Windows Infrastructure instance correctly configured?

The first thing to check when Splunk App for Windows Infrastructure data is incomplete or incorrect is to confirm that the central Splunk instance is properly configured and is receiving data.

1. Confirm that every indexer in the central Splunk instance is configured to receive data.

Note: Review "Enable a receiver" in the core Splunk documentation for instructions.

2. If present, confirm that every search head in the central Splunk instance is configured properly. Search heads must search all available indexers to get all indexed AD data.

Note:

• Review "Install a dedicated search head" and "Configure distributed search" in the core Splunk Enterprise documentation for specific instructions on configuring search heads and search peers (indexers).

• Review "How to deploy the Splunk App for Windows Infrastructure" for specific instructions on what specifically to install on the central Splunk instance's search heads.

3. Confirm that the Splunk App for Windows Infrastructure is installed properly.

• The Splunk App for Windows Infrastructure must reside on all indexers and search heads in the central Splunk instance.
• SA-ldapsearch must reside on all search heads in the central Splunk instance.
• Indexes that the Splunk App for Windows Infrastructure requires must be present on all indexers:
  • msad: for AD health metrics
  • winevents: for Directory Service, Replication Service, DNS server event logs
  • perfmon: for performance metrics
  • Review "About managing indexes" in the core Splunk Enterprise documentation for instructions on how to confirm that the proper indexes exist.

• eventtypes.conf (in %SPLUNK_HOME%\etc\apps\splunk_app_windows_infrastructure\local) must be configured with the proper indexes for the defined event types.
  • Check for typos in the configuration file.
  • Review "Install the app onto the central Splunk instance" for specific configuration instructions.

• ldap.conf (in %SPLUNK_HOME%\etc\apps\SA-ldapsearch\local) must be configured with the proper credentials to bind to Active Directory:
  • Check for typos in the configuration file.
  • Confirm that the credentials are valid, and that the account is not locked out, does not have an expired password, and is allowed full access to the AD schema which you are trying to monitor.
  • Review the table below for specific credential troubleshooting instructions.

4. Confirm that the app's lookup tables have been properly created.

• From the "Tools and Settings" menu, select "Build lookups".

5. Confirm that PowerShell is installed and enabled on all domain controllers and DNS servers, and that GPOs are in place to enable local script execution.

• Review "Enable auditing and local PowerShell script execution on Active Directory servers" for specific instructions.

6. Confirm that the attribute names in the base Distinguished Name have the proper capitalization.

Troubleshoot issues with ldapsearch

When the Splunk App for Windows Infrastructure cannot complete a search using the SA-ldapsearch supporting add-on, it notifies you by displaying an error message in Manager's status bar (at the top of your browser window), as follows:

External search command 'ldapsearch' returned error code 1. ERROR: com.unboundid.ldap.sdk.LDAPException: 80090308: LdapErr: DSID-0C090334, comment: AcceptSecurityContext error, data 525, vece

The Splunk App for Windows Infrastructure also writes a message to $SPLUNK_HOME/var/log/splunk/SA-ldapsearch.log, similar to the following:

2012-08-10 14:58:34.108 -0700 pid=877 com.splunk.program.LDAPSearch:main#-1 ERROR Exception com.unboundid.ldap.sdk.LDAPException thrown: 80090308: LdapErr: DSID-0C090334, comment: AcceptSecurityContext error, data 525, vece^@

If you see an error message similar to this when performing a search, use the following table to decode the data value and figure out how to resolve the error.

Data value	What it means	What you should do
255	Either the domain was not found or there was a syntax error in the search command.	Confirm that the domain that you want to monitor exists and is configured properly, or that your search string is properly formatted and syntactically correct.
525	The username provided in ldap.conf is not valid.	Edit ldap.conf and provide the correct user, then restart your central Splunk instance.
52E	The password provided in ldap.conf is not valid.	Edit ldap.conf and provide the correct password, then restart your central Splunk instance.
530	The user account provided is not allowed to log into Active Directory at this time.	Remove the user's log on time restrictions from within Active Directory, then try again.
531	The user account provided is not allowed to log into Active Directory from the current server.	Modify the local security policy of the server from which the specified user is trying to log in to Active Directory, then try again.
532	The user account provided has an expired password.	Change the user's password or set the "Password never expires" bit from within Active Directory, then try again.
533	The user account provided is disabled.	Re-enable the user account from within Active Directory, then try again.
701	The user account provided has expired.	Re-enable the user account from within Active Directory, then try again.
773	The user account provided has the "User must reset password at next logon" bit set.	Un-set the "User must reset password at next logon" bit for the user account from within Active Directory, then try again.
775	The user account provided is locked because an incorrect password has been entered too many times.	Re-enable the user account from within Active Directory and change the password to a known good one, then try again.

Windows event log or performance events from universal forwarders go to the 'main' index

When you install a universal forwarder on a Windows server, you must not select any options in the Enable Inputs screen of the installer. Doing so enables the scripted inputs that come with the forwarder by default. Those inputs send data to the "default" index as specified in their configuration files, which, on a standard Splunk installation, is main.

After you install the universal forwarder onto your exchange server, you must then install the appropriate technology add-ons included in the Splunk App for Windows Infrastructure's installation package. You must place the TAs in %SPLUNK_HOME%\etc\apps within the universal forwarder. Review "Deploy configurations for all server roles" for specific instructions.

Dashboards fail to load after upgrading the app or Splunk Enterprise

If you experience an issue where some dashboard panels or menus fail to load after you upgrade either Splunk Enterprise or the Splunk App for Windows Infrastructure, clear your web browser's cache, log out of Splunk Enterprise, then log back in. This should fix the problem.