Troubleshoot the Splunk App for Windows Infrastructure

This topic discusses how you can troubleshoot your Splunk App for Windows Infrastructure deployment if you aren't seeing the data that you expect.

Is the Splunk App for Windows Infrastructure deployment correctly configured?

The first thing to check when Splunk App for Windows Infrastructure data is incomplete or incorrect is to confirm that the central Splunk instance is properly configured and is receiving data.

1. Confirm that every indexer in the deployment has been configured to receive data. See Install and configure a Splunk Enterprise indexer in this manual or Enable a receiver in the core Splunk documentation for instructions.
2. If present, confirm that every search head in the central Splunk instance has been configured properly. Search heads must search all available indexers to get all indexed AD data.
   See Distributed search deployment overview and Configure distributed search in the core Splunk Enterprise documentation for specific instructions on configuring search heads and search peers (indexers).
   
   See How to deploy the Splunk App for Windows Infrastructure for specific instructions on what to install on search heads.
   
3. Confirm that you have installed and configured all the components properly.
   1. Indexes that the Splunk App for Windows Infrastructure requires must be present on all indexers.
      1. msad: for AD health metrics
      2. wineventlog: for Directory Service, Replication Service, DNS server event logs
      3. perfmon: for performance metrics
   2. The Splunk App for Windows Infrastructure must reside on all search heads in the deployment.
   3. The Splunk Supporting Add-on for Active Directory (SA-ldapsearch) must be configured properly and reside on all search heads in the deployment. See Troubleshoot issues with SA-LDAPsearch.
   4. eventtypes.conf (in %SPLUNK_HOME%\etc\apps\splunk_app_windows_infrastructure\local) must be configured with the proper indexes for the defined event types.
4. Confirm that the lookup tables for the app have been properly created.
   From the Tools and Settings menu, select Build lookups.
   

Troubleshoot issues with ldapsearch

When the Splunk App for Windows Infrastructure cannot complete a search using the SA-ldapsearch supporting add-on, it notifies you by displaying an error message in Manager's status bar (at the top of your browser window), as follows:

External search command 'ldapsearch' returned error code 1. ERROR: com.unboundid.ldap.sdk.LDAPException: 80090308: LdapErr: DSID-0C090334, comment: AcceptSecurityContext error, data 525, vece

The Splunk App for Windows Infrastructure also writes a message to $SPLUNK_HOME/var/log/splunk/SA-ldapsearch.log, similar to the following:

2012-08-10 14:58:34.108 -0700 pid=877 com.splunk.program.LDAPSearch:main#-1 ERROR Exception com.unboundid.ldap.sdk.LDAPException thrown: 80090308: LdapErr: DSID-0C090334, comment: AcceptSecurityContext error, data 525, vece^@

If you see an error message similar to this when performing a search, use the following table to decode the data value and figure out how to resolve the error.

Data value	What it means	What you should do
255	Either the domain was not found or there was a syntax error in the search command.	Confirm that the domain that you want to monitor exists and is configured properly, or that your search string is properly formatted and syntactically correct.
525	The username provided in ldap.conf is not valid.	Edit ldap.conf and provide the correct user, then restart your central Splunk instance.
52E	The password provided in ldap.conf is not valid.	Edit ldap.conf and provide the correct password, then restart your central Splunk instance.
530	The user account provided is not allowed to log into Active Directory at this time.	Remove the user's log on time restrictions from within Active Directory, then try again.
531	The user account provided is not allowed to log into Active Directory from the current server.	Modify the local security policy of the server from which the specified user is trying to log in to Active Directory, then try again.
532	The user account provided has an expired password.	Change the user's password or set the "Password never expires" bit from within Active Directory, then try again.
533	The user account provided is disabled.	Re-enable the user account from within Active Directory, then try again.
701	The user account provided has expired.	Re-enable the user account from within Active Directory, then try again.
773	The user account provided has the "User must reset password at next logon" bit set.	Un-set the "User must reset password at next logon" bit for the user account from within Active Directory, then try again.
775	The user account provided is locked because an incorrect password has been entered too many times.	Re-enable the user account from within Active Directory and change the password to a known good one, then try again.

Windows event log or performance events from universal forwarders go to the 'main' index

When you install a universal forwarder on a Windows server, you must not select any options in the Enable Inputs screen of the installer. Doing so enables the scripted inputs that come with the forwarder by default. Those inputs send data to the "default" index as specified in their configuration files, which, on a standard Splunk installation, is main.

After you install the universal forwarder onto your Windows server, deploy the appropriate add-ons included in the Splunk App for Windows Infrastructure installation package. See "Deploy the Splunk Add-on for Windows" and "Deploy the Splunk Add-ons for Active Directory".

Dashboards fail to load after upgrading the app or Splunk Enterprise

If you experience an issue where some dashboard panels or menus fail to load after you upgrade either Splunk Enterprise or the Splunk App for Windows Infrastructure, clear your web browser's cache, log out of Splunk Enterprise, then log back in. This should fix the problem.

No data types found after upgrade

If you experience a problem where the first-time run process detects no data after an upgrade, make sure you have added the new winfra-admin role to the user that runs the app.

1. In the upper right hand of the Splunk Web page, click Settings.
2. In the window that pops up, under "Users and Authentication", click Access Controls. Splunk loads the "Access Controls" page.
3. Click Users. Splunk loads the "Users" page.
4. In the Username column, click the name of the user that runs the Splunk App for Windows Infrastructure. Splunk Enterprise loads the settings page for that user.
5. Scroll down to Assign to roles.
6. Add the winfra-admin role to the Selected roles pane by clicking on its entry in the Available roles pane.
7. Click Save. Splunk saves the changes and returns you to the Users page.

Error message: Configuration file settings may be duplicated in multiple apps

Your deployment receives the following error.

Configuration file settings may be duplicated in multiple apps: stanza="ActiveDirectory: Create Computer Lookup" file="savedsearches" apps="splunk_app_microsoft_exchange,splunk_app_windows_infrastructure"

This error happens when the Splunk App for Windows Infrastructure and the Splunk App for Microsoft Exchange are installed on the same search head. The Splunk App for Windows Infrastructure and the Splunk App for Microsoft Exchange contain identical knowledge objects that cause a conflict when installed on the same search head deployment.

Removing either the Splunk App for Windows Infrastructure or the Splunk App for Microsoft Exchange will resolve this issue.

Lookup errors on all searches for WinEventLog:System:IAS and source:::Security' If running Windows App For Infrastructure 1.5.0 with Splunk_TA_windows 4.8.4.

If you experience the following search errors on UI while running Windows App For Infrastructure 1.5.0 with the Splunk_TA_windows 4.8.4, then remove windows_apps lookup and its transforms definition from Windows App For Infrastructure.

Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'source::*:Security' and lookup table 'windows_app_lookup'.

Remove windows_apps.csv lookup from main app.

1. Remove windows_apps lookup from /etc/shcluster/apps/splunk_app_windows_infrastructure/lookups on the searchhead deployer (In case of standalone searchhead, remove it from /etc/apps/splunk_app_windows_infrastructure/lookups).
2. Remove following windows_apps lookup definition from /etc/shcluster/apps/splunk_app_windows_infrastructure/default/transforms.conf on the searchhead deployer (In case of standalone searchhead, remove it from /etc/apps/splunk_app_windows_infrastructure/default/transforms.conf).

   [windows_app_lookup]
   filename = windows_apps.csv
   
   [windows_apps]
   filename=windows_apps.csv
   max_matches=1
   

3. Push bundle from deployer to SHs (In case of standalone searchhead, restart Splunk on SH).

Precheck error in Splunk App for Windows Infrastructure guided setup for Splunk version 7.0.x

If you're using Splunk version 7.0.x and the Splunk App for Windows Infrastructure version 1.5.2, you may experience an error while running the guided setup precheck step. Bypass the the precheck step and continue. Bypassing the step does not impact app functionality.