Troubleshoot the Splunk App for Microsoft Exchange

This topic discusses how you can troubleshoot your Splunk App for Microsoft Exchange deployment if you aren't seeing the data that you expect.

Is the central Splunk App for Microsoft Exchange instance correctly configured?

The first thing to check when Splunk App for Microsoft Exchange data is incomplete or incorrect is to confirm that the central Splunk instance is properly configured and is receiving data.

1. Confirm that every indexer in the central Splunk instance is configured to receive data.

Note: Review "Enable a receiver" in the core Splunk documentation for instructions.

2. If present, confirm that every search head in the central Splunk instance is configured properly. Search heads must search all available indexers to get all indexed AD data.

Note:

• Review "Install a dedicated search head" and "Configure distributed search" in the core Splunk Enterprise documentation for specific instructions on configuring search heads and search peers (indexers).

• Review "How to deploy the Splunk App for Microsoft Exchange" for specific instructions on what specifically to install on the central Splunk instance's search heads.

3. Confirm that the Splunk App for Microsoft Exchange is installed properly.

• The Splunk App for Microsoft Exchange must reside on all indexers and search heads in the central Splunk instance.
• SA-ldapsearch must reside on all search heads in the central Splunk instance.
• Indexes that the Splunk App for Microsoft Exchange requires must be present on all indexers:
  • msad: for AD health metrics
  • winevents: for Directory Service, Replication Service, DNS server event logs
  • perfmon: for performance metrics
  • Review "About managing indexes" in the core Splunk Enterprise documentation for instructions on how to confirm that the proper indexes exist.

• eventtypes.conf (in %SPLUNK_HOME%\etc\apps\splunk_app_microsoft_exchange\local) must be configured with the proper indexes for the defined event types.
  • Check for typos in the configuration file.
  • Review "Install the app onto the central Splunk instance" for specific configuration instructions.

• ldap.conf (in %SPLUNK_HOME%\etc\apps\SA-ldapsearch\local) must be configured with the proper credentials to bind to Active Directory:
  • Check for typos in the configuration file.
  • Confirm that the credentials are valid, and that the account is not locked out, does not have an expired password, and is allowed full access to the AD schema which you are trying to monitor.
  • Review the table below for specific credential troubleshooting instructions.

4. Confirm that the app's lookup tables have been properly created.

• From the "Tools and Settings" menu, select "Build lookups".

5. Confirm that PowerShell is installed and enabled on all domain controllers and DNS servers, and that GPOs are in place to enable local script execution.

• Review "Enable auditing and local PowerShell script execution on Active Directory servers" for specific instructions.

6. Confirm that the attribute names in the base Distinguished Name have the proper capitalization.

Troubleshoot issues with ldapsearch

When the Splunk App for Microsoft Exchange cannot complete a search using the SA-ldapsearch supporting add-on, it notifies you by displaying an error message in Manager's status bar (at the top of your browser window), as follows:

External search command 'ldapsearch' returned error code 1. ERROR: com.unboundid.ldap.sdk.LDAPException: 80090308: LdapErr: DSID-0C090334, comment: AcceptSecurityContext error, data 525, vece

The Splunk App for Microsoft Exchange also writes a message to $SPLUNK_HOME/var/log/splunk/SA-ldapsearch.log, similar to the following:

2012-08-10 14:58:34.108 -0700 pid=877 com.splunk.program.LDAPSearch:main#-1 ERROR Exception com.unboundid.ldap.sdk.LDAPException thrown: 80090308: LdapErr: DSID-0C090334, comment: AcceptSecurityContext error, data 525, vece^@

If you see an error message similar to this when performing a search, use the following table to decode the data value and figure out how to resolve the error.

Data value	What it means	What you should do
255	Either the domain was not found or there was a syntax error in the search command.	Confirm that the domain that you want to monitor exists and is configured properly, or that your search string is properly formatted and syntactically correct.
525	The username provided in ldap.conf is not valid.	Edit ldap.conf and provide the correct user, then restart your central Splunk instance.
52E	The password provided in ldap.conf is not valid.	Edit ldap.conf and provide the correct password, then restart your central Splunk instance.
530	The user account provided is not allowed to log into Active Directory at this time.	Remove the user's log on time restrictions from within Active Directory, then try again.
531	The user account provided is not allowed to log into Active Directory from the current server.	Modify the local security policy of the server from which the specified user is trying to log in to Active Directory, then try again.
532	The user account provided has an expired password.	Change the user's password or set the "Password never expires" bit from within Active Directory, then try again.
533	The user account provided is disabled.	Re-enable the user account from within Active Directory, then try again.
701	The user account provided has expired.	Re-enable the user account from within Active Directory, then try again.
773	The user account provided has the "User must reset password at next logon" bit set.	Un-set the "User must reset password at next logon" bit for the user account from within Active Directory, then try again.
775	The user account provided is locked because an incorrect password has been entered too many times.	Re-enable the user account from within Active Directory and change the password to a known good one, then try again.

Windows event log or performance events from universal forwarders go to the 'main' index

When you install a universal forwarder on your Exchange server, you must not select any options in the Enable Inputs screen of the installer. Doing so enables the scripted inputs that come with the forwarder by default. Those inputs send data to the "default" index as specified in their configuration files, which, on a standard Splunk installation, is main.

After you install the universal forwarder onto your exchange server, you must then install the appropriate technology add-ons included in the Splunk App for Microsoft Exchange's installation package. You must place the TAs in %SPLUNK_HOME%\etc\apps within the universal forwarder. Review "Deploy configurations for all server roles" for specific instructions.

Dashboards do not populate correctly or at all due to missing search results

If you encounter an issue where some dashboards do not populate properly even though data is coming into the app, make sure that the "user" role searches the four indexes that the Splunk App for Microsoft Exchange uses:

1. In the upper right hand of the Splunk Web page, click Settings.

2. In the window that pops up, under "Users and Authentication", click Access Controls. Splunk loads the "Access Controls" page.

3. Click Roles. Splunk loads the "Roles" page.

4. Click user. Splunk loads the "user" page.

5. Scroll down to Indexes searched by default.

6. Add the following indexes to the Selected Indexes pane by clicking on their entries in the Available Indexes pane:

• msexchange
• msad
• perfmon
• winevents

7. Click Save. Splunk saves the changes and returns you to the Roles page.

Dashboards do not populate due to errors with PowerShell modules

If you run Exchange 2010 Service Pack 2, you might experience an issue where dashboards do not populate due to errors with PowerShell. This is due to a bug that occurs when you apply Exchange 2010 SP2. This is an issue with Exchange and not with Splunk.

To fix the problem, follow the instructions at "Microsoft.Exchange.Management.PowerShell.E2010 is not installed on this machine" (http://blogs.technet.com/b/nawar/archive/2014/02/06/microsoft-exchange-management-powershell-e2010-39-is-not-installed-on-this-machine.aspx) on MS TechNet Blogs.

Dashboards fail to load after upgrading the app or Splunk Enterprise

If you experience an issue where some dashboard panels or menus fail to load after you upgrade either Splunk Enterprise or the Splunk App for Microsoft Exchange, clear your web browser's cache, log out of Splunk Enterprise, then log back in. This should fix the problem.