Troubleshoot add-ons

Stuck? Try these techniques to troubleshoot add-ons built by Splunk Inc.

Useful searches

Search the internal index for logs specific to the add-on. For the source, use a wild-carded version of the brief name of the technology that the add-on supports. The source should usually match the last element in the add-on folder name. See the examples below.

Add-on	Folder name	Search
Splunk Add-on for Amazon Web Services	Splunk_TA_aws	index = _internal source=*aws*
Splunk Add-on for Cisco IPS	Splunk_TA_cisco-ips	index = _internal source=*cisco-ips*
Splunk Add-on for Java Management Extensions	Splunk_TA_jmx	index = _internal source=*jmx*
Splunk Add-on for ServiceNow	Splunk_TA_snow	index = _internal source=*snow*

Once you have narrowed to the proper type of logs to find issues, here are some troubleshooting techniques that might be useful for a given scenario.

Keyword searching

Add useful keywords to your search, like "ERROR" or the name of the service or server that you are trying to connect to.

Find unusual logs

Finding rare events is easier when all the similar things are hidden. If you add

   | dedup punct

to the end of a search, it will temporarily reduce repetitive "noise" and make unusual lines stand out.

Find transactions

Some data sources log in a repetitive pattern across many entries. If a common field value is available, such as *pid*, you can use

   | transaction pid

at the end of a search to collapse these patterns into single events. That makes them easier to analyze with tools like stats or dedup. Transaction can use multiple fields as well:

   index=_internal source="*s3_main.log" | transaction pid tid | search "Previous run is not done yet"

Increase timeout settings

If you are configuring accounts or inputs in Splunk Web and you experience errors or trouble saving, go to $SPLUNK_HOME/etc/system/local/web.conf and change your timeout settings as shown below.

  [settings]
  splunkdConnectionTimeout = 300

Turn on debug logging

Some add-ons allow you to configure the log level in the add-on setup page. Toggle this to DEBUG to review more verbose logs to help you find the root cause. If the add-on that you are using does not support this through the setup in Splunk Web, it may be possible to configure this in the configuration files instead. Check the documentation for each add-on for guidance.

Check if the add-on is intended to be visible or not

Some add-ons have a custom configuration UI that you can use in Splunk Web to set up a connection to your data and configure inputs. Other add-ons do not have a custom UI, and instead rely on the Splunk platform's native data collection and configuration capabilities. If the add-on you are using does not have any custom views, or if it does but you want to turn those views off on certain nodes of your distributed deployment to prevent confusion, make sure that visibility of the add-on is set to False.

If you are trying to launch or load views for an add-on in Splunk Web and you are experiencing results you do not expect, the add-on may not be intended to be visible. You can determine this by looking in the add-on's default/app.conf file. If you see visible = false in the ui stanza, this add-on is not intended to be treated as a visible app in Splunk Web on any nodes of your architecture.

To turn off visibility of an add-on:

1. Go to Apps > Manage Apps.
2. Find the row for the add-on you are working with and click Edit properties.
3. Under Visible, choose No, then click Save.

If the add-on includes prebuilt panels, you can access those through your existing dashboards. See Access prebuilt panels included with add-ons.

Check for conflicting configurations

Some add-ons are intended to replace older versions of add-ons with different package names. In these cases, you should remove the old, differently named package of the add-on when you install the newer version to avoid conflicts in your configuration files. Review the release notes for specific add-ons to find upgrade guides to walk you through this process.

For example, if a workflow action is not behaving as expected, try running btool.

$SPLUNK_HOME/bin/splunk btool --debug workflow_actions list [workflow_action_stanza] [--app=search]

The output will show which configurations came from which app. They should all originate from the add-on you are troubleshooting. Other conflicting configurations may be causing the unexpected behavior. For example, Splunk Web attempts to render the workflow action result as Splunk view instead of as an external site.

Communication with external systems

Many Splunk developed add-ons that have modular inputs use a third-party API to communicate with an external system. You need to ensure that communication is allowed by your networking infrastructure. If you use a firewall in your environment, you need to open the correct firewall port. For an SSL enabled API, that port is typically 443. For example, the Splunk Add-on for F5 BIG-IP requires port 443 to be open on your firewall to enable F5 BIG-IP to communicate with the iControl API over SSL. If the correct port is not open, you may get "Destination unreachable" errors.

Prevent event data loss

When a Splunk add-on with modular inputs is collecting data on a heavy forwarder and the forwarder is stopped, event data can be lost.
Workaround: If you need to stop a forwarder for any reason, first disable all add-ons that might be collecting data on it, and wait for at least 10 seconds before stopping the forwarder.

On Windows, disable an add-on with modular inputs before upgrading it

On Windows, upgrading an add-on with modular inputs may fail due to some processes that keep running in the background and prevent some files from being overwritten during the upgrade. As a workaround, disable an add-on with modular inputs before upgrading it and enable it after upgrade is complete.