Troubleshoot the Splunk Add-on for ServiceNow
Cannot launch add-on
This add-on does not have views and is not intended to be visible in Splunk Web. If you are trying to launch or load views for this add-on and you are experiencing results you do not expect, turn off visibility for the add-on.
For more details about add-on visibility and instructions for turning visibility off, see the Check if the add-on is intended to be visible or not section of the Splunk Add-ons Troubleshooting topic.
Cannot access configuration page
If you are trying to reach the setup page but cannot see a link to it on your instance, confirm that you are signed in with an account that is a member of the admin or sc_admin role.
Find relevant errors
Search for the following event types to find errors relevant to the Splunk Add-on for ServiceNow.
eventtype=snow_ta_collector_error for errors related to data collection from ServiceNow.
eventtype=snow_ticket_error for errors related to creating events or incidents in ServiceNow from the Splunk platform.
eventtype=ta_frwk_error for errors related to low-level functions of the add-on.
eventtype=snow_ta_log_error for errors related to the add-on as well as account and input configuration.
If you are not getting data from all of the inputs that you have enabled, check that the ServiceNow account that you are using to connect to your ServiceNow instance from the Splunk platform has, at minimum, read-only access to all of the database tables from which you are attempting to collect data. Then, disable and re-enable the inputs for which you are not receiving data.
To validate that you do not have a permissions issue:
- Edit the following URL to use your ServiceNow instance name:
service_now_tableto the ServiceNow table you are trying to query
2016-01-01to the actual date you want to query from.
- Paste the URL into a browser.
- When prompted, log in with the same username and password that you use for the integration account in the add-on.
If you receive the historical data you expect and a
sys_updated_on field for each event, you have the correct permissions.
SSL certificate issue
If you encounter a
SSLHandshakeError, the SSL certificate entry might be missing from your operating system's certificate store. Resolve the issue by adding the certificate to your operating system's trust list.
- Navigate to the certificate store for your operating system. Certificate store locations vary by operating system.
- Add the SSL certificate.
- Save your changes.
By default, the SSL handshake is configured to function normally. The certificate used by ServiceNow is signed by the Entrust Certification Authority.
For more information on adding SSL certificates to your operating system's certificate store, see the How can I trust CAcert's root certificate? page in the CAcert Certificate Authority wiki.
Turn off SSL certificate communication
Communication to ServiceNow is performed via HTTPS. SSL certificate validation is enabled by default. If your ServiceNow data collection is over unencrypted communication (without certificate checks), you must disable the SSL check flag in
splunk_ta_snow_account.conf when upgrading the Splunk Add-on for ServiceNow.
Follow these steps:
- Navigate to
$SPLUNK_HOME/Splunk_TA_snow/localand create a
splunk_ta_snow_account.conffile if it does not already exist.
- Save your changes.
Custom search commands or alert-triggered scripts fail with no results
Check that you have successfully integrated your ServiceNow instance with your Splunk platform instances. If the configuration is unsuccessful, your searches will return "No results found" and the Splunk software logs a
u_splunk_incident does not exist error, which you can find by searching for
If your integration is successful, but incident and event creation fails, run the search
"eventtype=snow_ticket_error" to see what errors are reported. If the failure reason is error code 302, review the ServiceNow URL that you entered in the Setup page to make sure it is correct and does not end with any special characters or trailing slashes.
See Configure ServiceNow to integrate with the Splunk platform to learn more
Errors for data collection for specific database tables
If you are missing data for a specific database table, check your
- "Failure occurred...Not Found" means that the database table might not have any records.
- "Failure occurred...bad request" means that the database table might not exist.
Missing fields after upgrading to Splunk Add-on for ServiceNow 4.0.0
If you have ServiceNow data indexed into your Splunk instance after upgrading to Splunk Add-on for ServiceNow
3.0.04.0.0 from an earlier version, the following panels in the Splunk App for ServiceNow do not display the existing data correctly. Any newly indexed data is not impacted.
- Change Ticket Lookup under cmdb
- Incident Ticket Lookup under cmdb
- Incident Count by Location under Incidents > Open Incidents by Geography
If fields are missing or new fields start with "dv" after upgrading, see Upgrade.
Remove deleted configuration items from the configuration management database lookups
Service Now API for configuration management database (CMDB) does not tell you what configuration items (CI) have been deleted from CMDB. As a result, Splunk does not remove CIs from the CMDB lookups that are deleted. You can manually delete the CIs from the CMDB:
- Enable the data collection for sys_audit_delete:
- Navigate to the Inputs tab in the Splunk Add-on for ServiceNow.
- Configure and enable the
- Create a saved search:
- Create a saved search with the name "ServiceNow Sys Delete List"
sourcetype="snow:sys_audit_delete" | stats count by tablename,documentkey | rename documentkey as sys_id
- Set the
Earliestas 0 and
- Check the Accelerate this search check box and select All Time as Summary Range.
- Save the search.
- Set the saved search to Global.
- Set the
- After creating the saved search, update the existing savedsearch. This change should match the lookup ids with the
sys_audit_deletetable ids and remove it from the lookup. Update the saved search of cmdb tables. In this example, the saved search is named "ServiceNow CMDB CI Server":
eventtype=snow_cmdb_ci_server | dedup sys_id | fields - _bkt, _cd,_indextime,_kv,_raw,_serial,_si,_sourcetype,_subsecond, punct, index, source, sourcetype | inputlookup append=t cmdb_ci_server_lookup | dedup sys_id | outputlookup cmdb_ci_server_lookup
Add the following to each query:
| join max=0 type=left sys_id [ | savedsearch "ServiceNow Sys Delete List" | eval sys_id_delete=sys_id | table sys_id,sys_id_delete ] | where isnull(sys_id_delete)
eventtype=snow_cmdb_ci_server | dedup sys_id | fields - _bkt, _cd,_indextime,_kv,_raw,_serial,_si,_sourcetype,_subsecond, punct, index, source, sourcetype | join max=0 type=left sys_id [ | savedsearch "ServiceNow Sys Delete List" | eval sys_id_delete=sys_id | table sys_id,sys_id_delete ] | where isnull(sys_id_delete) | dedup sys_id | outputlookup cmdb_ci_server_lookup
Repeat this procedure for each of the following saved searches:
- ServiceNow CMDB CI List
- ServiceNow CMDB CI Server
- ServiceNow CMDB CI VM
- ServiceNow CMDB CI Infra Services
- ServiceNow CMDB CI Database Instances
- ServiceNow CMDB CI App Servers
- ServiceNow CMDB CI Relation
- ServiceNow CMDB CI Services
ServiceNow data collection stops after upgrading Splunk Add-on for ServiceNow to 4.0.0
See SSL certificate issues to collect data over encrypted communication.
For an on-premises installation using data collection over unencrypted communication, a message—"Data collection over unencrypted communication is unsecured"—displays. See Turn off SSL certificate communication.Make sure you have followed the steps in Upgrade the Splunk Add-on for ServiceNow. To check whether data is indexing, run this search:
If a configuration is missing, one of the following log messages displays:
No configured inputs found. To collect data from ServiceNow, configure new input(s) or update existing input(s) either from Inputs page of the Add-on or manually from inputs.conf.This message indicates that no inputs are enabled. Go to the Inputs page and configure new inputs or update existing inputs.
No account configurations found for this add-on. To start data collection, configure new account on Configurations page and link it to an input on Inputs page. Exiting TA.This message indicates that no account is configured. You must configure an account and link it to input.
No ServiceNow account linked to the data input <input_name>. To resume data collection, either configure new account on Configurations page or link an existing account to the input on Inputs page.This indicates that an account is configured, but not linked to your input. You must link the specified input to the account.
- 2018-12-28 17:41:58,703 ERROR pid=2953 tid=MainThread file=snow.py:stream_events:471 | Error Traceback (most recent call last): This indicates that the user has changed the account name from the back-end. This is not a best practice. In this case, re-enter the password for this account to resume the data collection.
File "/opt/splunk_snow_test/splunk/etc/apps/Splunk_TA_snow/bin/snow.py", line 352, in stream_events splunk_ta_snow_account_conf = account_cfm.get_conf("splunk_ta_snow_account", refresh=True).get_all() File "/opt/splunk_snow_test/splunk/etc/apps/Splunk_TA_snow/bin/Splunk_TA_snow/solnlib/utils.py", line 154, in wrapper return func(*args, **kwargs) File "/opt/splunk_snow_test/splunk/etc/apps/Splunk_TA_snow/bin/Splunk_TA_snow/solnlib/conf_manager.py", line 241, in get_all key_values = self._decrypt_stanza(name, stanza_mgr.content) File "/opt/splunk_snow_test/splunk/etc/apps/Splunk_TA_snow/bin/Splunk_TA_snow/solnlib/conf_manager.py", line 126, in _decrypt_stanza self._cred_mgr.get_password(stanza_name)) File "/opt/splunk_snow_test/splunk/etc/apps/Splunk_TA_snow/bin/Splunk_TA_snow/solnlib/utils.py", line 154, in wrapper return func(*args, **kwargs) File "/opt/splunk_snow_test/splunk/etc/apps/Splunk_TA_snow/bin/Splunk_TA_snow/solnlib/credentials.py", line 126, in get_password (self._realm, user)) CredentialNotExistException: Failed to get password of realm=__REST_CREDENTIAL__#Splunk_TA_snow#configs/conf-splunk_ta_snow_account, user=<account_name>.
Unable to create an incident/event on your ServiceNow instance
If you are unable to create an incident, complete the following steps:
- Perform the following search to check the error message in the internal logs for ServiceNow:
index=_internal sourcetype="ta_snow_ticket" "One of the possible causes of failure is absence of event management plugin or Splunk Integration plugin"
- Check for this error message:
Failed to create ticket. Return code is 400. Reason is Bad Request. One of the possible causes of failure is absence of event management plugin or Splunk Integration plugin on the ServiceNow instance. To fix the issue install the plugin(s) on ServiceNow instance.
- When you see this message, you need to install the Splunk Integration/Event Management plugin on your ServiceNow instance. See Configure ServiceNow to integrate with Splunk Enterprise.
Edit the display values for the ServiceNow API
About the commands, alert actions, and scripts available with the Splunk Add-on for ServiceNow
This documentation applies to the following versions of Splunk® Supported Add-ons: released