Splunk® Supported Add-ons

Splunk Add-on for ServiceNow

Download manual as PDF

Download topic as PDF

Troubleshoot the Splunk Add-on for ServiceNow

For troubleshooting tips that you can apply to all add-ons, see Troubleshoot add-ons in Splunk Add-ons. For additional resources, see Support and resource links for add-ons in Splunk Add-ons.

Cannot launch add-on

This add-on does not have views and is not intended to be visible in Splunk Web. If you are trying to launch or load views for this add-on and you are experiencing results you do not expect, turn off visibility for the add-on.

For more details about add-on visibility and instructions for turning visibility off, see the Check if the add-on is intended to be visible or not section of the Splunk Add-ons Troubleshooting topic.

Cannot access configuration page

If you are trying to reach the setup page but cannot see a link to it on your instance, confirm that you are signed in with an account that is a member of the admin or sc_admin role.

Find relevant errors

Search for the following event types to find errors relevant to the Splunk Add-on for ServiceNow.

Search eventtype=snow_ta_collector_error for errors related to data collection from ServiceNow.

Search eventtype=snow_ticket_error for errors related to creating events or incidents in ServiceNow from the Splunk platform.

Search eventtype=ta_frwk_error for errors related to low-level functions of the add-on.

Search eventtype=snow_ta_log_error for errors related to the add-on as well as account and input configuration.

Missing data

If you are not getting data from all of the inputs that you have enabled, check that the ServiceNow account that you are using to connect to your ServiceNow instance from the Splunk platform has, at minimum, read-only access to all of the database tables from which you are attempting to collect data. Then, disable and re-enable the inputs for which you are not receiving data.

To validate that you do not have a permissions issue:

  1. Edit the following URL to use your ServiceNow instance name:

https://<myservicenowinstance>.service-now.com/<service_now_table>.do?JSONv2&sysparm_query=sys_created_on>=2016-01-01+00:00:00^ORDERBYsys_created_on&sysparm_record_count=50 <myservicenowinstance>.service-now.com

  1. Change service_now_table to the ServiceNow table you are trying to query
  2. Change 2016-01-01 to the actual date you want to query from.
  3. Paste the URL into a browser.
  4. When prompted, log in with the same username and password that you use for the integration account in the add-on.

If you receive the historical data you expect and a sys_updated_on field for each event, you have the correct permissions.

SSL certificate issue

If you encounter a SSLHandshakeError, the SSL certificate entry might be missing from your operating system's certificate store. Resolve the issue by adding the certificate to your operating system's trust list.

  1. Navigate to the certificate store for your operating system. Certificate store locations vary by operating system.
  2. Add the SSL certificate.
  3. Save your changes.

By default, the SSL handshake is configured to function normally. The certificate used by ServiceNow is signed by the Entrust Certification Authority.

For more information on adding SSL certificates to your operating system's certificate store, see the How can I trust CAcert's root certificate? page in the CAcert Certificate Authority wiki.

Turn off SSL certificate communication

Communication to ServiceNow is performed via HTTPS. SSL certificate validation is enabled by default. If your ServiceNow data collection is over unencrypted communication (without certificate checks), you must disable the SSL check flag in splunk_ta_snow_account.conf when upgrading the Splunk Add-on for ServiceNow. Follow these steps:

  1. Navigate to $SPLUNK_HOME/Splunk_TA_snow/local and create a splunk_ta_snow_account.conf file if it does not already exist.
  2. Set disable_ssl_certificate_validation=1.
  3. Save your changes.

Custom search commands or alert-triggered scripts fail with no results

Check that you have successfully integrated your ServiceNow instance with your Splunk platform instances. If the configuration is unsuccessful, your searches will return "No results found" and the Splunk software logs a u_splunk_incident does not exist error, which you can find by searching for eventtype=snow_ticket_error.

If your integration is successful, but incident and event creation fails, run the search "eventtype=snow_ticket_error" to see what errors are reported. If the failure reason is error code 302, review the ServiceNow URL that you entered in the Setup page to make sure it is correct and does not end with any special characters or trailing slashes.

See Configure ServiceNow to integrate with the Splunk platform to learn more

Errors for data collection for specific database tables

If you are missing data for a specific database table, check your splunk_ta_snow_main.log.

  • "Failure occurred...Not Found" means that the database table might not have any records.
  • "Failure occurred...bad request" means that the database table might not exist.

Missing fields after upgrading to Splunk Add-on for ServiceNow 4.0.0

If you have ServiceNow data indexed into your Splunk instance after upgrading to Splunk Add-on for ServiceNow 3.0.04.0.0 from an earlier version, the following panels in the Splunk App for ServiceNow do not display the existing data correctly. Any newly indexed data is not impacted.

  • Change Ticket Lookup under cmdb
  • Incident Ticket Lookup under cmdb
  • Incident Count by Location under Incidents > Open Incidents by Geography

If fields are missing or new fields start with "dv" after upgrading, see Upgrade.

Remove deleted configuration items from the configuration management database lookups

Service Now API for configuration management database (CMDB) does not tell you what configuration items (CI) have been deleted from CMDB. As a result, Splunk does not remove CIs from the CMDB lookups that are deleted. You can manually delete the CIs from the CMDB:

  1. Enable the data collection for sys_audit_delete:
    1. Navigate to the Inputs tab in the Splunk Add-on for ServiceNow.
    2. Configure and enable the sys_audit_delete data input.
  2. Create a saved search:
    1. Create a saved search with the name "ServiceNow Sys Delete List"
sourcetype="snow:sys_audit_delete" | stats count by tablename,documentkey | rename documentkey as sys_id
    1. Set the Earliest as 0 and Latest as now.
    2. Check the Accelerate this search check box and select All Time as Summary Range.
    3. Save the search.
    4. Set the saved search to Global.
  1. After creating the saved search, update the existing savedsearch. This change should match the lookup ids with the sys_audit_delete table ids and remove it from the lookup. Update the saved search of cmdb tables. In this example, the saved search is named "ServiceNow CMDB CI Server":
 eventtype=snow_cmdb_ci_server | dedup sys_id | fields - _bkt, _cd,_indextime,_kv,_raw,_serial,_si,_sourcetype,_subsecond, punct, index, source, sourcetype  | inputlookup append=t cmdb_ci_server_lookup | dedup sys_id | outputlookup cmdb_ci_server_lookup 

Add the following to each query:

 | join max=0 type=left sys_id [ | savedsearch "ServiceNow Sys Delete List" | eval sys_id_delete=sys_id | table sys_id,sys_id_delete ]  | where isnull(sys_id_delete)

Modified query:

 eventtype=snow_cmdb_ci_server | dedup sys_id | fields - _bkt, _cd,_indextime,_kv,_raw,_serial,_si,_sourcetype,_subsecond, punct, index, source, sourcetype  | join max=0 type=left sys_id [ | savedsearch "ServiceNow Sys Delete List" | eval sys_id_delete=sys_id | table sys_id,sys_id_delete ]  | where isnull(sys_id_delete) | dedup sys_id | outputlookup cmdb_ci_server_lookup 

Repeat this procedure for each of the following saved searches:

  • ServiceNow CMDB CI List
  • ServiceNow CMDB CI Server
  • ServiceNow CMDB CI VM
  • ServiceNow CMDB CI Infra Services
  • ServiceNow CMDB CI Database Instances
  • ServiceNow CMDB CI App Servers
  • ServiceNow CMDB CI Relation
  • ServiceNow CMDB CI Services

ServiceNow data collection stops after upgrading Splunk Add-on for ServiceNow to 4.0.0

See SSL certificate issues to collect data over encrypted communication.

For an on-premises installation using data collection over unencrypted communication, a message—"Data collection over unencrypted communication is unsecured"—displays. See Turn off SSL certificate communication.

Make sure you have followed the steps in Upgrade the Splunk Add-on for ServiceNow. To check whether data is indexing, run this search:

index="_internal" sourcetype="ta_snow"

.

If a configuration is missing, one of the following log messages displays:

  1. No configured inputs found. To collect data from ServiceNow, configure new input(s) or update existing input(s) either from Inputs page of the Add-on or manually from inputs.conf. This message indicates that no inputs are enabled. Go to the Inputs page and configure new inputs or update existing inputs.
  2. No account configurations found for this add-on. To start data collection, configure new account on Configurations page and link it to an input on Inputs page. Exiting TA. This message indicates that no account is configured. You must configure an account and link it to input.
  3. No ServiceNow account linked to the data input <input_name>. To resume data collection, either configure new account on Configurations page or link an existing account to the input on Inputs page.This indicates that an account is configured, but not linked to your input. You must link the specified input to the account.
  4. 2018-12-28 17:41:58,703 ERROR pid=2953 tid=MainThread file=snow.py:stream_events:471 | Error Traceback (most recent call last):
      File "/opt/splunk_snow_test/splunk/etc/apps/Splunk_TA_snow/bin/snow.py", line 352, in stream_events
        splunk_ta_snow_account_conf = account_cfm.get_conf("splunk_ta_snow_account", refresh=True).get_all()
      File "/opt/splunk_snow_test/splunk/etc/apps/Splunk_TA_snow/bin/Splunk_TA_snow/solnlib/utils.py", line 154, in wrapper
        return func(*args, **kwargs)
      File "/opt/splunk_snow_test/splunk/etc/apps/Splunk_TA_snow/bin/Splunk_TA_snow/solnlib/conf_manager.py", line 241, in get_all
        key_values = self._decrypt_stanza(name, stanza_mgr.content)
      File "/opt/splunk_snow_test/splunk/etc/apps/Splunk_TA_snow/bin/Splunk_TA_snow/solnlib/conf_manager.py", line 126, in _decrypt_stanza
        self._cred_mgr.get_password(stanza_name))
      File "/opt/splunk_snow_test/splunk/etc/apps/Splunk_TA_snow/bin/Splunk_TA_snow/solnlib/utils.py", line 154, in wrapper
        return func(*args, **kwargs)
      File "/opt/splunk_snow_test/splunk/etc/apps/Splunk_TA_snow/bin/Splunk_TA_snow/solnlib/credentials.py", line 126, in get_password
        (self._realm, user))
    CredentialNotExistException: Failed to get password of realm=__REST_CREDENTIAL__#Splunk_TA_snow#configs/conf-splunk_ta_snow_account, user=<account_name>.
    This indicates that the user has changed the account name from the back-end. This is not a best practice. In this case, re-enter the password for this account to resume the data collection.

Unable to create an incident/event on your ServiceNow instance

If you are unable to create an incident, complete the following steps:

  1. Perform the following search to check the error message in the internal logs for ServiceNow:

    index=_internal sourcetype="ta_snow_ticket" "One of the possible causes of failure is absence of event management plugin or Splunk Integration plugin"

  2. Check for this error message:
    Failed to create ticket. Return code is 400. Reason is Bad Request. One of the possible causes of failure is absence of event management plugin or Splunk Integration plugin on the ServiceNow instance. To fix the issue install the plugin(s) on ServiceNow instance.
PREVIOUS
Edit the display values for the ServiceNow API
  NEXT
About the commands, alert actions, and scripts available with the Splunk Add-on for ServiceNow

This documentation applies to the following versions of Splunk® Supported Add-ons: released


Comments

Thanks for your comment, Christian. Yes, this was a typo. I have corrected the mistake.

Jrevell splunk, Splunker
December 12, 2017

In step #1, under "Missing Data", I believe the "mysinstance.service-now.com" part is a typo. That would fail in any case.

Christian.cloutier
December 4, 2017

Hi Ddearmond, Thanks for your comment about the inaccuracy. I have created a ticket for our engineering team to address the verbiage in this add-on's props.conf file.

Jrevell splunk, Splunker
September 13, 2017

The comments "# For display_value = all, comment the following." in the TA's props.conf are misleading. Doing this reverts the LOOKUP definitions to the default props.conf values. Blanking out the definitions as step 6 advises is the correct way to avoid using the automatic lookup definitions. Please adjust the props.conf comments.

Ddearmond splunk, Splunker
September 12, 2017

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters