Troubleshoot the Splunk Add-on for BMC Remedy

General troubleshooting

For helpful troubleshooting tips that you can apply to all add-ons, see Troubleshoot add-ons in Splunk Add-ons. For additional resources, see Support and resource links for add-ons in Splunk Add-ons.

Find relevant errors

The Splunk Add-on for BMC Remedy contains the following internal logs in the $SPLUNK_HOME/var/log/splunk directory:

Data source name	Source type	Description	Collection method	API
splunk_ta_remedy_alert.log	ta:remedy:alert	Logs related to creating or updating incidents in Remedy using Splunk software alert actions.	File Monitoring	None
splunk_ta_remedy_ticket.log	ta:remedy:ticket	Logs related to creating or updating incidents in Remedy using Splunk software custom commands.	File Monitoring	None
splunk_ta_remedy_incident.log	ta:remedy:incident	Logs related to Splunk software endpoint which fetches incident details from correlation id in Remedy.	File Monitoring	None
remedy_incident_modalert.log	ta:remedy:modalert	Audit logs of activities related to Splunk software alert actions.	File Monitoring	None
splunk_ta_remedy_rest_account_validation.log	ta:remedy:rest:account_validation	Logs related to Server Validation using REST on REST Account tab.	File Monitoring	None
splunk_ta_remedy_soap_account_validation.log	ta:remedy:soap:account_validation	Logs related to Server Validation using SOAP on SOAP Account tab.	File Monitoring	None
splunk_ta_remedy_input.log	ta:remedy:input	Logs related to Data Collection.	File Monitoring	None
splunk_ta_remedy_data_input_checkpoint.log	ta:remedy:data_input:checkpoint	Logs related to checkpoint mechanism during data collection.	File Monitoring	None

Configure the logging verbosity level in the add-on setup page.

Search for the following event types to find errors relevant to the Splunk Add-on for BMC Remedy.

• Search index=_internal sourcetype=ta:remedy:alert for errors related to creating or updating incidents in Remedy using alert actions.
• Search index=_internal sourcetype=ta:remedy:ticket for errors related to creating or updating incidents in Remedy using commands.
• Search index=_internal sourcetype=ta:remedy:incident for errors related to the Splunk platform endpoint which fetches incident details from the correlation id in Remedy.
• Search index=_internal sourcetype=ta:remedy:modalert for errors related to the execution of alert actions.
• Search index=_internal sourcetype=ta:remedy:rest:account_validation for errors related to saving account on REST Account Tab.
• Search index=_internal sourcetype=ta:remedy:soap:account_vlidationticket for errors related to saving account on SOAP Account Tab.
• Search index=_internal sourcetype=ta:remedy:input for errors related to data collection for inputs configured.
• Search index=_internal sourcetype=ta:remedy:data_input:checkpoint for errors related to checkpoint during data collection.

SSLError after add-on upgrade

If you are using a self signed certificate for your remedy server and see the following error in the TA logs of the add-on after upgrading, refer to the steps in the Splunk Add-on for BMC Remedy.

SSLError occurred. If you are using self signed certificate and your certificate is at /etc/ssl/ca-bundle.crt, please refer the troubleshooting section in add-on documentation.

splunk_ta_remedy_settings.conf not replicated to other members in a search head cluster

If you update the required fields in the splunk_ta_remedy_settings.conf file on a search head member in a search head cluster, the file will not be synced to the other search head members. You must manually update the required fields in the splunk_ta_remedy_settings.conf file on each search head in the cluster separately.

Custom command errors

<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:676)>

If you receive a <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:676)> error, check your add-on's configuration in splunk_ta_remedy_settings.conf. This issue comes when SSL communication and certificate validation is enabled, but the server certificate cannot be verified from available certificates with the client.

[Errno 2] No such file or directory

If you receive a [Errno 2] No such file or directory error, then you may have specified a certificate path for the ca_certs_path field in the additional_parameters stanza of splunk_ta_remedy_settings.conf. Check if the server's CA certificate is available at the specified location.is you need to disable the certificate validation or add/verify ca cert path as mentioned in Encrypted/Unencrypted Communication Section.

For Splunk Cloud users, the ca_certs_path located at ca_certs_path = /etc/ssl/certs/ca-certificates.crt. Location may vary, based on the operating system (OS) used.

IOError:[Errno 13] Permission Denied error

If you receive a IOError:[Errno 13] Permission Denied error, then you need the folder permissions where server ca certificate is present and of local folder of addon where wsdl files from Remedy server will be downloaded.

Server raised fault: 'ERROR (48255): ; The contact for the creation of an Incident cannot be found, please verify the contact information and try again.'

If you receive a Server raised fault: 'ERROR (48255): ; The contact for the creation of an Incident cannot be found, please verify the contact information and try again. error while creating an incident, check whether the account of whose first name and last name are provided in the command has rights to create an incident on the Remedy server.

Server raised fault: 'ERROR (326): Required field cannot be blank.; 2506'

If you receive a Server raised fault: 'ERROR (326): Required field cannot be blank.; 2506 error while creating/updating an incident, check all the mandatory fields required to be passed while creating or updating the incidents on Remedy server.

Server raised fault: 'ERROR (1291018): ; You do not have the authority to modify incident, "INCXXXXX"

If you receive a Server raised fault: 'ERROR (1291018): ; You do not have the authority to modify incident, "INCXXXXX" , check if the provided user is a member or an associate member of either the owner group or the assigned group in Remedy server.

Server raised fault: 'ERROR (1291205): ; The Assignee field must contain a value. Choose an assignee from the associated field menu, then save the Incident.'

If you receive a Server raised fault: 'ERROR (1291205): ; The Assignee field must contain a value. Choose an assignee from the associated field menu, then save the Incident., verify on your Remedy server that the assignee field is not empty for that specific incident. If it is blank, then add the assignee value.

'Remedy Web Service has not been setup' error

If you receive a Remedy Web Service has not been setup error, check Configuration > Account to verify whether remedy server configuration is configured. If it is blank, then add the valid configuration of your BMC Remedy server.

urlopen error [SSL: UNKNOWN_PROTOCOL] unknown protocol

If you receive a urlopen error [SSL: UNKNOWN_PROTOCOL] unknown protocol error, verify that your Remedy Server URL as mentioned in Configuration > Account is encrypted via HTTPS with a valid SSL certificate.

Add-on using protected URL instead of public URL

If you see that the Splunk Add-on for BMC Remedy is still using protected in WSDL URL, although the add-on has modified it to public, check if you have explicitly specified it in the [remedy_ws] stanza in splunk_ta_remedy_settings.conf. Modify the splunk_ta_remedy_settings.conf file, if required.

Alert action errors

Alert action error: No groups were found using automated routing. You need to manually select a group.

If you receive a No groups were found using automated routing. You need to manually select a group error, after configuring an alert action, this issue is due to the configuration in your BMC environment. You need to make sure that routing rules are configured in your BMC Remedy deployment. The assignment routing rules can be configured in your Remedy environment by navigating to the "Administrator Console > Application Administration Console >> Custom Configuration tab" section.