Troubleshoot script errors in Splunk Enterprise Security

Troubleshoot script errors from modular inputs in Splunk Enterprise Security.

Issue

Error message about a script exiting abnormally or a script that is in an unknown state

Cause

The Audit - Script Errors search replaces a configuration check script and creates Splunk messages to warn about non-zero exit codes that result from scripts in your Splunk deployment.

Additionally, how apps are exported globally in Splunk Enterprise Security can also cause script errors. Splunk Enterprise Security does not selectively import apps and add-ons based on the name of the app or add-on. Knowledge objects in apps and add-ons that are installed on the same search head as Splunk Enterprise Security and exported to other apps or globally are visible in Splunk Enterprise Security.

Apps that are not exported globally are flagged by the confcheck_app_exports.py health check. This creates health check messages, especially after a Splunk ES upgrade, as shown in the following figure:

Solutions

Investigate the script and stanza that produced the error

The following table provides information on possible root cause analysis for script errors:

Possible root cause	Verification	Mitigation
The script did not run successfully.	Review the log files for the script. Run the script manually to see if it runs successfully, and review the exit code that results.	Address the reasons why the script exited with a non-zero exit code.
The script ran successfully with a non-zero exit code.	Run the script manually to see if it runs successfully, and review the exit code that results.	Include the script in the suppression for the search so that it does not display messages for this script.
The script is in an unknown state. There is a stop time for the script, but no exit status or start time.	Check the modular input settings to confirm they are correct.	Correct the modular input settings.

Reduce the frequency of script error messages

You can customize the display of script errors by either reducing the frequency of all script errors, ignoring certain script errors by editing the `script_error_msg_ignore` macro, or reducing the frequency of specific script errors.

Follow these steps to edit the cron schedule of the search to reduce the frequency of all script errors:

1. In Splunk Enterprise Security, select Security content and then select Content Management.
2. Search for the "Audit - Script Errors" saved search and select it to navigate to the saved search editor page.
3. Edit the search by selecting Edit and then selecting Edit alert.
4. Scroll to the Cron schedule section and edit the value using standard cron notation.

Script error messages can occur based on the cron schedule that is set with the search.

Edit macro to ignore specific script errors

Ignore the script error messages about specific scripts by modifying the syntax in the `script_error_msg_ignore` macro.

For example: If you want to ignore the following health check error:

Health Check: msg="A script exited abnormally with exit status: 1" input=".$SPLUNK_HOME/etc/apps/splunk-dashboard-studio/bin/save_image_and_icon_on_install.py" stanza="default" Learn more.

You can edit the `script_error_msg_ignore` macro as follows:

(match(script, "(streamfwd|splunk-(wmi\.path|MonitorNoHandle\.exe|winevtlog\.exe|netmon\.exe|perfmon\.exe|regmon\.exe|winprintmon\.exe|admon\.exe|powershell\.exe))") AND exit_status=1) OR (script LIKE "%instrumentation.py" AND exit_status=114) OR (script LIKE "%save_image_and_icon_on_install.py" AND exit_status=1)

Throttle script errors

Reduce the frequency of messages about specific scripts instead of preventing them from appearing by throttling the alerts. Set up alert throttling for the Audit - Script Errors search based on the necessary values, such as the script field.

• For Splunk Enterprise, see Throttle alerts in the Alerting Manual.
• For Splunk Cloud Platform, see Throttle alerts in the Alerting Manual.

Turn off the configuration checker

To stop the messages by disabling the configuration checks, such as confcheck_app_exports.py, do the following:

1. On the Enterprise Security menu bar, select Configure > General > Configuration Checker.
2. Find the name of the script and click Deactivate / Turn off.

Though in the case of confcheck_app_exports.py specifically, see Export apps globally to verify if you want to export the apps or turn off the configuration checker.

Turn off health check messages

Follow these steps if you receive a health check warning after you upgrade Splunk ES:

1. Verify the app is exported globally. To verify a global export from the search head, check the following setting in the user directory or the local.meta file of the app or add-on.

   [server]
   export = system
   

   The export setting indicates that the app is globally available unless you've explicitly restricted it by role or user.

   The user directory is located at $SPLUNK_HOME/etc/users/<user_name>/<app_name>/local.
2. When installing Splunk ES in a search head cluster environment, verify that your server.conf shclustering is in the following location: $SPLUNK_HOME/etc/system/local/server.conf or is in an app that exports the server configuration globally using metadata.
3. If the app is not globally available, you have the following options:
   • Make the app globally available.
   • Stop the health check messages.
   • Suppress configuration checks.

Export apps globally

Prerequisite User has permission to write to the original app.

Add the following stanza to $SPLUNK_HOME/etc/apps/unix/metadata/local.meta to export apps globally:

[eventtypes/rhallen]
export = system

If you do not want to make the app globally available and stop the display of health check messages, turn off the configuration checker confcheck_app_exports.py.

1. On the Enterprise Security menu bar, select Configure > General > Configuration Checker.
2. Find the name of the script and click Deactivate / Turn off.
3. Suppress configuration checks selectively. To selectively suppress configuration checks and turn off the configuration checker, apply a regex for "Suppress" in confcheck_app_exports.py. For more information on suppressing configuration checks, see Turn off the configuration checker.

See also

For more information on troubleshooting script errors, see the product documentation:

• Configure a script for an alert action in the Splunk Enterprise Alerting Manual
• What Splunk software logs about itself in the Splunk Enterprise Troubleshooting Manual.
• Prerequisites for installing Enterprise Security in a search head cluster environment in the Splunk Enterprise Installation and Upgrade Manual.
• Make Splunk knowledge objects globally available in the Splunk Enterprise Admin Manual.