Troubleshoot script errors in Splunk Enterprise Security
Troubleshoot script errors from modular inputs in Splunk Enterprise Security. If you see a message about a script exiting abnormally or a script that is in an unknown state, investigate the script and stanza that produced the error.
Audit - Script Errors search replaces a configuration check script and creates Splunk messages to warn about non-zero exit codes that result from scripts in your Splunk deployment.
|Possible root cause||Verification||Mitigation|
|The script did not run successfully.||Review the log files for the script. Run the script manually to see if it runs successfully, and review the exit code that results.||Address the reasons why the script exited with a non-zero exit code.|
|The script ran successfully with a non-zero exit code.||Run the script manually to see if it runs successfully, and review the exit code that results.||Include the script in the suppression for the search so that it does not display messages for this script.|
|The script is in an unknown state. There is a stop time for the script, but no exit status or start time.||Check the modular input settings to confirm they are correct.||Correct the modular input settings.|
Customize messages about specific scripts
You can customize the display of script errors by either reducing the frequency of all script errors, ignoring certain script errors by editing the
`script_error_msg_ignore` macro, or reducing the frequency of specific script errors.
Edit the cron schedule to reduce the frequency of all script errors
Reduce the frequency of all script errors by editing the cron schedule of the search.
- From the ES menu bar, select Configure > Content > Content Management.
- Search for the "Audit - Script Errors" saved search and click on it to navigate to the saved search editor page.
- Edit the search by clicking Edit > Edit Alert.
- Scroll to the Cron Schedule section and edit the value using standard cron notation.
Script error messages can occur based on the cron schedule that is set with the search.
Edit macro to ignore specific script errors
Ignore the script error messages about specific scripts by modifying syntax in the
For example: If you want to ignore the following health check error:
Health Check: msg="A script exited abnormally with exit status: 1" input=".$SPLUNK_HOME/etc/apps/splunk-dashboard-studio/bin/save_image_and_icon_on_install.py" stanza="default" Learn more.
You can edit the
`script_error_msg_ignore` macro as follows:
(match(script, "(streamfwd|splunk-(wmi\.path|MonitorNoHandle\.exe|winevtlog\.exe|netmon\.exe|perfmon\.exe|regmon\.exe|winprintmon\.exe|admon\.exe|powershell\.exe))") AND exit_status=1) OR (script LIKE "%instrumentation.py" AND exit_status=114) OR (script LIKE "%save_image_and_icon_on_install.py" AND exit_status=1)
Throttle script errors
Reduce the frequency of messages about specific scripts instead of preventing them from appearing by throttling the alerts. Set up alert throttling for the
Audit - Script Errors search based on the necessary values, such as the
- For Splunk Enterprise, see Throttle alerts in the Alerting Manual.
- For Splunk Cloud Platform, see Throttle alerts in the Alerting Manual.
Disable the configuration checker
To stop the messages by disabling the configuration checks, such as
confcheck_app_exports.py, do the following:
- On the Enterprise Security menu bar, select Configure > General > Configuration Checker.
- Find the name of the script and click Disable.
Though in the case of
confcheck_app_exports.py specifically, see Export apps globally to verify if you want to export the apps or disable the configuration checker.
Export apps globally
Splunk Enterprise Security no longer selectively imports apps and add-ons based on the name of the app or add-on. Knowledge objects in apps and add-ons that are installed on the same search head as Splunk Enterprise Security and exported to other apps or globally are visible in Splunk Enterprise Security. Apps that are not exported globally are flagged by the
confcheck_app_exports.py health check.
To verify a global export from the search head, check the
local.meta file of the app or add-on for
export = system. For further details, see the "Make Splunk knowledge objects globally available" section of App architecture and object ownership in the Splunk Enterprise Admin Manual.
Or when installing ES in a search head cluster environment, verify that your
server.conf shclustering configuration is in
$SPLUNK_HOME/etc/system/local/server.conf or is in an app that exports the server configuration globally via metadata:
export = system
See Prerequisites for installing Enterprise Security in a search head cluster environment in the Splunk Enterprise Installation and Upgrade Manual.
Dashboard requirements matrix for Splunk Enterprise Security
Troubleshoot messages about default indexes searched by the admin role
This documentation applies to the following versions of Splunk® Enterprise Security: 6.6.2