Log files in Splunk Enterprise Security
Splunk Enterprise Security uses many custom log files to log errors and activity specific to the application.
Use the log files to check for activity
You can check the log files for errors and activity. The path for all log files is $SPLUNK_HOME/var/log/splunk/.
You can also use log files from the Splunk platform to audit Splunk Enterprise Security activity using these log files: splunkd_access.log and audit.log.
analyticstory_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| analyticstory_rest_handler |
Analytic Stories: REST Handler |
SA-ThreatIntelligence |
Logs create, read, update, and delete (CRUD) operations for analytics stories.
|
app_certs_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| app_certs_rest_handler |
Application Certificates: REST Handler |
SA-Utils |
Logs CRUD options for certificates uploaded via the "Credential Management" page.
|
app_imports_update.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| app_imports_update |
App Imports Update: REST Handler |
SA-Utils |
Checks if apps, which had previously been imported, are not exporting their knowledge objects globally so that they are visible within ES. The output is complementary to the configuration_check.log file.
|
app_permissions_manager.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| app_permissions_manager |
App Permissions: Modular Input |
SplunkEnterpriseSecuritySuite |
Logs when permissions policies are changed or enforced.
|
app_permissions_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| app_permissions_rest_handler |
App Permissions: REST Handler |
SplunkEnterpriseSecuritySuite |
Persistent rest handler for returning a list of ES permissions related to the the ess_permissions page.
|
appmaker_base_class.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| appmaker:base_class |
App Maker: Base |
SA-Utils |
Super class for all the appmaker scripts. The make_on_prem.py script is used on Distributed Conf Management, which also has its own log file. The make_index_time_properties.py script is used by Distribute Conf Download. Th make_content_pack.py script is used on Content Management when exporting knowledge objects.
|
appmaker_make_content_pack.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| appmaker:make_content_pack |
App Maker: Make Content Pack |
SA-Utils |
Logs when exporting from Content Management into an app.
|
appmaker_make_on_prem.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| appmaker:make_on_prem |
App Maker: Make On Prem |
SA-Utils |
Logs when downloading the distributed configuration management application "Splunk_TA_AROnPrem" in General Settings.
|
appmaker_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| appmaker:rest_handler |
App Maker: REST Handler |
SA-Utils |
Logs export requests from the Content Management page, including the export package name as well as the download requests for exported packages.
|
apps_shc_es_deployer_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| apps_shc_es_deployer_rest_handler |
SHC Installer: REST Handler |
SplunkEnterpriseSecuritySuite |
Persistent rest handler for managing apps on a search head cluster deployer.
|
configuration_check.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| configuration_check |
Configuration Check: Modular Input |
SA-Utils |
Logs output messages of the confcheck migration scripts, such as when migration from the correlationsearches.conf file to the savedsearches.conf file fails.
|
contentinfo.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| contentinfo |
ContentInfo: Search Command |
SA-Utils |
Logs the data sources referenced by contentinfo search-related objects.
|
contentinfo_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| contentinfo_rest_handler |
ContentInfo: REST Handler |
SA-Utils |
Logs errors and successful operations to the contentinfo REST handler and associated components, as used mostly by the Use Case Library and Analytic Story pages.
|
correlationmigration_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| correlationsearches:migration_rest_handler |
Correlation Migration: REST Handler |
SA-ThreatIntelligence |
Logs when migration from correlationsearches.conf to savedsearches.conf fails.
|
customsearchbuilder_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| customsearchbuilder:rest_handler |
Custom Search Builder: REST Handler |
SA-ThreatIntelligence |
Logs when the search syntax of a detection, a lookup generating search, or an Assets and Identities LDAP search cannot be created or is incorrect.
|
data_migrator.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| data_migrator |
Data Migrator: Modular Input |
SA-Utils |
Logs migration operations during ES upgrades. For example, when searches are executed as first-time run tasks or when a CSV lookup table is migrated to a KV store collection during an app upgrade.
|
datamodelsimple.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| datamodelsimple |
Data Model Simple: Search Command |
Splunk_SA_CIM |
Logs when datamodelsimple starts and finishes processing in a search command.
|
entity_merge.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| identity_correlation:merge |
Identity Correlation Merge: Search Command |
SA-IdentityManagement |
Logs the status of the search process during asset and identity merge.
|
es_investigations_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| es_investigations_rest_handler |
ES Investigations Conf: REST Handler |
SplunkEnterpriseSecuritySuite |
Returns knowledge objects and handles change request for them, also enforces schemas and other stanza-specific prefixes and so on.
|
esconfighealth.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| esconfighealth |
ES Configuration Health: Search Command |
SplunkEnterpriseSecuritySuite |
For installation and upgrade, logs the health of ES configurations against a manifest file that ships with each ES release. This typically logs as a result of running a config health check through the ES Configuration Health custom search command feature.
|
ess_configured_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| ess_configured_handler |
ES Configured: REST Handler |
SplunkEnterpriseSecuritySuite |
Logs current configured version state of search head cluster captains and search head cluster members for ES during setup and reset.
|
ess_content_importer.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| ess_content_importer |
ES Content Importer: Modular Input |
SplunkEnterpriseSecuritySuite |
Logs when importing content from installed apps.
|
essinstaller2.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| essinstall2 |
ES Installer: Search Command |
SplunkEnterpriseSecuritySuite |
Logs installation status after setup completes.
|
event_sequencing_engine.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| event_sequencing_engine_log |
Event Sequencing Engine: Search Command |
SplunkEnterpriseSecuritySuite |
Logs event sequencing engine operations such as terminate for sequence templates.
|
expectedactivity.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| expectedactivity |
Expected Activity: Search Command |
SA-Utils |
Pertains to the Expected Activity custom search command. Logs when filling in gaps in results in preparation for use in statistical calculations. For example in stats, chart, or timechart.
|
governance_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| governance:rest_handler |
Governance: REST Handler |
SA-ThreatIntelligence |
Logs when handling governance configurations and collections.
|
identdelete.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| identity_correlation:delete |
Identity Correlation Delete: Search Command |
SA-IdentityManagement |
Logs when pruning identities marked for deletion from the assets_by_str, assets_by_cidr, or identities_expanded collections.
|
identity_correlation_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| identity_correlation:rest_handler |
Identity Correlation: REST Handler |
SA-IdentityManagement |
Logs when creating, editing, validating, and deleting correlations for automatic lookups.
|
identity_manager.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| identity_correlation:modular_input |
Identity Correlation: Modular Input |
SA-IdentityManagement |
Logs when asset and identity information is merged into Splunk asset and identity lookup tables.
|
identitymapper.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| identity_correlation:identitymapper |
Identity Mapper: REST Handler |
SA-IdentityManagement |
Logs during reverse lookup searches for assets or identities.
|
investigation_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| investigation_rest_handler |
Investigation Workbench: REST Handler |
SplunkEnterpriseSecuritySuite |
Logs errors and such related to investigations, such as investigation data, entries, attachments, and cross-references to investigations from the Mission Control dashboard.
|
log_review_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| log_review_rest_handler |
Log Review Conf: REST Handler |
SA-ThreatIntelligence |
Logs management information for REST changes made to log_review.conf, which is used by the Mission Control dashboard.
|
lookup_table_custom_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| lookup_table_custom_rest_handler |
Lookup Table Custom: REST Handler |
SA-Utils |
Logs interactions with ES-managed csv lookups, including uploading new lookups through content management, as well as editing lookups in the lookup editor.
|
managed_lookups_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| managed_lookups_rest_handler |
Managed Lookups: REST Handler |
SA-Utils |
Logs internal operations such as settings checks for managed lookups.
|
managed_nav_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| managed_nav_rest_handler |
Managed Navigation: REST Handler |
SA-Utils |
Logs CRUD operations for the ES navigation menu, typically through the Navigation editor page.
|
modaction_adhoc_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| modaction:adhoc_rest_handler |
Modular Action Adhoc: REST Handler |
Splunk_SA_CIM |
CIM: Adaptive Response actions execution. Logs when ad hoc searches result in adaptive response actions.
|
modaction_invocations_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| modaction:invocations_rest_handler |
Modular Action Invocations: REST Handler |
Splunk_SA_CIM |
CIM: Adaptive Response actions execution
|
modaction_queue_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| modaction:queue_handler |
Modular Action Queue: REST Handler |
Splunk_SA_CIM |
Logs when handling the queue for Common Action Model properties.
|
notable_event_suppression.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| notable_event_suppression |
Notable Event Suppression: Base |
SA-ThreatIntelligence |
Logs when managing notable event suppressions.
|
notable_event_suppression_autoDisable.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| notable_event_suppression:autoDisable |
Notable Event Suppression: Auto Disable |
SA-ThreatIntelligence |
Logs on auto-disable for finding suppressions of adhoc risk events.
|
notable_update_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| notable_update_rest_handler |
Notable Event Update: REST Handler |
SA-ThreatIntelligence |
Logs when changing findings in the analyst queue on the Mission Control dashboard.
|
outputcheckpoint.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| outputcheckpoint |
Output Checkpoint: Search Command |
SA-Utils |
Logs when outputting the results of the previous search pipeline to a modular input checkpoint directory.
|
per_panel_filtering.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| per_panel_filtering |
Per Panel Filtering |
SA-Utils |
Logs per panel filtering changes.
|
relaymodaction.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| relaymodaction |
Modular Action Relay: Modular Input |
Splunk_SA_CIM |
Logs when managing remote Splunk instance modular actions.
|
reviewstatuses_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| reviewstatuses:rest_handler |
Reviewstatuses: REST Handler |
SA-ThreatIntelligence |
Logs when handling knowledge objects for configuring notable statuses and investigation statuses.
|
sequence_instance_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| sequence_instance_rest_handler |
Sequence Instance: REST Handler |
SplunkEnterpriseSecuritySuite |
Logs when handling an instance of a running sequenced event.
|
sequence_templates_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| sequence_templates_rest_handler |
Sequence Templates: REST Handler |
SplunkEnterpriseSecuritySuite |
Logs when making CRUD operations to the configuration of sequence templates.
|
sorttimecols.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| sorttimecols |
Sort Time Columns: Search Command |
SA-Utils |
Pertains to the sorttimecols custom search command. Logs when using the sorttimecols commands to sort columns in a result set by time.
|
suppressions_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| notable_event_suppression:rest_handler |
Notable Event Suppression: REST Handler |
SA-ThreatIntelligence |
REST handler for notable suppression create and edit. For use in conjunction with the notable_event_suppression.log file.
|
threat_intel_file_upload_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| threatintel:file_upload_rest_handler |
Threat Intel Upload: REST Handler |
DA-ESS-ThreatIntelligence |
rest handler for uploading threat intelligence files
|
threat_intelligence_manager.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| threatintel:manager |
Threat Intel Manager: Modular Input |
DA-ESS-ThreatIntelligence |
Logs when the modular input parses the threat sources and updates the KV Store threat collections with any new intelligence.
|
threat_intelligence_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| threatintel:rest_handler |
Threat Intel: REST Handler |
DA-ESS-ThreatIntelligence |
Logs activity of threat intel endpoints.
|
threatlist.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| threatintel:download |
Intelligence Download: Modular Input |
SA-ThreatIntelligence |
Logs the status of threat intel downloads, including success and failure.
|
transitioners_rest_handler.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| transitioners_rest_handler |
Transitioners: REST Handler |
SA-ThreatIntelligence |
notable status handler, checking permission who can change status, also migrates from authorize.conf to reviewstatuses.conf.
|
uba_rest.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| uba:rest_handler |
UBA: REST Handler |
SA-UEBA |
Pertains to the UBA Integration rest handler.
|
whois_manager.log
| Sourcetype |
Component |
Eai:acl.app |
Description
|
| whois_manager |
Whois Manager: Modular Input |
SA-NetworkProtection |
Logs when executing the whois modular input data.
|
Use search to check for activity
You can use search to check for errors and activity. The majority of sourcetypes can be searched in the _internal index. The notable_update_rest_handler can also be searched for as a source in the _audit index.
Searching the _internal index for notable_update_rest_handler will show you, for example, what happens during the handler review process.
Example search:
index=_internal sourcetype="notable_update_rest_handler"
Example response:
| i |
Time |
Event
|
| >
|
12/2/19 3:07:16.525 PM
|
2019-12-02 20:07:16,525+0000 INFO pid=8649 tid=MainThread file=rest_handler.py:handle:728 NotableEventUpdate.handle_post duration=4.474
host = hostname = /usr/local/bamboo/splunk-install/current/var/log/splunk/notable_update_rest_handler.log sourcetype = notable_update_rest_handler
|
| >
|
12/2/19 3:07:16.524 PM
|
2019-12-02 20:07:16,524+0000 INFO pid=8649 tid=MainThread file=notable_update_rest_handler.py:setStatuses:957 Done editing events matching search admin__admin__SplunkEnterpriseSecuritySuite__RMD57f02abc0263583b0
_1575317218.11939
host = hostname = /usr/local/bamboo/splunk-install/current/var/log/splunk/notable_update_rest_handler.log sourcetype = notable_update_rest_handler
|
| >
|
12/2/19 3:07:16.524 PM
|
2019-12-02 20:07:16,524+0000 INFO pid=8649 tid=MainThread file=cim_actions.py:message:425 I sendmodaction - worker="soln-esnightly1" signature="Successfully created splunk events" action_name="notable_event_edit" digest_mode="1" action_mode="adhoc" event_count="1"
host = hostname source = /usr/local/bamboo/splunk-install/current/var/log/splunk/notable_update_rest_handler.log sourcetype = notable_update_rest_handler
|
Searching the _audit index for the source of notable_update_rest_handler will show you, for example, what was saved to the KV Store during the handler processing. This is not necessarily for troubleshooting, but more specific to the activity on the Mission Control page.
Example search:
index=_audit sourcetype="incident_review"
Example response:
| i |
Time |
Event
|
| >
|
12/2/19 3:07:13.090 PM
|
1575317233.09,19E67472-762C-4636-9A91-E4CF6B4BD885@@notable@@15c339addb8d09e6d8a24176beafd9792bd84f45,Host With Multiple Infections,4,esadmin,high,comment,admin,True
host = hostname source = notable_update_rest_handler sourcetype = incident_review
|
Download manual
Feedback submitted, thanks!