Log files in Splunk Enterprise Security

Splunk Enterprise Security uses many custom log files to log errors and activity specific to the application.

Use the log files to check for activity

You can check the log files for errors and activity. The path for all log files is $SPLUNK_HOME/var/log/splunk/.

You can also use log files from the Splunk platform to audit Splunk Enterprise Security activity using these log files: splunkd_access.log and audit.log.

analyticstory_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
analyticstory_rest_handler	Analytic Stories: REST Handler	SA-ThreatIntelligence	Logs create, read, update, and delete (CRUD) operations for analytics stories.

app_certs_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
app_certs_rest_handler	Application Certificates: REST Handler	SA-Utils	Logs CRUD options for certificates uploaded via the "Credential Management" page.

app_imports_update.log

Sourcetype	Component	Eai:acl.app	Description
app_imports_update	App Imports Update: REST Handler	SA-Utils	Checks if apps, which had previously been imported, are not exporting their knowledge objects globally so that they are visible within ES. The output is complementary to the configuration_check.log file.

app_permissions_manager.log

Sourcetype	Component	Eai:acl.app	Description
app_permissions_manager	App Permissions: Modular Input	SplunkEnterpriseSecuritySuite	Logs when permissions policies are changed or enforced.

app_permissions_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
app_permissions_rest_handler	App Permissions: REST Handler	SplunkEnterpriseSecuritySuite	Persistent rest handler for returning a list of ES permissions related to the the ess_permissions page.

appmaker_base_class.log

Sourcetype	Component	Eai:acl.app	Description
appmaker:base_class	App Maker: Base	SA-Utils	Super class for all the appmaker scripts. The make_on_prem.py script is used on Distributed Conf Management, which also has its own log file. The make_index_time_properties.py script is used by Distribute Conf Download. Th make_content_pack.py script is used on Content Management when exporting knowledge objects.

appmaker_make_content_pack.log

Sourcetype	Component	Eai:acl.app	Description
appmaker:make_content_pack	App Maker: Make Content Pack	SA-Utils	Logs when exporting from Content Management into an app.

appmaker_make_on_prem.log

Sourcetype	Component	Eai:acl.app	Description
appmaker:make_on_prem	App Maker: Make On Prem	SA-Utils	Logs when downloading the distributed configuration management application "Splunk_TA_AROnPrem" in General Settings.

appmaker_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
appmaker:rest_handler	App Maker: REST Handler	SA-Utils	Logs export requests from the Content Management page, including the export package name as well as the download requests for exported packages.

apps_shc_es_deployer_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
apps_shc_es_deployer_rest_handler	SHC Installer: REST Handler	SplunkEnterpriseSecuritySuite	Persistent rest handler for managing apps on a search head cluster deployer.

configuration_check.log

Sourcetype	Component	Eai:acl.app	Description
configuration_check	Configuration Check: Modular Input	SA-Utils	Logs output messages of the confcheck migration scripts, such as when migration from correlationsearches.conf to savedsearches.conf fails.

contentinfo.log

Sourcetype	Component	Eai:acl.app	Description
contentinfo	ContentInfo: Search Command	SA-Utils	Logs the data sources referenced by contentinfo search-related objects.

contentinfo_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
contentinfo_rest_handler	ContentInfo: REST Handler	SA-Utils	Logs errors and successful operations to the contentinfo REST handler and associated components, as used mostly by the Use Case Library and Analytic Story pages.

correlationmigration_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
correlationsearches:migration_rest_handler	Correlation Migration: REST Handler	SA-ThreatIntelligence	Logs when migration from correlationsearches.conf to savedsearches.conf fails.

customsearchbuilder_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
customsearchbuilder:rest_handler	Custom Search Builder: REST Handler	SA-ThreatIntelligence	Logs when the search syntax of a correlation search, a lookup generating search, or an Assets and Identities LDAP search cannot be created or is incorrect.

data_migrator.log

Sourcetype	Component	Eai:acl.app	Description
data_migrator	Data Migrator: Modular Input	SA-Utils	Logs migration operations during ES upgrades. For example, when searches are executed as first-time run tasks or when a CSV lookup table is migrated to a KV store collection during an app upgrade.

datamodelsimple.log

Sourcetype	Component	Eai:acl.app	Description
datamodelsimple	Data Model Simple: Search Command	Splunk_SA_CIM	Logs when datamodelsimple starts and finishes processing in a search command.

entity_merge.log

Sourcetype	Component	Eai:acl.app	Description
identity_correlation:merge	Identity Correlation Merge: Search Command	SA-IdentityManagement	Logs the status of the search process during asset and identity merge.

es_investigations_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
es_investigations_rest_handler	ES Investigations Conf: REST Handler	SplunkEnterpriseSecuritySuite	Returns knowledge objects and handles change request for them, also enforces schemas and other stanza-specific prefixes and so on.

esconfighealth.log

Sourcetype	Component	Eai:acl.app	Description
esconfighealth	ES Configuration Health: Search Command	SplunkEnterpriseSecuritySuite	For installation and upgrade, logs the health of ES configurations against a manifest file that ships with each ES release. This typically logs as a result of running a config health check through the ES Configuration Health custom search command feature.

ess_configured_handler.log

Sourcetype	Component	Eai:acl.app	Description
ess_configured_handler	ES Configured: REST Handler	SplunkEnterpriseSecuritySuite	Logs current configured version state of search head cluster captains and search head cluster members for ES during setup and reset.

ess_content_importer.log

Sourcetype	Component	Eai:acl.app	Description
ess_content_importer	ES Content Importer: Modular Input	SplunkEnterpriseSecuritySuite	Logs when importing content from installed apps.

essinstaller2.log

Sourcetype	Component	Eai:acl.app	Description
essinstall2	ES Installer: Search Command	SplunkEnterpriseSecuritySuite	Logs installation status after setup completes.

event_sequencing_engine.log

Sourcetype	Component	Eai:acl.app	Description
event_sequencing_engine_log	Event Sequencing Engine: Search Command	SplunkEnterpriseSecuritySuite	Logs event sequencing engine operations such as terminate for sequence templates.

expectedactivity.log

Sourcetype	Component	Eai:acl.app	Description
expectedactivity	Expected Activity: Search Command	SA-Utils	Pertains to the Expected Activity custom search command. Logs when filling in gaps in results in preparation for use in statistical calculations. For example in stats, chart, or timechart.

governance_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
governance:rest_handler	Governance: REST Handler	SA-ThreatIntelligence	Logs when handling governance configurations and collections.

identdelete.log

Sourcetype	Component	Eai:acl.app	Description
identity_correlation:delete	Identity Correlation Delete: Search Command	SA-IdentityManagement	Logs when pruning identities marked for deletion from the assets_by_str, assets_by_cidr, or identities_expanded collections.

identity_correlation_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
identity_correlation:rest_handler	Identity Correlation: REST Handler	SA-IdentityManagement	Logs when creating, editing, validating, and deleting correlations for automatic lookups.

identity_manager.log

Sourcetype	Component	Eai:acl.app	Description
identity_correlation:modular_input	Identity Correlation: Modular Input	SA-IdentityManagement	Logs when asset and identity information is merged into Splunk asset and identity lookup tables.

identitymapper.log

Sourcetype	Component	Eai:acl.app	Description
identity_correlation:identitymapper	Identity Mapper: REST Handler	SA-IdentityManagement	Logs during reverse lookup searches for assets or identities.

investigation_handler.log

Sourcetype	Component	Eai:acl.app	Description
investigation_rest_handler	Investigation Workbench: REST Handler	SplunkEnterpriseSecuritySuite	Logs errors and such related to investigations, such as investigation data, entries, attachments, and cross-references to investigations from the Incident Review dashboard.

log_review_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
log_review_rest_handler	Log Review Conf: REST Handler	SA-ThreatIntelligence	Logs management information for REST changes made to log_review.conf, which is used by the Incident Review dashboard and Incident Review Settings page.

lookup_table_custom_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
lookup_table_custom_rest_handler	Lookup Table Custom: REST Handler	SA-Utils	Logs interactions with ES-managed csv lookups, including uploading new lookups through content management, as well as editing lookups in the lookup editor.

managed_lookups_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
managed_lookups_rest_handler	Managed Lookups: REST Handler	SA-Utils	Logs internal operations such as settings checks for managed lookups.

managed_nav_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
managed_nav_rest_handler	Managed Navigation: REST Handler	SA-Utils	Logs CRUD operations for the ES navigation menu, typically through the Navigation editor page.

modaction_adhoc_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
modaction:adhoc_rest_handler	Modular Action Adhoc: REST Handler	Splunk_SA_CIM	CIM: Adaptive Response actions execution. Logs when ad hoc searches result in adaptive response actions.

modaction_invocations_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
modaction:invocations_rest_handler	Modular Action Invocations: REST Handler	Splunk_SA_CIM	CIM: Adaptive Response actions execution

modaction_queue_handler.log

Sourcetype	Component	Eai:acl.app	Description
modaction:queue_handler	Modular Action Queue: REST Handler	Splunk_SA_CIM	Logs when handling the queue for Common Action Model properties.

notable_event_suppression.log

Sourcetype	Component	Eai:acl.app	Description
notable_event_suppression	Notable Event Suppression: Base	SA-ThreatIntelligence	Logs when managing notable event suppressions.

notable_event_suppression_autoDisable.log

Sourcetype	Component	Eai:acl.app	Description
notable_event_suppression:autoDisable	Notable Event Suppression: Auto Disable	SA-ThreatIntelligence	Logs on auto-disable for notable event suppressions of Adhoc Risk Events.

notable_update_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
notable_update_rest_handler	Notable Event Update: REST Handler	SA-ThreatIntelligence	Logs when changing notable events in Incident Review.

outputcheckpoint.log

Sourcetype	Component	Eai:acl.app	Description
outputcheckpoint	Output Checkpoint: Search Command	SA-Utils	Logs when outputting the results of the previous search pipeline to a modular input checkpoint directory.

per_panel_filtering.log

Sourcetype	Component	Eai:acl.app	Description
per_panel_filtering	Per Panel Filtering	SA-Utils	Logs per panel filtering changes.

relaymodaction.log

Sourcetype	Component	Eai:acl.app	Description
relaymodaction	Modular Action Relay: Modular Input	Splunk_SA_CIM	Logs when managing remote Splunk instance modular actions.

reviewstatuses_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
reviewstatuses:rest_handler	Reviewstatuses: REST Handler	SA-ThreatIntelligence	Logs when handling knowledge objects for configuring notable statuses and investigation statuses.

sequence_instance_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
sequence_instance_rest_handler	Sequence Instance: REST Handler	SplunkEnterpriseSecuritySuite	Logs when handling an instance of a running sequenced event.

sequence_templates_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
sequence_templates_rest_handler	Sequence Templates: REST Handler	SplunkEnterpriseSecuritySuite	Logs when making CRUD operations to the configuration of sequence templates.

sorttimecols.log

Sourcetype	Component	Eai:acl.app	Description
sorttimecols	Sort Time Columns: Search Command	SA-Utils	Pertains to the sorttimecols custom search command. Logs when using the sorttimecols commands to sort columns in a result set by time.

suppressions_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
notable_event_suppression:rest_handler	Notable Event Suppression: REST Handler	SA-ThreatIntelligence	REST handler for notable suppression create and edit. For use in conjunction with the notable_event_suppression.log file.

threat_intel_file_upload_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
threatintel:file_upload_rest_handler	Threat Intel Upload: REST Handler	DA-ESS-ThreatIntelligence	rest handler for uploading threat intelligence files

threat_intelligence_manager.log

Sourcetype	Component	Eai:acl.app	Description
threatintel:manager	Threat Intel Manager: Modular Input	DA-ESS-ThreatIntelligence	Logs when the modular input parses the threat sources and updates the KV Store threat collections with any new intelligence.

threat_intelligence_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
threatintel:rest_handler	Threat Intel: REST Handler	DA-ESS-ThreatIntelligence	Logs activity of threat intel endpoints.

threatlist.log

Sourcetype	Component	Eai:acl.app	Description
threatintel:download	Intelligence Download: Modular Input	SA-ThreatIntelligence	Logs the status of threat intel downloads, including success and failure.

transitioners_rest_handler.log

Sourcetype	Component	Eai:acl.app	Description
transitioners_rest_handler	Transitioners: REST Handler	SA-ThreatIntelligence	notable status handler, checking permission who can change status, also migrates from authorize.conf to reviewstatuses.conf.

uba_rest.log

Sourcetype	Component	Eai:acl.app	Description
uba:rest_handler	UBA: REST Handler	SA-UEBA	Pertains to the UBA Integration rest handler.

whois_manager.log

Sourcetype	Component	Eai:acl.app	Description
whois_manager	Whois Manager: Modular Input	SA-NetworkProtection	Logs when executing the whois modular input data.

Use search to check for activity

You can use search to check for errors and activity. The majority of sourcetypes can be searched in the _internal index. The notable_update_rest_handler can also be searched for as a source in the _audit index.

Searching the _internal index for notable_update_rest_handler will show you, for example, what happens during the handler review process. Example search:

index=_internal sourcetype="notable_update_rest_handler"

Example response:

i	Time	Event
>	12/2/19 3:07:16.525 PM	2019-12-02 20:07:16,525+0000 INFO pid=8649 tid=MainThread file=rest_handler.py:handle:728 NotableEventUpdate.handle_post duration=4.474 host = hostname = /usr/local/bamboo/splunk-install/current/var/log/splunk/notable_update_rest_handler.log sourcetype = notable_update_rest_handler
>	12/2/19 3:07:16.524 PM	2019-12-02 20:07:16,524+0000 INFO pid=8649 tid=MainThread file=notable_update_rest_handler.py:setStatuses:957 Done editing events matching search admin__admin__SplunkEnterpriseSecuritySuite__RMD57f02abc0263583b0 _1575317218.11939 host = hostname = /usr/local/bamboo/splunk-install/current/var/log/splunk/notable_update_rest_handler.log sourcetype = notable_update_rest_handler
>	12/2/19 3:07:16.524 PM	2019-12-02 20:07:16,524+0000 INFO pid=8649 tid=MainThread file=cim_actions.py:message:425 I sendmodaction - worker="soln-esnightly1" signature="Successfully created splunk events" action_name="notable_event_edit" digest_mode="1" action_mode="adhoc" event_count="1" host = hostname source = /usr/local/bamboo/splunk-install/current/var/log/splunk/notable_update_rest_handler.log sourcetype = notable_update_rest_handler

Searching the _audit index for the source of notable_update_rest_handler will show you, for example, what was saved to the KV Store during the handler processing. This is not necessarily for troubleshooting, but more specific to incident review activity.

Example search:

index=_audit sourcetype="incident_review"

Example response:

i	Time	Event
>	12/2/19 3:07:13.090 PM	1575317233.09,19E67472-762C-4636-9A91-E4CF6B4BD885@@notable@@15c339addb8d09e6d8a24176beafd9792bd84f45,Host With Multiple Infections,4,esadmin,high,comment,admin,True host = hostname source = notable_update_rest_handler sourcetype = incident_review