Known Issues

The following are known issues and workarounds for this version of the Splunk App for Enterprise Security.

Highlighted issues

• On Splunk Enterprise version 6.0.1, Splunk may begin skipping searches when "indexed real-time" searching is enabled. This can cause dashboards to appear empty and data models to be incomplete. To resolve this issue, upgrade to Splunk Enterprise 6.0.2 or obtain the patch for Splunk Enterprise 6.0.1 from Splunk Support. (SOLNESS-4504)

• On Splunk Enterprise version 6.1 and later, a Windows server can experience a crash when using INDEXED_EXTRACTIONS on introspection logs. (SPL-83975) (SOLNESS-5245)

Workaround:

Modify the content in $SPLUNK_HOME/etc/apps/introspection_generator_addon/local/props.conf and override the value for INDEXED_EXTRACTIONS on all Windows search head and indexer instances.

[splunk_disk_objects]

INDEXED_EXTRACTIONS =

[splunk_resource_usage]

INDEXED_EXTRACTIONS =

Restart Splunk Enterprise.

• Enterprise Security implements data model acceleration. The data model acceleration process will automatically run backfill searches up to the default retention period set. The default retention for a data model can be up to 1 year. While the backfill process is running, the search head and indexers will experience very high load and the Enterprise Security dashboards may not populate (SOLNESS-4644) (SPL-73529). The workaround is to decrease the retention period in each data model used by Enterprise Security until a maintenance window is available. During the maintenance window, the data model retention should be set back to an appropriate value for your organization and the backfill process allowed to complete. Every Enterprise Security implementation has different data volumes and resources available. There is no estimate for the time it will take to complete the backfill process for every data model.

Example: After installing Enterprise Security, go to Settings > Data Models and isolate the Data Models to the Enterprise Security app. In each active data model marked by a yellow lightning bolt, edit the Acceleration and lower the Summary Range to 1 month or less. Allow the backfill processes to complete by checking the data model Status. Repeat the process, lowering the Summary Range on all active data models until complete.

When a maintenance window is available, return to to Settings > Data Models and isolate the Data Models to view those in Enterprise Security app. Begin with one active data model. Edit the Acceleration and raise the Summary Range to an appropriate value for your organization or back to the default. Allow the backfill processes to complete by checking the data model Status. Splunk Enterprise will experience a large increase in search load across the search head and indexers until the backfill process is complete. Repeat the process, changing the Summary Range on all active data models

Enterprise Security Data Models default retention

Data Model	Summary Range	Data Model	Summary Range
Application State	1 month	Authentication	1 year
Change Analysis	1 year	Domain Analysis	3 months
Incident Management	All Time	Intrusion Detection	1 year
Malware	1 year	Network Sessions	3 months
Network Traffic	3 months	Performance	1 month
Splunk Audit Logs	1 year	Updates	1 year
Vulnerabilities	1 year	Web	3 months

Hardware prerequisites

Note: See "Prerequisites" in the Splunk App for Enterprise Security Installation and Configuration Manual for specific hardware requirements information.

• The Splunk App for Enterprise Security may not run on virtualized machines with insufficient hardware. (SOLNESS-1118)

• Running Windows on under-provisioned virtualized hardware may cause Enterprise Security setup to fail. If you have well-provisioned physical hardware, retry the setup if it fails the first time. (SOLNESS-4256)

Install / Upgrade

• Enterprise Security changes the default settings for real-time indexing. The setting is global and applied to all other apps on the same search head. (SPL-76910) (SOLNESS-4435)

• Large lookups fail in a distributed environment. With default settings, any lookup > 10MB will create an index (.tsidx) alongside the lookup file. (SPL-74438)

• Installing the Splunk App for Enterprise Security causes real-time searches to use the backfill feature in Splunk. (SOLNESS-831)

• With Windows, the Enterprise Security Install App reports false positives for modified default files during an upgrade. After upgrading, check your modified files to verify if they have been customized. (SOLNESS-3141)

• Enterprise Security implements data model acceleration. Data model acceleration defaults to using the $SPLUNK_DB path on the indexers for storage. Review the indexes.conf on the indexers and verify the tstatsHomePath setting to synchronize the storage for accelerated data models and indexes. If the indexes use storage volumes that are not referenced as $SPLUNK_DB, the configuration for accelerated data model storage should be changed to match the storage used by the indexes.

• Enterprise Security implements data model acceleration. The data model acceleration process will automatically run backfill searches up to the defined retention period. When Splunk Enterprise services are restarted, the backfill searches may not be cancelled automatically and would become orphaned processes. An orphaned processes can occur on either the indexers or search heads. The symptoms are seen when the data model backfill is attempting to meet a very long retention requirement during a restart of Splunk Enterprise services. (SOLNESS-4644)

Browsers

• The cache in the Chrome browser prevents some panels in the Enterprise Security Install App from expanding. To workaround, refresh the browser cache. (SOLNESS-2939)

Incident Review

• Contributing events from any notable event in the Incident Review dashboard will default to "All Time" and may take a long time to return results. To workaround this issue, cancel the search and rerun with the desired time window. (SOLNESS-1784)

• The Incident Review dashboard feature does not work on the Solaris operating system. (SOLNESS-2508)

• When viewing the Incident Review dashboard using Internet Explorer 9, if you finalize the search, the word "events" ("Edit all _ matching events.") is wrapped to the next line. The workaround is to increase or decrease the page size. (SOLNPCI-1038)

• A user attempting a notable event suppression against the Application State data model tag=service object will fail. (SOLNESS-4580)

Configuration

• Clicking on a configuration item in the App Settings page takes the user to the Search Macros Manager page. The Cancel button does not work. The Save button takes the user to the list of macros in the Search Macros Manager page instead of back to the App Settings Configuration page. (SOLNPCI-375)

Dashboards

• When working with individual Reports (Search > Reports), some drill down functionality may not produce desired behavior. This is dependent on the structure of the search, and the search commands being used. This should not affect shipped dashboards. If adding a report to ones own dashboard, for best results use Simple XML to define explicit drill down. (SOLNESS-4387)

• When drilling down from the Traffic Center dashboard to the Traffic Search dashboard, the specified time window is passed via the URL. If you change the time window, the URL is not updated, which forces the dashboard to run with the same parameters submitted during the previous drill-down. To workaround, click on Network > Traffic Search to reset the URL. The time picker will resume normal expected behavior. (SOLNESS-2827)

• Some summary index data and lookup table data in the Splunk App for Enterprise Security is generated using a custom post-processing mechanism, which permits multiple searches to be executed as a single alert action, reducing overall search load. Post-processing is controlled by "postprocess.conf" configuration file(s).

We generally recommended that you not edit these files without the involvement of Splunk Support. However, if you do find it necessary to edit a "postprocess.conf" file on the filesystem, a refresh of the postprocess REST endpoint is required for the change take effect. This can be done one of two ways:

1. By issuing a refresh request using curl, wget, or the browser to one of the following URLs:

          https://<splunk_server_ip>:8089/en-US/debug/refresh?entity=saved/postprocess 
          https://<splunk_server_ip>:8000/en-US/debug/refresh?entity=saved/postprocess 

2. Issuing a Splunk restart using any method. (SOLNPCI-868)

• Enterprise Security implements data model acceleration. If the data models included in Enterprise Security are modified to include additional sources or sourcetypes, the backfill job will begin. While the backfill process is running, the search head and indexers will experience very high load and the Enterprise Security dashboards may not populate. (SPL-81167)

• When adding a custom created key indicator to a dashboard through the dashboard UI, the indicator panel will not stay pinned to the dashboard after navigating away to another view. Creating a custom key indicator requires direct editing of the savedsearches.conf file. The custom key indicator stanza in savedsearches.conf must include the following settings:

   action.keyindicator.group.0.name =

   action.keyindicator.group.0.order =

After adding the settings to the key indicator stanza, a custom indicator UI panel can be added to a dashboard and will persist. (SOLNESS-5001)

• Changing the title of an existing entity investigator or swimlane search will break all swimlane searches used on the same dashboard. Changing the title of the search back to the default will fix the display issue. (SOLNESS-5194)

Reports

• In any Individual Reports window, selecting a real-time Time Range such as: 24 hour window, 30 minute window, etc. will cause a display error: (SOLNESS-3536)

Error in 'tstats' command: This command is not supported in a real-time search.

Workaround:

Use a relative "Time Range" such as: Last 24 hours or Last 15 minutes.

• When adding a report to a custom dashboard in the Enterprise Security app, the report's drilldown search may not produce the desired behavior. This includes pre-defined reports included with the Enterprise Security app. (SOLNESS-4387)

Workaround:

Test all report drilldown behaviors on custom dashboards, and use Simple XML to define the drilldown search for each report as desired.

Search

• A notable event created from a search shows object details in the Original Event field in raw events. This is caused by an underlying bug. (SOLNESS-4470)

Logs

• The tsidx_clean_up.log may display "Error generated when running TSIDX clean up" and "TypeError: can't compare datetime.datetime to NoneType". (SOLNESS-4589)

Inputs

• A modular input script on Windows may report "A script exited abnormally" input="path\to\file\Splunk\bin\splunk-perfmon.exe" stanza="default" status="exited with code -1". The error is occurring because the scripted inputs included with the Splunk Technology Add-on for Windows use non-zero exit codes even when they exit successfully. The error is benign and can be ignored. (SOLNESS-4629)

• TA-mcafee uses python scripts to collect McAfee EPO data. The script mcafee_epo.py has a dependency to the python bundled with Splunk Enterprise that prevents it from running on other python installations. (ADDON-894)

• The implementation of the WHOIS modular input used on the New Domain Analysis dashboard is inefficient for large deployments. The methodology used to identify and parse top level domains from URLs can place excessive requests for information from the WHOIS provider. A workaround is available to reduce the WHOIS query volume until a solution is delivered. Please contact Splunk Support for the patch and reference SOLNESS-4554. (SOLNESS-4554)