Known Issues

The following are known issues and workarounds for this version of the Splunk App for Enterprise Security.

Highlighted issues

• Always review the latest Known Issues topic in the Splunk Enterprise Release Notes.

• On Splunk Enterprise version 6.1 and later, a Windows server can experience a crash when using INDEXED_EXTRACTIONS on introspection logs. (SPL-83975) (SOLNESS-5245)

Workaround:

Modify the content in $SPLUNK_HOME/etc/apps/introspection_generator_addon/local/props.conf and override the value for INDEXED_EXTRACTIONS on all Windows search head and indexer instances.

[splunk_disk_objects]

INDEXED_EXTRACTIONS =

[splunk_resource_usage]

INDEXED_EXTRACTIONS =

Restart Splunk Enterprise.

• Users who install or upgrade TA-Bro will find that the add-on does not work by default. The command line will display warnings when you start Splunk Enterprise. This issue is due to a known problem in Splunk Enterprise and will be fixed in a future release. (ADDON-1104)

Workaround

For a new installation:

1. After installing TA-Bro, rename the $SPLUNK_HOME/etc/apps/Splunk_TA_bro/SPEC directory to $SPLUNK_HOME/etc/apps/Splunk_TA_bro/README.

2. Restart Splunk Enterprise.

For an upgrade:

1. Stop Splunk Enterprise.

2. Remove the existing README file, if it exists: $SPLUNK_HOME/etc/apps/Splunk_TA_bro/README).

3. Rename the $SPLUNK_HOME/etc/apps/Splunk_TA_bro/SPEC directory to $SPLUNK_HOME/etc/apps/Splunk_TA_bro/README.

4. Restart Splunk Enterprise.

• Enterprise Security implements data model acceleration. The data model acceleration process will automatically run backfill searches up to the default retention period set. The default retention for a data model can be up to 1 year. While the backfill process is running, the search head and indexers will experience very high load and the Enterprise Security dashboards may not populate (SOLNESS-4644) (SPL-73529). The workaround is to decrease the retention period in each data model used by Enterprise Security until a maintenance window is available. During the maintenance window, the data model retention should be set back to an appropriate value for your organization and the backfill process allowed to complete. Every Enterprise Security implementation has different data volumes and resources available. There is no estimate for the time it will take to complete the backfill process for every data model.

Example: After installing Enterprise Security, go to Settings > Data Models and isolate the Data Models to the Enterprise Security app. In each active data model marked by a yellow lightning bolt, edit the Acceleration and lower the Summary Range to 1 month or less. Allow the backfill processes to complete by checking the data model Status. Repeat the process, lowering the Summary Range on all active data models until complete.

When a maintenance window is available, return to to Settings > Data Models and isolate the Data Models to view those in Enterprise Security app. Begin with one active data model. Edit the Acceleration and raise the Summary Range to an appropriate value for your organization or back to the default. Allow the backfill processes to complete by checking the data model Status. Splunk Enterprise will experience a large increase in search load across the search head and indexers until the backfill process is complete. Repeat the process, changing the Summary Range on all active data models

Enterprise Security Data Models default retention

Data Model	Summary Range	Data Model	Summary Range
Application State	1 month	Authentication	1 year
Change Analysis	1 year	Domain Analysis	3 months
Incident Management	All Time	Intrusion Detection	1 year
Malware	1 year	Network Sessions	3 months
Network Traffic	3 months	Performance	1 month
Splunk Audit Logs	1 year	Updates	1 year
Vulnerabilities	1 year	Web	3 months

Hardware prerequisites

Note: See "Prerequisites" in the Splunk App for Enterprise Security Installation and Configuration Manual for specific hardware requirements information.

• The Splunk App for Enterprise Security may not run on virtualized machines with insufficient hardware. (SOLNESS-1118)

• Running Splunk Enterprise on a combination of virtualized hardware and Windows Server may cause the Enterprise Security setup to fail. If the virtualized system is properly provisioned to Splunk Enterprise specifications and setup is unable to complete, increase the splunkdConnectionTimeout setting in the web.conf to 120 seconds or more until the setup process is complete. (SOLNESS-4256) (SPL-82837)

Install / Upgrade

• Enterprise Security changes the default settings for real-time searching to utilize the indexed real-time option in Splunk Enterprise. The setting is global and will be applied to all searches running on the same search head. (SPL-76910) (SOLNESS-4435)

• Large lookups fail in a distributed environment. With default settings, any lookup > 10MB will create an index (.tsidx) alongside the lookup file. (SPL-74438)

• Installing the Splunk App for Enterprise Security causes real-time searches to use the backfill feature in Splunk. (SOLNESS-831)

• With Windows, the Enterprise Security Install App reports false positives for modified default files during an upgrade. After upgrading, check your modified files to verify if they have been customized. (SOLNESS-3141)

• Enterprise Security implements data model acceleration. Data model acceleration defaults to using the $SPLUNK_DB path on the indexers for data model storage. The data models will write data to the default $SPLUNK_DB path on the indexers until the tstatsHomePath is changed. Review the indexes.conf on the indexers, and verify the tstatsHomePath setting to synchronize the storage for accelerated data models and indexes.

• Enterprise Security implements data model acceleration. The data model acceleration process will automatically run backfill searches up to the defined retention period. When Splunk Enterprise services are restarted, the backfill searches may not be cancelled automatically and would become orphaned processes. An orphaned processes can occur on either the indexers or search heads. The symptoms are seen when the data model backfill is attempting to meet a very long retention requirement during a restart of Splunk Enterprise services. (SOLNESS-4644)

• The Identity Manager modular input may not re-execute if the previous run failed. This condition can occur during the scripted setup, or during a rapid sequence of manual restarts of the splunkd service. (SOLNESS-4555)

Browsers

• The cache in the Chrome browser prevents some panels in the Enterprise Security Install App from expanding. To workaround, refresh the browser cache. (SOLNESS-2939)

Incident Review

• Contributing events from any notable event in the Incident Review dashboard will default to "All Time" and may take a long time to return results. To workaround this issue, cancel the search and rerun with the desired time window. (SOLNESS-1784)

• The Incident Review dashboard feature does not work on the Solaris operating system. (SOLNESS-2508)

• When viewing the Incident Review dashboard using Internet Explorer 9, if you finalize the search, the word "events" ("Edit all _ matching events.") is wrapped to the next line. The workaround is to increase or decrease the page size. (SOLNPCI-1038)

Configuration

• Clicking on a configuration item in the App Settings page takes the user to the Search Macros Manager page. The Cancel button does not work. The Save button takes the user to the list of macros in the Search Macros Manager page instead of back to the App Settings Configuration page. (SOLNPCI-375)

• When the Splunk App for Enterprise Security is running on Splunk Enterprise 6.x with FIPS compatibility enabled, using the Correlation Search Configuration view will cause Splunk Enterprise to crash. Please contact Splunk Support for a patch and reference SOLNESS-4616. (SOLNESS-4616)

Dashboards

• When working with individual Reports (Search > Reports), some drill down functionality may not produce desired behavior. This is dependent on the structure of the search, and the search commands being used. This should not affect shipped dashboards. If adding a report to ones own dashboard, for best results use Simple XML to define explicit drill down. (SOLNESS-4387)

• When drilling down from the Traffic Center dashboard to the Traffic Search dashboard, the specified time window is passed via the URL. If you change the time window, the URL is not updated, which forces the dashboard to run with the same parameters submitted during the previous drill-down. To workaround, click on Network > Traffic Search to reset the URL. The time picker will resume normal expected behavior. (SOLNESS-2827)

• Some summary index data and lookup table data in the Splunk App for Enterprise Security is generated using a custom post-processing mechanism, which permits multiple searches to be executed as a single alert action, reducing overall search load. Post-processing is controlled by "postprocess.conf" configuration file(s).

We generally recommended that you not edit these files without the involvement of Splunk Support. However, if you do find it necessary to edit a "postprocess.conf" file on the filesystem, a refresh of the postprocess REST endpoint is required for the change take effect. This can be done one of two ways:

1. By issuing a refresh request using curl, wget, or the browser to one of the following URLs:

          https://<splunk_server_ip>:8089/en-US/debug/refresh?entity=saved/postprocess 
          https://<splunk_server_ip>:8000/en-US/debug/refresh?entity=saved/postprocess 

2. Issuing a Splunk restart using any method. (SOLNPCI-868)

• Enterprise Security implements data model acceleration. If the data models included in Enterprise Security are modified to include additional sources or sourcetypes, the backfill job will begin. While the backfill process is running, the search head and indexers will experience very high load and the Enterprise Security dashboards may not populate. (SPL-81167)

• Threat lists containing CIDR notation are being rejected by the threatlist framework. Single IP notation and ranges work properly. Please contact Splunk Support for a patch and reference SOLNESS-4819. (SOLNESS-4819)

• On a Windows search head, the asset and identity center shows no results. Error messages will be displayed on the search head about missing lookup files. The python_modular_inputs.log reports errors:

      ERROR pid=4040 tid=asset file=writers.py:_move_lookup:108 | FAILURE: Temporary output file was not created: C:\Program Files\Splunk\var\run\splunk\lookup_tmp\lookup_conv6jppog.txt
      ERROR pid=4040 tid=asset file=writers.py:move_lookups:156 | FAILURE: A lookup table could not be created: (key: cidr, tempfile: C:\Program Files\Splunk\var\run\splunk\lookup_tmp\lookup_conv6jppog.txt)

The asset and identity lookup creation and expansion process is not working correctly due to an issue with a python script on Windows. Please contact Splunk Support for a replacement script and reference SOLNESS-4642. (SOLNESS-4642)

Once the script is obtained, follow the instructions below:

1. Replace the writers.py script in $SPLUNK_HOME\etc\apps\SA-Utils\lib\SolnCommon\lookup_conversion

2. Make sure all the *.csv's in SA-IdentityManagement\lookups are there, and if not create a new copy from the *.csv.default files.

3. Delete all the contents under $SPLUNK_HOME\var\lib\splunk\modinputs\identity_manager

4. Restart Splunk Enterprise

• Attempts to move external lookup files into the Asset and Identities Management system will generate an error in splunkd.log and in python_modular_input.log:

ExecProcessor - message from "python splunk/etc/apps/SA-ThreatIntelligence/bin/threatlist.py" OSError: [Errno 2] No such file or directory

ExecProcessor-messagefrom "splunk/etc/apps/SA-ThreatIntelligence/bin/threatlist.py" OSError: [Errno18] Invalid cross-devicelink

ERROR tid=MainThread file=lookup_modinput.py:collect_files:141 | status="Checkpoint file error" err="unknown path or update time"

The Asset and Identities Management system is implemented using a modular input. If the Splunk instance was configured to utilize a $SPLUNK_DB path that resides outside of /opt/splunk, the modular input fails when trying to perform file management.

Please contact Splunk Support for a patch and reference SOLNESS-4830. (SOLNESS-4830)

Once the patch is obtained, follow the instructions below:

1. Stop Splunk services.

2. Unzip the splunk_app_es-3.0.1.2.zip file into the $SPLUNK_HOME/etc/apps directory, overwriting the files: app.conf and lookups.py in the SA-Utils app.

• When using Advanced Threat dashboards, some dashboard views show a yellow warning sign triangle even if the view displays results. The warning reports:

Empty csv lookup file (contains only a header) for table 'ppf_http_category': /splunk/etc/apps/DA-ESS-NetworkProtection/lookups/ppf_http_user_agent.csv

Empty csv lookup file (contains only a header) for table 'ppf_url_length': /splunk/etc/apps/DA-ESS-NetworkProtection/lookups/ppf_new_domains.csv

This is expected behavior and is harmless. The lookup files referenced in the warning message manages the per-panel filtering feature in Enterprise Security. Per-panel filtering is used to filter or whitelist items out of dashboard views that are deemed unimportant or non-threatening.

Until the per-panel filter lookup is used, the file is empty and contains only a header. This status does not affect the functioning of the dashboard panel.

See Per-Panel Filter Audit in the Enterprise Security Installation and Configuration Manual for more information. (SOLNESS-4631)

• When adding a custom created key indicator to a dashboard through the dashboard UI, the indicator panel will not stay pinned to the dashboard after navigating away to another view. Creating a custom key indicator requires direct editing of the savedsearches.conf file. The custom key indicator stanza in savedsearches.conf must include the following settings:

   action.keyindicator.group.0.name =

   action.keyindicator.group.0.order =

After adding the settings to the key indicator stanza, a custom indicator UI panel can be added to a dashboard and will persist. (SOLNESS-5001)

• Changing the title of an existing entity investigator or swimlane search will break all swimlane searches used on the same dashboard. Changing the title of the search back to the default will fix the display issue. (SOLNESS-5194)

Reports

• In any Individual Reports window, selecting a real-time Time Range such as: 24 hour window, 30 minute window, etc. will cause a display error: (SOLNESS-3536)

Error in 'tstats' command: This command is not supported in a real-time search.

Workaround:

Use a relative "Time Range" such as: Last 24 hours or Last 15 minutes.

• When adding a report to a custom dashboard in the Enterprise Security app, the report's drilldown search may not produce the desired behavior. This includes pre-defined reports included with the Enterprise Security app. (SOLNESS-4387)

Workaround:

Test all report drilldown behaviors on custom dashboards, and use Simple XML to define the drilldown search for each report as desired.

Search

• A notable event created from a search shows object details in the Original Event field in raw events. This is fixed with Splunk Enterprise 6.0.2. (SOLNESS-4470)

Inputs

• A modular input script on Windows may report "A script exited abnormally" input="path\to\file\Splunk\bin\splunk-perfmon.exe" stanza="default" status="exited with code -1". The error is occurring because the scripted inputs included with the Splunk Technology Add-on for Windows use non-zero exit codes even when they exit successfully. The error is benign and can be ignored. (SOLNESS-4629)

• TA-mcafee uses python scripts to collect McAfee EPO data. The script mcafee_epo.py has a dependency to the python bundled with Splunk Enterprise that prevents it from running on other python installations. (ADDON-894)

• The implementation of the WHOIS modular input used on the New Domain Analysis dashboard is inefficient for large deployments. The methodology used to identify and parse top level domains from URLs can place excessive requests for information from the WHOIS provider. A workaround is available to reduce the WHOIS query volume until a solution is delivered. Please contact Splunk Support for the patch and reference SOLNESS-4554. (SOLNESS-4554)

• The threat list emerging_threats_malvertisers_blocklist has been obsoleted. The threat list input should be disabled. (SOLNESS-4785)