Known issues

Known issues

The following are issues and workarounds for version 4.0.9 of Splunk.

Epoch timestamps not parsed correctly after March 12, 2011

This problem affects ALL Splunk versions: 3.x, 4.x, and 4.2.

In Splunk's datetime.xml, the regular expression for parsing epoch time assumes values from 2002 through to March 12th 2011. Those values started with 10,11,12. On March 12th, 2011, the seconds since 1970 became 1300000000, starting with 13.

First, make a backup copy of $SPLUNK_HOME/etc/datetime.xml, and then modify it. Change the _utcepoch regex (at around line 200) to the following:

<define name="_utcepoch" extract="utcepoch, subsecond">
    <!-- update regex before 2017! :) -->
    <text><![CDATA[((?<=^|[\s#,"=([\|{])(?:1[012345]|9)\d{8}|^@[\da-fA-F]{16,24})(?:.?(\d{1,6}))?(?![\d(])]]></text>
</define>

Alternatively, for your sources that use epoch time, explicitly specify a strptime format in props.conf, by using TIME_FORMAT and TIME_PREFIX fields.

Example:

[asterisk]
TIME_FORMAT = %s

Security issues

This version of Splunk contains several security flaws described on this page in the Splunk Security Portal. Splunk strongly recommends that you upgrade to 4.0.11 (or 4.1.2 or later) as soon as possible.

General issues

• License violation warning message may not be displayed when a license violation occurs. (SPL-29454)
• When you save a top or rare search with the argument showperc, the showperc argument disappears when you run the search. (SPL-27694)
• On shutdown, many WARN lines are displayed in splunk.log that should actually be INFO. These lines can be safely ignored. (SPL-24862)
• You must manually distribute certificates to a host before you can successfully add it as a distributed search peer using the CLI. (SPL-24786)
• If you expand the view of a large event to the full event and back again to the summary view, subsequent attempts to expand to view the entire event will be restricted to 500 lines. (SPL-27109)
• web_access.log and web_service.log grow forever, and consume unbounded disk space. (SPL-27588)
• Summary indexing does not work if var/run/splunk and var/spool/splunk are on different filesystems. (SPL-26631)
• The SplunkLightForwarder app *requires* an outputs.conf-style choice of server to forward to. If SplunkLightForwarder is configured on while no target server to transmit data to is specified, the Splunk instance will not forward the data, nor block, it will null-route the dataflow. (SPL-22247)
• Splunk search is limited to lists of OR terms around 415 long, eg "1 OR 2 OR 3.... OR 415". If more cases than this are needed, a lookup may be an effective workaround. (SPL-28301)
• CLI help is missing some commands (SPL-28396)
• The CLI resurrect command for restoring frozen data to the index does not work for 4.x. See http://www.splunk.com/base/Community:MoveIndexes for considerations for manually moving buckets. (SPL-26202)
• Deploying apps that do not contain a local directory will cause Splunk to crash on the client. (SPL-29019)
• CSV files used for field lookups cannot use ^M line terminators. No error is generated. CSV files generated on MacOS version 9 have this problem. (SPL-29434)
• Indexing performance can be impacted if you are running anti-virus scans on Splunk's index directories. This is caused primarily by the additional CPU and disk overhead that the anti-virus accessing these files at the same time as Splunk.
• Events generated by the internal auditing feature, which creates events for user-actions such as fired searches are undesirably counted against the license. In a distributed environment this can be worked around by configuring the search head as a forwarder and adding the following stanza to your inputs.conf
  

[fschange:$SPLUNK_HOME/etc]
index = _audit
_TCP_ROUTING = *
Typically this is not a problem because the volume is so tiny.(SPL-28462)

• Scripted auth scripts and search scripts in perl do not work. (SPL-28532)

Data input issues

• monitor inputs using the followTail setting sometimes will index some older events or all events from log files which are updated when not intended. (SPL-23555)
• Splunk generates misleading warnings for plain text logfiles, eg Using charset UTF-8 for events from '....', as the monitor is believed over the raw text which may be ASCII. ASCII is a proper subset of UTF8 so this is a non-problem. (SPL-27498)
• When configuring a TCP input the host= setting is not respected. If you want the host value to be something other than the sending server you will need to modify the host value through props.conf and transform.conf. (SPL-27735)
• Fschange used with recurse=true and whitelist+blacklist filter results in blacklisting of all files in subdirectories despite whitelisting. Workaround is to disable recurse and create a separate fschange stanza for each directory. (SPL-29335)

Localization, internationalization, and character set issues

• Certain Japanese language OSes, including most versions of Windows, use the ¥ (Yen) symbol to denote backslashes in path names. This can cause issues when monitoring or spooling files, and may require custom regex configurations where a file path is part of the dataset. (SPL-23307)

(This issue is also present in the Japanese PDFs of the documentation.)

• The default locale settings on OS X 10.6 (Snow Leopard) trigger an error in Splunk Web. To work around, unset LC_CTYPE, eg 'unset LC_CTYPE; splunk start' (SPL-28896)

App and App development issues

• An issue exists in the first time run experience around input collisions: if you enable the *Nix App, the inputs it adds put their data in the "os" index, which by default is only searchable from the *Nix App interface. If you then try to add /var/log as an input (through the Getting Started App or any other App), an error is displayed stating that this input already exists. (SPL-25138)
• It's possible to get to the setup page for an App without enabling it first. (SPL-24852)
• No dashboards are added to the navigation menus for the Windows and *Nix Apps. (SPL-24933)
• Old modules, templates, and other App components are not deleted on upgrade. (SPL-22494)
• The *Nix App is not supported on AIX. (ENH-3001)
• Simple XML searchPostProcess doesn't work with <chart> and <fields>. (SPL-27248)
• Deploying apps that do not contain a local directory will cause Splunk to crash on the client. (SPL-29019)
• in custom form search views, Lister modules like SearchSelectLister can not be configured to run their internal searches over the time range selected in a TimeRangePicker. (SPL-31706)

Search and search App issues

• Creating an invalid event type does not generate an error. (SPL-25091)
• The All indexed data dashboard count for number of sources stops incrementing at 10,000 sources. (SPL-27300)
• For events which contain literal asterisks, there are some search irregularities. An event that contains "*foo*bar*" can be found with a search for 'foo', but cannot be found with 'sourcetype=thesourcetype foo'. A workaround is 'sourcetype=thesourcetype *foo*" (SPL-28232)
• Recovery from hitting srchDiskQuota limit or max concurrent searches is not graceful. Splunk must be restarted in order for scheduled searches to resume. (SPL-28999)

Alerting and scheduled search issues

• Custom alert scripts that do not complete will stall further scheduled searches. Be sure your alert scripts will complete promptly. (SPL-28421)
• Installing the Splunk for Bluecoat app v 1.0 dated December 9 may break other scheduled search functionality by flooding the scheduler. This app version unintentionally contains a large number of frequently scheduled saved searches.
• Quotes in saved searches are incorrectly being escaped by an internal sub-system producing an incorrect search and returning zero results. (SPL-28734)
• Scheduled summary searches that hit disk quota OR hit max allowed concurrent searches do not recover without a restart even if the condition that caused the failure is cleared. (SPL-28999)
• Searches using the parameter 'use starthoursago= # ' produce a correct result set but a misleading time range message a the bottom. (SPL-30250)

Splunk Web and Manager display issues

• The number of users to display per page in Manager > Users does not retain its state if you change it. (SPL-24896)
• Pausing a search job in the job manager does not update the job's displayed status (SPL-24999)
• If you have cookies disabled or if the server and/or client CPU time are not in sync, you will be returned to the login page. Both machines must have the correct time set when cookie timestamp is verified. (SPL-22393)

Windows-specific issues

• The crawl feature is not applicable on Windows. (SPL-24843)
• The Windows App was enabled by default in its app.conf file in versions 4.0-4.0.2. Starting in version 4.0.3, it is disabled in this file by default. (SPL-25487) Read on for important details:
  • If you're upgrading from 4.0-4.0.2 to 4.0.3 or later, the Windows App will be disabled, even if it was enabled in the version you're upgrading from.
  • If you're doing a fresh installation of 4.0.3 or later, the Windows App is enabled by default via the MSI and if you want to install it in a disabled state, you must specify this using the SPLUNK_APP msiexec command as described in "Install on Windows via the commandline" .
• Show Source is not available for monitor inputs specified as a UNC path on a remote volume. (SPL-28455)
• A perpetual license may report expiration in approximately 1.5 years. (SPL-27005)
• Indexing .evt files that originated on a different machine may not always work, as the Windows API doesn't include consistent .dlls across the different OS versions. You will likely see the following message in your index when this occurs - Splunk could not get the description for this event. Either the component that raises this event is not installed on your local computer or the installation is corrupt. The workaround for this is to use a Splunk forwarder on the originating machine. Splunk cannot be modified to simply "work" in this situation. The problem originates from the lack of information provided by the Windows OS.

Issues related to Splunk Free

• After migrating from Splunk Enterprise to Splunk Free, certain searches (eg top 5 sourceypes) will not work, producing an error about action.summary_index._name. The message is correct, Splunk free doesn't support summary indexing, but the UI breakage is unintended (SPL-28470)

Migrating your license

Splunk 4.x does not work with licenses from older releases. When you install Splunk 4.0.2 or later, your existing 3.x license will be moved aside and replaced with a 4.x Enterprise trial license, which you can use while you procure an updated license.

• If you are an current Enterprise customer, check your splunk.com orders page for an updated license.
• If you are running with a 3.x Free or Enterprise trial license, delete the $SPLUNK_HOME/etc/splunk.license file before you start Splunk 4.x. The instance will then pick up the 60-day Enterprise trial license.
• If you see your license expiration date in year 2283, you will notice that days remaining appear to be off. This is due to the Year 2038 problem. Please request a new license key by submitting a support case.

Considerations for users of Splunk 3.4.x

Splunk 4 is a huge stride forward in performance and flexibility, but there are a few interaction changes vs. 3.4.x which upgraders should be aware of, and even some reasons why you might want to wait for a future release before upgrading. Below are some capabilities that have changed with the introduction of Splunk 4:

Live tail

• With Splunk 4's dramatically improved search and indexing speed, along with the ability to provide intermediate search results, you don't really need a separate live event console to see data in near real-time. However, if your use case relies on version 3.4.x's "Live tail" feature, you may want to wait on upgrading to Splunk 4. Future roadmap plans involve re-architecting the live tail functionality to scale across much larger data flows, and across distributed environments. Additionally, look out for improve real-time alerting and dashboard updates down the road as a result of these upcoming architectural changes.

Custom field actions

• Based on customer feedback, we decided to re-architect this feature to improve flexibility and allow for event actions based on multiple fields. Expect this functionality to be reintroduced in a near term 4.x release. If you rely on this functionality, but still want to upgrade, you may want to consider Splunk 4's new "Dynamic field lookups" as an alternative which allows you to map data from external databases and lists into Splunk.

Snapshots

• In Splunk 4, we've improved upon 3.x's ability to take a timeline snapshots of individual searches. Try out Splunk 4's new job manager which allows you to retrieve the entire cached search result, including reports, from existing searches.

Event scrolling

• In Splunk 4, the new page selector allows you to hop between results with greater flexibility, even as a search runs. However, for those who still prefer a scroll bar, expect this capability to be re-introduced as an option in a future 4.x release.

Timeline and timestamp interaction

• In Splunk 4, we improved the timeline to allow users to quickly view any time range within search results, without having to rerun a search. Also try clicking "zoom-in" on the timeline, which now allows you to lock-in a time range, and specify follow on search.
• We're also planning to improve the usability of some related 3.4.x functionality including clicking on timestamps, and double clicking on timeline bars in future versions of 4.x.

Crawl

• Crawl is no longer configurable via the UI, but is still available as a search command. Based on customer feedback, we have decided to re-architect this feature to make it easier and more effective. Expect improved functionality, along with a new user interface to be introduced in a future release.

FIFO inputs

• This input type has been depreciated with Splunk 4, and we do not recommend using it as a best practice due to data loss considerations. Please contact support@splunk.com if you currently rely on this input type for alternative input methods.

Deployment

• Splunk 4.0.x Deployment server is not compatible with Splunk Deployment client 3.x.

• Was this topic useful?
• Post a comment