Known issues

Known issues

The following are issues and workarounds for version 4.0.3 of Splunk.

Epoch timestamps not parsed correctly after March 12, 2011

This problem affects ALL Splunk versions: 3.x, 4.x, and 4.2.

In Splunk's datetime.xml, the regular expression for parsing epoch time assumes values from 2002 through to March 12th 2011. Those values started with 10,11,12. On March 12th, 2011, the seconds since 1970 became 1300000000, starting with 13.

First, make a backup copy of $SPLUNK_HOME/etc/datetime.xml, and then modify it. Change the _utcepoch regex (at around line 200) to the following:

<define name="_utcepoch" extract="utcepoch, subsecond">
    <!-- update regex before 2017! :) -->
    <text><![CDATA[((?<=^|[\s#,"=([\|{])(?:1[012345]|9)\d{8}|^@[\da-fA-F]{16,24})(?:.?(\d{1,6}))?(?![\d(])]]></text>
</define>

Alternatively, for your sources that use epoch time, explicitly specify a strptime format in props.conf, by using TIME_FORMAT and TIME_PREFIX fields.

Example:

[asterisk]
TIME_FORMAT = %s

Security issues

This version of Splunk contains several security flaws described on this page in the Splunk Security Portal. Splunk strongly recommends that you upgrade to 4.0.11 (or 4.1.2 or later) as soon as possible.

Events dated 2010 not returned by searches

Splunk is not auto-recognizing some timestamps from the year 2010. The problem is specific to two-digit year representations; the timestamp for these events are not correctly indexed by Splunk and so the events are not returned correctly by search. This is a particular issue with Windows Event Log events, but affects all events with timestamps that use two digits to represent the year.

If events from 2010 are not returned by searches, replace the datetime.xml file in your Splunk installation with this one:

http://download.splunk.com/support/config/2010fixed.datetime.xml.gz

The datetime.xml file is located in $SPLUNK_HOME/etc. You must apply this file to all indexers, and to regular Splunk forwarders (but you do not have to apply it to light forwarders, since indexing is not occurring on them.)

To apply this file to your instance:

1. Download the file.
2. Decompress it: gzip -d 2010fixed.datetime.xml.gz
3. copy it to your install: cp 2010fixed.datetime.xml path/to/splunk/etc/datetime.xml

This issue will be resolved in 4.0.8. We are currently working on a step-by-step procedure for recovering events between 01/Jan/2010 00:00:00 and the time you replace datetime.xml, but the general recommendation is to examine your buckets, locate those that include events for the timeframe in question, export them, and re-import them. For more information on buckets and how to identify their timeranges, refer to this topic on the Splunk Wiki.

Security

• A cross-site scripting vulnerability has been identified in Splunkweb. To remediate the issue download this file and untar into your $SPLUNK_HOME directory. (SPL-27560)

General issues

• Deleting events with the "| delete" operator does not work in the CLI and Splunk Web. (SPL-25751)
• Field extractions through IFX are limited to Admin role. (SPL-25471)
• On shutdown, many WARN lines are displayed in splunk.log that should actually be INFO. These lines can be safely ignored. (SPL-24862)
• Editing an existing role to inherit capabilities from another role causes those capabilities to be removed from the first role if it already had them. (SPL-24194)
• You must manually distribute certificates to a host before you can successfully add it as a distributed search peer using the CLI. (SPL-24786)
• LDAP auth to AD may not list all users if realNameAttribute = cn. Set realNameAttribute to 'name' or 'displayName' as a workaround. (SPL-25462)
• If your username has a space in it, you cannot edit views. (SPL-25537)
• Data retirement policies are not being respected. (SPL-23415)
• When creating new users, only 30 roles are shown in the field picker. (SPL-26185)
• DATETIME_CONFIG=CURRENT uses input file modtime instead of current time. (SPL-26311)
• Deployment clients behind a NAT firewall are not supported. (SPL-26237)
• For syslog input, if no sourcetype is defined in inputs.conf, you can end up with "unkown#" sourcetypes. These sourcetypes are unsearchable. (SPL-26213)
• Saved searches created in Splunk Web may reach configuration files with the value 'qualifiedSearch' set. This value is not used and should be ignored or optionally deleted. This should be resolved in 4.0.4. (SPL-26321)
• If you disable Splunk Web from the CLI, you cannot re-enable it without editing web.conf. (SPL-25669)
• Splunk has stability and behavior problems on OS X 10.6 "snow leopard". It may be possible to work around them by running splunkd under gdb. (SPL-25434)
• The troubleshooting tool 'splunk diag' will not work on Windows if the *Nix App is installed (or any app with a path in indexes.conf containing forward slashes). (SPL-25568). Using info_gather.py from 4.0.5 and later will work.
• splunk-search may crash (in factorCommonTerms) when processing moderately complex boolean compound expressions, especially those involving tags. (SPL-27495)
• splunkd's httpd interface may stall when fetching the list of apps for the launcher in unusual network conditions. Workaround by configuring off remote_tab in launcher. http://www.splunk.com/base/Community:ConfigureNoInternet (SPL-28210)

Data input issues

• The .spec file for inputs.conf states an inaccurate default value for rcvbuf. (SPL-24860)
• Uploading a local file over 105MB is currently not supported. (SPL-24292)
• monitor inputs using the followTail setting sometimes will index some older events or all events from log files which are updated when not intended. (SPL-23555)
• 'Automatic' sourcetyping is not supported for network inputs (UDP, TCP), and should not be available in Splunk Web. (SPL-22451)
• Selecting input choice "from list" may revert to 'automatic'. (SPL-25549)
• followTail is ignored. The whole log file gets indexed. (SPL-26010)
• Forwarding data in syslog format does not work. (SPL-26153)

Migrating your license

Splunk 4.x does not work with licenses from older releases. When you install 4.0.2, your existing 3.x license will be moved aside and replaced with a 4.x Enterprise trial license, which you can use while you procure an updated license.

• If you are an current Enterprise customer, check your splunk.com orders page for an updated license.
• If you are running with a 3.x Free or Enterprise trial license, delete the $SPLUNK_HOME/etc/splunk.license file before you start Splunk 4.x. The instance will then pick up the 60-day Enterprise trial license.

Localization, internationalization, and character set issues

• Certain Japanese language OSes, including most versions of Windows, use the ¥ (Yen) symbol to denote backslashes in path names. This can cause issues when monitoring or spooling files, and may require custom regex configurations where a file path is part of the dataset. (SPL-23307)

App and App development issues

• An issue exists in the first time run experience around input collisions: if you enable the *Nix App, the inputs it adds put their data in the "os" index, which by default is only searchable from the *Nix App interface. If you then try to add /var/log as an input (through the Getting Started App or any other App), an error is displayed stating that this input already exists. (SPL-25138)
• It's possible to get to the setup page for an App without enabling it first. (SPL-24852)
• No dashboards are added to the navigation menus for the Windows and *Nix Apps. (SPL-24933)
• It's not possible to delete views from Manager. (SPL-24908)
• Old modules, templates, and other App components are not deleted on upgrade. (SPL-22494)
• If you delete a saved search that is referred to in a dashboard, the dashboard will not load. (SPL-25145)
• Enabling the SplunkForwarder or SplunkDesktop Apps disables deployment server and deployment client functionality. (SPL-26024)
• You may not add more than 10 rows or 3 columns in a view's layoutPanel. (SPL-26177)
• On initial install, the sample app writes its data to the main index. (SPL-25707)
• On Windows, cannot deploy the default SplunkLightForwarder, SplunkForwarder, SplunkDesktop app. Create a custom app instead. (SPL-25716)
• The SplunkLightForwarder app is not forwarding the Splunk internal logs. (APPS-2)
• The *Nix App is not supported on AIX.
• Turning on SplunkForwarder, SplunkLightForwader, SplunkDesktop disables both deployment server and deployment client. (SPL-26024)
• Simple XML searchPostProcess doesn't work with <chart> and <fields>. (SPL-27248)
• in custom form search views, Lister modules like SearchSelectLister can not be configured to run their internal searches over the time range selected in a TimeRangePicker. (SPL-31706)

Search and search app issues

• Clicking on a field value that contains a quote character does not escape it in the search string. (SPL-25001)
• Creating an invalid event type does not generate an error. (SPL-25091)
• The error displayed ("Your entry was not saved. The following error was reported: undefined.") when trying to save a search with a name that already exists is too generic. (SPL-24267)
• The 'uniq' command does not work for large datasets and using '| uniq' in a search will give 0 results. Use '| dedup _raw' instead. (SPL-25852)
• If you schedule a search and leave the 'Include results in email' box un-checked, results are still attached to the email. (SPL-25862)
• The xpath operator does not work. (SPL-26985)

Splunk Web and Manager display issues

• Scrolling in panel layout view does not work correctly in IE7. (SPL-24861)
• The number of users to display per page in Manager > Users does not retain its state if you change it. (SPL-24896)
• Pausing a search job in the job manager does not update the job's displayed status (SPL-24999)
• Splunk Web produces an error saying AttributeError: 'int' object has no attribute 'strip', if updateCheckerBaseURL = 0 in web.conf. Change this setting to "none" or an empty value to disable the updateChecker, or remove the line to enable it (SPL-25319)
• If you have cookies disabled or if the server and/or client CPU time are not correct, you will be returned to the login page. Both machines must be on same GMT time when cookie timestamp is verified. (SPL-22393)
• If you schedule a search that's a report and have it emailed, the link that is included in the email will link to something that does not have the chart formatting you specified. (SPL-25671)
• The web.conf setting 'startwebserver' is assumed to be false for values other than '1'. eg 'startwebserver = true' results in no web server. (SPL-25794)
• Setting tools.sessions.timeout in web.conf is ignored. (SPL-26243)

Windows-specific issues

• The crawl feature is not applicable on Windows. (SPL-24843)
• The Windows App was enabled by default in its app.conf file in versions 4.0-4.0.2. Starting in version 4.0.3, it is disabled in this file by default. (SPL-25487) Read on for important details:
  • If you're upgrading from 4.0-4.0.2 to 4.0.3 or later, the Windows App will be disabled, even if it was enabled in the version you're upgrading from.
  • If you're doing a fresh installation of 4.0.3 or later, the Windows App is enabled by default via the MSI and if you want to install it in a disabled state, you must specify this using the SPLUNK_APP msiexec command as described in "Install on Windows via the commandline" .
• In props.conf the source stanza is ignored. Use sourcetype as a workaround. (SPL-25898)
• Splunk Web becomes unavailable when a user logs out of Windows. To work around this, restart Splunk ( splunk restart) from the CLI. (SPL-25861)
• Forcing roll from hot to warm from the command line requires a backslash on the pipe character before the debug command, eg splunk search "\|debug cmd=roll index=your_index". (SPL-27534)

Considerations for users of Splunk 3.4.x

Splunk 4 is a huge stride forward in performance and flexibility, but there are a few interaction changes vs. 3.4.x which upgraders should be aware of, and even some reasons why you might want to wait for a future release before upgrading. Below are some capabilities that have changed with the introduction of Splunk 4:

Live tail

• With Splunk 4's dramatically improved search and indexing speed, along with the ability to provide intermediate search results, you don't really need a separate live event console to see data in near real-time. However, if your use case relies on version 3.4.x's "Live tail" feature, you may want to wait on upgrading to Splunk 4. Future roadmap plans involve re-architecting the live tail functionality to scale across much larger data flows, and across distributed environments. Additionally, look out for improve real-time alerting and dashboard updates down the road as a result of these upcoming architectural changes.

Custom field actions

• Based on customer feedback, we decided to re-architect this feature to improve flexibility and allow for event actions based on multiple fields. Expect this functionality to be reintroduced in a near term 4.x release. If you rely on this functionality, but still want to upgrade, you may want to consider Splunk 4's new "Dynamic field lookups" as an alternative which allows you to map data from external databases and lists into Splunk.

Snapshots

• In Splunk 4, we've improved upon 3.x's ability to take a timeline snapshots of individual searches. Try out Splunk 4's new job manager which allows you to retrieve the entire cached search result, including reports, from existing searches.

Event scrolling

• In Splunk 4, the new page selector allows you to hop between results with greater flexibility, even as a search runs. However, for those who still prefer a scroll bar, expect this capability to be re-introduced as an option in a future 4.x release.

Timeline and timestamp interaction

• In Splunk 4, we improved the timeline to allow users to quickly view any time range within search results, without having to rerun a search. Also try clicking "zoom-in" on the timeline, which now allows you to lock-in a time range, and specify follow on search.
• We're also planning to improve the usability of some related 3.4.x functionality including clicking on timestamps, and double clicking on timeline bars in future versions of 4.x.

Crawl

• Crawl is no longer configurable via the UI, but is still available as a search command. Based on customer feedback, we have decided to re-architect this feature to make it easier and more effective. Expect improved functionality, along with a new user interface to be introduced in a future release.

FIFO inputs

• This input type has been deprecated with Splunk 4, and we do not recommend using it as a best practice due to data loss considerations. If you currently rely on this input type the recommended input method is to have the FIFO write to a flat file that Splunk is monitoring.

RSS Feed alerts

• Splunk 4 now has improved capabilities for creating email alerts based off searches on your data. However, the RSS feed alerting option from 3.x is currently being re-architected based on customer feedback, and will be reintroduced as an option in a future 4.x release.

Deployment

• Splunk 4.0.x Deployment server is not compatible with Splunk Deployment client 3.x.

• Was this topic useful?
• Post a comment