Known issues

Known issues

The following are issues and workarounds for version 4.1.2 of Splunk.

Epoch timestamps not parsed correctly after March 12, 2011

This problem affects ALL Splunk versions: 3.x, 4.x, and 4.2.

In Splunk's datetime.xml, the regular expression for parsing epoch time assumes values from 2002 through to March 12th 2011. Those values started with 10,11,12. On March 12th, 2011, the seconds since 1970 became 1300000000, starting with 13.

First, make a backup copy of $SPLUNK_HOME/etc/datetime.xml, and then modify it. Change the _utcepoch regex (at around line 200) to the following:

<define name="_utcepoch" extract="utcepoch, subsecond">
    <!-- update regex before 2017! :) -->
    <text><![CDATA[((?<=^|[\s#,"=([\|{])(?:1[012345]|9)\d{8}|^@[\da-fA-F]{16,24})(?:.?(\d{1,6}))?(?![\d(])]]></text>
</define>

Alternatively, for your sources that use epoch time, explicitly specify a strptime format in props.conf, by using TIME_FORMAT and TIME_PREFIX fields.

Example:

[asterisk]
TIME_FORMAT = %s

Data input issues

• monitor inputs using the followTail setting sometimes will index some older events or all events from log files which are updated when not intended. (SPL-23555)
• Adding an input directory without the trailing slash can produce an error: "Encountered the following error while trying to save: In handler 'monitor': Path must be absolute." (SPL-30011)
• If you configure two different indexes with the same paths to the cold and thawed dbs, Splunk will crash, even if one of the indexes is disabled. (SPL-29281)
• TIME_FORMAT/strptime ignores the hour component of a timestamp if minutes are not provided. (SPL-23777)
• When configuring file system change monitor (fschange) on a forwarder, if signedaudit = true and index=_audit are not explicitly set, fschange events do not get forwarded. (SPL-25294)
• File system change monitor does not work and generates a "Monitoring file or directory that doesn't exist at startup time" in splunkd.log when you monitor the root directory. (SPL-27107)
• The MAX_DAYS_AGO setting sometimes fails to ignore timestamps beyond the set parameter. (SPL-27817)
• Time elapsed between startup and indexing of events can be > 15 minutes due to either Hosts.data file containing thousands of duplicate hosts because some inputs are adding a space to the hostname or due to e metadata files in your index. Contact Support for workaround. (SPL-31033, SPL-31035)
• Sources.Data filesize grows (hundreds of MBs) to where the indexqueue can cause forwarder connections to be dropped/temporarily refused. Contact Splunk Support for workaround. (SPL-31196)
• The props.conf settings CHARSET, CHECK_FOR_HEADER, and NO_BINARY_CHECK are not resepected on the basis of host stanzas. Use source or sourcetype for these settings. (SPL-31303)
• Two equivalent monitor entries with various spellings (eg. variations on slashes on windows, use of .. expressions in paths) are not reasonably supported. For overlapping cases, the behavior is likely to surprise. (SPL-31576)
• fschange sometimes starts to generate events for action=add or action=delete even when there is no such action (SPL-49536)

Splunk Web and Manager interface issues

• If you have cookies disabled or if the server and/or client CPU time are not in sync, you will be returned to the login page. Both machines must have the correct time set when cookie timestamp is verified. (SPL-22393)
• Pressing Enter on the interactive field extractor "Save Field Extraction" form closes the form and does not save the field extraction. (SPL-30419)
• Creating a field via the interactive field extractor displays a redundant error. (SPL-30417)
• Creating a tag with uncommon characters results in undesired behavior such as duplicate tags. (SPL-26414)
• Default auto header extraction (CHECK_FOR_HEADER) is not consistently maintaing sourcetypes when there is no change in the header. (SPL-30466)
• Pressing Enter on the event type builder "Save Event Type" form closes form and does not save the eventtype. (SPL-30407)
• Saving an event type in the event type builder generates an error stating that the proposed event type has already been defined. (SPL-30406)
• Event type builder save-window produces strange behavior in Firefox. (SPL-30104, SPL-30103)
• Clicking on an event term in Splunk Web to add it to the search fails when the term ends with a parenthesis. (SPL-30465)
• The field summary popup windows doesn't show all of the field values if you have 10 events per page selected. (SPL-30464)
• If you type a new search into Splunk Web after your session has timed out (but before you've re-authenticated, click >, and re-log in as requested, the search you ran last before the one you just typed in will be run. (SPL-30460)
• When configuring Splunk Web to use SSL, an 404 Not Found error is displayed. (SPL-30333)
• Drop-down menus are obscured by selected values in fields onscreen on IE6. (SPL-30056)
• Hover-over labels in Flash timecharts are too narrow to display the entire timestamp. (SPL-30251)
• The interactive field extractor mistakenly interprets < and > in field names as comparison operators and will generate an error "Unable to get sample events: Error in 'UnifiedSearch': Unable to parse the 'Invalid LHS for comparison' search." (SPL-30148)
• Using Manager>Search Macros in Splunk Web to define search macros that take arguments results in a non-working configuration. To work around this issue, define these types of search macros in $SPLUNK_HOME/etc/system/local/macros.conf (or in the appropriate app directory). (SPL-30227)
• When using real-time search, various display issues sometimes occur with the timeline, fields picker, and the events view. (SPL-29400)
• Disabling a deployment server server class via Manager generates an error. (SPL-30398)
• Using the browser's Back button to get back to a form view doesn't work properly; you have to re-run the search to redisplay the graph. (SPL-27179)
• Zooming out in the flashtimeline only zooms out the previous time region, not the subsequent one. (SPL-30554)
• Splunk Web still thinks your license is expired if you replace it behind the scenes. To work around this issue, choose 'Enter a new license number' and then log in. (SPL-28582)
• The success message when uploading a file in Splunk Web does not correctly display the filename. (SPL-29855)
• Using jquery before 1.3.2 with changeset 6268 results in false activeX warnings (see http://dev.jquery.com/changeset/6268/trunk). A patch is available, to apply the patch:
  • Download the patch file.
  • Unzip the patch file.
  • cd $SPLUNK_HOME/share/splunk/search_mrsparkle/exposed/js/contrib
  • patch jquery-1.3.2.js jquery-activex.patch
  • Because Splunk Web aggressively caches content, you must change the URI signature:
  • Open http://localhost:8000/_bump
  • Click the 'bump version' button.
• When you filter a list of objects in Manager by App context or owner, then perform an operation on an object in the list, the filter is reset. (SPL-27623)
• On the Field Transformations page in Manager "Delete" links are not presented for objects that are deletable but not editable. (SPL-30899)
• All dashboard panels that are rendered from any scheduled job that was run by the scheduler, appear to get 'last refreshed time' of NaN NaN NaN (SPL-31162)

Charting and drill-down issues

• When a chart displays an "NULL" bucket of values, drilling down into it adds myfield="NULL" to the search string. (SPL-30400)
• When a chart displays an "OTHER" bucket of values, drilling down into it adds myfield="OTHER" to the search string. (SPL-30399)
• SimpleResultsTable and Flashchart do not honor "showperc=f" with a search using "top" as the transform command. (SPL-29635)
• Chart/table drill down goes to an incorrect follow-on search when using discretized ranges in a chart. (SPL-29571, SPL-30553)
• Options for advanced charting display briefly when loading the basic charting page. To access the advanced options, click the 'x-axis' and 'y-axis' links. (SPL-26611)
• FlashCharts in views that use the simplified <form> syntax will not render. Workaround is to find the resize bar with your mouse and drag the chart open from a size of "0" to a fixed size. (SPL-31483)

Search, saved search, scheduling, and job management issues

• When you save a top or rare search with the argument showperc, the showperc argument disappears when you run the search. (SPL-27694)
• Deleting events via the delete operator does not seem to work. (SPL-30499, SPL-30468)
• Pausing a search job in the Job Manager does not update the job's displayed status (SPL-24999)
• There is no notification in Splunk Web that a job has expired or been deleted when you try to interact with the job elsewhere in Splunk Web (SPL-30114)
• The audit.log contains random search_ids for saved searches that have been run manually. (SPL-29566)
• When running a search with 'use starthoursago', the displayed time range message is misleading (although the results are correct). (SPL-30250)
• Searching through a bucket with one or more events in the distant future (such as 2012) can cause no results to be returned unless 'over all time' is selected. (SPL-28444)
• It is possible to create a dashboard which has a real-time search on it and then have it scheduled for delivery. This is not actually supported. (SPL-29782)
• There is no way to escape an asterisk (*) in the search language. (SPL-30079)
• Resurrection issue (saved searches, dashboards) with searches that use | sort with multiple arguments. All arguments past first arg are dropped on resurrect (SPL-30980)
• Clicking on links from saved search alerts gives blank results. (SPL-31337)
• The file operator crashes splunkd and is no longer supported for 4.1.x. (SPL-36953)
• Results using the perc* and median functions for stats/chart/timechart are off by 1 rank. For any dataset larger than a few hundred values, the error is negliable or non-existent (because the value at rank N and at rank N+1 are very likely to be the same or very close to being the same). (SPL-40331)

Localization, internationalization, and character set issues

• Certain Japanese language OSes, including most versions of Windows, use the ¥ (Yen) symbol to denote backslashes in path names. This can cause issues when monitoring or spooling files, and may require custom regex configurations where a file path is part of the dataset. (SPL-23307)

(This issue is also present in the Japanese PDFs of the documentation.)

• Email alert link fails when savedsearch is in Japanese. (SPL-31477)

App and app development issues

• An issue exists in the first time run experience around input collisions: if you enable the *Nix App, the inputs it adds put their data in the "os" index, which by default is only searchable from the *Nix App interface. If you then try to add /var/log as an input (through the Getting Started App or any other App), an error is displayed stating that this input already exists. (SPL-25138)
• It's possible to get to the setup page for an App without enabling it first. (SPL-24852)
• No dashboards are added to the navigation menus for the Windows and *Nix Apps. (SPL-24933)
• Old modules, templates, and other App components are not deleted on upgrade. (SPL-22494)
• The *Nix app is not supported on AIX. (ENH-3001)
• Timeline in the Windows app is overly compressed. (SPL-29932)
• Some 'form search' visual styling elements and contextual styles rely on custom css that only comes with the Search app. (SPL-29816)
• When accessing the "Longest Running Logins" and "The Most Frequent Logons" searches from the Windows app, Splunk displays an error about the keepevicted flag being required. To work around this issue, edit $SPLUNK_HOME/etc/apps/windows/default/data/ui/views/systemmgmt.xml and add keepevicted=1 right after the transaction command (transaction Logon_ID ) for both of the searches. (SPL-30350)
• The *Nix app does not run on Windows. (SPL-25576)
• Navigation menus do not support nesting below 2 levels deep. (SPL-29475)
• If you specify more than the 3-column maximum for layoutPanel, the error message is not very helpful. (SPL-29295)
• You can create/update/clone/delete 'Navigation menus', but Splunk Web only uses default.xml. (SPL-30024)
• Migration from 3.4.x to 4.1 should handle the enabling/disabling of apps correctly. eg. Splunk Desktop is automatically enabled in 4.1 but previously disabled. (SPL-31280)
• in custom form search views, Lister modules like SearchSelectLister can not be configured to run their internal searches over the time range selected in a TimeRangePicker. (SPL-31706)
• 3.x apps can become enabled or disabled (counter to how they were previously configured) when migrationg to 4.1. (SPL-32180)

Windows-specific issues

• The crawl feature is not applicable on Windows. (SPL-24843)
• Adding a Windows-specific input (such as Event Log, Windows Registry, or WMI) in Manager takes longer than adding a file or directory input. This issue will be resolved in a future release. (SPL-26235)
• Splunk Windows services (both splunkweb and splunkd) are installed by default with Startup Type set to "automatic", which means that if you have deployed light forwarders on Windows and haven't explicitly set Startup Type to "manual", the splunkweb process gets started every time you reboot your forwarders. (SPL-22434)
• WMI collection time counters are rounded to whole numbers. It's not possible to improve the precision on the log events time counter, but the performance data can be brought up to sub-second precision. (SPL-28456)
• The Message field is not extracted and therefore missing from imported Windows .evt data. (SPL-24947)
• Timestamps are not set correctly for comment lines in W3C (aka Windows IIS and Exchange) log files. (SPL-29111)
• The splunkd.exe executable on Windows is generating about 4,000 page faults/sec page faults when running the Windows app (only) with all the inputs turned on. This is not necessarily a real problem since most of them will be cache hits and won't end up as hard (on-disk) page faults. However, if the machine is under memory pressure (perhaps from another RAM-hungry app) then splunkd's behavior may cause lots of hard page faults/sec. (SPL-30343)
• On Windows XP/2003 systems, Event Log checkpointing fails if you stop Splunk, clean the events, and restart Splunk. To work around this issue, don't stop Splunk when you clean the events. (SPL-29594)
• Windows installer MSI flags place objects (such as input definitions) in conf files in $SPLUNK_HOME/etc/system/local, which can interfere with deployment server operations. (SPL-29378)
• When translating GUIDs/SIDs in event logs, if the DC Splunk is connected to goes down and then comes back up, Splunk doesn't recover the connection. (SPL-30368)
• When using the commandline to install, if FORWARD_SERVER is set but SPLUNK_APP is not set, forwarding is not enabled. (SPL-29304)
• By default Windows App is enabled, but no windows inputs are enabled (wmi:* and wineventlog:*). (SPL-30979 )
• WinEventLog:Security logs stop indexing with Splunkd.log reporting: ERROR WinEventLogChannel - initOld: Failed to initialize checkpoint for Windows Event Log channel 'Security'. To work around this issue, check to see if this file: %SPLUNK_HOME%\var\lib\splunk\persistentstorage\WinEventLog\Security_checkpoint is empty (1KB). Remove the file and restart Splunkd. The Security_checkpoint file will get recreated and the logs will get processed. (SPL-31339)
• The file operator does not work at all on Windows. When used on a directory, it crashes splunkd. It will not be fixed. Instead, it is no longer supported for 4.1.x and removed in later versions. (SPL-33897, SPL-36953)
• When reinstalling or upgrading Splunk on Windows, the Splunk installer overwrites custom certification authority (CA) certificates in %SPLUNK_HOME%\etc\auth, which can cause SSL communication to fail between a forwarder and an indexer. (SPL-43373)

CLI issues

• CLI output does not recognize "dumb" terminals. (SPL-30432)
• The -output raw option for the CLI search command does not work. (SPL-30404)
• CLI search using -uri from a 3.3 host to a 4.x host produces an error. (SPL-29681)
• Can't enable and disable inputs via the CLI. (SPL-30555)
• CLI 'help' information is only available if Splunk is running. (SPL-30576)
• Running splunk _internal command rebuild-metadata against non-existent index crashes splunkd (SPL-31072)
• "splunk train sourcetype" command does not work. See http://www.splunk.com/base/Documentation/latest/Admin/Commandlinetools#classify to call classify directly. You should use this command only if you have been informed by Support that you require it. (SPL-31078)

Lookup issues

• Doing a field lookup from a csv file that uses ^M does not work. (SPL-29434)
• Making any changes to an existing automatic lookup table in Manager (or hitting Save on an existing configuration without making any changes) leaves garbage behind and creates undesired configs in props.conf. To work around this issue, create a new automatic lookup configuration and delete the old one. (SPL-30617)
• Uploading a too-large ( > 500MB) file (such as a lookup table) via Splunk Web fails without an error. (SPL-30595)

Distributed deployment and deployment server issues

• You must manually distribute certificates to a host before you can successfully add it as a distributed search peer using the CLI. (SPL-24786)
• Splunk Web is unreachable if an enabled deployment server in the same instance cannot access DNS. (SPL-28471)
• Deployment server does not deploy apps whose names include non-ascii characters. To work around this issue, you can rename the app on the client side after it has been deployed. (SPL-30065)
• When transferring configuration files from one system to another, you must either bring along your splunk.secret, or revert your hashed fields to cleartext. (SPL-26529)
• You can't specify an app for deployment server in Manager, only server classes. (SPL-29903)
• Repository Location should not be optional at Manager > Deployment > Deployment server > Add New. (SPL-29901)
• The message displayed when a distributed search peer is unreachable doesn't tell you if there are more than one. (SPL-28399)
• Splunk2Splunk HTTP forwarding ("httpoutput" stanza in outputs.conf) is not currently functional. (SPL-32830)
• In Manager, the deployment server serverclass status view incorrectly lists other deployment client hosts configured. Only those hosts associated with a specific serverclass's whitelist should be displayed. (SPL-40731)
• An attribute, syslogSourceType, for syslog routing does not work. (SPL-64400)

Unsorted issues

• On shutdown, many WARN lines are displayed in splunk.log that should actually be INFO. These lines can be safely ignored. (SPL-24862)
• If a single scripted authentication request hangs, no other authentication requests can be served until the original process is killed. (SPL-30265)
• Exporting multiline events only writes the first 100 lines to the csv file. (SPL-29261)
• HTML results in email alerts does not properly sort fields. (SPL-28474)
• The value of maxlen in limits.conf is ignored, which can result in poor performance over long events. (SPL-30080)
• Splunk Web does not notify you if you specify an invalid port number in web.conf. (SPL-25584)
• The locktest utility should produce human-readable output. (SPL-27664)
• No warning message is displayed when a license violation is committed. (SPL-29454)
• Manually rolling buckets generates a "FATAL" error, although the rolling works fine. (SPL-29045)
• Persistent queue functionality is not working, and will be reworked in a future release. (SPL-27545, SPL-28957)
• Splunk doesn't run on FreeBSD with ZFS. (SPL-30317)
• Some punct tag syntax from pre-4.1 tags.conf files may not be recognized in 4.1 and searches for those tags may not return expected results. To work around this issue, recreate your punct tags using Splunk Web in 4.1. (SPL-28353)
• If the disk Splunk uses fills up, eventually users will not be able to log in because the audit log cannot be written to. (SPL-30162)
• Indexing performance can be impacted if you are running anti-virus scans on Splunk's index directories. This is caused primarily by the additional CPU and disk overhead of the anti-virus accessing these files at the same time as Splunk.
• Version number on all conf/spec/example files is 4.0. (SPL-30714)
• Setting the default app for user or role from UI fails because Splunk creates the setting under the wrong stanza, [general]. The correct stanza setting is [general_default]. Workaround is to edit the user-prefs.conf directly to correct this. (SPL-31580,SPL-30790)
• Splunk cannot be setup using kickstart. (SPL-31683)
• Solaris silent install method does not work. (SPL-30861)
• In some situations, migration may produce an exception when trying to create a migration preview, or perform a migration, for example: IOError: [Errno 2] No such file or directory: '/opt/splunk/etc/apps/user-prefs/local/user-prefs.conf.migratePreview'. To work around, create the directory (and any necessary parent directories) and retry the migration. (SPL-31732)
• Scripted auth scripts and search scripts in perl do not work. (SPL-28532)
• Splunk-forwarder.license is associated with an expiration date of "2011-03-07" This was fixed in 4.1.4 and later (SPL-31628)
• Lowering the maxDataSizeMB or homePath.maxDataSizeMB in indexes.conf might freeze more buckets than the size specified in these attributes. Avoid to use these attributes to reduce the already-indexed volume. (SPL-40220, SPL-39849)

• Was this topic useful?
• Post a comment