FAQ

Filtering by business unit includes unexpected systems. Why is that?

This is happening because the search also considers the business unit of system owners is also considered. A filter of APAC includes systems based in APAC and systems based in other business units that are owned by APAC managers.

I want a custom field in the Incident Review dashboard in order to see the variable data in the body of a notable event. How do I that?

1. Open the $SPLUNK_HOME/etc/apps/SplunkEnterpriseSecuritySuite/appserver/event_renderers/notable2.html file.

2. Add the field orig_sourcetype to the list in the "Event Fields List" section. For example:

###################
##Event Fields List
###################

<%def name="event_additional_fields(job, event, request, options, xslt)">

   <%
     def get_random_id():
         import random
         return int(random.random() * 1000000000)
         
     rand_id = get_random_id()
             
     # The following dictionary defines the fields that will be shown in the daily log review list field_inclusion_list = {

'action'  : 'Action',

...

<snip>

'orig_host_requires_av'  : 'Host Requires Av',

'orig_sourcetype'  : 'Original Sourcetype',

'os'  : 'Operating System',

</snip> 'vlan_id'  : 'VLAN Identifier',

'vlan_name'  : 'VLAN Name'

...

}

 %>

Save the file.

3. Switch back to the Incident Review dashboard and look for the orig_sourcetype field values in notable events.

I enter "EMEA" or "Emea" in a field, but the search does not find either one. Why is that?

Use "emea" (lowercase) in the filter field to see search results. All dashboard filters require lowercase text values. This true for all dashboard filters where text values are used.

I cannot search on extracted fields. Why is that?

I setup a regex based field extraction, for example the field name is MyField. When I run a search, sourcetype=MyEvents I see that the field is extracted correctly. However, when I run a search based on a value of MyField, say sourcetype=MyEvents MyField=ValidValue, nothing gets returned.

The solution

  $SPLUNK_HOME/etc/system/local/fields.conf
  [MyField]
  INDEXED_VALUE = false

For more details about this solutions, see blog entry "Cannot search based on an extracted field".

There is incorrect data (or no data) from a distributed environment showing up in System Center dashboard. Why?

If add-ons corresponding to the data forwarded from full forwarders are not installed on full forwarders, incorrect data will show up in dashboards.

If a forwarder is a full forwarder, then data forwarded from it goes through the parsing queue (but not through the indexing queue) on the forwarder. That means that add-ons containing knowledge needed for parsing that data need to be installed on the forwarders.

When I search on Identity Center it doesn't filter identities. Why?

The filter for the Identity Center dashboard uses a search field that needs key=value pairs to be specified in this filter to work correctly, not text. You need to enter a key=value pair into the filter, instead of a name or text string.

Sample key=value pairs would be email=*acmetech.com, nick=anickname

See "Identity Center dashboard" in this document for more information.

How do I manually enable eventgen?

Manually enable the eventgen application by editing the inputs.conf file and setting the state disabled to false in $SPLUNK_HOME/etc/apps/SA-Eventgen/default/inputs.conf.

Here is an example of inputs.conf with the application enabled:

[script://$SPLUNK_HOME/etc/apps/SA-Eventgen/bin/eventgen.py]
disabled = false
interval = 300
passAuth = splunk-system-user
sourcetype = eventgen
index = _internal

Restart Splunk to make the event generator begin producing events.

Eventgen was enabled and I need to stop it and remove the data. How do I do that?

To stop sample event generation, go to Apps > Manage Apps > SA-Eventgen and click Disable.

To remove data generated by eventgen, do the following from the command-line:

  $SPLUNK_HOME/bin ./splunk stop
  $SPLUNK_HOME/bin ./splunk clean all -f

Important note: This will remove all of your Splunk data, not just the eventgen data.

Some searches are using a great deal of memory. Why is that?

Adding the notable index or summary indexes to the default indexes to be searched causes correlation searches to re-detect another finding based on the content of a previous correlation search firing.

The solution is to remove the notable index (or summary index) from the list of indexes to be searched by default.

See "Configure multiple indexes" in the Enterprise Security Installation and Configuration Manual for more information.

Why does importing large XML files, such as Nessus or NMAP scan logs, take such a long time?

XML parsing of complex documents requires a large amount of resources. Splunk recommends splitting XML output into small blocks for optimal performance.

Sometimes notable events include raw event material and sometimes not. Why is that?

Contributing or original events may be displayed in the notable event viewer, depending on formatting rules, or they may be linked. You will always be able to access the raw material behind a notable event.

Are there new dashboard colors in Enterprise Security?

Yes. A new color palette has been introduced for Enterprise Security that does not conflict with priority colors in the dashboards. The randomly assigned colors for event data are now easily distinguishable from these priority colors.

I added a new user but I do not see the user in the Enterprise Security dashboards. Why is that?

It may take several minutes for new Splunk users added to Enterprise Security to be reflected in Enterprise Security dashboards.

When I click a "View Full Results" link or chart to show raw events, it finds no events. Why?

Enterprise Security leverages summary indexes and lookup tables extensively. Summary indexes and lookups may contain data that summarizes raw events that are no longer in Splunk, since raw events can be rolled out of the system. Talk to your System Administrator about how Splunk is set up to handle data retention if desired raw events are not accessible.

My chart displays the status name instead of the status label. Why is that?

If you use the report builder to build a report on notable event status (and generate your chart from this report) you should be using "status_label" instead of "status" (which is an id).

A drilldown on notable events finds more events than displayed on the Notable Event dashboard. Why?

By default, notable event drilldown is configured to display all related events at the time you drill down. You can change this window by editing the associated correlation search. See the Splunk App for Enterprise Security Installation and Configuration Manual for more information.

I install Enterprise Security on Splunk 4.2.3 or older and Enterprise Security becomes unresponsive. What should I do?

Splunk 4.2.3 is not a supported platform. The Splunk App for Enterprise Security 2.2 requires Splunk 5.0 or later. See "Splunk App for Enterprise Security prerequisites" for details.

I create a suppression for events in the Incident Review dashboard but the notable events are still visible in the Security Posture dashboard. What is going on?

Event suppression only suppresses events from Incident Review, as there is no need to for an analyst to review them. They do still represent load on the system and will continue to be represented in Security Posture, Audit, and other screens.

I have turned off some dashboards via the Configure > Dashboards and Domains tool, but my system's performance seems unchanged. Why?

Turning off domains does not stop the searches that provide the data and metrics for those domains; see the Search View Matrix in this manual for more information.

After performing a search in Incident Review, I cannot expand or manage notable events. What is happening?

The search may not have completed or may be running in real time. Searches in the Incident Review dashboard must be finalized before working with notable events. To finalize a search, click the green checkmark icon. More information can be found in the "Perform search actions" topic in the core Splunk product documentation.

I'm getting unexpected data in some of my fields. What is going on?

If you have edited any of the lookup files via Configure > Lists and Lookups, you may have introduced a typo. There is no validation in this editor panel at this time.

Why does my lookup file produce an error?

Excel files created on any platform produce CSV files with Windows line endings. The dos2unix command can be used to correct this. See "Create user-populated lists" for more information.

In the Malware Center view, "allowed" is indicated in green and not red, even though it is often indicating a "bad" thing such as an EPP failure. Why is this?

This is a known issue with color mapping.

How does Enterprise Security detect PII?

The Splunk App for Enterprise Security provides a correlation search, Audit - Personally Identifiable Information Detection - Rule, to look for Personally Identifiable Information (PII) within log data. The search uses the Luhn algorithm and an Issuer Identification Number (IIN) lookup to identify suspect integer sequences.

This search is turned off by default because many customers filter potential PII sources from being indexed. When it is turned on, it should be tuned to specific sourcetypes so that it will only send the optimal strings to the Luhn algorithm.

Can I edit correlation searches in the Splunk Search editor: Settings > Searches?

Technically, you can edit the searches from the Settings menu but you should not. Editing the search this way could break the correlation search or you might not be able to edit other necessary, related settings. Correlation searches are more complex than regular searches in Splunk. Use the Splunk App for Enterprise Security editor -- Configure > Correlation Searches -- to edit correlation searches.

Why do I have "dateparserverbose" errors in my Splunk internal log?

W3C Extended Log Format files, such as those from BlueCoat or MS ISA, contain header sections that do not have timestamps. This causes the Splunk configuration (that is correct for the log format), to warn that these header lines cannot be parsed for a date. The warnings can be prevented by sending lines beginning with a hashmark (#) to the NullQueue. If you do this, potentially interesting information such as software versions, will not be indexed.