FAQ

Filtering by business unit includes unexpected systems. Why is that?

This is happening because the search also considers the business unit of system owners is also considered. A filter of APAC includes systems based in APAC and systems based in other business units that are owned by APAC managers.

I enter "EMEA" or "Emea" in a field, but the search does not find either one. Why is that?

Use "emea" (lowercase) in the filter field to see search results. All dashboard filters require lowercase text values. This true for all dashboard filters where text values are used.

I cannot search on extracted fields. Why is that?

I setup a regex based field extraction, for example the field name is MyField. When I run a search, sourcetype=MyEvents I see that the field is extracted correctly. However, when I run a search based on a value of MyField, say sourcetype=MyEvents MyField=ValidValue, nothing gets returned.

The solution

  $SPLUNK_HOME/etc/system/local/fields.conf
  [MyField]
  INDEXED_VALUE = false

For more details about this solutions, see blog entry "Cannot search based on an extracted field".

There is incorrect data (or no data) from a distributed environment showing up in System Center dashboard. Why?

If add-ons corresponding to the data forwarded from full forwarders are not installed on full forwarders, incorrect data will show up in dashboards.

If a forwarder is a full forwarder, then data forwarded from it goes through the parsing queue (but not through the indexing queue) on the forwarder. That means that add-ons containing knowledge needed for parsing that data need to be installed on the forwarders.

When I search on Identity Center it doesn't filter identities. Why?

The filter for the Identity Center dashboard uses a search field that needs key=value pairs to be specified in this filter to work correctly, not text. You need to enter a key=value pair into the filter, instead of a name or text string.

Sample key=value pairs would be email=*acmetech.com, nick=anickname

See "Identity Center dashboard" in this document for more information.

How do I manually enable eventgen?

Manually enable the eventgen application by editing the inputs.conf file and setting the state disabled to false in $SPLUNK_HOME/etc/apps/SA-Eventgen/default/inputs.conf.

Here is an example of inputs.conf with the application enabled:

[script://$SPLUNK_HOME/etc/apps/SA-Eventgen/bin/eventgen.py]
disabled = false
interval = 300
passAuth = splunk-system-user
sourcetype = eventgen
index = _internal

Restart Splunk to make the event generator begin producing events.

Eventgen was enabled and I need to stop it and remove the data. How do I do that?

To stop sample event generation, go to Apps > Manage Apps > SA-Eventgen and click Disable.

To remove data generated by eventgen, do the following from the command-line:

  $SPLUNK_HOME/bin ./splunk stop
  $SPLUNK_HOME/bin ./splunk clean all -f

Important note: This will remove all of your Splunk data, not just the eventgen data.

Some searches are using a great deal of memory. Why is that?

Adding the notable index or summary indexes to the default indexes to be searched causes correlation searches to re-detect another finding based on the content of a previous correlation search firing.

The solution is to remove the notable index (or summary index) from the list of indexes to be searched by default.

See "Configure multiple indexes" in the Enterprise Security Installation and Configuration Manual for more information.

Why does importing large XML files, such as Nessus or NMAP scan logs, take such a long time?

XML parsing of complex documents requires a large amount of resources. Splunk recommends splitting XML output into small blocks for optimal performance.

Sometimes notable events include raw event material and sometimes not. Why is that?

Contributing or original events may be displayed in the notable event viewer, depending on formatting rules, or they may be linked. You will always be able to access the raw material behind a notable event.

Are there new dashboard colors in Enterprise Security?

Yes. A new color palette has been introduced for Enterprise Security that does not conflict with priority colors in the dashboards. The randomly assigned colors for event data are now easily distinguishable from these priority colors.

I added a new user but I do not see the user in the Enterprise Security dashboards. Why is that?

It may take several minutes for new Splunk users added to Enterprise Security to be reflected in Enterprise Security dashboards.

When I click a "View Full Results" link or chart to show raw events, it finds no events. Why?

Enterprise Security leverages summary indexes and lookup tables extensively. Summary indexes and lookups may contain data that summarizes raw events that are no longer in Splunk, since raw events can be rolled out of the system. Talk to your System Administrator about how Splunk is set up to handle data retention if desired raw events are not accessible.

My chart displays the status name instead of the status label. Why is that?

If you use the report builder to build a report on notable event status (and generate your chart from this report) you should be using "status_label" instead of "status" (which is an id).

A drilldown on notable events finds more events than displayed on the Notable Event dashboard. Why?

By default, notable event drilldown is configured to display all related events at the time you drill down. You can change this window by editing the associated correlation search.

I create a suppression for events in the Incident Review dashboard but the notable events are still visible in the Security Posture dashboard. What is going on?

Event suppression only suppresses events from Incident Review, as there is no need to for an analyst to review them. They do still represent load on the system and will continue to be represented in Security Posture, Audit, and other screens.

After performing a search in Incident Review, I cannot expand or manage notable events. What is happening?

The search may not have completed or may be running in real time. Searches in the Incident Review dashboard must be finalized before working with notable events. To finalize a search, click the green checkmark icon. More information can be found in the "Perform search actions" topic in the core Splunk product documentation.

I'm getting unexpected data in some of my fields. What is going on?

If you have edited any of the lookup files via Configure > Lists and Lookups, you may have introduced a typo. There is no validation in this editor panel at this time.

Why does my lookup file produce an error?

Excel files created on any platform produce CSV files with Windows line endings. The dos2unix command can be used to correct this. See "Create user-populated lists" for more information.

In the Malware Center view, "allowed" is indicated in green and not red, even though it is often indicating a "bad" thing such as an EPP failure. Why is this?

This is a known issue with color mapping.

Does Splunk App for Enterprise Security use Internet threat lists?

The Splunk App for Enterprise Security performs daily downloads of threat lists that are used to support the following correlation rules:

   Network - Internet Proxy Server Activity - Rule
   Network - Known Web Attacker Activity - Rule
   Network - LogMeIn Activity - Rule
   Network - PirateBay Activity - Rule
   Network - RapidShare Activity - Rule
   Network - SANS Block List Activity - Rule
   Network - Spyware Activity - Rule
   Network - Tor Router Activity - Rule
   

See "Configure threat lists" in the Splunk App for Enterprise Security Installation and Configuration Manual for more information about threat lists.

Does Splunk App for Enterprise Security detect Personally Identifiable Information?

The Splunk App for Enterprise Security provides a correlation search, Audit - Personally Identifiable Information Detection - Rule, to look for Personally Identifiable Information (PII) within log data. The search identify suspect integer sequences that could be credit card numbers, then passes them to the Luhn algorithm and an Issuer Identification Number (IIN) lookup to confirm before generating a notable event.

Note: This search is turned off by default in order to avoid inadvertent testing of integer sequences that match the format but are known not to be suspect. To enable the search, ensure that it will only review data where suspect integer sequences are possible.

The Luhn algorithm is used to validate identification numbers. Most commonly, it is used for credit card numbers. It is used to determine if numbers that look like credit card numbers actually are credit card numbers.

The issuers list matches credit card numbers (which match the Luhn algorithm) with the organization that has issued them.

The Luhn algorithm search can be tuned by copying the [luhn_lookup] section from default/transforms.conf to local/transforms.conf in $SPLUNK_HOME/etc/apps/SA-AuditAndDataProtection/ and tuning the external_cmd field. The separators, minStrength, maxStrength, and offset are settings that help improve the detection of suspect sequences.

• separators specify what type of special characters may separate integer sequences
• minStrength tells the script to ignore any sequences that are not at least X integers in length
• maxStrength tells the script to ignore any sequences that exceed X integers in length
• offset tells the script to ignore the first X characters in its integer sequence evaluation

How does Enterprise Security detect PII?

The Splunk App for Enterprise Security provides a correlation search, Audit - Personally Identifiable Information Detection - Rule, to look for Personally Identifiable Information (PII) within log data. The search uses the Luhn algorithm and an Issuer Identification Number (IIN) lookup to identify suspect integer sequences.

This search is turned off by default because many customers filter potential PII sources from being indexed. When is it turned on, it should be tuned to specific sourcetypes so that it will only send the optimal strings to the Luhn algorithm.

Can I edit correlation searches in the Splunk Search editor: Settings > Searches?

Technically, you can edit the searches from the Settings menu but you should not. Editing the search this way could break the correlation search or you might not be able to edit other necessary, related settings. Correlation searches are more complex than regular searches in Splunk. Use the Splunk App for Enterprise Security editor -- Configure > Correlation Searches -- to edit correlation searches.

Why do I have "dateparserverbose" errors in my Splunk internal log?

W3C Extended Log Format files, such as those from BlueCoat or MS ISA, contain header sections that do not have timestamps. This causes the Splunk configuration (that is correct for the log format), to warn that these header lines cannot be parsed for a date. The warnings can be prevented by sending lines beginning with a hashmark (#) to the NullQueue. If you do this, potentially interesting information such as software versions, will not be indexed.