Splunk® Enterprise Security

Use Splunk Enterprise Security

Download manual as PDF

This documentation does not apply to the most recent version of ES. Click here for the latest version.
Download topic as PDF

Configure threat intelligence sources

Splunk Enterprise Security provides threat intelligence and context for security monitoring and investigation by comparing the contents of multiple threat sources to indexed events. Enterprise Security supports multiple threat intelligence collection sources, and includes a selection of threat intelligence sources.

Each source represents a collection of knowledge about a specific type of security threat or activity. Splunk administrators can add new sources to ES if those sources are available as a structured data file or stream.

Supported threat collection methods

Any content that is available as a structured text file can be added to ES as a threat source. This includes support for:

In a Splunk Cloud deployment, customers will work with Splunk Professional Services to implement, and enable threat intelligence sources.

Supported threat intelligence groups

Threat collections are defined in the collections.conf, and stored in KVStore. Each threat collection has an associated local lookup file to allow you to manually enter threat intelligence data. Select Configure > Data Enrichment > Lists and Lookups to view the local threat intelligence lookups and add data.

Supported IOC data types Threat Collection Local lookup file
X509 Certificates certificate_intel Local Certificate Intel
Email email_intel Local Email Intel
Files names/hashes file_intel Local File Intel
HTTP http_intel Local HTTP Intel
IP/Domains ip_intel Local IP Intel, Local Domain Intel
Processes process_intel Local Process Intel
Registry entries registry_intel Local Registry Intel
Services service_intel Local Service Intel
Users user_intel Local User Intel

Review the provided threat sources

Splunk Enterprise Security includes preconfigured threat intelligence sources. Each threat source site offers suggestions for polling intervals and other configuration requirements independent of Enterprise Security. Use the source site links to review the provider's documentation for a threat source.

  1. On the Enterprise Security menu bar, open Configure > Data Enrichment > Threat Intelligence Downloads and review the Description field in all defined threat intelligence sources.
  2. Use the Threat Intelligence Download Settings page to enable and configure threatlist settings.
  3. Use the Threat Intelligence Audit dashboard to review the status of all threat download sources.
  4. Use the Threat Artifacts dashboard to find or query on individual threat elements.

Threat sources included with ES

These threat sources are included with Splunk Enterprise Security and retrieve information across the internet.

Threat Source Threat Provider Source site
Alexa Top 1 Million Sites Alexa Internet http://s3.amazonaws.com/alexa-static/
Emerging Threats compromised IPs blocklist Emerging Threats http://rules.emergingthreats.net/blockrules
Emerging Threats fwip rules Emerging Threats http://rules.emergingthreats.net/fwrules
Malware domain host list Hail a TAXII.com http://hailataxii.com
iblocklist Logmein I-Blocklist http://list.iblocklist.com
iblocklist Piratebay I-Blocklist http://list.iblocklist.com
iblocklist Proxy I-Blocklist http://list.iblocklist.com
iblocklist Rapidshare I-Blocklist http://list.iblocklist.com
iblocklist Spyware I-Blocklist http://list.iblocklist.com
iblocklist Tor I-Blocklist http://list.iblocklist.com
iblocklist Web attacker I-Blocklist http://list.iblocklist.com
ICANN Top-level Domains List IANA http://data.iana.org
Malware Domain Blocklist Malware Domains http://mirror1.malwaredomains.com
Mozilla Public Suffix List Mozilla https://publicsuffix.org
abuse.ch Palevo C&C IP Blocklist abuse.ch https://palevotracker.abuse.ch
Phishtank Database Phishtank http://data.phishtank.com
SANS blocklist SANS http://isc.sans.edu
abuse.ch ZeuS blocklist (bad IPs only) abuse.ch https://zeustracker.abuse.ch
abuse.ch ZeuS blocklist (standard) abuse.ch https://zeustracker.abuse.ch

Note: If you determine that your Splunk Enterprise Security installation is retrieving data from unexpected IP addresses, perform a WHOIS or nslookup to determine if the IP address matches that of one of the threat sources configured in your environment.

Threat Intelligence Download Settings

As an analyst, use the Threat Intelligence Download page to implement, configure, and enable threat intelligence sources that require an API or URL connection, or are lookup based. On the Enterprise Security menu bar, open Configure > Data Enrichment and select Threat Intelligence Downloads.

ES33 Threat Intel Download.png

Select the threat source name field to view the settings.

Threat Intelligence Download Setting
  • Type: The type of threat source. (Required)
  • Description: Details about the source. (Required)
  • URL: A link to the source download. Use the base URL defined by the feed provider. (Required)
  • Weight: The threat source weight multiplier used when making Risk Analysis calculations. (Required) A higher number implies an increased relevance. See Assigning a risk score through search for an example.
  • Interval: The download interval in seconds.
  • POST Arguments: Optional arguments that can be passed to the URL for threat source downloading.
  • Maximum age: The retention period for this threat source. If the last change time exceeds the maximum age, the threat source data will be purged from the collection. To enable this feature, see Configure threat source retention in this topic.
Parsing Options
  • Delimiting regular expression: A delimiter used to split lines in a threat source document.
  • Extracting regular expression: A regular expression used to extract fields from individual lines of a threat source document.
  • Fields: A transforms.conf-style expression used to rename or combine fields. (Required)
  • Ignoring regular expression: A regular expression used to IGNORE lines in a threat source.
  • Skip header lines: The number of header lines to skip when processing the threat source. Set to "1" for lookup tables.
Download Options
  • Retry interval: Number of seconds to wait between download retry attempts. Review the recommended poll interval of the threat source provider before changing the retry interval. (Required)
  • Remote site user: A username to use in remote authentication, if required. The user must correspond to the name of a Splunk secure stored credential in Credential Management
  • Retries: The maximum number of retry attempts.
  • Timeout: Number of seconds to wait before marking a download attempt as failed. (Required)
Proxy Options: (Optional)
  • Proxy Server: The proxy server address. Cannot be a URL. For example, 10.10.10.10 or server.example.com
  • Proxy Port: The proxy server port.
  • Proxy User: The proxy server user credential. Supported authentication methods are Basic and Digest. The user must correspond to the name of a Splunk secure stored credential in Credential Management
Important: If you remove an existing proxy user and password in the Threat Intelligence Download Setting editor, the download process will no longer reference the stored credentials. Removing the referenced to credential does not delete the stored credentials from Credential Management.

Edit a threat source

Use the Threat Intelligence Download Settings editor to modify information about an existing threat source.

To edit a threat source:

  1. Select Configure > Data Enrichment > Threat Intelligence Downloads.
  2. Select the name of the threat source you want to edit, which opens the Threat Intelligence Download Settings editor.
  3. Make changes to the fields as required.
  4. Save your changes.

Note: To allow non-admin users to edit threat sources, see Adding capabilities to a role in the Installation and Upgrade manual.

Configure threat source retention

To remove threat source data from a threat collection based on date, the threat source must have a Maximum age setting and a retention tracking search enabled.

  1. From the Enterprise Security menu bar, select Configure > Data Enrichment > Threat Intelligence Downloads.
  2. Select a threat source.
  3. Change the Maximum age setting using a relative time specifier. Example: -7d, -30d. TAXII feeds do not support this setting.
  4. Enable the retention search for the collection. On the Splunk platform menu bar, select Settings and click Searches, reports, and alerts.
  5. Search for "retention" using the search filter.
  6. Enable the retention search for the collection that hosts the threat source. All retention searches are disabled by default.

Add an OpenIOC file

All file-based threat sources must be uploaded to the monitored path on the on-premises Enterprise Security search head, or made available on a mounted local network share. In a search head cluster, add it to this location on the deployer and deploy it to the search head cluster members. An OpenIOC file must have a .ioc extension to be read and parsed.

Default settings

The modular input da_ess_threat_local is configured to monitor the $SPLUNK_HOME/etc/apps/DA-ESS-ThreatIntelligence/local/data/threat_intel folder, ingest any .ioc files found, and delete the documents after processing.

To review or modify the da_ess_threat_local inputs settings:

  1. On the Enterprise Security menu bar, select Configure > Data Enrichment > Threat Intelligence Management.
  2. Select the da_ess_threat_local modular input.
  3. Review or modify the settings as required.

Note: Do not modify the default da_ess_threat_default, or sa_threat_local inputs.

Add a STIX file

All file-based threat sources must be uploaded to the monitored path on an on-premises Splunk Enterprise Security search head, or made available on a mounted local network share. In a search head cluster, add it to this location on the deployer and deploy it to the search head cluster members. A STIX file must have a .xml extension to be read and parsed. Splunk Cloud customers must work with Splunk Support to import STIX threat intelligence data.

Default settings

The modular input da_ess_threat_local is configured to monitor the $SPLUNK_HOME/etc/apps/DA-ESS-ThreatIntelligence/local/data/threat_intel folder, ingest any .xml files found, and delete the documents after processing.

To review or modify the da_ess_threat_local inputs settings:

  1. On the Enterprise Security menu bar, open Configure > Data Enrichment > Threat Intelligence Management
  2. Select the da_ess_threat_local modular input.
  3. Review or modify the settings as required.

Note: Do not modify the default da_ess_threat_default, or sa_threat_local inputs.

Add a TAXII feed

The threat intel modular input threatlist.py downloads content from the TAXII feed as a single document and places the file into the $SPLUNK_HOME/etc/apps/SA-ThreatIntelligence/local/data/threat_intel folder for processing.

  1. On the Enterprise Security menu bar, select Configure > Data Enrichment > Threat Intelligence Downloads.
  2. Click New to add a new TAXII feed.
  3. Type a Name for the threat intelligence feed.
  4. Type a Type of taxii.
  5. Type a Description for the threat intelligence feed.
  6. Type a URL to use to download the TAXII feed.
  7. (Optional) Change the default Weight for the threat intelligence feed. Increase the weight if the threats on the threat feed are high-confidence and malicious threats that should increase the risk score for assets and identities that interact with the indicators from the threat source.
  8. (Optional) Adjust the interval at which to download the threat intelligence. Defaults to 43200 seconds, or twice a day.
  9. Type TAXII-specific space-delimited POST arguments for the threat intelligence feed.
    <POST argument>="<POST argument value>"
    Example POST argument Description Example value
    collection Name of the data collection from a TAXII feed. collection="A_TAXII_Feed_Name"
    earliest The earliest threat data to pull from the TAXII feed. earliest="-1y"
    taxii_username An optional method to provide a TAXII feed username. taxii_username="user"
    taxii_password An optional method to provide a TAXII feed password. If you provide a username without providing a password, the threat intelligence modular input attempts to find the password in Credential Management. taxii_password="password"
    cert_file Add the certificate file name if the TAXII feed uses certificate authentication. The file name must match exactly and is case sensitive. cert_file="cert.crt"
    key_file Add the key file name for the certificate if the TAXII feed uses certificate authentication. The file name must match exactly and is case sensitive. key_file="cert.key"
  10. TAXII feeds do not use the Maximum age setting.
  11. TAXII feeds do not use the Parsing Options settings.
  12. (Optional) Change the Download Options.
  13. (Optional) Change the Proxy Options. See Configure a proxy for retrieving threat intelligence.
  14. Save the changes.

You cannot use an authenticated proxy with a TAXII feed because the libtaxii library used by Enterprise Security does not support authenticated proxies. If possible, use an unauthenticated proxy instead.

Add a TAXII feed with certificate authentication

In a Splunk Cloud deployment, customers must work with Splunk Support to add or change files on cloud-based nodes. Add the certificate and keys to the same app directory that you define the TAXII feed in, for example, DA-ESS-ThreatIntelligence.

  1. Follow the steps for adding a TAXII feed to Splunk Enterprise Security.
  2. Add the certificate to the $SPLUNK_HOME/etc/apps/<app_name>/auth directory.
  3. Add the private key for the certificate to the same /auth directory.
  4. Add the TAXII feed, using the cert_file and key_file arguments to specify the file names of the certificate and private key file.

Add a URL-based threat source

  1. On the Enterprise Security menu bar, select Configure > Data Enrichment > Threat Intelligence Downloads.
  2. Click New to open the Threat Intelligence Download Settings editor.
  3. Enter the information about the threat source, filling in all required fields. For examples, see the preconfigured threat sources provided in Enterprise Security.
  4. Save the changes when you are done.
  5. Validate the source is properly configured and downloading by reviewing the Threat Intelligence Audit dashboard.

Adding a custom source

Add a custom threat source to Splunk Enterprise Security as a local lookup file. A lookup-based threat source can add data to any of the supported threat intelligence groups, such as file or IP intelligence.

Prerequisite

Identify whether the custom threat source is certificate, domain, email, file, HTTP, IP, process, registry, service, or user intelligence.

Steps

Based on the type of intelligence you add to Splunk Enterprise Security, you need to identify the headers for the csv file.

  1. Select Configure > Data Enrichment > Lists and Lookups.
  2. Locate the lookup file that matches the local threat intel you are providing. For example, Local File Intel.
  3. Open the relevant lookup to view the required headers.
  4. Create a new .csv file with a header row containing the required fields.
  5. Add the threat data to the .csv file.

After you create the lookup file, you need to add it to Splunk Enterprise Security.

  1. On the Splunk platform menu bar, select Settings > Lookups
  2. Next to Lookup table files, click Add New.
  3. Select a Destination App of SA-ThreatIntelligence.
  4. Upload the .csv file you created.
  5. Type a Destination filename for the file. For example, threatindicatorszerodayattack.csv.
  6. Save.

After adding the threat intel lookup to ES, set appropriate permissions so ES can use the file.

  1. Open Lookup table files.
  2. Locate the lookup file that you added and select Permissions.
  3. Set Object should appear in to All apps.
  4. Select Read access for Everyone.
  5. Select Write access for admin.
  6. Save.

Define the lookup so that Splunk ES can import it and understand what type of intelligence you are adding.

  1. On the Splunk platform menu bar, select Settings > Lookups.
  2. Next to Lookup definitions, click Add New.
  3. Select a Destination App of SA-ThreatIntelligence.
  4. Enter a name for the threat source. The name you enter here is used to define the threatlist in the input stanza. For example, zero_day_attack_threat_indicators_list.
  5. Select a Type: of File based.
  6. Select the Lookup File: that you added in step one. For example, threatindicatorszerodayattack.csv.
  7. Save.

Set permissions on the lookup definition so that the lookup functions properly.

  1. Open Lookup definitions
  2. Locate the definition you added in step four and select Permissions.
  3. Set Object should appear in to All apps.
  4. Set Read access for Everyone.
  5. Set Write access for admin.
  6. Save.

Add a threat source input stanza that corresponds to the lookup file so that ES knows where to find the new threat intelligence.

  1. Select Configure > Data Enrichment > Threat Intelligence Downloads.
  2. Choose a threat source input that matches your new content. For example, local_file_intel.
  3. Click Clone in the Actions column.
  4. Type a Name. The name cannot include spaces. For example, zero_day_attack_threat_indicators.
  5. Type a Type. For example, IOCs.
  6. Type a Description. For example, File-based threat indicators from zero day malware.
  7. Type a URL that references the lookup definition you created in step three: lookup://zero_day_attack_threat_indicators_list.
  8. (Optional) Change the default Weight for the threat data.
  9. (Optional) Change the default Retry interval for the lookup.

Verify that you added the custom threat source successfully and ES is ingesting it accurately.

  1. Select Audit > Threat Intelligence Audit.
  2. Review the Threat Intelligence Downloads to confirm that the threat source is enabled.
  3. Review the Threat Intelligence Audit Events to determine if there are errors associated with the threat lookup name.
  4. Select Security Intelligence > Threat Intelligence > Threat Artifacts.
  5. Search for an object from the threat source to confirm that Splunk Enterprise Security can find it.

Note: If one of the pre-configured inputs isn't a match for your threat intel source, select New. Fill out the required fields in the newly created threat source input.

Add database-hosted content

For an example of how to add database-hosted content, see this blog post discussing custom threat feed integration with Enterprise Security (http://blogs.splunk.com/2014/03/10/custom-threat-feed-integration-with-enterprise-security).

Add threat data manually

Each threat collection has a pre-configured lookup file to accept manually entered threat data.

  1. On the Enterprise Security menu bar, select Configure > Data Enrichment > Lists and Lookups.
  2. Find the preconfigured lookup that matches the threat collection data you are providing. Example: Local Certificate intel.
  3. Select the lookup to open the lookup editor.
  4. Update the lookup, populating all of the fields for each row (right-click on the row, and choose Insert Row Below).
  5. (Optional) Populate the weight value with a numeric value to change the risk scoring calculation for an entry.
  6. Click Save when finished.

Configure a custom folder for threat sources

Enterprise Security provides inputs configured to monitor local folders on the search head and ingest threat intelligence sources placed there. Use the Threat Intelligence Management page to review, add, or change inputs for monitoring folders containing threat sources. In a Splunk Cloud deployment, customers will work with Splunk Support to add or change files or folders on cloud-based nodes.

Default settings

The modular input da_ess_threat_local is configured to monitor the $SPLUNK_HOME/etc/apps/DA-ESS-ThreatIntelligence/local/data/threat_intel folder, ingest any .csv, .ioc and .xml files found, and delete the documents after processing.

To review or modify the da_ess_threat_local inputs settings:

  1. From the Enterprise Security menu bar, select Configure > Data Enrichment > Threat Intelligence Management.
  2. Select the da_ess_threat_local modular input.
  3. Review or modify the settings as required.

Note: Do not modify the default da_ess_threat_default, or sa_threat_local inputs.

Create a new input monitor for threat sources

  1. From the Enterprise Security menu bar, select Configure > Data Enrichment > Threat Intelligence Management.
  2. Select New
  3. Provide a descriptive name for the modular input.
  4. Provide a path to the file repository. The file repository must be $SPLUNK_HOME/etc/apps/<app_name>/local/threat_intel
  5. (Optional): Provide a maximum file size in bytes.
  6. (Optional): Set the sinkhole option. The default is to retain all documents.
  7. (Optional): Set the option to remove unusable threat intelligence. This deletes a file after processing it if it contains no actionable threat intelligence.
  8. (Optional): Set the default weight for all threat intelligence documents consumed from this directory.

How Splunk Enterprise Security processes threat intelligence data

A high-level overview of the steps that Splunk Enterprise Security takes to process threat intelligence data.

  1. Splunk ES polls the configured threat intelligence sources for new changes.
  2. The updated threat source data is placed into folders for processing.
    • You can configure these folders on the Threat Intelligence Manager page. By default, these folders are local/data/threat_intel and default/data/threat_intel
  3. The modular input collects and parses the threat sources. After reviewing a threat source, the file is deleted if it contains no actionable threat data
  4. The results are placed into threat collections.
    • You can view the data in the threat collections using the Threat Artifacts dashboard.
  5. The "Lookup Gen" searches extract threat data from the collections and create lookup files.
  6. The "Threat Gen" searches compare the lookup files to the events in the data models. By default, all "Threat Gen" searches look back over the last 45 minutes of events. To match threat intelligence data against events older than 45 minutes, modify the earliest time for the "Threat Gen" searches.
  7. When the "Threat Gen" searches find a match, the search stores the results in the threat_activity index and presented through the Threat Intelligence data model.
  8. A correlation search creates notable events from the threat source matches, and optionally changes risk scores or performs other alert actions.
  9. You can view the notable events using the Incident review dashboard.
PREVIOUS
Threat Intelligence dashboards
  NEXT
Web Intelligence dashboards

This documentation applies to the following versions of Splunk® Enterprise Security: 4.5.0, 4.5.1, 4.5.2, 4.5.3


Comments

The name of the threat intelligence source cannot have whitespace - it will be the name of the file on the filesystem, you may want to indicate that in the documentation.

Niemesrw
December 1, 2016

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters