Configure threat intelligence sources
Splunk Enterprise Security provides threat intelligence and context for security monitoring and investigation by comparing the contents of multiple threat sources to indexed events. Enterprise Security supports multiple threat intelligence collection sources, and includes a selection of threat intelligence sources.
Each source represents a collection of knowledge about a specific type of security threat or activity. Splunk administrators can add new sources to ES if those sources are available as a structured data file or stream.
Supported threat collection methods
Any content that is available as a structured text file can be added to ES as a threat source. This includes support for:
- STIX documents and TAXII feeds
- OpenIOC documents
- Content collected via API or URL
- File-based content
- Database-hosted content.
Supported threat intelligence groups
Threat collections are defined in the collections.conf
, and stored in KVStore. To provide for manually entered threat data, each threat collection also has an associated local lookup file. To manually add threat intelligence data open Configure > Data Enrichment on the Enterprise Security menu bar, and select Lists and Lookups.
Supported IOC data types | Threat Collection | Local lookup file |
---|---|---|
X509 Certificates | certificate_intel | Local Certificate Intel |
email_intel | Local Email Intel | |
Files names/hashes | file_intel | Local File Intel |
HTTP | http_intel | Local HTTP Intel |
IP/Domains | ip_intel | Local IP Intel, Local Domain Intel |
Processes | process_intel | Local Process Intel |
Registry entries | registry_intel | Local Registry Intel |
Services | service_intel | Local Service Intel |
Users | user_intel | Local User Intel |
Review the provided threat sources
Splunk Enterprise Security includes pre-configured threat intelligence sources. Each threat source site offers suggestions for polling intervals and other configuration requirements independent of Enterprise Security. Use the source site links to review the provider's documentation for a threat source.
- On the Enterprise Security menu bar, open Configure > Data Enrichment > Threat Intelligence Downloads and review the Description field in all defined threat intelligence sources.
- Use the Threat Intelligence Download Settings page to enable and configure threatlist settings.
- Use the Threat Intelligence Audit dashboard to review the status of all threat download sources.
- Use the Threat Artifacts dashboard to find or query on individual threat elements.
Threat sources included with ES
Threat Source | Threat Provider | Source site |
---|---|---|
Alexa Top 1 Million Sites | Alexa Internet | http://s3.amazonaws.com/alexa-static/ |
Emerging Threats compromised IPs blocklist | Emerging Threats | http://rules.emergingthreats.net/blockrules |
Emerging Threats fwip rules | Emerging Threats | http://rules.emergingthreats.net/fwrules |
Malware domain host list | Hail a TAXII.com | http://hailataxii.com |
iblocklist Logmein | I-Blocklist | http://list.iblocklist.com |
iblocklist Piratebay | I-Blocklist | http://list.iblocklist.com |
iblocklist Proxy | I-Blocklist | http://list.iblocklist.com |
iblocklist Rapidshare | I-Blocklist | http://list.iblocklist.com |
iblocklist Spyware | I-Blocklist | http://list.iblocklist.com |
iblocklist Tor | I-Blocklist | http://list.iblocklist.com |
iblocklist Web attacker | I-Blocklist | http://list.iblocklist.com |
ICANN Top-level Domains List | IANA | http://data.iana.org |
Malware Domain Blocklist | Malware Domains | http://mirror1.malwaredomains.com |
MaxMind GeoIP ASN IPv4 Database | MaxMind | http://download.maxmind.com |
MaxMind GeoIP ASN IPv6 Database | MaxMind | http://download.maxmind.com |
Mozilla Public Suffix List | Mozilla | https://publicsuffix.org |
abuse.ch Palevo C&C IP Blocklist | abuse.ch | https://palevotracker.abuse.ch |
Phishtank Database | Phishtank | http://data.phishtank.com |
SANS blocklist | SANS | http://isc.sans.edu |
abuse.ch ZeuS blocklist (bad IPs only) | abuse.ch | https://zeustracker.abuse.ch |
abuse.ch ZeuS blocklist (standard) | abuse.ch | https://zeustracker.abuse.ch |
Threat Intelligence Download Settings
As an analyst, use the Threat Intelligence Download page to implement, configure, and enable threat intelligence sources that require an API or URL connection, or are lookup based. On the Enterprise Security menu bar, open Configure > Data Enrichment and select Threat Intelligence Downloads.
Select the threat source name field to view the settings.
Threat Intelligence Download Setting |
|
---|---|
Parsing Options |
|
Download Options |
|
Proxy Options: (Optional) |
|
Edit a threat source
Use the Threat Intelligence Download Settings editor to modify information about an existing threat source.
To edit a threat source:
- Go to Configure > Data Enrichment > Threat Intelligence Downloads.
- Select the name of the threat source you want to edit, which opens the Threat Intelligence Download Settings editor.
- Make changes to the fields as required.
- Save changes.
Note: To allow non-admin users to edit threat sources, see "Adding capabilities to a role" in the Installation and Upgrade manual.
Configure threat source retention
To remove threat source data from a threat collection based on date, the threat source must have a Maximum age setting and a retention tracking search enabled.
- On the Enterprise Security menu bar, open Configure > Data Enrichment and select Threat Intelligence Downloads.
- Select a threat source.
- Change the Maximum age setting using a relative time specifier. Example: -7d, -30d. TAXII feeds do not support this setting.
- Enable the retention search for the collection. On the Enterprise Security menu bar, open Settings and select Content Management.
- Search for "retention" using the search filter.
- Enable the retention search for the collection that hosts the threat source. All retention searches are disabled by default.
Add an OpenIOC file
All file-based threat sources must be uploaded to the monitored path on the Enterprise Security search head, or made available on a mounted network share. An OpenIOC file must have a .ioc
extension to be read and parsed.
Default settings
The modular input da_ess_threat_local is configured to monitor the $SPLUNK_HOME/etc/apps/DA-ESS-ThreatIntelligence/local/data/threat_intel
folder, ingest any .ioc
files found, and delete the documents after processing.
To review or modify the da_ess_threat_local inputs settings:
- 1. On the Enterprise Security menu bar, open Settings > Data Inputs and select Threat Intelligence Manager.
- 2. Select the da_ess_threat_local modular input.
- 3. Review or modify the settings as required.
Note: Do not modify the default da_ess_threat_default, or sa_threat_local inputs.
Add a STIX file
All file-based threat sources must be uploaded to the monitored path on the Enterprise Security search head, or made available on a mounted network share. A STIX file must have a .xml
extension to be read and parsed.
Default settings
The modular input da_ess_threat_local is configured to monitor the $SPLUNK_HOME/etc/apps/DA-ESS-ThreatIntelligence/local/data/threat_intel
folder, ingest any .xml
files found, and delete the documents after processing.
To review or modify the da_ess_threat_local inputs settings:
- On the Enterprise Security menu bar, open Settings > Data Inputs and open Threat Intelligence Manager
- Select the da_ess_threat_local modular input.
- Review or modify the settings as required.
Note: Do not modify the default da_ess_threat_default, or sa_threat_local inputs.
Add a TAXII feed
- On the Enterprise Security menu bar, open Configure > Data Enrichment and choose Threat Intelligence Downloads.
- Click New to open the Threat Intelligence Download Settings editor.
- Enter a Type of taxii.
- Enter taxii-specific POST arguments in line, space delimited.
- collection: The name of the data collection from a TAXII feed. Example:
collection="A_TAXII_Feed_Name"
- earliest: The earliest threat data to pull from the TAXII feed. Example:
earliest="-1y"
- (Optional) taxii_username: An optional method to provide a TAXII feed username. Example:
taxii_username="user"
- (Optional) taxii_password: An optional method to provide a TAXII feed password. If username is set without a providing a password, the input will attempt to find the password in Credential Management. Example:
taxii_password="password"
- (Optional) cert_file: Add the certificate file name if the TAXII feed uses certificate authentication. The file name must match exactly and is case sensitive. Example:
cert_file="cert.crt"
- (Optional) key_file: Add the key file name for the certificate if the TAXII feed uses certificate authentication. The file name must match exactly and is case sensitive. Example:
key_file="cert.key"
- collection: The name of the data collection from a TAXII feed. Example:
- Save the changes.
- Review the "Threat Intelligence Audit" dashboard to validate that you properly configured the TAXII feed and Splunk Enterprise Security is downloading it.
Add a TAXII feed with certificate authentication
In a Splunk Cloud deployment, customers must work with Splunk Support to add or change files on cloud-based nodes. Add the certificate and keys to the same app directory that you define the TAXII feed in, for example, DA-ESS-ThreatIntelligence.
- Follow the steps for adding a TAXII feed to Splunk Enterprise Security.
- Add the certificate to the
$SPLUNK_HOME/etc/apps/<app_name>/auth
directory. - Add the private key for the certificate to the same
/auth
directory. - Add the TAXII feed, using the
cert_file
andkey_file
arguments to specify the file names of the certificate and private key file.
Add a URL-based threat source
- On the Enterprise Security menu bar, open Configure > Data Enrichment and choose Threat Intelligence Downloads.
- Click New to open the Threat Intelligence Download Settings editor.
- Enter the information about the threat source, filling in all required fields. For examples, use the pre-configured threat sources provided in Enterprise Security.
- Save the changes when you are done.
- Validate the source is properly configured and downloading by reviewing the "Threat Intelligence Audit" dashboard.
Adding a custom source
Add a custom threat source to Splunk Enterprise Security as a local lookup file. A lookup-based threat source can add data to any of the supported threat intelligence groups, such as file or IP intelligence.
Prerequisite: Identify whether the custom threat source is certificate, domain, email, file, HTTP, IP, process, registry, service, or user intelligence.
Add a custom threat source
Based on the type of intelligence you add to Splunk Enterprise Security, you need to identify the headers for the csv file.
- Select Configure > Data Enrichment > Lists and Lookups.
- Locate the lookup file that matches the local threat intel you are providing. For example, Local File Intel.
- Open the relevant lookup to view the required headers.
- Create a new
.csv
file with a header row containing the required fields. - Add the threat data to the
.csv
file.
After you create the lookup file, you need to add it to Splunk Enterprise Security.
- On the Splunk platform menu bar, select Settings > Lookups
- Next to Lookup table files, click Add New.
- Select a Destination App of SA-ThreatIntelligence.
- Upload the
.csv
file you created. - Enter a Destination filename for the file. For example,
threatindicatorszerodayattack.csv
. - Save.
After adding the threat intel lookup to ES, set appropriate permissions so ES can use the file.
- Open Lookup table files.
- Locate the lookup file that you added and select Permissions.
- Set Object should appear in to All apps.
- Select Read access for Everyone.
- Select Write access for admin.
- Save.
Define the lookup so that Splunk ES can import it and understand what type of intelligence you are adding.
- On the Splunk platform menu bar, select Settings > Lookups.
- Next to Lookup definitions, click Add New.
- Select a Destination App of SA-ThreatIntelligence.
- Enter a name for the threat source. The name you enter here is used to define the threatlist in the input stanza. For example,
zero_day_attack_threat_indicators_list
. - Select a Type: of File based.
- Select the Lookup File: that you added in step one. For example,
threatindicatorszerodayattack.csv
. - Save.
Set permissions on the lookup definition so that the lookup functions properly.
- Open Lookup definitions
- Locate the definition you added in step four and select Permissions.
- Set Object should appear in to All apps.
- Set Read access for Everyone.
- Set Write access for admin.
- Save.
Add a threat source input stanza that corresponds to the lookup file so that ES knows where to find the new threat intelligence.
- Select Configure > Data Enrichment > Threat Intelligence Downloads.
- Choose a threat source input that matches your new content. For example,
local_file_intel
. - Click Clone in the Actions column.
- Enter a Name. For example, Zero day attack threat indicators.
- Enter a Type. For example, IOCs.
- Enter a Description. For example, File-based threat indicators from zero day malware.
- Enter a URL that references the lookup definition you created in step three:
lookup://zero_day_attack_threat_indicators_list
. - (Optional) Change the default Weight for the threat data.
- (Optional) Change the default Retry interval for the lookup.
Verify that you added the custom threat source successfully and ES is ingesting it accurately.
- Select Audit > Threat Intelligence Audit.
- Review the Threat Intelligence Downloads to confirm that the threat source is enabled.
- Review the Threat Intelligence Audit Events to determine if there are errors associated with the threat lookup name.
- Select Advanced Threat > Threat Artifacts.
- Search for an object from the threat source to confirm that Splunk ES can find it.
Note: If one of the pre-configured inputs isn't a match for your threat intel source, select New. Fill out the required fields in the newly created threat source input.
Add database-hosted content
For an example of how to add database-hosted content, see this blog post discussing custom threat feed integration with Enterprise Security (http://blogs.splunk.com/2014/03/10/custom-threat-feed-integration-with-enterprise-security).
Add threat data manually
Each threat collection has a pre-configured lookup file to accept manually entered threat data.
- On the Enterprise Security menu bar, open Configure > Data Enrichment and select Lists and Lookups.
- Find the pre-configured lookup that matches the threat collection data you are providing. Example: Local Certificate intel.
- Select the lookup to open the lookup editor.
- Update the lookup, populating all of the fields for each row (right-click on the row, and choose Insert Row Below).
- Click Save when finished.
Configure a custom folder for threat sources
Enterprise Security provides inputs configured to monitor folders and ingest threat intelligence sources. Use the Threat Intelligence Manager page to review, add, or change inputs for monitoring folders containing threat sources.
Default settings
The modular input da_ess_threat_local is configured to monitor the $SPLUNK_HOME/etc/apps/DA-ESS-ThreatIntelligence/local/data/threat_intel
folder, ingest any .csv
, .ioc
and .xml
files found, and delete the documents after processing.
To review or modify the da_ess_threat_local inputs settings:
- Navigate to Settings > Data Input > Threat Intelligence Manager
- Select the da_ess_threat_local modular input.
- Review or modify the settings as required.
Note: Do not modify the default da_ess_threat_default, or sa_threat_local inputs.
Create a new input monitor for threat sources
- Navigate to Settings > Data Input > Threat Intelligence Manager
- Select New
- Provide a descriptive name for the modular input.
- Provide a path to the file repository.
- (Optional): Provide a maximum file size in bytes.
- (Optional): Set the sinkhole option. The default is to retain all documents.
- (Optional): Set the default weight for all threat intelligence documents consumed from this directory.
How Splunk Enterprise Security processes threat intelligence data
A high-level overview of the steps that Splunk Enterprise Security takes to process threat intelligence data.
- Splunk ES polls the configured threat intelligence sources for new changes.
- You can configure new threat intelligence sources on the Threat Intelligence Downloads dashboard.
- You can view the latest status changes to the threat sources using the Threat Intelligence Audit dashboard.
- The updated threat source data is placed into folders for processing.
- You can configure these folders on the Threat Intelligence Manager page. By default, these folders are
local/data/threat_intel
anddefault/data/threat_intel
- You can configure these folders on the Threat Intelligence Manager page. By default, these folders are
- The modular input collects and parses the threat sources.
- The results are placed into threat collections.
- You can view the data in the threat collections using the Threat Artifacts dashboard.
- The "Lookup Gen" searches extract threat data from the collections and create lookup files.
- The "Threat Gen" searches compare the lookup files to the events in the data models. By default, all "Threat Gen" searches look back over the last 24 hours of events. To match threat intelligence data against events older than 24 hours, modify the earliest time for the "Threat Gen" searches.
- When the "Threat Gen" searches find a match, the search stores the results in the
threat_intel
index and presented through theThreat Intelligence
data model.- You can view the results using the Threat Activity dashboard.
- A correlation search creates notable events from the threat source matches, and optionally changes risk scores or performs other alert actions.
- You can view the notable events using the Incident review dashboard.
Threat Intelligence dashboards | Advanced Threat dashboards |
This documentation applies to the following versions of Splunk® Enterprise Security: 4.1.0, 4.1.1, 4.1.2, 4.1.3, 4.1.4
Feedback submitted, thanks!