Download an intelligence feed from the Internet in Splunk Enterprise Security

Splunk Enterprise Security can periodically download an intelligence feed available from the Internet and store it in the $SPLUNK_DB/modinput/threatlist directory. You can then use the inputintelligence search command to use the intelligence in reports, searches, or dashboards. See Example: Add a generic intelligence source to Splunk Enterprise Security.

1. (Optional) Configure a proxy for retrieving intelligence.
2. Add a URL-based intelligence source.

Configure a proxy for retrieving intelligence

If you use a proxy server to send intelligence to Splunk Enterprise Security, configure the proxy options for the intelligence source.

The user must correspond to the name of a Splunk secure stored credential in Credential Management. If you remove an existing proxy user and password in the Intelligence Download Setting editor, the download process no longer references the stored credentials. Removing the reference to the credential does not delete the stored credentials from Credential Management. For more information, see Manage credentials in Splunk Enterprise Security.

1. On the Enterprise Security menu bar, select Configure > Data Enrichment > Intelligence Downloads.
2. Select the download source.
3. Configure the proxy options.
   1. Type a proxy server address. The Proxy Server cannot be a URL. For example, 10.10.10.10 or server.example.com.
   2. Type a proxy server port to use to access the proxy server address.
   3. Type a proxy user credential for the proxy server. Only basic and digest authentication methods are supported. The user must correspond to the name of a credential stored in Credential Management.
   4. (Optional) Type a proxy user realm for the proxy user credential. Use this to specify a proxy user realm for the user credential.
4. Save your changes.

Add a URL-based intelligence source

Add a non-TAXII source of intelligence that is available from a URL on the Internet. For an example of adding a URL-based generic intelligence source, see Example: Add a generic intelligence source to Splunk Enterprise Security.

1. On the Enterprise Security menu bar, select Configure > Data Enrichment > Intelligence Downloads.
2. Type a Name for the download. The name can only contain alphanumeric characters, hyphens, and underscores. The name cannot contain spaces.
3. Click New to add a new intelligence source.
4. Do not select the check box for Sinkhole.
5. Deselect the check box for Is Threat Intelligence.
6. Type a Type for the download. The type identifies the type of information that the feed contains.
7. Type a Description. Describe the information in the feed.
8. Leave the default Weight because the field does not matter for the generic intelligence source.
9. (Optional) Change the default download Interval for the feed. Defaults to 43200 seconds, or every 12 hours.
10. (Optional) Type POST arguments for the feed. You can use POST arguments to retrieve user credentials from Credential Management. Use the format key=$user:<username>$ or key=$user:<username>,realm:<realm>$ to specify a username and realm.
11. Do not use the Maximum age setting.
12. (Optional) If you need to specify a custom User agent string to bypass network security controls in your environment, type it in the format <user-agent>/<version>. For example, Mozilla/5.0 or AppleWebKit/602.3.12. The value in this field must match this regex: ([A-Za-z0-9_.-]+)/([A-Za-z0-9_.-]+). Check with your security device administrator to ensure the string you type here is accepted by your network security controls.
13. Fill out the Parsing Options fields to make sure that your list parses successfully. You must fill out either a delimiting regular expression or an extracting regular expression. You cannot leave both fields blank.

    Field	Description	Example
    Delimiting regular expression	A regular expression string used to split, or delimit, lines in an intelligence source. For complex delimiters, use an extracting regular expression. For parsing options, you can either use a delimiting regular expression or an extracting regular expression, but not both.	, or : or \t
    Extracting regular expression	A regular expression used to extract fields from individual lines of an intelligence source document. Use to extract values in the intelligence source. For parsing options, you can either use a delimiting regular expression or an extracting regular expression, but not both.	^(\S+)\t+(\S+)\t+\S+\t+\S+\t*(\S*)
    Fields	Required if your document is line-delimited. Comma-separated list of fields to be extracted from the intelligence list. Can also be used to rename or combine fields. Description is a required field. Additional acceptable fields are the fields in the corresponding KV Store collection for the threat intelligence, visible in the local lookup files or the DA-ESS-ThreatIntelligence/collections.conf file. Defaults to description:$1,ip:$2.	<fieldname>:$<number>,<field name>.$<number> ip:$1,description:domain_blocklist
    Ignoring regular expression	A regular expression used to ignore lines in an intelligence source. Defaults to ignoring blank lines and comments that begin with #.	^\s*$)
    Skip header lines	The number of header lines to skip when processing the intelligence source.	0
    Intelligence file encoding	If the file encoding is something other than ASCII or UTF8, specify the encoding here. Leave blank otherwise.	latin1

14. (Optional) Change the Download Options fields to make sure that your list downloads successfully.

    Field	Description	Example
    Retry interval	Number of seconds to wait between download retry attempts. Review the recommended poll interval of the intelligence source provider before changing the retry interval.	60
    Remote site user	If the threat feed requires authentication, type the user name to use in remote authentication, if required. The user name you add in this field must match the name of a credential in Credential Management. See Manage input credentials in Splunk Enterprise Security.	buttercup
    Remote site user realm	If the threat feed requires authentication, type the user name to use in remote authentication, if required. The realm you add in this field must match the realm of a credential in Credential Management. See Manage input credentials in Splunk Enterprise Security.	paddock
    Retries	The maximum number of retry attempts.	3
    Timeout	Number of seconds to wait before marking a download attempt as failed.	30

15. (Optional) If you are using a proxy server, fill out the Proxy Options for the feed. See Configure a proxy for retrieving intelligence.
16. Save your changes.

If you are finished adding intelligence sources, see Verify that you have added intelligence successfully in Splunk Enterprise Security.