Splunk® Enterprise

Knowledge Manager Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Configure external lookups

External lookups invoke a script that matches fields in your events with fields in an external source and outputs corresponding fields from that external source and adds them to your events.

External lookups are often referred to as scripted lookups, because they are facilitated through the use of a script. See About the external lookup script for information about how these scripts work.

Create an external lookup

The following is the steps required to create an external lookup for a Splunk Enterprise deployment. If you have Splunk Cloud and want to define external lookups, file a Support ticket.

Prerequisities

Steps

  1. Add the script for the lookup to your Splunk deployment.
    The script must be located in one of two places:
    • $SPLUNK_HOME/etc/searchscripts
    • $SPLUNK_HOME/etc/apps/<app_name>/bin
  2. Add an external lookup stanza to transforms.conf.
    If you want the lookup to be available globally, add its lookup stanza to the version of transforms.conf in $SPLUNK_HOME/etc/system/local/. If you want the lookup to be specific to a particular app, add its stanza to the version of transforms.conf in $SPLUNK_HOME/etc/apps/<app_name>/local/.
    Caution: Do not edit configuration files in $SPLUNK_HOME/etc/system/default.
    The external lookup stanza names the lookup table, provides the script and argument to perform lookups, identifies the script type, and supplies a list of fields that are supported by the script. It uses these required attributes.
    • [<lookup_name>]: The name of the lookup.
    • external_cmd = <string>: The command and arguments issued to perform the lookup. The command must be the name of the script, such as external_lookup.py. The arguments are the names of the fields that you want to pass to the script, separated by spaces, like this: clienthost clientip.
    • external_type = [python|executable|kvstore|geo]: The type of script being used for the lookup. Can be python, for a Python script, or executable, for a binary executable. The kvstore and geo values are reserved for KV store lookups and geospatial lookups, respectively.
    • fields_list = <string>: is a list of all fields that are supported by the external lookup. The fields must be delimited by a comma followed by a space.
  3. (Optional) Set up field/value matching rules for the external lookup.
  4. (Optional) If the data source for the external lookup contains time fields, make the external lookup time-bounded.
  5. (Optional) Make the external lookup automatic by adding a configuration to props.conf.
    If you want the automatic lookup to be available globally, add its lookup stanza to the version of props.conf in $SPLUNK_HOME/etc/system/local/. If you want the lookup to be specific to a particular app, add its stanza to the version of props.conf in $SPLUNK_HOME/etc/apps/<app_name>/local/.
    Caution: Do not edit configuration files in $SPLUNK_HOME/etc/system/default.
  6. Restart Splunk Enterprise to implement your changes.
    If you have set up an automatic lookup, after restart you should see the output fields from your lookup table listed in the fields sidebar. From there, you can select the fields to display in each of the matching search results.

External lookup example

Here's an example of an external lookup that is delivered with Splunk software. It matches with information from a DNS server. It does not have a props.conf component, so it is not an automatic lookup. You access it by running a search with the lookup command.

Splunk Enterprise ships with a script located in $SPLUNK_HOME/etc/system/bin/ called external_lookup.py, which is a DNS lookup script that:

  • if given a host, returns the IP address.
  • if given an IP address, returns the host name.

The configuration for this script resides in $SPLUNK_HOME/etc/system/default/transforms.conf.

[dnslookup]
external_cmd = external_lookup.py clienthost clientip 
fields_list = clienthost,clientip

You can run a search with the lookup command that uses the [dnslookup] stanza from the default transforms.conf.

sourcetype=access_combined | lookup dnslookup clienthost AS host | stats count by clientip

This search:

  • Matches the clienthost field in the external lookup table with the host field in your events</code>
  • Returns a table that provides a count for each of the clientip values that corresponds with the clienthost matches.

This search does not add fields to your events.

You can also design a search that performs a reverse lookup, which in this case returns a host value for each IP address it receives.

sourcetype=access_combined | lookup dnslookup clientip | stats count by clienthost

Note that this reverse lookup search does not include an AS clause. This is because Splunk automatically extracts IP addresses as clientip.

About the external lookup script

Your external lookup script must take in a partially empty CSV file and output a filled-in CSV file. The arguments that you pass to the script are the headers for these input and output files.

In the DNS lookup example above, the CSV file contains two fields: clienthost and clientip. The fields that you pass to this script are the ones you specify in transforms.conf using the external_cmd attribute. If you do not pass these arguments, the script returns an error.

external_cmd = external_lookup.py clienthost clientip

When you run this search string:

... | lookup dnsLookup clienthost

You are telling Splunk software to:

  1. Use the lookup table that you defined in transforms.conf as [dnsLookup]
  2. Pass the values for the clienthost field into the external command script as a CSV file. The CSV file looks like this.
clienthost,clientip
work.com
home.net

This is a CSV file with clienthost and clientip as column headers, but without values for clientip. The script includes the two headers because they are the fields you specified in the fields_list attribute of the [dnslookup] stanza in the default transforms.conf.

The script then outputs the following CSV file, which is used to populate the clientip field in your results:

host,ip
work.com,127.0.0.1
home.net,127.0.0.2

Note: When writing your script, if you refer to any external resources (such as a file), the reference must be relative to the directory where the script is located.

See also

In addition to using external lookups to add fields from external sources to events, you might use a scripted input to send data from non-standard sources for indexing or to prepare this data for parsing. For more information, see the Scripted inputs overview in Developing Views and Apps for Splunk Web.

PREVIOUS
Configure CSV lookups
  NEXT
Configure KV Store lookups

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters