Splunk® Enterprise

Knowledge Manager Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Define an external lookup in Splunk Web

External lookups use python scripts or binary executables to populate events with field values from an external source.

External lookups are often referred to as scripted lookups, because they are facilitated through the use of a script. See About the external lookup script.

Create an external lookup

If you have Splunk Cloud and want to define external lookups, file a Support ticket in order to add a script, or use an existing Splunk software script.

Prerequisites

  • You must be an admin user with .conf and file directory access to upload a script for the lookup.

Review

Steps

  1. Add the script for the lookup to your Splunk deployment.
    The script must be located in either one of two places:
    • $SPLUNK_HOME/etc/searchscripts
    • $SPLUNK_HOME/etc/apps/<app_name>/bin
  2. Select Settings > Lookups.
  3. Select Lookup definitions.
  4. Click New.
  5. Change the Type to External.
  6. Select the destination app.
  7. Type a unique Name for your external lookup.
  8. Type the command and arguments for the lookup. The command must be the name of the script, for example external_lookup.py. The arguments are the names of the fields that you want to pass to the script, separated by spaces, for example: clienthost clientip.
  9. List all of the fields that are supported by the external lookup. The fields must be delimited by a comma followed by a space.
  10. (Optional) Make this lookup a time-based lookup.
    Time-based options Description
    Name of time field The minimum number of matches for each input lookup value. The default value is true.
    Time format Enter a number from 1-1000 to specify the maximum number of matches for each lookup value. If time-based, the default is 1; otherwise, the default is 1000.
    Minimum offset When fewer than the minimum number of matches are present for any given input, the Splunk software provides this value one or more times until the minimum is reached.
    Maximum offset If the check box is selected, case-sensitive matching will be performed for all fields in a lookup table. The default value is true.
  11. (Optional) To define advanced options for your lookup, select the Advanced options check box.
    Advanced options Description
    Minimum matches The minimum number of matches for each input lookup value. The default value is 0.
    Maximum matches Enter a number from 1-1000 to specify the maximum number of matches for each lookup value. If time-based, the default value is 1; otherwise, the default value is 1000.
    Default matches When fewer than the minimum number of matches are present for any given input, the Splunk software provides this value one or more times until the minimum is reached.
    Case sensitive match If the check box is unselected, case-insensitive matching will be performed for all fields in a lookup table. Defaults to true.
    Allow caching Allows output from lookup scripts to be cached. The default value is true.
    Match type A comma and space-delimited list of <match_type>(<field_name>) specification to allow for non-exact matching. The available match_type values are WILDCARD, CIDR, and EXACT. EXACT is the default. Specify the fields that use WILDCARD or CIDR in this list.
    Filter lookup Filter results from the lookup table before returning data. Create this filter like you would a typical search query using Boolean expressions and/or comparison operators.

    For CSV lookups, filtering is done in memory.

  12. Click Save.

Your lookup is now defined as an external lookup and will show up in the list of lookup definitions.

Share the lookup definition

Now that you have created the lookup definition, you need to specify in which apps you want to use the definition.

  1. In the Lookup definitions list, for the lookup definition you created, click Permissions.
  2. In the Permissions dialog box, under Object should appear in, select All apps to share globally. If you want the lookup to be specific to this app only, select This app only. You can also keep your lookup private by selecting Keep private.
  3. Click Save.
    In the Lookup definitions page, your lookup now has the permissions you have set.

Permissions for lookup table files must be at the same level or higher than those of the lookup definitions that use those files.

External lookup example

The following is an example of an external lookup that is delivered with Splunk software. It matches with information from a DNS server. It is not an automatic lookup. You can access it by running a search with the lookup command.

Splunk Enterprise ships with a script located in $SPLUNK_HOME/etc/system/bin/ called external_lookup.py, which is a DNS lookup script that:

  • if given a host, returns the IP address.
  • if given an IP address, returns the host name.

In the following section, you will use the default script external_lookup.py to create a lookup.
Define the external lookup

  1. Select Settings > Lookups.
  2. Select Lookup definitions.
  3. Click New.
  4. Type the lookup name dnslookup in the Name field.
  5. Change the Type to External.
  6. For the Command, enter the python script name external_lookup.py and the arguments clienthost and clientip as shown below.
  7. external_lookup.py clienthost clientip
  8. For the Supported fields, enter clienthost, clientip
  9. Click Save.

Share the external lookup

  1. In the lookup definitions list, click Permissions.
  2. Select All apps for the lookup definition to be shared globally.
  3. Click Save.

You can now run a search with the lookup command that uses the dnslookup lookup definition that you created.

sourcetype=access_combined | lookup dnslookup clienthost AS host | stats count by clientip

This search:

  • Matches the clienthost field in the external lookup table with the host field in your events.
  • Returns a table that provides a count for each of the clientip values that corresponds with the clienthost matches.

This search does not add fields to your events.

You can also design a search that performs a reverse lookup, which in this case returns a host value for each IP address it receives.

sourcetype=access_combined | lookup dnslookup clientip | stats count by clienthost

This reverse lookup search does not include an AS clause. This is because Splunk automatically extracts IP addresses as clientip.

About the external lookup script

Your external lookup script must take in an incomplete CSV file and output a complete CSV file. The arguments that you pass to the script are the headers for these input and output files.

In the DNS lookup example, the CSV file contains the two fields clienthost and clientip. The fields that you pass to this script are specified in the lookup definition that you have created. If you do not pass these arguments, the script returns an error:

Screen Shot 2017-04-14 at 10.45.58 AM.png

When you run this search string:

... | lookup dnsLookup clienthost

You are telling Splunk software to:

  1. Use the lookup table that you defined in Splunk Web as dnslookup.
  2. Pass the values for the clienthost field into the external command script as a CSV file. The CSV file appears as follows:
clienthost,clientip
work.com
home.net

This is a CSV file with clienthost and clientip as column headers, but without values for clientip. The script includes the two headers because they are the fields you specified in the fields_list attribute of the [dnslookup] stanza in the default transforms.conf.

The script outputs the following CSV file, which is used to populate the clientip field in your results:

host,ip
work.com,127.0.0.1
home.net,127.0.0.2

Note: When writing your script, if you refer to any external files, the reference must be relative to the directory where the script is located.

See also

In addition to using external lookups to add fields from external sources to events, you can use a scripted input to send data from non-standard sources for indexing or to prepare this data for parsing. For more information, see the Scripted inputs overview in Developing Views and Apps for Splunk Web.

Make the lookup automatic

Instead of using the lookup command in your search when you want to apply a field lookup to your events, you can set the lookup to run automatically. See Define an automatic lookup for more information.

Configure external lookups with .conf files

External lookups can also be configured using .conf files. See Configure external lookups for more information.

PREVIOUS
Define a CSV lookup in Splunk Web
  NEXT
Define a KV Store lookup in Splunk Web

This documentation applies to the following versions of Splunk® Enterprise: 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters