Define an external lookup in Splunk Web

External lookups use python scripts or binary executables to populate events with field values from an external source.

External lookups are often referred to as scripted lookups, because they are facilitated through the use of a script. See About the external lookup script.

Create an external lookup

If you have Splunk Cloud Platform and want to define external lookups, use an existing Splunk software script or create a private app that contains your custom script. If you are a Splunk Cloud Platform administrator with experience creating private apps, see Manage private apps on your Splunk Cloud Platform deployment in the Splunk Cloud Platform Admin Manual. If you have not created private apps, contact your Splunk account representative for help with this customization.

Prerequisites

• You must be an admin user with .conf and file directory access to upload a script for the lookup.
  

Review

• About lookups
• About the external lookup script
• Configure a time-bounded lookup
• Make your lookup automatic

Steps

1. Add the script for the lookup to the $SPLUNK_HOME/etc/apps/<app_name>/bin directory of your Splunk deployment.
2. Select Settings > Lookups.
3. Select Lookup definitions.
4. Click New.
5. Change the Type to External.
6. Select the destination app.
7. Type a unique Name for your external lookup.
8. Type the command and arguments for the lookup. The command must be the name of the script, for example external_lookup.py. The arguments are the names of the fields that you want to pass to the script, separated by spaces, for example: clienthost clientip.
9. List all of the fields that are supported by the external lookup. The fields must be delimited by a comma followed by a space.
10. (Optional) Make this lookup a time-based lookup.

    Time-based options	Description
    Name of time field	The name of the field in the lookup table that represents the timestamp. This defaults to 0.
    Time format	The strptime format of the timestamp field. You can include subseconds but the Splunk platform will ignore them. This defaults to %s.%Q or seconds from unix epoch in UTC and optional milliseconds.
    Minimum offset	The minimum time (in seconds) that the event timestamp can be later than the lookup entry timestamp for a match to occur. This defaults to 0.
    Maximum offset	The maximum time (in seconds) that the event timestamp can be later than the lookup entry time for a match to occur. This defaults to 2000000000.

11. (Optional) To define advanced options for your lookup, select the Advanced options check box.

    Advanced options	Description
    Minimum matches	The minimum number of matches for each input lookup value. The default value is 0.
    Maximum matches	Enter a number from 1-1000 to specify the maximum number of matches for each lookup value. If time-based, the default value is 1; otherwise, the default value is 1000.
    Default matches	When fewer than the minimum number of matches are present for any given input, the Splunk software provides this value one or more times until the minimum is reached. Splunk software treats NULL values as matching values and does not replace them with the Default matches value.
    Case sensitive match	If the check box is unselected, case-insensitive matching will be performed for all fields in a lookup table. Defaults to true.
    Allow caching	Allows output from lookup scripts to be cached. The default value is true.
    Match type	A comma and space-delimited list of <match_type>(<field_name>) specification to allow for non-exact matching. The available match_type values are WILDCARD, CIDR, and EXACT. EXACT is the default. Specify the fields that use WILDCARD or CIDR in this list.
    Filter lookup	Filter results from the lookup table before returning data. Create this filter like you would a typical search query using Boolean expressions and/or comparison operators. For CSV lookups, filtering is done in memory.

12. Click Save.

Your lookup is now defined as an external lookup and will show up in the list of lookup definitions.

Share the lookup definition

Now that you have created the lookup definition, you need to specify in which apps you want to use the definition.

1. In the Lookup definitions list, for the lookup definition you created, click Permissions.
2. In the Permissions dialog box, under Object should appear in, select All apps to share globally. If you want the lookup to be specific to this app only, select This app only. You can also keep your lookup private by selecting Keep private.
3. Click Save.

   In the Lookup definitions page, your lookup now has the permissions you have set.

Permissions for lookup table files must be at the same level or higher than those of the lookup definitions that use those files.

External lookup example

The following is an example of an external lookup that is delivered with Splunk software. It matches with information from a DNS server. It is not an automatic lookup. You can access it by running a search with the lookup command.

Splunk Enterprise ships with a script located in $SPLUNK_HOME/etc/system/bin/ called external_lookup.py, which is a DNS lookup script that:

• if given a host, returns the IP address.
• if given an IP address, returns the host name.

In the following section, you will use the default script external_lookup.py to create a lookup.
Define the external lookup

1. Select Settings > Lookups.
2. Select Lookup definitions.
3. Click New.
4. Type the lookup name dnslookup in the Name field.
5. Change the Type to External.
6. For the Command, enter the python script name external_lookup.py and the arguments clienthost and clientip as shown below.

external_lookup.py clienthost clientip

7. For the Supported fields, enter clienthost, clientip
8. Click Save.

Share the external lookup

1. In the lookup definitions list, click Permissions.
2. Select All apps for the lookup definition to be shared globally.
3. Click Save.

You can now run a search with the lookup command that uses the dnslookup lookup definition that you created.

sourcetype=access_combined | lookup dnslookup clienthost AS host | stats count by clientip

This search:

• Matches the clienthost field in the external lookup table with the host field in your events.
• Returns a table that provides a count for each of the clientip values that corresponds with the clienthost matches.

This search does not add fields to your events.

You can also design a search that performs a reverse lookup, which in this case returns a host value for each IP address it receives.

sourcetype=access_combined | lookup dnslookup clientip | stats count by clienthost

This reverse lookup search does not include an AS clause. This is because Splunk automatically extracts IP addresses as clientip.

About the external lookup script

External lookups defined in Splunk Web require Python scripts. If you want to create an external lookup that uses a binary executable script, such as a C++ executable, you need to have configuration file access. For more information about writing the external lookup script, see Create external lookups for apps in Splunk Cloud Platform or Splunk Enterprise on the Splunk Developer Portal.

Your external lookup script must input data in the format of an incomplete CSV table and output data in the format of a complete CSV table. The arguments that you pass to the script are the headers for these input and output CSV tables.

In the DNS lookup example, the CSV table contains the two fields clienthost and clientip. The fields that you pass to this script are specified in the lookup definition that you have created. If you do not pass these arguments, the script returns an error:

When you run this search string:

... | lookup dnsLookup clienthost

You are telling Splunk software to:

1. Use the lookup table that you defined in Splunk Web as dnslookup.
2. Pass the values for the clienthost field into the external command script as a CSV table. The CSV table appears as follows:

clienthost,clientip
work.com
home.net

This is a CSV table with clienthost and clientip as column headers, but without values for clientip. The script includes the two headers because they are the fields you specified in the fields_list attribute of the [dnslookup] stanza in the default transforms.conf.

The script outputs the following CSV table, which is used to populate the clientip field in your results:

host,ip
work.com,127.0.0.1
home.net,127.0.0.2

Your script does not have to refer to actual external CSV files. But if it does refer to external CSV files, the filepath references must be relative to the directory where the scripts are located.

See also

In addition to using external lookups to add fields from external sources to events, you can use a scripted input to send data from non-standard sources for indexing or to prepare this data for parsing. For more information, see Create custom data inputs for Splunk Cloud Platform or Splunk Enterprise on the Splunk Developer Portal.

Make the lookup automatic

Instead of using the lookup command in your search when you want to apply a field lookup to your events, you can set the lookup to run automatically. See Define an automatic lookup for more information.

Configure external lookups with .conf files

External lookups can also be configured using .conf files. See Create external lookups for apps in Splunk Cloud Platform or Splunk Enterprise on the Splunk Developer Portal for more information.