Splunk® Enterprise

Knowledge Manager Manual

Download manual as PDF

Download topic as PDF

Configure geospatial lookups

You use geospatial lookups to create queries that return results that Splunk software can use to generate a choropleth map visualization. Choropleth maps cannot be rendered without the data generated by corresponding geospatial lookups.

A geospatial lookup matches location coordinates in your events to location coordinate ranges in a geographic feature collection (a KMZ, or Keyhole Markup Language, file) and outputs fields to your events that provide corresponding geographic feature information that is encoded in the feature collection. This geographic information usually represents a geographic region that can share borders with geographic regions of the same type, like a country, state, province, or county.

Splunk provides two geospatial lookups for the United States and for world countries, enabling you to render choropleth maps of:

  • The USA, divided up into state regions.
  • The world, divided up into countries.

This topic shows you how to create additional geospatial lookups that break up choropleth maps into other types of regions (counties, provinces, timezones, and so on).

For more information about choropleth maps and geographic data visualizations, see Mapping data, in the Dashboards and Visualizations manual.

The FeatureId and featureCollection fields

Geospatial lookups differ from other lookup types in that they are designed to always output these two fields: featureId and featureCollection. The featureId is the "name" of the feature, like "California" or "CA" or whatever is encoded in the feature collection. The featureCollection field provides the name of the lookup in which the feature was found.

If you pipe the output of a geospatial lookup directly into a geom command, the command does not need to be given the lookup name. The geom command detects the featureId and featureCollection fields in the event and uses the lookup to generate the geographic data structures that the Splunk software requires to generate a choropleth map. However, be aware that geographic data structures can be large; so much so that it is strongly discouraged to pipe events into the geom command, as geographic data structures will be attached to every event. Instead, you should first perform stats on the results of your geographic lookup, and only perform geom on a "boiled down" aggregated statistic like count by featureId.

Define a geospatial lookup stanza in transforms.conf

The geospatial lookup stanza provides the location of the geographic feature collection that is to be used as a lookup table. It can optionally include:

  • a feature_id_element attribute.
  • field matching rules.
  • rules for time-bounded lookups.

See "The geospatial lookup stanza format" for details.

If you want a geospatial lookup to be available globally, add its lookup stanza to the version of transforms.conf in $SPLUNK_HOME/etc/system/local/. If you want the lookup to be specific to a particular app, add its stanza to the version of transforms.conf in $SPLUNK_HOME/etc/apps/<app_name>/local/.

Caution: Do not edit configuration files in $SPLUNK_HOME/etc/system/default.

The geospatial lookup stanza format

When you add a geospatial lookup stanza to transforms.conf it should follow this format.

[<lookup_name>]
external_type = geo
filename = <name_of_KMZ_file>
feature_id_element = <XPath_expression>
  • [<lookup_name>] is the name of the lookup.
  • external_type should be set to geo if you are defining a geospatial lookup.
  • filename is the name of the KMZ file that you are using. KMZ files are also referred to as "geographic feature collections."
    • Two feature collections are provided: geo_us_states for the United States, and geo_countries for the countries of the world.
    • You can optionally upload geographic feature collections for other regions and feature types, such as US counties or European provinces.
  • feature_id_element is an optional attribute. It is an XPath expression that defines a path from a Polygon element in the KML file to some other XML element or attribute that contains the name of the feature. This is needed in cases where the typical style of named Placemark element is not in use.
    • feature_id_element may be required in cases where the featureID field generated by the lookup is an empty string, or when the feature collection returns incorrect features by default. In the latter case the feature that you want may be a peer of the default feature or is located relative to the default feature.
    • To determine what path you need, study the geographic feature collection. Each feature in the collection is tagged with Placemark, and each Placemark contains a name that the lookup writes out as featureId fields.
    • The default setting for feature_id_element is /Placemark/name.

Configure a geospatial lookup

Prerequisities

Steps

  1. (Optional) Upload a geographic feature collection to your Splunk deployment, if you need to use a collection other than geo_us_states or geo_countries.
    Geographic feature collections are encoded as KMZ (Keyhole Markup Language) files.
    Upload the feature collection in Settings. Navigate to Settings > Lookups > Lookup table files.
    If you have a KML file, you can convert it to a KMZ file by compressing it and replacing the .zip extension with .kmz.
  2. Create a geospatial lookup stanza in transforms.conf, following the stanza format described in The geospatial lookup stanza format above.
    If you want the lookup to be available globally, add its lookup stanza to the version of transforms.conf in $SPLUNK_HOME/etc/system/local/. If you want the lookup to be specific to a particular app, add its stanza to the version of transforms.conf in $SPLUNK_HOME/etc/apps/<app_name>/local/.
    Caution: Do not edit configuration files in $SPLUNK_HOME/etc/system/default.
  3. (Optional) Set up field/value matching rules for the geospatial lookup.
  4. (Optional) Make the geospatial lookup an automatic lookup by adding a configuration to props.conf.
    If you want the automatic lookup to be available globally, add its lookup stanza to the version of props.conf in $SPLUNK_HOME/etc/system/local/. If you want the lookup to be specific to a particular app, add its stanza to the version of props.conf in $SPLUNK_HOME/etc/apps/<app_name>/local/.
    Caution: Do not edit configuration files in $SPLUNK_HOME/etc/system/default.
  5. Save your .conf file changes.
  6. Restart Splunk Enterprise to implement your changes.
    If you have set up an automatic lookup, after restart you should see the output fields from your lookup table listed in the fields sidebar. From there, you can select the fields to display in each of the matching search results.

Geospatial lookup example

Here is a geospatial lookup called geo_us_states. It is located in $SPLUNK_HOME/etc/system/bin/.

[geo_us_states]
external_type=geo
filename=geo_us_states.kmz

This lookup deals with a geographic feature collection that contains US states.

To use this lookup to build a choropleth map, you need to create a search that queries it in a manner that returns results that can be used to generate the map. The search needs to do all of these things:

  • Indicate an events data source.
  • Query the lookup by matching fields in your events to fields in the KMZ file.
  • Include a transforming command that aggregates the data using featureId, the lookup's geographic output field.
  • Use the geom search command to generate data that can be used to create a choropleth map.

This is a partial choropleth map query. It meets the first two of the four requirements listed above for a choropleth map lookup search. It returns the latitude and longitude for features in the feature collection.

sourcetype=crime_data cc=USA | lookup geo_us_states latitude, longitude

The output of that lookup should look something like this:

_featureIdField featureId featureCollection latitude longitude
featureId AK geo_us_states y x
featureId AL geo_us_states y x

You can update this search to display results for the geom command. Note that the geom command should be preceded by a transforming command operation, such as this one involving count.

This is a full choropleth map query. It retrieves crime event counts by US state and adds the geometry of each state as a geom column.

sourcetype=crime_data cc=USA | lookup geo_us_states latitude, longitude | stats count by featureId | geom

The output of that search should look something like this:

_featureIdField featureId geom count
featureId AK {...} 10
featureId AL {...} 15

_featureIdField is a hidden field that works with the geomfilter post-process search command, when you run a search that contains it. It allows geomfilter to know which field contains the featureId values, even when featureId is renamed to something else.

For example, say you rename featureId to state. If you run geomfilter, it consults the stored search results in the search dispatch folder and looks in the _featureIdField column, where it finds the value state. This causes it to seek the featureId values it needs for its calculations in the state column.

For more information about geospatial lookup search queries, see "Mapping data" in the Dashboards and Visualizations manual.

PREVIOUS
Configure KV Store lookups
  NEXT
Add field matching rules to your lookup configuration

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3


Comments

Hi Geoff,

I checked the conf file, and you are correct about the default path. I've gone ahead and fixed it. Thanks for pointing it out!

I'll follow up with you in a separate email about the more advanced example.

Thanks!
-Maria

Myu splunk, Splunker
February 3, 2017

I believe the following statement is incorrect: "The default setting for feature_id_element is /<placemark>/<name>."

I believe the correct valid XPath is /Placemark/name

Ghendrey splunk, Splunker
February 3, 2017

more complex example on us zip code boundaries. Note the "feature_id_element" Path which extracts the ZCTA5CE10 attribute to use as the feature id:
transforms.conf:
[zip]
external_type = geo
filename = cb_2014_us_zcta510_500k.kmz
feature_id_element = /Placemark/ExtendedData/SchemaData/SimpleData[@name='ZCTA5CE10']

KML/KMZ file:
<Placemark id="cb_2014_us_zcta510_500k">
<visibility>1</visibility>
<description><![CDATA[<center><table><tr><th colspan='2' align='center'><em>Attributes</em></th></tr><tr bgcolor="#E3E3F3">
<th>ZCTA5CE10</th>
<td>36426</td>
</tr><tr bgcolor="">
<th>AFFGEOID10</th>
<td>8600000US36426</td>
</tr><tr bgcolor="#E3E3F3">
<th>GEOID10</th>
<td>36426</td>
</tr><tr bgcolor="">
<th>ALAND10</th>
<td>1258584220</td>
</tr><tr bgcolor="#E3E3F3">
<th>AWATER10</th>
<td>12694318</td>
</tr></table></center>]]></description>
<styleUrl>#KMLStyler</styleUrl>
<ExtendedData>
<SchemaData schemaUrl="#kml_schema_ft_cb_2014_us_zcta510_500k">
<SimpleData name="ZCTA5CE10">36426</SimpleDat

Ghendrey splunk, Splunker
February 3, 2017

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters