Configure geospatial lookups
Use geospatial lookups to create queries that return results that Splunk software can use to generate a choropleth map visualization. Choropleth maps cannot be rendered without the data generated by corresponding geospatial lookups.
A geospatial lookup matches location coordinates in your events to location coordinate ranges in a geographic feature collection known as a Keyhole Markup Zipped (KMZ) or Keyhole Markup Language (KML) file and outputs fields to your events that provide corresponding geographic feature information that is encoded in the feature collection. This information represents a geographic region that shares borders with geographic regions of the same type, such as a country, state, province, or county.
Splunk software provides two geospatial lookups that enable you to render choropleth maps at two levels of granularity:
- The USA, divided into state regions.
- The world, divided into countries.
This topic shows you how to create additional geospatial lookups that break up choropleth maps into other types of regions such as counties, provinces, timezones, and so on.
For more information about choropleth maps and geographic data visualizations, see Mapping data in the Dashboards and Visualizations manual.
You can also define geospatial lookups through Splunk Web. See Define a geospatial lookup in Splunk Web for details.
The FeatureId and featureCollection fields
Geospatial lookups differ from other lookup types in that they are designed to output these two fields: featureId
and featureCollection
. The featureId
is the name of the feature, such as California, CA, or whatever is encoded in the feature collection. The featureCollection
field provides the name of the lookup in which the feature was found.
If you pipe the output of a geospatial lookup directly into a geom
command, the command does not need to be given the lookup name. The geom
command detects the featureId
and featureCollection
fields in the event and uses the lookup to generate the geographic data structures that Splunk software requires to generate a choropleth map. However, geographic data structures can be large. It is strongly discouraged to pipe events into the geom
command, because geographic data structures will be attached to every event. Instead, you should first perform stats
on the results of your geographic lookup, and only perform geom on an aggregated statistic such as count by featureId
.
The Feature Id Element field
The Feature Id Element is an XPath expression that defines a path from a <Placemark>
element in the KML file to some other XML element or attribute that contains the name of the feature. A typical <Placemark>
element associates a name
element with one or more <Polygon>
elements. Each <Placemark>
element is considered a geographic feature, and Splunk software uses its name as the unique Feature Id value that matches the geographic feature to an event in your data.
The default Feature Id Element setting is /Placemark/name
, because the name element is typically tagged with <name>
and located immediately beneath <Placemark>
in the XML hierarchy. If the KML file you are using has a different architecture, provide an XPath expression for Feature Id Element that indicates where the <name>
element is located relative to the <Placemark>
element.
- A
feature_id_element
may be required in cases where thefeatureID
field generated by the lookup is an empty string, or when the feature collection returns incorrect features by default. In the latter case, the<name>
feature may be a peer of the default feature or be located relative to the default feature. - To determine what path you need, study the geographic feature collection. Each feature in the collection is tagged with
Placemark
, and eachPlacemark
contains aname
that the lookup writes out asfeatureId
fields. For an example, see feature_id_element.
- A
Define a geospatial lookup stanza in transforms.conf
The geospatial lookup stanza provides the location of the geographic feature collection that is to be used as a lookup table. It can optionally include:
- a
feature_id_element
attribute. - field matching rules.
- rules for time-bounded lookups.
If you want a geospatial lookup to be available globally, add its lookup stanza to the version of transforms.conf
in $SPLUNK_HOME/etc/system/local/
. If you want the lookup to be specific to a particular app, add its stanza to the version of transforms.conf
in $SPLUNK_HOME/etc/apps/<app_name>/local/
.
Do not edit configuration files in $SPLUNK_HOME/etc/system/default
.
The geospatial lookup stanza format
When you create a geospatial lookup definition, it should follow this format.
[<lookup_name>] external_type = geo filename = <name_of_KMZ_file> feature_id_element = <XPath_expression>
[<lookup_name>]
is the name of the lookup.external_type
should be set togeo
if you are defining a geospatial lookup.filename
is the name of the KMZ file that you are using. KMZ files are also called geographic feature collections.- Two feature collections are provided:
geo_us_states
for the United States, andgeo_countries
for the countries of the world. - You can optionally upload geographic feature collections for other regions and feature types, such as US counties or European provinces.
- Two feature collections are provided:
feature_id_element
is an optional attribute. The default setting is/Placemark/name
. See The Feature Id Element field for details.
XPath and feature_id_element example
The following is an example Placemark
element extracted from a KML file.
<Placemark> <name>Bayview Park</name> <visibility>0</visibility> <styleUrl>#msn_ylw-pushpin15</styleUrl> <Polygon> <tessellate>1</tessellate> <outerBoundaryIs> <LinearRing> <coordinates> -122.3910323868129,37.70819686392956,0 -122.3902274700583,37.71036559447013,0 -122.3885849520798,37.71048623150828,0 -122.38693857563,37.71105170319798,0 -122.3871609118563,37.71133850017815,0 -122.3878019922009,37.71211354629052,0 -122.3879259342663,37.7124647025671,0 -122.3880447162415,37.71294302956619,0 -122.3881688500638,37.71332710079798,0 -122.3883793249067,37.71393746998403,0 -122.3885493512011,37.71421419664032,0 -122.3889255249081,37.71472750657607,0 -122.3887475583787,37.71471048572742,0 -122.3908349856662,37.71698679132378,0 -122.3910091960123,37.71714685034637,0 -122.3935812625442,37.71844151854729,0 -122.3937942835165,37.71824640920892,0 -122.3943907534492,37.71841931125917,0 -122.3946363652554,37.71820562249533,0 -122.3945665820268,37.71790603321808,0 -122.3949430786581,37.71764553372926,0 -122.3953167478058,37.71742547689517,0 -122.3958076264322,37.71693521458138,0 -122.3960283880498,37.7166859403894,0 -122.3987339294558,37.71607634724589,0 -122.3964526840739,37.71310454861037,0 -122.396237742007,37.71265453835174,0 -122.3959650878797,37.7123218368849,0 -122.3955644372275,37.71122536767665,0 -122.3949262649838,37.7082082656386,0 -122.3910323868129,37.70819686392956,0 </coordinates> </LinearRing> </outerBoundaryIs> </Polygon> </Placemark>
Let's take a look at a Placemark
element extracted from a KML file with a different XPath expression than the default.
<Placemark> <name>MyFeature</name> <ExtendedData> <SchemaData> <SimpleData name="placename">foo</SimpleData> <SimpleData name="bar">baz</SimpleData> </SchemaData> ...
The XPath expression for this Placemark fragment would be feature_id_element=/Placemark/ExtendedData/SchemaData/SimpleData[@name='placename']
.
Configure a geospatial lookup
Prerequisities
- See About lookups for more information on lookups.
- See Add field matching rules to your lookup configuration for information on field/value matching rules.
- See Make your lookup automatic for information on configuring an automatic lookup.
- Your role must have the upload_lookup_files capability. Without it you cannot manage geospatial lookups in Splunk Web after you configure them. See Define roles with capabilities in Securing Splunk Enterprise.
Steps
- (Optional) Upload a geographic feature collection to your Splunk deployment, if you need to use a collection other than
geo_us_states
orgeo_countries
.- Geographic feature collections are encoded as KMZ (Keyhole Markup Language) files.
- Upload the feature collection in Settings. Navigate to Settings > Lookups > Lookup table files.
- If you have a KML file, you can convert it to a KMZ file by compressing it and replacing the
.zip
extension with.kmz
.
- Create a geospatial lookup stanza in
transforms.conf
, following the stanza format described in "The geospatial lookup stanza format," above.- If you want the lookup to be available globally, add its lookup stanza to the version of
transforms.conf
in$SPLUNK_HOME/etc/system/local/
. If you want the lookup to be specific to a particular app, add its stanza to the version oftransforms.conf
in$SPLUNK_HOME/etc/apps/<app_name>/local/
.
- Caution: Do not edit configuration files in
$SPLUNK_HOME/etc/system/default
.
- If you want the lookup to be available globally, add its lookup stanza to the version of
- (Optional) Set up field/value matching rules for the geospatial lookup.
- (Optional) Make the geospatial lookup an automatic lookup by adding a configuration to
props.conf
.- If you want the automatic lookup to be available globally, add its lookup stanza to the version of
props.conf
in$SPLUNK_HOME/etc/system/local/
. If you want the lookup to be specific to a particular app, add its stanza to the version ofprops.conf
in$SPLUNK_HOME/etc/apps/<app_name>/local/
.
- Caution: Do not edit configuration files in
$SPLUNK_HOME/etc/system/default
.
- If you want the automatic lookup to be available globally, add its lookup stanza to the version of
- Save your
.conf
file changes. - Restart Splunk Enterprise to implement your changes.
- If you have set up an automatic lookup, after restart you should see the
output
fields from your lookup table listed in the fields sidebar. From there, you can select the fields to display in each of the matching search results.
- If you have set up an automatic lookup, after restart you should see the
Search commands and geospatial lookups
After you save a geospatial lookup stanza and restart Splunk Enterprise, you can interact with the new geospatial lookup through the inputlookup
search command. You can use inputlookup
to quickly check the featureIds of your geospatial lookup or show all geographic features on a Choropleth map visualization.
This example uses the default geo_us_states
lookup.
Prerequisites
- A geospatial lookup, see above.
Steps:
- From the Search and Reporting app, use the
inputlookup
command to search on the contents of your geospatial lookup. | inputlookup geo_us_states
- Check to make sure that your featureIds are in the lookup with the featureId column.
- Click on the Visualization tab.
- Click on Cluster Map and select Chloropleth Map for your visualization.
A chloropleth map displaying the featureIds of your geospatial lookup appears. For more information on chloropleth maps, see Generate a chloropleth map in the Dashboards and Visualizations manual.
Geospatial lookup example
Here is a geospatial lookup called geo_us_states
. It is located in $SPLUNK_HOME/etc/system/bin/
.
[geo_us_states] external_type=geo filename=geo_us_states.kmz
This lookup deals with a geographic feature collection that contains US states.
To use this lookup to build a choropleth map, you need to create a search that queries it in a manner that returns results that can be used to generate the map. The search needs to do all of these things:
- Indicate an events data source.
- Query the lookup by matching fields in your events to fields in the KMZ file.
- Include a transforming command that aggregates the data using
featureId
, the lookup's geographic output field. - Use the
geom
search command to generate data that can be used to create a choropleth map.
This is a partial choropleth map query. It meets the first two of the four requirements listed above for a choropleth map lookup search. It returns the latitude and longitude for features in the feature collection.
sourcetype=crime_data cc=USA | lookup geo_us_states latitude, longitude
The output of that lookup should look something like this:
_featureIdField | featureId | featureCollection | latitude | longitude |
---|---|---|---|---|
featureId | AK | geo_us_states | y | x |
featureId | AL | geo_us_states | y | x |
You can update this search to display results for the geom
command. Note that the geom
command should be preceded by a transforming command operation, such as this one involving count
.
This is a full choropleth map query. It retrieves crime event counts by US state and adds the geometry of each state as a geom
column.
sourcetype=crime_data cc=USA | lookup geo_us_states latitude, longitude | stats count by featureId | geom
The output of that search should look something like this:
_featureIdField | featureId | geom | count |
---|---|---|---|
featureId | AK | {...} | 10 |
featureId | AL | {...} | 15 |
_featureIdField
is a hidden field that works with the geomfilter
post-process search command, when you run a search that contains it. It allows geomfilter
to know which field contains the featureId
values, even when featureId
is renamed to something else.
For example, say you rename featureId
to state
. If you run geomfilter
, it consults the stored search results in the search dispatch folder and looks in the _featureIdField
column, where it finds the value state
. This causes it to seek the featureId
values it needs for its calculations in the state
column.
For more information about geospatial lookup search queries, see Mapping data in the Dashboards and Visualizations manual.
Configure KV Store lookups | Add field matching rules to your lookup configuration |
This documentation applies to the following versions of Splunk® Enterprise: 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.3.0, 9.3.1
Feedback submitted, thanks!