Configure Hunk to use a streaming resource library
To use a streaming resource library you modify
indexes.conf to add a provider family and one or more providers.
[provider-family] stanza encapsulates the common configurations for a family of ERPs. It determines all of the connection information for the type of system your ERPs are running on. The [provider] stanza then provides more specific information such as the path to your Java installation, the path to your Hadoop library, and other MapReduce configurations that you want to use when running searches against that cluster.
indexes.conf to establish a virtual index. This is where you tell Splunk about your Hadoop cluster and about the data you want to access via virtual indexes.
If you do not already have a local copy of
indexes.conf, create a copy and place it into your local directory. In this example we are using:
Note: The following changes to indexes.conf become effective at search time, no restart is necessary.
Create a provider family
[provider-family] stanza encapsulates the connection information for the type of system streaming resource libraries run on.
You can configure multiple streaming resource libraries that share the same executable and certain basic environment variables (defined in the provider family) but then differ in the cluster details (defined in the provider), such as the JobTracker host/port or paths to data.
You must configure the provider family first. You may configure multiple providers for a provider family. Note that provider configurations can override the family.
Tell Hunk about your system by adding the following to a
[provider-family] stanza in
[provider-family:<your provider type, for example: mongodb>] vix.mode=stream vix.command = java vix.command.arg.1 = -Xmx512m vix.command.arg.2 = -classpath
For each different cluster in your system (provider family), you create a separate
When you create your providers, you can use and/or override the information configured for the provider family. In Hunk, the attributes in the
provider stanza are merged with the
family stanza, which it inherits from. The
vix. prefix is stripped from each attribute and the values are passed to the MapReduce job configuration.
[provider:MyMongoProvider] vix.family= <the family configured in provider family, in this example, Mongodb> vix.mongodb.host = localhost:<xxxx>
Set provider configuration variables
Hunk also provides preset configuration variables for each provider you create. You can leave the preset variables in place or edit them as needed. If you want to edit them, see Provider Configuration Variables in the reference section of this manual.
Note: If you are configuring Hunk to work with YARN, you must add new settings. See "Required configuration variables for YARN" in this manual.
Set up your indexes
Once your provider family and providers are configured, you can configure one or more virtual indexes for each provider. You can implement varying degrees of filtering for virtual indexes that use streaming libraries, keeping in mind that some event fields could be generated via: lookups, calculated fields, field aliases, eventtypes, tags etc are specific to Splunk indexing, which the external system might not know how to generate.
For more information about creating virtual indexes, see the following topics:
About streaming resource libraries
About pass-through authentication
This documentation applies to the following versions of Hunk®(Legacy): 6.1, 6.1.1, 6.1.2, 6.1.3, 6.2, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11