Data Source Integration Manual

 


Overview

This documentation does not apply to the most recent version of ES. Click here for the latest version.

Overview

The Splunk App for Enterprise Security gives the security practitioner visibility into security-relevant data captured and indexed within Splunk.

The Splunk App for Enterprise Security's reports and correlation searches are designed to present a unified view of security across heterogeneous vendor data formats. Unlike traditional approaches to doing so that are based on normalizing the data into a common schema at time of data collection, Splunk does so based on search-time mappings to a common set of field names and tags that can be defined at any time after the data is already captured, indexed, and available for ad hoc search.

This means you don't need to write parsers up front before you can start collecting and searching the data. However, you do need to define the field extractions and tags for each data format before the Enterprise Security's reports and correlation searches will work on that data. These tags and field extractions for data formats are defined in technology add-ons. The Splunk App for Enterprise Security ships with a starting set of these add-ons. This guide explains how to create your own.

Technology add-ons contain the Splunk "knowledge" - field extractions, tags, and source types - necessary to extract and normalize detailed information from the data sources at search time and make the resulting information available for reporting. By creating your own technology add-ons, you can easily add new or custom types of data and fully integrate them with the existing dashboards and reports within the Splunk App for Enterprise Security.

Once you have created a technology add-on, you can add it to your Enterprise Security deployment or post it to Splunk Apps to share with others.

What is a technology add-on?

A technology add-on is a Splunk app that extracts knowledge from IT data so that it can be processed by Enterprise Security, as well as other apps that leverage the Common Information Model (CIM). The technology add-on may pull data into Splunk or simply map data that is already coming in. Technology add-ons may conflict with or duplicate other Splunk apps that are already pulling in the same sort of data if they disagree on the source type. The difference between a technology add-on and another Splunk app is compliance with the Common Information Model.

Note: The technology add-on will not require a user interface because reporting will be handled by existing dashboard, centers, and searches in Enterprise Security.

Define a source type for the data

By default Splunk automatically sets a source type for a given data input. Each technology add-on should have at least one source type defined for the data that is captured and indexed within Splunk. This will require an override of the automatic source type that Splunk will attempt to assign to the data source, because the primary source type must be set in the technology add-on in order to apply the right field extractions used by Enterprise Security. A technology add-on can extrapolate key data within the raw text of logs to extract "fields," that are fully compliant with the Common Information Model.

Specifically, a technology add-on performs the following functions:

  • Capture and index the data: If necessary, the technology add-on can import and source type the data into Splunk. This is not required if the data is already in Splunk and source-typed properly.
  • Identify the relevant events that should be visible for security purposes (such as a successful login to a server).
  • Extract fields and aliases that match the CIM so that notable events can be generated and dashboards will function properly.
  • Create tags to categorize the data (for example, tagging all data indicating network communication with the tags "network" and "communicate").
  • Create any additional required fields that are not in the original data source (such as fields that describe the vendor or product).
  • Normalize field values to a common standard (such as changing "Accepted public key" or "Success Audit" into "action=success").

Each technology add-on is designed for a specific data format, such as a particular vendor's firewall or router. Once the technology add-on is created, data sources simply need to be assigned the corresponding source type for the technology add-on to begin processing the data.

Things you need to know to build a technology add-on

See the Knowledge Manager Manual in the core Splunk product documentation for more information about these tasks:

The Splunk App for Enterprise Security package includes a TA-template.zip in the $SPLUNK_HOME/etc/apps directory. This ZIP file includes templates for the common configuration or ".conf" files you will need to create your own technology add-on.

See the "Out-of-the-box source types" in this document for a list of tags and source types that are already available with the Splunk App for Enterprise Security.

Available technology add-ons

Each Enterprise Security technology add-on is specific to a single technology, or portion of a technology that provides all the Splunk knowledge necessary to incorporate that technology into the Splunk App for Enterprise Security. You can use pre-packaged add-ons when available.

Technology add-ons for a number of common source types are bundled with the Splunk App for Enterprise Security. Some of these add-ons may need to be configured for your environment. Each add-on contains a README file that details the required configurations.

This documentation applies to the following versions of ES: 2.0 , 2.2 , 2.2.1 , 2.4 , 2.4.1 View the Article History for its revisions.


You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!