Example 1: Blue Coat proxy logs
- Define a source type for the data
- Handle timestamp recognition
- Configure line breaking
- Understand your data and Enterprise Security dashboards
- Create extractions
- Verify field extractions
- Document technology add-on
- Blue Coat ProxySG technology add-on
- Using this technology add-on
- Package the technology add-on
Example 1: Blue Coat proxy logs
This example shows how to create a technology add-on for Blue Coat web proxy logs from hosts running SGOS 22.214.171.124. (Earlier versions of Blue Coat require additional work to extract some of the required fields.)
Choose a folder name for the technology add-on
Name this technology add-on
Splunk_TA-bluecoat. Create the technology add-on folder at
$SPLUNK_HOME/etc/apps/Splunk_TA-bluecoat. Extract the template located at
Use the sample files in the
template.zip to create the configuration files needed for this technology add-on.
Step 1: Capture and index the data
To get started, set up a data input in order to get Blue Coat web proxy data into the Splunk App for Enterprise Security. Blue Coat submits logs via syslog over port UDP\514, so use a network-based data input. Since this add-on will not be able to determine the source type based on the content of the logs (because it is using network-based data input), the source type must be manually defined as "bluecoat".
Manually define the source type as "bluecoat".
Define a source type for the data
This add-on only handles one type of logs, so we only need a single source type, which we will name "bluecoat". We define a source type in
default/props.conf of Splunk_TA-bluecoat:
[source::....bluecoat] sourcetype = bluecoat [bluecoat]
Handle timestamp recognition
Looking at the events, we see that Splunk successfully parses the date and time so there is no need to customize the timestamp recognition.
Configure line breaking
Each log message is separated by an end-line, therefore, we need to disable line merging to prevent multiple messages from being combined. Line merging is disabled by setting
SHOULD_LINEMERGE to false in
[source::....bluecoat] sourcetype = bluecoat [bluecoat] SHOULD_LINEMERGE = false
Step 2: Identify relevant IT security events
Next identify which Enterprise Security dashboards we want to display the Blue Coat web proxy data.
Understand your data and Enterprise Security dashboards
The data from the Blue Coat web proxy should go into the Proxy Center and the Proxy Search dashboards.
Step 3: Create field extractions and aliases
Now create the field extractions that will populate the fields according to the Common Information Model. Before we begin, restart Splunk so that it will recognize the technology add-on and source type defined in the previous steps.
Review the Common Information Model and the Dashboard Requirements Matrix and determine that the Blue Coat technology add-on needs to define the following fields to work with the Proxy Center and Proxy Search:
|Domain||Sub-Domain||Field Name||Data Type|
Blue Coat data consists of fields separated by spaces. To parse this data we can use automatic key/value pair extraction and define the name of the fields using the associated location.
Start by analyzing the data and identifying the available fields. We see that the data contains a lot of duplicate or similar fields that describe activity between different devices. For clarity, create the following temporary naming convention to help characterize the fields:
|s||Relates to the proxy server|
|c||Relates to the client requesting access through the proxy server|
|cs||Relates to the activity between the client and the proxy server|
|sc||Relates to the activity between the proxy server and the client|
|rs||Relates to the activity between the remote server and the proxy server|
Identify the existing fields and give them temporary names, listed here in the order in which they occur.
Once we know what the fields are and what they contain, we can map the relevant fields to the fields required in the Common Information Model (CIM).
|Blue Coat Field||Field in CIM|
Next create the field extractions. Since the Blue Coat data is space-delimited, we start by setting the delimiter to the single space character. Then define the order of the field names. Add the following to
defaults/transforms.conf in the Splunk_TA-bluecoat folder.
[auto_kv_for_bluecoat] DELIMS = " " FIELDS = date,time,src,bytes_in,duration,action,status,rs_status,rs_bytes, bytes_out,cs_auth_type,user,sc_filter_result,http_method,dest, app_version,sr_bytes,url,http_referrer,http_content_type, http_user_agent,http_cookie
DELIMS: defines the delimiter between fields (in this case a space)
FIELDS: defines the fields in the order in which they occur
Add a reference to the name of the transform to
default/props.conf in the Splunk_TA-bluecoat folder to enable it:
[bluecoat] SHOULD_LINEMERGE = false REPORT-0auto_kv_for_bluecoat = auto_kv_for_bluecoat
Now define the product and vendor. Since these fields are not included in the data, they will be populated using a lookup.
## props.conf REPORT-vendor_for_bluecoat = vendor_static_bluecoat REPORT-product_for_bluecoat = product_static_proxy ## transforms.conf [vendor_static_bluecoat] REGEX = (.) FORMAT = vendor::"Bluecoat" [product_static_proxy] REGEX = (.) FORMAT = product::"Proxy" ## bluecoat_vendor_info.csv sourcetype,vendor,product bluecoat,Bluecoat,Proxy
Note: Do not under any circumstances assign static strings to event fields as this will prevent these fields from being searched in Splunk. Instead, fields that do not exist should be mapped with a lookup. See "Static strings and event fields" in this document for more information.
Now we enable the transforms in
default/props.conf using REPORT lines to call out the
transforms.conf sections that will enable proper field extractions:
[bluecoat] SHOULD_LINEMERGE = false REPORT-0auto_kv_for_bluecoat = auto_kv_for_bluecoat REPORT-product_for_bluecoat = product_static_Proxy REPORT-vendor_for_bluecoat = vendor_static_Bluecoat
Generally, it is a good idea to add a field extraction that will process data that is uploaded as a file into Splunk directly. To do this, add a property to
default/props.conf that indicates that any files ending with the extension
bluecoat should be processed as Blue Coat data:
[source::.bluecoat] sourcetype = bluecoat [bluecoat] SHOULD_LINEMERGE = false REPORT-0auto_kv_for_bluecoat = auto_kv_for_bluecoat REPORT-product_for_bluecoat = product_static_Proxy REPORT-vendor_for_bluecoat = vendor_static_Bluecoat
Verify field extractions
Now that the field extractions have been create, we need to verify that the extractions are working correctly. First, restart Splunk so that the changes can be applied. Next, search for the source type in the Search dashboard:
When the results are displayed, select Pick Fields and choose the fields that ought to be populated. Once the fields are selected, click the field name to display a list of the values appear in this field:
We created the field extractions, now we need to add the tags.
The Common Information Model requires that proxy data be tagged with "web" and "proxy":
|Network Protection||Proxy||proxy||web proxy|
Create tag properties
Now that we know what tags to create, we can create the tags.
First, we need to create an event type to which we can assign the tags. To do so, we create a stanza in
default/eventtypes.conf that assigns an event type of bluecoat to data with the source type of bluecoat:
[bluecoat] search = sourcetype=bluecoat
Next, we assign the tags in
default/tags.confin our technology add-on:
[eventtype=bluecoat] web = enabled proxy = enabled
Now that the tags have been created, we can verify that they are being applied. We search for the source type in the search view:
Use the field picker to display the field "tag::eventtype" at the bottom of each event. Review the entries and look for the tag statements at the bottom of the log message.
Check Enterprise Security dashboards
Now that the tags and field extractions are complete, the data should show up in the Splunk App for Enterprise Security. The extracted fields and the defined tags fit into the Proxy Center dashboard; therefore, the
bluecoat data ought to be visible on this dashboard. However, the
bluecoat data will not be immediately available since Enterprise Security uses summary indexing. It may take up to an hour after Splunk has been restarted for the data to appear. After an hour or so, the dashboard begins populating with Blue Coat data:
Step 5: Document and package the technology add-on
Packaging the technology add-on makes it easy to deploy to other Splunk installs in your organization, or to post to Splunkbase for use in other organizations. To package a technology add-on you only need to provide some simple documentation an compress the files.
Document technology add-on
Now that the technology add-on is complete, we create a
README file under the root technology add-on directory:
===Blue Coat ProxySG technology add-on=== Author: John Doe Version/Date: 1.4 October 2012 Supported product(s): This technology add-on supports Blue Coat ProxySG 126.96.36.199 Source type(s): This technology add-on will process data that is source-typed as "bluecoat". Input requirements: N/A ===Using this technology add-on=== Configuration: Manual To use this technology add-on, manually configure the data input with the corresponding source type and it will optimize the data automatically.
Package the technology add-on
Package up the Blue Coat technology add-on by converting it into a zip archive named
Splunk_TA-bluecoat.zip. To share the technology add-on, go to Splunkbase, click upload an app and follow the instructions for the upload.