Use the CIM to normalize OSSEC data
This example demonstrates how to create an add-on for OSSEC, an open-source host-based intrusion detection system (IDS).
Note: Splunk offers an add-on that provides the capabilities in this example for OSSEC data, so you do not need to build one yourself. Find the add-on on Splunkbase at https://splunkbase.splunk.com/app/2808/.
This example illustrates how to perform the following tasks:
- Evaluate data in the context of the CIM and Splunk Enterprise Security requirements.
- Use regular expressions to extract the necessary fields.
- Convert the values in the severity field to match the format required in the Common Information Model.
- Create multiple event types to identify different types of events within a single data source.
- Package the results as an add-on to share with the community.
Step 1: Get the data in
To get started, set up a data input in order to get OSSEC data into Splunk Enterprise Security. OSSEC submits logs via syslog
over port UDP:514, so you can use a network-based data input. Once you have built and installed the add-on, it will detect OSSEC data and automatically assign it the correct source type when it receives data over UDP port 514.
1. Configure folder and source type naming. Create a folder for the new add-on at $SPLUNK_HOME/etc/apps/Splunk_TA-ossec
. (The name of this add-on is Splunk_TA-ossec
.) For this add-on, use the source type ossec
to identify data associated with the OSSEC intrusion detection system.
2. Configure line breaking. Because each log message separates itself with an end-line, you must disable line-merging to prevent the add-on from combining multiple messages. To do so, set SHOULD_LINEMERGE
to false in the default/props.conf
.
For example:
[source::....ossec] sourcetype=ossec [ossec] SHOULD_LINEMERGE = false [source::udp:514] TRANSFORMS-force_sourcetype_for_ossec_syslog = force_sourcetype_for_ossec
3. Restart the Splunk platform so that it recognizes the add-on and source type you defined.
Step 2: Examine your data to identify relevant IT security events
1. Identify which events you want to display in the Intrusion Center dashboard in Splunk Enterprise Security. Use the CIM reference tables to find fields that are relevant for intrusion detection data. The data maps to the Intrusion Detection data model.
2. Open the reference table for that model to use as a reference. In Splunk Web, open the Data Model Editor for the IDS model to refer to the dataset structure and constraints.
Step 3: Tag events
1. Identify the tags you must create. The Common Information Model dictates that you must tag intrusion detection data with "attack
" and "ids
" to indicate that the data comes from an attack detection event.
2. Create the event types to which you can assign tags. To do so, create an event type in the eventtypes.conf
file that assigns the "ossec_attack" event type to all data with the source type ossec
and a severity_id
greater than or equal to 6.
[ossec_attack] search = sourcetype=ossec severity_id >=6 #tags = ids attack
3. Assign the tags in the tags.conf
file.
[eventtype=ossec_attack] attack = enabled ids = enabled
Step 4: Verify tags
1. Verify that your Splunk platform applies the tags correctly. In the Searching and Reporting app, search for the source type as follows:
sourcetype="ossec"
2. Review the entries to find the tag statements under the log message.
Step 5: Normalize fields
1. Create the field extractions that populate the fields according to the Common Information Model. First, review the Common Information Model and the Dashboard requirements matrix for Splunk Enterprise Security in Administer Splunk Enterprise Security to determine that the OSSEC add-on needs to include the following fields in order to populate the Intrusion Center and Intrusion Search dashboards in Splunk Enterprise Security:
Domain | Field Name | Data Type |
---|---|---|
Intrusion Detection | signature | string |
Intrusion Detection | category | string |
Intrusion Detection | severity | string |
Intrusion Detection | src | string |
Intrusion Detection | dest | string |
Intrusion Detection | ids_type
ids_type of |
string |
You can also populate additional CIM fields, if they are available in your data.
2. Create extractions. OSSEC data is in a proprietary format that does not use key-value pairs or any kind of standard delimiter between the fields. Therefore, you have to write a regular expression to parse the individual fields. The following outlines a log message highlighting the relevant fields.
The severity field includes an integer, while the Common Information Model requires a string. Therefore, extract this into a different field, severity_id
, then perform the necessary conversion later to produce the severity field.
3. Extract the Location, Message, severity_id, signature and src_ip fields. To do so, edit the default/transforms.conf
file to add a stanza that extracts the fields you need to the following:
[force_sourcetype_for_ossec] DEST_KEY = MetaData:Sourcetype REGEX = ossec\: FORMAT = sourcetype::ossec [kv_for_ossec] REGEX = Alert Level\:\s+([^;]+)\;\s+Rule\:\s+([^\s]+)\s+- \s+([^\.]+)\.{0,1}\;\s+Location\:\s+([^;]+)\;\s*(srcip\:\s+(\d{1,3} \.\d{1,3}\.\d{1,3}\.\d{1,3})\;){0,1}\s*(user\:\s+([^;]+)\;){0,1}\s*(.*) FORMAT = severity_id::"$1" signature_id::"$2" signature::"$3" Location::"$4" src_ip::"$6" user::"$8" Message::"$9"
4. Enable the statement in the default/props.conf
file in your add-on folder.
[source::....ossec] sourcetype=ossec [ossec] SHOULD_LINEMERGE = false REPORT-0kv_for_ossec = kv_for_ossec [source::udp:514] TRANSFORMS-force_sourcetype_for_ossec_syslog = force_sourcetype_for_ossec
5. Extract the dest
field. Some of the fields need additional field extraction to fully match the Common Information Model. The Location
field includes several separate fields within a single field value. Create the following stanza in the default/props.conf
file to extract the destination DNS name, destination IP address, and original source address.
[source::....ossec] sourcetype=ossec [ossec] SHOULD_LINEMERGE = false [source::udp:514] TRANSFORMS-force_sourcetype_for_ossec_syslog = force_sourcetype_for_ossec [kv_for_ossec] REGEX = Alert Level\:\s+([^;]+)\;\s+Rule\:\s+([^\s]+)\s+- \s+([^\.]+)\.{0,1}\;\s+Location\:\s+([^;]+)\;\s*(srcip\:\s+(\d{1,3 }\.\d{1,3}\.\d{1,3}\.\d{1,3})\;){0,1}\s*(user\:\s+([^;]+)\;){0,1}\s*(.*) FORMAT = severity_id::"$1" signature_id::"$2" signature::"$3" Location::"$4" src_ip::"$6" user::"$8" Message::"$9" [Location_kv_for_ossec] SOURCE_KEY = Location REGEX = (\(([^\)]+)\))*\s*(.*?)(->)(.*) FORMAT = dest_dns::"$2" dest_ip::"$3" orig_source::"$5"
6. Enable the statement in the default/props.conf
file in the add-on folder:
[source::....ossec] sourcetype=ossec [ossec] SHOULD_LINEMERGE = false REPORT-0kv_for_ossec = kv_for_ossec, Location_kv_for_ossec [source::udp:514] TRANSFORMS-force_sourcetype_for_ossec_syslog = force_sourcetype_for_ossec
7. The "Location_kv_for_ossec
" stanza creates two fields that represent the destination (either by the DNS name or destination IP address). You need a single field named "dest
" that represents the destination. To handle this, add stanzas to default/transforms.conf
that populate the destination field if the dest_ip
or dest_dns
is not empty. Note that the regular expressions below work only if the string has at least one character. This ensures that the destination is not an empty string.
[source::....ossec] sourcetype=ossec [ossec] SHOULD_LINEMERGE = false [source::udp:514] TRANSFORMS-force_sourcetype_for_ossec_syslog = force_sourcetype_for_ossec [kv_for_ossec] REGEX = Alert Level\:\s+([^;]+)\;\s+Rule\:\s+([^\s]+)\s+- \s+([^\.]+)\.{0,1}\;\s+Location\:\s+([^;]+)\;\s*(srcip\:\s+(\d{1,3}\.\d{1,3 }\.\d{1,3}\.\d{1,3})\;){0,1}\s*(user\:\s+([^;]+)\;){0,1}\s*(.*) FORMAT = severity_id::"$1" signature_id::"$2" signature::"$3" Location::"$4" src_ip::"$6" user::"$8" Message::"$9" [Location_kv_for_ossec] SOURCE_KEY = Location REGEX = (\(([^\)]+)\))*\s*(.*?)(->)(.*) FORMAT = dest_dns::"$2" dest_ip::"$3" orig_source::"$5" [dest_ip_as_dest] SOURCE_KEY = dest_ip REGEX = (.+) FORMAT = dest::"$1" [dest_dns_as_dest] SOURCE_KEY = dest_dns REGEX = (.+) FORMAT = dest::"$1"
8. Enable the field extractions you created in the default/transforms.conf
file by adding them to the default/props.conf
file. Set up your field extractions to ensure that you get the DNS name instead of the IP address if both are available. To do so, place the "dest_dns_as_dest
" transform first; the Splunk platform processes field extractions in order, stopping on the first one that matches.
[source::....ossec] sourcetype=ossec [ossec] SHOULD_LINEMERGE = false REPORT-0kv_for_ossec = kv_for_ossec, Location_kv_for_ossec REPORT-dest_for_ossec = dest_dns_as_dest,dest_ip_as_dest [source::udp:514] TRANSFORMS-force_sourcetype_for_ossec_syslog = force_sourcetype_for_ossec
9. Extract the src
field. You populated the source IP into the field "src_ip
", but the CIM requires a separate "src
" field as well. To create the separate field, add a field alias in the default/props.conf
file that populates the "src
" field with the value in "src_ip
".
[source::....ossec] sourcetype=ossec [ossec] SHOULD_LINEMERGE = false REPORT-0kv_for_ossec = kv_for_ossec, Location_kv_for_ossec REPORT-dest_for_ossec = dest_dns_as_dest,dest_ip_as_dest FIELDALIAS-src_for_ossec = src_ip as src [source::udp:514] TRANSFORMS-force_sourcetype_for_ossec_syslog = force_sourcetype_for_ossec
10. Normalize the severity field. The OSSEC data includes a field that contains an integer value for the severity. However, the Common Information Model requires a string value for the severity. Therefore, you need to convert the input value to a value that matches the Common Information Model. Do this using a lookup table. Map the "severity_id
" values to the corresponding severity string, then create a CSV file in lookups/ossec_severities.csv
.
severity_id,severity 0,informational 1,informational 2,informational 3,informational 4,error 5,error 6,low 7,low 8,low 9,medium 10,medium 11,medium 12,high 13,high 14,high 15,critical
11. Add the lookup file definition to the default/transforms.conf
file.
[source::....ossec] sourcetype=ossec [ossec] SHOULD_LINEMERGE = false [source::udp:514] TRANSFORMS-force_sourcetype_for_ossec_syslog = force_sourcetype_for_ossec [kv_for_ossec] REGEX = Alert Level\:\s+([^;]+)\;\s+Rule\:\s+([^\s]+)\s+- \s+([^\.]+)\.{0,1}\;\s+Location\:\s+([^;]+)\;\s*(srcip\:\s+(\d{1,3 }\.\d{1,3}\.\d{1,3}\.\d{1,3})\;){0,1}\s*(user\:\s+([^;]+)\;){0,1}\s*(.*) FORMAT = severity_id::"$1" signature_id::"$2" signature::"$3" Location::"$4" src_ip::"$6" user::"$8" Message::"$9" [Location_kv_for_ossec] SOURCE_KEY = Location REGEX = (\(([^\)]+)\))*\s*(.*?)(->)(.*) FORMAT = dest_dns::"$2" dest_ip::"$3" orig_source::"$5" [dest_ip_as_dest] SOURCE_KEY = dest_ip REGEX = (.+) FORMAT = dest::"$1" [dest_dns_as_dest] SOURCE_KEY = dest_dns REGEX = (.+) FORMAT = dest::"$1" [ossec_severities_lookup] filename = ossec_severities.csv
12. Add the lookup to default/props.conf
:
[source::....ossec] sourcetype=ossec [ossec] SHOULD_LINEMERGE = false REPORT-0kv_for_ossec = kv_for_ossec, Location_kv_for_ossec REPORT-dest_for_ossec = dest_dns_as_dest,dest_ip_as_dest FIELDALIAS-src_for_ossec = src_ip as src LOOKUP-severity_for_ossec = ossec_severities_lookup severity_id OUTPUT severity [source::udp:514] TRANSFORMS-force_sourcetype_for_ossec_syslog = force_sourcetype_for_ossec
13. Define the vendor and product fields. The last fields to populate are the vendor and product fields. To populate these, add stanzas to the default/transforms.conf
file to statically define them:
[source::....ossec] sourcetype=ossec [ossec] SHOULD_LINEMERGE = false [source::udp:514] TRANSFORMS-force_sourcetype_for_ossec_syslog = force_sourcetype_for_ossec [kv_for_ossec] REGEX = Alert Level\:\s+([^;]+)\;\s+Rule\:\s+([^\s]+)\s+- \s+([^\.]+)\.{0,1}\;\s+Location\:\s+([^;]+)\;\s*(srcip\:\s+(\d{1,3}\.\d{1,3}\.\d{1,3 }\.\d{1,3})\;){0,1}\s*(user\:\s+([^;]+)\;){0,1}\s*(.*) FORMAT = severity_id::"$1" signature_id::"$2" signature::"$3" Location::"$4" src_ip::"$6" user::"$8" Message::"$9" [Location_kv_for_ossec] SOURCE_KEY = Location REGEX = (\(([^\)]+)\))*\s*(.*?)(->)(.*) FORMAT = dest_dns::"$2" dest_ip::"$3" orig_source::"$5" [dest_ip_as_dest] SOURCE_KEY = dest_ip REGEX = (.+) FORMAT = dest::"$1" [dest_dns_as_dest] SOURCE_KEY = dest_dns REGEX = (.+) FORMAT = dest::"$1" [ossec_severities_lookup] filename = ossec_severities.csv [product_static_hids] REGEX = (.) FORMAT = product::"HIDS" [vendor_static_open_source_security] REGEX = (.) FORMAT = vendor::"Open Source Security"
14. Enable the stanzas in the default/props.conf
file.
[source::....ossec] sourcetype=ossec [ossec] SHOULD_LINEMERGE = false REPORT-0kv_for_ossec = kv_for_ossec, Location_kv_for_ossec REPORT-dest_for_ossec = dest_dns_as_dest,dest_ip_as_dest FIELDALIAS-src_for_ossec = src_ip as src LOOKUP-severity_for_ossec = ossec_severities_lookup severity_id OUTPUT severity REPORT-product_for_ossec = product_static_hids REPORT-vendor_for_ossec = vendor_static_open_source_security
Step 6: Validate your CIM compliance
1. Verify that your field extractions function correctly. First, restart the Splunk platform so that it recognizes the lookups you created.
2. In the Searching and Reporting app, search for the source type.
sourcetype="ossec"
3. From the search results, select Pick Fields to choose the fields that the Splunk platform ought to populate. Hover over the field name to display the values (see example below).
- Optional You can further validate using one of the following methods:
- Use Pivot. See Validate using Pivot for details.
- Use the
datamodel
command. See Validate using thedatamodel
command for details. - Use the Splunk Enterprise Security dashboard in which you expect the data to appear. In this example, the OSSEC data ought to display in the Intrusion Center dashboard. The OSSEC data is not immediately available in the dashboard because Splunk Enterprise Security uses summary indexing. Therefore, the data may not be available on the dashboard for up to an hour after you have completed the add-on.
Step 7: (Optional) Document and package your configurations as an add-on
1. Create a README file. In the file, include information necessary for others to use the add-on. Create the following README.txt
file under the root add-on directory.
===OSSEC add-on=== Author: John Doe Version/Date: 1.3 September 2013 Supported product(s): This add-on supports Open Source Security (OSSEC) IDS 2.5 Source type(s): This add-on will process data that is source typed as "ossec". Input requirements: Syslog data sent to port UDP\514 ===Using this add-on=== Configuration: Automatic Syslog data sent to port UDP\514 will automatically be detected as OSSEC data and processed accordingly. To process data that is sent to another port, configure the data input with a source type of "ossec".
2. Package the OSSEC add-on by converting it into a zip archive named Splunk_TA-ossec.zip
.
3. To share your add-on, go to https://splunkbase.splunk.com/develop/, then click Submit your app.
Use the common action model to build custom alert actions | Use the CIM to normalize CPU performance metrics |
This documentation applies to the following versions of Splunk® Common Information Model Add-on: 4.8.0, 4.9.0, 4.9.1, 4.10.0, 4.11.0, 4.12.0, 4.13.0, 4.14.0, 4.15.0, 4.16.0, 4.17.0, 4.18.0, 4.18.1, 4.19.0, 4.20.0, 4.20.2, 5.0.0, 5.0.1, 5.0.2, 5.1.0, 5.1.1, 5.1.2, 5.2.0, 5.3.1, 5.3.2, 6.0.0
Feedback submitted, thanks!