Example inline field extraction configurations

The following are examples of inline field extraction, using props.conf.

Add an error code field

Create an error code field by configuring a field extraction in props.conf. The field is identified by the occurrence of device_id= followed by a word within brackets and a text string terminating with a colon. The field is extracted from the testlog source type.

In props.conf, add the following line:

[testlog]
EXTRACT-errors = device_id=\[w+\](?<err_code>[^:]+)

Extract multiple fields by using one regular expression

The following is an example of a field extraction of five fields. A sample of the event data follows.

#%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet9/16, changed state to down

The stanza in props.conf for the extraction looks like this:

[syslog]
EXTRACT-port_flapping = Interface\s(?<interface>(?<media>[^\d]+)(?<slot>\d+)\/(?<port>\d+))\,\schanged
\sstate\sto\s(?<port_status>up|down)

Five fields are extracted as named groups: interface, media, slot, port, and port_status.

Use extracted fields to report port flapping events.

1. Use tags to define event types in eventtypes.conf:

   [cisco_ios_port_down]
   search = "changed state to down"
   
   [cisco_ios_port_up]
   search = "changed state to up"
   

2. Create a report in savedsearches.conf that ties much of the above together to find port flapping and report on the results:

   [port flapping]
   search = eventtype=cisco_ios_port_down OR eventtype=cisco_ios_port_up starthoursago=3 
   | stats count by interface,host,port_status 
   | sort -count
   

You can then use these fields with some event types to help you find port flapping events and report on them.

Create a field from a subtoken

You may run into problems if you are extracting a field value that is a subtoken--a part of a larger token. Tokens are chunks of event data that have been run through event processing prior to being indexed. During event processing, events are broken up into segments, and each segment created is a token.

Example

Tokens are never smaller than a complete word or number. For example, you may have the word foo123 in your event. If it has been run through event processing and indexing, it is a token, and it can be a value of a field. However, if your extraction pulls out the foo as a field value unto itself, you're extracting a subtoken. The problem is that while foo123 exists in the index, foo does not, which means that you'll likely get few results if you search on that subtoken, even though it may appear to be extracted correctly in your search results.

Because tokens cannot be smaller than individual words within strings, a field extraction of a subtoken (a part of a word) can cause problems because subtokens will not themselves be in the index, only the larger word of which they are a part.

1. (Optional) If your field value is a smaller part of a token, you must configure props.conf as explained here.
2. Add an entry to fields.conf.

   [<fieldname>]
   INDEXED = False
   INDEXED_VALUE = False
   

   • Fill in <fieldname> with the name of your field.
     • For example, [url] if you've configured a field named "url."
   • Set INDEXED and INDEXED_VALUE to false.
     • This setting specifies that the value you're searching for is not a token in the index.

You do not need to add this entry to fields.conf for cases where you are extracting a field's value from the value of a default field (such as host, source, sourcetype, or timestamp) that is not indexed and therefore not tokenized.

For more information on the tokenization of event data, see About segmentation in the Getting Data In Manual.