Docs » Manage the logs pipeline » Log processing rules

Log processing rules πŸ”—

Add value to your raw logs by creating log processing rules, also known as processors, to transform your data or a subset of your data as it arrives.

To add more control to processors, you can add filters that determine which logs a processor will be applied to.

On the pipeline lister page, you can adjust the order in which your processing rules run, edit processors, or delete processors.

Note

You can’t edit or delete prepackaged processors.

Prepackaged processors appear at the beginning of the list of processors, and they’re identified by a lock icon. These prepackaged processors always execute before any processors you define. You can’t modify or reorder prepackaged processors.

One example of a prepackaged processor is the Level to severity attributed remapper.

Splunk Observability Cloud includes prepackaged processors for Kubernetes and Cassandra.

Observability Cloud provides three types of log processors:

Field extraction processors πŸ”—

Field extraction lets you find an existing field in your incoming logs and create a processor based on the format of the field’s value.

Field extraction helps you do the following tasks:

Consider the following raw log record

10.4.93.105 - - [04/Feb/2021:16:57:05 +0000] β€œGET /metrics HTTP/1.1” 200 73810 β€œ-” β€œGo-http-client/1.1” 23

If you have not defined any processors in your logs pipeline, you can only do a keyword search on the sample log, which searches the _raw field. The following table shows how you can extract fields to define processing rules:

Example of value to extract

Processor definition to use

IP address (10.4.93.105)

IP

04/Feb/2021:16:57:05 +0000

time

GET

method

/metrics

path

Creating Regex and Event Time field extractions allows you to filter and aggregate on the fields: IP, time, method, and path. This enables you to create the query β€œDisplay a Visual Analysis of the number of requests from {IP} broken down by {method}”.

Additionally, the extracted fields begin appearing in the Fields summary panel along with their top values and other statistics.

There are three types of field extraction. These are:

  • Regex

  • JSON

  • Event Time

To start creating a field extraction, follow these steps:

  1. From the navigation menu, go to Organization Settings > Logs Pipeline Management. A list of existing processors is displayed with the prepackaged processors displaying first.

  2. Click New Processing Rule. Alternatively, you can launch the processor wizard from Log Observer. To do this, click into a log in the Raw Logs Table. The Log Details panel appears on the right. Click a field value then select Extract field. This takes you to Define Processor, the second step of the processor wizard. Skip to step 7.

  3. Select Field Extraction as the processor type, then click Continue. This takes you to Select sample, the first step in the processor wizard.

  4. To find a log that contains the field you want to extract, add filters to the filter bar until the Raw Logs Table displays a log with the desired field.

  5. Click the log containing the field you want. A list of fields and values appears below the log line.

  6. Click Use as sample next to the field you want to extract, then click Next. This takes you to Define Processor, the second step of the processor wizard.

  7. Select the extraction processor type that you want to use.

  8. From here, follow the steps to create the extraction processor type you selected:

Create a Regex processor πŸ”—

The regular expression workspace lets you to extract fields from your data and then create a new processor using regex. Pipeline Management makes suggestions to help you write the appropriate regex for your processor. You can modify the regex within the processor wizard.

To create a Regex processor, follow these steps:

  1. Highlight one or more values in your sample and select Extract field from the drop-down menu.

  2. Enter the name for your new field, then click Validate. Results display in a table.

  3. Preview your rule in the table to ensure that the correct fields are extracted.

  4. To apply your new rule to only a subset of incoming logs, add filters to the filter bar. The new rule will apply only to logs matching this filter.

  5. In step 3 of the processor wizard entitled Name, Save, and Review, give your new rule a name and description.

  6. Review your configuration choices, then click Save. The Logs Pipeline Management homepage displays a list of existing processors. Your new processor appears at the end of the list. It defaults to Active and immediately begins processing incoming logs. To disable your processor, click Inactive.

  7. On the rule lister page, you can reorder, edit, or delete all processors except those that are prepackaged (shown with a lock).

Create a JSON processor πŸ”—

To create a JSON processor, follow these steps:

  1. Add filters to the filter bar to define a matching condition. Pipeline Management only applies the new processor to log events that match this filter.

  2. Preview your rule to ensure that Pipeline Management is extracting the correct field values.

  3. If you see the correct field values in the results table, click Next. Otherwise, adjust your filter.

  4. Add a name and description for your new rule, then click Save. The Logs Pipeline Management homepage displays a list of existing processors. Your new processor appears at the end of the list. It defaults to Active and immediately begins processing incoming logs. To disable your processor, click Inactive.

  5. On the rule lister page, you can edit, reorder, or delete all processors except those that are prepackaged (shown with a lock).

Create an Event Time processor πŸ”—

To create an Event Time processor, follow these steps:

  1. Select a time format from the drop-down list. The wizard looks for the selected format within your sample.

  2. From the matches you see, select the time when the sample event occurred, then click Next.

  3. Add filters to the filter bar to define a matching condition, then click Next. Pipeline Management only applies the new processor to log events that match this filter.

  4. Give your new rule a name and description.

  5. Review your configuration choices, then click Save.

  6. The Logs Pipeline Management homepage displays a list of existing processors. Your new processor appears at the end of the list. It defaults to Active and immediately begins processing incoming logs. To disable your processor, click Inactive.

  7. On the rule lister page, you can edit, reorder, or delete all processors except those that are prepackaged (shown with a lock).

Field copy processors πŸ”—

Field copy processors let you define a new relationship between new or existing fields. One way to use field copy processors is to use OpenTelemetry mappings to help power your related content suggestions.

To create a field copy processor, follow these steps:

  1. From the navigation menu, go to Organization Settings > Logs Pipeline Management.

  2. Click New Processing Rule.

  3. Select Field Copy, then click Continue.

  4. Enter a target field in the first text box. You can choose from available extracted fields in the drop-down list.

  5. In the second text box, choose a field to which you want to map your target field. The drop-down list options suggest OpenTelemetry mappings, which help power your Related Content suggestions.

  6. If you want to create multiple mappings, click + Add another mapping and repeat steps 4 and 5; otherwise, click Next.

  7. To apply your new rule to only a subset of incoming logs, add filters to the filter bar. The new rule is applied only to logs matching this filter. If you do not add a filter, the rule is applied to all incoming log events.

  8. Preview your rule to ensure that Pipeline Management is extracting the correct field values, then click Next.

  9. Give your new rule a name and description, then click Save.

  10. The Logs Pipeline Management homepage displays a list of existing processors. Your new processor appears at the end of the list. It defaults to Active and immediately begins processing incoming logs. To disable your processor, click Inactive.

  11. On the rule lister page, you can edit, reorder, or delete all processors except those that are prepackaged (shown with a lock).

Field redaction processors πŸ”—

Field redaction lets you mask data, including personally identifiable information.

To create a field redaction processor, follow these steps:

  1. From the navigation menu, go to Organization Settings > Logs Pipeline Management.

  2. Click New Processing Rule.

  3. Select Field Redaction, then click Continue. This takes you to the first step in the processor wizard, Select Sample.

  4. To find a log that contains the field you want to redact, add filters to the filter bar until the Raw Logs Table displays a log with the desired field.

  5. Click the log containing the field you want. A list of fields and values appears below the log line.

  6. Click Use as sample next to the field you want to redact, then click Next. This takes you to Define Processor, the second step of the processor wizard.

  7. Select if you want to redact an entire field value or a partial field value. If you want to redact a partial field value, highlight the portion you want to redact. You can edit the Regex here.

  8. Define a matching condition. To apply your new rule to only a subset of incoming logs, add filters to the filter bar. The new rule will apply only to logs matching this filter.

  9. Give your new rule a name and description.

  10. Review your configuration choices, then click Save. The Logs Pipeline Management homepage displays a list of existing processors. Your new processor appears at the end of the list. It defaults to Active and immediately begins processing incoming logs. To disable your processor, click Inactive.

  11. On the rule lister page, you can reorder, edit, or delete all processors except those that are prepackaged (shown with a lock).

Note

If the field you redacted also appears in _raw, it is still available in _raw. Redact the field in _raw in addition to redacting the field itself.