Splunk Cloud Platform Field alias behavior change
When you upgrade to version 7.2.4+ of Splunk Cloud Platform, the behavior of certain field alias configurations changes.
A field alias is a way of setting up an alternate name for a field. You can then use that alternate name to search for events that contain that field. Ideally, you should be able to define multiple aliases for a single field, but each alias you define should apply only to one source field. Additionally, when you apply a field alias configuration to a search, the expectation is that the source field is in the events, but the alias field is not in the events.
This issue involves events that already include the alias field, but which are missing the source field or have no value for the source field.
Before the 7.2.4 field alias fix
In versions of Splunk Cloud Platform previous to 7.2.4, when you applied a field alias configuration to events that had the alias field but not the source field, no changes were made to those events. The alias fields were allowed to stay.
This behavior is an erroneous application of the field alias concept. It allows users to have alias field values that do not correspond to source fields.
Example of the old field alias behavior
Here are four events of sample log data. This is what they look like before we apply field alias processing to them. Each of these events has a sourcetype
of st1
and a source
of example.log
.
02-19-2019 19:46:56.122 user=jessica id=123456789 uid=1241 message=this is just a simple example 02-19-2019 19:46:11.342 user=joe id=123777789 message=this is just a simple example 02-18-2019 11:12:56.854 uid=7788 message=this is just a simple example 02-18-2019 11:11:25.478 user=adam message=this is just a simple example
Now, say you want to apply this pair of props.conf
field alias configurations to that set of events.
[st1] FIELDALIAS-class1 = uid AS user [source::example.log] FIELDALIAS-class2 = id AS user
With this pair of configurations, events that share a sourcetype
of st1
and a source
of example.log
have the user
field aliased to two different source fields: uid
and id
. These colliding configurations are problematic because field aliases are supposed to reference only one source field at a time.
In addition, you know that user
, the alias field, already exists in the events. If your field alias configurations say that the value of user
should match a value of either uid
or id
but the user
field in the event already has a value of jessica
, how does the search head resolve this? It replaces the user
field value with one of the alias field values, according to lexicographical sort order logic.
But the real issue here is with the fourth event, where the alias field exists, but no source field exists. The pre-7.2.4 rules allowed the alias field to stay in an event when there were no source fields in the event.
Here is what the sample events look like after field alias processing with the pre-7.2.4 rules.
02-19-2019 19:46:56.122 user=123456789 id=123456789 uid=1241 message=this is just a simple example 02-19-2019 19:46:11.342 user=123777789 id=123777789 message=this is just a simple example 02-18-2019 11:12:56.854 user=7788 uid=7788 message=this is just a simple example 02-18-2019 11:11:25.478 user=adam message=this is just a simple example
This results in the overwriting of user
values in the first two events. The search head resolves the conflict between id
and uid
in the first event by selecting id
.
The search head resolves collisions between two or more AS configurations by applying each of the FIELDALIAS class names in lexicographical sort order. It uses the last class that it applies. So in the first event of this example, it first applies class1 and then applies class2. Because class2 is the last class applied, the user field takes on the value of the id field.
The third event gets a new field. Before processing, it had a source field, but no alias field. After processing it has an alias field with the value of the source field. This is how field aliases are supposed to work.
But the fourth event has an alias field without a source field. After the field alias configuration is applied, the alias field should not appear in events that do not have the corresponding source fields. The logic set by the configuration is not consistent.
After the 7.2.4 field alias fix
In Splunk Cloud Platform 7.2.4, this bug was fixed. The fix changed the behavior when you apply a field alias configuration to an event where the alias field is already present but no source fields exist. This table explains how the behavior has changed and why.
Event contains source field? | Event contains alias field? | Before 7.2.4, what happened after application of the field alias configuration to the event? | After 7.2.4, what happens after application of the field alias configuration to the event? | Why does this happen? |
---|---|---|---|---|
Yes | Yes | The search head replaces the value of the alias field with the value of the source field. | The search head replaces the value of the alias field with the value of the source field. | If a source field and its alias field are both present in an event they should share the value of the source field. |
No | Yes | Nothing. The search head does not change or remove the alias field. | The search head removes the alias field from the event. | If a source field is missing from an event, its alias field should not be present in that event. |
Yes | No | The search head adds an alias to the event. It is given the value of the source field in the event. | The search head adds an alias to the event. It is given the value of the source field in the event. | This is the ideal environment for a field alias. It enables you to search for events using a field name that is an alias for a source field that is already present in those events. |
No | No | Nothing happens. The search head leaves the event unchanged. | Nothing happens. The search head leaves the event unchanged. | If the event does not have the source field, and the alias field is also not present, there is no need to change the event. |
Example of the 7.2.4 fix
This example shows you how the 7.2.4 fix changed the results of some searches. Say you start with the same two field alias configurations:
[st1] FIELDALIAS-class1 = uid AS user [source::example.log] FIELDALIAS-class2 = id AS user
You apply those configurations to the same four events as the preceding examples. Here are the results:
02-19-2019 19:46:56.122 user=123456789 id=123456789 uid=1241 message=this is just a simple example 02-19-2019 19:46:11.342 user=123777789 id=123777789 message=this is just a simple example 02-18-2019 11:12:56.854 user=7788 uid=7788 message=this is just a simple example 02-18-2019 11:11:25.478 message=this is just a simple example
As you can see, the difference in this set of events is that the user
field is removed from the fourth event. It is removed because it is an alias field and there is no source field in the event.
The introduction of ASNEW in 7.2.4
Version 7.2.4 of the Splunk Cloud Platform introduced the ASNEW
field alias configuration. ASNEW
allows you to combine field aliases without overriding or removing values.
For example, say you have a search that runs over events that include the dst
field, and you want to apply the following props.conf
field alias configuration to it:
[sv1] FIELDALIAS-classx src AS dst
In the case of events that already have dst
, you want the field and its values to be undisturbed by the field alias processing. You do not want dst
to be removed, and you do not want the value of dst
to be altered. In this case you must change the configuration from AS
to ASNEW
:
[sv1] FIELDALIAS-classx src ASNEW dst
When you apply this configuration, the search head passes over instances of dst
that are already present in your events. It does not remove them or overwrite them.
Configure settings on Splunk Web
You can use the Overwite field values setting to determine how alias fields are treated when they are already present in events at the time that field alias processing takes place.
Select Overwrite field values for a field alias that uses the corrected field alias behavior. This means that it does what it takes during processing to ensure that the alias fields in the event share the values of their corresponding source fields. See the table in After the 7.2.4 field alias fix for more information.
When Overwrite field values is not selected, the field alias uses the uncorrected behavior, which means that the alias field is not changed or removed if it exists in an event without a source field when field alias processing takes place.
When you create a new field alias, Overwrite field values is not selected by default.
See Create field aliases in Splunk Web for more information about the workflow for field alias creation with the Settings pages.
Using calculated fields to apply an alias field to multiple source fields
Calculated fields provide a more versatile method for applying an alias field to multiple source fields. Use eval
functions such as coalesce
to determine the order in which colliding source fields are applied to your alias fields.
Calculated fields that use functions like mvappend
and mvdedup
also enable you to deal with situations where your field alias configuration collides with a field extraction. For example, say you have this combination of a field alias configuration and a field extraction configuration:
[st1] FIELDALIAS-class1 = uid AS user [source::example.log] EXTRACT-class2 = 123(?<user>[0-9]+)789
During a search, the EXTRACT-class2
configuration extracts user
field values for events with a source of example.log
. Later in the search pipeline, the FIELDALIAS-class1
configuration applies a field alias to events with a source type of st1
. FIELDALIAS-class1
gives the user
field the same value as uid
even when uid
is null. As a result, events with a source of example.log
and a source type of st1
have the extracted value of the user
field overwritten by the contents of the uid
field.
Here is the same set of results used earlier in this topic, after they have been processed through these two configurations:
02-19-2019 19:46:56.122 user=1241 id=123456789 uid=1241 message=this is just a simple example 02-19-2019 19:46:11.342 id=123777789 message=this is just a simple example 02-18-2019 11:12:56.854 user=7788 uid=7788 message=this is just a simple example 02-18-2019 11:11:25.478 message=this is just a simple example
This configuration is fine if you intend for the extracted value of user
to be overwritten. But if that is not the case, one of the following three calculated field configurations would be a better choice than the FIELDALIAS-class1
configuration, depending on the effect you are trying to achieve:
Calculated field configuration | What it does |
---|---|
EVAL-user = coalesce(user, uid)
|
Retains only one field value. Prioritizes the extracted value over the aliased value. |
EVAL-user = mvappend(user, uid)
|
Maintains both the extracted and aliased values. Could lead to duplicated values. |
EVAL-user = mvdedup(mvappend(user, uid))
|
Maintains both the extracted and aliased values. No duplicates. |
Known and fixed issues for | Cloud Monitoring Console |
This documentation applies to the following versions of Splunk Cloud Platform™: 8.2.2112, 8.2.2201, 8.2.2202, 8.2.2203, 9.0.2205, 9.0.2208, 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308, 9.1.2312, 9.2.2403, 9.2.2406 (latest FedRAMP release)
Feedback submitted, thanks!