Knowledge Manager Manual

 


Organize and administrate knowledge objects

Understand and use the Common Information Model

NOTE - Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.

Understand and use the Common Information Model

The Common Information Model is based on the idea that you can break down most log files into three components:

  • fields
  • event type tags
  • host tags

With these three components a savvy knowledge manager should be able to set up their log files in a way that makes them easily processable by Splunk and which normalizes noncompliant log files and forces them to follow a similar schema. The Common Information model details the standard fields, event type tags, and host tags that Splunk uses when it processes most IT data.

Normalizing the standard event format

This is the recommended format that should be used when events are generated or written to a system:

<timestamp> name="<name>" event_id=<event_id> <key>=<value>

Any number of field key-value pairs are allowed. For example:

2008-11-06 22:29:04 name="Failed Login" event_id=sshd:failure src_ip=10.2.3.4 src_port=12355 dest_ip=192.168.1.35 dest_port=22

The keys are ones that are listed in the "Standard fields below". name and event_id are mandatory.

When events coming from a CISCO PIX log are compliant with the Common Information Model format, the following PIX event:

Sep 2 15:14:11 10.235.224.193 local4:warn|warning fw07 %PIX-4-106023: Deny icmp src internet:213.208.19.33 dst eservices-test-ses-public:193.8.50.70 (type 8, code 0) by access-group "internet_access_in"

looks as follows:

2009-09-02 15:14:11 name="Deny icmp" event_id=106023 vendor=CISCO product=PIX log_level=4 dvc_ip=10.235.224.193 dv_host=fw07 syslog_facility=local4 syslog_priority=warn src_ip=213.208.19.33 dest_ip=193.8.50.70 src_network=internet dest_network=eservices-test-ses-public icmp_type=8 icmp_code=0 proto=icmp rule_number="internet_access_in"

Standard fields

This section presents lists of standard fields that can be extracted from event data as custom search-time field extractions.

Please note that we strongly recommend that all of these field extractions be performed at search time. There is no need to add these fields to the set of default fields that Splunk extracts at index time.

For more information about the index time/search time distinction, see "Index time versus search time" in the Admin manual. For more information about performing field extractions at search time, see "Create search-time field extractions" in this manual.

Note that some of these field extractions are fields that have a narrowly defined set of possible values. For example, in most cases an action field can have only two values: success or failure. Most fields have a wide range of possible values, however. For example, affected_user_id, a six-digit user id number, has a large number of possible values. While the set of possible values for a six-digit user id are finite, you wouldn't try to list all of them.

We've also grouped fields together into event categories. You'll see that in some cases the same field appears in several different categories. This is because the meaning of a field can change depending on the context of the event type it belongs to. For example, in an authentication event, the dest field represents the target involved in the authentication. But in a malware detection event, dest usually refers to the target that has been affected or infected by malware.

Account management

Field name Data type Description Possible values
dest_nt_domain string The domain containing the user that is affected by the account management event.
signature string Description of the account management change performed.
src_nt_domain string The NT source of the destination. In the case of an account management event, this is the domain that contains the user that generated the event.


Authentication - Access protection

Field name Data type Description Possible values
action string The action performed on the resource. success, failure
app string The application involved in the event (such as ssh, splunk, win:local).
dest string The target involved in the authentication. If your field is named dest_host, dest_ip, dest_ipv6, or dest_nt_host you can alias it as dest to make it CIM-compliant.
src string The source involved in the authentication. In the case of endpoint protection authentication the src is the client. If your field is named src_host, src_ip, src_ipv6, or src_nt_host you can alias it as src to make it CIM-compliant.. It is required for all events dealing with endpoint protection (Authentication, change analysis, malware, system center, and update).

Note: Do not confuse this with the event source or sourcetype fields.
src_user string In privilege escalation events, src_user represents the user who initiated the privilege escalation.
user string The name of the user involved in the event, or who initiated the event. For authentication privilege escalation events this should represent the user targeted by the escalation.

Change analysis - Endpoint protection

Field name Data type Description Possible values
action string The action performed on the resource.
data string Data associated with the change event
dest string The host that was affected by the change. If your field is named dest_host,dest_ip,dest_ipv6, or dest_nt_host you can alias it as dest to make it CIM-compliant.
msg string Message associated with the event
object string Name of affected object
object_attrs MV string Attributes changed on object, if applicable
object_category string Generic name for class of changed object directory, file, registry, unknown
object_id string Unique affected object ID as presented to system, if applicable
(SID in Windows, UUID in UNIX if in use)
object_path string Full path to object , if applicable
severity string Severity of change, if applicable
status string Status of the change
user string User or entity performing the change (can be UID or PID)
user_type string Type of user performing change

Change analysis - Network protection

Field name Data type Description Possible values
action string The type of change observed.
command string The command that initiated the change.
dvc string The device that is directly affected by the change.
user string The user that initiated the change.


Common event fields

Field name Data type Description Possible values
category string A device-specific classification provided as part of the event.
count string A device-specific classification provided as part of the event.
desc string The free-form description of a particular event.
dhcp_pool string The name of a given DHCP pool on a DHCP server.
duration int The amount of time the event lasted.
dvc_host string The fully qualified domain name of the device transmitting or recording the log record.
dvc_ip string The IPv4 address of the device reporting the event.
dvc_ip6 string The IPv6 address of the device reporting the event.
dvc_location string The free-form description of the device's physical location.
dvc_mac string The MAC (layer 2) address of the device reporting the event.
dvc_nt_domain string The Windows NT domain of the device recording or transmitting the event.
dvc_nt_host string The Windows NT host name of the device recording or transmitting the event.
dvc_time timestamp Time at which the device recorded the event.
end_time timestamp The event's specified end time.
event_id int A unique identifier that identifies the event. This is unique to the reporting device.
length int The length of the datagram, event, message, or packet.
log_level string The log-level that was set on the device and recorded in the event.
name string The name of the event as reported by the device. The name should not contain information that's already being parsed into other fields from the event, such as IP addresses.
pid int An integer assigned by the device operating system to the process creating the record.
priority int An environment-specific assessment of the event's importance, based on elements such as event severity, business function of the affected system, or other locally defined variables.
product string The product that generated the event.
product_version int The version of the product that generated the event.
reason string The result root cause, such as connection refused, timeout, crash, and so on.
result string The action result. Often is a binary choice: succeeded and failed, allowed and denied, and so on.
severity string The severity (or priority) of an event as reported by the originating device.
start_time timestamp The event's specified start time.
transaction_id string The transaction identifier.
url string A uniform record locator (a web address, in other words) included in a record.
vendor string The vendor who made the product that generated the event.


DNS protocol

Field name Data type Description Possible values
record_class string The DNS resource record class. IN (internet - default), HS (Hesiod - historic), or CH (Chaos - historic)


DNS protocol

Field name Data type Description Possible values
dest_domain string The DNS domain that has been queried.
dest_record string The remote DNS resource record being acted upon.
dest_zone string The DNS zone that is being received by the slave as part of a zone transfer.
record_class string The DNS resource record class. IN (internet - default), HS (Hesiod - historic), or CH (Chaos - historic)
record_type string The DNS resource record type (see this Wikipedia article on DNS record types).
src_domain string The local DNS domain that is being queried.
src_record string The local DNS resource record being acted upon.
src_zone string The DNS zone that is being transferred by the master as part of a zone transfer.


Email tracking

Field name Data type Description Possible values
recipient string The person to whom an email is sent.
sender string The person responsible for sending an email.
subject string The email subject line.


File management

Field name Data type Description Possible values
file_access_time timestamp The time the file (the object of the event) was accessed.
file_create_time timestamp The time the file (the object of the event) was created.
file_hash string A cryptographic identifier assigned to the file object affected by the event.
file_modify_time timestamp The time the file (the object of the event) was altered.
file_name string The name of the file that is the object of the event (without location information related to local file or directory structure).
file_path string The location of the file that is the object of the event, in terms of local file and directory structure.
file_permission string Access controls associated with the file affected by the event.
file_size int The size of the file that is the object of the event. Indicate whether Bytes, KB, MB, GB.


Intrusion detection

Field name Data type Description Possible values
category string The category of the triggered signature.
dest string The destination of the attack detected by the intrusion detection system (IDS). If your field is named dest_host, dest_ip, dest_ipv6, or dest_nt_host you can alias it as dest to make it CIM-compliant.
dvc string The device that detected the intrusion event.
ids_type string The type of IDS that generated the event. network, host, application
product string The product name of the vendor technology generating network protection data, such as IDP, Providentia, and ASA.

Note: Required for all events dealing with network protection (Change analysis, proxy, malware, intrusion detection, packet filtering, and vulnerability).
severity string The severity of the network protection event (such as critical, high, medium, low, or informational).

Note: This field is a string. Please use a severity_id field for severity ID fields that are integer data types.
signature string The name of the intrusion detected on the client (the src), such as PlugAndPlay_BO and JavaScript_Obfuscation_Fre.
src string The source involved in the attack detected by the IDS. If your field is named src_host, src_ip, src_ipv6, or src_nt_host you can alias it as src to make it CIM-compliant.
user string The user involved with the intrusion detection event.
vendor string The vendor technology used to generate network protection data, such as IDP, Providentia, and ASA.

Note: Required for all events dealing with network protection (Change analysis, proxy, malware, intrusion detection, packet filtering, and vulnerability).


Malware - Endpoint protection

Field name Data type Description Possible values
action string The outcome of the infection allowed, blocked, deferred
dest_nt_domain string The NT domain of the destination (the dest_bestmatch).
file_hash string The cryptographic hash of the file associated with the malware event (such as the malicious or infected file).
file_name string The name of the file involved in the malware event (such as the infected or malicious file).
file_path string The path of the file involved in the malware event (such as the infected or malicious file).
product string The product name of the vendor technology (the vendor field) that is generating malware data (such as Antivirus or EPO).
product_version string The product version number of the vendor technology installed on the client (such as 10.4.3 or 11.0.2).
signature string The name of the malware infection detected on the client (the src), such as Trojan.Vundo,Spyware.Gaobot,W32.Nimbda).

Note: This field is a string. Please use a signature_id field for signature ID fields that are integer data types.
signature_version string The current signature definition set running on the client, such as 11hsvx)
dest string The target affected or infected by the malware. If your field is named dest_host, dest_ip, dest_ipv6, or dest_nt_host you can alias it as dest to make it CIM-compliant.
src_nt_domain string The NT domain of the source (the src).
user string The name of the user involved in the malware event.
vendor string The name of the vendor technology generating malware data, such as Symantec or McAfee.


Malware - Network protection

Field name Data type Description Possible values
product string The product name of the vendor technology generating network protection data, such as IDP, Proventia, and ASA.

Note: Required for all events dealing with network protection (Change analysis, proxy, malware, intrusion detection, packet filtering, and vulnerability).
severity string The severity of the network protection event (such as critical, high, medium, low, or informational).

Note: This field is a string. Please use a severity_id field for severity ID fields that are integer data types.
vendor string The vendor technology used to generate network protection data, such as IDP, Proventia, and ASA.

Note: Required for all events dealing with network protection (Change analysis, proxy, malware, intrusion detection, packet filtering, and vulnerability).


Network traffic - ESS

Field name Data type Description Possible values
action string The action of the network traffic.
dest_port int The destination port of the network traffic.
product string The product name of the vendor technology generating NetworkProtection data, such as IDP, Proventia, and ASA.

Note: Required for all events dealing with network protection (Change analysis, proxy, malware, intrusion detection, packet filtering, and vulnerability).
src_port int The source port of the network traffic.
vendor string The vendor technology used to generate NetworkProtection data, such as IDP, Proventia, and ASA.

Note: Required for all events dealing with network protection (Change analysis, proxy, malware, intrusion detection, packet filtering, and vulnerability).


Network traffic - Generic

Field name Data type Description Possible values
app_layer string The ISO layer 7 (application layer) protocol, such as HTTP, HTTPS, SSH, and IMAP.
bytes_in int How many bytes this device/interface received.
bytes_out int How many bytes this device/interface transmitted.
channel string 802.11 channel number used by a wireless network.
cve string The Common Vulnerabilities and Exposures (CVE) reference value.
dest_app string The destination application being targeted.
dest_cnc_channel string The destination command and control service channel.
dest_cnc_name string The destination command and control service name.
dest_cnc_port string The destination command and control service port.
dest_country string The country associated with a packet's recipient.
dest_host string The fully qualified host name of a packet's recipient. For HTTP sessions, this is the host header.
dest_int string The interface that is listening remotely or receiving packets locally.
dest_ip string The IPv4 address of a packet's recipient.
dest_ipv6 string The IPv6 address of a packet's recipient.
dest_lat int The (physical) latitude of a packet's destination.
dest_long int The (physical) longitude of a packet's destination.
dest_mac string The destination TCP/IP layer 2 Media Access Control (MAC) address of a packet's destination.
dest_nt_domain string The Windows NT domain containing a packet's destination.
dest_nt_host string The Windows NT host name of a packet's destination.
dest_port int TCP/IP port to which a packet is being sent.
dest_translated_ip string The NATed IPv4 address to which a packet has been sent.
dest_translated_port int The NATed port to which a packet has been sent.
ip_version int The numbered Internet Protocol version. 4, 6
outbound_interface string The network interface through which a packet was transmitted.
packets_in int How many packets this device/interface received.
packets_out int How many packets this device/interface transmitted.
proto string The OSI layer 3 (Network Layer) protocol, such as IPv4/IPv6, ICMP, IPsec, IGMP or RIP.
session_id string The session identifier. Multiple transactions build a session.
ssid string The 802.11 service set identifier (ssid) assigned to a wireless session.
src_country string The country from which the packet was sent.
src_host string The fully qualified host name of the system that transmitted the packet. For Web logs, this is the HTTP client.
src_int string The interface that is listening locally or sending packets remotely.
src_ip string The IPv4 address of the packet's source. For Web logs, this is the http client.
src_ipv6 string The IPv6 address of the packet's source.
src_lat int The (physical) latitude of the packet's source.
src_long int The (physical) longitude of the packet's source.
src_mac string The Media Access Control (MAC) address from which a packet was transmitted.
src_nt_domain string The Windows NT domain containing the machines that generated the event.
src_nt_host string The Windows NT hostname of the system that generated the event.
src_port int The network port from which a packet originated.
src_translated_ip string The NATed IPv4 address from which a packet has been sent.
src_translated_port int The NATed network port from which a packet has been sent.
syslog_id string The application, process, or OS subsystem that generated the event.
syslog_priority string The criticality of an event, as recorded by UNIX syslog.
tcp_flag string The TCP flag(s) specified in the event. Can be one or more of SYN, ACK, FIN, RST, URG, or PSH.
tos string The hex bit that specifies TCP 'type of service' (see http://en.wikipedia.org/wiki/Type_of_Service).
transport string The transport protocol. TCP, UDP
ttl int The "time to live" of a packet or datagram.
vlan_id int The numeric identifier assigned to the virtual local area network (VLAN) specified in the record.
vlan_name string The name assigned to the virtual local area network (VLAN) specified in the record.


Packet filtering

Field name Data type Description Possible values
action string The action the filtering device (the dvc_bestmatch field) performed on the communication. allowed, blocked
dest_port int The IP port of the packet's destination, such as 22.
direction string The direction the packet is traveling. inbound, outbound
dvc string The name of the packet filtering device. If your field is named dvc_host, dvc_ip, or dvc_nt_host you can alias it as dvc to make it CIM-compliant.
rule string The rule which took action on the packet, such as 143.
svc_port int The IP port of the packet's source, such as 34541.

Proxy

Field name Data type Description Possible values
action string The action taken by the proxy.
dest string The destination of the network traffic (the remote host).
http_content_type string The content-type of the requested HTTP resource.
http_method string The HTTP method used to request the resource. GET, POST, DELETE, and so on.
http_refer string The HTTP referrer used to request the HTTP resource.
http_response int The HTTP response code.
http_user_agent string The user agent used to request the HTTP resource.
product string The product name of the vendor technology generating Network Protection data, such as IDP, Providentia, and ASA.
src string The source of the network traffic (the client requesting the connection).
status int The HTTP response code indicating the status of the proxy request. 404, 302, 500, and so on.
user string The user that requested the HTTP resource.
url string The URL of the requested HTTP resource.
vendor string The vendor technology generating Network Protection data, such as IDP, Providentia, and ASA.


System center

Field name Data type Description Possible values
selinux string Values from the SE Linux configuration file. disabled, enforcing
Startmode string The start mode of the given service. disabled, enabled, auto


System center

Field name Data type Description Possible values
app string The running application or service on the system (the src field), such as explorer.exe or sshd.
FreeMBytes int The amount of disk space available per drive or mount (the mount field) on the system (the src field).
kernel_release string The version of operating system installed on the host (the src field), such as 6.0.1.4 or 2.6.27.30-170.2.82.fc10.x86_64.
label string Human-readable version of the SystemUptime value.
mount string The drive or mount reporting available disk space (the FreeMBytes field) on the system (the src field).
os string The name of the operating system installed on the host (the src), such as Microsoft Windows Server 2003 or GNU/Linux).
PercentProcessorTime int The percentage of processor utilization.
setlocaldefs int The setlocaldefs setting from the SE Linux configuration.
selinux string Values from the SE Linux configuration file. disabled, enforcing
selinuxtype string The SE Linux type (such as targeted).
shell string The shell provided to the User Account (the user field) upon logging into the system (the src field).
src_port int The TCP/UDP source port on the system (the src field).
sshd_protocol string The sshd protocol version.
Startmode string The start mode of the given service. disabled, enabled, auto
SystemUptime int The number of seconds since the system (the src) has been "up."
TotalMBytes int The total amount of available memory on the system (the src).
UsedMBytes int The amount of used memory on the system (the src).
user string The User Account present on the system (the src).
updates int The number of updates the system (the src) is missing.


Traffic

Field name Data type Description Possible values
dest string The destination of the network traffic. If your field is named dest_host, dest_ip, dest_ipv6, or dest_nt_host you can alias it as dest to make it CIM-compliant.
dvc string The name of the packet filtering device. If your field is named dvc_host, dvc_ip, or dvc_nt_host you can alias it as dvc to make it CIM-compliant.
src string The source of the network traffic. If your field is named src_host, src_ip, src_ipv6, or src_nt_host you can alias it as src to make it CIM-compliant.


Update

Field name Data type Description Possible values
package string The name of the installed update.


User information updates

Field name Data type Description Possible values
affected_user string A user that has been affected by a change. For example, user fflanda changed the name of user rhallen, so affected_user=rhallen.
affected_user_group string The user group affected by a change.
affected_user_group_id int The identifier of the user group affected by a change.
affected_user_id int The identifier of the user affected by a change.
affected_user_privilege enumeration The security context associated with the user affected by a change. administrator, user, guest/anonymous
user string The name of the user affected by the recorded event.
user_group string A user group that is the object of an event, expressed in human-readable terms.
user_group_id int The numeric identifier assigned to the user group event object.
user_id int The system-assigned identifier for the user affected by an event.
user_privilege enumeration The security context associated with the object of an event (the affected user). administrator, user, guest/anonymous
user_subject string The name of the user that is the subject of an event--the user executing the action, in other words.
user_subject_id int The ID number of the user that is the subject of an event.
user_subject_privilege enumeration The security context associated with the subject of an event (the user causing a change). administrator, user, guest/anonymous

Vulnerability

Field name Data type Description Possible values
category string The category of the discovered vulnerability.
dest string The host with the discovered vulnerability. If your field is named dest_host, dest_ip, dest_ipv6, or dest_nt_host you can alias it as dest to make it CIM-compliant.
os string The operating system of the host containing the vulnerability detected on the client (the src field), such as SuSE Security Update, or cups security update.
severity string The severity of the discovered vulnerability.
signature string The name of the vulnerability detected on the client (the src field), such as SuSE Security Update, or cups security update.


Windows administration

Field name Data type Description Possible values
object_name string The object name (associated only with Windows).
object_type string The object type (associated only with Windows).
object_handle string The object handle (associated only with Windows).

Standardize your event type tags

The Common Information Model suggests that you use a specific convention when tagging your event types. This convention requires that you set up two categories of tags, and that you give each event type in your system a single tag from both of these categories. The categories are object and status.

This arrangement enables precise event type classification. The object tag denotes what the event is about. What object has been targeted? Is the event talking about a host, a resource, a file, or what? And the status tag provides the status of the action. Was it successful? Failed? Or was it simply an attempt? In addition to these two standard tags, you can add other tags as well.

The three tags in discussion here are:

<objecttag> <statustag>

Some examples of using the standard tags are:

  • For a firewall deny event type:
host failure
  • For a firewall accept event :
host success
  • For a successful database login:
database success

Object event type tags

Use one of these object tags in the first position as defined above.

TagExplanation
application An application-level event.
application av An anti virus event.
application backdoor An event using an application backdoor.
application database A database event.
application database data An event related to database data.
application dosclient An event involving a DOS client.
application firewall An event involving an application firewall.
application im An instant message-related event.
application peertopeer A peer to peer-related event.
host A host-level event.
group A group-level event
resource An event involving system resources.
resource cpu An event involving the CPU.
resource file An event involving a file.
resources interface An event involving network interfaces.
resource memory An event involving memory.
resource registry An event involving the system registry.
os An OS-level event.
os process An event involving an OS-related process
os service An event involving an OS service.
user A user-level event


Status event type tags

Use one of these status tags in the third position as defined above.

TagExplanation
attempt An event marking an attempt at something.
deferred A deferred event.
failure A failed event.
inprogress An event marking something progress.
report A report of a status.
success A successful event.

Optional tags

For those who want to use standard additional tags when they apply, some suggestions are below.

TagExplanation
attack An event marking an attack.
attack exploit An event marking the use of an exploit.
attack bruteforce An event marking a brute force attack.
attack dosAn event marking a denial of service attack.
attack escalation An event indicating a privilege escalation attack.
infoleak An event indicating an information leak.
malware An event marking malware action.
malware dosclient An event marking malware utilizing a DOS client.
malware spyware An event marking spyware.
malware trojan An event marking a trojan.
malware virus An event marking a virus.
malware worm An event marking a worm.
recon An event marking recon probes.
suspicious An event indicating suspicious activity.

Standardize your host tags

As you may know, it can be problematic to rename hosts directly. Because hosts are identified before event data is indexed, changes to host names are not applied to data that has already been indexed. It's far easier to use tags to group together events from particular hosts.

You can use standardized tags to describe specific hosts and what they do. There are a variety of approaches to host tagging, all of which can be used where appropriate. Some of these methods include:

  • What service(s) the host is running.
  • What OS the host is running.
  • The department the host belongs to.
  • What data the host contains.
  • What cluster/round robin the host belongs to.

General host tags

These host tags are useful across the board. You can also develop lists of host tags that are appropriate for specific apps.

TagExplanation
dbThis host is a database.
developmentThis host is a development box.
dmzThis host is in the DMZ.
dns This host is a DNS server.
email This host is an email server.
finance This host contains financial information.
firewallThis host is a firewall.
highly_criticalThis host is highly critical for business purposes.
web This host is a Web server.

This documentation applies to the following versions of Splunk: 4.1 , 4.1.1 , 4.1.2 , 4.1.3 , 4.1.4 , 4.1.5 , 4.1.6 , 4.1.7 , 4.1.8 , 4.2 , 4.2.1 , 4.2.2 , 4.2.3 , 4.2.4 , 4.2.5 , 4.3 , 4.3.1 , 4.3.2 , 4.3.3 , 4.3.4 , 4.3.5 , 4.3.6 , 4.3.7 View the Article History for its revisions.


You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!