Understand and use the Common Information Model

Understand and use the Common Information Model

The Common Information Model is based on the idea that you can break down most log files into three components:

• fields
• event type tags
• host tags

With these three components a savvy knowledge manager should be able to set up their log files in a way that makes them easily processable by Splunk and which normalizes noncompliant log files and forces them to follow a similar schema. The Common Information model details the standard fields, event type tags, and host tags that Splunk uses when it processes most IT data.

Normalizing the standard event format

This is the recommended format that should be used when events are generated or written to a system:

<timestamp> name="<name>" event_id=<event_id> <key>=<value>

Any number of field key-value pairs are allowed. For example:

2008-11-06 22:29:04 name="Failed Login" event_id=sshd:failure src_ip=10.2.3.4 src_port=12355 dest_ip=192.168.1.35 dest_port=22

The keys are ones that are listed in the "Standard fields below". name and event_id are mandatory.

When events coming from a CISCO PIX log are compliant with the Common Information Model format, the following PIX event:

Sep 2 15:14:11 10.235.224.193 local4:warn|warning fw07 %PIX-4-106023: Deny icmp src internet:213.208.19.33 dst eservices-test-ses-public:193.8.50.70 (type 8, code 0) by access-group "internet_access_in"

looks as follows:

2009-09-02 15:14:11 name="Deny icmp" event_id=106023 vendor=CISCO product=PIX log_level=4 dvc_ip=10.235.224.193 dv_host=fw07 syslog_facility=local4 syslog_priority=warn src_ip=213.208.19.33 dest_ip=193.8.50.70 src_network=internet dest_network=eservices-test-ses-public icmp_type=8 icmp_code=0 proto=icmp rule_number="internet_access_in"

Standard fields

This section presents lists of standard fields that can be extracted from event data as custom search-time field extractions.

Please note that we strongly recommend that all of these field extractions be performed at search time. There is no need to add these fields to the set of default fields that Splunk extracts at index time.

For more information about the index time/search time distinction, see "Index time versus search time" in the Admin manual. For more information about performing field extractions at search time, see "Create search-time field extractions" in this manual.

Note that some of these field extractions are fields that have a narrowly defined set of possible values. For example, in most cases an action field can have only two values: success or failure. Most fields have a wide range of possible values, however. For example, affected_user_id, a six-digit user id number, has a large number of possible values. While the set of possible values for a six-digit user id are finite, you wouldn't try to list all of them.

We've also grouped fields together into event categories. You'll see that in some cases the same field appears in several different categories. This is because the meaning of a field can change depending on the context of the event type it belongs to. For example, in an authentication event, the dest field represents the target involved in the authentication. But in a malware detection event, dest usually refers to the target that has been affected or infected by malware.

Account management

Field name	Data type	Description	Possible values
dest_nt_domain	string	The domain containing the user that is affected by the account management event.
signature	string	Description of the account management change performed.
src_nt_domain	string	The NT source of the destination. In the case of an account management event, this is the domain that contains the user that generated the event.

Authentication - Access protection

Field name	Data type	Description	Possible values
action	string	The action performed on the resource.	success, failure
app	string	The application involved in the event (such as ssh, splunk, win:local).
dest	string	The target involved in the authentication. If your field is named dest_host, dest_ip, dest_ipv6, or dest_nt_host you can alias it as dest to make it CIM-compliant.
src	string	The source involved in the authentication. In the case of endpoint protection authentication the src is the client. If your field is named src_host, src_ip, src_ipv6, or src_nt_host you can alias it as src to make it CIM-compliant.. It is required for all events dealing with endpoint protection (Authentication, change analysis, malware, system center, and update). Note: Do not confuse this with the event source or sourcetype fields.
src_user	string	In privilege escalation events, src_user represents the user who initiated the privilege escalation.
user	string	The name of the user involved in the event, or who initiated the event. For authentication privilege escalation events this should represent the user targeted by the escalation.

Change analysis - Endpoint protection

Field name	Data type	Description	Possible values
action	string	The action performed on the resource.
data	string	Data associated with the change event
dest	string	The host that was affected by the change. If your field is named dest_host,dest_ip,dest_ipv6, or dest_nt_host you can alias it as dest to make it CIM-compliant.
msg	string	Message associated with the event
object	string	Name of affected object
object_attrs	MV string	Attributes changed on object, if applicable
object_category	string	Generic name for class of changed object	directory, file, registry, unknown
object_id	string	Unique affected object ID as presented to system, if applicable (SID in Windows, UUID in UNIX if in use)
object_path	string	Full path to object , if applicable
severity	string	Severity of change, if applicable
status	string	Status of the change
user	string	User or entity performing the change (can be UID or PID)
user_type	string	Type of user performing change

Change analysis - Network protection

Field name	Data type	Description	Possible values
action	string	The type of change observed.
command	string	The command that initiated the change.
dvc	string	The device that is directly affected by the change.
user	string	The user that initiated the change.

Common event fields

Field name	Data type	Description	Possible values
category	string	A device-specific classification provided as part of the event.
count	string	A device-specific classification provided as part of the event.
desc	string	The free-form description of a particular event.
dhcp_pool	string	The name of a given DHCP pool on a DHCP server.
duration	int	The amount of time the event lasted.
dvc_host	string	The fully qualified domain name of the device transmitting or recording the log record.
dvc_ip	string	The IPv4 address of the device reporting the event.
dvc_ip6	string	The IPv6 address of the device reporting the event.
dvc_location	string	The free-form description of the device's physical location.
dvc_mac	string	The MAC (layer 2) address of the device reporting the event.
dvc_nt_domain	string	The Windows NT domain of the device recording or transmitting the event.
dvc_nt_host	string	The Windows NT host name of the device recording or transmitting the event.
dvc_time	timestamp	Time at which the device recorded the event.
end_time	timestamp	The event's specified end time.
event_id	int	A unique identifier that identifies the event. This is unique to the reporting device.
length	int	The length of the datagram, event, message, or packet.
log_level	string	The log-level that was set on the device and recorded in the event.
name	string	The name of the event as reported by the device. The name should not contain information that's already being parsed into other fields from the event, such as IP addresses.
pid	int	An integer assigned by the device operating system to the process creating the record.
priority	int	An environment-specific assessment of the event's importance, based on elements such as event severity, business function of the affected system, or other locally defined variables.
product	string	The product that generated the event.
product_version	int	The version of the product that generated the event.
reason	string	The result root cause, such as connection refused, timeout, crash, and so on.
result	string	The action result. Often is a binary choice: succeeded and failed, allowed and denied, and so on.
severity	string	The severity (or priority) of an event as reported by the originating device.
start_time	timestamp	The event's specified start time.
transaction_id	string	The transaction identifier.
url	string	A uniform record locator (a web address, in other words) included in a record.
vendor	string	The vendor who made the product that generated the event.

DNS protocol

Field name	Data type	Description	Possible values
record_class	string	The DNS resource record class.	IN (internet - default), HS (Hesiod - historic), or CH (Chaos - historic)

DNS protocol

Field name	Data type	Description	Possible values
dest_domain	string	The DNS domain that has been queried.
dest_record	string	The remote DNS resource record being acted upon.
dest_zone	string	The DNS zone that is being received by the slave as part of a zone transfer.
record_class	string	The DNS resource record class.	IN (internet - default), HS (Hesiod - historic), or CH (Chaos - historic)
record_type	string	The DNS resource record type (see this Wikipedia article on DNS record types).
src_domain	string	The local DNS domain that is being queried.
src_record	string	The local DNS resource record being acted upon.
src_zone	string	The DNS zone that is being transferred by the master as part of a zone transfer.

Email tracking

Field name	Data type	Description	Possible values
recipient	string	The person to whom an email is sent.
sender	string	The person responsible for sending an email.
subject	string	The email subject line.

File management

Field name	Data type	Description	Possible values
file_access_time	timestamp	The time the file (the object of the event) was accessed.
file_create_time	timestamp	The time the file (the object of the event) was created.
file_hash	string	A cryptographic identifier assigned to the file object affected by the event.
file_modify_time	timestamp	The time the file (the object of the event) was altered.
file_name	string	The name of the file that is the object of the event (without location information related to local file or directory structure).
file_path	string	The location of the file that is the object of the event, in terms of local file and directory structure.
file_permission	string	Access controls associated with the file affected by the event.
file_size	int	The size of the file that is the object of the event. Indicate whether Bytes, KB, MB, GB.

Intrusion detection

Field name	Data type	Description	Possible values
category	string	The category of the triggered signature.
dest	string	The destination of the attack detected by the intrusion detection system (IDS). If your field is named dest_host, dest_ip, dest_ipv6, or dest_nt_host you can alias it as dest to make it CIM-compliant.
dvc	string	The device that detected the intrusion event.
ids_type	string	The type of IDS that generated the event.	network, host, application
product	string	The product name of the vendor technology generating network protection data, such as IDP, Providentia, and ASA. Note: Required for all events dealing with network protection (Change analysis, proxy, malware, intrusion detection, packet filtering, and vulnerability).
severity	string	The severity of the network protection event (such as critical, high, medium, low, or informational). Note: This field is a string. Please use a severity_id field for severity ID fields that are integer data types.
signature	string	The name of the intrusion detected on the client (the src), such as PlugAndPlay_BO and JavaScript_Obfuscation_Fre.
src	string	The source involved in the attack detected by the IDS. If your field is named src_host, src_ip, src_ipv6, or src_nt_host you can alias it as src to make it CIM-compliant.
user	string	The user involved with the intrusion detection event.
vendor	string	The vendor technology used to generate network protection data, such as IDP, Providentia, and ASA. Note: Required for all events dealing with network protection (Change analysis, proxy, malware, intrusion detection, packet filtering, and vulnerability).

Malware - Endpoint protection

Field name	Data type	Description	Possible values
action	string	The outcome of the infection	allowed, blocked, deferred
dest_nt_domain	string	The NT domain of the destination (the dest_bestmatch).
file_hash	string	The cryptographic hash of the file associated with the malware event (such as the malicious or infected file).
file_name	string	The name of the file involved in the malware event (such as the infected or malicious file).
file_path	string	The path of the file involved in the malware event (such as the infected or malicious file).
product	string	The product name of the vendor technology (the vendor field) that is generating malware data (such as Antivirus or EPO).
product_version	string	The product version number of the vendor technology installed on the client (such as 10.4.3 or 11.0.2).
signature	string	The name of the malware infection detected on the client (the src), such as Trojan.Vundo,Spyware.Gaobot,W32.Nimbda). Note: This field is a string. Please use a signature_id field for signature ID fields that are integer data types.
signature_version	string	The current signature definition set running on the client, such as 11hsvx)
dest	string	The target affected or infected by the malware. If your field is named dest_host, dest_ip, dest_ipv6, or dest_nt_host you can alias it as dest to make it CIM-compliant.
src_nt_domain	string	The NT domain of the source (the src).
user	string	The name of the user involved in the malware event.
vendor	string	The name of the vendor technology generating malware data, such as Symantec or McAfee.

Malware - Network protection

Field name	Data type	Description	Possible values
product	string	The product name of the vendor technology generating network protection data, such as IDP, Proventia, and ASA. Note: Required for all events dealing with network protection (Change analysis, proxy, malware, intrusion detection, packet filtering, and vulnerability).
severity	string	The severity of the network protection event (such as critical, high, medium, low, or informational). Note: This field is a string. Please use a severity_id field for severity ID fields that are integer data types.
vendor	string	The vendor technology used to generate network protection data, such as IDP, Proventia, and ASA. Note: Required for all events dealing with network protection (Change analysis, proxy, malware, intrusion detection, packet filtering, and vulnerability).

Network traffic - ESS

Field name	Data type	Description	Possible values
action	string	The action of the network traffic.
dest_port	int	The destination port of the network traffic.
product	string	The product name of the vendor technology generating NetworkProtection data, such as IDP, Proventia, and ASA. Note: Required for all events dealing with network protection (Change analysis, proxy, malware, intrusion detection, packet filtering, and vulnerability).
src_port	int	The source port of the network traffic.
vendor	string	The vendor technology used to generate NetworkProtection data, such as IDP, Proventia, and ASA. Note: Required for all events dealing with network protection (Change analysis, proxy, malware, intrusion detection, packet filtering, and vulnerability).

Network traffic - Generic

Field name	Data type	Description	Possible values
app_layer	string	The ISO layer 7 (application layer) protocol, such as HTTP, HTTPS, SSH, and IMAP.
bytes_in	int	How many bytes this device/interface received.
bytes_out	int	How many bytes this device/interface transmitted.
channel	string	802.11 channel number used by a wireless network.
cve	string	The Common Vulnerabilities and Exposures (CVE) reference value.
dest_app	string	The destination application being targeted.
dest_cnc_channel	string	The destination command and control service channel.
dest_cnc_name	string	The destination command and control service name.
dest_cnc_port	string	The destination command and control service port.
dest_country	string	The country associated with a packet's recipient.
dest_host	string	The fully qualified host name of a packet's recipient. For HTTP sessions, this is the host header.
dest_int	string	The interface that is listening remotely or receiving packets locally.
dest_ip	string	The IPv4 address of a packet's recipient.
dest_ipv6	string	The IPv6 address of a packet's recipient.
dest_lat	int	The (physical) latitude of a packet's destination.
dest_long	int	The (physical) longitude of a packet's destination.
dest_mac	string	The destination TCP/IP layer 2 Media Access Control (MAC) address of a packet's destination.
dest_nt_domain	string	The Windows NT domain containing a packet's destination.
dest_nt_host	string	The Windows NT host name of a packet's destination.
dest_port	int	TCP/IP port to which a packet is being sent.
dest_translated_ip		string	The NATed IPv4 address to which a packet has been sent.
dest_translated_port	int	The NATed port to which a packet has been sent.
ip_version	int	The numbered Internet Protocol version.	4, 6
outbound_interface	string	The network interface through which a packet was transmitted.
packets_in	int	How many packets this device/interface received.
packets_out	int	How many packets this device/interface transmitted.
proto	string	The OSI layer 3 (Network Layer) protocol, such as IPv4/IPv6, ICMP, IPsec, IGMP or RIP.
session_id	string	The session identifier. Multiple transactions build a session.
ssid	string	The 802.11 service set identifier (ssid) assigned to a wireless session.
src_country	string	The country from which the packet was sent.
src_host	string	The fully qualified host name of the system that transmitted the packet. For Web logs, this is the HTTP client.
src_int	string	The interface that is listening locally or sending packets remotely.
src_ip	string	The IPv4 address of the packet's source. For Web logs, this is the http client.
src_ipv6	string	The IPv6 address of the packet's source.
src_lat	int	The (physical) latitude of the packet's source.
src_long	int	The (physical) longitude of the packet's source.
src_mac	string	The Media Access Control (MAC) address from which a packet was transmitted.
src_nt_domain	string	The Windows NT domain containing the machines that generated the event.
src_nt_host	string	The Windows NT hostname of the system that generated the event.
src_port	int	The network port from which a packet originated.
src_translated_ip	string	The NATed IPv4 address from which a packet has been sent.
src_translated_port	int	The NATed network port from which a packet has been sent.
syslog_id	string	The application, process, or OS subsystem that generated the event.
syslog_priority	string	The criticality of an event, as recorded by UNIX syslog.
tcp_flag	string	The TCP flag(s) specified in the event.	Can be one or more of SYN, ACK, FIN, RST, URG, or PSH.
tos	string	The hex bit that specifies TCP 'type of service' (see http://en.wikipedia.org/wiki/Type_of_Service).
transport	string	The transport protocol.	TCP, UDP
ttl	int	The "time to live" of a packet or datagram.
vlan_id	int	The numeric identifier assigned to the virtual local area network (VLAN) specified in the record.
vlan_name	string	The name assigned to the virtual local area network (VLAN) specified in the record.

Packet filtering

Field name	Data type	Description	Possible values
action	string	The action the filtering device (the dvc_bestmatch field) performed on the communication.	allowed, blocked
dest_port	int	The IP port of the packet's destination, such as 22.
direction	string	The direction the packet is traveling.	inbound, outbound
dvc	string	The name of the packet filtering device. If your field is named dvc_host, dvc_ip, or dvc_nt_host you can alias it as dvc to make it CIM-compliant.
rule	string	The rule which took action on the packet, such as 143.
svc_port	int	The IP port of the packet's source, such as 34541.

Proxy

Field name	Data type	Description	Possible values
action	string	The action taken by the proxy.
dest	string	The destination of the network traffic (the remote host).
http_content_type	string	The content-type of the requested HTTP resource.
http_method	string	The HTTP method used to request the resource.	GET, POST, DELETE, and so on.
http_refer	string	The HTTP referrer used to request the HTTP resource.
http_response	int	The HTTP response code.
http_user_agent	string	The user agent used to request the HTTP resource.
product	string	The product name of the vendor technology generating Network Protection data, such as IDP, Providentia, and ASA.
src	string	The source of the network traffic (the client requesting the connection).
status	int	The HTTP response code indicating the status of the proxy request.	404, 302, 500, and so on.
user	string	The user that requested the HTTP resource.
url	string	The URL of the requested HTTP resource.
vendor	string	The vendor technology generating Network Protection data, such as IDP, Providentia, and ASA.

System center

Field name	Data type	Description	Possible values
selinux	string	Values from the SE Linux configuration file.	disabled, enforcing
Startmode	string	The start mode of the given service.	disabled, enabled, auto

System center

Field name	Data type	Description	Possible values
app	string	The running application or service on the system (the src field), such as explorer.exe or sshd.
FreeMBytes	int	The amount of disk space available per drive or mount (the mount field) on the system (the src field).
kernel_release	string	The version of operating system installed on the host (the src field), such as 6.0.1.4 or 2.6.27.30-170.2.82.fc10.x86_64.
label	string	Human-readable version of the SystemUptime value.
mount	string	The drive or mount reporting available disk space (the FreeMBytes field) on the system (the src field).
os	string	The name of the operating system installed on the host (the src), such as Microsoft Windows Server 2003 or GNU/Linux).
PercentProcessorTime	int	The percentage of processor utilization.
setlocaldefs	int	The setlocaldefs setting from the SE Linux configuration.
selinux	string	Values from the SE Linux configuration file.	disabled, enforcing
selinuxtype	string	The SE Linux type (such as targeted).
shell	string	The shell provided to the User Account (the user field) upon logging into the system (the src field).
src_port	int	The TCP/UDP source port on the system (the src field).
sshd_protocol	string	The sshd protocol version.
Startmode	string	The start mode of the given service.	disabled, enabled, auto
SystemUptime	int	The number of seconds since the system (the src) has been "up."
TotalMBytes	int	The total amount of available memory on the system (the src).
UsedMBytes	int	The amount of used memory on the system (the src).
user	string	The User Account present on the system (the src).
updates	int	The number of updates the system (the src) is missing.

Traffic

Field name	Data type	Description	Possible values
dest	string	The destination of the network traffic. If your field is named dest_host, dest_ip, dest_ipv6, or dest_nt_host you can alias it as dest to make it CIM-compliant.
dvc	string	The name of the packet filtering device. If your field is named dvc_host, dvc_ip, or dvc_nt_host you can alias it as dvc to make it CIM-compliant.
src	string	The source of the network traffic. If your field is named src_host, src_ip, src_ipv6, or src_nt_host you can alias it as src to make it CIM-compliant.

Update

Field name	Data type	Description	Possible values
package	string	The name of the installed update.

User information updates

Field name	Data type	Description	Possible values
affected_user	string	A user that has been affected by a change. For example, user fflanda changed the name of user rhallen, so affected_user=rhallen.
affected_user_group	string	The user group affected by a change.
affected_user_group_id	int	The identifier of the user group affected by a change.
affected_user_id	int	The identifier of the user affected by a change.
affected_user_privilege	enumeration	The security context associated with the user affected by a change.	administrator, user, guest/anonymous
user	string	The name of the user affected by the recorded event.
user_group	string	A user group that is the object of an event, expressed in human-readable terms.
user_group_id	int	The numeric identifier assigned to the user group event object.
user_id	int	The system-assigned identifier for the user affected by an event.
user_privilege	enumeration	The security context associated with the object of an event (the affected user).	administrator, user, guest/anonymous
user_subject	string	The name of the user that is the subject of an event--the user executing the action, in other words.
user_subject_id	int	The ID number of the user that is the subject of an event.
user_subject_privilege	enumeration	The security context associated with the subject of an event (the user causing a change).	administrator, user, guest/anonymous

Vulnerability

Field name	Data type	Description	Possible values
category	string	The category of the discovered vulnerability.
dest	string	The host with the discovered vulnerability. If your field is named dest_host, dest_ip, dest_ipv6, or dest_nt_host you can alias it as dest to make it CIM-compliant.
os	string	The operating system of the host containing the vulnerability detected on the client (the src field), such as SuSE Security Update, or cups security update.
severity	string	The severity of the discovered vulnerability.
signature	string	The name of the vulnerability detected on the client (the src field), such as SuSE Security Update, or cups security update.

Windows administration

Field name	Data type	Description	Possible values
object_name	string	The object name (associated only with Windows).
object_type	string	The object type (associated only with Windows).
object_handle	string	The object handle (associated only with Windows).

Standardize your event type tags

The Common Information Model suggests that you use a specific convention when tagging your event types. This convention requires that you set up two categories of tags, and that you give each event type in your system a single tag from both of these categories. The categories are object and status.

This arrangement enables precise event type classification. The object tag denotes what the event is about. What object has been targeted? Is the event talking about a host, a resource, a file, or what? And the status tag provides the status of the action. Was it successful? Failed? Or was it simply an attempt? In addition to these two standard tags, you can add other tags as well.

The three tags in discussion here are:

<objecttag> <statustag>

Some examples of using the standard tags are:

• For a firewall deny event type:

host failure

• For a firewall accept event :

host success

• For a successful database login:

database success

Object event type tags

Use one of these object tags in the first position as defined above.

Tag	Explanation
application	An application-level event.
application av	An anti virus event.
application backdoor	An event using an application backdoor.
application database	A database event.
application database data	An event related to database data.
application dosclient	An event involving a DOS client.
application firewall	An event involving an application firewall.
application im	An instant message-related event.
application peertopeer	A peer to peer-related event.
host	A host-level event.
group	A group-level event
resource	An event involving system resources.
resource cpu	An event involving the CPU.
resource file	An event involving a file.
resources interface	An event involving network interfaces.
resource memory	An event involving memory.
resource registry	An event involving the system registry.
os	An OS-level event.
os process	An event involving an OS-related process
os service	An event involving an OS service.
user	A user-level event

Status event type tags

Use one of these status tags in the third position as defined above.

Tag	Explanation
attempt	An event marking an attempt at something.
deferred	A deferred event.
failure	A failed event.
inprogress	An event marking something progress.
report	A report of a status.
success	A successful event.

Optional tags

For those who want to use standard additional tags when they apply, some suggestions are below.

Tag	Explanation
attack	An event marking an attack.
attack exploit	An event marking the use of an exploit.
attack bruteforce	An event marking a brute force attack.
attack dos	An event marking a denial of service attack.
attack escalation	An event indicating a privilege escalation attack.
infoleak	An event indicating an information leak.
malware	An event marking malware action.
malware dosclient	An event marking malware utilizing a DOS client.
malware spyware	An event marking spyware.
malware trojan	An event marking a trojan.
malware virus	An event marking a virus.
malware worm	An event marking a worm.
recon	An event marking recon probes.
suspicious	An event indicating suspicious activity.

Standardize your host tags

As you may know, it can be problematic to rename hosts directly. Because hosts are identified before event data is indexed, changes to host names are not applied to data that has already been indexed. It's far easier to use tags to group together events from particular hosts.

You can use standardized tags to describe specific hosts and what they do. There are a variety of approaches to host tagging, all of which can be used where appropriate. Some of these methods include:

• What service(s) the host is running.
• What OS the host is running.
• The department the host belongs to.
• What data the host contains.
• What cluster/round robin the host belongs to.

General host tags

These host tags are useful across the board. You can also develop lists of host tags that are appropriate for specific apps.

Tag	Explanation
db	This host is a database.
development	This host is a development box.
dmz	This host is in the DMZ.
dns	This host is a DNS server.
email	This host is an email server.
finance	This host contains financial information.
firewall	This host is a firewall.
highly_critical	This host is highly critical for business purposes.
web	This host is a Web server.

• Was this topic useful?
• Post a comment