Customize log and field extractions for supported sourcetypes

You may need to perform additional customization to the log subscription or the add-on configuration, if, for example:

• The Access Logs type subscription are customized to provide all data necessary for CIM normalization
• The W3C Logs type subscription require a specific field sequence to be configured at the WSA device to make it work.
• Changes that Cisco makes to its Access Log format in future releases require that you update fields order in the add-on.

.

Recommendations for data to be included into Cisco WSA Access and W3C Log types of log subscriptions

By default, events generated by Cisco WSA Access and W3C Logs subscriptions do not contain all the data necessary for event CIM normalization. To make CIM normalization complete, the add-on requires the following data.

Log Field in W3C Logs	W3C specifier	Field description	Field name for extraction
timestamp	%t	Timestamp in UNIX epoch (Gives you date and time).	timestamp
x-elapsed-time	%e	Elapsed time (duration)	x_elapsed_time
cs(Referer)	%<Referer:	Referer	cs_Referer
c-ip	%a	Client IP Address	c_ip
x-resultcode-httpstatus	%w/%h	Result code and the HTTP response code, with a slash (/) in between	x_resultcode_httpstatus
sc-bytes	%s	Response size (header + body)	sc_bytes
cs-method	%y	Method	cs_method
cs-url	%Y	The entire URL	cs_url
s-hostname	%d	Data source or server IP address.	s_hostname
cs-uri	%U	Request URI	cs_uri
cs-username	%A	Authenticated user name	cs_username
cs-mime-type	%y	Response body MIME type	cs_mime_type
x-acltag	%D	ACL decision tag	x_acltag
cs(X-Forwarded-For)	%f	X-Forwarded-For header	cs_X_Forwarded_For
c-port	%F	Client source port	c_port
s-computerName	%N	Server name or destination hostname	s_computerName
s-port	%p	Destination port number	s_port
cs-version	%P	Protocol	cs_version
x-webcat-code-abbr	%XC	URL category abbreviation for the custom URL category assigned to the transaction.	x_webcat_code_abbr
x-wbrs-score	%XW	Decoded WBRS score <-10.0-10.0>	x_wbrs_score
x-webroot-scanverdict	%Xv	Malware scanning verdict from Webroot	x_webroot_scanverdict
x-webroot-threat-name	%Xn	Webroot specific identifier: (Threat name)	x_webroot_threat_name
x-webroot-trr	%Xt	Webroot specific identifier: (Threat Risk Ratio [TRR])	x_webroot_trr
x-webroot-spyid	%Xs	Webroot specific identifier: (Spy ID)	x_webroot_spyid
x-webroot-trace-id	%Xi	Webroot specific scan identifier: (Trace ID)	x_webroot_trace_id
x-mcafee-scanverdict	%Xd	McAfee specific identifier: (scan verdict)	x_mcafee_scanverdict
x-mcafee-filename	%Xe	McAfee specific identifier: (File name yielding verdict)	x_mcafee_filename
x-mcafee-av-scanerror	%Xf	McAfee specific identifier: (scan error)	x_mcafee_av_scanerror
x-mcafee-av-detecttype	%Xg	McAfee specific identifier: (detect type)	x_mcafee_av_detecttype
x-mcafee-av-virustype	%Xh	McAfee specific identifier: (virus type)	x_mcafee_av_virustype
x-mcafee-virus-name	%Xj	McAfee specific identifier: (virus name)	x_mcafee_virus_name
x-sophos-scanverdict	%XY	Sophos specific identifier: (scan verdict)	x_sophos_scanverdict
x-sophos-scanerror	%Xx	Sophos specific identifier: (scan return code)	x_sophos_scanerror
x-sophos-file-name	%Xy	The name of the file in which Sophos found the objectionable content. Applies to responses detected by Sophos only.	x_sophos_file_name
x-sophos-virus-name	%Xz	Sophos specific identifier: (threat name)	x_sophos_virus_name
x-ids-verdict	%Xl	Cisco Data Security Policy scanning verdict.	x-ids-verdict
x-icap-verdict	%Xp	External DLP server scanning verdict	x_icap_verdict
x-webcat-req-code-abbr	%XQ	The predefined URL category verdict determined during request-side scanning, abbreviated	x_webcat_req_code_abbr
x-webcat-resp-code-abbr	%XA	The URL category verdict determined during response-side scanning, abbreviated	x_webcat_resp_code_abbr
x-wbrs-threat-type	%Xk	Web reputation threat type	x_wbrs_threat_type
x-avc-app	%XO	The web application identified by the AVC engine	x_avc_app
x-avc-type	%Xu	The web application type identified by the AVC engine	x_avc_type
x-avc-behavior	%Xb	The web application behavior identified by the AVC engine	x_avc_behavior
x-request-rewrite	%XS	Safe browsing scanning verdict	x_request_rewrite
x-avg-bw	%XB	Average bandwidth of the user if bandwidth limits are defined by the AVC engine	x_avg_bw
x-bw-throttled	%XT	Flag that indicates whether bandwidth limits were applied to the transaction	x_bw_throttled
user-type	%l	Type of user, either local or remote	user_type
x-resp-dvs-threat-name	%X1	Unified response-side anti-malware scanning verdict that provides the malware threat name independent of which scanning engines are enabled	x_resp_dvs_threat_name
x-resp-dvs-scanverdict	%X0	Unified response-side anti-malware scanning verdict that provides the malware category number independent of which scanning engines are enabled	x_resp_dvs_scanverdict
x-resp-dvs-verdictname	%XZ	Unified response-side anti-malware scanning verdict that provides the malware category independent of which scanning engines are enabled	x_resp_dvs_verdictname
x-req-dvs-threat-name	%X4	Request side DVS threat name	x_req_dvs_threat_name
x-req-dvs-scanverdict	%X2	Request side DVS Scan verdict	x_req_dvs_scanverdict
x-req-dvs-verdictname	%X3	Request side DVS verdict name	x_req_dvs_verdictname
x-amp-verdict	%X#1#	Verdict from Advanced Malware Protection file scanning	x_amp_verdict
x-amp-malware-name	%X#2#	Threat name, as determined by Advanced Malware	x_amp_malware_name
x-amp-score	%X#3#	Reputation score from Advanced Malware Protection file scanning	x_amp_score
x-amp-upload-indicator	%X#4#	Indicator of upload and analysis request	x_amp_upload_indicator
x-amp-filename	%X#5#	The name of the file being downloaded and analyzed	x_amp_filename
x-amp-sha	%X#6#	The SHA-256 identifier for this file	x_amp_sha
cs(User-Agent)	%u	User agent	cs_User_Agent
cs-bytes	%q	Response size (header + body)	cs_bytes

Customization for cisco:wsa:squid and cisco:wsa:squid:new sourcetypes (Access Log type subscription)

In version 4.0.0 of the Splunk Add-on for Cisco WSA cisco:wsa:squid and cisco:wsa:squid:new sourcetypes use the same extractions. For new installations use cisco:wsa:squid and keep cisco:wsa:squid:new for backward compatibility.

cisco:wsa:squid and cisco:wsa:squid:new are implemented to consume Access Log type log subscriptions in Squid style. Here is a sample of such an event generated by Cisco WSA v14.5:

1653327021.889 64 10.160.161.111 TCP_MISS/200 657 GET http://10.160.220.221:8080/HelpBarYUI_js?action=GetHelpScreen&urldata=http%3A%2F%2F10.160.220.221%3A8080%2Fcommit%3Freferrer%3Dhttp%3A%2F%2F10.160.220.221%3A8080%2Fsystem_administration%2Flog_subscriptions&parms=&screen=commit&task_name=&task_description=&CSRFKey=14866fd4-fcae-479c-d4c1-98731c748e55 - DIRECT/10.160.220.221 text/plain DEFAULT_CASE_12-DefaultGroup-eun_internal_group-NONE-NONE-NONE-DefaultGroup-NONE <"nc",-3.0,1,"-",0,0,0,1,"-",-,-,-,"-",0,0,"-","-",-,-,"nc",-,"Unknown","-","-","Unknown","Unknown","-","-",82.12,0,-,"Unknown","-",-,"-",-,-,"-","-",-,-,"-",-,-> - -

In general, the standard message itself is sufficient so no additional configuration is required. However, if you want to configure event CIM normalization, you can enrich events with additional data by defining Custom Fields in the log subscription. According to the WSA Access Log file header, default logs map to W3C specifiers in the following order:

W3C specifier	Log Field in W3C Logs
%t	timestamp
%e	x-elapsed-time
%a	c-ip
%w/%h	sc-result-code/sc-http-status
%s	sc-bytes
%2r	cs-method cs-url (equivalent to: %y %Y)
%A	cs-username
%H/%d	s-hierarchy/s-hostname
%c	cs-mime-type
%D	x-acltag
%Xr	scan verdict information
%?BLOCK_SUSPECT_USER_AGENT,MONITOR_SUSPECT_USER_AGENT?%<User-Agent:%!%-%. %)	x-suspect-user-agent

In addition, %Xr specifier represents scan verdict information that expands into the following sequence of specifiers and log field names.

W3C specifier	Log Field in W3C Logs
%XC	x-webcat-code-abbr
%XW	x-wbrs-score
%Xv	x-webroot-scanverdict
"%Xn"	x-webroot-threat-name
%Xt	x-webroot-trr
%Xs	x-webroot-spyid
%Xi	x-webroot-trace-id
%Xd	x-mcafee-scanverdict
"%Xe"	x-mcafee-filename
%Xf	x-mcafee-av-scanerror
%Xg	x-mcafee-av-detecttype
%Xh	x-mcafee-av-virustype
"%Xj"	x-mcafee-virus-name
%XY	x-sophos-scanverdict
%Xx	x-sophos-scanerror
"%Xy"	x-sophos-file-name
"%Xz"	x-sophos-virus-name
%Xl	x-ids-verdict
%Xp	x-icap-verdict
%XQ	x-webcat-req-code-abbr
%XA	x-webcat-resp-code-abbr
"%XZ"	x-resp-dvs-verdictname
"%Xk"	x-wbrs-threat-type
%X#10#	google_translate_enc_url
%XO	x-avc-app
"%Xu"	x-avc-type
"%Xb"	x-avc-behavior
"%XS"	x-request-rewrite
%XB	x-avg-bw
%XT	x-bw-throttled
%l	user-type
"%X3"	x-req-dvs-verdictname
"%X4"	x-req-dvs-threat-name
%X#1#	x-amp-verdict
%X#2#	x-amp-malware-name
%X#3#	x-amp-score
%X#4#	x-amp-upload
%X#5#	x-amp-filename
%X#6#	x-amp-sha
%X#8#	ext_archivescan_blockedfiletype
%Xo	ext_archivescan_verdict
%Xm	ext_archivescan_threatdetail
%XU	ext_wtt_behavior
%X#29#	ext_youtube_url_category

Compare the recommended fields to make sure the following additional specifiers/fields are present for CIM normalization to work correctly. In order to enrich resulting WSA logs with this data, put the following sequence of specifiers into Custom Fields of Access Log subscription configuration: %<Referer: %U %f %F %N %p %P %X1 %X0 %XZ %X2 %u %q

W3C specifier	Log Field in W3C Logs
%<Referer:	cs(Referer)
%U	cs-uri
%f	cs(X-Forwarded-For)
%F	c-port
%N	s-computerName
%p	s-port
%P	cs-version
%X1	x-resp-dvs-threat-name
%X0	x-resp-dvs-scanverdict
%XZ	x-resp-dvs-verdictname
%X2	x-req-dvs-scanverdict
%u	cs(User-Agent)
%q	cs-bytes

As this is a delimiter/position-based format it is important to keep the order as specified above.

Here is an example of a WSA Access Logs enriched event:

1655211836.080 304 10.160.161.111 TCP_MISS_SSL/200 39 CONNECT tunnel://safebrowsing.googleapis.com:443/ - DIRECT/safebrowsing.googleapis.com - DECRYPT_AVC_7-DefaultGroup-DefaultGroup-NONE-NONE-NONE-DefaultGroup-NONE <"IW_srch",9.2,1,"-",-,-,-,-,"-",-,-,-,"-",-,-,"-","-",-,-,"IW_srch",-,"-","Search Engines and Portals","-","Google","Search Engine","Encrypted","-",1.03,0,-,"-","-",-,"-",-,-,"-","-",-,-,"-",-,-> - - - - - 56003 safebrowsing.googleapis.com 443 2 - - - - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" 246

Customization for cisco:wsa:w3c:recommended sourcetype (Access Log type subscription)

In version 4.0.0 of the Splunk Add-on for Cisco WSA cisco:wsa:w3c:recommended is not compatible with the 3.5.0 version of cisco:wsa:w3c:recommended.

To ingest Cisco WSA events as cisco:wsa:w3c:recommended sourcetype, put the following string into Custom Fields setting of an Access Log type of log subscription:

>timestamp: %t x-elapsed-time: %e cs(Referer): %<Referer: c-ip: %a sc-result-code: %w sc-http-status: %h sc-bytes: %s cs-method: %y cs-url: %Y s-hostname: %d cs-uri: %U cs-username: %A cs-mime-type: %c x-acltag: %D cs(X-Forwarded-For): %f c-port: %F s-computerName: %N s-port: %p cs-version: %P x-webcat-code-abbr: %XC x-wbrs-score: %XW x-webroot-scanverdict: %Xv x-webroot-threat-name: %Xn x-webroot-trr: %Xt x-webroot-spyid: %Xs x-webroot-trace-id: %Xi x-mcafee-scanverdict: %Xd x-mcafee-filename: %Xe x-mcafee-av-scanerror: %Xf x-mcafee-av-detecttype: %Xg x-mcafee-av-virustype: %Xh x-mcafee-virus-name: %Xj x-sophos-scanverdict: %XY x-sophos-scanerror: %Xx x-sophos-file-name: %Xy x-sophos-virus-name: %Xz x-ids-verdict: %Xl x-icap-verdict: %Xp x-webcat-req-code-abbr: %XQ x-webcat-resp-code-abbr: %XA x-wbrs-threat-type: %Xk x-avc-app: %XO x-avc-type: %Xu x-avc-behavior: %Xb x-request-rewrite: %XS x-avg-bw: %XB x-bw-throttled: %XT user-type: %l x-resp-dvs-threat-name: %X1 x-resp-dvs-scanverdict: %X0 x-resp-dvs-verdictname: %XZ x-req-dvs-threat-name: %X4 x-req-dvs-scanverdict: %X2 x-req-dvs-verdictname: %X3 x-amp-verdict: %X#1# x-amp-malware-name: %X#2# x-amp-score: %X#3# x-amp-upload: %X#4# x-amp-filename: %X#5# x-amp-sha: %X#6# cs(User-Agent): %u cs-bytes: %q

Above configuration will append the key-value sequence to the standard Access Log events and make them look as follows:

1654615437.003 55976 10.160.161.111 NONE/503 0 POST http://update.googleapis.com/service/update2?cup2key=11:pGuIn0TEz3CvKQMxp6_Pw9qkGmMRHPk4WCUsV8L0imA&cup2hreq=734699c8b380b746ecef4bdf9bdde2e6a99f6e7e7a0fdb9ebf433749b9d801bf - NONE/update.googleapis.com - OTHER-NONE-DefaultGroup-NONE-NONE-NONE-DefaultGroup-NONE <"err",-0.8,0,"-",0,0,0,1,"-",-,-,-,"-",-,-,"-","-",-,-,"err",-,"-","-","-","Unknown","Unknown","-","-",0.00,0,-,"Unknown","-",-,"-",-,-,"-","-",-,-,"-",-,-> - - timestamp: 1654615437.003 x-elapsed-time: 55976 cs(Referer): - c-ip: 10.160.161.111 sc-result-code: NONE sc-http-status: 503 sc-bytes: 0 cs-method: POST cs-url: http://update.googleapis.com/service/update2?cup2key=11:pGuIn0TEz3CvKQMxp6_Pw9qkGmMRHPk4WCUsV8L0imA&cup2hreq=734699c8b380b746ecef4bdf9bdde2e6a99f6e7e7a0fdb9ebf433749b9d801bf s-hostname: update.googleapis.com cs-uri: service/update2?cup2key=11:pGuIn0TEz3CvKQMxp6_Pw9qkGmMRHPk4WCUsV8L0imA&cup2hreq=734699c8b380b746ecef4bdf9bdde2e6a99f6e7e7a0fdb9ebf433749b9d801bf cs-username: - cs-mime-type: - x-acltag: OTHER-NONE-DefaultGroup-NONE-NONE-NONE-DefaultGroup-NONE cs(X-Forwarded-For): - c-port: 54017 s-computerName: update.googleapis.com s-port: 80 cs-version: 1 x-webcat-code-abbr: err x-wbrs-score: -0.8 x-webroot-scanverdict: 0 x-webroot-threat-name: - x-webroot-trr: "0" x-webroot-spyid: 0 x-webroot-trace-id: 0 x-mcafee-scanverdict: 1 x-mcafee-filename: - x-mcafee-av-scanerror: - x-mcafee-av-detecttype: - x-mcafee-av-virustype: - x-mcafee-virus-name: - x-sophos-scanverdict: - x-sophos-scanerror: - x-sophos-file-name: - x-sophos-virus-name: - x-ids-verdict: - x-icap-verdict: - x-webcat-req-code-abbr: err x-webcat-resp-code-abbr: - x-wbrs-threat-type: - x-avc-app: "Unknown" x-avc-type: "Unknown" x-avc-behavior: - x-request-rewrite: - x-avg-bw: 0.00 x-bw-throttled: 0 user-type: - x-resp-dvs-threat-name: - x-resp-dvs-scanverdict: - x-resp-dvs-verdictname: - x-req-dvs-threat-name: - x-req-dvs-scanverdict: 0 x-req-dvs-verdictname: "Unknown" x-amp-verdict: - x-amp-malware-name: - x-amp-score: - x-amp-upload: - x-amp-filename: - x-amp-sha: - cs(User-Agent): "Google Update/1.3.36.132;winhttp;cup-ecdsa" cs-bytes: 1722

This format relies on the custom (key-value) part. It expects that each value specifier in the custom part is added together with the corresponding token. The token must precede the specifier and be separated from it by column and space, i.e. '<token>: <specifier>'. The values for tokens should be taken from the WSA documentation section "Access Log Format Specifiers and W3C Log File Fields" from the table column "Log Field in W3C Logs" according to the specifier used, or from the table of recommended fields located earlier in this documentation. This way tokens are actually W3C field names representing keys while specifiers represent field values. For example, if it is required to add the specifier %c, which means 'response body MIME type', the following string should be added to format: cs-mime-type: %c.

The key-value approach allows the implementation of field extractions separately and independent of the order of values in the event, which makes TA more resistant to human mistakes and possible future event format changes made by the vendor. However due to information duplication in standard and custom event parts and inclusion of field names into events, customers should expect higher Splunk license usage in comparison to position based formats

Customization for cisco:wsa:w3c sourcetype (W3C Log type subscription)

cisco:wsa:w3c sourcetype expects logs generated by W3C Log type subscription. Just like squid-styled Access Log type subscriptions, this type of log subscription is also position-based. This format lets you customize data that should be included into events by defining the list of required W3C fields and their exact position in an event.

Adjusting log subscription configuration

The easiest way to start ingesting events for cisco:wsa:w3c sourcetype is to configure Cisco WSA W3C Log subscription by replacing Selected Log Fields list with the below recommended field sequence:

timestamp
x-elapsed-time
cs(Referer)
c-ip
x-resultcode-httpstatus
sc-bytes
cs-method
cs-url
s-hostname
cs-uri
cs-username
cs-mime-type
x-acltag
cs(X-Forwarded-For)
c-port
s-computerName
s-port
cs-version
x-webcat-code-abbr
x-wbrs-score
x-webroot-scanverdict
x-webroot-threat-name
x-webroot-trr
x-webroot-spyid
x-webroot-trace-id
x-mcafee-scanverdict
x-mcafee-filename
x-mcafee-av-scanerror
x-mcafee-av-detecttype
x-mcafee-av-virustype
x-mcafee-virus-name
x-sophos-scanverdict
x-sophos-scanerror
x-sophos-file-name
x-sophos-virus-name
x-ids-verdict
x-icap-verdict
x-webcat-req-code-abbr
x-webcat-resp-code-abbr
x-wbrs-threat-type
x-avc-app
x-avc-type
x-avc-behavior
x-request-rewrite
x-avg-bw
x-bw-throttled
user-type
x-resp-dvs-threat-name
x-resp-dvs-scanverdict
x-resp-dvs-verdictname
x-req-dvs-threat-name
x-req-dvs-scanverdict
x-req-dvs-verdictname
x-amp-verdict
x-amp-malware-name
x-amp-score
x-amp-upload-indicator
x-amp-filename
x-amp-sha
cs(User-Agent)
cs-bytes

See Cisco WSA 14.5 documentation for more information:

1. Clean up Selected Log Fields list.
2. Copy-paste the above sequence to the Custom Fields edit box.
3. Click the Add >> button to populate fields into Selected Log Fields list.
4. Verify that the field order is not broken.

Adjusting the add-on configuration

To keep the default fields order or specify any other field order you want in the W3C Log subscription, specify the correct field sequence for the FIELDS parameter in cisco_wsa_latest_delim_w3c stanza. By default, this stanza is configured to accept the recommended field sequence mentioned above and it is defined as follows

[cisco_wsa_latest_delim_w3c]
DELIMS = " "
FIELDS = timestamp,x_elapsed_time,cs_Referer,c_ip,x_resultcode_httpstatus,sc_bytes,cs_method,cs_url,s_hostname,cs_uri,cs_username,cs_mime_type,x_acltag,cs_X_Forwarded_For,c_port,s_computerName,s_port,cs_version,x_webcat_code_abbr,x_wbrs_score,x_webroot_scanverdict,x_webroot_threat_name,x_webroot_trr,x_webroot_spyid,x_webroot_trace_id,x_mcafee_scanverdict,x_mcafee_filename,x_mcafee_av_scanerror,x_mcafee_av_detecttype,x_mcafee_av_virustype,x_mcafee_virus_name,x_sophos_scanverdict,x_sophos_scanerror,x_sophos_file_name,x_sophos_virus_name,x_ids_verdict,x_icap_verdict,x_webcat_req_code_abbr,x_webcat_resp_code_abbr,x_wbrs_threat_type,x_avc_app,x_avc_type,x_avc_behavior,x_request_rewrite,x_avg_bw,x_bw_throttled,user_type,x_resp_dvs_threat_name,x_resp_dvs_scanverdict,x_resp_dvs_verdictname,x_req_dvs_threat_name,x_req_dvs_scanverdict,x_req_dvs_verdictname,x_amp_verdict,x_amp_malware_name,x_amp_score,x_amp_upload_indicator,x_amp_filename,x_amp_sha,cs_User_Agent,cs_bytes

To specify a custom field sequence that fits your W3C logs:

1. Create new transforms.conf file in TA local folder $SPLUNK_HOME/etc/apps/Splunk_TA_cisco-wsa/default/transforms.conf
2. Copy-paste cisco_wsa_latest_delim_w3c stanza with it's content from the above example or from default tratsforms.conf file $SPLUNK_HOME/etc/apps/Splunk_TA_cisco-wsa/default/transforms.conf into the newly created transfroms.conf file
3. Adjust comma-separated field list assigned to FIELDS parameter to fit the order defined in the Selected Log Fields list of W3C Logs subscription.
4. Save the file.
5. Restart the Splunk platform for the changes to take effect.