Configure enhanced log formatting on the Apache Web Server using httpd.conf

Configure log formatting on the Apache Web Server using httpd.conf. This lets the Splunk Add-on for Apache Web Server monitor your deployment's log files.

To make sure you have all the required fields present in apache access_log, Splunk best practice is to use an enhanced custom log format in the apache httpd.conf file. For more information, see Configure monitor inputs for the Splunk Add-on for Apache Web Server.

The Splunk Add-on for Apache Web Server now supports the default out-of-the-box log format of Apache Web Server; the add-on extracts all possible CIM fields where present and makes them searchable. However, this log format is missing many critical CIM fields (e.g. user, hostname, etc), so note that the add-on does not apply CIM Data Model tags to those events. Splunk best practice is still to use an enhanced custom log format which requires modifications to httpd.conf, and use the apache:access:kv or apache:access:json sourcetypes. To use the default out-of-the-box events, ignore the below steps & proceed with Configure monitor inputs for the Splunk Add-on for Apache Web Server.
1. Open the httpd.conf in the Apache Web Server installation folder $APACHE_HOME/etc/apache/conf/httpd.conf

   The default location of httpd.conf might be different on different platforms.

2. Look for the statement <IfModule log_config_module> and replace the whole block with the following stanza:

   <IfModule log_config_module>
       #
       # The following directives define some format nicknames for use with
       # a CustomLog directive (see below).
       #
       LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
       LogFormat "%h %l %u %t \"%r\" %>s %b" common
   
    <IfModule logio_module>
       # You need to enable mod_logio.c to use %I and %O
       LogFormat "time=%{%s}t.%{usec_frac}t, bytes_in=%I, bytes_out=%O, cookie=\"%{Cookie}i\", server=%v, dest_port=%p, http_content_type=\"%{Content-type}i\", http_method=\"%m\", http_referrer=\"%{Referer}i\", http_user_agent=\"%{User-agent}i\", ident=\"%l\", response_time_microseconds=%D, client=%h, status=%>s, uri_path=\"%U\", uri_query=\"%q\", user=\"%u\"" splunk_kv
   
       #LogFormat "{\"time\":\"%{%s}t.%{usec_frac}t\", \"bytes_in\":\"%I\", \"bytes_out\":\"%O\", \"cookie\":\"%{Cookie}i\", \"server\":\"%v\", \"dest_port\":\"%p\", \"http_content_type\":\"%{Content-type}i\", \"http_method\":\"%m\", \"http_referrer\":\"%{Referer}i\", \"http_user_agent\":\"%{User-agent}i\", \"ident\":\"%l\", \"response_time_microseconds\":\"%D\", \"client\":\"%h\", \"status\":\"%>s\", \"uri_path\":\"%U\", \"uri_query\":\"%q\", \"user\":\"%u\"}" splunk_json    
     
     #LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
   
   </IfModule>
       #
       # The location and format of the access logfile (Common Logfile Format).
       # If you do not define any access logfiles within a <VirtualHost>
       # container, they will be logged here.  Contrariwise, if you *do*
       # define per-<VirtualHost> access logfiles, transactions will be
       # logged therein and *not* in this file.
       #
       # CustomLog "logs/access_log" common
       #
       # If you prefer a logfile with access, agent, and referer information
       # (Combined Logfile Format) you can use the following directive.
       #
       CustomLog "logs/access_log" splunk_kv
       #CustomLog "logs/access_log" splunk_json
       #CustomLog "logs/access_log" combined
   </IfModule>
   

3. Choose either the splunk_kv or splunk_json format for access logs. Only one format can be enabled at a time. The information provided by either splunk_kv and splunk_json is the same. The difference is only in formatting. By default, splunk_kv is enabled and the splunk_json is disabled. To enable the splunk_json format:
   1. Comment out the splunk_kv definition and log file directives.
   2. Uncomment the splunk_json definition and log file directives.

The KV pair formatting is simpler when compared to JSON formatting. The best practice is to include new custom fields, if required, in KV format. If applied in JSON format, then it must comply with strict JSON formatting.

4. Validate the syntax of the conf file after you make your changes.

   sudo apache2ctl configtest

   or
   

   sudo httpd -t

   If the output says syntax ok, proceed.
5. Restart Apache Web Server. If the log format in Apache Web Server is configured correctly, you receive log files that look like this:
   • For splunk_kv

     time=###TIME###.000000, bytes_in=###BYTES_IN###, bytes_out=###BYTES_OUT###, cookie="###COOKIE###", server=C6852495051.domain, dest_port=###DEST_PORT###, http_content_type="###HTTP_CONTENT_TYPE###", http_method="###HTTP_METHOD###", http_referrer="###HTTP_REFERRER###", http_user_agent="###HTTP_USER_AGENT###", ident="1", response_time_microseconds=###RESPONSE_TIME_MICROSECONDS###, client=###CLIENT###, status=###STATUS###, uri_path="###URI_PATH###", uri_query="###URI_QUERY###", user="xyz123"
     

   • For splunk_json

     {"time":"###TIME###.000000", "bytes_in":"###BYTES_IN###", "bytes_out":"###BYTES_OUT###", "cookie":"###COOKIE###", "server":"C6852495051.domain", "dest_port":"###DEST_PORT###", "http_content_type":"###HTTP_CONTENT_TYPE###", "http_method":"###HTTP_METHOD###", "http_referrer":"###HTTP_REFERRER###", "http_user_agent":"###HTTP_USER_AGENT###", "ident":"1", "response_time_microseconds":"###RESPONSE_TIME_MICROSECONDS###", "client":"###CLIENT###", "status":"###STATUS###", "uri_path":"###URI_PATH###", "uri_query":"###URI_QUERY###", "user":"xyz123"}
     

The new fields will be auto extracted without making any change in the add-on's field extraction.