Configure NGINX logging and monitoring

You need to set up the NGINX logging and monitoring to enable the Splunk Add-on for NGINX to collect data from the NGINX server including access log, error log, and performance metrics.

Configure NGINX access log

NGINX writes information about client requests in the access log right after the request is processed. By default, the access log is located at /var/log/nginx/access.log, and the information is written to the log in the predefined combined format. You can override the default settings and change the format of logged messages by editing the NGINX configuration file (/etc/nginx/nginx.conf by default).
The Splunk Add-on for NGINX can ingest the NGINX access log in both the predefined combined format and the custom key-value pair format. Splunk recommends using the custom key-value pair format, which contains more verbose information and is easier to parse.

Default NGINX access log

For information about setting up the default NGINX access log, refer to the NGINX documentation:
https://www.nginx.com/resources/admin-guide/logging-and-monitoring/#access_log .

Custom NGINX access log

Edit the NGINX configuration file (/etc/nginx/nginx.conf by default) and use the log_format directive to define the format of logged messages based on your requirements.

Here is an example of logging in raw format for nginx:plus:access source type:

log_format main '$remote_addr $server_name $remote_user [$time_local] "$request" '
                           '$status $body_bytes_sent "$http_referer" '
                           '"$http_user_agent" "$http_x_forwarded_for" $server_port '
                           '$upstream_bytes_received "$sent_http_content_type" $host "$https" "$http_cookie"';

Here is an example of logging in kv format for nginx:plus:kv source type:

log_format kv 'site="$server_name" server="$host" dest_port="$server_port" dest_ip="$server_addr" '
                       'src="$remote_addr" src_ip="$realip_remote_addr" user="$remote_user" '
                       'time_local="$time_local" protocol="$server_protocol" status="$status" '
                       'bytes_out="$bytes_sent" bytes_in="$upstream_bytes_received" '
                       'http_referer="$http_referer" http_user_agent="$http_user_agent" '
                       'nginx_version="$nginx_version" http_x_forwarded_for="$http_x_forwarded_for" '
                       'http_x_header="$http_x_header" uri_query="$query_string" uri_path="$uri" '
                       'http_method="$request_method" response_time="$upstream_response_time" '
                        'cookie="$http_cookie" request_time="$request_time" category="$sent_http_content_type" https="$https"';

Note: It is recommended to use kV format instead of a raw format for the access log.

See the full list of variables that can you can capture in the log.

For more information about configuring ngx_http_log_module, refer to the official NGINX documentation.

Set up NGINX error log

NGINX writes information about encountered issues of different severity levels to the error log. For information about setting up the NGINX error log, refer to https://www.nginx.com/resources/admin-guide/logging-and-monitoring/#error_log .

Set up NGINX live activity monitoring

NGINX Plus provides a real-time live activity monitoring interface that shows key load and performance metrics of your server infrastructure. These metrics can be represented as a RESTful JSON interface and live JSON data can be ingested into Splunk. You need to enable collecting statistics in the NGINX Plus configuration file. For information about setting live activity monitoring, see https://www.nginx.com/resources/admin-guide/Monitoring/ .

Configure the NGINX App Protect Security log

Security logs (also known as Request logs or Traffic logs) contain information on HTTP requests and responses, how App Protect processes them, and the final decision made based on the configured policy parameters. The policy configuration defines the information contained in the Security log, such as whether requests are passed, blocked or alerted, due to violations, attack signatures, and other criteria.

For information about setting up the default NGINX App Protect Security log, refer to the NGINX documentation.

Edit the /etc/app_protect/conf/log_default.json file and change the format from default to splunk.

For example:

{
    "filter": {
        "request_type": "illegal"
    },
 
    "content": {
        "format": "splunk",
        "max_request_size": "any",
        "max_message_size": "5k"
    }
}