nginx series 3: log configuration

**

preface

**

One of nginx series: getting started with nginx
nginx Series II: Interpretation of configuration files
nginx series 3: log configuration
nginx series 4: web server
nginx Series 5: load balancing
nginx Series 6: cache Service
nginx Series 7: current limiting configuration
nginx series 8: load balancing using upsync module

Transferred from: Thanks to the original blogger for sorting and sharing

1, nginx access log configuration

1.1 access_log log configuration

access_log is used to define the log level and log location. The syntax is as follows:
Log level: debug > info > notice > warn > error > crit > alert > emerg

Syntax format: access_ Log path [format [buffer=size] [gzip[=level][flush=time] [if=condition]];
access_log off;

Default: access_ Log logs/access Log combined;
Scope: http, server, location, if in location, limit_except

Example 1:

access_log /var/logs/nginx-access.log compression buffer=32k;
  • 1

1.2 log_format defines the log format

Syntax format: log_format name [escape=default|json] string;
Default: log_format combined "...";
Scope: http

Example 1:

log_format compression '$remote_addr - $remote_user [$time_local] '
                       '"$request" $status $bytes_sent '
                       '"$http_referer" "$http_user_agent" "$gzip_ratio"';
access_log /var/logs/nginx-access.log compression buffer=32k;
  • 1
  • 2
  • 3
  • 4

1.3 open_log_file_cache

Using open_log_file_cache to set the log file cache (off by default).

  • max: set the maximum number of file descriptors in the cache. If the cache is full, use the LRU algorithm to close the descriptor.
  • inactive: set the survival time, which is 10s by default
  • min_uses: sets the minimum number of times the log file is used in the inactive time period, and the log file descriptor is recorded in the cache. The default is 1 time
  • valid: set the inspection frequency, 60s by default
  • off: disable caching

Syntax format: open_log_file_cache max=N [inactive=time] [min_uses=N] [valid=time];
open_log_file_cache off;
Default: open_log_file_cache off;
Scope: http, server, location

Example 1

open_log_file_cache max=1000 inactive=20s valid=1m min_uses=2;
  • 1

1.4 common global variables in logs

  • $remote_addr, $http_x_forwarded_for record client IP address
  • $remote_user record client user name
  • $request record the URL and HTTP protocol of the request (GET,POST,DEL, etc.)
  • $status record request status
  • $body_ Bytes_ The number of bytes sent by sent to the client, excluding the size of the response header; This variable is associated with Apache module Mod_ Log_ The "%B" parameter in config is compatible.
  • $bytes_ The total number of bytes sent to the client by sent.
  • $connection serial number of the connection.
  • $connection_requests the current number of requests obtained through a connection.
  • $msec log write time. The unit is seconds and the precision is milliseconds.
  • $pipe if the request is sent through HTTP pipelined, the pipe value is "p", otherwise it is ".".
  • $http_ The referer records the page link from which it is accessed
  • $http_user_agent records client browser related information
  • $request_length the length of the request (including the request line, request header and request body).
  • $request_time request processing time, in seconds, with precision of milliseconds; It starts from reading the first byte of the client until the log is written after the last character is sent to the client.
  • $time_ Iso8601 local time in iso8601 standard format.
  • $time_local local time in the common log format.

2, nginx log debugging skills

2.1 record only fixed IP errors

When you set the log level to debug, if you are debugging an online high traffic website, your error log may record many messages of each request, which will become meaningless.

On events{...} Configure the following in to enable Nginx to record error logs only from your IP.

events {
        debug_connection 1.2.3.4;
}
  • 1
  • 2
  • 3

2.2 debugging nginx rewrite rules

When debugging the rewrite rule, if the rule is written incorrectly, you will only see a 404 page. You can open the nginx rewrite log in the configuration file for debugging.

server {
        error_log    /var/logs/nginx/example.com.error.log;
        rewrite_log on;
}
  • 1
  • 2
  • 3
  • 4

rewrite_log on; When enabled, it will send all rewrite related log information to error_log file, use the [notice] level. Then you can_ Log to view the rewrite information.

2.3 use location to record the log of the specified URL

server {
        error_log    /var/logs/nginx/example.com.error.log;
        location /static/ {
        error_log /var/logs/nginx/static-error.log debug; 
    }         
}
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6

After configuring the above configuration, /static/ related logs will be recorded separately in static error Log file.

nginx log has three parameters

  1. access_log: defines the path and format of the log.
  2. log_format: defines the template of the log.
  3. open_log_file_cache: defines the log file cache.

proxy_set_header X-Forwarded-For: if the program on the back-end Web server needs to obtain the user IP, obtain it from the header. proxy_set_header X-Forwarded-For $remote_addr;

3, Common examples

3.1 main format

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"'
                       '$upstream_addr $upstream_response_time $request_time ';
access_log  logs/access.log  main;
  • 1
  • 2
  • 3
  • 4
  • 5

3.2 json format

log_format logstash_json '{"@timestamp":"$time_iso8601",'
       '"host": "$server_addr",'
       '"client": "$remote_addr",'
       '"size": $body_bytes_sent,'
       '"responsetime": $request_time,'
       '"domain": "$host",'
       '"url":"$request_uri",'
       '"referer": "$http_referer",'
       '"agent": "$http_user_agent",'
       '"status":"$status",'
       '"x_forwarded_for":"$http_x_forwarded_for"}';
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11

Pit:
You can use $uri to change or rewrite the URL in nginx, but you can use $request for log output_ Uri can be replaced. If there are no special business requirements, it can be replaced completely.

3.3 compression format

Compressed information is added to the log.

http {
    log_format compression '$remote_addr - $remote_user [$time_local] '
                           '"$request" $status $body_bytes_sent '
                           '"$http_referer" "$http_user_agent" "$gzip_ratio"';
    server {
        gzip on;
        access_log /spool/logs/nginx-access.log compression;
        ...
    }
}
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10

3.4 upstream format

Increase the time consumed by upstream.

http {
    log_format upstream_time '$remote_addr - $remote_user [$time_local] '
                             '"$request" $status $body_bytes_sent '
                             '"$http_referer" "$http_user_agent"'
                             'rt=$request_time uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';
server {
    access_log /spool/logs/nginx-access.log upstream_time;
    ...
}

}

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11

4, Appendix: reference documents

4.1 count the number of status occurrences

awk '{print $9}' access.log | sort | uniq -c | sort -rn

36461 200
483 500
87 404
9 400
3 302
1 499
1 403
1 301

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10

4.2 display the URL that returns the 302 status code.

awk '($9 ~ /302/)' access.log | awk '{print $7}' | sort | uniq -c | sort -rn
1 /wp-login.php
1 /wp-admin/plugins.php?action=activate&plugin=ewww-image-optimizer%2Fewww-image-optimizer.php&_wpnonce=cc4a379131
1 /wp-admin/
  • 1
  • 2
  • 3
  • 4

Reference documents

How to Configure Custom Access and Error Log Formats in Nginx
How to configure custom access and error log formats in Nginx

Posted by utahcon on Wed, 01 Jun 2022 01:49:09 +0530