Log management using logrotate provided with Linux

During the daily operation and maintenance, it is often necessary to manage, clean up and monitor all kinds of logs. In particular, because of the application bug, dozens of G logs can be written within one hour, resulting in full disk and system failure.

nohup.out,access.log,catalina.out

This article briefly introduces the use of Linux's own logrotate to manage various logs in the operating system.

1. Introduction to logrotate

The logrotate utility is designed to simplify the administration of log files on a system which generates a lot of log files. Logrotate allows for the automatic rotation compression, removal and mailing of log files. Logrotate can be set to handle a log file daily, weekly, monthly or when the log file gets to a certain size.

In order to use it, there are two main areas that need to be modified: one is /etc/logrotate Conf and /etc/logrotate D/ the following documents.

You can either Conf directly defines how to handle your log file. You can also use /logrotate D/ create a new file corresponding to your log to define the log processing behavior. It is recommended to use the directory /logrotate D/ create your own file below to process personalized logs.

logrotate defines how to handle logs, which itself is called periodically by crond.

A production example I used:

/usr/local/nginx/logs/*.log {
    create 0644 root root
    daily
    rotate 2
    missingok
    copytruncate
    ifempty
    compress
    noolddir
}

Save the above contents to nginxlog file and Directory: /etc/logrotate D/nginxlog
Setting permissions: owner=root group=root mode=0644

Test whether the configuration is correct:

lograte -d /etc/logrotate.d/nginxlog

2. logrotate configuration parameters

Logrotate global configuration file: /etc/logrotate Conf

configuration parameter Function description
compress Compress and dump logs through gzip
nocompress Use this parameter when compression is not required
copytruncate It is used to back up and truncate the current log files that are still open; It is a method of copying first and then emptying. There is a time difference between copying and emptying, and some log data may be lost.
nocopytruncate Backup log files without truncation
create mode owner group  Dump file to create a new log file using the specified file mode. Specify the attribute to create a new file during rotation, such as create 0777 nobody nobody
nocreate Do not create a new log file
delaycompress When used with compress, the dumped log file is not compressed until the next dump
nodelaycompress Override the delaycompress option, dump and compress at the same time
errors address  Send the error message during exclusive storage to the specified Email address
ifempty Dump even empty files. This is the default option of logrotate.
notifempty If it is an empty file, do not dump it
mail address  Send the dumped log file to the specified E-mail address
nomail Do not send log files during dump
olddir directory The dumped log file is placed in the specified directory and must be in the same file system as the current log file
noolddir The dumped log file and the current log file are placed in the same directory

prerotate/endscript

Instructions to be executed before logrotate dump, such as modifying file attributes; These two keywords must be on separate lines;
postrotate/endscript Instructions to be executed after logrotate dump, such as restarting (kill -HUP) a service! Must travel independently;
daily Specify dump period as per day
weekly Specify dump cycle as weekly
monthly Specifies that the dump cycle is monthly
rotate count Specify the number of dumps before deleting the log file. 0 means no backup, and 5 means 5 backups are reserved
tabootext [+] list let logrotate Do not dump files with the specified extension. The default extension is: RPM Orig Rpmsave, V, and ~
size Size the log file is dumped when it reaches the specified size. Size can specify bytes (default) and KB (sizek) or MB (sizem)
missingok If the log is lost, continue to scroll to the next log without error
notifempty Do not rotate when the log file is empty
sharedscripts Run the postrotate script to execute the script once after all logs are rotated. If this is not configured, the script will be executed once after each log rotation
dateext Use current date as naming format
dateformat .%s  It is used in conjunction with dateext and appears immediately on the next line. It defines the file name after file cutting. It must be used in conjunction with dateext. Only the four parameters%y%m%d%s are supported
Size (or minsize) log size The log file is dumped when it reaches the specified size. The log size can specify bytes (default) and KB (sizek) or MB(sizem)

Description:

Dump when the log file > = log size.
The following is the legal format: (the unit case of other formats has not been tried)
size = 5 or size 5 (> = dump after 5 bytes)
size = 100k or size 100k
size = 100M or size 100M

Example:

/home/deploy/apps/production.log {
missingok
copytruncate
rotate 10
notifempty
sharedscripts
dateext
dateformat -%Y-%m-%d-%s
size=10M
postrotate
mv /home/deploy/apps/production.log-* /data1/log/rails
gzip /data1/log/rails/production.log-*
endscript
}

Question: what is the difference between rotate and maxage?
Both of them are used to control how many log files are saved. The difference is that rotate is in number, while maxage is in days. If we rotate the logs by day, the difference between the two is not big.

4. nginx log cutting instance

vim /etc/logrotate.d/nginx   #Create nginx log cutting profile

/application/nginx/logs/*.log{
daily
rotate 10
create
dateext
}

logrotate -d /etc/logrotate.d/nginx    Commissioning test   -d debug
logrotate -d /etc/logrotate.d/nginx    Manual cutting log test

ls /application/nginx/logs/           The date indicates the cut log
access.log           bbs.log-20180228   error.log           www.log
access.log-20180228  blog.log           error.log-20180228  www.log-20180228
bbs.log              blog.log-20180228  nginx.pid

Configure the effective time of nginx cutting log

# cat /etc/anacrontab    #This document has an effective date
# /etc/anacrontab: configuration file for anacron

# See anacron(8) and anacrontab(5) for details.
 
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root

# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45

# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22   #The effective time range is from 3:00 to 22:00

#period in days   delay in minutes   job-identifier   command
1       5       cron.daily              nice run-parts /etc/cron.daily
7       25      cron.weekly             nice run-parts /etc/cron.weekly
@monthly 45     cron.monthly            nice run-parts /etc/cron.monthly

In other words, the configured nginx cutting log takes effect between 3:00 a.m. and 22:00 a.m., and the random delay time is 45 minutes

5. Other configuration examples

/var/log/htmlaccess.log {
 errors jim
 notifempty
 nocompress
 weekly
 prerotate
 /usr/bin/chattr -a /var/log/htmlaccess.log
 endscript
 
 postrotate
 /usr/bin/chattr +a /var/log/htmlaccess.log
 endscript
}

Continuous integration system log processing configuration

/var/log/jenkins/jenkins.log /var/log/jenkins/access_log {
    compress
    dateext
    maxage 365      #Retention maximum 365 days
    rotate 99       #Retain up to 99 backups
    size=+4096k
    notifempty
    missingok
    create 644
    copytruncate
}

Custom log processing

/medialog/*.log {
    create 0644 root root
    daily
    rotate 30
    missingok
    copytruncate
    notifempty
    compress
    delaycompress
    olddir /medialog/backlog    # Store archived logs in separate directories
}

 

People will never have enough time to do it well, but they will always have enough time to start over. However, because there is not always a chance to do it again, you must do it better. In other words, people will never have enough time to think about whether they want it, but will always have enough time to regret it

Posted by gls2ro on Wed, 01 Jun 2022 12:33:45 +0530