运维记录 - 业务日志清理功能

2019-09-29 10:58:16 浏览数 (1)

线上某些系统业务跑一段时间后，日志就会越来越多，考虑到业务机器磁盘有限，需要添加业务日志清理功能。根据日志所在分区磁盘使用情况来判断是否清理日志，比如当日志分区磁盘空间使用超过90%时，将一周前的日志打包转移到别处 (别的分区下或远程存储设备上)。脚本 (/opt/script/log_clear.sh) 如下:

代码语言：javascript复制
#!/bin/bash

#定义日志所在分区当前空间所占比例数(去掉%)。grep -w表示精准匹配，只匹配"/"这个分区
LOG_PARTITION=$(`which df` -h|awk '{print $5,$6}'|grep -w "/"|cut -d" " -f1|awk -F"%" '{print $1}')
#定义一周前的日期，用于日志分区空间超过设定的阈值后保留的份数(这里保留一周的日志)
KEEP_DATE=`date -d '-7 days'  %Y%m%d`
#定义日志路径
LOG_DIR=/opt/log/kevin
#定义日志备份路径(即当日志分区空间超过阈值后，将日志转移到该处。这里就转移到了本地home分区下，也可以转移到别的机器或远程存储设备上)
LOG_BACK_DIR=/home/log/kevin

#确保日志备份路径存在，如果不存在，则创建该路径。
if [ ! -d ${LOG_BACK_DIR} ];then
   `which mkdir` -p ${LOG_BACK_DIR}
else
   echo "${LOG_BACK_DIR} is exist" >/dev/null 2>&1
fi

#当日志分区当前空间超过90%时执行的动作
if [ ${LOG_PARTITION} -ge 90 ];then
   
   #切换到日志路径下
   cd ${LOG_DIR}
   #截取日志文件中的日期，比如stepweb_20190915.log日志文件，则截取20190915
   for LOG_DATE in $(ls -l ${LOG_DIR}|awk '{print $9}'|cut -d"_" -f2|awk -F"." '{print $1}')
   do
     #当日志文件中截取的日期是一周之前的日志时执行的动作
     if [ ${LOG_DATE} -lt ${KEEP_DATE} ];then
       #打包，转移，删除
       #注意这里tar使用-P参数，因为tar默认为相对路径，使用绝对路径的话会报错"Removing leading `/’ from member names",使用-P参数就解决了该问题
       `which tar` -zvPcf ${LOG_DIR}/stepweb_${LOG_DATE}.log.tar.gz ${LOG_DIR}/stepweb_${LOG_DATE}.log
       mv ${LOG_DIR}/stepweb_${LOG_DATE}.log.tar.gz ${LOG_BACK_DIR}/
       rm -rf ${LOG_DIR}/stepweb_${LOG_DATE}.log
     else
       echo "keep stepweb_${LOG_DATE}.log" >/dev/null 2>&1
     fi
   done
else
   echo "${LOG_PARTITION} is available" >/dev/null 2>&1
fi

根据上面脚本验证下：

代码语言：javascript复制
查看本机磁盘使用情况，日志所在分区磁盘当前使用率达到92%了
[root@yyweb kevin]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  46G   4G     92% /
devtmpfs                 3.9G     0  3.9G   0% /dev
tmpfs                    3.9G     0  3.9G   0% /dev/shm
tmpfs                    3.9G   34M  3.8G   1% /run
tmpfs                    3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1               1014M  183M  832M  19% /boot
/dev/mapper/centos-home  142G   33M  142G   1% /home

日志所在路径
[root@yyweb kevin]# pwd
/opt/log/kevin

日志情况
[root@yyweb kevin]# ls
stepweb_20190810.log  stepweb_20190817.log  stepweb_20190824.log  stepweb_20190901.log  stepweb_20190908.log  stepweb_20190915.log
stepweb_20190811.log  stepweb_20190818.log  stepweb_20190825.log  stepweb_20190902.log  stepweb_20190909.log  stepweb_20190916.log
stepweb_20190812.log  stepweb_20190819.log  stepweb_20190826.log  stepweb_20190903.log  stepweb_20190910.log  stepweb_20190917.log
stepweb_20190813.log  stepweb_20190820.log  stepweb_20190827.log  stepweb_20190904.log  stepweb_20190911.log  stepweb_20190918.log
stepweb_20190814.log  stepweb_20190821.log  stepweb_20190828.log  stepweb_20190905.log  stepweb_20190912.log  stepweb_20190919.log
stepweb_20190815.log  stepweb_20190822.log  stepweb_20190829.log  stepweb_20190906.log  stepweb_20190913.log
stepweb_20190816.log  stepweb_20190823.log  stepweb_20190830.log  stepweb_20190907.log  stepweb_20190914.log

[root@yyweb kevin]# ls /home/
[root@yyweb kevin]# 

执行日志清理脚本
[root@yyweb kevin]# sh /opt/script/log_clear.sh 
/opt/log/kevin/stepweb_20190810.log
/opt/log/kevin/stepweb_20190811.log
/opt/log/kevin/stepweb_20190812.log
/opt/log/kevin/stepweb_20190813.log
/opt/log/kevin/stepweb_20190814.log
/opt/log/kevin/stepweb_20190815.log
/opt/log/kevin/stepweb_20190816.log
/opt/log/kevin/stepweb_20190817.log
/opt/log/kevin/stepweb_20190818.log
/opt/log/kevin/stepweb_20190819.log
/opt/log/kevin/stepweb_20190820.log
/opt/log/kevin/stepweb_20190821.log
/opt/log/kevin/stepweb_20190822.log
/opt/log/kevin/stepweb_20190823.log
/opt/log/kevin/stepweb_20190824.log
/opt/log/kevin/stepweb_20190825.log
/opt/log/kevin/stepweb_20190826.log
/opt/log/kevin/stepweb_20190827.log
/opt/log/kevin/stepweb_20190828.log
/opt/log/kevin/stepweb_20190829.log
/opt/log/kevin/stepweb_20190830.log
/opt/log/kevin/stepweb_20190901.log
/opt/log/kevin/stepweb_20190902.log
/opt/log/kevin/stepweb_20190903.log
/opt/log/kevin/stepweb_20190904.log
/opt/log/kevin/stepweb_20190905.log
/opt/log/kevin/stepweb_20190906.log
/opt/log/kevin/stepweb_20190907.log
/opt/log/kevin/stepweb_20190908.log
/opt/log/kevin/stepweb_20190909.log
/opt/log/kevin/stepweb_20190910.log
/opt/log/kevin/stepweb_20190911.log
/opt/log/kevin/stepweb_20190912.log

日志清理后，日志路径下只保留了最近一周的日志
[root@yyweb kevin]# ls
stepweb_20190913.log  stepweb_20190915.log  stepweb_20190917.log  stepweb_20190919.log
stepweb_20190914.log  stepweb_20190916.log  stepweb_20190918.log

一周之前的日志被打包转移到/home/log/kevin下了
[root@yyweb kevin]# ls /home/log/kevin/
stepweb_20190810.log.tar.gz  stepweb_20190817.log.tar.gz  stepweb_20190824.log.tar.gz  stepweb_20190901.log.tar.gz  stepweb_20190908.log.tar.gz
stepweb_20190811.log.tar.gz  stepweb_20190818.log.tar.gz  stepweb_20190825.log.tar.gz  stepweb_20190902.log.tar.gz  stepweb_20190909.log.tar.gz
stepweb_20190812.log.tar.gz  stepweb_20190819.log.tar.gz  stepweb_20190826.log.tar.gz  stepweb_20190903.log.tar.gz  stepweb_20190910.log.tar.gz
stepweb_20190813.log.tar.gz  stepweb_20190820.log.tar.gz  stepweb_20190827.log.tar.gz  stepweb_20190904.log.tar.gz  stepweb_20190911.log.tar.gz
stepweb_20190814.log.tar.gz  stepweb_20190821.log.tar.gz  stepweb_20190828.log.tar.gz  stepweb_20190905.log.tar.gz  stepweb_20190912.log.tar.gz
stepweb_20190815.log.tar.gz  stepweb_20190822.log.tar.gz  stepweb_20190829.log.tar.gz  stepweb_20190906.log.tar.gz
stepweb_20190816.log.tar.gz  stepweb_20190823.log.tar.gz  stepweb_20190830.log.tar.gz  stepweb_20190907.log.tar.gz

再贴一个简单的日志处理脚本

代码语言：javascript复制
#!/usr/bin/sh
#根据系统/服务/日志保留天数三个参数压缩日志
#usage: sh clearlog.sh sysname appname keepdays
sysName=$1
appName=$2
keepDay=$3
logDir=/var/log/${sysName}/${appName}
logFile=${appName}.log
cd ${logDir}
find ./ -name "${logFile}.*[0-9][0-9]" -mtime  ${keepDay} -exec gzip {} ;
script sh 磁盘 脚本 日志

0 人点赞