CDN-批量下载日志

2021-10-01 08:07:19 浏览数 (1)

获取访问日志下载链接：https://cloud.tencent.com/document/api/228/39232

DescribeCdnDomainLogs接口返回内容DescribeCdnDomainLogs接口返回内容

收集 LogPath 中的URL链接就可以了，将这些URL链接写到 url.list 文件中，通过 SHELL 脚本批量下载访问日志

SHELL 脚本内容

代码语言：shell复制
#!/bin/bash

# url.list 文件格式
# 可批量下载，每行一条日志下载链接
# https://log-download.cdn.zijiebao.com/20210329/22/2021032922-test.followme.com.gz?st=MUi0MC1UTjMN1QpaSOcTNw&e=3234152804
# https://log-download.cdn.zijiebao.com/20210329/20/2021032920-test.followme.com.gz?st=PUrkOwLNapq0Yz5f3Zul-A&e=3234152804
#
if [ $# != 1 ];then
    echo "Usage: $0 url.list"
    exit 1
fi
File=$1
if [ ! -f $File ];then
    echo "文件 $File 未找到"
    exit 1
else
    for i in $(cat $File);do
        mkdir -p ./fwlog
        wget "$i" -O ./fwlog/$(echo $i | awk -F'/|?' '{print $6}')
    done
fi

脚本使用方法

执行命令：

代码语言：javascript复制
chmod  x cdnlogw.sh && ./cdnlogdw.sh url.list
代码语言：javascript复制
[root@tx-cdn fw]# ls
cdnlogdw.sh  url.list
[root@tx-cdn fw]# 
[root@tx-cdn fw]# cat cdnlogdw.sh # 查看脚本内容
#!/bin/bash

# url.list 文件格式
# 可批量下载，每行一条日志下载链接
# https://log-download.cdn.zijiebao.com/20210329/22/2021032922-test.followme.com.gz?st=MUi0MC1UTjMN1QpaSOcTNw&e=3234152804
# https://log-download.cdn.zijiebao.com/20210329/20/2021032920-test.followme.com.gz?st=PUrkOwLNapq0Yz5f3Zul-A&e=3234152804
#
if [ $# != 1 ];then
    echo "Usage: $0 url.list"
    exit 1
fi
File=$1
if [ ! -f $File ];then
    echo "文件 $File 未找到"
    exit 1
else
    for i in $(cat $File);do
	mkdir -p ./fwlog
        wget "$i" -O ./fwlog/$(echo $i | awk -F'/|?' '{print $6}')
    done
fi
[root@tx-cdn fw]# 
[root@tx-cdn fw]# cat url.list # 查看 url.list 文件内容
https://log-download.cdn.zijiebao.com/20210929/23/2021092923-test.followme.com.gz?st=Uyk-CuK65q6n31pkq2OuWQ&e=3266108368
https://log-download.cdn.zijiebao.com/20210929/22/2021092922-test.followme.com.gz?st=xPAqlc12DU0bHQxkjqH5vQ&e=3266108368
[root@tx-cdn fw]# 
[root@tx-cdn fw]# ./cdnlogdw.sh url.list # 执行脚本批量下载访问日志
--2021-09-30 22:28:42--  https://log-download.cdn.zijiebao.com/20210929/23/2021092923-test.followme.com.gz?st=Uyk-CuK65q6n31pkq2OuWQ&e=3266108368
Resolving log-download.cdn.zijiebao.com (log-download.cdn.zijiebao.com)... 109.244.212.181
Connecting to log-download.cdn.zijiebao.com (log-download.cdn.zijiebao.com)|109.244.212.181|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 812 [application/octet-stream]
Saving to: ‘./fwlog/2021092923-test.followme.com.gz’

100%[===================================================================================================================================>] 812         --.-K/s   in 0s      

2021-09-30 22:28:42 (206 MB/s) - ‘./fwlog/2021092923-test.followme.com.gz’ saved [812/812]

--2021-09-30 22:28:42--  https://log-download.cdn.zijiebao.com/20210929/22/2021092922-test.followme.com.gz?st=xPAqlc12DU0bHQxkjqH5vQ&e=3266108368
Resolving log-download.cdn.zijiebao.com (log-download.cdn.zijiebao.com)... 109.244.212.181
Connecting to log-download.cdn.zijiebao.com (log-download.cdn.zijiebao.com)|109.244.212.181|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 334 [application/octet-stream]
Saving to: ‘./fwlog/2021092922-test.followme.com.gz’

100%[===================================================================================================================================>] 334         --.-K/s   in 0s      

2021-09-30 22:28:42 (76.2 MB/s) - ‘./fwlog/2021092922-test.followme.com.gz’ saved [334/334]

[root@tx-cdn fw]# 
[root@tx-cdn fw]# ls
cdnlogdw.sh  fwlog  url.list
[root@tx-cdn fw]# ls fwlog/ # 访问日志存放在 fwlog/ 目录中
2021092922-test.followme.com.gz  2021092923-test.followme.com.gz
[root@tx-cdn fw]# 
[root@tx-cdn fw]#

也可以直接在命令行运行以下命令

代码语言：javascript复制
for i in $(cat url.list);do mkdir -p ./fwlog;wget "$i" -O ./fwlog/$(echo $i | awk -F'/|?' '{print $6}');done
cdn

0 人点赞