第十一章·Filebeat-使用Filebeat收集日志

2022-09-26 11:30:40 浏览数 (1)
  • Filebeat介绍及部署
  • Filebeat收集单类型日志到本地文件
  • Filebeat收集单类型多个日志到Logstash
  • Filebeat收集单类型多个日志到Redis
  • Filebeat收集多类型日志到Redis
  • Filebeat收集多类型日志输出到多个目标

-曾老湿, 江湖人称曾老大。

-多年互联网运维工作经验，曾负责过大规模集群架构自动化运维管理工作。 -擅长Web集群架构与自动化运维，曾负责国内某大型金融公司运维工作。 -devops项目经理兼DBA。 -开发过一套自动化运维平台（功能如下）： 1)整合了各个公有云API，自主创建云主机。 2)ELK自动化收集日志功能。 3)Saltstack自动化运维统一配置管理工具。 4)Git、Jenkins自动化代码上线及自动化测试平台。 5)堡垒机，连接Linux、Windows平台及日志审计。 6)SQL执行及审批流程。 7)慢查询日志分析web界面。

Filebeat介绍及部署

Filebeat介绍

Filebeat附带预构建的模块，这些模块包含收集、解析、充实和可视化各种日志文件格式数据所需的配置，每个Filebeat模块由一个或多个文件集组成，这些文件集包含摄取节点管道、Elasticsearch模板、Filebeat勘探者配置和Kibana仪表盘。

Filebeat模块很好的入门，它是轻量级单用途的日志收集工具，用于在没有安装java的服务器上专门收集日志，可以将日志转发到logstash、elasticsearch或redis等场景中进行下一步处理。

Filebeat和Logstash使用内存对比

Logstash内存占用

代码语言：javascript复制
[root@elkstack03 ~]# ps -ef |  grep -v grep |    grep logstash  | awk '{print $2}'
12628
[root@elkstack03 ~]# cat /proc/12628/status | grep -i vm
VmPeak:  6252788 kB
VmSize:  6189252 kB
VmLck:         0 kB
VmHWM:    661168 kB
VmRSS:    661168 kB
VmData:  6027136 kB
VmStk:        88 kB
VmExe:         4 kB
VmLib:     16648 kB
VmPTE:      1888 kB
VmSwap:        0 kB

Filebeat内存占用

代码语言：javascript复制
[root@test ~]# cat  /proc/12750/status  /proc/12751/status |  grep -i vm 
VmPeak:    11388 kB
VmSize:    11388 kB
VmLck:         0 kB
VmHWM:       232 kB
VmRSS:       232 kB
VmData:    10424 kB
VmStk:        88 kB
VmExe:       864 kB
VmLib:         0 kB
VmPTE:        16 kB
VmSwap:        0 kB


VmPeak:    25124 kB
VmSize:    25124 kB
VmLck:         0 kB
VmHWM:     15144 kB
VmRSS:     15144 kB
VmData:    15496 kB
VmStk:        88 kB
VmExe:      4796 kB
VmLib:         0 kB
VmPTE:        68 kB
VmSwap:        0 kB

Filebeat部署

官方文档:https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-details.html

官网下载地址:https://www.elastic.co/downloads/beats/filebeat

代码语言：javascript复制
#下载Filebeat安装包
[root@elkstack03 ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.3.2-x86_64.rpm
#安装Filebeat
[root@elkstack03 ~]# yum localinstall -y filebeat-5.3.2-x86_64.rpm

Filebeat收集单类型日志到本地文件

配置Filebeat

代码语言：javascript复制
#编辑Filebeat配置文件
[root@elkstack03 ~]# vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /usr/local/nginx/logs/access_json.log
#不收集的行
  exclude_lines: ["^DBG","^$"]
#日志类型
  document_type: ngx_log

output.file:
  path: "/tmp"
  filename: "zls_filebeat.txt"
  
#启动Filebeat(CentOS6)
[root@elkstack03 ~]# /etc/init.d/filebeat start
#启动Filebeat(CentOS7)
[root@elkstack03 ~]# systemctl start filebeat
#检测进程
[root@elkstack03 ~]# ps -ef|grep filebeat
root      10881      1  0 01:06 pts/1    00:00:00 /usr/share/filebeat/bin/filebeat-god -r / -n -p /var/run/filebeat.pid -- /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
root      10882  10881  0 01:06 pts/1    00:00:00 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat



检测本地数据文件

代码语言：javascript复制
#查看本地/tmp目录下内容
[root@elkstack03 ~]# ll /tmp/
总用量 8
-rw------- 1 root  root  143953 4月   9 01:06 zls_filebeat.txt
#查看日志内容
[root@elkstack03 ~]# cat /tmp/zls_filebeat.txt
{"@timestamp":"2019-04-08T17:06:09.591Z","beat":{"hostname":"elkstack03","name":"elkstack03","version":"5.3.2"},"input_type":"log","message":"{"@timestamp":"2019-04-08T10:47:41 08:00","host":"10.0.0.53","clientip":"10.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.53","url":"/index.html","domain":"10.0.0.53","xff":"-","referer":"-","status":"304"}","offset":256,"source":"/usr/local/nginx/logs/access_json.log","type":"ngx_log"}
{"@timestamp":"2019-04-08T17:06:09.591Z","beat":{"hostname":"elkstack03","name":"elkstack03","version":"5.3.2"},"input_type":"log","message":"{"@timestamp":"2019-04-08T10:47:42 08:00","host":"10.0.0.53","clientip":"10.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.53","url":"/index.html","domain":"10.0.0.53","xff":"-","referer":"-","status":"304"}","offset":512,"source":"/usr/local/nginx/logs/access_json.log","type":"ngx_log"}



Filebeat收集单类型多个日志到Logstash

配置Filebeat

代码语言：javascript复制
#编辑Filebeat配置文件
[root@elkstack03 ~]# vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /usr/local/nginx/logs/access_json.log
    - /usr/local/nginx/logs/access.log
  exclude_lines: ["^DBG","^$"]
  document_type: ngx_zls

output.logstash:
#logstash 服务器地址，可以是多个
  hosts: ["10.0.0.53:6666"]
#是否开启输出至logstash，默认即为true
  enabled: true
#工作线程数
  worker: 1
#压缩级别
  compression_level: 3
#多个输出的时候开启负载
#  loadbalance: true

#重启Filebeat
[root@elkstack03 ~]# /etc/init.d/filebeat stop
Stopping filebeat:                                         [确定]
[root@elkstack03 ~]# rm -f /var/lib/filebeat/registry
[root@elkstack03 ~]# /etc/init.d/filebeat start

配置Logstash输出到ES

代码语言：javascript复制
#进入Logstash配置文件目录
[root@elkstack03 ~]# cd /etc/logstash/conf.d/
#编辑Logstash配置文件
[root@elkstack03 conf.d]# vim beats.conf
input {
        beats {
        port => 6666
        codec => "json"
        }
}

output {
    elasticsearch {
      hosts => ["10.0.0.51:9200"]
      index => "%{type}-%{ YYYY.MM.dd}"
    }
}

#启动Logstash
[root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/beats.conf &

验证数据

打开浏览器，访问：http://10.0.0.51:9100/



Filebeat收集单类型多个日志到Redis

配置Filebeat

代码语言：javascript复制
#编辑Filebeat配置文件
[root@elkstack03 ~]# vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /usr/local/nginx/logs/access_json.log
    - /usr/local/nginx/logs/access.log
#不收集的行
  exclude_lines: ["^DBG","^$"]
#日志类型
  document_type: blog.driverzeng.com

output.redis:
  hosts: ["10.0.0.54:6379"]
#Redis中的key名称
  key: "nginx"
#使用1库
  db: 0
#设置超时时间
  timeout: 5
#redis密码
  password: zls

#重启Filebeat
[root@elkstack03 ~]# /etc/init.d/filebeat stop
Stopping filebeat:                                         [确定]
[root@elkstack03 ~]# rm -f /var/lib/filebeat/registry
[root@elkstack03 ~]# /etc/init.d/filebeat start

登录Redis验证数据

代码语言：javascript复制
#登录Redis
[root@elkstack04 ~]# redis-cli -a zls
#查看所有key
127.0.0.1:6379> KEYS *
1) "nginx"
#查看nginx key长度
127.0.0.1:6379> LLEN nginx
(integer) 218
#取出一条日志
127.0.0.1:6379> LPOP nginx
"{"@timestamp":"2019-04-08T17:40:14.675Z","beat":{"hostname":"elkstack03","name":"elkstack03","version":"5.3.2"},"input_type":"log","message":"10.0.0.1 - - [08/Apr/2019:10:29:11  0800] \"GET / HTTP/1.1\" 404 571 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36\"","offset":194,"source":"/usr/local/nginx/logs/access.log","type":"blog.driverzeng.com"}"



Filebeat收集多类型日志到Redis

配置Filebeat

代码语言：javascript复制
#修改Filebeat配置文件
[root@elkstack03 conf.d]# vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /usr/local/nginx/logs/access_json.log
#不收集的行
  exclude_lines: ["^DBG","^$"]
#日志类型
  document_type: ngx_log

- input_type: log
  paths:
    - /usr/local/tomcat/logs/tomcat_access_log.*.log
#不收集的行
  exclude_lines: ["^DBG","^$"]
#日志类型
  document_type: tc_log

output.redis:
  hosts: ["10.0.0.54:6379"]
#Redis中的key名称
  key: "tomcat_nginx"
#使用1库
  db: 1
#设置超时时间
  timeout: 5
#redis密码
  password: zls

#重启Filebeat
[root@elkstack03 ~]# /etc/init.d/filebeat stop
Stopping filebeat:                                         [确定]
[root@elkstack03 ~]# rm -f /var/lib/filebeat/registry
[root@elkstack03 ~]# /etc/init.d/filebeat start

登录Redis验证数据

代码语言：javascript复制
#登录Redis
[root@elkstack04 ~]# redis-cli -a zls
#切换成1库
127.0.0.1:6379> SELECT 1
OK
#查看所有key
127.0.0.1:6379[1]> KEYS *
1) "tomcat_nginx"
#查看key长度
127.0.0.1:6379[1]> LLEN tomcat_nginx
(integer) 7
#取出日志
127.0.0.1:6379[1]> LPOP tomcat_nginx
"{"@timestamp":"2019-04-08T17:50:48.599Z","beat":{"hostname":"elkstack03","name":"elkstack03","version":"5.3.2"},"input_type":"log","message":"{\"@timestamp\":\"2019-04-09T01:50:47 08:00\",\"host\":\"10.0.0.53\",\"clientip\":\"10.0.0.53\",\"size\":0,\"responsetime\":0.000,\"upstreamtime\":\"-\",\"upstreamhost\":\"-\",\"http_host\":\"www.elk.com\",\"url\":\"/index.html\",\"domain\":\"www.elk.com\",\"xff\":\"10.0.0.1\",\"referer\":\"-\",\"status\":\"304\"}","offset":25894,"source":"/usr/local/nginx/logs/access_json.log","type":"ngx_log"}"
127.0.0.1:6379[1]> LPOP tomcat_nginx
"{"@timestamp":"2019-04-08T17:50:48.599Z","beat":{"hostname":"elkstack03","name":"elkstack03","version":"5.3.2"},"input_type":"log","message":"{\"@timestamp\":\"2019-04-09T01:50:47 08:00\",\"host\":\"10.0.0.53\",\"clientip\":\"10.0.0.53\",\"size\":0,\"responsetime\":0.000,\"upstreamtime\":\"-\",\"upstreamhost\":\"-\",\"http_host\":\"www.elk.com\",\"url\":\"/index.html\",\"domain\":\"www.elk.com\",\"xff\":\"10.0.0.1\",\"referer\":\"-\",\"status\":\"304\"}","offset":26162,"source":"/usr/local/nginx/logs/access_json.log","type":"ngx_log"}"
127.0.0.1:6379[1]> LPOP tomcat_nginx
"{"@timestamp":"2019-04-08T17:50:48.599Z","beat":{"hostname":"elkstack03","name":"elkstack03","version":"5.3.2"},"input_type":"log","message":"{\"@timestamp\":\"2019-04-09T01:50:47 08:00\",\"host\":\"10.0.0.53\",\"clientip\":\"10.0.0.53\",\"size\":0,\"responsetime\":0.000,\"upstreamtime\":\"-\",\"upstreamhost\":\"-\",\"http_host\":\"www.elk.com\",\"url\":\"/index.html\",\"domain\":\"www.elk.com\",\"xff\":\"10.0.0.1\",\"referer\":\"-\",\"status\":\"304\"}","offset":26430,"source":"/usr/local/nginx/logs/access_json.log","type":"ngx_log"}"
127.0.0.1:6379[1]> LPOP tomcat_nginx
"{"@timestamp":"2019-04-08T17:50:58.601Z","beat":{"hostname":"elkstack03","name":"elkstack03","version":"5.3.2"},"input_type":"log","message":"{\"clientip\":\"10.0.0.53\",\"ClientUser\":\"-\",\"authenticated\":\"-\",\"AccessTime\":\"[09/Apr/2019:01:50:49  0800]\",\"method\":\"GET / HTTP/1.1\",\"status\":\"304\",\"SendBytes\":\"-\",\"Query?string\":\"\",\"partner\":\"-\",\"AgentVersion\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36\"}","offset":6328,"source":"/usr/local/tomcat/logs/tomcat_access_log.2019-04-09.log","type":"tc_log"}"
127.0.0.1:6379[1]> LPOP tomcat_nginx
"{"@timestamp":"2019-04-08T17:50:58.601Z","beat":{"hostname":"elkstack03","name":"elkstack03","version":"5.3.2"},"input_type":"log","message":"{\"clientip\":\"10.0.0.53\",\"ClientUser\":\"-\",\"authenticated\":\"-\",\"AccessTime\":\"[09/Apr/2019:01:50:49  0800]\",\"method\":\"GET / HTTP/1.1\",\"status\":\"304\",\"SendBytes\":\"-\",\"Query?string\":\"\",\"partner\":\"-\",\"AgentVersion\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36\"}","offset":6661,"source":"/usr/local/tomcat/logs/tomcat_access_log.2019-04-09.log","type":"tc_log"}"

看的出来，tomcat日志和nginx日志都在一个key中



那么有人会问了，都在一个key中，日志不就混乱了么？我们该如何查看日志呢？接下来我们就来解决这个问题。

使用Logstash将beat放入redis的数据输出到ES

代码语言：javascript复制
#进入Logstash配置文件目录
[root@elkstack03 ~]# cd /etc/logstash/conf.d/
#编辑Logstash配置文件
[root@elkstack03 conf.d]# vim beats_redis_es.conf
input {
  redis {
    host => "10.0.0.54"
    port => "6379"
    db => "1"
    key => "tomcat_nginx"
    data_type => "list"
    password => "zls"
    codec  => "json"
 }
}

output {
    elasticsearch {
      hosts => ["10.0.0.51:9200"]
      index => "%{type}-%{ YYYY.MM.dd}"
  }
}
#启动Logstash
[root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/beats_redis_es.conf &

验证数据

打开浏览器，访问：http://10.0.0.51:9100/



可以看到，我们通过Logstash利用type将日志区分开，分别输出到ES中，虽然在Redis中没有区分开，但是最终在ES中区分开了，那么添加到Kibana中，同样是两个日志。

Filebeat收集多类型日志输出到多个目标

配置Filebeat

我们将nginx日志 tomcat日志同时输出到Redis和本地文件中

代码语言：javascript复制
[root@elkstack03 conf.d]# vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /usr/local/nginx/logs/access_json.log
#不收集的行
  exclude_lines: ["^DBG","^$"]
#日志类型
  document_type: ngx_log

- input_type: log
  paths:
    - /usr/local/tomcat/logs/tomcat_access_log.*.log
#不收集的行
  exclude_lines: ["^DBG","^$"]
#日志类型
  document_type: tc_log

output.redis:
#redis 服务器地址，可以是多个
  hosts: ["10.0.0.54:6379"]
  key: "tn"
  db: 2
  timeout: 5
  password: zls

output.file:
  path: "/tmp"
  filename: "zls.txt"
#工作线程数
  worker: 1
#压缩级别
  compression_level: 3
#多个输出的时候开启负载
  loadbalance: true
  
#重启Filebeat
[root@elkstack03 ~]# /etc/init.d/filebeat stop
Stopping filebeat:                                         [确定]
[root@elkstack03 ~]# rm -f /var/lib/filebeat/registry
[root@elkstack03 ~]# /etc/init.d/filebeat start

验证Redis数据和本地文件数据

代码语言：javascript复制
#登录redis
[root@elkstack04 ~]# redis-cli -a zls
#切换2库
127.0.0.1:6379> SELECT 2
OK
#查看所有key
127.0.0.1:6379[2]> KEYS *
1) "tn"
#查看key长度
127.0.0.1:6379[2]> LLEN tn
(integer) 260
#取出日志
127.0.0.1:6379[2]> LPOP tn
"{"@timestamp":"2019-04-08T18:47:12.133Z","beat":{"hostname":"elkstack03","name":"elkstack03","version":"5.3.2"},"input_type":"log","message":"{\"@timestamp\":\"2019-04-08T10:47:41 08:00\",\"host\":\"10.0.0.53\",\"clientip\":\"10.0.0.1\",\"size\":0,\"responsetime\":0.000,\"upstreamtime\":\"-\",\"upstreamhost\":\"-\",\"http_host\":\"10.0.0.53\",\"url\":\"/index.html\",\"domain\":\"10.0.0.53\",\"xff\":\"-\",\"referer\":\"-\",\"status\":\"304\"}","offset":256,"source":"/usr/local/nginx/logs/access_json.log","type":"ngx_log"}"



代码语言：javascript复制
#进入tmp目录
[root@elkstack03 conf.d]# cd /tmp/
#查看文件是否生成
[root@elkstack03 tmp]# ll
总用量 2284
drwxr-xr-x 2 root  root     4096 4月   9 02:20 hsperfdata_root
-rw-r--r-- 1 root  root      638 4月   9 01:35 keyutil_example.com_5881860801300609526.crt
-rw-r--r-- 1 root  root      910 4月   9 01:35 keyutil_example.com_672012091508350129.key
-rw-r--r-- 1 root  root  2010104 4月   9 01:35 libnetty-tcnative-linux-x86_647457201629343237428.so
-rw-r--r-- 1 root  root      702 3月  30 23:05 message2_2019.03.30.log
srwxrwxrwx 1 mysql mysql       0 3月   8 06:28 mysql.sock
-rw-r--r-- 1 root  root     2523 3月  31 01:39 secure_2019.03.30.log
-rw------- 1 root  root   148281 4月   9 01:06 zls_filebeat.txt
-rw------- 1 root  root   154598 4月   9 02:47 zls.txt
#查看文件内容
[root@elkstack03 tmp]# cat zls.txt
{"@timestamp":"2019-04-08T18:47:12.133Z","beat":{"hostname":"elkstack03","name":"elkstack03","version":"5.3.2"},"input_type":"log","message":"{"@timestamp":"2019-04-08T10:47:41 08:00","host":"10.0.0.53","clientip":"10.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.53","url":"/index.html","domain":"10.0.0.53","xff":"-","referer":"-","status":"304"}","offset":256,"source":"/usr/local/nginx/logs/access_json.log","type":"ngx_log"}
{"@timestamp":"2019-04-08T18:47:12.133Z","beat":{"hostname":"elkstack03","name":"elkstack03","version":"5.3.2"},"input_type":"log","message":"{"@timestamp":"2019-04-08T10:47:42 08:00","host":"10.0.0.53","clientip":"10.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.53","url":"/index.html","domain":"10.0.0.53","xff":"-","referer":"-","status":"304"}","offset":512,"source":"/usr/local/nginx/logs/access_json.log","type":"ngx_log"}



运维 es 自动化 nginx http

0 人点赞