环境说明 架构说明及架构图
filebeat部署在客户端用于收集日志并把收集到的日志发送到logstash. logstash把收集到的日志处理之后交给elasticsearch. kibana从elasticsearch中提取数据并进行展示. 之所以使用filebeat进行日志收集是因为filebeat不会像logstash使用大量的资源,影响业务服务器.
环境需求 需要Java环境和redis
yum install java yum install redis使用版本 java 1.8.0_111
redis 2.8.16
filebeat 5.1.2
logstash 5.1.2
elasticsearch 5.1.1
kibana 5.1.1
安装配置 filebeat安装及配置 安装filebeat
rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch cat > /etc/yum.repos.d/elk.repo <<EOF [elasticsearch-5.x] name=Elasticsearch repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF yum clean all yum install filebeat -y
配置filebeat
vim /etc/filebeat/filebeat.yml paths: - /var/log/nginx/access.log tags: ["nginx"] output.logstash: hosts: ["1.8.101.53:5044"]
其中path选项为filebeat发送给logstash的路径, 多个日志可以使用*.log通配. file不会自动递归日志目录下的子目录, 如果需要递归子目录可以使用类似 /var/log/*/*.log 的结构. tags选项会向log中添加一个标签, 此标签可以提供给logstash用于区分不同客户端不同业务的log. output指定发送log到哪台服务器的哪个服务, 默认输出到elasticsearch. 本例使用logstash, 所以需要注释掉发送到elasticsearch的配置, 并启用发送到logstash的配置.
启动filebeat
/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -e &logstash
安装及配置 安装logstash
rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch cat > /etc/yum.repos.d/elk.repo <<EOF [elasticsearch-5.x] name=Elasticsearch repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF yum clean all yum install logstash ln -s /usr/share/logstash/bin/logstash /usr/bin/logstash配置logstash
vim /etc/logstash/conf.d/nginx.conf input { beats { port => 5044 } } filter { if "nginx" in [tags] { grok { match => [ "message","%{COMBINEDAPACHELOG} %{GREEDYDATA:extra_fields}"] overwrite => [ "message"] } mutate { convert => ["response","integer"] convert => ["bytes","integer"] convert => ["responsetime","float"] } geoip { source=>"clientip" target => "geoip" } date { match => [ "timestamp","dd/MMM/YYYY:HH:mm:ss Z"] remove_field => [ "timestamp"] } useragent { source=>"agent" } } } output { if "nginx" == [tags][0] { elasticsearch { hosts => ["1.8.101.53:9200"] index => "access-%{ YYYY.MM.dd}" } } }
启动logstash
logstash --path.settings /etc/logstashelasticsearch安装及配置 安装elasticsearch
yum install elasticsearch启动elasticsearch
service elasticsearch startelasticsearch本身不需要过多配置, 用包管理工具启动即可
kibana安装及配置
安装kibana
yum intall kibana启动kibana
/usr/share/kibana/bin/kibana &
