logstash收集日志到elasticsearch

2021-08-05 10:46:50 浏览数 (1)

1.logstash下载安装

(图片来自:https://www.elastic.co/cn/downloads/logstash)

(图片来自:https://www.elastic.co/guide/en/logstash/current/configuration.html)

其余配置文件的说明,如下:

(图片来自:https://www.elastic.co/guide/en/logstash/current/config-setting-files.html)

配置收集tomcat日志,如下:

代码语言:javascript复制
input { 
beats {
    port => "5044"
}
file {
      path => ".../localhost_access_log*.log"
      type => "tomcat-access-log-ceshi"
      start_position => "beginning"
      stat_interval => "2"
    }
}
output {
  elasticsearch { 
  hosts => ["***.***.***.***:9200"] 
  index => "logstash-tomcat-access-log-ceshi-%{ YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

详细:ELK 架构之 Logstash 和 Filebeat 安装配置

2.filebeat连接logstash使用elasticsearch记日志的过程

(ME:filebeat和logstash有什么区别和联系?)

代码语言:javascript复制
[root@node1 ~]# vi /etc/logstash/conf.d/logstash-filebeat-syslog.conf
input {
 beats {
   port => 10515
  }
}
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
output {
 elasticsearch {
  hosts => [ "node1:9200" ]
    index => "%{[@metadata][beat]}-%{ YYYY.MM.dd}"
  }
}

(代码来自:https://www.cnblogs.com/xishuai/p/elk-logstash-filebeat.html)

(ME:这里的grok是什么用法?)

(ME:为什么filebeat不把日志直接存储到elasticsearch,而是发给logstash服务器?)

这里对过滤logstash的过滤的职责描述的更清晰一点

Reference

https://blog.csdn.net/z1017915048/article/details/102971692

https://www.cnblogs.com/Applogize/p/13545754.html

https://www.cnblogs.com/yanshicheng/articles/9431335.html

0 人点赞