ELK7日志分析系统基础(二)
版本区别
- ELK6： 默认对外开放访问，需要xpack之类的插件才能开启认证
- ELK7： 默认开启安全验证功能
基本环境需求
- centos7
- 关闭防火墙
- 关闭selinux
- 时间同步
- 磁盘分区
- yum源
curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel ntpdate
ntpdate time.windows.com- java环境安装
- 下载地址： https://www.oracle.com/java/technologies/javase/javase-jdk8-downloads.html
- yum安装： yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel
- java环境的验证： java -version- ELK下载准备
https://www.elastic.co/guide/en/elasticsearch/reference/7.6/index.htmlELK7的基础概念
- Elasticsearch： 搜索数据库服务器，提供restful web接口，简称ES
- Logstash：数据采集和过滤分析以及字段提取
- kibana： 主要是页面展示，ES操作简化等
ElasticSearch集群部署与使用实战
ES数据库单节点部署
官方安装文档参考： https://www.elastic.co/guide/en/elasticsearch/reference/7.6/install-elasticsearch.html
- 基础环境准备好
- elasticsearch rpm下载与安装
[root@centos7-node1 ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.2-x86_64.rpm
[root@centos7-node1 ~]# yum -y localinstall elasticsearch-7.6.2-x86_64.rpm - jvm配置文件修改（默认要填写本机内存的60%）
[root@centos7-node1 ~]# vim /etc/elasticsearch/jvm.options
-Xms200M
-Xmx200M- ES单实例配置与启动(特别要注意)
[root@centos7-node1 ~]# cp /etc/elasticsearch/elasticsearch.yml /usr/local/src/
[root@centos7-node1 ~]# vim /etc/elasticsearch/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.56.11
http.port: 9200
xpack.security.enabled: true
discovery.type: single-node
[root@centos7-node1 ~]# systemctl restart elasticsearch # 服务启动- 访问（需要密码）
- 设置密码(试验环境： elastic)
[root@centos7-node1 ~]# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
Enter password for [elastic]: elastic
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]- 访问
- 验证是否启动成功
[root@centos7-node1 ~]# curl -u elastic:elastic http://192.168.56.11:9200
{
"name" : "centos7-node1",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "WUehKqv3TyudTo_IKMNNlA",
"version" : {
"number" : "7.6.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
"build_date" : "2020-03-26T06:34:37.794943Z",
"build_snapshot" : false,
"lucene_version" : "8.4.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
[root@centos7-node1 ~]# curl -u elastic:elastic http://192.168.56.11:9200/_cat/nodes?v #查看节点信息
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.56.11 56 89 0 0.00 0.01 0.05 dilm * centos7-node1
[root@centos7-node1 ~]# curl -u elastic:elastic http://192.168.56.11:9200/_cat/indices?v #列出索引信息
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .security-7 ByCOlBeLSfStcZVe8Ne0-Q 1 0 6 0 19.6kb 19.6kb- 写入数据
[root@centos7-node1 ~]# curl -u elastic:elastic -X POST http://192.168.56.11:9200/test-index/_doc -H 'Content-Type:application/json' -d '{"name": "test-data1","age": 20}' #插入数据
{"_index":"test-index","_type":"_doc","_id":"1Ip3xnUB0a1lyJIOfMTO","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}- 查看数据
[root@centos7-node1 ~]# curl -u elastic:elastic http://192.168.56.11:9200/test-index/_search?q=* | python -m json.tool #查看数据
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 284 100 284 0 0 2273 0 --:--:-- --:--:-- --:--:-- 2290
{
"_shards": {
"failed": 0,
"skipped": 0,
"successful": 1,
"total": 1
},
"hits": {
"hits": [
{
"_id": "1Ip3xnUB0a1lyJIOfMTO",
"_index": "test-index",
"_score": 1.0,
"_source": {
"age": 20,
"name": "test-data1"
},
"_type": "_doc"
}
],
"max_score": 1.0,
"total": {
"relation": "eq",
"value": 1
}
},
"timed_out": false,
"took": 104
}ES数据库加密集群部署
ES的分布式集群
索引的分片可以把数据分配到不同节点上
每个分片可设置值0个或者多个副本
副本的功能： 备份，提高查询效率，与集群中任何一个节点的通信结果都是一致的
ES分布式集群的部署
- 部署节点信息
主机名 | ip | 软件 |
|---|---|---|
centos7-node1 | 192.168.56.11 | java-1.8.0-openjdk java-1.8.0-openjdk-devel elasticsearch7.6 |
centos7-node2 | 192.168.56.12 | java-1.8.0-openjdk java-1.8.0-openjdk-devel elasticsearch7.6 |
centos7-node3 | 192.168.56.13 | java-1.8.0-openjdk java-1.8.0-openjdk-devel elasticsearch7.6 |
- 软件和基础环境的问题参考第一小节
curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.2-x86_64.rpm
yum -y localinstall elasticsearch-7.6.2-x86_64.rpm- 置(所有节点)
[root@centos7-node1 ~]# vim /etc/elasticsearch/jvm.options
-Xms200M
-Xmx200MES加密集群
- 创建集群交互证书
[root@centos7-node1 ~]# /usr/share/elasticsearch/bin/elasticsearch-certutil ca #直接回车两次即可
[root@centos7-node1 ~]# /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /usr/share/elasticsearch/elastic-stack-ca.p12 #直接回车三次即可- 拷贝证书(拷贝到每台es服务器上)
[root@centos7-node1 ~]# cp /usr/share/elasticsearch/elastic-certificates.p12 /etc/elasticsearch/elastic-certificates.p12
# scp 发送证书到其余两个节点上
[root@centos7-node1 ~]# scp /usr/share/elasticsearch/elastic-certificates.p12 192.168.56.12:/etc/elasticsearch/elastic-certificates.p12
[root@centos7-node1 ~]# scp /usr/share/elasticsearch/elastic-certificates.p12 192.168.56.13:/etc/elasticsearch/elastic-certificates.p12- 修改证书权限(三个节点都需要修改)
[root@centos7-node1 ~]# chown elasticsearch:elasticsearch /etc/elasticsearch/elastic-certificates.p12
[root@centos7-node1 ~]# ssh 192.168.56.12 "chown elasticsearch:elasticsearch /etc/elasticsearch/elastic-certificates.p12"
[root@centos7-node1 ~]# ssh 192.168.56.13 "chown elasticsearch:elasticsearch /etc/elasticsearch/elastic-certificates.p12" - elasticsearch配置
其中node.master用来做数据汇聚
其中node.data是用来做数据存储和查询，压力相对比较大，生产建议将master和data做分离
代码语言：javascript复制###### centos7-node1配置文件
[root@centos7-node1 ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: cropy
node.name: node1
node.master: true
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.56.11
http.port: 9200
discovery.seed_hosts: ["192.168.56.11","192.168.56.12","192.168.56.13"]
cluster.initial_master_nodes: ["192.168.56.11","192.168.56.12","192.168.56.13"]
xpack.security.enabled: true
xpack.monitoring.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12
###### centos7-node2配置文件
[root@centos7-node2 ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: cropy
node.name: node2
node.master: true
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.56.12
http.port: 9200
discovery.seed_hosts: ["192.168.56.11","192.168.56.12","192.168.56.13"]
cluster.initial_master_nodes: ["192.168.56.11","192.168.56.12","192.168.56.13"]
xpack.security.enabled: true
xpack.monitoring.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12
###### centos7-node3配置文件
[root@centos7-node31 ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: cropy
node.name: node3
node.master: true
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.56.13
http.port: 9200
discovery.seed_hosts: ["192.168.56.11","192.168.56.12","192.168.56.13"]
cluster.initial_master_nodes: ["192.168.56.11","192.168.56.12","192.168.56.13"]
xpack.security.enabled: true
xpack.monitoring.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12- 启动服务
[root@centos7-node1 ~]# systemctl enable elasticsearch && systemctl restart elasticsearch
[root@centos7-node1 ~]# tail -f /var/log/elasticsearch/cropy.log #集群日志查看（最后会有Yello状态）
........
[2020-11-14T21:39:53,107][INFO ][o.e.c.r.a.AllocationService] [node1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.security-7][0]]]).
[root@centos7-node2 ~]# systemctl enable elasticsearch && systemctl restart elasticsearch
[root@centos7-node3 ~]# systemctl enable elasticsearch && systemctl restart elasticsearch- 状态查询
systemctl status elasticsearch
ps -ef | grep elasticsearch
netstat -tanlp | grep 9200
[root@centos7-node1 ~]# curl -u elastic:elastic http://192.168.56.11:9200/_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.56.11 60 92 1 0.00 0.01 0.05 dilm * node1
192.168.56.12 45 93 1 0.00 0.01 0.05 dilm - node2
192.168.56.13 60 94 1 0.05 0.03 0.05 dilm - node3
[root@centos7-node1 ~]# curl -u elastic:elastic http://192.168.56.11:9200/_cat/indices?v - 重要事项
ES集群开启xpack的时候只要有一个节点设置密码即可，否则不能访问 /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive #参考命令
ES集群安全交互抓包
- 需要安装软件(集群所有节点)
yum -y install ngrep tcpdump- 测试
[root@centos7-node1 ~]# ngrep -d ens33 port 9200
interface: ens33 (192.168.56.0/255.255.255.0)
filter: ( port 9200 ) and ((ip || ip6) || (vlan && (ip || ip6)))
#
T 192.168.56.1:50907 -> 192.168.56.11:9200 [AF] #1
......
##
T 192.168.56.1:50907 -> 192.168.56.11:9200 [A] #3
...... 
9300端口测试结果
代码语言：javascript复制[root@centos7-node1 ~]# ngrep -d ens33 port 9300
ES数据库的基础操作
ES概念
- 索引： 类似于数据库，索引在写入数据时会自动创建，可按天
- 文档： 类似于表数据，存储在ES里面的数据
ES的基础操作
- curl的方式： 相对比较麻烦
### 写入指定ID数据(1 是id)
[root@centos7-node1 ~]# curl -u elastic:elastic -X PUT http://192.168.56.11:9200/test-index/_doc/1 -H 'Content-Type: application/json' -d '{"name":"zhangsan","age":30}'
### 查询数据
[root@centos7-node1 ~]# curl -u elastic:elastic http://192.168.56.13:9200/test-index/_search?q=*
[root@centos7-node1 ~]# curl -u elastic:elastic http://192.168.56.13:9200/test-index/_doc/1
### 写入随机ID数据（_doc/ 之后不添加id）
[root@centos7-node1 ~]# curl -u elastic:elastic -X POST http://192.168.56.11:9200/test-index/_doc -H 'Content-Type: application/json' -d '{"name":"lisi","age":33}'
### 更新数据（id 为1 的数据更新）
[root@centos7-node1 ~]# curl -u elastic:elastic -X POST http://192.168.56.11:9200/test-index/_update/1 -H 'Content-Type: application/json' -d '{"age":100}'
### 删除数据
[root@centos7-node1 ~]# curl -u elastic:elastic -X DELETE http://192.168.56.11:9200/test-index/_doc/1 #删除单条数据
[root@centos7-node1 ~]# curl -u elastic:elastic -X DELETE http://192.168.56.11:9200/test-index #删除所有数据
### 测试集群数据同步
[root@centos7-node1 ~]# curl -u elastic:elastic -X POST http://192.168.56.12:9200/test-index/_doc -H 'Content-Type: application/json' -d '{"name": "wanghui", "age": 29}'- kibana： 提供简化的操作界面
Kibana的部署与使用实战
Kibana用于做数据展示，es的操作简化
Kibana的安装与配置
- 找其中的一台ES节点做安装
[root@centos7-node1 ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-7.6.2-x86_64.rpm
[root@centos7-node1 ~]# yum -y localinstall kibana-7.6.2-x86_64.rpm - 配置kibana
[root@centos7-node1 ~]# cp /etc/kibana/kibana.yml /usr/local/src/
[root@centos7-node1 ~]# vim /etc/kibana/kibana.yml
server.port: 5601
server.host: "192.168.56.11"
elasticsearch.hosts: ["http://192.168.56.11:9200","http://192.168.56.12:9200","http://192.168.56.13:9200"]
elasticsearch.username: "elastic"
elasticsearch.password: "elastic"
logging.dest: "/tmp/kibaba.log"- 启动kibana
[root@centos7-node1 ~]# systemctl start kibana && systemctl enable kibana
[root@centos7-node1 ~]# tail -f /tmp/kibaba.log #启动日志查看
....
{"type":"log","@timestamp":"2020-11-14T15:07:29Z","tags":["listening","info"],"pid":13907,"message":"Server running at http://192.168.56.11:5601"}- 访问kibana 用户密码都是elastic （之前es部署的时候添加的）
Kibana简化ES数据库的操作
- 简单查询
GET /
GET /_cat/nodes?v
GET /_cat/indices?v- 数据插入
# 插入数据
PUT /nginx-logs/_doc/1
{
"server_name": "www.baidu.com",
"IP": "101.11.203.12"
}
## 查询数据
GET /nginx-logs/_doc/1
GET /nginx-logs/_search?q=*
# 插入随机id数据
POST /nginx-logs/_doc
{
"server_name": "tianyancha.com",
"IP": "180.21.33.41"
}
# 查询所有数据
GET /nginx-logs/_search?q=*
# 更新数据
POST /nginx-logs/_update/1
{
"doc": {
"IP": "111.23.33.23"
}
}
# 修改所有数据
POST /nginx-logs/_update_by_query
{
"script": {
"source": "ctx._source['IP']='111.23.133.123'"
},
"query": {
"match_all": {}
}
}
GET /nginx-logs/_search?q=*
# 增加字段
POST /nginx-logs/_update_by_query
{
"script": {
"source": "ctx._source['city']='beijing'"
},
"query": {
"match_all": {}
}
}
# 删除数据
DELETE /nginx-logs/_doc/1
# 插入数据
POST /nginx-logs1/_doc
{
"server_name": "tianyancha.com",
"IP": "180.21.33.41"
}
POST /nginx-logs2/_doc
{
"server_name": "tianyancha.com",
"IP": "180.21.33.41"
}
# 查询索引
GET /_cat/indices?v
# 删除索引
DELETE /nginx-logs*ElasticSearch模板使用与Python操作
索引的分片及副本的设置
索引的分片以及副本数的设置： 三台ES，最多两个副本，其余的一个要用来存储主数据
代码语言：javascript复制# 设置分片和副本
PUT /nginx-log
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 2
}
}
GET /_cat/indices?v
#获取分片信息
GET /nginx-log/_search_shards
# 插入数据
POST /nginx-logs/_doc
{
"server_name": "tianyancha.com",
"IP": "180.21.33.41"
}
# 查询数据分片所在位置(routing也就是数据的ID)
GET /nginx-log/_search_shards?routing=GUmH0XUBiqEQwQWjL5hD代码语言：javascript复制索引创建完成之后分片不可修改，副本数可以修改
# 修改副本数量
PUT /nginx-log/_settings
{
"number_of_replicas": 1
}索引的模板
代码语言：javascript复制# 获取索引模板
GET /_template
# 简单索引模板创建
PUT _template/test-nginx
{
"index_patterns": ["nginx*"],
"settings": {
"number_of_shards": 2,
"number_of_replicas": 2
}
}
# 插入数据
POST /nginx-logs/_doc
{
"server_name": "www.tianyancha.com",
"IP": "180.21.33.41"
}
POST /nginx-logs/_doc/1
{
"server_name": "shanghai.tianyancha.com",
"IP": "180.21.33.42"
}
POST /nginx-logs/_doc/2
{
"server_name": "shanghai.tianyancha.com",
"IP": "180.21.33.42"
}python操作ES数据库集群
环境准备
- 安装python36
[root@centos7-node2 ~]# yum -y install python36 python36-devel- 升级pip3
[root@centos7-node2 ~]# pip3 install --upgrade pip -i https://mirrors.aliyun.com/pypi/simple- 安装es7.6 的python库
[root@centos7-node2 ~]# pip3 install elasticsearch==7.6.0 -i https://mirrors.aliyun.com/pypi/simple - 检查ES模块是否安装成功
[root@centos7-node2 ~]# python3
Python 3.6.8 (default, Apr 2 2020, 13:34:55)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import elasticsearchpython操作ES集群
- 添加数据
[root@centos7-node2 ~]# vim add_data.py
#!/use/bin/python3
from elasticsearch import Elasticsearch
es = Elasticsearch(['http://elastic:elastic@192.168.56.11:9200','http://elastic:elastic@192.168.56.12:9200','http://elastic:elastic@192.168.56.13
:9200'])
body = {"server_name":"https://baidu.com","IP":"120.99.220.23"}
es.index(index='nginx-logs',body=body)
print("insert data success!!")
- 查询数据
[root@centos7-node2 ~]# cat search_data.py
#!/use/bin/python3
from elasticsearch import Elasticsearch
es = Elasticsearch(['http://elastic:elastic@192.168.56.11:9200','http://elastic:elastic@192.168.56.12:9200','http://elastic:elastic@192.168.56.13:9200'])
print(es.search(index='nginx-logs'))
[root@centos7-node2 ~]# python3 search_data.py #查询数据
{'took': 2, 'timed_out': False, '_shards': {'total': 1, 'successful': 1, 'skipped': 0, 'failed': 0}, 'hits': {'total': {'value': 4, 'relation': 'eq'}, 'max_score': 1.0, 'hits': [{'_index': 'nginx-logs', '_type': '_doc', '_id': 'GUmH0XUBiqEQwQWjL5hD', '_score': 1.0, '_source': {'server_name': 'tianyancha.com', 'IP': '180.21.33.41'}}, {'_index': 'nginx-logs', '_type': '_doc', '_id': 'GkmQ0XUBiqEQwQWjDpiD', '_score': 1.0, '_source': {'server_name': 'www.tianyancha.com', 'IP': '180.21.33.41'}}, {'_index': 'nginx-logs', '_type': '_doc', '_id': '1', '_score': 1.0, '_source': {'server_name': 'shanghai.tianyancha.com', 'IP': '180.21.33.42'}}, {'_index': 'nginx-logs', '_type': '_doc', '_id': 'BA-e0XUB4OFcF5NOHHFn', '_score': 1.0, '_source': {'server_name': 'https://baidu.com', 'IP': '120.99.220.23'}}]}}- 删除索引
[root@centos7-node2 ~]# vim delete_index.py
#!/use/bin/python3
from elasticsearch import Elasticsearch
es = Elasticsearch(['http://elastic:elastic@192.168.56.11:9200','http://elastic:elastic@192.168.56.12:9200','http://elastic:elastic@192.168.56.13
:9200'])
print(es.indices.delete(index='nginx-logs'))- 循环添加数据
#!/use/bin/python3
from elasticsearch import Elasticsearch
import time
es = Elasticsearch(['http://elastic:elastic@192.168.56.11:9200','http://elastic:elastic@192.168.56.12:9200','http://elastic:elastic@192.168.56.13
:9200'])
for i in range(1,10000):
body = {"server_name":"https://{0}.baidu.com".format(i),"IP":"120.99.220.23","count":i}
es.index('nginx-logs',body=body)
time.sleep(0.1)
print("insert {0}".format(i))LogStash实用技能实战
Logstash的安装和简单使用
官方文档： https://www.elastic.co/guide/en/logstash/7.6/index.html
Logstash的功能
- 对日志进行过滤处理
- 也能用于日志收集（一般不这么用）
- 输入支持： 标准输入，文本日志输入等
- 输出支持： 标准输出，ES输出等
Logstash的安装部署
- 节点： 192.168.56.14 centos7-node4
- 基本环境
- centos7
- 时间同步
- yum源
curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel ntpdate
ntpdate time.windows.com- 安装logstash
参考： https://www.elastic.co/guide/en/logstash/7.6/installing-logstash.html
代码语言：javascript复制[root@centos7-node4 ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-7.6.2.rpm
[root@centos7-node4 ~]# yum localinstall logstash-7.6.2.rpm -y- 配置logstash JVM参数
[root@centos7-node4 ~]# vim /etc/logstash/jvm.options
-Xms200M
-Xmx200M- 配置logstash
[root@centos7-node4 ~]# cp /etc/logstash/logstash-sample.conf /etc/logstash/conf.d/logstash.conf
[root@centos7-node4 ~]# vim /etc/logstash/conf.d/logstash.conf
input {
stdin {}
}
output {
stdout {
codec=>rubydebug
}
}- logstash的启动和测试
[root@centos7-node4 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf
....
INFO ] 2019-11-09 05:17:20.482 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
logstash
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"@version" => "1",
"message" => "logstash",
"host" => "centos7-node4",
"@timestamp" => 2019-11-08T21:17:41.189Z
}
hello word
{
"@version" => "1",
"message" => "hello word",
"host" => "centos7-node4",
"@timestamp" => 2019-11-08T21:17:47.686Z
....Logstash读取日志文件实战
- 环境准备： 安装ngixn
[root@centos7-node4 ~]# yum -y install nginx
[root@centos7-node4 ~]# vim /usr/lib/systemd/system/nginx.service 修改nginx启动文件
[Unit]
Description=The nginx HTTP and reverse proxy server
After=network.target remote-fs.target nss-lookup.target
[Service]
Type=forking
PIDFile=/run/nginx.pid
# Nginx will fail to start if /run/nginx.pid already exists but has the wrong
# SELinux context. This might happen when running `nginx -t` from the cmdline.
# https://bugzilla.redhat.com/show_bug.cgi?id=1268621
ExecStartPre=/usr/bin/rm -f /run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
[Install]
WantedBy=multi-user.target
[root@centos7-node4 ~]# systemctl start nginx && systemctl enable nginx #服务启动
[root@centos7-node4 ~]# curl localhost
[root@centos7-node4 ~]# tail /var/log/nginx/access.log - logstash配置
[root@centos7-node4 ~]# vim /etc/logstash/conf.d/logstash.conf
input {
file {
path => "/var/log/nginx/access.log"
}
}
output {
stdout {
codec=>rubydebug
}
}- 测试logstash
[root@centos7-node4 ~]# systemctl restart logstash
[root@centos7-node4 ~]# tail -f /var/log/logstash/logstash-plain.log
[2019-11-09T05:29:03,147][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.6.2"}
[2019-11-09T05:29:03,205][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"93219590-7b15-41b6-b262-c2c4077278f9", :path=>"/var/lib/logstash/uuid"}
[2019-11-09T05:29:05,203][INFO ][org.reflections.Reflections] Reflections took 54 ms to scan 1 urls, producing 20 keys and 40 values
[2019-11-09T05:29:06,349][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.RubyArray) has been created for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2019-11-09T05:29:06,375][INFO ][logstash.javapipeline 此时logstash不能正常采集日志，因为对nginx日志权限存在问题，所以要改一下nginx日志权限
代码语言：javascript复制[root@centos7-node4 ~]# chmod 775 -R /var/log/nginx/- 查看message日志(一定要有新日志产生)
[root@centos7-node4 ~]# tail -f /var/log/messagesLogstash读取日志内容输出到ES
说明：
logstash支持读取日志发送到ES 但是Logstash用来收集日志比较重，后面将对此做优化
操作实战
- 清理原先ES集群数据
- 配置logstash配置发送日志到ES的配置
[root@centos7-node4 ~]# vim /etc/logstash/conf.d/logstash.conf
elasticsearch {
input {
file {
path => "/var/log/nginx/access.log"
}
}
output {
elasticsearch {
hosts => ["http://192.168.56.11:9200","192.168.56.12:9200","192.168.56.13:9200"]
user => "elastic"
password => "elastic"
index => "nginx-logs-%{ YYYY-MM-dd}"
}
}- 重启logstash
[root@centos7-node4 ~]# systemctl restart logstash
[root@centos7-node4 ~]# ps -ef | grep logstash- 多次访问http://192.168.56.14 (这是我的nginx部署所在)
- kibana查看index是否存在
- 加入kibana
- 数据写入查看
Logstash正则提取Nginx日志
为什么要提取nginx日志？
- 使用一整行日志无法分析，需要提取单独的字段
- 分析哪个IP的访问量最大
- 分析nginx的相应状态码
nginx默认日志的格式与配置
- 日志格式
192.168.56.1 - - [09/Nov/2019:05:24:08 0800] "GET / HTTP/1.1" 200 4833 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36" "-"- 日志格式配置
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';Grok日志提取神器
需要掌握正则表达式，借助kibana的grok工具验证提取
- 自写正则提取
- 内置规则提取（提取简化）
[root@centos7-node4 ~]# cat /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns - 普通正则表达式符号
.表示任意一个字符*表示前面一个字符出现0次或者多次[abc]表示中括号内的任意一个字符[^abc]表示非中括号内的字符[0-9]表示数字[a-z]表示小写字母[A-Z]表示大写字母[a-zA-Z]表示所有字母[a-zA-Z0-9]表示所有字母 数字[^0-9]表示非数字^xxx表示以xxx开头xxx$表示以xxx结尾s表示空白字符S表示非空白字符d表示数字
1.1 kibana日志提取操作
- 拓展正则表达式，在普通正则表达式的基础上再进行拓展
%{IP:remote_addr} - (%{WORD:remote_user}|-) [%{HTTPDATE:time_local}] "%{WORD:method} %{NOTSPACE:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:body_byte_sent} %{QS} %{QS:http_user_agent}
logstash grok将日志写入ES
- 配置logstash提取日志配置
[root@centos7-node4 ~]# vim /etc/logstash/conf.d/logstash.conf
input {
file {
path => "/var/log/nginx/access.log"
}
}
filter {
grok {
match => {
"message" => '%{IP:remote_addr} - (%{WORD:remote_user}|-) [%{HTTPDATE:time_local}] "%{WORD:method} %{NOTSPACE:request} HTTP/%{NUMBER}" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS} %{QS:http_user_agent}'
}
remove_field => ["message"]
}
}
output {
elasticsearch {
hosts => ["http://192.168.56.11:9200","192.168.56.12:9200","192.168.56.13:9200"]
user => "elastic"
password => "elastic"
index => "nginx-logs-%{ YYYY-MM-dd}"
}
}- 重启logstash
[root@centos7-node4 ~]# systemctl restart logstash
[root@centos7-node4 ~]# tail -f /var/log/logstash/logstash-plain.log
[root@centos7-node4 ~]# ps -ef | grep logstash- 访问nginx(192.168.56.14)
Nginx模拟用户访问
while true;do
curl 192.168.56.14/wanghui666
curl 127.0.0.1
sleep 2
done- 查看kibana
kibana显示感叹号问题的处理
出现感叹号的原因就是重新加入分词，日志字段出现多个的场景
- kibana索引刷新
- Kibana索引的操作并不会影响到数据，删除重建也没问题
- 查看索引
Logstash特殊字段处理与替换
- 去除字段中的引号
mutate {
gsub => ["http_user_agent",'"',""]
}- 数字类型的字符串转换成整型
mutate {
convert => { "status" => "integer" }
convert => { "body_bytes_sent" => "integer" }
}- 完整logstash如下：
[root@centos7-node4 ~]# vim /etc/logstash/conf.d/logstash.conf
input {
file {
path => "/var/log/nginx/access.log"
}
}
filter {
grok {
match => {
"message" => '%{IP:remote_addr} - (%{WORD:remote_user}|-) [%{HTTPDATE:time_local}] "%{WORD:method} %{NOTSPACE:request} HTTP/%{
NUMBER}" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS} %{QS:http_user_agent}'
}
remove_field => ["message"]
}
mutate {
gsub => ["http_user_agent",'"',""]
convert => { "status" => "integer" }
convert => { "body_bytes_sent" => "integer" }
}
}
output {
elasticsearch {
hosts => ["http://192.168.56.11:9200","192.168.56.12:9200","192.168.56.13:9200"]
user => "elastic"
password => "elastic"
index => "nginx-logs-%{ YYYY-MM-dd}"
}
}- 重启logstash
[root@centos7-node4 ~]# kill -1 $(ps -ef | grep logstash | grep -v grep | awk '{print $2}')- 访问nginx
- 查看kibana结果
Logstash替换时间戳timestamp
存在的问题：
处理过程如下：
- 删除index
- 修改logstash
[root@centos7-node4 ~]# vim /etc/logstash/conf.d/logstash.conf
input {
file {
path => "/var/log/nginx/access.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
grok {
match => {
"message" => '%{IP:remote_addr} - (%{WORD:remote_user}|-) [%{HTTPDATE:time_local}] "%{WORD:method} %{NOTSPACE:request} HTTP/%{
NUMBER}" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS} %{QS:http_user_agent}'
}
remove_field => ["message"]
}
mutate {
gsub => ["http_user_agent",'"',""]
convert => { "status" => "integer" }
convert => { "body_bytes_sent" => "integer" }
}
}
output {
elasticsearch {
hosts => ["http://192.168.56.11:9200","192.168.56.12:9200","192.168.56.13:9200"]
user => "elastic"
password => "elastic"
index => "nginx-logs-%{ YYYY-MM-dd}"
}
}- 重启logstash
[root@centos7-node4 ~]# kill -1 $(ps -ef | grep logstash | grep -v grep | awk '{print $2}')
[root@centos7-node4 ~]# tail -f /var/log/messages- 问题浮现
使用nginx日志中的访问日期覆盖kibana上的时间
- logstash配置如下
[root@centos7-node4 ~]# vim /etc/logstash/conf.d/logstash.conf
input {
file {
path => "/var/log/nginx/access.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
grok {
match => {
"message" => '%{IP:remote_addr} - (%{WORD:remote_user}|-) [%{HTTPDATE:time_local}] "%{WORD:method} %{NOTSPACE:request} HTTP/%{
NUMBER}" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS} %{QS:http_user_agent}'
}
remove_field => ["message"]
}
date {
match => ["time_local", "dd/MMM/yyyy:HH:mm:ss Z"]
target => "@timestamp"
}
mutate {
gsub => ["http_user_agent",'"',""]
convert => { "status" => "integer" }
convert => { "body_bytes_sent" => "integer" }
remove_field => ["time_local"]
}
}
output {
elasticsearch {
hosts => ["http://192.168.56.11:9200","192.168.56.12:9200","192.168.56.13:9200"]
user => "elastic"
password => "elastic"
index => "nginx-logs-%{ YYYY-MM-dd}"
}
}- 删除日志索引
- 重启logstash
[root@centos7-node4 ~]# kill -1 $(ps -ef | grep logstash | grep -v grep | awk '{print $2}')
[root@centos7-node4 ~]# tail -f /var/log/messages- kibana展示查看
注意
日志里如果有不同的时间格式，覆盖的时候格式要对应 20/Feb/2019:14:50:06 -> dd/MMM/yyyy:HH:mm:ss 2016-08-24 18:05:39,830 -> yyyy-MM-dd HH:mm:ss,SSS
Logstash正则提取与异常处理
- 在nginx日志中加入异常字段
[root@centos7-node4 ~]# echo "sdadsadsdsadsfac sadf fwekfwekwfadfaofcr" >> /var/log/nginx/access.log
[root@centos7-node4 ~]# echo "sdadsadsdsadsfac sadf fwekfwekwfadfaofcr" >> /var/log/nginx/access.log
[root@centos7-node4 ~]# echo "sdadsadsdsadsfac sadf fwekfwekwfadfaofcr" >> /var/log/nginx/access.log
[root@centos7-node4 ~]# echo "sdadsadsdsadsfac sadf fwekfwekwfadfaofcr" >> /var/log/nginx/access.log
[root@centos7-node4 ~]# echo "sdadsadsdsadsfac sadf fwekfwekwfadfaofcr" >> /var/log/nginx/access.log
[root@centos7-node4 ~]# echo "sdadsadsdsadsfac sadf fwekfwekwfadfaofcr" >> /var/log/nginx/access.log
[root@centos7-node4 ~]# echo "sdadsadsdsadsfac sadf fwekfwekwfadfaofcr" >> /var/log/nginx/access.log- kibana查看到异常的日志
- 将异常数据发送到另外的index
[root@centos7-node4 ~]# vim /etc/logstash/conf.d/logstash.conf
input {
file {
path => "/var/log/nginx/access.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
hosts => ["http://192.168.238.90:9200", "http://192.168.238.92:9200"]
hosts => ["http://192.168.238.90:9200", "http://192.168.238.92:9200"]
output {
elasticsearch {
hosts => ["http://192.168.56.11:9200","192.168.56.12:9200","192.168.56.13:9200"]
user => "elastic"
password => "elastic"
index => "nginx-logs-%{ YYYY-MM-dd}"
}
}
input {
file {
path => "/var/log/nginx/access.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
grok {
match => {
"message" => '%{IP:remote_addr} - (%{WORD:remote_user}|-) [%{HTTPDATE:time_local}] "%{WORD:method} %{NOTSPACE:request} HTTP/%{
NUMBER}" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS} %{QS:http_user_agent}'
}
remove_field => ["message"]
}
date {
match => ["time_local", "dd/MMM/yyyy:HH:mm:ss Z"]
target => "@timestamp"
}
mutate {
gsub => ["http_user_agent",'"',""]
convert => { "status" => "integer" }
convert => { "body_bytes_sent" => "integer" }
remove_field => ["time_local"]
}
}
output {
if "_grokparsefailure" not in [tags] and "_dateparsefailure" not in [tags] {
elasticsearch {
hosts => ["http://192.168.56.11:9200","192.168.56.12:9200","192.168.56.13:9200"]
user => "elastic"
password => "elastic"
index => "nginx-logs-%{ YYYY-MM-dd}"
}
}
else{
elasticsearch {
hosts => ["http://192.168.56.11:9200","192.168.56.12:9200","192.168.56.13:9200"]
user => "elastic"
password => "elastic"
index => "nginx-err-logs-%{ YYYY-MM-dd}"
}
}
}- 重启logstash
[root@centos7-node4 ~]# kill -1 $(ps -ef | grep logstash | grep -v grep | awk '{print $2}')
[root@centos7-node4 ~]# tail -f /var/log/messages - 再次写入异常数据
[root@centos7-node4 ~]# echo "sdadsadsdsadsfac sadf fwekfwekwfadfaofcr" >> /var/log/nginx/access.log
[root@centos7-node4 ~]# echo "sdadsadsdsadsfac sadf fwekfwekwfadfaofcr" >> /var/log/nginx/access.log
[root@centos7-node4 ~]# echo "sdadsadsdsadsfac sadf fwekfwekwfadfaofcr" >> /var/log/nginx/access.log
[root@centos7-node4 ~]# echo "sdadsadsdsadsfac sadf fwekfwekwfadfaofcr" >> /var/log/nginx/access.log- 新增kibana 数据
Kibana图形使用简介
- 模拟数据(nginx 所在机器)
[root@centos7-node4 ~]# while true;do curl 192.168.56.14/cropy666; curl 127.0.0.1; sleep 2; done- 首页区域 可以根据时间查看访问量：每分钟访问量 可以根据某个字段查询 可以单独看某个字段的统计
- Kibana图形有建立，选择terms去查看对应的数据 饼图的创建 pie_remote_addr
表的创建 table_remote_addr
- Kibana面板的创建cropy_dash 创建面板 在面板上添加图形
- 建议采用Grafana展示
Logstash分析Linux系统日志
- 系统日志
[root@centos7-node4 ~]# cat /var/log/secure
Nov 21 20:47:54 centos7-node4 polkitd[712]: Registered Authentication Agent for unix-process:2314:2063304 (system bus name :1.177 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Nov 21 20:47:56 centos7-node4 polkitd[712]: Unregistered Authentication Agent for unix-process:2314:2063304 (system bus name :1.177, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)- 修改系统日志让支持年份
[root@centos7-node4 ~]# vim /etc/rsyslog.conf
# Use default timestamp format
# $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$template tycformat,"%$NOW% %TIMESTAMP:8:15% %hostname% %syslogtag% %msg%n"
$ActionFileDefaultTemplate tycformat
[root@centos7-node4 ~]# systemctl restart rsyslog- 验证是否生效
[root@centos7-node4 ~]# tail -f /var/log/secure
Nov 21 20:47:54 centos7-node4 polkitd[712]: Registered Authentication Agent for unix-process:2314:2063304 (system bus name :1.177 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Nov 21 20:47:56 centos7-node4 polkitd[712]: Unregistered Authentication Agent for unix-process:2314:2063304 (system bus name :1.177, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Nov 22 00:05:36 centos7-node4 polkitd[712]: Registered Authentication Agent for unix-process:5177:3249450 (system bus name :1.308 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
2020-11-22 00:05:36 centos7-node4 polkitd[712]: Unregistered Authentication Agent for unix-process:5177:3249450 (system bus name :1.308, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
2020-11-22 00:06:23 centos7-node4 sshd[5191]: Accepted password for root from 192.168.56.1 port 57635 ssh2
2020-11-22 00:06:23 centos7-node4 sshd[5191]: pam_unix(sshd:session): session opened for user root by (uid=0)- kibana grok debug 解决日志格式问题
2020-11-22 00:06:23 centos7-node4 sshd[5191]: Accepted password for root from 192.168.56.1 port 57635 ssh2
%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE} %{NOTSPACE:procinfo}: (?<secinfo>.*)
- logstash采集系统日志(message 日志也类似)
[root@centos7-node4 ~]# chmod r /var/log/secure
[root@centos7-node4 ~]# vim /etc/logstash/conf.d/logstash-sys.conf
input {
file {
path => "/var/log/secure"
}
}
filter {
grok {
match => {
"message" => '%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE} %{NOTSPACE:procinfo}: (?<secinfo>.*)'
}
remove_field => ["message"]
}
date {
match => ["timestamp", "yyyy-MM-dd HH:mm:ss"]
target => "@timestamp"
}
mutate {
remove_field => ["timestamp"]
}
}
output {
elasticsearch {
hosts => ["http://192.168.56.11:9200","192.168.56.12:9200","192.168.56.13:9200"]
user => "elastic"
password => "elastic"
index => "system-secure-%{ YYYY.MM.dd}"
}
}- 重启logstash
[root@centos7-node4 ~]# systemctl restart logstash
[root@centos7-node4 ~]# tail -f /var/log/messages- kibana加入index
