Kubernetes K8S之Helm部署ELK日志分析系统；由于Logstash比较消耗资源，因此本次我们使用Fluentd实现日志收集（EFK）。
主机配置规划
服务器名称(hostname) | 系统版本 | 配置 | 内网IP | 外网IP(模拟) |
|---|---|---|---|---|
k8s-master | CentOS7.7 | 2C/4G/20G | 172.16.1.110 | 10.0.0.110 |
k8s-node01 | CentOS7.7 | 2C/4G/20G | 172.16.1.111 | 10.0.0.111 |
k8s-node02 | CentOS7.7 | 2C/4G/20G | 172.16.1.112 | 10.0.0.112 |
备注：由于EFK部署比较消耗内存；所以每台机器的内存最好大于等于4G。
ELK概述
ELK是Elasticsearch、Logstash、Kibana的简称，这三者是核心套件，但并非全部。
Elasticsearch是实时全文搜索和分析引擎，提供搜集、分析、存储数据三大功能；是一套开放REST和JAVA API等接口，提供高效搜索功能，可扩展的分布式系统。它构建于Apache Lucene搜索引擎库之上。
Logstash是一个用来搜集、分析、过滤日志的工具。它支持几乎任何类型的日志，包括系统日志、错误日志和自定义应用程序日志。它可以从许多来源接收日志，这些来源包括 syslog、消息传递（例如 RabbitMQ）和JMX，它能够以多种方式输出数据，包括电子邮件、websockets和Elasticsearch。
Kibana是一个基于Web的图形界面，用于搜索、分析和可视化存储在 Elasticsearch指标中的日志数据。它利用Elasticsearch的REST接口来检索数据，不仅允许用户创建他们自己数据定制仪表板的视图，还允许他们以特殊的方式查询和过滤数据。
由于Logstash比较消耗资源，因此本次我们使用Fluentd实现日志收集（EFK）。
EFK镜像下载
由于镜像都在国外，因此我们在国内下载镜像，然后tag为对应的镜像名称。执行如下脚本【集群所有机器都执行】：
代码语言：javascript复制 1 [root@k8s-node02 software]# vim download_efk_image.sh
2 #!/bin/sh
3
4 ##### 在 master 节点和 worker 节点都要执行 【所有机器执行】
5
6 # 加载环境变量
7 . /etc/profile
8 . /etc/bashrc
9
10 # 变量设置
11 elasticsearch_iamge="elasticsearch-oss:6.7.0"
12 busybox_image="busybox:latest"
13 bats_image="bats:0.4.0"
14 fluentd_image="fluentd-elasticsearch:v2.3.2"
15 kibana_image="kibana-oss:6.7.0"
16
17 # 集群所有机器执行
18 # elasticsearch镜像下载
19 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge}
20 docker tag registry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge} docker.elastic.co/elasticsearch/${elasticsearch_iamge}
21 docker rmi registry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge}
22 # busybox镜像下载
23 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${busybox_image}
24 docker tag registry.cn-beijing.aliyuncs.com/google_registry/${busybox_image} ${busybox_image}
25 docker rmi registry.cn-beijing.aliyuncs.com/google_registry/${busybox_image}
26 # bats镜像下载
27 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${bats_image}
28 docker tag registry.cn-beijing.aliyuncs.com/google_registry/${bats_image} dduportal/${bats_image}
29 docker rmi registry.cn-beijing.aliyuncs.com/google_registry/${bats_image}
30 # fluentd-elasticsearch镜像下载
31 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image}
32 docker tag registry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image} gcr.io/google-containers/${fluentd_image}
33 docker rmi registry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image}
34 # kibana-oss镜像下载
35 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${kibana_image}
36 docker tag registry.cn-beijing.aliyuncs.com/google_registry/${kibana_image} docker.elastic.co/kibana/${kibana_image}
37 docker rmi registry.cn-beijing.aliyuncs.com/google_registry/${kibana_image}Elasticsearch部署
本次部署EFK，创建一个efk名称空间。
chart下载与配置修改
代码语言：javascript复制 1 # 当前目录
2 [root@k8s-master efk]# pwd
3 /root/k8s_practice/efk
4 # 创建 efk 名称空间
5 [root@k8s-master efk]# kubectl create namespace efk
6 [root@k8s-master efk]#
7 # ES版本查看，本次我们部署chart 1.30.0版本，ES 6.7.0版本
8 [root@k8s-master efk]# helm search stable/elasticsearch -l
9 NAME CHART VERSION APP VERSION DESCRIPTION
10 stable/elasticsearch 1.32.5 6.8.6 DEPRECATED Flexible and powerful open source, distributed...
11 stable/elasticsearch 1.32.4 6.8.6 Flexible and powerful open source, distributed real-time ...
12 stable/elasticsearch 1.32.3 6.8.6 Flexible and powerful open source, distributed real-time ...
13 ………………
14 [root@k8s-master efk]#
15 [root@k8s-master efk]# helm fetch stable/elasticsearch --version 1.30.0
16 [root@k8s-master efk]# tar xf elasticsearch-1.30.0.tgz
17 # 修改配置文件1
18 [root@k8s-master efk]# vim elasticsearch/values.yaml
19 initImage:
20 repository: "busybox"
21 tag: "latest"
22 pullPolicy: "IfNotPresent" # 从Always 改为IfNotPresent
23 ………………
24 client:
25 name: client
26 replicas: 1 # 从2改为1，因为是在自己PC机操作的，内存有限
27 serviceType: ClusterIP
28 ………………
29 master:
30 name: master
31 exposeHttp: false
32 replicas: 3 # 不要修改
33 heapSize: "512m"
34 persistence:
35 enabled: false # 没有多余的PVC，因此从true改为false
36 accessMode: ReadWriteOnce
37 ………………
38 data:
39 name: data
40 exposeHttp: false
41 replicas: 1 # 从2改为1，因为是在自己PC机操作的，内存有限
42 heapSize: "1024m" # 从1536m改为1024m，因为是在自己PC机操作的，内存有限
43 persistence:
44 enabled: false # 没有多余的PVC，因此从true改为false
45 accessMode: ReadWriteOnce
46 [root@k8s-master efk]#
47 # 修改配置文件2
48 [root@k8s-master efk]# vim elasticsearch/templates/client-deployment.yaml
49 apiVersion: apps/v1 # 从 apps/v1beta1 改为 apps/v1
50 kind: Deployment
51 ………………
52 spec:
53 replicas: {{ .Values.client.replicas }}
54 # 添加信息 如下
55 selector:
56 matchLabels:
57 app: {{ template "elasticsearch.name" . }}
58 component: "{{ .Values.client.name }}"
59 release: {{ .Release.Name }}
60 # 添加信息 如上
61 [root@k8s-master efk]#
62 # 修改配置文件3
63 [root@k8s-master efk]# vim elasticsearch/templates/data-statefulset.yaml
64 apiVersion: apps/v1 # 从 apps/v1beta1 改为 apps/v1
65 kind: StatefulSet
66 ………………
67 spec:
68 serviceName: {{ template "elasticsearch.data.fullname" . }}
69 replicas: {{ .Values.data.replicas }}
70 # 添加信息 如下
71 selector:
72 matchLabels:
73 app: {{ template "elasticsearch.name" . }}
74 component: "{{ .Values.data.name }}"
75 release: {{ .Release.Name }}
76 role: data
77 # 添加信息 如上
78 [root@k8s-master efk]#
79 # 修改配置文件4
80 [root@k8s-master efk]# vim elasticsearch/templates/master-statefulset.yaml
81 apiVersion: apps/v1 # 从 apps/v1beta1 改为 apps/v1
82 kind: StatefulSet
83 ………………
84 spec:
85 serviceName: {{ template "elasticsearch.master.fullname" . }}
86 replicas: {{ .Values.master.replicas }}
87 # 添加信息 如下
88 selector:
89 matchLabels:
90 app: {{ template "elasticsearch.name" . }}
91 component: "{{ .Values.master.name }}"
92 release: {{ .Release.Name }}
93 role: master
94 # 添加信息 如上
95 [root@k8s-master efk]#Elasticsearch部署
步骤如下：
代码语言：javascript复制 1 # 当前目录
2 [root@k8s-master efk]# pwd
3 /root/k8s_practice/efk
4 # 部署ES
5 [root@k8s-master efk]# helm install --name es01 --namespace=efk elasticsearch/
6 [root@k8s-master efk]#
7 # 状态查看
8 [root@k8s-master ~]# helm list
9 NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
10 es01 1 Sat Jul 25 12:18:50 2020 DEPLOYED elasticsearch-1.30.0 6.7.0 efk
11 [root@k8s-master efk]#
12 # 等待一会儿后【估计几分钟】，查看pod状态信息如下
13 [root@k8s-master ~]# kubectl get deploy -n efk
14 NAME READY UP-TO-DATE AVAILABLE AGE
15 es01-elasticsearch-client 1/1 1 1 6m13s
16 [root@k8s-master ~]#
17 [root@k8s-master ~]# kubectl get pod -n efk
18 NAME READY STATUS RESTARTS AGE
19 es01-elasticsearch-client-646f8f866d-rt2wp 1/1 Running 0 6m21s
20 es01-elasticsearch-data-0 1/1 Running 0 6m21s
21 es01-elasticsearch-master-0 1/1 Running 0 6m21s
22 es01-elasticsearch-master-1 1/1 Running 0 5m30s
23 es01-elasticsearch-master-2 1/1 Running 0 5m3s
24 [root@k8s-master efk]#
25 [root@k8s-master efk]# kubectl get svc -n efk
26 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
27 es01-elasticsearch-client ClusterIP 10.100.237.152 <none> 9200/TCP 6m34s
28 es01-elasticsearch-discovery ClusterIP None <none> 9300/TCP 6m42s
29 [root@k8s-master efk]#
30 [root@k8s-master efk]# kubectl get sts -n efk
31 NAME READY AGE
32 es01-elasticsearch-data 1/1 7m4s
33 es01-elasticsearch-master 3/3 7m4s
34 [root@k8s-master efk]#Elasticsearch访问
其中IP来源于ES的svc。
代码语言：javascript复制 1 [root@k8s-master ~]# curl 10.100.237.152:9200/
2 {
3 "name" : "es01-elasticsearch-client-646f8f866d-rt2wp",
4 "cluster_name" : "elasticsearch",
5 "cluster_uuid" : "S4t_UDOuRye9mtK22VWxLw",
6 "version" : {
7 "number" : "6.7.0",
8 "build_flavor" : "oss",
9 "build_type" : "docker",
10 "build_hash" : "8453f77",
11 "build_date" : "2019-03-21T15:32:29.844721Z",
12 "build_snapshot" : false,
13 "lucene_version" : "7.7.0",
14 "minimum_wire_compatibility_version" : "5.6.0",
15 "minimum_index_compatibility_version" : "5.0.0"
16 },
17 "tagline" : "You Know, for Search"
18 }
19 [root@k8s-master ~]#
20 [root@k8s-master ~]# curl 10.100.237.152:9200/_cluster/health?pretty
21 {
22 "cluster_name" : "elasticsearch",
23 "status" : "green", # 可见状态正常
24 "timed_out" : false,
25 "number_of_nodes" : 5,
26 "number_of_data_nodes" : 1,
27 "active_primary_shards" : 0,
28 "active_shards" : 0,
29 "relocating_shards" : 0,
30 "initializing_shards" : 0,
31 "unassigned_shards" : 0,
32 "delayed_unassigned_shards" : 0,
33 "number_of_pending_tasks" : 0,
34 "number_of_in_flight_fetch" : 0,
35 "task_max_waiting_in_queue_millis" : 0,
36 "active_shards_percent_as_number" : 100.0
37 }至此，elasticsearch部署完毕
elasticsearch-client域名获取
根据es01-elasticsearch-client的svc信息获取到es01-elasticsearch-client的域名；用于后面的fluentd 和kibana。
启动一个pod
代码语言：javascript复制 1 [root@k8s-master test]# pwd
2 /root/k8s_practice/test
3 [root@k8s-master test]# cat myapp_demo.yaml
4 apiVersion: v1
5 kind: Pod
6 metadata:
7 name: myapp-demo
8 namespace: default
9 labels:
10 k8s-app: myapp
11 spec:
12 containers:
13 - name: myapp
14 image: registry.cn-beijing.aliyuncs.com/google_registry/myapp:v1
15 imagePullPolicy: IfNotPresent
16 ports:
17 - name: httpd
18 containerPort: 80
19 protocol: TCP
20 [root@k8s-master test]#
21 [root@k8s-master test]# kubectl apply -f myapp_demo.yaml
22 pod/myapp-demo created
23 [root@k8s-master test]#
24 [root@k8s-master test]# kubectl get pod -o wide
25 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
26 myapp-demo 1/1 Running 0 6s 10.244.2.84 k8s-node02 <none> <none>进入pod并得到elasticsearch-client域名信息
代码语言：javascript复制 1 # 进入一个pod容器
2 [root@k8s-master test]# kubectl exec -it myapp-demo sh
3 ##### 格式 nslookup svc-cluster-ip
4 / # nslookup 10.100.237.152
5 nslookup: can't resolve '(null)': Name does not resolve
6
7 Name: 10.100.237.152
8 Address 1: 10.100.237.152 es01-elasticsearch-client.efk.svc.cluster.local
9 / #
10 / #
11 ### 通过ping判断域名是否通畅
12 / # ping es01-elasticsearch-client.efk.svc.cluster.local
13 PING es01-elasticsearch-client.efk.svc.cluster.local (10.100.237.152): 56 data bytes
14 64 bytes from 10.100.237.152: seq=0 ttl=64 time=0.094 ms
15 64 bytes from 10.100.237.152: seq=1 ttl=64 time=0.081 ms
16 64 bytes from 10.100.237.152: seq=2 ttl=64 time=0.243 ms由上可得，Service中es01-elasticsearch-client的域名为：es01-elasticsearch-client.efk.svc.cluster.local
Service的域名格式为：(service name). (namespace).svc.cluster.local，其中 cluster.local 指定的集群的域名
Fluentd部署
chart下载与配置修改
代码语言：javascript复制 1 [root@k8s-master efk]# pwd
2 /root/k8s_practice/efk
3 [root@k8s-master efk]#
4 # fluentd版本信息查看
5 [root@k8s-master efk]# helm search stable/fluentd-elasticsearch -l
6 NAME CHART VERSION APP VERSION DESCRIPTION
7 stable/fluentd-elasticsearch 2.0.7 2.3.2 DEPRECATED! - A Fluentd Helm chart for Kubernetes with El...
8 stable/fluentd-elasticsearch 2.0.6 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
9 stable/fluentd-elasticsearch 2.0.5 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou...
10 ………………
11 # 获取fluentd-elasticsearch 并解压
12 [root@k8s-master efk]# helm fetch stable/fluentd-elasticsearch --version 2.0.7
13 [root@k8s-master efk]# tar xf fluentd-elasticsearch-2.0.7.tgz
14 # 配置修改
15 [root@k8s-master efk]# vim fluentd-elasticsearch/values.yaml
16 ### 为什么使用域名而不是IP，因此每次重启ES的svc，对应IP都会改变。而域名是不变的
17 elasticsearch:
18 host: 'es01-elasticsearch-client.efk.svc.cluster.local' # 修改处，域名获取参见上文
19 port: 9200
20 scheme: 'http'
21 [root@k8s-master efk]#fluentd-elasticsearch部署
步骤如下：
代码语言：javascript复制 1 ################ 部署fluentd-elasticsearch
2 # 当前目录
3 [root@k8s-master efk]# pwd
4 /root/k8s_practice/efk
5 # 部署fluentd-elasticsearch
6 [root@k8s-master efk]# helm install --name fluentd-es01 --namespace=efk fluentd-elasticsearch
7 [root@k8s-master efk]#
8 # 状态查看
9 [root@k8s-master efk]# helm list
10 NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
11 es01 1 Sat Jul 25 12:18:50 2020 DEPLOYED elasticsearch-1.30.0 6.7.0 efk
12 fluentd-es01 1 Sat Jul 25 12:36:01 2020 DEPLOYED fluentd-elasticsearch-2.0.7 2.3.2 efk
13 [root@k8s-master efk]#
14 #查看pod状态信息如下
15 [root@k8s-master efk]# kubectl get ds -n efk
16 NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
17 fluentd-es01-fluentd-elasticsearch 2 2 2 2 2 <none> 113s
18 [root@k8s-master efk]#
19 [root@k8s-master efk]# kubectl get pod -n efk -o wide
20 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
21 es01-elasticsearch-client-646f8f866d-rt2wp 1/1 Running 0 17m 10.244.2.57 k8s-node02 <none> <none>
22 es01-elasticsearch-data-0 1/1 Running 0 17m 10.244.2.58 k8s-node02 <none> <none>
23 es01-elasticsearch-master-0 1/1 Running 0 17m 10.244.4.241 k8s-node01 <none> <none>
24 es01-elasticsearch-master-1 1/1 Running 0 17m 10.244.2.59 k8s-node02 <none> <none>
25 es01-elasticsearch-master-2 1/1 Running 0 16m 10.244.4.242 k8s-node01 <none> <none>
26 fluentd-es01-fluentd-elasticsearch-qnmf9 1/1 Running 0 43s 10.244.4.243 k8s-node01 <none> <none>
27 fluentd-es01-fluentd-elasticsearch-xmw5f 1/1 Running 0 43s 10.244.2.60 k8s-node02 <none> <none>Kibana部署
kibana的主版本和大版本必须和elasticsearch（ES）一致，小版本可以不同；但两者版本最好一致，这样可以避免一些因版本不同导致的意外情况。
由于elasticsearch（ES）使用的是6.7.0，因此kibana我们也是用该版本。
chart下载与配置修改
代码语言：javascript复制 1 [root@k8s-master efk]# pwd
2 /root/k8s_practice/efk
3 # 所有版本查看
4 [root@k8s-master efk]# helm search stable/kibana -l
5 NAME CHART VERSION APP VERSION DESCRIPTION
6 stable/kibana 3.2.7 6.7.0 Kibana is an open source data visualization plugin for El...
7 stable/kibana 3.2.6 6.7.0 Kibana is an open source data visualization plugin for El...
8 ………………
9 # 获取kibana，并解压缩
10 [root@k8s-master efk]# helm fetch stable/kibana --version 3.2.7
11 [root@k8s-master efk]# tar xf kibana-3.2.7.tgz
12 # 配置修改1
13 [root@k8s-master efk]# vim kibana/values.yaml
14 ### 为什么使用域名而不是IP，因此每次重启ES的svc，对应IP都会改变。而域名是不变的
15 files:
16 kibana.yml:
17 ## Default Kibana configuration from kibana-docker.
18 server.name: kibana
19 server.host: "0"
20 elasticsearch.url: http://es01-elasticsearch-client.efk.svc.cluster.local:9200 # 修改处，域名获取参见上文
21 ………………
22 service:
23 type: NodePort # 修改内容 从ClusterIP改为NodePort
24 externalPort: 443
25 internalPort: 5601
26 nodePort: 30601 # 添加处，Service端口范围：30000-32767
27 [root@k8s-master efk]#
28 # 配置修改2
29 [root@k8s-master efk]# vim kibana/templates/deployment.yaml
30 apiVersion: apps/v1 # 从 apps/v1beta1 改为 apps/v1
31 kind: Deployment
32 metadata:
33 ………………
34 spec:
35 replicas: {{ .Values.replicaCount }}
36 revisionHistoryLimit: {{ .Values.revisionHistoryLimit }}
37 # 添加信息 如下
38 selector:
39 matchLabels:
40 app: {{ template "kibana.name" . }}
41 release: "{{ .Release.Name }}"
42 # 添加信息 如上kibana部署
步骤如下：
代码语言：javascript复制 1 ################ 部署kibana-oss
2 # 当前目录
3 [root@k8s-master efk]# pwd
4 /root/k8s_practice/efk
5 # 部署kibana-oss
6 [root@k8s-master efk]# helm install --name kibana01 --namespace=efk kibana
7 [root@k8s-master efk]#
8 # 状态查看
9 [root@k8s-master efk]# helm list
10 NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
11 es01 1 Sat Jul 25 12:18:50 2020 DEPLOYED elasticsearch-1.30.0 6.7.0 efk
12 fluentd-es01 1 Sat Jul 25 12:36:01 2020 DEPLOYED fluentd-elasticsearch-2.0.7 2.3.2 efk
13 kibana01 1 Sat Jul 25 12:38:18 2020 DEPLOYED kibana-3.2.7 6.7.0 efk
14 [root@k8s-master efk]#
15 #查看pod状态信息如下
16 [root@k8s-master efk]# kubectl get deploy -n efk
17 NAME READY UP-TO-DATE AVAILABLE AGE
18 es01-elasticsearch-client 1/1 1 1 19m
19 kibana01 1/1 1 1 27s
20 [root@k8s-master efk]#
21 [root@k8s-master efk]# kubectl get pod -n efk -o wide
22 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
23 es01-elasticsearch-client-646f8f866d-rt2wp 1/1 Running 0 20m 10.244.2.57 k8s-node02 <none> <none>
24 es01-elasticsearch-data-0 1/1 Running 0 20m 10.244.2.58 k8s-node02 <none> <none>
25 es01-elasticsearch-master-0 1/1 Running 0 20m 10.244.4.241 k8s-node01 <none> <none>
26 es01-elasticsearch-master-1 1/1 Running 0 19m 10.244.2.59 k8s-node02 <none> <none>
27 es01-elasticsearch-master-2 1/1 Running 0 19m 10.244.4.242 k8s-node01 <none> <none>
28 fluentd-es01-fluentd-elasticsearch-qnmf9 1/1 Running 0 3m10s 10.244.4.243 k8s-node01 <none> <none>
29 fluentd-es01-fluentd-elasticsearch-xmw5f 1/1 Running 0 3m10s 10.244.2.60 k8s-node02 <none> <none>
30 kibana01-bc479f8c7-kr2n2 1/1 Running 0 53s 10.244.4.244 k8s-node01 <none> <none>
31 [root@k8s-master efk]#
32 # 查看svc信息
33 [root@k8s-master efk]# kubectl get svc -n efk -o wide
34 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
35 es01-elasticsearch-client ClusterIP 10.100.237.152 <none> 9200/TCP 20m app=elasticsearch,component=client,release=es01
36 es01-elasticsearch-discovery ClusterIP None <none> 9300/TCP 20m app=elasticsearch,component=master,release=es01
37 kibana01 NodePort 10.101.200.177 <none> 443:30601/TCP 71s app=kibana,release=kibana01浏览器访问
代码语言：javascript复制http://172.16.1.110:30601/页面访问
创建索引
数据查看
相关阅读
1、Kubernetes K8S之Helm部署、使用与示例
完毕！
