Kubernetes(k8s)1.14 离线版集群 - 部署高可用的Etcd（3.3.17版）集群并搭建Flannel网络

2020-06-11 16:23:01 浏览数 (1)

声明： 如果您有更好的技术与作者分享，或者商业合作； 请访问作者个人网站 http://www.esqabc.com/view/message.html 留言给作者。 如果该案例触犯您的专利，请在这里：http://www.esqabc.com/view/message.html 留言给作者说明原由 作者一经查实，马上删除。

. .

1、Etcd集群各节点的名称和ip如下：

前提提条件、服务器，请查看这个地址：https://blog.csdn.net/esqabc/article/details/102726771

内网IP

名称

172.26.16.249

k8s-01

172.26.16.250

k8s-02

172.26.16.251

k8s-03

2、下载etcd

a、官方下载地址：https://github.com/etcd-io/etcd/releases/tag/v3.3.17/etcd-v3.3.17-linux-amd64.tar.gz 下载后上传到：/opt/k8s/work

[root@k8s-01 ~]# cd /opt/k8s/work [root@k8s-01 ~]# tar -xvf etcd-v3.3.17-linux-amd64.tar.gz

b、分发二进制文件到集群节点

[root@k8s-01 ~]# cd /opt/k8s/work [root@k8s-01 ~]# source /opt/k8s/bin/environment.sh

代码语言：javascript复制
for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp etcd-v3.3.17-linux-amd64/etcd* root@${node_ip}:/opt/k8s/bin
    ssh root@${node_ip} "chmod  x /opt/k8s/bin/*"
  done

3、创建etcd证书和私钥

a、创建证书json

[root@k8s-01 ~]# cd /opt/k8s/work [root@k8s-01 ~]# cat > etcd-csr.json <<EOF 添加下面内容：

代码语言：javascript复制
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "172.26.16.249",
    "172.26.16.250",
    "172.26.16.251"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF

说明一下：

  • hosts：字段指定授权使用该证书的etcd节点IP或域名列表，需要将etcd集群的3个节点都添加其中

b、生成证书和私钥

[root@k8s-01 ~]# cd /opt/k8s/work

代码语言：javascript复制
cfssl gencert -ca=/opt/k8s/work/ca.pem 
    -ca-key=/opt/k8s/work/ca-key.pem 
    -config=/opt/k8s/work/ca-config.json 
    -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

[root@k8s-01 work]# ls etcd*pem

c、分发证书和私钥到etcd各个节点

[root@k8s-01 ~]# cd /opt/k8s/work [root@k8s-01 ~]# source /opt/k8s/bin/environment.sh

代码语言：javascript复制
for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /etc/etcd/cert"
    scp etcd*.pem root@${node_ip}:/etc/etcd/cert/
  done

d、创建etcd的启动文件

[root@k8s-01 ~]# cd /opt/k8s/work [root@k8s-01 ~]# cat > etcd.service.template <<EOF

代码语言：javascript复制
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=${ETCD_DATA_DIR}
ExecStart=/opt/k8s/bin/etcd \
  --data-dir=${ETCD_DATA_DIR} \
  --wal-dir=${ETCD_WAL_DIR} \
  --name=##NODE_NAME## \
  --cert-file=/etc/etcd/cert/etcd.pem \
  --key-file=/etc/etcd/cert/etcd-key.pem \
  --trusted-ca-file=/etc/kubernetes/cert/ca.pem \
  --peer-cert-file=/etc/etcd/cert/etcd.pem \
  --peer-key-file=/etc/etcd/cert/etcd-key.pem \
  --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --listen-peer-urls=https://##NODE_IP##:2380 \
  --initial-advertise-peer-urls=https://##NODE_IP##:2380 \
  --listen-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \
  --advertise-client-urls=https://##NODE_IP##:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster=${ETCD_NODES} \
  --initial-cluster-state=new \
  --auto-compaction-mode=periodic \
  --auto-compaction-retention=1 \
  --max-request-bytes=33554432 \
  --quota-backend-bytes=6442450944 \
  --heartbeat-interval=250 \
  --election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

说明一下：

  • WorkDirectory、–data-dir 指定etcd工作目录和数据存储为${ETCD_DATA_DIR},需要在启动前创建这个目录
  • –wal-dir 指定wal目录，为了提高性能，一般使用SSD和–data-dir不同的盘
  • –name 指定节点名称，当–initial-cluster-state值为new时，–name的参数值必须位于–initial-cluster列表中
  • –cert-file、–key-file ETCD server与client通信时使用的证书和私钥
  • –trusted-ca-file 签名client证书的CA证书，用于验证client证书
  • –peer-cert-file、–peer-key-file ETCD与peer通信使用的证书和私钥
  • –peer-trusted-ca-file 签名peer证书的CA证书，用于验证peer证书

e、分发启动文件到各个节点

(1）、分发会将配置文件中的#替换成ip

[root@k8s-01 ~]# cd /opt/k8s/work

代码语言：javascript复制
for (( i=0; i < 3; i   ))
  do
    sed -e "s/##NODE_NAME##/${MASTER_NAMES[i]}/" -e "s/##NODE_IP##/${ETCD_IPS[i]}/" etcd.service.template > etcd-${ETCD_IPS[i]}.service 
  done

[root@k8s-01 ~]# ls *.service

(2）、分发会将配置文件中的#替换成ip

[root@k8s-01 ~]# cd /opt/k8s/work [root@k8s-01 ~]# source /opt/k8s/bin/environment.sh

代码语言：javascript复制
for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp etcd-${node_ip}.service root@${node_ip}:/etc/systemd/system/etcd.service
  done

4、启动etcd服务

a、重命名etcd启动文件并启动etcd服务

[root@k8s-01 ~]# cd /opt/k8s/work [root@k8s-01 ~]# source /opt/k8s/bin/environment.sh

代码语言：javascript复制
for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p ${ETCD_DATA_DIR} ${ETCD_WAL_DIR}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd " &
  done

b、检查etcd启动结果

代码语言：javascript复制
for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status etcd|grep Active"
  done

如果启动正常，如下图：

如果etcd集群状态不是active (running)，请使用下面命令查看etcd日志

[root@k8s-01 ~]# journalctl -fu etcd

c、验证ETCD集群状态，在任意etcd节点执行

[root@k8s-01 ~]# cd /opt/k8s/work

代码语言：javascript复制
for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ETCDCTL_API=3 /opt/k8s/bin/etcdctl 
    --endpoints=https://${node_ip}:2379 
    --cacert=/etc/kubernetes/cert/ca.pem 
    --cert=/etc/etcd/cert/etcd.pem 
    --key=/etc/etcd/cert/etcd-key.pem endpoint health
  done

正常状态，如下图：

d、查看当前etcd集群leader

[root@k8s-01 ~]# cd /opt/k8s/work

代码语言：javascript复制
ETCDCTL_API=3 /opt/k8s/bin/etcdctl 
  -w table --cacert=/etc/kubernetes/cert/ca.pem 
  --cert=/etc/etcd/cert/etcd.pem 
  --key=/etc/etcd/cert/etcd-key.pem 
  --endpoints=${ETCD_ENDPOINTS} endpoint status

正常，如下图：

.

5、部署Flannel网络

a、下载分发flanneld二进制文件

[root@k8s-01 ~]# cd /opt/k8s/work [root@k8s-01 work]# mkdir flannel [root@k8s-01 work]# wget http://down.i4t.com/k8s1.14/flannel-v0.11.0-linux-amd64.tar.gz [root@k8s-01 work]# tar -xzvf flannel-v0.11.0-linux-amd64.tar.gz -C flannel

b、分发二进制文件到所有集群的节点

[root@k8s-01 ~]# cd /opt/k8s/work

代码语言：javascript复制
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp flannel/{flanneld,mk-docker-opts.sh} root@${node_ip}:/opt/k8s/bin/
    ssh root@${node_ip} "chmod  x /opt/k8s/bin/*"
  done

c、创建Flannel证书和私钥

[root@k8s-01 ~]# cd /opt/k8s/work [root@k8s-01 ~]# cat > flanneld-csr.json <<EOF 添加下面内容：

代码语言：javascript复制
{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF

d、生成证书和私钥

[root@k8s-01 ~]# cd /opt/k8s/work

代码语言：javascript复制
cfssl gencert -ca=/opt/k8s/work/ca.pem 
  -ca-key=/opt/k8s/work/ca-key.pem 
  -config=/opt/k8s/work/ca-config.json 
  -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

[root@k8s-01 ~]# ls flanneld*pem

e、将生成的证书和私钥分发到所有节点

[root@k8s-01 ~]# cd /opt/k8s/work

代码语言：javascript复制
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /etc/flanneld/cert"
    scp flanneld*.pem root@${node_ip}:/etc/flanneld/cert
  done

f、向etcd写入Pod网段信息

[root@k8s-01 ~]# cd /opt/k8s/work

代码语言：javascript复制
etcdctl 
  --endpoints=${ETCD_ENDPOINTS} 
  --ca-file=/opt/k8s/work/ca.pem 
  --cert-file=/opt/k8s/work/flanneld.pem 
  --key-file=/opt/k8s/work/flanneld-key.pem 
  mk ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 21, "Backend": {"Type": "vxlan"}}'

g、创建flanneld的启动文件

[root@k8s-01 ~]# cd /opt/k8s/work [root@k8s-01 work]# cat > flanneld.service << EOF

代码语言：javascript复制
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
ExecStart=/opt/k8s/bin/flanneld \
  -etcd-cafile=/etc/kubernetes/cert/ca.pem \
  -etcd-certfile=/etc/flanneld/cert/flanneld.pem \
  -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \
  -etcd-endpoints=${ETCD_ENDPOINTS} \
  -etcd-prefix=${FLANNEL_ETCD_PREFIX} \
  -iface=${IFACE} \
  -ip-masq
ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF

h、分发启动文件到所有节点

[root@k8s-01 ~]# cd /opt/k8s/work

代码语言：javascript复制
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp flanneld.service root@${node_ip}:/etc/systemd/system/
  done

i、启动flanneld服务

[root@k8s-01 ~]# cd /opt/k8s/work

代码语言：javascript复制
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld"
  done

j、检查启动结果

[root@k8s-01 ~]# cd /opt/k8s/work

代码语言：javascript复制
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status flanneld|grep Active"
  done

正常结果：

k、检查分配给flanneld的Pod网段信息

[root@k8s-01 ~]# cd /opt/k8s/work

代码语言：javascript复制
etcdctl 
  --endpoints=${ETCD_ENDPOINTS} 
  --ca-file=/etc/kubernetes/cert/ca.pem 
  --cert-file=/etc/flanneld/cert/flanneld.pem 
  --key-file=/etc/flanneld/cert/flanneld-key.pem 
  get ${FLANNEL_ETCD_PREFIX}/config

l、查看已分配的Pod子网网段列表

[root@k8s-01 ~]# cd /opt/k8s/work

代码语言：javascript复制
etcdctl 
  --endpoints=${ETCD_ENDPOINTS} 
  --ca-file=/etc/kubernetes/cert/ca.pem 
  --cert-file=/etc/flanneld/cert/flanneld.pem 
  --key-file=/etc/flanneld/cert/flanneld-key.pem 
  ls ${FLANNEL_ETCD_PREFIX}/subnets

m、查看节点flannel网络信息

[root@k8s-01 ~]# ip addr show

n、检查是否创建了 flannel 接口

[root@k8s-01 ~]# cd /opt/k8s/work

代码语言：javascript复制
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh ${node_ip} "/usr/sbin/ip addr show flannel.1|grep -w inet"
  done

. . .

也可以参考这篇文章：https://i4t.com/4253.html

kubernetes etcd linux bash bash指令

0 人点赞