二进制部署k8s教程03 - 部署etcd集群

!TIP 二进制部署 k8s - 部署 etcd 集群

转载请注明出处：https://janrs.com/qtcn 有任何问题欢迎在底部评论区发言。

Alma 版本 8.6

etcd 版本 3.5.5

部署 etcd 集群

1.初始化系统

!NOTE 每台 etcd 服务器都要设置

1-1.设置 hostname

!NOTE 设置后重启生效

代码语言：shell复制

hostnamectl set-hostname etcd-01
hostnamectl set-hostname etcd-02
hostnamectl set-hostname etcd-03

1-2.设置 /etc/hosts

代码语言：shell复制

cat >> /etc/hosts <<EOF
172.16.222.111  etcd-01
172.16.222.112  etcd-02
172.16.222.113  etcd-03
EOF

1-3.禁用 selinux

临时禁用和永久禁用

代码语言：shell复制

setenforce 0 && 
sed -ri '/^SELINUX/s/(SELINUX=)(.*)/1disabled/' /etc/selinux/config

1-4.禁用 firewalld 和 iptables

!NOTE 如果 iptables 没有安装则不用管

代码语言：shell复制

systemctl stop firewalld && systemctl disable firewalld && 
systemctl stop iptables && systemctl disable iptables

1-5.创建目录

代码语言：shell复制

mkdir -p /etc/etcd/ssl/ && 
mkdir -p /ssl/

1-6.设置接口版本变量

!NOTE 新版本的 etcd 使用的是 v3 版本的接口，有提供向下兼容 v2 版本的接口。如果需要，设置成使用 v3 版本的。这里使用的是 v3 版本。每个节点都需要设置。

代码语言：shell复制

cat >> /etc/profile <<EOF
export ETCDETC_API=3
EOF
source /etc/profile

2.创建 ssl 证书

!NOTE 使用的是 cloudflare 提供的 cfssl 证书创建工具。在 etcd 集群中需要创建 server , peer 两种证书： server 证书用于 etcd 开启 ssl 认证 peer 证书用于集群内节点双向通信
创建证书可以在任何地方创建，创建后上传到 etcd 服务器就行。

2-1.下载 cfssl

代码语言：shell复制

cd /home && 

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 && 

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 && 

chmod  x cfssl_linux-amd64 cfssljson_linux-amd64 && 

mv cfssl_linux-amd64 /usr/local/bin/cfssl && 

mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

2-2.创建证书

2-2-1.创建 ca 配置文件

代码语言：shell复制

cat > /ssl/ca-config.json <<EOF
{
    "signing": {
        "default": {
            "expiry": "43800h"
        },
        "profiles": {
            "server": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}
EOF

2-2-2.创建 ca 证书

!NOTE 该操作将生成 ca-key.pem , ca.csr , ca.pem 三个证书。 *.csr 证书在整个过程中将不会用到。要确保 ca.key 证书的安全。根证书用 4096 位保证安全。

代码语言：shell复制

cat > /ssl/etcd-ca-csr.json <<EOF
{
    "key": {
        "algo": "rsa",
        "size": 4096
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF
cd /ssl/ && 
cfssl gencert -initca etcd-ca-csr.json | 
cfssljson -bare etcd-ca - && 
ls etcd-ca* | grep etcd-ca

2-2-3.创建 server 证书

!NOTE 这里需要把 etcd 集群的 ip 地址都写进去。

代码语言：shell复制

cat > /ssl/etcd-server-csr.json <<EOF
{
    "CN": "etcd-server",
    "hosts": [
        "127.0.0.1",
        "172.16.222.111",
        "172.16.222.112",
        "172.16.222.113"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "O": "etcd-cluster",
            "OU": "System",
            "ST": "Beijing"
        }
    ]
}
EOF
cd /ssl/ && 
cfssl gencert 
-ca=etcd-ca.pem 
-ca-key=etcd-ca-key.pem 
-config=ca-config.json 
-profile=server etcd-server-csr.json | 
cfssljson -bare etcd-server && 
ls etcd-server* | grep etcd-server

2-2-4.创建 peer 双向对等证书

!NOTE 对于 etcd 集群，每个节点既是作为 server 又是作为 client 端，所以需要生成 peer 对等证书。同样需要把所有节点的 ip 地址都写进去。

代码语言：shell复制

cat > /ssl/etcd-peer-csr.json <<EOF
{
    "CN": "etcd-peer",
    "hosts": [
        "127.0.0.1",
        "172.16.222.111",
        "172.16.222.112",
        "172.16.222.113"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "O": "etcd-cluster",
            "OU": "System",
            "ST": "Beijing"
        }
    ]
}
EOF
cd /ssl/ && 
cfssl gencert 
-ca=etcd-ca.pem -ca-key=etcd-ca-key.pem 
-config=ca-config.json 
-profile=peer etcd-peer-csr.json | 
cfssljson -bare etcd-peer && 
ls etcd-peer* | grep etcd-peer

2-3.分发证书

2-3-1.分发 ca 证书

代码语言：shell复制

scp /ssl/etcd-ca.pem root@172.16.222.111:/etc/etcd/ssl/ && 
scp /ssl/etcd-ca.pem root@172.16.222.112:/etc/etcd/ssl/ && 
scp /ssl/etcd-ca.pem root@172.16.222.113:/etc/etcd/ssl/

2-3-2.分发 server 证书

代码语言：shell复制

scp /ssl/etcd-server*.pem root@172.16.222.111:/etc/etcd/ssl/ && 

scp /ssl/etcd-server*.pem root@172.16.222.112:/etc/etcd/ssl/ && 

scp /ssl/etcd-server*.pem root@172.16.222.113:/etc/etcd/ssl/

2-3-3.分发 peer 证书

代码语言：shell复制

scp /ssl/etcd-peer*.pem root@172.16.222.111:/etc/etcd/ssl/ && 

scp /ssl/etcd-peer*.pem root@172.16.222.112:/etc/etcd/ssl/ && 

scp /ssl/etcd-peer*.pem root@172.16.222.113:/etc/etcd/ssl/

3.部署 etcd

3-1.创建目录

!NOTE 每台服务器都要创建。工作目录为：/var/lib/etcd 数据存放目录：/data/etcd 配置文件地址：/etc/etcd/etcd.conf 集群部署方式为：static。其他部署方式还有：etcd discovery 以及 DNS discovery

代码语言：shell复制

mkdir /var/lib/etcd/ && 
mkdir -p /data/etcd/ && 
chmod -R 777 /data/etcd/

3-2.下载 etcd

下载对应硬件架构的 etcd 二进制文件

代码语言：shell复制

cd /home && 
wget https://github.com/etcd-io/etcd/releases/download/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz

解压

代码语言：shell复制

tar -zxvf etcd-v3.5.5-linux-amd64.tar.gz

复制命令到 /usr/local/bin/

代码语言：shell复制

cd /home/etcd-v3.5.5-linux-amd64/ && 
cp etcd etcdctl etcdutl /usr/local/bin/

分发二进制文件到每台服务器

代码语言：shell复制

scp etcd etcdctl etcdutl root@172.16.222.111:/usr/local/bin/ && 

scp etcd etcdctl etcdutl root@172.16.222.112:/usr/local/bin/ && 

scp etcd etcdctl etcdutl root@172.16.222.113:/usr/local/bin/

3-3.创建 etcd.conf

!NOTE 每台服务器都要创建。 ip 地址改为自己对应的。

3-3-1.在 etcd-01 服务器创建

代码语言：shell复制

cat > /etc/etcd/etcd.conf <<EOF
ETCD_NAME=etcd-01
ETCD_DATA_DIR="/data/etcd/"

ETCD_LISTEN_CLIENT_URLS="https://172.16.222.111:2379,https://127.0.0.1:2379"

ETCD_LISTEN_PEER_URLS="https://172.16.222.111:2380"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.222.111:2380"

ETCD_INITIAL_CLUSTER="etcd-01=https://172.16.222.111:2380,etcd-02=https://172.16.222.112:2380,etcd-03=https://172.16.222.113:2380"

ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="https://172.16.222.111:2379"

ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"
ETCD_CERT_FILE="/etc/etcd/ssl/etcd-server.pem"
ETCD_KEY_FILE="/etc/etcd/ssl/etcd-server-key.pem"

ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd-peer.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-peer-key.pem"
EOF

3-3-2.在 etcd-02 服务器创建

代码语言：shell复制

cat > /etc/etcd/etcd.conf <<EOF
ETCD_NAME=etcd-02
ETCD_DATA_DIR="/data/etcd/"

ETCD_LISTEN_CLIENT_URLS="https://172.16.222.112:2379,https://127.0.0.1:2379"

ETCD_LISTEN_PEER_URLS="https://172.16.222.112:2380"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.222.112:2380"

ETCD_INITIAL_CLUSTER="etcd-01=https://172.16.222.111:2380,etcd-02=https://172.16.222.112:2380,etcd-03=https://172.16.222.113:2380"

ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="https://172.16.222.112:2379"

ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"
ETCD_CERT_FILE="/etc/etcd/ssl/etcd-server.pem"
ETCD_KEY_FILE="/etc/etcd/ssl/etcd-server-key.pem"

ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd-peer.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-peer-key.pem"
EOF

3-3-4.在 etcd-03 服务器创建

代码语言：shell复制

cat > /etc/etcd/etcd.conf <<EOF
ETCD_NAME=etcd-03
ETCD_DATA_DIR="/data/etcd/"

ETCD_LISTEN_CLIENT_URLS="https://172.16.222.113:2379,https://127.0.0.1:2379"

ETCD_LISTEN_PEER_URLS="https://172.16.222.113:2380"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.222.113:2380"

ETCD_INITIAL_CLUSTER="etcd-01=https://172.16.222.111:2380,etcd-02=https://172.16.222.112:2380,etcd-03=https://172.16.222.113:2380"

ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="https://172.16.222.113:2379"

ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"
ETCD_CERT_FILE="/etc/etcd/ssl/etcd-server.pem"
ETCD_KEY_FILE="/etc/etcd/ssl/etcd-server-key.pem"

ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd-peer.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-peer-key.pem"
EOF

3-4.创建启动服务

!NOTE 每台服务器都创建，启动项都一样。

代码语言：shell复制

cat > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
Before=kube-apiserver.service

[Service]
Type=simple
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd"
Restart=always
RestartSec=10
Type=notify

[Install]
WantedBy=multi-user.target
EOF

3-5.启动服务

!NOTE 每台服务器都执行。由于是部署集群而不是单节点部署，所以要打开三个 ssh 标签都连接到服务器并且尽可能同时执行启动。因为每个服务都会在一定时间内监听其他服务的状态把集群节点加入进来。

代码语言：shell复制

systemctl start etcd && 
systemctl enable etcd

3-6.检查状态

检查健康状态

代码语言：shell复制

etcdctl --cacert=/etc/etcd/ssl/etcd-ca.pem 
--cert=/etc/etcd/ssl/etcd-peer.pem 
--key=/etc/etcd/ssl/etcd-peer-key.pem 
--endpoints="https://172.16.222.111:2379,https://172.16.222.112:2379,https://172.16.222.113:2379" 
endpoint health --write-out="table"

显示。可以看到 HEALTH 状态为 true

代码语言：text复制

 ----------------------------- -------- ------------- ------- 
|          ENDPOINT           | HEALTH |    TOOK     | ERROR |
 ----------------------------- -------- ------------- ------- 
| https://172.16.222.111:2379 |   true | 20.329616ms |       |
| https://172.16.222.112:2379 |   true | 24.033545ms |       |
| https://172.16.222.113:2379 |   true | 23.918639ms |       |
 ----------------------------- -------- ------------- -------

查看成员列表

代码语言：shell复制

etcdctl --cacert=/etc/etcd/ssl/etcd-ca.pem 
--cert=/etc/etcd/ssl/etcd-peer.pem 
--key=/etc/etcd/ssl/etcd-peer-key.pem 
--endpoints="https://172.16.222.111:2379,https://172.16.222.112:2379,https://172.16.222.113:2379" 
member list --write-out="table"

显示

代码语言：text复制

 ------------------ --------- --------- ----------------------------- ----------------------------- ------------ 
|        ID        | STATUS  |  NAME   |         PEER ADDRS          |        CLIENT ADDRS         | IS LEARNER |
 ------------------ --------- --------- ----------------------------- ----------------------------- ------------ 
| 1cdccba2db807b88 | started | etcd-01 | https://172.16.222.111:2380 | https://172.16.222.111:2379 |      false |
| 58badf9cfae9256f | started | etcd-03 | https://172.16.222.113:2380 | https://172.16.222.113:2379 |      false |
| 7cf1d6781c958345 | started | etcd-02 | https://172.16.222.112:2380 | https://172.16.222.112:2379 |      false |
 ------------------ --------- --------- ----------------------------- ----------------------------- ------------

3-7.其他操作

停止服务

代码语言：shell复制

systemctl stop etcd && 
systemctl disable etcd && 
systemctl daemon-reload

如果重启报错需要删除旧的数据

代码语言：shell复制

rm -rvf /data/etcd/* && rm -rvf /var/lib/etcd/*

删除 ssl 证书

代码语言：shell复制

rm -rvf /etc/etcd/ssl/*

删除配置文件

代码语言：shell复制

rm -rvf /etc/etcd/etcd.conf

至此。 etcd 集群部署成功。

转载请注明出处：https://janrs.com/qtcn 有任何问题欢迎在底部评论区发言。

etcd 数据分析 SSL证书 https tcp/ip

0 人点赞