K8S(V1.10.1)高可用集群超详细版本(包含Dashboard、Rancher)

2022-11-21 18:48:47 浏览数 (1)

IP

主机名

CPU

运行内存

备注

192.168.1.10

node01

2

4

Master and etcd

192.168.1.20

node02

2

4

Master and etcd

192.168.1.30

node03

2

4

Master and etcd

192.168.1.40

node04

1

2

node

192.168.1.50

node05

1

2

node

192.168.1.60

node06

1

2

node

软件

版本

kubernetes等组件

V1.10.1

docker

V1.13.1

①环境初始化

分别在6台主机设置主机名称

代码语言：javascript复制
hostnamectl set-hostname node01
hostnamectl set-hostname node02
hostnamectl set-hostname node03
hostnamectl set-hostname node04
hostnamectl set-hostname node05
hostnamectl set-hostname node06

配置主机映射

代码语言：javascript复制
echo '192.168.1.10 node01
192.168.1.20 node02
192.168.1.30 node03
192.168.1.40 node04
192.168.1.50 node05
192.168.1.60 node06' >> /etc/hosts

node01上执行ssh免密码登陆配置

代码语言：javascript复制
ssh-keygen  #一路回车即可
ssh-copy-id  -i node01
ssh-copy-id  -i node02
ssh-copy-id  -i node03
ssh-copy-id  -i node04
ssh-copy-id  -i node05
ssh-copy-id  -i node06

六台主机配置、停防火墙、关闭Swap、关闭Selinux、设置内核、安装依赖包、配置ntp（配置完后建议重启一次）

代码语言：javascript复制
systemctl stop firewalld
systemctl disable firewalld

swapoff -a 
sed -i 's/.*swap.*/#&/' /etc/fstab

setenforce  0 
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux 
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config 
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/sysconfig/selinux 
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/selinux/config  

modprobe br_netfilter
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf
ls /proc/sys/net/bridge


yum install -y epel-release
yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools conntrack-tools wget vim  ntpdate libseccomp libtool-ltdl 

systemctl enable ntpdate.service
echo '*/30 * * * * /usr/sbin/ntpdate time7.aliyun.com >/dev/null 2>&1' > /tmp/crontab2.tmp
crontab /tmp/crontab2.tmp
systemctl start ntpdate.service
 
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536"  >> /etc/security/limits.conf
echo "* hard nproc 65536"  >> /etc/security/limits.conf
echo "* soft  memlock  unlimited"  >> /etc/security/limits.conf
echo "* hard memlock  unlimited"  >> /etc/security/limits.conf
-----------------------------------------------------------------------------------------------------------------------
=======================================================================================================================
-----------------------------------------------------------------------------------------------------------------------
重启过后建议在执行一次
sysctl -p /etc/sysctl.d/k8s.conf
如果报错：
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: 没有那个文件或目录
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: 没有那个文件或目录
再次执行以下命令：
modprobe br_netfilter
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf
ls /proc/sys/net/bridge

--------------------------------------------------------------------------------
修改iptables的另一种方法 区别：
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
1.临时开启，（写入内存，在内存中开启）
echo "1" > /proc/sys/net/ipv4/ip_forward

2.永久开启，（写入内核）
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf 
sysctl -p

②创建etcd证书(node01上执行即可)

设置cfssl环境

代码语言：javascript复制
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod  x cfssl_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
chmod  x cfssljson_linux-amd64
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
chmod  x cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
export PATH=/usr/local/bin:$PATH

创建 CA 配置文件（下面配置的IP为etc节点的IP）

代码语言：javascript复制
mkdir /root/ssl
cd /root/ssl
cat >  ca-config.json <<EOF
{
"signing": {
"default": {
  "expiry": "8760h"
},
"profiles": {
  "kubernetes-Soulmate": {
    "usages": [
        "signing",
        "key encipherment",
        "server auth",
        "client auth"
    ],
    "expiry": "8760h"
  }
}
}
}
EOF

cat >  ca-csr.json <<EOF
{
"CN": "kubernetes-Soulmate",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
  "C": "CN",
  "ST": "shanghai",
  "L": "shanghai",
  "O": "k8s",
  "OU": "System"
}
]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.1.10",
    "192.168.1.20",
    "192.168.1.30"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "shanghai",
      "L": "shanghai",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem 
  -ca-key=ca-key.pem 
  -config=ca-config.json 
  -profile=kubernetes-Soulmate etcd-csr.json | cfssljson -bare etcd

node01分发etcd证书到node02、node03上面

代码语言：javascript复制
mkdir -p /etc/etcd/ssl
cp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/
ssh -n node02 "mkdir -p /etc/etcd/ssl && exit"
ssh -n node03 "mkdir -p /etc/etcd/ssl && exit"
scp -r /etc/etcd/ssl/*.pem node02:/etc/etcd/ssl/
scp -r /etc/etcd/ssl/*.pem node03:/etc/etcd/ssl/

安装配置etcd (三主节点） 安装etcd

代码语言：javascript复制
yum install etcd -y
mkdir -p /var/lib/etcd

node01的etcd.service

代码语言：javascript复制
cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd 
  --name node01 
  --cert-file=/etc/etcd/ssl/etcd.pem 
  --key-file=/etc/etcd/ssl/etcd-key.pem 
  --peer-cert-file=/etc/etcd/ssl/etcd.pem 
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem 
  --trusted-ca-file=/etc/etcd/ssl/ca.pem 
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem 
  --initial-advertise-peer-urls https://192.168.1.10:2380 
  --listen-peer-urls https://192.168.1.10:2380 
  --listen-client-urls https://192.168.1.10:2379,http://127.0.0.1:2379 
  --advertise-client-urls https://192.168.1.10:2379 
  --initial-cluster-token etcd-cluster-0 
  --initial-cluster node01=https://192.168.1.10:2380,node02=https://192.168.1.20:2380,node03=https://192.168.1.30:2380 
  --initial-cluster-state new 
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

node02的etcd.service

代码语言：javascript复制
cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd 
  --name node02 
  --cert-file=/etc/etcd/ssl/etcd.pem 
  --key-file=/etc/etcd/ssl/etcd-key.pem 
  --peer-cert-file=/etc/etcd/ssl/etcd.pem 
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem 
  --trusted-ca-file=/etc/etcd/ssl/ca.pem 
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem 
  --initial-advertise-peer-urls https://192.168.1.20:2380 
  --listen-peer-urls https://192.168.1.20:2380 
  --listen-client-urls https://192.168.1.20:2379,http://127.0.0.1:2379 
  --advertise-client-urls https://192.168.1.20:2379 
  --initial-cluster-token etcd-cluster-0 
  --initial-cluster node01=https://192.168.1.10:2380,node02=https://192.168.1.20:2380,node03=https://192.168.1.30:2380 
  --initial-cluster-state new 
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

node03的etcd.service

代码语言：javascript复制
cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd 
  --name node03 
  --cert-file=/etc/etcd/ssl/etcd.pem 
  --key-file=/etc/etcd/ssl/etcd-key.pem 
  --peer-cert-file=/etc/etcd/ssl/etcd.pem 
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem 
  --trusted-ca-file=/etc/etcd/ssl/ca.pem 
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem 
  --initial-advertise-peer-urls https://192.168.1.30:2380 
  --listen-peer-urls https://192.168.1.30:2380 
  --listen-client-urls https://192.168.1.30:2379,http://127.0.0.1:2379 
  --advertise-client-urls https://192.168.1.30:2379 
  --initial-cluster-token etcd-cluster-0 
--initial-cluster node01=https://192.168.1.10:2380,node02=https://192.168.1.20:2380,node03=https://192.168.1.30:2380 
  --initial-cluster-state new 
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

添加自启动（etc集群最少2个节点才能启动，启动报错看mesages日志）

代码语言：javascript复制
 mv /etc/systemd/system/etcd.service /usr/lib/systemd/system/
 systemctl daemon-reload
 systemctl enable etcd
 systemctl start etcd
 systemctl status etcd

在三个etcd节点执行一下命令检查

代码语言：javascript复制
etcdctl --endpoints=https://192.168.1.10:2379,https://192.168.1.20:2379,https://192.168.1.30:2379 
  --ca-file=/etc/etcd/ssl/ca.pem 
  --cert-file=/etc/etcd/ssl/etcd.pem 
  --key-file=/etc/etcd/ssl/etcd-key.pem  cluster-health

③所有节点安装配置docker

因为本次要部署的K8S版本为1.10.1版本，版本较低，所以docker版本不要安装太高的版本（亲测19.03版本不兼容）

代码语言：javascript复制
yum install docker -y
systemctl start docker && systemctl enable docker

默认安装的是1.13版本（目前最新版本是1.13版本），不要安装docker-ce，docker-ce会安装19.03版本（截止目前的最新版本） 查看docker cgroup

代码语言：javascript复制
docker info

1.10.1版本的K8S使用的是cgroup driver，1.14之后的版本才推荐使用systemd，docker与K8S两者必须要保持一致，否则会报错

如果查看到的cgroup不为cgroupfs，需要修改

代码语言：javascript复制
vim /usr/lib/systemd/system/docker.service

可以在添加一个加速

修改完成后重载配置，重启docker

代码语言：javascript复制
systemctl daemon-reload && systemctl restart docker

查看docker版本

代码语言：javascript复制
# docker --version
Docker version 1.13.1, build 0be3e21/1.13.1

④安装kubeadm，kubectl，kubelet

下载必要离线包，因为不下载的话，后面在初始化kubeadm时，会去自动拉去必要镜像，有些镜像需要翻墙才能下得下来

etcd 容器镜像服务 node.js tcp/ip

0 人点赞