Kubernetes-v1.24版安装部署之Mater节点安装部署

2023-04-24 17:33:55 浏览数 (1)

Kubernetes Master

Kubernetes Master节点部署三个服务：kube-apiserver、kube-controller-manager、kube-scheduler和一个命令工具kubectl.

Master节点来负责整个集群的管理和控制，其中

kube-apiserver: 服务提供了HTTP Rest接口的关键服务进程，是Kuberneters里所有资源的增删改查等操作的唯一入口，也是集群控制的入口进程.

kube-controller-manager: 服务 是kubernetes里面所有资源对象的自动化控制中心，可以理解为资源对象的”大总管”

kube-scheduler: 服务负责资源调度(pod调度)的进程，相当于公交公司的”调度室”。

安装Kube-apiserver

相关参数介绍

• —logtostderr：启用日志 • —-v：日志等级 • —log-dir：日志目录 • —etcd-servers：etcd集群地址 • —bind-address：监听地址 • —secure-port：https安全端口 • —advertise-address：集群通告地址 • —allow-privileged：启用授权 • —service-cluster-ip-range：Service虚拟IP地址段 • —enable-admission-plugins：准入控制模块 • —authorization-mode：认证授权，启用RBAC授权和节点自管理 • —enable-bootstrap-token-auth：启用TLS bootstrap机制 • —token-auth-file：bootstrap token文件 • —service-node-port-range：Service nodeport类型默认分配端口范围 • —kubelet-client-xxx：apiserver访问kubelet客户端证书 • —tls-xxx-file：apiserver https证书 • 1.20版本后必须加的参数：—service-account-issuer，—service-account-signing-key-file • —etcd-xxxfile：连接Etcd集群证书 • —audit-log-xxx：审计日志 • 启动聚合层相关配置：—requestheader-client-ca-file，—proxy-client-cert-file，—proxy-client-key-file，—requestheader-allowed-names，—requestheader-extra-headers-prefix，—requestheader-group-headers，—requestheader-username-headers，—enable-aggregator-routing 更多参数介绍：https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-apiserver/

创建证书

  • apiserver证书
  • ServiceAccount证书
代码语言：javascript复制
cat > /opt/certs/apiserver-csr.json <<EOF
{
    "CN": "kubernetes",
    "hosts": [
        "127.0.0.1",
        "192.168.0.1",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local",
        "10.1.1.50",
        "10.1.1.100",
        "10.1.1.110",
        "10.1.1.120",
        "10.1.1.130"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "apiserver",
            "OU": "kubernetes"
        }
    ]
}
EOF

## 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes apiserver-csr.json |cfssljson -bare apiserver
代码语言：javascript复制
cat > /opt/certs/sa-csr.json <<EOF
{
    "CN": "ServiceAccount",
    "hosts": [
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "ServiceAccount",
            "OU": "kubernetes"
        }
    ]
}
EOF

## 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes sa-csr.json |cfssljson -bare sa

安装kube-apiserver

下载地址： https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md

注：打开链接你会发现里面有很多包，下载一个server包就够了，包含了Master和Worker Node二进制文件。

代码语言：javascript复制
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} 
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kube-apiserver /opt/kubernetes/bin
cp kubectl /usr/bin/
scp 10.1.1.11:/opt/certs/apiserver*.pem /opt/kubernetes/ssl
scp 10.1.1.11:/opt/certs/ca*.pem /opt/kubernetes/ssl
scp 10.1.1.11:/opt/certs/sa*.pem /opt/kubernetes/ssl

配置apiserver文件

代码语言：javascript复制
cat > /opt/kubernetes/cfg/kube-apiserver.conf <<EOF
KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--etcd-servers=https://10.1.1.100:2379,https://10.1.1.130:2379,https://10.1.1.120:2379 \
--bind-address=10.1.1.100 \
--secure-port=6443 \
--advertise-address=10.1.1.100 \
--allow-privileged=true \
--service-cluster-ip-range=192.168.0.0/16 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-32767 \
--kubelet-client-certificate=/opt/kubernetes/ssl/apiserver.pem \
--kubelet-client-key=/opt/kubernetes/ssl/apiserver-key.pem \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/sa.pem \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--service-account-signing-key-file=/opt/kubernetes/ssl/sa-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/etcd.pem \
--etcd-keyfile=/opt/etcd/ssl/etcd-key.pem \
--tls-cert-file=/opt/kubernetes/ssl/apiserver.pem  \
--tls-private-key-file=/opt/kubernetes/ssl/apiserver-key.pem \
--proxy-client-cert-file=/opt/kubernetes/ssl/apiserver.pem \
--proxy-client-key-file=/opt/kubernetes/ssl/apiserver-key.pem \
--requestheader-allowed-names=aggregator \
--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--enable-aggregator-routing=true \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF

配置kube-apiserver启动文件

代码语言：javascript复制
cat > /usr/lib/systemd/system/kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

创建配置token文件

代码语言：javascript复制
 cat > /opt/kubernetes/cfg/token.csv <<EOF
bc43e407e311d78b60da186fdd347fc8,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

格式：token，用户名，UID，用户组

token也可自行生成替换：

代码语言：javascript复制
head -c 16 /dev/urandom | od -An -t x | tr -d ' '

启动apiserver

代码语言：javascript复制
systemctl daemon-reload
systemctl enable --now kube-apiserver

## 检查配置是否启动
# netstat -lnpt|grep 6443
tcp        0      0 10.1.1.100:6443         0.0.0.0:*               LISTEN      6905/kube-apiserver

常见错误

代码语言：javascript复制
# 此处为etcd正常关闭报错，故可忽略。
[transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"

授权apiserver访问kubelet

应用场景：例如kubectl logs

代码语言：javascript复制
cat > /opt/kubernetes/cfg/apiserver-to-kubelet-rbac.yaml <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - "*"
    resources:
      - nodes
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF

kubectl apply -f /opt/kubernetes/cfg/apiserver-to-kubelet-rbac.yaml

安装Kube-controller-manager

创建证书

代码语言：javascript复制
cat > /opt/certs/kube-controller-manager-csr.json << EOF
{
  "CN": "system:kube-controller-manager",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing", 
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "kubernetes"
    }
  ]
}
EOF
# 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

生成kubeconfig

代码语言：javascript复制
scp 10.1.1.11:/opt/certs/kube-controller-manager*.pem /opt/kubernetes/ssl/
KUBE_CONFIG="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig"
KUBE_APISERVER="https://10.1.1.100:6443"

# 设置集群参数
kubectl config set-cluster kubernetes 
  --certificate-authority=/opt/kubernetes/ssl/ca.pem 
  --embed-certs=true 
  --server=${KUBE_APISERVER} 
  --kubeconfig=${KUBE_CONFIG}
# 设置客户端认证参数 
kubectl config set-credentials kube-controller-manager 
  --client-certificate=/opt/kubernetes/ssl/kube-controller-manager.pem 
  --client-key=/opt/kubernetes/ssl/kube-controller-manager-key.pem 
  --embed-certs=true 
  --kubeconfig=${KUBE_CONFIG}
# 设置上下文参数  
kubectl config set-context default 
  --cluster=kubernetes 
  --user=kube-controller-manager 
  --kubeconfig=${KUBE_CONFIG}
# 设置默认上下文 
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

kube-controller-manager配置

代码语言：javascript复制
cd /server/tools/kubernetes/server/bin
cp kube-controller-manager /opt/kubernetes/bin

cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF        
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect=true \
--kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \
--cluster-cidr=172.7.0.0/16 \
--service-cluster-ip-range=192.168.0.0/16 \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/sa-key.pem \
--cluster-signing-duration=87600h0m0s"
EOF

• —kubeconfig：连接apiserver配置文件 • —leader-elect：当该组件启动多个时，自动选举（HA） • —cluster-signing-cert-file/—cluster-signing-key-file：自动为kubelet颁发证书的CA，与apiserver保持一致

配置kube-controller-manager启动文件

代码语言：javascript复制
cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

启动kube-controller-manager

代码语言：javascript复制
systemctl daemon-reload
systemctl enable --now kube-controller-manager

# 检查
netstat -lnpt|grep kube
tcp        0      0 10.1.1.100:6443         0.0.0.0:*               LISTEN      6905/kube-apiserver
tcp6       0      0 :::10257                :::*                    LISTEN      7253/kube-controlle

安装kube-scheduler

生成kube-scheduler证书

代码语言：javascript复制
cat > /opt/certs/kube-scheduler-csr.json << EOF
{
  "CN": "system:kube-scheduler",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "kubernetes"
    }
  ]
}
EOF

# 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

生成kubeconfig

代码语言：javascript复制
scp 10.1.1.11:/opt/certs/kube-scheduler*.pem /opt/kubernetes/ssl/

KUBE_CONFIG="/opt/kubernetes/cfg/kube-scheduler.kubeconfig"
KUBE_APISERVER="https://10.1.1.100:6443"

# 设置集群参数
kubectl config set-cluster kubernetes 
  --certificate-authority=/opt/kubernetes/ssl/ca.pem 
  --embed-certs=true 
  --server=${KUBE_APISERVER} 
  --kubeconfig=${KUBE_CONFIG}
# 设置客户端认证参数 
kubectl config set-credentials kube-scheduler 
  --client-certificate=/opt/kubernetes/ssl/kube-scheduler.pem 
  --client-key=/opt/kubernetes/ssl/kube-scheduler-key.pem 
  --embed-certs=true 
  --kubeconfig=${KUBE_CONFIG}
# 设置上下文参数  
kubectl config set-context default 
  --cluster=kubernetes 
  --user=kube-scheduler 
  --kubeconfig=${KUBE_CONFIG}
# 设置默认上下文 
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

创建kube-scheduler配置

代码语言：javascript复制
cd /server/tools/kubernetes/server/bin
cp kube-scheduler /opt/kubernetes/bin

cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect \
--kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \
--bind-address=127.0.0.1"
EOF

• —kubeconfig：连接apiserver配置文件

• —leader-elect：当该组件启动多个时，自动选举（HA）

kube-scheduler启动文件

代码语言：javascript复制
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

启动kube-scheduler

代码语言：javascript复制
systemctl daemon-reload
systemctl enable --now kube-scheduler

# 检查
netstat -lnpt|grep kube
tcp        0      0 10.1.1.100:6443         0.0.0.0:*               LISTEN      6905/kube-apiserver 
tcp        0      0 127.0.0.1:10259         0.0.0.0:*               LISTEN      7378/kube-scheduler 
tcp6       0      0 :::10257                :::*                    LISTEN      7253/kube-controlle 

[root@k8s-master1 ~]# tailf /opt/kubernetes/logs/kube-scheduler.INFO 
I0516 22:16:14.820411    7378 leaderelection.go:258] successfully acquired lease kube-system/kube-scheduler

查看集群状态

生成kubectl连接集群的证书

代码语言：javascript复制
cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

# 创建证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

生成kubeconfig文件

代码语言：javascript复制
scp 10.1.1.11:/opt/certs/admin*.pem /opt/kubernetes/ssl/ 

mkdir /root/.kube

KUBE_CONFIG="/root/.kube/config"
KUBE_APISERVER="https://10.1.1.100:6443"

# 设置集群参数
kubectl config set-cluster kubernetes 
  --certificate-authority=/opt/kubernetes/ssl/ca.pem 
  --embed-certs=true 
  --server=${KUBE_APISERVER} 
  --kubeconfig=${KUBE_CONFIG}
# 设置客户端认证参数 
kubectl config set-credentials cluster-admin 
  --client-certificate=/opt/kubernetes/ssl/admin.pem 
  --client-key=/opt/kubernetes/ssl/admin-key.pem 
  --embed-certs=true 
  --kubeconfig=${KUBE_CONFIG}
# 设置上下文参数 
kubectl config set-context default 
  --cluster=kubernetes 
  --user=cluster-admin 
  --kubeconfig=${KUBE_CONFIG}
# 设置默认上下文
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

查询

代码语言：javascript复制
[root@k8s-master1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19 
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
controller-manager   Healthy   ok                  
etcd-2               Healthy   {"health":"true"}

如上输出说明Master节点组件运行正常。

授权kubelet-bootstrap用户允许请求证书

代码语言：javascript复制
kubectl create clusterrolebinding kubelet-bootstrap 
--clusterrole=system:node-bootstrapper 
--user=kubelet-bootstrap
kubernetes 部署 服务 集群 配置

0 人点赞