[TOC]
0x00 前言简述
kubernetes 源码编译，分为本地二进制可执行文件编译和 docker 镜像编译两种， 之前演示的 minikube 方式或 kubeadm 方式安装，都是基于第二种 docker 镜像方式运行，当然也可以基于二进制文件方式安装，不管哪种方式，都是直接使用并不需要修改任何 k8s 代码。
不过当我们有特殊需求时，比如需要修改 kube-proxy 对 service 的代理逻辑，kube-scheduler 对 pod 的调度逻辑等，这个时候就需要修改 k8s 源码了，为了让修改的代码生效，就需要对 k8s 代码执行编译了。当然 k8s 也为我们提供了 CRD 等可扩展插件，在不修改 k8s 源码的基础上实现自定义功能，但是对于一些底层逻辑策略需要修改的话，还是办不到的。
kubernetes 开发者参考: https://github.com/kubernetes/community/blob/master/contributors/devel/development.md
注意：因为 kubernetes 源码是由 go 语言编写的，所以要编译其源码，需要安装 go 环境，若我们想基于 docker 镜像编译 kubernetes 的话，那么还需要安装 docker 环境。
0x01 开发环境
描述: kubernetes 开发依赖于Go所以它是必须的，其次是Git为了拉取官方的kubernetes项目；
Kunernetes 常用开发环境安装方式:
- 本地二进制可执行文件编译
- Docker镜像安装(Google 提供了 kube-core 但需要访问国外网站)
1.本地开发
环境准备:
Ubuntu & Git
代码语言：javascript复制> lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal
> uname -r
5.4.0-56-generic
> git version
git version 2.25.Go 语言安装
代码语言：javascript复制# Go官网：https://golang.org/dl/
# Go中文社区：https://studygolang.com/dl
wget https://dl.google.com/go/go1.15.5.linux-amd64.tar.gz
tar -zxvf go1.15.5.linux-amd64.tar.gz -C /home/weiyigeek/app/program/go/
vi /etc/profile
export GOPATH=/home/weiyigeek/app/program/go/src/ # GOPATH 作为 go 执行的工作路径
export GOROOT=/home/weiyigeek/app/program/go # go 安装目录
export PATH=$PATH:${GOROOT}/bin
source /etc/profile
# 环境验证
$ env | grep "GO"
GOPATH=/home/weiyigeek/app/program/go/src/
GOROOT=/home/weiyigeek/app/program/go
$ go version
go version go1.15.5 linux/amd64Kubernetes 项目源码
代码语言：javascript复制# 方式1: 拉取项目所有分支
mkdir -p $GOPATH/src/k8s.io && cd $GOPATH/src/k8s.io
> git clone https://github.com/kubernetes/kubernetes.git
# $ git clone https://github.com/kubernetes/kubernetes -b release-1.19.3 # 拉取Kubernetes源码指定分支
$ ls kubernetes
# api cluster go.mod logo pkg SUPPORT.md WORKSPACE
# build cmd go.sum Makefile plugin test
# BUILD.bazel code-of-conduct.md hack Makefile.generated_files README.md third_party
# CHANGELOG CONTRIBUTING.md LICENSE OWNERS SECURITY_CONTACTS translation
# 方式2: 拉取指定K8S版本源代码
K8S_VERSION=1.19.3
wget -c --no-check-certificate https://github.com/kubernetes/kubernetes/archive/v${K8S_VERSION}.tar.gz
> tar -tf v1.19.3.tar.gz | more
kubernetes-1.19.3/
kubernetes-1.19.3/.bazelrc
kubernetes-1.19.3/.bazelversion
kubernetes-1.19.3/.generated_filesDocker 镜像安装
描述: 安装Docker以及加速源的配置这里不再累述了，不会的读者参考官网或者本博客中的Docker安装流程;
代码语言：javascript复制# 运行容器，并进入到容器内部
docker run --rm -it -v /home/weiyigeek/kubernetes/k8s-src:/go/src/k8s.io/kubernetes mirrorgooglecontainers/kube-cross:v1.12.10-1 bash复制代码
vim staging/src/k8s.io/client-go/util/cert/cert.go # kubeadm 1.14 版本之前 vim cmd/kubeadm/app/util/pkiutil/pki_helpers.go # kubeadm 1.14 至今 const duration365d = time.Hour 24 365 * 10 #设置为10年 NotAfter: time.Now().Add(duration365d).UTC(), #替换 make WHAT=cmd/kubeadm GOFLAGS=-v #设置只编译kubeadm cp _output/bin/kubeadm /root/kubeadm-new
复制代码
⒌更新 kubeadm
#将kubeadm 进行替换 cp /usr/bin/kubeadm /usr/bin/kubeadm.old cp /root/kubeadm-new /usr/bin/kubeadm chmod a x /usr/bin/kubeadm
⒍更新各节点证书至Master节点
cp -r /etc/kubernetes/pki /etc/kubernetes/pki.old cd /etc/kubernetes/pki kubeadm alpha certs renew all –config=/root/kubeadm-config.yaml openssl x509 -in apiserver.crt -text -noout | grep Not
⒎HA集群其余 mater节点证书更新 复制代码
#!/bin/bash masterNode=”192.168.66.20 192.168.66.21”
#for host in ${masterNode}; do
scp /etc/kubernetes/pki/{ca.crt,ca.key,sa.key,sa.pub,front-proxy-ca.crt,front-proxy-ca.key}
“${USER}”@$host:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/etcd/{ca.crt,ca.key} “root”@$host:/etc/kubernetes/pki/etcd
scp /etc/kubernetes/admin.conf “root”@$host:/etc/kubernetes/
#donefor host in {CONTROL_PLANE_IPS}; do scp /etc/kubernetes/pki/{ca.crt,ca.key,sa.key,sa.pub,front-proxy-ca.crt,front-proxy-ca.key}“{USER}”@host:/root/pki/ scp /etc/kubernetes/pki/etcd/{ca.crt,ca.key} “root”@host:/root/etcd scp /etc/kubernetes/admin.conf “root”@
https://blog.csdn.net/aixiaoyang168/article/details/87917184
weiyigeek@Ubuntu-PC:~/app/program/go/src/k8s.io/kubernetes/cmd/kubeadm
go build -v .
weiyigeek@Ubuntu-PC:~/app/program/go/src/k8s.io/kubernetes/cmd/kubeadm
ls app BUILD kubeadm kubeadm.go OWNERS test
weiyigeek@Ubuntu-PC:~/app/program/go/src/k8s.io/kubernetes/cmd/kubeadm
./kubeadm version kubeadm version: &version.Info{Major:””, Minor:””, GitVersion:”v0.0.0-master 1e11e4a210802”, GitCommit:”1e11e4a2108024935ecfcb2912226cedeafd99df”, GitTreeState:””, BuildDate:”1970-01-01T00:00:00Z”, GoVersion:”go1.15.5”, Compiler:”gc”, Platform:”linux/amd64”}
实操1.Kubeadm / etcd证书过期或者延期处理流程
描述: 下述操作主要用于处理已过期或者即将过期的kubernetes集群证书;
Kubernetes 与 Etcd 版本与证书关联信息说明:
- Etcd 版本小于等于v1.9版本，etcd默认是不使用TLS连接，没有etcd相关证书，只需要更新master证书即可。
- Etcd 版本大于等于v1.10版本，etcd默认开启TLS，需要更新etcd证书和master证书。
/etc/kubernetes/pki/etcd$ ls
# ca.crt ca.key healthcheck-client.crt healthcheck-client.key peer.crt peer.key server.crt server.key
/etc/kubernetes/pki/etcd$ sudo openssl x509 -in server.crt -text -noout
# Certificate:
# Data:
# Version: 3 (0x2)
# Serial Number: 4411755188724745942 (0x3d39b3f83becc6d6)
# Signature Algorithm: sha256WithRSAEncryption
# Issuer: CN = etcd-ca
# Validity
# Not Before: Nov 5 08:47:32 2020 GMT
# Not After : Dec 12 09:17:18 2021 GMT # 关键点PS : 如果 Etcd 是由Kubeadm创建和托管的此时也可以通过下面的方式进行证书的续期, 如果是外部管理需要手动进行配置;
操作流程: 1.采用以下命令查询证书可用时间
代码语言：javascript复制# apiserver.crt (证书一年)
/etc/kubernetes/pki# openssl x509 -in apiserver.crt -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 3597153141676254659 (0x31eba7b94c4b25c3)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = kubernetes
Validity
Not Before: Nov 5 08:47:32 2020 GMT
Not After : Nov 5 08:47:32 2021 GMT
# ca.crt (证书十年)
/etc/kubernetes/pki# openssl x509 -in ca.crt -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 0 (0x0)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = kubernetes
Validity
Not Before: Nov 5 08:47:32 2020 GMT
Not After : Nov 3 08:47:32 2030 GMT
Subject: CN = kubernetes2.查看当前集群证书相关信息
代码语言：javascript复制$ sudo kubeadm alpha certs check-expiration
# [check-expiration] Reading configuration from the cluster...
# [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
# CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY(认证中心) EXTERNALLY MANAGED(外部管理)
# admin.conf Nov 05, 2021 08:47 UTC 328d no
# apiserver Nov 05, 2021 08:47 UTC 328d ca no
# apiserver-etcd-client Nov 05, 2021 08:47 UTC 328d etcd-ca no
# apiserver-kubelet-client Nov 05, 2021 08:47 UTC 328d ca no
# controller-manager.conf Nov 05, 2021 08:47 UTC 328d no
# etcd-healthcheck-client Nov 05, 2021 08:47 UTC 328d etcd-ca no
# etcd-peer Nov 05, 2021 08:47 UTC 328d etcd-ca no
# etcd-server Nov 05, 2021 08:47 UTC 328d etcd-ca no
# front-proxy-client Nov 05, 2021 08:47 UTC 328d front-proxy-ca no
# scheduler.conf Nov 05, 2021 08:47 UTC 328d no
# CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
# ca Nov 03, 2030 08:47 UTC 9y no
# etcd-ca Nov 03, 2030 08:47 UTC 9y no
# front-proxy-ca Nov 03, 2030 08:47 UTC 9y no3.采用renew子命令刷新证书的到期时间进行续期
代码语言：javascript复制~/.k8s$ sudo kubeadm alpha certs renew all --config=./kubeadm-init-config.yaml
# W1212 17:17:16.721037 1306627 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
# certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed # 嵌入在kubeconfig文件中的证书，供管理员使用，并对kubeadm本身进行更新 (admin.conf )
# certificate for serving the Kubernetes API renewed # 更新Kubernetes API服务证书
# certificate the apiserver uses to access etcd renewed # 服务器访问etcd所使用的证书已更新
# certificate for the API server to connect to kubelet renewed # API服务器连接到kubelet的证书已更新
# certificate embedded in the kubeconfig file for the controller manager to use renewed # 证书嵌入在kubeconfig文件中，供控制器管理器使用更新 (controller-manager.conf)
# certificate for liveness probes to healthcheck etcd renewed # 健康检查etcd激活探针证书续期
# certificate for etcd nodes to communicate with each other renewed # 用于etcd节点之间通信的证书更新
# certificate for serving etcd renewed # 续期etcd“服务证书”
# certificate for the front proxy client renewed # 前代理客户端的证书更新
# certificate embedded in the kubeconfig file for the scheduler manager to use renewed #证书嵌入在kubeconfig文件中，供调度器管理器使用更新 (scheduler.conf )
# 补充：如果环境里中etcd不是由Kubeadm创建并托管而是外部二进制部署的就不能采用此种方式更新etcd的证书
# 更新指定集群组件进行更新集群
~/.k8s$ kubeadm alpha certs renew
# admin.conf apiserver apiserver-kubelet-client etcd-healthcheck-client etcd-server scheduler.conf
# all apiserver-etcd-client controller-manager.conf etcd-peer front-proxy-client
kubeadm alpha certs renew apiserver --config=./kubeadmin-config.yaml
kubeadm alpha certs renew apiserver-kubelet-client --config=./kubeadmin-config.yaml
kubeadm alpha certs renew front-proxy-client --config=./kubeadmin-config.yaml4.验证刷新证书后的到期时间以及证书文件属性查看
代码语言：javascript复制~/.k8s$ sudo kubeadm alpha certs check-expiration
# [check-expiration] Reading configuration from the cluster...
# [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
# PS -> 显示顺序与上面刷新证书的顺序是一致的
# CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
# admin.conf Dec 12, 2021 09:17 UTC 364d no
# apiserver Dec 12, 2021 09:17 UTC 364d ca no
# apiserver-etcd-client Dec 12, 2021 09:17 UTC 364d etcd-ca no
# apiserver-kubelet-client Dec 12, 2021 09:17 UTC 364d ca no
# controller-manager.conf Dec 12, 2021 09:17 UTC 364d no
# etcd-healthcheck-client Dec 12, 2021 09:17 UTC 364d etcd-ca no
# etcd-peer Dec 12, 2021 09:17 UTC 364d etcd-ca no
# etcd-server Dec 12, 2021 09:17 UTC 364d etcd-ca no
# front-proxy-client Dec 12, 2021 09:17 UTC 364d front-proxy-ca no
# scheduler.conf Dec 12, 2021 09:17 UTC 364d no
# CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
# ca Nov 03, 2030 08:47 UTC 9y no
# etcd-ca Nov 03, 2030 08:47 UTC 9y no
# front-proxy-ca Nov 03, 2030 08:47 UTC 9y no
/etc/kubernetes/pki$ stat apiserver.key
# File: apiserver.key
# Size: 1675 Blocks: 8 IO Block: 4096 regular file
# Device: fd00h/64768d Inode: 2099948 Links: 1
# Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root)
# Access: 2020-12-12 20:30:32.703551276 0800 最近访问：
# Modify: 2020-12-12 17:17:17.055077737 0800 最近更改：
# Change: 2020-12-12 17:17:17.055077737 0800 最近改动：
# Birth: -
/etc/kubernetes/pki$ stat apiserver.crt
# File: apiserver.crt
# Size: 1294 Blocks: 8 IO Block: 4096 regular file
# Device: fd00h/64768d Inode: 2099949 Links: 1
# Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
# Access: 2020-12-12 17:19:54.202163815 0800
# Modify: 2020-12-12 17:17:17.099078602 0800
# Change: 2020-12-12 17:17:17.099078602 0800
# Birth: -5.基于新证书重新生成各个组件的配置文件
代码语言：javascript复制# (1) 主Master节点上运行生成各组件的配置文件
kubeadm alpha kubeconfig user --org system:masters --client-name kubernetes-admin > /etc/kubernetes/admin.conf
kubeadm alpha kubeconfig user --client-name system:kube-controller-manager > /etc/kubernetes/controller-manager.conf
kubeadm alpha kubeconfig user --client-name system:kube-scheduler > /etc/kubernetes/scheduler.conf
# (2) 要开始使用您的集群，您需要运行以下作为一个常规用户:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# (3 )复制证书和配置文件到其他master
scp -r /etc/kubernetes/pki/* root@master@master-01:/etc/kubernetes/pki/
scp /etc/kubernetes/scheduler.conf root@master@master-01:/etc/kubernetes/
scp /etc/kubernetes/controller-manager.conf root@master@master-01:/etc/kubernetes/
scp /etc/kubernetes/admin.conf root@master@master-01:/etc/kubernetes/6.重启kubelet组件,其他master同样重启kubelet和3个组件docker容器
代码语言：javascript复制systemctl restart kubelet
# docker容器重启controller、scheduler、apiserver3个组件
# docker restart该脚本只处理master节点上的证书：kubeadm默认配置了kubelet证书自动更新，node节点kubelet.conf所指向的证书会自动更新
小于v1.17版本的master初始化节点(执行kubeadm init的节点) kubelet.conf里的证书并不会自动更新，这算是一个bug，该脚本会一并处理更新master节点的kubelet.conf所包含的证书
kubeadm生成的证书有效期为为1年，该脚本可将kubeadm生成的证书有效期更新为10年
Kubernetes集群版本在更新时，就会自动更新apiserver.crt证书的使用期限，这可能也是k8s官方设置这一年期限的原因 —— 为了让使用者跟上版本更新的步伐。
证书修改流程 由于k8s是基于kubeadm安装的，所以只需修改kubeadm源码中的证书期限即可。 kubeadm是采用go语言编写，所以我们要现在本机安装go语言环境，然后将对应版本的kubeadm源码进行修改，修改后重新编译，再用新编译的kubeadm生成新证书即可。
Kubeadm/etcd证书修改可用年限
描述: 在Vallidity节点下Kubernetes有两种机制去创建证书，有一部分是1年的(apiServer.key)，有1部分是10年的比如我们的(CA.CERT); Kubernetes集群版本在更新时，就会自动更新apiserver.crt证书的使用期限，这可能也是k8s官方设置这一年期限的原因为了让使用者跟上版本更新的步伐。但是有一些公司就会选取一个稳定版本一直使用下去，这便会使用本节所介绍的内容如何修改证书的使用年限。
操作流程:
代码语言：javascript复制kub~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
> git checkout -b remotes/origin/release-1.19.3 v1.19.3 master [2bd115e3648]
切换到一个新分支 'remotes/origin/release-1.19.3'
weiyigeek@Ubuntu-PC:~/app/program/go/src/k8s.io/kubernetes
> vim staging/src/k8s.io/client-go/util/cert/cert.go remotes/origin/release-1.19.3 [1e11e4a2108]
38 const duration365d = time.Hour * 24 * 365
56 // NewSelfSignedCACert creates a CA certificate
57 func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) {
65 NotBefore: now.UTC(),
66 NotAfter: now.Add(duration365d * 10).UTC(),
86 // GenerateSelfSignedCertKeyWithFixtures creates a self-signed certificate and key for the given host.
94 func GenerateSelfSignedCertKeyWithFixtures(host string, alternateIPs []net.IP, alternateDNS []string, fixtureDirector y string) ([]byte, []byte, error) {
96 maxAge := time.Hour * 24 * 365 * 10 // one year self-signed certs
123 NotBefore: validFrom,
124 NotAfter: validFrom.Add(maxAge),
// NewSignedCert creates a signed certificate using the given CA certificate and key
func NewSignedCert(cfg *CertConfig, key crypto.Signer, caCert *x509.Certificate, caKey crypto.Signer) (*x509.Certificate, error) {
const duration365d = time.Hour * 24 * 365 * 10
591 NotBefore: caCert.NotBefore,
592 NotAfter: time.Now().Add(duration365d).UTC(),
}
结果在这里找到kubeadmconstants.CertificateValidity的定义
vim cmd/kubeadm/app/util/pkiutil/pki_helpers.go
# 这个方法里面看到NotAfter: time.Now().Add(kubeadmconstants.CertificateValidity).UTC()
# 参数里面是一个常量kubeadmconstants.CertificateValidity
# 所以这里可以不修改，我去看看源码能不能找到这个常量的赋值位置
// CertificateValidity defines the validity for all the signed certificates generated by kubeadm
CertificateValidity = time.Hour * 24 * 365 * 10
NotBefore: caCert.NotBefore,
NotAfter: time.Now().Add(kubeadmconstants.CertificateValidity).UTC(),
vim ./cmd/kubeadm/app/constants/constants.go
const (
// KubernetesDir is the directory Kubernetes owns for storing various configuration files
KubernetesDir = "/etc/kubernetes"
// ManifestsSubDirName defines directory name to store manifests
ManifestsSubDirName = "manifests"
// TempDirForKubeadm defines temporary directory for kubeadm
// should be joined with KubernetesDir.
TempDirForKubeadm = "tmp"
// CertificateValidity defines the validity for all the signed certificates generated by kubeadm
CertificateValidity = time.Hour * 24 * 365 * 10
// CACertAndKeyBaseName defines certificate authority base name
CACertAndKeyBaseName = "ca"
// CACertName defines certificate name
CACertName = "ca.crt"
// CAKeyName defines certificate name
CAKeyName = "ca.key"
// APIServerCertAndKeyBaseName defines API's server certificate and key base name
APIServerCertAndKeyBaseName = "apiserver"
// APIServerCertName defines API's server certificate name
APIServerCertName = "apiserver.crt"
// APIServerKeyName defines API's server key name
APIServerKeyName = "apiserver.key"
// APIServerCertCommonName defines API's server certificate common name (CN)
APIServerCertCommonName = "kube-apiserver"
// APIServerKubeletClientCertAndKeyBaseName defines kubelet client certificate and key base name
APIServerKubeletClientCertAndKeyBaseName = "apiserver-kubelet-client"
// APIServerKubeletClientCertName defines kubelet client certificate name
APIServerKubeletClientCertName = "apiserver-kubelet-client.crt"
// APIServerKubeletClientKeyName defines kubelet client key name
APIServerKubeletClientKeyName = "apiserver-kubelet-client.key"
// APIServerKubeletClientCertCommonName defines kubelet client certificate common name (CN)
APIServerKubeletClientCertCommonName = "kube-apiserver-kubelet-client"
// EtcdCACertAndKeyBaseName defines etcd's CA certificate and key base name
EtcdCACertAndKeyBaseName = "etcd/ca"
// EtcdCACertName defines etcd's CA certificate name
EtcdCACertName = "etcd/ca.crt"
// EtcdCAKeyName defines etcd's CA key name
EtcdCAKeyName = "etcd/ca.key"
// EtcdServerCertAndKeyBaseName defines etcd's server certificate and key base name
EtcdServerCertAndKeyBaseName = "etcd/server"
// EtcdServerCertName defines etcd's server certificate name
EtcdServerCertName = "etcd/server.crt"
// EtcdServerKeyName defines etcd's server key name
EtcdServerKeyName = "etcd/server.key"
# 编译kubeadm, 这里主要编译kubeadm 即可
make WHAT=cmd/kubeadm GOFLAGS=-vweiyigeek@master:~$ sudo cp /usr/bin/kubeadm{,.bak}
sudo cp kubeadm /usr/bin/
weiyigeek@master:~$ kubeadm version
kubeadm version: &version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-master 1e11e4a210802", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
weiyigeek@master:~$ kubeadm.bak version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:47:53Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
weiyigeek@master:~/.k8s$ kubeadm alpha certs check-expiration
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Dec 12, 2021 09:17 UTC 364d no
apiserver Dec 12, 2021 09:17 UTC 364d ca no
apiserver-etcd-client Dec 12, 2021 09:17 UTC 364d etcd-ca no
apiserver-kubelet-client Dec 12, 2021 09:17 UTC 364d ca no
controller-manager.conf Dec 12, 2021 09:17 UTC 364d no
etcd-healthcheck-client Dec 12, 2021 09:17 UTC 364d etcd-ca no
etcd-peer Dec 12, 2021 09:17 UTC 364d etcd-ca no
etcd-server Dec 12, 2021 09:17 UTC 364d etcd-ca no
front-proxy-client Dec 12, 2021 09:17 UTC 364d front-proxy-ca no
scheduler.conf Dec 12, 2021 09:17 UTC 364d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Nov 03, 2030 08:47 UTC 9y no
etcd-ca Nov 03, 2030 08:47 UTC 9y no
front-proxy-ca Nov 03, 2030 08:47 UTC 9y no
kubeadm alpha certs renew all --config=/root/kubeadm-config.yaml
weiyigeek@master:~/.k8s$ sudo kubeadm alpha certs renew all --config=kubeadm-init-config.yaml
W1213 13:42:52.803441 2059631 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
weiyigeek@master:~/.k8s$ sudo kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Dec 11, 2030 05:42 UTC 9y no
apiserver Dec 11, 2030 05:42 UTC 9y ca no
apiserver-etcd-client Dec 11, 2030 05:42 UTC 9y etcd-ca no
apiserver-kubelet-client Dec 11, 2030 05:42 UTC 9y ca no
controller-manager.conf Dec 11, 2030 05:42 UTC 9y no
etcd-healthcheck-client Dec 11, 2030 05:42 UTC 9y etcd-ca no
etcd-peer Dec 11, 2030 05:42 UTC 9y etcd-ca no
etcd-server Dec 11, 2030 05:42 UTC 9y etcd-ca no
front-proxy-client Dec 11, 2030 05:42 UTC 9y front-proxy-ca no
scheduler.conf Dec 11, 2030 05:42 UTC 9y no
openssl x509 -in apiserver.crt -text -noout | grep Not
⒎HA集群其余 mater节点证书更新
#!/bin/bash
masterNode="192.168.66.20 192.168.66.21"
#for host in ${masterNode}; do
# scp /etc/kubernetes/pki/{ca.crt,ca.key,sa.key,sa.pub,front-proxy-ca.crt,front-proxy-ca.key}
# "${USER}"@$host:/etc/kubernetes/pki/
# scp /etc/kubernetes/pki/etcd/{ca.crt,ca.key} "root"@$host:/etc/kubernetes/pki/etcd
# scp /etc/kubernetes/admin.conf "root"@$host:/etc/kubernetes/
#done
for host in ${CONTROL_PLANE_IPS}; do
scp /etc/kubernetes/pki/{ca.crt,ca.key,sa.key,sa.pub,front-proxy-ca.crt,front-proxy-ca.key}
"${USER}"@$host:/root/pki/
scp /etc/kubernetes/pki/etcd/{ca.crt,ca.key} "root"@$host:/root/etcd
scp /etc/kubernetes/admin.conf "root"@$host:/root/kubernetes/
done入坑出坑
问题 : Unable to connect to the server: x509: certificate has expired or is not yet valid 故障描述: 使用kubeadm部署的集群，在运行了一年之后今天，出现k8s api无法调取的现象，使用kubectl命令获取资源均返回如下报错
代码语言：javascript复制Unable to connect to the server: x509: certificate has expired or is not yet valid故障原因: 由于默认的apiserver.crt证书有效期只有一年其错误信息表示证书到期无法正常使用；
解决办法: 现在证书过期的问题已经发生了，只能通过更换证书的办法解决，在github上有详细的说明 issue链接如下:
https://github.com/kubernetes/kubeadm/issues/581
