k8s的各个组件与apiServer交互操作各种资源对象，最终都会落入到etcd中。k8s为所有对外提供服务的Restful资源实现了一套通用的符合Restful要求的etcd操作接口，每个服务接口负责处理一类(Kind)资源对象。k8s主要把自己的数据注册在/registry/前缀下面（在ETCD-v3版本后没有了目录的概念，只能一切皆前缀了）。
弄清k8s数据存储的结构，对我们快速掌握k8s和学习k8s源码非常有帮助，那么我们如何k8s的查看etcd中的存储形式呢？
代码语言：javascript复制% kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-558bd4d5db-qc6px 1/1 Running 13 69d
coredns-558bd4d5db-wzcgb 1/1 Running 13 69d
etcd-docker-desktop 1/1 Running 13 69d
kube-apiserver-docker-desktop 1/1 Running 13 69d
kube-controller-manager-docker-desktop 1/1 Running 13 69d
kube-proxy-k6shm 1/1 Running 13 69d
kube-scheduler-docker-desktop 1/1 Running 15 69d
storage-provisioner 1/1 Running 26 69d
vpnkit-controller 1/1 Running 1944 69d我们可以看到，有一个运行etcd的pod：etcd-docker-desktop ，进一步查看下它的详细信息：
代码语言：javascript复制% kubectl describe pod etcd-docker-desktop -n kube-system
Name: etcd-docker-desktop
Namespace: kube-system
Priority: 2000001000
Priority Class Name: system-node-critical
Node: docker-desktop/192.168.65.4
Start Time: Fri, 29 Oct 2021 16:23:09 0800
Labels: component=etcd
tier=control-plane
Annotations: kubeadm.kubernetes.io/etcd.advertise-client-urls: https://192.168.65.4:2379
kubernetes.io/config.hash: 5d9d97b8d8daed31d6fd5c6d386c29c5
kubernetes.io/config.mirror: 5d9d97b8d8daed31d6fd5c6d386c29c5
kubernetes.io/config.seen: 2021-08-23T03:18:46.987692420Z
kubernetes.io/config.source: file
Status: Running
IP: 192.168.65.4
IPs:
IP: 192.168.65.4
Controlled By: Node/docker-desktop
Containers:
etcd:
Container ID: docker://9d2f3cf9d7bc9a933a216f6560eb22a4c62d59555bd449161ce8a58afce29d4e
Image: k8s.gcr.io/etcd:3.4.13-0
Image ID: docker://sha256:05b738aa1bc6355db8a2ee8639f3631b908286e43f584a3d2ee0c472de033c28
Port: <none>
Host Port: <none>
Command:
etcd
--advertise-client-urls=https://192.168.65.4:2379
--cert-file=/run/config/pki/etcd/server.crt
--client-cert-auth=true
--data-dir=/var/lib/etcd
--initial-advertise-peer-urls=https://192.168.65.4:2380
--initial-cluster=docker-desktop=https://192.168.65.4:2380
--key-file=/run/config/pki/etcd/server.key
--listen-client-urls=https://127.0.0.1:2379,https://192.168.65.4:2379
--listen-metrics-urls=http://127.0.0.1:2381
--listen-peer-urls=https://192.168.65.4:2380
--name=docker-desktop
--peer-cert-file=/run/config/pki/etcd/peer.crt
--peer-client-cert-auth=true
--peer-key-file=/run/config/pki/etcd/peer.key
--peer-trusted-ca-file=/run/config/pki/etcd/ca.crt
--snapshot-count=10000
--trusted-ca-file=/run/config/pki/etcd/ca.crt
State: Running
Started: Fri, 29 Oct 2021 16:23:10 0800
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Mon, 25 Oct 2021 09:55:39 0800
Finished: Fri, 29 Oct 2021 16:21:13 0800
Ready: True
Restart Count: 13
Requests:
cpu: 100m
memory: 100Mi
Liveness: http-get http://127.0.0.1:2381/health delay=10s timeout=15s period=10s #success=1 #failure=8
Startup: http-get http://127.0.0.1:2381/health delay=10s timeout=15s period=10s #success=1 #failure=24
Environment: <none>
Mounts:
/run/config/pki/etcd from etcd-certs (rw)
/var/lib/etcd from etcd-data (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
etcd-certs:
Type: HostPath (bare host directory volume)
Path: /run/config/pki/etcd
HostPathType: DirectoryOrCreate
etcd-data:
Type: HostPath (bare host directory volume)
Path: /var/lib/etcd
HostPathType: DirectoryOrCreate
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: :NoExecute op=Exists
Events: <none>可以看到容器ID，我们进入容器内部，用etcd客户端先访问一下试试
代码语言：javascript复制% docker exec -it 05b738aa1bc6 sh%ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/run/config/pki/etcd/ca.crt --key=/run/config/pki/etcd/peer.key --cert=/run/config/pki/etcd/peer.crt get /registry/namespaces --prefix -w=jso可以看到返回的数据
代码语言：javascript复制{"header":{"cluster_id":4202879228857769416,"member_id":16554063148076462710,"revision":1911370,"raft_term":15},"kvs":[{"key":"L3JlZ2lzdHJ5L25hbWVzcGFjZXMvZGVmYXVsdA==","create_revision":208,"mod_revision":208,"version":1,"value":"azhzAAoPCgJ2MRIJTmFtZXNwYWNlEoYCCusBCgdkZWZhdWx0EgAaACIAKiQzYjg0ZWMzMy02MzQ2LTRiMjgtOTI2MS0yNzU3ZWYzNWM4NWIyADgAQggIo6eMiQYQAFomChtrdWJlcm5ldGVzLmlvL21ldGFkYXRhLm5hbWUSB2RlZmF1bHR6AIoBewoOa3ViZS1hcGlzZXJ2ZXISBlVwZGF0ZRoCdjEiCAijp4yJBhAAMghGaWVsZHNWMTpJCkd7ImY6bWV0YWRhdGEiOnsiZjpsYWJlbHMiOnsiLiI6e30sImY6a3ViZXJuZXRlcy5pby9tZXRhZGF0YS5uYW1lIjp7fX19fRIMCgprdWJlcm5ldGVzGggKBkFjdGl2ZRoAIgA="},{"key":"L3JlZ2lzdHJ5L25hbWVzcGFjZXMva3ViZS1ub2RlLWxlYXNl","create_revision":52,"mod_revision":52,"version":1,"value":"azhzAAoPCgJ2MRIJTmFtZXNwYWNlEpYCCvsBCg9rdWJlLW5vZGUtbGVhc2USABoAIgAqJDQ5OTMxMzYxLTkzMzYtNGIwZC04MDQxLWQ3YzRiZDc2ZmIyNjIAOABCCAihp4yJBhAAWi4KG2t1YmVybmV0ZXMuaW8vbWV0YWRhdGEubmFtZRIPa3ViZS1ub2RlLWxlYXNlegCKAXsKDmt1YmUtYXBpc2VydmVyEgZVcGRhdGUaAnYxIggIoaeMiQYQADIIRmllbGRzVjE6SQpHeyJmOm1ldGFkYXRhIjp7ImY6bGFiZWxzIjp7Ii4iOnt9LCJmOmt1YmVybmV0ZXMuaW8vbWV0YWRhdGEubmFtZSI6e319fX0SDAoKa3ViZXJuZXRlcxoICgZBY3RpdmUaACIA"},{"key":"L3JlZ2lzdHJ5L25hbWVzcGFjZXMva3ViZS1wdWJsaWM=","create_revision":45,"mod_revision":45,"version":1,"value":"azhzAAoPCgJ2MRIJTmFtZXNwYWNlEo4CCvMBCgtrdWJlLXB1YmxpYxIAGgAiACokNTIxM2Q4NzAtNWRmNC00MzhlLWI1ZDgtMzc4OTA0ZmI5ZGQ5MgA4AEIICKGnjIkGEABaKgoba3ViZXJuZXRlcy5pby9tZXRhZGF0YS5uYW1lEgtrdWJlLXB1YmxpY3oAigF7Cg5rdWJlLWFwaXNlcnZlchIGVXBkYXRlGgJ2MSIICKGnjIkGEAAyCEZpZWxkc1YxOkkKR3siZjptZXRhZGF0YSI6eyJmOmxhYmVscyI6eyIuIjp7fSwiZjprdWJlcm5ldGVzLmlvL21ldGFkYXRhLm5hbWUiOnt9fX19EgwKCmt1YmVybmV0ZXMaCAoGQWN0aXZlGgAiAA=="},{"key":"L3JlZ2lzdHJ5L25hbWVzcGFjZXMva3ViZS1zeXN0ZW0=","create_revision":13,"mod_revision":13,"version":1,"value":"azhzAAoPCgJ2MRIJTmFtZXNwYWNlEo4CCvMBCgtrdWJlLXN5c3RlbRIAGgAiACokZWJhZDc5YzMtNmI3OC00OGQwLWI1ZGItZmI4YzNiYjk1MWVhMgA4AEIICKGnjIkGEABaKgoba3ViZXJuZXRlcy5pby9tZXRhZGF0YS5uYW1lEgtrdWJlLXN5c3RlbXoAigF7Cg5rdWJlLWFwaXNlcnZlchIGVXBkYXRlGgJ2MSIICKGnjIkGEAAyCEZpZWxkc1YxOkkKR3siZjptZXRhZGF0YSI6eyJmOmxhYmVscyI6eyIuIjp7fSwiZjprdWJlcm5ldGVzLmlvL21ldGFkYXRhLm5hbWUiOnt9fX19EgwKCmt1YmVybmV0ZXMaCAoGQWN0aXZlGgAiAA=="}],"count":4}我们可以看下全部的key
代码语言：javascript复制% ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/run/config/pki/etcd/ca.crt --key=/run/config/pki/etcd/peer.key --cert=/run/config/pki/etcd/peer.crt get "" --prefix --keys-only |grep -Ev "^$"
/registry/apiregistration.k8s.io/apiservices/v1.
/registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.apps
/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.autoscaling
/registry/apiregistration.k8s.io/apiservices/v1.batch
/registry/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.events.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.networking.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.node.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.policy
/registry/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.storage.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.apiextensions.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.authentication.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.batch
/registry/apiregistration.k8s.io/apiservices/v1beta1.certificates.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.coordination.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.discovery.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.events.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.extensions
/registry/apiregistration.k8s.io/apiservices/v1beta1.flowcontrol.apiserver.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.networking.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.node.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.policy
/registry/apiregistration.k8s.io/apiservices/v1beta1.rbac.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.scheduling.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io
/registry/apiregistration.k8s.io/apiservices/v2beta1.autoscaling
/registry/apiregistration.k8s.io/apiservices/v2beta2.autoscaling
/registry/clusterrolebindings/cluster-admin
/registry/clusterrolebindings/docker-for-desktop-binding
/registry/clusterrolebindings/ingress-nginx
/registry/clusterrolebindings/ingress-nginx-admission
/registry/clusterrolebindings/kubeadm:get-nodes
/registry/clusterrolebindings/kubeadm:kubelet-bootstrap
/registry/clusterrolebindings/kubeadm:node-autoapprove-bootstrap
/registry/clusterrolebindings/kubeadm:node-autoapprove-certificate-rotation
/registry/clusterrolebindings/kubeadm:node-proxier
/registry/clusterrolebindings/storage-provisioner
/registry/clusterrolebindings/system:basic-user
/registry/clusterrolebindings/system:controller:attachdetach-controller
/registry/clusterrolebindings/system:controller:certificate-controller
/registry/clusterrolebindings/system:controller:clusterrole-aggregation-controller
/registry/clusterrolebindings/system:controller:cronjob-controller
/registry/clusterrolebindings/system:controller:daemon-set-controller
/registry/clusterrolebindings/system:controller:deployment-controller
/registry/clusterrolebindings/system:controller:disruption-controller
/registry/clusterrolebindings/system:controller:endpoint-controller
/registry/clusterrolebindings/system:controller:endpointslice-controller
/registry/clusterrolebindings/system:controller:endpointslicemirroring-controller
/registry/clusterrolebindings/system:controller:ephemeral-volume-controller
/registry/clusterrolebindings/system:controller:expand-controller
/registry/clusterrolebindings/system:controller:generic-garbage-collector
/registry/clusterrolebindings/system:controller:horizontal-pod-autoscaler
/registry/clusterrolebindings/system:controller:job-controller
/registry/clusterrolebindings/system:controller:namespace-controller
/registry/clusterrolebindings/system:controller:node-controller
/registry/clusterrolebindings/system:controller:persistent-volume-binder
/registry/clusterrolebindings/system:controller:pod-garbage-collector
/registry/clusterrolebindings/system:controller:pv-protection-controller
/registry/clusterrolebindings/system:controller:pvc-protection-controller
/registry/clusterrolebindings/system:controller:replicaset-controller
/registry/clusterrolebindings/system:controller:replication-controller
/registry/clusterrolebindings/system:controller:resourcequota-controller
/registry/clusterrolebindings/system:controller:root-ca-cert-publisher
/registry/clusterrolebindings/system:controller:route-controller
/registry/clusterrolebindings/system:controller:service-account-controller
/registry/clusterrolebindings/system:controller:service-controller
/registry/clusterrolebindings/system:controller:statefulset-controller
/registry/clusterrolebindings/system:controller:ttl-after-finished-controller
/registry/clusterrolebindings/system:controller:ttl-controller
/registry/clusterrolebindings/system:coredns
/registry/clusterrolebindings/system:discovery
/registry/clusterrolebindings/system:kube-controller-manager
/registry/clusterrolebindings/system:kube-dns
/registry/clusterrolebindings/system:kube-scheduler
/registry/clusterrolebindings/system:monitoring
/registry/clusterrolebindings/system:node
/registry/clusterrolebindings/system:node-proxier
/registry/clusterrolebindings/system:public-info-viewer
/registry/clusterrolebindings/system:service-account-issuer-discovery
/registry/clusterrolebindings/system:volume-scheduler
/registry/clusterrolebindings/vpnkit-controller
/registry/clusterroles/admin
/registry/clusterroles/cluster-admin
/registry/clusterroles/edit
/registry/clusterroles/ingress-nginx
/registry/clusterroles/ingress-nginx-admission
/registry/clusterroles/kubeadm:get-nodes
/registry/clusterroles/secret-reader
/registry/clusterroles/system:aggregate-to-admin
/registry/clusterroles/system:aggregate-to-edit
/registry/clusterroles/system:aggregate-to-view
/registry/clusterroles/system:auth-delegator
/registry/clusterroles/system:basic-user
/registry/clusterroles/system:certificates.k8s.io:certificatesigningrequests:nodeclient
/registry/clusterroles/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
/registry/clusterroles/system:certificates.k8s.io:kube-apiserver-client-approver
/registry/clusterroles/system:certificates.k8s.io:kube-apiserver-client-kubelet-approver
/registry/clusterroles/system:certificates.k8s.io:kubelet-serving-approver
/registry/clusterroles/system:certificates.k8s.io:legacy-unknown-approver
/registry/clusterroles/system:controller:attachdetach-controller
/registry/clusterroles/system:controller:certificate-controller
/registry/clusterroles/system:controller:clusterrole-aggregation-controller
/registry/clusterroles/system:controller:cronjob-controller
/registry/clusterroles/system:controller:daemon-set-controller
/registry/clusterroles/system:controller:deployment-controller
/registry/clusterroles/system:controller:disruption-controller
/registry/clusterroles/system:controller:endpoint-controller
/registry/clusterroles/system:controller:endpointslice-controller
/registry/clusterroles/system:controller:endpointslicemirroring-controller
/registry/clusterroles/system:controller:ephemeral-volume-controller
/registry/clusterroles/system:controller:expand-controller
/registry/clusterroles/system:controller:generic-garbage-collector
/registry/clusterroles/system:controller:horizontal-pod-autoscaler
/registry/clusterroles/system:controller:job-controller
/registry/clusterroles/system:controller:namespace-controller
/registry/clusterroles/system:controller:node-controller
/registry/clusterroles/system:controller:persistent-volume-binder
/registry/clusterroles/system:controller:pod-garbage-collector
/registry/clusterroles/system:controller:pv-protection-controller
/registry/clusterroles/system:controller:pvc-protection-controller
/registry/clusterroles/system:controller:replicaset-controller
/registry/clusterroles/system:controller:replication-controller
/registry/clusterroles/system:controller:resourcequota-controller
/registry/clusterroles/system:controller:root-ca-cert-publisher
/registry/clusterroles/system:controller:route-controller
/registry/clusterroles/system:controller:service-account-controller
/registry/clusterroles/system:controller:service-controller
/registry/clusterroles/system:controller:statefulset-controller
/registry/clusterroles/system:controller:ttl-after-finished-controller
/registry/clusterroles/system:controller:ttl-controller
/registry/clusterroles/system:coredns
/registry/clusterroles/system:discovery
/registry/clusterroles/system:heapster
/registry/clusterroles/system:kube-aggregator
/registry/clusterroles/system:kube-controller-manager
/registry/clusterroles/system:kube-dns
/registry/clusterroles/system:kube-scheduler
/registry/clusterroles/system:kubelet-api-admin
/registry/clusterroles/system:monitoring
/registry/clusterroles/system:node
/registry/clusterroles/system:node-bootstrapper
/registry/clusterroles/system:node-problem-detector
/registry/clusterroles/system:node-proxier
/registry/clusterroles/system:persistent-volume-provisioner
/registry/clusterroles/system:public-info-viewer
/registry/clusterroles/system:service-account-issuer-discovery
/registry/clusterroles/system:volume-scheduler
/registry/clusterroles/view
/registry/clusterroles/vpnkit-controller
/registry/configmaps/default/ingress-controller-leader-nginx
/registry/configmaps/default/ingress-nginx-controller
/registry/configmaps/default/kube-root-ca.crt
/registry/configmaps/default/redis-config
/registry/configmaps/kube-node-lease/kube-root-ca.crt
/registry/configmaps/kube-public/cluster-info
/registry/configmaps/kube-public/kube-root-ca.crt
/registry/configmaps/kube-system/coredns
/registry/configmaps/kube-system/extension-apiserver-authentication
/registry/configmaps/kube-system/kube-proxy
/registry/configmaps/kube-system/kube-root-ca.crt
/registry/configmaps/kube-system/kubeadm-config
/registry/configmaps/kube-system/kubelet-config-1.21
/registry/controllerrevisions/kube-system/kube-proxy-7cdcb64568
/registry/csinodes/docker-desktop
/registry/daemonsets/kube-system/kube-proxy
/registry/deployments/default/ingress-nginx-controller
/registry/deployments/default/minio-deployment
/registry/deployments/default/redis
/registry/deployments/kube-system/coredns
/registry/endpointslices/default/apple-service-5f9zk
/registry/endpointslices/default/ingress-nginx-controller-admission-shdx6
/registry/endpointslices/default/ingress-nginx-controller-jjm9l
/registry/endpointslices/default/kubernetes
/registry/endpointslices/default/minio-service-fd4dl
/registry/endpointslices/default/redis-gztm7
/registry/endpointslices/kube-system/kube-dns-wqqx2
/registry/events/kube-system/vpnkit-controller.16b272c39baa616d
/registry/events/kube-system/vpnkit-controller.16b272c39cb40bae
/registry/events/kube-system/vpnkit-controller.16b272c3a1d78e07
/registry/events/kube-system/vpnkit-controller.16b2742b54d7628e
/registry/flowschemas/catch-all
/registry/flowschemas/exempt
/registry/flowschemas/global-default
/registry/flowschemas/kube-controller-manager
/registry/flowschemas/kube-scheduler
/registry/flowschemas/kube-system-service-accounts
/registry/flowschemas/probes
/registry/flowschemas/service-accounts
/registry/flowschemas/system-leader-election
/registry/flowschemas/system-nodes
/registry/flowschemas/workload-leader-election
/registry/ingress/default/ingress-with-auth
/registry/jobs/default/ingress-nginx-admission-create
/registry/jobs/default/ingress-nginx-admission-patch
/registry/leases/kube-node-lease/docker-desktop
/registry/leases/kube-system/kube-scheduler
/registry/masterleases/192.168.65.4
/registry/minions/docker-desktop
/registry/namespaces/default
/registry/namespaces/kube-node-lease
/registry/namespaces/kube-public
/registry/namespaces/kube-system
/registry/persistentvolumeclaims/default/minio-pv-claim
/registry/persistentvolumeclaims/default/redis
/registry/persistentvolumes/minio
/registry/persistentvolumes/redis
/registry/pods/default/apple-app
/registry/pods/default/ingress-nginx-admission-create-vjn92
/registry/pods/default/ingress-nginx-admission-patch-wlq6p
/registry/pods/default/ingress-nginx-controller-57648496fc-84wl8
/registry/pods/default/minio-deployment-55bf5bff5d-cvq7v
/registry/pods/default/redis-f9f74787-tq6tw
/registry/pods/kube-system/coredns-558bd4d5db-qc6px
/registry/pods/kube-system/coredns-558bd4d5db-wzcgb
/registry/pods/kube-system/etcd-docker-desktop
/registry/pods/kube-system/kube-apiserver-docker-desktop
/registry/pods/kube-system/kube-controller-manager-docker-desktop
/registry/pods/kube-system/kube-proxy-k6shm
/registry/pods/kube-system/kube-scheduler-docker-desktop
/registry/pods/kube-system/storage-provisioner
/registry/pods/kube-system/vpnkit-controller
/registry/priorityclasses/system-cluster-critical
/registry/priorityclasses/system-node-critical
/registry/prioritylevelconfigurations/catch-all
/registry/prioritylevelconfigurations/exempt
/registry/prioritylevelconfigurations/global-default
/registry/prioritylevelconfigurations/leader-election
/registry/prioritylevelconfigurations/system
/registry/prioritylevelconfigurations/workload-high
/registry/prioritylevelconfigurations/workload-low
/registry/ranges/serviceips
/registry/ranges/servicenodeports
/registry/replicasets/default/ingress-nginx-controller-57648496fc
/registry/replicasets/default/minio-deployment-55bf5bff5d
/registry/replicasets/default/minio-deployment-6cfc69548
/registry/replicasets/default/minio-deployment-857cc8bbdf
/registry/replicasets/default/redis-f9f74787
/registry/replicasets/kube-system/coredns-558bd4d5db
/registry/rolebindings/default/ingress-nginx
/registry/rolebindings/default/ingress-nginx-admission
/registry/rolebindings/default/read-pods
/registry/rolebindings/kube-public/kubeadm:bootstrap-signer-clusterinfo
/registry/rolebindings/kube-public/system:controller:bootstrap-signer
/registry/rolebindings/kube-system/kube-proxy
/registry/rolebindings/kube-system/kubeadm:kubelet-config-1.21
/registry/rolebindings/kube-system/kubeadm:nodes-kubeadm-config
/registry/rolebindings/kube-system/system::extension-apiserver-authentication-reader
/registry/rolebindings/kube-system/system::leader-locking-kube-controller-manager
/registry/rolebindings/kube-system/system::leader-locking-kube-scheduler
/registry/rolebindings/kube-system/system:controller:bootstrap-signer
/registry/rolebindings/kube-system/system:controller:cloud-provider
/registry/rolebindings/kube-system/system:controller:token-cleaner
/registry/roles/default/ingress-nginx
/registry/roles/default/ingress-nginx-admission
/registry/roles/default/pod-reader
/registry/roles/kube-public/kubeadm:bootstrap-signer-clusterinfo
/registry/roles/kube-public/system:controller:bootstrap-signer
/registry/roles/kube-system/extension-apiserver-authentication-reader
/registry/roles/kube-system/kube-proxy
/registry/roles/kube-system/kubeadm:kubelet-config-1.21
/registry/roles/kube-system/kubeadm:nodes-kubeadm-config
/registry/roles/kube-system/system::leader-locking-kube-controller-manager
/registry/roles/kube-system/system::leader-locking-kube-scheduler
/registry/roles/kube-system/system:controller:bootstrap-signer
/registry/roles/kube-system/system:controller:cloud-provider
/registry/roles/kube-system/system:controller:token-cleaner
/registry/secrets/default/basic-auth
/registry/secrets/default/default-token-cg2vq
/registry/secrets/default/ingress-nginx-admission
/registry/secrets/default/ingress-nginx-admission-token-xblnc
/registry/secrets/default/ingress-nginx-token-7clh8
/registry/secrets/default/tls-secret
/registry/secrets/kube-node-lease/default-token-q97sv
/registry/secrets/kube-public/default-token-jq4g9
/registry/secrets/kube-system/attachdetach-controller-token-6lg9w
/registry/secrets/kube-system/bootstrap-signer-token-nnrn5
/registry/secrets/kube-system/certificate-controller-token-mwsrf
/registry/secrets/kube-system/clusterrole-aggregation-controller-token-27mp8
/registry/secrets/kube-system/coredns-token-5r4fz
/registry/secrets/kube-system/cronjob-controller-token-bvgp6
/registry/secrets/kube-system/daemon-set-controller-token-xx2z8
/registry/secrets/kube-system/default-token-pbdlw
/registry/secrets/kube-system/deployment-controller-token-8j72x
/registry/secrets/kube-system/disruption-controller-token-w9mmh
/registry/secrets/kube-system/endpoint-controller-token-t8dwq
/registry/secrets/kube-system/endpointslice-controller-token-dnbfm
/registry/secrets/kube-system/endpointslicemirroring-controller-token-plknt
/registry/secrets/kube-system/ephemeral-volume-controller-token-xm679
/registry/secrets/kube-system/expand-controller-token-9wgwf
/registry/secrets/kube-system/generic-garbage-collector-token-smjfw
/registry/secrets/kube-system/horizontal-pod-autoscaler-token-6jsj2
/registry/secrets/kube-system/job-controller-token-cft9x
/registry/secrets/kube-system/kube-proxy-token-c9bth
/registry/secrets/kube-system/namespace-controller-token-6rtph
/registry/secrets/kube-system/node-controller-token-ztkld
/registry/secrets/kube-system/persistent-volume-binder-token-zntw5
/registry/secrets/kube-system/pod-garbage-collector-token-qt297
/registry/secrets/kube-system/pv-protection-controller-token-fq7jx
/registry/secrets/kube-system/pvc-protection-controller-token-4krd7
/registry/secrets/kube-system/replicaset-controller-token-jjtjx
/registry/secrets/kube-system/replication-controller-token-kxdqb
/registry/secrets/kube-system/resourcequota-controller-token-n9zqv
/registry/secrets/kube-system/root-ca-cert-publisher-token-wldpn
/registry/secrets/kube-system/service-account-controller-token-wzgng
/registry/secrets/kube-system/service-controller-token-djmtc
/registry/secrets/kube-system/statefulset-controller-token-96msw
/registry/secrets/kube-system/storage-provisioner-token-n575g
/registry/secrets/kube-system/token-cleaner-token-vqxd2
/registry/secrets/kube-system/ttl-after-finished-controller-token-lmxng
/registry/secrets/kube-system/ttl-controller-token-xg5kg
/registry/secrets/kube-system/vpnkit-controller-token-8ztzx
/registry/serviceaccounts/default/default
/registry/serviceaccounts/default/ingress-nginx
/registry/serviceaccounts/default/ingress-nginx-admission
/registry/serviceaccounts/kube-node-lease/default
/registry/serviceaccounts/kube-public/default
/registry/serviceaccounts/kube-system/attachdetach-controller
/registry/serviceaccounts/kube-system/bootstrap-signer
/registry/serviceaccounts/kube-system/certificate-controller
/registry/serviceaccounts/kube-system/clusterrole-aggregation-controller
/registry/serviceaccounts/kube-system/coredns
/registry/serviceaccounts/kube-system/cronjob-controller
/registry/serviceaccounts/kube-system/daemon-set-controller
/registry/serviceaccounts/kube-system/default
/registry/serviceaccounts/kube-system/deployment-controller
/registry/serviceaccounts/kube-system/disruption-controller
/registry/serviceaccounts/kube-system/endpoint-controller
/registry/serviceaccounts/kube-system/endpointslice-controller
/registry/serviceaccounts/kube-system/endpointslicemirroring-controller
/registry/serviceaccounts/kube-system/ephemeral-volume-controller
/registry/serviceaccounts/kube-system/expand-controller
/registry/serviceaccounts/kube-system/generic-garbage-collector
/registry/serviceaccounts/kube-system/horizontal-pod-autoscaler
/registry/serviceaccounts/kube-system/job-controller
/registry/serviceaccounts/kube-system/kube-proxy
/registry/serviceaccounts/kube-system/namespace-controller
/registry/serviceaccounts/kube-system/node-controller
/registry/serviceaccounts/kube-system/persistent-volume-binder
/registry/serviceaccounts/kube-system/pod-garbage-collector
/registry/serviceaccounts/kube-system/pv-protection-controller
/registry/serviceaccounts/kube-system/pvc-protection-controller
/registry/serviceaccounts/kube-system/replicaset-controller
/registry/serviceaccounts/kube-system/replication-controller
/registry/serviceaccounts/kube-system/resourcequota-controller
/registry/serviceaccounts/kube-system/root-ca-cert-publisher
/registry/serviceaccounts/kube-system/service-account-controller
/registry/serviceaccounts/kube-system/service-controller
/registry/serviceaccounts/kube-system/statefulset-controller
/registry/serviceaccounts/kube-system/storage-provisioner
/registry/serviceaccounts/kube-system/token-cleaner
/registry/serviceaccounts/kube-system/ttl-after-finished-controller
/registry/serviceaccounts/kube-system/ttl-controller
/registry/serviceaccounts/kube-system/vpnkit-controller
/registry/services/endpoints/default/apple-service
/registry/services/endpoints/default/ingress-nginx-controller
/registry/services/endpoints/default/ingress-nginx-controller-admission
/registry/services/endpoints/default/kubernetes
/registry/services/endpoints/default/minio-service
/registry/services/endpoints/default/redis
/registry/services/endpoints/kube-system/docker.io-hostpath
/registry/services/endpoints/kube-system/kube-dns
/registry/services/specs/default/apple-service
/registry/services/specs/default/ingress-nginx-controller
/registry/services/specs/default/ingress-nginx-controller-admission
/registry/services/specs/default/kubernetes
/registry/services/specs/default/minio-service
/registry/services/specs/default/redis
/registry/services/specs/kube-system/kube-dns
/registry/storageclasses/hostpath
/registry/validatingwebhookconfigurations/ingress-nginx-admission
compact_rev_key通过观察k8s中deployment、namespace、pod等在ETCD中的表示，可以知道这部分资源的key的格式为/registry/#{k8s对象}/#{命名空间}/#{具体实例名}。存在一个与众不同的key值compact_rev_key，搜索可以知道这是apiserver/compact.go中用来记录无效数据版本使用的；运行etcdctl get compact_rev_key可以发现，输出的是一个整形数值。在查看ETCD时，k8s中除了必要的网络插件canal，未部署其他的应用，此时ETCD中只有240条数据。
我们还可以看下某个key里面存储的内容
代码语言：javascript复制% ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/run/config/pki/etcd/ca.crt --key=/run/config/pki/etcd/peer.key --cert=/run/config/pki/etcd/peer.crt get /registry/ranges/serviceips
/registry/ranges/serviceips
k8s
v1RangeAllocation�
"*28Bz
10.96.0.0/12�"可以很明显看出来，ETCD中保存的并不是输出友好的数据（比如json或xml等就是输出友好型数据）。当然，如果进一步研究可以知道，ETCD保存的是Protocol Buffers序列化后的值。如果像看到json格式的数据可以在后面加上命令
代码语言：javascript复制 -w=json我们需要进入docker内部去查看etcd存储的数据，显然是很不方便的，我们有没有办法在宿主机上看etcd的数据呢？答案是有，首先我们硬访问下，试试：
代码语言：javascript复制% ETCDCTL_API=3 /Users/xiazemin/source/etcd/bin/etcdctl --endpoints=https://127.0.0.1:2379
--cacert=/Users/xiazemin/Library/Group Containers/group.com.docker/pki/etcd/ca.crt
--key=/Users/xiazemin/Library/Group Containers/group.com.docker/pki/etcd/peer.key
--cert=/Users/xiazemin/Library/Group Containers/group.com.docker/pki/etcd/peer.crt
get /registry/namespaces --prefix -w=json
{"level":"warn","ts":"2021-11-01T11:59:31.114 0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x1400039c540/#initially=[https://127.0.0.1:2379]","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused""}
Error: context deadline exceeded连接超时，显然网络连不上。
代码语言：javascript复制kubectl get svc -n kube-system 我们发现没有etcd相关的svc，在k8s中，如果我们想访问某些服务，我们需要通过svc（服务发现），把pod的端口暴露出来，如果我们通过svc把etcd的pod端口暴露出来，我们是不是可以在宿主机上访问了？
创建一个service，通过NodePort的方式做端口转发：
代码语言：javascript复制apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2021-11-04T05:30:37Z"
labels:
component: etcd
tier: control-plane
name: etcd-svc-docker-desktop-xzm
namespace: kube-system
resourceVersion: "2055694"
uid: 9cc9abbf-2edc-44ba-b359-157a8b643c62
spec:
clusterIP: 10.111.136.178
clusterIPs:
- 10.111.136.178
externalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- nodePort: 32379
port: 2379
protocol: TCP
targetPort: 2379
selector:
component: etcd
tier: control-plane
sessionAffinity: None
type: NodePort
status:
loadBalancer:
ingress:
- hostname: localhost% kubectl apply -f etcd/etcd-svc-docker-desktop-xzm.svc.yaml
service/etcd-svc-docker-desktop-xzm created查看下我们的service
代码语言：javascript复制 % kubectl get svc etcd-svc-docker-desktop-xzm -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
etcd-svc-docker-desktop-xzm NodePort 10.105.187.254 <none> 2379:32379/TCP 35s发现端口已经转发了，接着我们尝试下是否连接成功，发现仍然超时，难道我们找的证书位置不对？把pod里面的证书copy出来对比下：
代码语言：javascript复制% docker cp 9d2f3cf9d7bc9a933a216f6560eb22a4c62d59555bd449161ce8a58afce29d4e:/run/config/pki/etcd/peer.key etcd vimdiff ./etcd/peer.key /Users/xiazemin/Library/Group Containers/group.com.docker/pki/etcd/peer.key发现完全没有diff，说明问题不在这里。是不是端口冲突了？
代码语言：javascript复制 % lsof |grep 32379
com.docke 71118 xiazemin 69u IPv6 0xb7573feed65f41d 0t0 TCP localhost:32379->localhost:62688 (CLOSE_WAIT)
com.docke 71118 xiazemin 89u IPv6 0xb7573fef567a0bd 0t0 TCP *:32379 (LISTEN)果然如此，我们该下转发的端口到32389，再试下？
代码语言：javascript复制
% lsof |grep 32389
% ETCDCTL_API=3 etcdctl --cacert=./etcd/ca.crt
--cert=./etcd/peer.crt
--key=./etcd/peer.key
--endpoints=https://127.0.0.1:32389
get /registry/namespaces --prefix -w=json
{"header":{"cluster_id":4202879228857769416,"member_id":16554063148076462710,"revision":2056701,"raft_term":18},"kvs":[{"key":"L3JlZ2lzdHJ5L25hbWVzcGFjZXMvZGVmYXVsdA==","create_revision":208,"mod_revision":208,"version":1,"value":"azhzAAoPCgJ2MRIJTmFtZXNwYWNlEoYCCusBCgdkZWZhdWx0EgAaACIAKiQzYjg0ZWMzMy02MzQ2LTRiMjgtOTI2MS0yNzU3ZWYzNWM4NWIyADgAQggIo6eMiQYQAFomChtrdWJlcm5ldGVzLmlvL21ldGFkYXRhLm5hbWUSB2RlZmF1bHR6AIoBewoOa3ViZS1hcGlzZXJ2ZXISBlVwZGF0ZRoCdjEiCAijp4yJBhAAMghGaWVsZHNWMTpJCkd7ImY6bWV0YWRhdGEiOnsiZjpsYWJlbHMiOnsiLiI6e30sImY6a3ViZXJuZXRlcy5pby9tZXRhZGF0YS5uYW1lIjp7fX19fRIMCgprdWJlcm5ldGVzGggKBkFjdGl2ZRoAIgA="},{"key":"L3JlZ2lzdHJ5L25hbWVzcGFjZXMva3ViZS1ub2RlLWxlYXNl","create_revision":52,"mod_revision":52,"version":1,"value":"azhzAAoPCgJ2MRIJTmFtZXNwYWNlEpYCCvsBCg9rdWJlLW5vZGUtbGVhc2USABoAIgAqJDQ5OTMxMzYxLTkzMzYtNGIwZC04MDQxLWQ3YzRiZDc2ZmIyNjIAOABCCAihp4yJBhAAWi4KG2t1YmVybmV0ZXMuaW8vbWV0YWRhdGEubmFtZRIPa3ViZS1ub2RlLWxlYXNlegCKAXsKDmt1YmUtYXBpc2VydmVyEgZVcGRhdGUaAnYxIggIoaeMiQYQADIIRmllbGRzVjE6SQpHeyJmOm1ldGFkYXRhIjp7ImY6bGFiZWxzIjp7Ii4iOnt9LCJmOmt1YmVybmV0ZXMuaW8vbWV0YWRhdGEubmFtZSI6e319fX0SDAoKa3ViZXJuZXRlcxoICgZBY3RpdmUaACIA"},{"key":"L3JlZ2lzdHJ5L25hbWVzcGFjZXMva3ViZS1wdWJsaWM=","create_revision":45,"mod_revision":45,"version":1,"value":"azhzAAoPCgJ2MRIJTmFtZXNwYWNlEo4CCvMBCgtrdWJlLXB1YmxpYxIAGgAiACokNTIxM2Q4NzAtNWRmNC00MzhlLWI1ZDgtMzc4OTA0ZmI5ZGQ5MgA4AEIICKGnjIkGEABaKgoba3ViZXJuZXRlcy5pby9tZXRhZGF0YS5uYW1lEgtrdWJlLXB1YmxpY3oAigF7Cg5rdWJlLWFwaXNlcnZlchIGVXBkYXRlGgJ2MSIICKGnjIkGEAAyCEZpZWxkc1YxOkkKR3siZjptZXRhZGF0YSI6eyJmOmxhYmVscyI6eyIuIjp7fSwiZjprdWJlcm5ldGVzLmlvL21ldGFkYXRhLm5hbWUiOnt9fX19EgwKCmt1YmVybmV0ZXMaCAoGQWN0aXZlGgAiAA=="},{"key":"L3JlZ2lzdHJ5L25hbWVzcGFjZXMva3ViZS1zeXN0ZW0=","create_revision":13,"mod_revision":13,"version":1,"value":"azhzAAoPCgJ2MRIJTmFtZXNwYWNlEo4CCvMBCgtrdWJlLXN5c3RlbRIAGgAiACokZWJhZDc5YzMtNmI3OC00OGQwLWI1ZGItZmI4YzNiYjk1MWVhMgA4AEIICKGnjIkGEABaKgoba3ViZXJuZXRlcy5pby9tZXRhZGF0YS5uYW1lEgtrdWJlLXN5c3RlbXoAigF7Cg5rdWJlLWFwaXNlcnZlchIGVXBkYXRlGgJ2MSIICKGnjIkGEAAyCEZpZWxkc1YxOkkKR3siZjptZXRhZGF0YSI6eyJmOmxhYmVscyI6eyIuIjp7fSwiZjprdWJlcm5ldGVzLmlvL21ldGFkYXRhLm5hbWUiOnt9fX19EgwKCmt1YmVybmV0ZXMaCAoGQWN0aXZlGgAiAA=="}],"count":4}果然成功了.从此我们就可以在pod外面查看etcd里面的存储内容了。
总结：在掌握了k8s svc的基础原理后，我们稍微联想发散一下，就可以解决很多我们看上去很茫然，很陌生的问题。很多看上去很复杂的事务的外衣下都有一个简单的内核。
