在不同应用业务环境下,对于平台可能有一些特殊的需求,这些需求可以抽象为 Kubernetes 的扩展资源,而 Kubernetes 的 CRD (CustomResourceDefinition)为这样的需求提供了轻量级的机制,保证新的资源的快速注册和使用。
我们希望在执行 kubectl create -f kafka-source.yaml 之后,在 kubernetes 里会启动一个 pod,这个 pod 会做下面的事情:它会从地址是 my-cluster-kafka-bootstrap.kafka:9092,topic 是 knative- demo-topic 的 kafka 机群里读取消;将读到的消息,发送到 kubernetes 的一个 Service 去执行;Kuberentes 里并没有 KafkaSource 这个资源可以使用,所以直接执行 kubectl create -f kafka-source.yaml 的时候,会出错。但是 kubernetes 提供的 CRD 机制可以让我们轻松的把上述功能添加到 kubernetes 里。
CRD 机制以上述 Kafkasource 为例,如下:(1)需要把 KafkaSource 这个资源注册到 kubernetes 中,这样 kubernetes 才会知道这个资源(2)注册之后,还需要开发一个 controller 组件,来监听用户是否创建了 KafkaSource,也就是部署、更新或者删除如上的 yaml 文件。(3)Controller 监听到有用户创建了 KafkaSource,就会创建一个 pod 来做相应的工作
归纳一下就是:用户向 Kubernetes API 服务注册一个带特定 schema 的资源,并定义相关 API;将扩展资源的数据存储到 Kubernetes 的 etcd 集群;借助 Kubernetes 提供的 controller 模式开发框架,实现新的 controller,并借助 APIServer 监听 etcd 集群关于该资源的状态并定义状态变化的处理逻辑.informer 会借助 APIServer 跟踪该扩展资源定义的变化,一旦被触发就会调用回调函数,并把变更的具体内容放到 Workqueue 中,自定义 controller 里面的 worker会获取Workqueue 里面内容,并进行相应的业务处理。
k8s官方有个crd的例子可以供我们研究学习:
代码语言:javascript复制https://github.com/kubernetes/sample-controller
1,编译:
代码语言:javascript复制% go build -o sample-controller .
../../../../go/pkg/mod/sigs.k8s.io/json@v0.0.0-20211020170558-c049b76a60c6/internal/golang/encoding/json/encode.go:1255:18: sf.IsExported undefined (type reflect.StructField has no field or method IsExported)
原因是最新版本用的是go 1.17,json包不兼容;
代码语言:javascript复制% git checkout release-1.21
重新编译,问题解决。
2,启动controller:
代码语言:javascript复制 % ./sample-controller -kubeconfig=$HOME/.kube/config
I1104 13:44:56.114207 58916 controller.go:115] Setting up event handlers
I1104 13:44:56.114648 58916 controller.go:156] Starting Foo controller
I1104 13:44:56.114654 58916 controller.go:159] Waiting for informer caches to sync
E1104 13:44:56.132686 58916 reflector.go:138] pkg/generated/informers/externalversions/factory.go:117: Failed to watch *v1alpha1.Foo: failed to list *v1alpha1.Foo: the server could not find the requested resource (get foos.samplecontroller.k8s.io)
E1104 13:44:57.427687 58916 reflector.go:138] pkg/generated/informers/externalversions/factory.go:117: Failed to watch *v1alpha1.Foo: failed to list *v1alpha1.Foo: the server
启动成功,中间报错,是因为我们没有通过kubectl apply向etcd里注册对应的资源,所以informers说没有对应的资源。
3,创建我们的foo资源
代码语言:javascript复制% kubectl create -f artifacts/examples/crd.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16 , unavailable in v1.22 ; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/foos.samplecontroller.k8s.io created
代码语言:javascript复制% kubectl create -f artifacts/examples/example-foo.yaml
foo.samplecontroller.k8s.io/example-foo created
4,查看资源:
代码语言:javascript复制 % kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
example-foo 0/1 1 0 14s
已经成功将foo注册进去了。
5,回头看我们controller的日志,已经不报错了
代码语言:javascript复制^@^@^@I1104 13:48:10.365898 58916 controller.go:228] Successfully synced 'default/example-foo'
I1104 13:48:10.365938 58916 event.go:291] "Event occurred" object="default/example-foo" kind="Foo" apiVersion="samplecontroller.k8s.io/v1alpha1" type="Normal" reason="Synced" message="Foo synced successfully"
6,我们尝试着获取资源
代码语言:javascript复制 % kubectl get CustomResourceDefinition foos.samplecontroller.k8s.io
NAME CREATED AT
foos.samplecontroller.k8s.io 2021-11-04T05:46:53Z
代码语言:javascript复制% kubectl get Foo example-foo
NAME AGE
example-foo 3m11s
说明已经成功运行了,我们定义的资源是否正常写入etcd了呢?根据上一讲的内容,我们尝试获取下:
7,查看etcd内容:
代码语言:javascript复制% ETCDCTL_API=3 etcdctl --cacert=./etcd/ca.crt
--cert=./etcd/peer.crt
--key=./etcd/peer.key
--endpoints=https://127.0.0.1:32389
get "" --prefix --keys-only |grep foo
/registry/apiextensions.k8s.io/customresourcedefinitions/foos.samplecontroller.k8s.io
/registry/deployments/default/example-foo
/registry/events/default/example-foo-9bbb75dc8-v7dzv.16b441bc25c1f3ba
/registry/events/default/example-foo-9bbb75dc8-v7dzv.16b441bc4b1a9b8f
/registry/events/default/example-foo-9bbb75dc8-v7dzv.16b441cbaffd4dc2
/registry/events/default/example-foo-9bbb75dc8-v7dzv.16b441cbbc571207
/registry/events/default/example-foo-9bbb75dc8-v7dzv.16b441cbc264acc2
/registry/events/default/example-foo-9bbb75dc8.16b441bc25872175
/registry/events/default/example-foo.16b441bc243a6ed8
/registry/events/default/example-foo.16b441bc24a13b4a
/registry/pods/default/example-foo-9bbb75dc8-v7dzv
/registry/replicasets/default/example-foo-9bbb75dc8
/registry/samplecontroller.k8s.io/foos/default/example-foo
查看key对应的具体内容:
代码语言:javascript复制% ETCDCTL_API=3 etcdctl --cacert=./etcd/ca.crt
--cert=./etcd/peer.crt
--key=./etcd/peer.key
--endpoints=https://127.0.0.1:32389
get /registry/pods/default/example-foo-9bbb75dc8-v7dzv -w json
{"header":{"cluster_id":4202879228857769416,"member_id":16554063148076462710,"revision":2058014,"raft_term":18},"kvs":[{"key":"L3JlZ2lzdHJ5L3BvZHMvZGVmYXVsdC9leGFtcGxlLWZvby05YmJiNzVkYzgtdjdkenY=","create_revision":2057101,"mod_revision":2057210,"version":4,"value":"azhzAAoJCgJ2MRIDUG9kEt4TCowMChtleGFtcGxlLWZvby05YmJiNzVkYzgtdjdkenYSFmV4YW1wbGUtZm9vLTliYmI3NWRjOC0aB2RlZmF1bHQiACokOTAxYzA5ODYtMTM0ZS00Y2I5LTliMjItYWI5N2QyMzlmZDkyMgA4AEIICOjnjYwGEABaDAoDYXBwEgVuZ2lueFoZCgpjb250cm9sbGVyEgtleGFtcGxlLWZvb1oeChFwb2QtdGVtcGxhdGUtaGFzaBIJOWJiYjc1ZGM4alYKClJlcGxpY2FTZXQaFWV4YW1wbGUtZm9vLTliYmI3NWRjOCIkYjhjZWQ5Y2YtMjViNS00NjhkLTg1MzYtODBlODRmZmIwMjY4KgdhcHBzL3YxMAE4AXoAigG5BQoXa3ViZS1jb250cm9sbGVyLW1hbmFnZXISBlVwZGF0ZRoCdjEiCAjo542MBhAAMghGaWVsZHNWMTr9BAr6BHsiZjptZXRhZGF0YSI6eyJmOmdlbmVyYXRlTmFtZSI6e30sImY6bGFiZWxzIjp7Ii4iOnt9LCJmOmFwcCI6e30sImY6Y29udHJvbGxlciI6e30sImY6cG9kLXRlbXBsYXRlLWhhc2giOnt9fSwiZjpvd25lclJlZmVyZW5jZXMiOnsiLiI6e30sIms6e1widWlkXCI6XCJiOGNlZDljZi0yNWI1LTQ2OGQtODUzNi04MGU4NGZmYjAyNjhcIn0iOnsiLiI6e30sImY6YXBpVmVyc2lvbiI6e30sImY6YmxvY2tPd25lckRlbGV0aW9uIjp7fSwiZjpjb250cm9sbGVyIjp7fSwiZjpraW5kIjp7fSwiZjpuYW1lIjp7fSwiZjp1aWQiOnt9fX19LCJmOnNwZWMiOnsiZjpjb250YWluZXJzIjp7Ims6e1wibmFtZVwiOlwibmdpbnhcIn0iOnsiLiI6e30sImY6aW1hZ2UiOnt9LCJmOmltYWdlUHVsbFBvbGljeSI6e30sImY6bmFtZSI6e30sImY6cmVzb3VyY2VzIjp7fSwiZjp0ZXJtaW5hdGlvbk1lc3NhZ2VQYXRoIjp7fSwiZjp0ZXJtaW5hdGlvbk1lc3NhZ2VQb2xpY3kiOnt9fX0sImY6ZG5zUG9saWN5Ijp7fSwiZjplbmFibGVTZXJ2aWNlTGlua3MiOnt9LCJmOnJlc3RhcnRQb2xpY3kiOnt9LCJmOnNjaGVkdWxlck5hbWUiOnt9LCJmOnNlY3VyaXR5Q29udGV4dCI6e30sImY6dGVybWluYXRpb25HcmFjZVBlcmlvZFNlY29uZHMiOnt9fX2KAbQECgdrdWJlbGV0EgZVcGRhdGUaAnYxIggIrOiNjAYQADIIRmllbGRzVjE6iAQKhQR7ImY6c3RhdHVzIjp7ImY6Y29uZGl0aW9ucyI6eyJrOntcInR5cGVcIjpcIkNvbnRhaW5lcnNSZWFkeVwifSI6eyIuIjp7fSwiZjpsYXN0UHJvYmVUaW1lIjp7fSwiZjpsYXN0VHJhbnNpdGlvblRpbWUiOnt9LCJmOnN0YXR1cyI6e30sImY6dHlwZSI6e319LCJrOntcInR5cGVcIjpcIkluaXRpYWxpemVkXCJ9Ijp7Ii4iOnt9LCJmOmxhc3RQcm9iZVRpbWUiOnt9LCJmOmxhc3RUcmFuc2l0aW9uVGltZSI6e30sImY6c3RhdHVzIjp7fSwiZjp0eXBlIjp7fX0sIms6e1widHlwZVwiOlwiUmVhZHlcIn0iOnsiLiI6e30sImY6bGFzdFByb2JlVGltZSI6e30sImY6bGFzdFRyYW5zaXRpb25UaW1lIjp7fSwiZjpzdGF0dXMiOnt9LCJmOnR5cGUiOnt9fX0sImY6Y29udGFpbmVyU3RhdHVzZXMiOnt9LCJmOmhvc3RJUCI6e30sImY6cGhhc2UiOnt9LCJmOnBvZElQIjp7fSwiZjpwb2RJUHMiOnsiLiI6e30sIms6e1wiaXBcIjpcIjEwLjEuMC4yMDNcIn0iOnsiLiI6e30sImY6aXAiOnt9fX0sImY6c3RhcnRUaW1lIjp7fX19Eo8ECoQBChVrdWJlLWFwaS1hY2Nlc3MtZGg0eHMSa9IBaAoOIgwKABCXHBoFdG9rZW4KKBomChIKEGt1YmUtcm9vdC1jYS5jcnQSEAoGY2EuY3J0EgZjYS5jcnQKKRInCiUKCW5hbWVzcGFjZRIYCgJ2MRISbWV0YWRhdGEubmFtZXNwYWNlEKQDEpUBCgVuZ2lueBIMbmdpbng6bGF0ZXN0KgBCAEpMChVrdWJlLWFwaS1hY2Nlc3MtZGg0eHMQARotL3Zhci9ydW4vc2VjcmV0cy9rdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50IgAyAGoUL2Rldi90ZXJtaW5hdGlvbi1sb2dyBkFsd2F5c4ABAIgBAJABAKIBBEZpbGUaBkFsd2F5cyAeMgxDbHVzdGVyRmlyc3RCB2RlZmF1bHRKB2RlZmF1bHRSDmRvY2tlci1kZXNrdG9wWABgAGgAcgCCAQCKAQCaARFkZWZhdWx0LXNjaGVkdWxlcrIBNgocbm9kZS5rdWJlcm5ldGVzLmlvL25vdC1yZWFkeRIGRXhpc3RzGgAiCU5vRXhlY3V0ZSisArIBOAoebm9kZS5rdWJlcm5ldGVzLmlvL3VucmVhY2hhYmxlEgZFeGlzdHMaACIJTm9FeGVjdXRlKKwCwgEAyAEA8AEB gEUUHJlZW1wdExvd2VyUHJpb3JpdHkaugMKB1J1bm5pbmcSIwoLSW5pdGlhbGl6ZWQSBFRydWUaACIICOjnjYwGEAAqADIAEh0KBVJlYWR5EgRUcnVlGgAiCAis6I2MBhAAKgAyABInCg9Db250YWluZXJzUmVhZHkSBFRydWUaACIICKzojYwGEAAqADIAEiQKDFBvZFNjaGVkdWxlZBIEVHJ1ZRoAIggI6OeNjAYQACoAMgAaACIAKgwxOTIuMTY4LjY1LjQyCjEwLjEuMC4yMDM6CAjo542MBhAAQtcBCgVuZ2lueBIMEgoKCAis6I2MBhAAGgAgASgAMgxuZ2lueDpsYXRlc3Q6X2RvY2tlci1wdWxsYWJsZTovL25naW54QHNoYTI1Njo2NDRhNzA1MTZhMjYwMDRjOTdkMGQ4NWM3ZmUxZDBjM2E2N2VhOGFiN2RkZjRhZmYxOTNkOWYzMDE2NzBjZjM2Qklkb2NrZXI6Ly85NWM3MjFiMzZhOTdiMzcwZjI4ZWZhN2U2MzllMjdkMjdjZDIyNmE5Zjc3NzIwYTQ1ODM0YWE3ZDI1N2RlNGZjSAFKCkJlc3RFZmZvcnRaAGIMCgoxMC4xLjAuMjAzGgAiAA=="}],"count":1}
至此我们的demo已经正常运行了。