OpenEBS Local Volume 测试

OpenEBS Local Volume

Local PV hostpath

对比Kubernetes Hostpath卷相比，OpenEBS本地PV Hostpath卷具有以下优势:

OpenEBS本地PV Hostpath允许您的应用程序通过StorageClass、PVC和PV访问Hostpath。这为您提供了更改PV提供者的灵活性，而无需重新设计应用程序YAML
使用Velero备份和恢复进行数据保护
通过对应用程序YAML和pod完全屏蔽主机路径来防范主机路径安全漏洞

挂载分区

创建 1T 的 LVM 挂载到 /var/openebs/local 目录（local pv hostpath 默认使用目录，也可以创建新的 StorageClass 指定新的目录）

1 2	lvcreate -n lv_openebs -L 1T datavg ;mkfs.xfs /dev/datavg/lv_openebs ;mkdir -p /var/openebs/local mount /dev/datavg/lv_openebs /var/openebs/local

安装

1	kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml

默认创建两个 storageclass ，我们主要使用 openebs-hostpath

1 2 3 4	# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE openebs-device openebs.io/local Delete WaitForFirstConsumer false 5m43s openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 5m43s

验证

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

# vim local-hostpath-sample.yaml --- #创建 pvc kind: PersistentVolumeClaim apiVersion: v1 metadata: name: local-hostpath-pvc spec: storageClassName: openebs-hostpath accessModes: - ReadWriteOnce resources: requests: storage: 5G --- # 创建 pod apiVersion: v1 kind: Pod metadata: name: hello-local-hostpath-pod spec: volumes: - name: local-storage persistentVolumeClaim: claimName: local-hostpath-pvc containers: - name: hello-container image: dockerhub.kubekey.local/library/busybox:latest command: - sh - -c - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 300)); done' volumeMounts: - mountPath: /mnt/store name: local-storage kubectl apply -f local-hostpath-sample.yaml

查看 pv 和 pvc

1 2 3 4 5 6 7	# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-a500bbdc-65d8-4b38-a342-c15828e4a358 5G RWO Delete Bound default/local-hostpath-pvc openebs-hostpath 6m50s # kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE local-hostpath-pvc Bound pvc-a500bbdc-65d8-4b38-a342-c15828e4a358 5G RWO openebs-hostpath 8m20s

对应节点的 /var/openebs/local 目录

1 2 3 4 5 6 7 8 9

# pod 路径 # kubectl exec -ti hello-local-hostpath-pod -- cat /mnt/store/greet.txt Fri May 27 07:51:13 UTC 2022 [hello-local-hostpath-pod] Hello from OpenEBS Local PV. Fri May 27 07:56:13 UTC 2022 [hello-local-hostpath-pod] Hello from OpenEBS Local PV. # 主机路径 # cat /var/openebs/local/pvc-a500bbdc-65d8-4b38-a342-c15828e4a358/greet.txt Fri May 27 07:51:13 UTC 2022 [hello-local-hostpath-pod] Hello from OpenEBS Local PV. Fri May 27 07:56:13 UTC 2022 [hello-local-hostpath-pod] Hello from OpenEBS Local PV.

清理 pod

1	kubectl delete pod hello-local-hostpath-pod

性能测试

下载性能测试文件：

performance-benchmark/fio-deploy.yaml at master · openebs/performance-benchmark

调整以下内容：

1 2 3 4	volumes: - name: dbench-pv persistentVolumeClaim: claimName: local-hostpath-pvc # 修改这里

创建应用

1	kubectl create -f fio-deploy.yaml

查看状态

1 2 3	# kubectl get pod NAME READY STATUS RESTARTS AGE dbench-zp7th--1-xjj8t 1/1 Running 0 5s

查看性能测试结果

1 2 3 4 5 6 7 8 9 10 11	kubectl logs dbench-zp7th--1-xjj8t …… All tests complete. ================== = Dbench Summary = ================== Random Read/Write IOPS: 130k/25.3k. BW: 2838MiB/s / 978MiB/s Average Latency (usec) Read/Write: 93.60/384.73 Sequential Read/Write: 4128MiB/s / 1673MiB/s Mixed Random Read/Write IOPS: 25.1k/8413

清理

1 2	kubectl delete pod dbench-zp7th--1-xjj8t kubectl delete pvc local-hostpath-pvc

Local PV device

对比Kubernetes本地持久卷，OpenEBS本地PV设备卷有以下优点:

OpenEBS本地PV设备卷provider是动态的，Kubernetes设备卷provider是静态的
OpenEBS NDM更好地管理用于创建本地pv的块设备。 NDM提供了发现块设备属性、设置设备筛选器、度量集合以及检测块设备是否已经跨节点移动等功能

创建 SC

默认安装已经自带了 local PV 这里可以跳过

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18	cat > local-device-sc.yaml <<EOF apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: local-device annotations: openebs.io/cas-type: local cas.openebs.io/config: \| - name: StorageType value: device - name: FSType value: xfs provisioner: openebs.io/local reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer EOF kubectl apply -f local-device-sc.yaml

验证

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

# vim local-pv-sample.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: local-device-pvc spec: storageClassName: local-device accessModes: - ReadWriteOnce resources: requests: storage: 5G --- apiVersion: v1 kind: Pod metadata: name: hello-local-device-pod spec: volumes: - name: local-storage persistentVolumeClaim: claimName: local-device-pvc containers: - name: hello-container image: dockerhub.kubekey.local/library/busybox:latest command: - sh - -c - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 300)); done' volumeMounts: - mountPath: /mnt/store name: local-storage kubectl apply -f local-device-pod.yaml

查看 pv 和 pvc

1 2 3 4 5 6 7	# kubectl get pod NAME READY STATUS RESTARTS AGE hello-local-device-pod 1/1 Running 0 10m # kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE local-device-pvc Bound pvc-f8dcfaa4-2d82-46c8-850d-8d92403aa7ce 5G RWO local-device 10m

查看磁盘（ndm会自动选择合适的数据盘）

如下所示，sdb 被使用了，自动选择了 sdc 创建分区挂载到 pod

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 446.1G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 444.1G 0 part ├─rootvg-lv_root 252:0 0 20G 0 lvm / ├─rootvg-lv_swap 252:1 0 16G 0 lvm ├─rootvg-lv_var 252:2 0 60G 0 lvm /var ├─rootvg-lv_home 252:3 0 20G 0 lvm /home └─rootvg-lv_etcd 252:6 0 100G 0 lvm /var/lib/etcd sdb 8:16 0 3.5T 0 disk ├─datavg-lv_docker 252:4 0 500G 0 lvm /var/lib/docker ├─datavg-lv_kubelet 252:5 0 100G 0 lvm /var/lib/kubelet └─datavg-lv_openebs 252:7 0 1T 0 lvm /var/openebs/local sdc 8:32 0 3.5T 0 disk /var/lib/kubelet/pods/bc10974f-1a8a-433f-84b9-b20d8dc908c4/volumes/kubernetes.io~local-volume/pvc-f8dcfaa4-2d82-46c8-850d-8d92403aa7ce

性能测试

创建 pvc 用于性能测试

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16	cat > dbench-pvc.yaml <<EOF --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: dbench spec: storageClassName: local-device accessModes: - ReadWriteOnce resources: requests: storage: 5G EOF kubectl apply -f dbench-pvc.yaml

下载性能测试文件：

performance-benchmark/fio-deploy.yaml at master · openebs/performance-benchmark

调整以下内容：

1 2 3 4	volumes: - name: dbench-pv persistentVolumeClaim: claimName: dbench # 修改这里

创建应用

1	kubectl create -f fio-deploy.yaml

查看状态

1 2 3	# kubectl get pod NAME READY STATUS RESTARTS AGE dbench-qzczk--1-kz6wf 1/1 Running 0 10s

查看磁盘挂载信息

自动选择了另外一台机器的 sdc 创建分区挂载到 pod

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

10.188.100.99 | CHANGED | rc=0 >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 446.1G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 444.1G 0 part ├─rootvg-lv_root 252:0 0 20G 0 lvm / ├─rootvg-lv_swap 252:1 0 16G 0 lvm ├─rootvg-lv_var 252:2 0 60G 0 lvm /var ├─rootvg-lv_home 252:3 0 20G 0 lvm /home └─rootvg-lv_etcd 252:6 0 100G 0 lvm /var/lib/etcd sdb 8:16 0 3.5T 0 disk ├─datavg-lv_docker 252:4 0 500G 0 lvm /var/lib/docker ├─datavg-lv_kubelet 252:5 0 100G 0 lvm /var/lib/kubelet └─datavg-lv_openebs 252:7 0 1T 0 lvm /var/openebs/local sdc 8:32 0 3.5T 0 disk /var/lib/kubelet/pods/0ebab420-cdfa-4421-8a50-3fcdbe59ac94/volumes/kubernetes.io~local-volume/pvc-c2206bd7-dbd9-45d2-ba4b-36224edbd0e4 …… 10.188.100.101 | CHANGED | rc=0 >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 446.1G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 444.1G 0 part ├─rootvg-lv_root 252:0 0 20G 0 lvm / ├─rootvg-lv_swap 252:1 0 16G 0 lvm ├─rootvg-lv_var 252:2 0 60G 0 lvm /var ├─rootvg-lv_home 252:3 0 20G 0 lvm /home └─rootvg-lv_etcd 252:6 0 100G 0 lvm /var/lib/etcd sdb 8:16 0 3.5T 0 disk ├─datavg-lv_docker 252:4 0 500G 0 lvm /var/lib/docker ├─datavg-lv_kubelet 252:5 0 100G 0 lvm /var/lib/kubelet └─datavg-lv_openebs 252:7 0 1T 0 lvm /var/openebs/local sdc 8:32 0 3.5T 0 disk /var/lib/kubelet/pods/e8ba48b4-957d-4d6f-bc5f-d850f058d603/volumes/kubernetes.io~local-volume/pvc-f8dcfaa4-2d82-46c8-850d-8d92403aa7ce

查看性能测试结果

性能与 hostpath 模式相差不大

1 2 3 4 5 6 7 8 9 10 11	kubectl logs dbench-qzczk--1-kz6wf …… All tests complete. ================== = Dbench Summary = ================== Random Read/Write IOPS: 149k/25.7k. BW: 2474MiB/s / 997MiB/s Average Latency (usec) Read/Write: 94.81/384.69 Sequential Read/Write: 4428MiB/s / 1821MiB/s Mixed Random Read/Write IOPS: 22.4k/7434

清理

1 2	kubectl delete pod dbench-qzczk--1-kz6wf hello-local-device-pod kubectl delete pvc dbench local-device-pvc

LVM Local PV

安装

1	kubectl apply -f https://openebs.github.io/charts/lvm-operator.yaml

查看 pod 状态

1 2 3 4 5 6 7 8 9 10 11 12

# kubectl get pod -A | grep openebs kube-system openebs-lvm-controller-0 5/5 Running 0 74s kube-system openebs-lvm-node-47nz2 2/2 Running 0 74s kube-system openebs-lvm-node-7nl47 2/2 Running 0 74s kube-system openebs-lvm-node-dwrkc 2/2 Running 0 74s kube-system openebs-lvm-node-g55vx 2/2 Running 0 74s kube-system openebs-lvm-node-gxwgr 2/2 Running 0 74s kube-system openebs-lvm-node-hkz69 2/2 Running 0 74s kube-system openebs-lvm-node-kv48x 2/2 Running 0 74s kube-system openebs-lvm-node-mqwm8 2/2 Running 0 74s kube-system openebs-lvm-node-tsxsl 2/2 Running 0 74s kube-system openebs-lvm-node-z2z6g 2/2 Running 0 74s

创建 vg

集群中部分机器有 sdc ，手动在 sdc 上面创建 lvmvg 用于 openebs lvm local pv

1 2	pvcreate /dev/sdc vgcreate lvmvg /dev/sdc

创建 SC

因为 vg 只在部分机器上面有，所以需要使用 allowedTopologies 选项，指定可以使用的节点

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

cat > lvm-local-pv-sc.yaml <<EOF apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: openebs-lvmpv allowVolumeExpansion: true parameters: storage: "lvm" volgroup: "lvmvg" #上面创建的 vg 名 provisioner: local.csi.openebs.io allowedTopologies: - matchLabelExpressions: - key: kubernetes.io/hostname values: #有 lvmvg 的节点 - wx-vpck8s-worker098 - wx-vpck8s-worker099 - wx-vpck8s-worker100 - wx-vpck8s-worker101 - wx-vpck8s-worker102 EOF kubectl apply -f lvm-local-pv-sc.yaml

部署应用

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

cat > lvm-local-pv-example.yaml <<EOF kind: PersistentVolumeClaim apiVersion: v1 metadata: name: csi-lvmpv spec: storageClassName: openebs-lvmpv accessModes: - ReadWriteOnce resources: requests: storage: 10Gi --- apiVersion: v1 kind: Pod metadata: name: lvm-pod spec: restartPolicy: Never containers: - name: perfrunner image: dockerhub.kubekey.local/library/busybox:latest command: - sh - -c - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 300)); done' volumeMounts: - mountPath: /datadir name: lvm-vol tty: true volumes: - name: lvm-vol persistentVolumeClaim: claimName: csi-lvmpv EOF kubectl apply -f lvm-local-pv-example.yaml

查看应用和lvm信息

1 2 3 4 5 6 7 8 9 10 11 12 13 14

# kubectl get pod,pv,pvc NAME READY STATUS RESTARTS AGE pod/lvm-pod 1/1 Running 0 27s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-9abc3a59-cefa-4c78-8189-ddc2ee09e58e 10Gi RWO Delete Bound default/csi-lvmpv openebs-lvmpv 26s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/csi-lvmpv Bound pvc-9abc3a59-cefa-4c78-8189-ddc2ee09e58e 10Gi RWO openebs-lvmpv 27s # lvm 信息 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdc 8:32 0 3.5T 0 disk └─lvmvg-pvc--9abc3a59--cefa--4c78--8189--ddc2ee09e58e 252:8 0 10G 0 lvm /var/lib/kubelet/pods/32dd581a-e163-4fe1-a8a5-a1eafb396d71/volumes/kubernetes.io~csi/pvc-9abc3a59-cefa-4c78-8189-ddc2ee09e58e/mount

性能测试

创建新的 pvc 用于性能测试

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15	cat > dbench-pvc.yaml <<EOF kind: PersistentVolumeClaim apiVersion: v1 metadata: name: dbench-lvm spec: storageClassName: openebs-lvmpv accessModes: - ReadWriteOnce resources: requests: storage: 50Gi EOF kubectl apply -f dbench-pvc.yaml

下载性能测试文件：

performance-benchmark/fio-deploy.yaml at master · openebs/performance-benchmark

调整以下内容：

1 2 3 4	volumes: - name: dbench-lvm persistentVolumeClaim: claimName: dbench # 修改这里

创建应用

1	kubectl create -f fio-deploy.yaml

查看状态

1 2 3 4	# kubectl get pod NAME READY STATUS RESTARTS AGE dbench-729c9--1-4clvm 1/1 Running 0 5s lvm-pod 1/1 Running 0 6m2s

查看磁盘挂载信息

自动选择了另外一台机器的 sdc 创建分区挂载到 pod

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

10.188.100.99 | CHANGED | rc=0 >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 446.1G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 444.1G 0 part ├─rootvg-lv_root 252:0 0 20G 0 lvm / ├─rootvg-lv_swap 252:1 0 16G 0 lvm ├─rootvg-lv_var 252:2 0 60G 0 lvm /var ├─rootvg-lv_home 252:3 0 20G 0 lvm /home └─rootvg-lv_etcd 252:6 0 100G 0 lvm /var/lib/etcd sdb 8:16 0 3.5T 0 disk ├─datavg-lv_docker 252:4 0 500G 0 lvm /var/lib/docker ├─datavg-lv_kubelet 252:5 0 100G 0 lvm /var/lib/kubelet └─datavg-lv_openebs 252:7 0 1T 0 lvm /var/openebs/local sdc 8:32 0 3.5T 0 disk └─lvmvg-pvc--763113e0--c96f--4487--b3ae--9a306c159231 252:8 0 50G 0 lvm /var/lib/kubelet/pods/61686ab9-0538-4a89-a8e7-84ce9af351b1/volumes/kubernetes.io~csi/pvc-763113e0-c96f-4487-b3ae-9a306c159231/mount 10.188.100.98 | CHANGED | rc=0 >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 446.1G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 444.1G 0 part ├─rootvg-lv_root 252:0 0 20G 0 lvm / ├─rootvg-lv_swap 252:1 0 16G 0 lvm ├─rootvg-lv_var 252:2 0 60G 0 lvm /var ├─rootvg-lv_home 252:3 0 20G 0 lvm /home └─rootvg-lv_etcd 252:6 0 100G 0 lvm /var/lib/etcd sdb 8:16 0 3.5T 0 disk ├─datavg-lv_docker 252:4 0 500G 0 lvm /var/lib/docker ├─datavg-lv_kubelet 252:5 0 100G 0 lvm /var/lib/kubelet └─datavg-lv_openebs 252:7 0 1T 0 lvm /var/openebs/local sdc 8:32 0 3.5T 0 disk └─lvmvg-pvc--9abc3a59--cefa--4c78--8189--ddc2ee09e58e 252:8 0 10G 0 lvm /var/lib/kubelet/pods/32dd581a-e163-4fe1-a8a5-a1eafb396d71/volumes/kubernetes.io~csi/pvc-9abc3a59-cefa-4c78-8189-ddc2ee09e58e/mount

查看性能测试结果

1 2 3 4 5 6 7 8 9 10 11	kubectl logs dbench-729c9--1-4clvm …… All tests complete. ================== = Dbench Summary = ================== Random Read/Write IOPS: 125k/23.9k. BW: 2623MiB/s / 962MiB/s Average Latency (usec) Read/Write: 95.08/385.63 Sequential Read/Write: 5042MiB/s / 1819MiB/s Mixed Random Read/Write IOPS: 23.7k/7938

清理

1 2 3	kubectl delete -f fio-deploy.yaml kubectl delete -f dbench-pvc.yaml kubectl delete -f lvm-local-pv-example.yaml

总结

三种模式pod磁盘性能差不多，lvm 模式下顺序读性能稍好。

Local PV hostpath 需要手动挂载磁盘到对应目录
Local PV device pod独占磁盘
LVM Local PV 需要手动创建vg，pod 使用 lvm 较为灵活，目前推荐使用

参考文章：

https://weiliang-ms.github.io/wl-awesome/2.容器/k8s/storage/OpenEBS.html

https://github.com/openebs/lvm-localpv

性能测试 local volume 测试磁盘

0 人点赞