一、安装
1.1、创建operator
代码语言：javascript
复制# 安装
git clone --single-branch --branch v1.8.7 https://github.com/rook/rook.git
cd rook/deploy/examples
kubectl create -f crds.yaml -f common.yaml
# 修改配置
vim operator.yaml
# 自动发现开启
ROOK_ENABLE_DISCOVERY_DAEMON: "true"
# 镜像
# 国外镜像，需访问国外网站
# 容忍，由于测试环境，需要部署到master节点
- name: DISCOVER_TOLERATIONS
value: |
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
- effect: NoExecute
key: node-role.kubernetes.io/master
operator: Exists
kubectl apply -f operator.yaml
1.2、创建集群
代码语言：javascript
复制# 创建集群
# 修改配置
vim cluster.yaml
skipUpgradeChecks: true
# mgr副本数
mgr:
count: 2
# 是否使用所有节点，所有磁盘
storage: # cluster level storage configuration and selection
useAllNodes: false
useAllDevices: false
# 自定义节点
nodes:
- name: "k8s-master01"
devices: # specific devices to use for storage can be specified for each node
- name: "sdb"
- name: "k8s-master02"
devices: # specific devices to use for storage can be specified for each node
- name: "sdb"
- name: "k8s-master03"
devices: # specific devices to use for storage can be specified for each node
- name: "sdb"
# 容忍，测试环境需要使用master节点
placement:
osd:
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/controlplane
operator: Exists
- effect: NoExecute
key: node-role.kubernetes.io/etcd
operator: Exists
prepareosd:
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/controlplane
operator: Exists
- effect: NoExecute
key: node-role.kubernetes.io/etcd
operator: Exists
mon:
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/controlplane
operator: Exists
- effect: NoExecute
key: node-role.kubernetes.io/etcd
operator: Exists
# 注：时间很长，需要30分钟左右
kubectl apply -f cluster.yaml
# 查看存储驱动
kubectl get csidrivers.storage.k8s.io
# 创建工具pod，用于执行ceph命令
kubectl apply -f toolbox.yaml
# 启用orchestrator
kubectl -n rook-ceph exec -it rook-ceph-tools-68955c4f99-8kmgb -- bash
ceph mgr module enable rook
ceph orch set backend rook
1.3、配置ceph控制台ingress
代码语言：javascript
复制# 配置ceph控制台ingress
# 生成https证书
kubectl create secret tls testsoft-secret --cert=www.testsoft.com.crt --key=www.testsoft.com.key -n rook-ceph
# 创建ingress
# 修改dashboard-ingress-https.yaml
vim dashboard-ingress-https.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: rook-ceph-mgr-dashboard
namespace: rook-ceph # namespace:cluster
annotations:
kubernetes.io/ingress.class: "nginx"
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/server-snippet: |
proxy_ssl_verify off;
spec:
tls:
- hosts:
- ceph.testsoft.com
secretName: testsoft-secret
rules:
- host: ceph.testsoft.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: rook-ceph-mgr-dashboard
port:
name: https-dashboard
# 创建ingress
kubectl apply -f dashboard-ingress-https.yaml -n rook-ceph
# 获取访问密码
# 访问地址：https://ceph.testsoft.com
kubectl get secrets -n rook-ceph rook-ceph-dashboard-password -o jsonpath="{['data']['password']}"|base64 --decode && echo
二、使用
2.1、配置storageclasses，提供块存储pvc
代码语言：javascript
复制# storageclasses
# 存储类型Block Storage
# 编辑配置文件，测试环境副本数2即可
vim /root/rook-1.8.7/deploy/examples/csi/rbd/storageclass.yaml
replicated:
size: 2
# 创建pool以及storageclass
kubectl apply -f /root/rook-1.8.7/deploy/examples/csi/rbd/storageclass.yaml
2.2、配置storageclasses，提供文件存储pvc
代码语言：javascript
复制# storageclasses
# 存储类型Shared Filesystem
# 编辑配置文件，测试环境副本数2即可
vim /root/rook-1.8.7/deploy/examples/filesystem.yaml
dataPools:
- name: replicated
failureDomain: host
replicated:
size: 2
# 创建Shared Filesystem pool
kubectl apply -f filesystem.yaml
# 创建storageclasses
kubectl apply -f /root/rook-1.8.7/deploy/examples/csi/cephfs/storageclass.yaml
2.3、配置storageclasses，通过对象存储bucket，支持s3协议
代码语言：javascript
复制# storageclasses
# 存储类型Object Storage
# 编辑配置文件，测试环境副本数2即可
vim /root/rook-1.8.7/deploy/examples/object.yaml
dataPool:
failureDomain: host
replicated:
size: 2
# 创建CephObjectStore
kubectl apply -f object.yaml
# 确认gw以及对应svc已经正常
kubectl -n rook-ceph get pod -l app=rook-ceph-rgw
kubectl -n rook-ceph get svc -l app=rook-ceph-rgw
# 创建storageclasses
kubectl apply -f /root/rook-1.8.7/deploy/examples/storageclass-bucket-delete.yaml
# 创建bucket
# 默认在default namespace，bucket和pvc属于同级，必须绑定namespace使用
# 存储类型Block Storage和存储类型Shared Filesystem创建的是pvc，存储类型Object Storage创建的是桶
kubectl apply -f /root/rook-1.8.7/deploy/examples/object-bucket-claim-delete.yaml
# s3 API访问信息
AWS_HOST=$(kubectl -n default get cm ceph-delete-bucket -o jsonpath='{.data.BUCKET_HOST}')
AWS_ACCESS_KEY_ID=$(kubectl -n default get secret ceph-delete-bucket -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 --decode)
AWS_SECRET_ACCESS_KEY=$(kubectl -n default get secret ceph-delete-bucket -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode)
2.4、配置snapshot，提供pvc快照功能
代码语言：javascript
复制# 快照支持
# 官方文档：https://rook.io/docs/rook/v1.8/ceph-csi-snapshot.html
# 下载对应external-snapshotter：https://github.com/kubernetes-csi/external-snapshotter
# 创建crd
kubectl kustomize client/config/crd | kubectl create -f -
# 创建snapshot-controller
kubectl -n kube-system kustomize deploy/kubernetes/snapshot-controller | kubectl create -f -
# 创建snapshotclass
# 存储类型Block Storage
kubectl apply -f /root/rook-1.8.6/deploy/examples/csi/rbd/snapshotclass.yaml
# 保存快照
# yaml示例
# name: 快照名称
# persistentVolumeClaimNam 要创建快照的pvc
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: rbd-pvc-snapshot
spec:
volumeSnapshotClassName: csi-rbdplugin-snapclass
source:
persistentVolumeClaimName: rbd-pvc
# 查看快照
kubectl get volumesnapshot
# 恢复快照（基于快照创建pvc）
# 示例yaml
# name：恢复出来的pvc名称
# rbd-pvc-snapshot： 快照名称
# storage: 必须大于等于之前快照pvc的大小
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pvc-restore
spec:
storageClassName: rook-ceph-block
dataSource:
name: rbd-pvc-snapshot
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
# 创建snapshotclass
# 存储类型Shared Filesystem
kubectl apply -f /root/rook-1.8.6/deploy/examples/csi/cephfs/snapshotclass.yaml
# 保存快照
# yaml示例
# name: 快照名称
# persistentVolumeClaimNam 要创建快照的pvc
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: cephfs-pvc-snapshot
spec:
volumeSnapshotClassName: csi-cephfsplugin-snapclass
source:
persistentVolumeClaimName: pvc_name
# 查看快照
kubectl get volumesnapshot
# # 恢复快照（基于快照创建pvc）
# 示例yaml
# name：恢复出来的pvc名称
# rbd-pvc-snapshot： 快照名称
# storage: 必须大于等于之前快照pvc的大小
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cephfs-pvc-restore
spec:
storageClassName: rook-cephfs
dataSource:
name: cephfs-pvc-snapshot
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
# 查看快照类
kubectl get VolumeSnapshotClass
2.5、pvc扩容
代码语言：javascript
复制# pvc 扩容
kubectl edit pvc pvc_name
# 修改storage大小
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
# 生效时间取决于容量大小
2.6、pvc克隆
代码语言：javascript
复制# pvc 克隆
# 存储类型Block Storage
# yaml示例
# metadata.name：复制出来的pvc名称
# dataSource.name： 源pvc名称
# storage: 必须大于等于之前快照pvc的大小
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pvc-clone
spec:
storageClassName: rook-ceph-block
dataSource:
name: rbd-pvc
kind: PersistentVolumeClaim
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
# pvc 克隆
# 存储类型Shared Filesystem
# yaml示例
# name：复制出来的pvc名称
# dataSource.name： 源pvc名称
# storage: 必须大于等于之前快照pvc的大小
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cephfs-pvc-clone
spec:
storageClassName: rook-cephfs
dataSource:
name: cephfs-pvc
kind: PersistentVolumeClaim
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
三、维护
3.1、增加osd
代码语言：javascript
复制# 增加osd
kubectl edit cephclusters.ceph.rook.io -n rook-ceph rook-ceph
- devices:
- name: sdb
name: k8s-node01
- devices:
- name: sdb
name: k8s-node02
# 重启operator
kubectl -n rook-ceph rollout restart deployment rook-ceph-operator
3.2、移除osd
代码语言：javascript
复制# 移除osd
# 需先确认移除osd后集群又足够的空间进行数据存储，否则先增加osd
# 确认剩余osd和pg正常
# 不要一次移除过多osd
# operator副本数设置为0，防止自动重建
kubectl -n rook-ceph scale deployment rook-ceph-operator --replicas=0
# 删除集群中对应osd
kubectl edit cephclusters.ceph.rook.io -n rook-ceph rook-ceph
# 从ceph集群清除osd
# 编辑移除yaml 修改OSD-IDs
vim osd-purge.yaml
# 执行删除job
kubectl create -f osd-purge.yaml
# 执行完成后删除该job
kubectl delete -f osd-purge.yaml
# 删除对应osd deployment
kubectl delete deployment -n rook-ceph rook-ceph-osd-<ID>
# operator副本数设置为1
kubectl -n rook-ceph scale deployment rook-ceph-operator --replicas=1
