Kubernetes 部署 NFS 持久存储
Nov 27, 2019 21:30 · 696 words · 2 minute read
我们自己测试使用的 Kubernetes 环境有时候需要外置的持久化存储，Ceph 或者 OpenStack Cinder 这些冗余外置存储也不是想要就要的，这时候自己搭建一个 NFS 来模拟外置存储无疑是最简单的方案。
搭建 NFS 服务器
1. 安装 NFS 工具
$ yum install nfs-utils
2. 创建 NFS 数据路径
$ mkdir -p /nfs/data
$ chmod -R 777 /nfs/data
3. 格式化磁盘并挂载（建议添加一块磁盘作为 NFS 数据盘）
$ mkfs.xfs /dev/sdb
$ mount /dev/sdb /nfs/data
$ echo "/dev/sdb /nfs/data xfs defaults 0 0" >> /etc/fstab
4. 编辑 NFS 配置文件
$ echo "/nfs/data *(rw,no_root_squash,sync)" > /etc/exports
$ exportfs -r
5. 启动 rpcbind、nfs 服务
$ systemctl restart rpcbind && systemctl enable rpcbind
$ systemctl restart nfs && systemctl enable nfs
6. 查看 RPC 服务的注册状况
$ rpcinfo -p localhost
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100005 1 udp 20048 mountd
100005 1 tcp 20048 mountd
100024 1 udp 60673 status
100024 1 tcp 60405 status
100005 2 udp 20048 mountd
100005 2 tcp 20048 mountd
100005 3 udp 20048 mountd
100005 3 tcp 20048 mountd
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100227 3 tcp 2049 nfs_acl
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100227 3 udp 2049 nfs_acl
100021 1 udp 44635 nlockmgr
100021 3 udp 44635 nlockmgr
100021 4 udp 44635 nlockmgr
100021 1 tcp 39475 nlockmgr
100021 3 tcp 39475 nlockmgr
100021 4 tcp 39475 nlockmgr
7. 查看 NFS 挂载信息
$ showmount -e ${NODE_IP}
Export list for ${NODE_IP}:
/nfs/data *
Kubernetes 使用动态 PersistentVolume
1. 部署 RBAC
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs-client/deploy/rbac.yaml
2. 部署 nfs-client-provisioner
$ cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
labels:
app: nfs-client-provisioner
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: nfs-client-provisioner
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-client-provisioner
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: ${NODE_IP}
- name: NFS_PATH
value: /nfs/data
volumes:
- name: nfs-client-root
nfs:
server: ${NODE_IP}
path: /nfs/data
修改 NFS 服务器地址 NFS_SERVER 对应的值后执行：
$ kubectl apply -f deployment.yaml
3. 创建 StorageClass
provisioner 要对应 驱动所传入的环境变量 PROVISIONER_NAME 的值。
$ cat storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs
provisioner: fuseim.pri/ifs
parameters:
archiveOnDelete: "false"
reclaimPolicy: Delete
$ kubectl apply -f storageclass.yaml
如果要把 nfs 设置成默认 StorageClass：
$ kubectl patch storageclass nfs -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
$ kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs (default) fuseim.pri/ifs Delete Immediate false 5m43s
4. 创建 PVC (PersistentVolumeClaim)
$ cat pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-pvc
annotations:
volume.beta.kubernetes.io/storage-class: "nfs"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
$ kubectl apply -f pvc.yaml
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-pvc Bound pvc-afa835af-f367-48be-804e-6e9820520063 1Gi RWX nfs 7s
kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-afa835af-f367-48be-804e-6e9820520063 1Gi RWX Delete Bound default/test-pvc nfs 5m10s
5. 测试
kind: Pod
apiVersion: v1
metadata:
name: test-pod
spec:
containers:
- name: test-pod
image: gcr.io/google_containers/busybox:1.24
command:
- "/bin/sh"
args:
- "-c"
- "touch /mnt/SUCCESS && exit 0 || exit 1"
volumeMounts:
- name: nfs-pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: nfs-pvc
persistentVolumeClaim:
claimName: test-pvc