参考：https://forums.cnrancher.com/q_445.html
问题2：挂载失败 MountVolume.WaitForAttach failed for volume "pvc-aa0d2e46-3df3-4c70-a318-ad95d4d0810a" : rbd: map failed exit status 110, rbd output: rbd: sysfs write failed In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (110) Connection timed out解决
ceph osd crush tunables hammer参考：
问题3：ceph HEALTH_WARN HEALTH_WARN application not enabled on 1 pool(s)解决
ceph health detail ceph osd pool application enable kube-pool rbd 部署cephfs文件系统k8s默认没有cephfs的provisioner，所以需要手动部署一个provisioner去对接cephfs
参考github链接：https://github.com/kubernetes-retired/external-storage/tree/master/ceph/cephfs
部署mds（元数据服务） ceph-deploy mds create ceph-01 创建两个存储池，用来存放实际的数据以及元数据 ceph osd pool create cephfs_data 64 ceph osd pool create cephfs_metadata 64 创建cephfs文件系统 ceph fs new cephfs cephfs_metadata cephfs_data 查看mds状态 ceph mds stat ceph -s 部署provisioner这里有两种方式部署provisioner，其中一种是直接docker run的方式部署，另一种是通过deployment的方式部署到k8s中
docker run方式部署ceph-provisioner docker run -tid -v /root/.kube:/kube -v /var/run/kubernetes:/var/run/kubernetes --privileged --net=host --name ceph-provisioner quay.io/external_storage/cephfs-provisioner /usr/local/bin/cephfs-provisioner -master=https://172.16.0.99:6443 -kubeconfig=http://www.likecs.com/kube/config -id=cephfs-provisioner-1 -disable-ceph-namespace-isolation deployment方式部署到k8s中rbac相关yaml
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cephfs-provisioner namespace: cephfs rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] - apiGroups: [""] resources: ["services"] resourceNames: ["kube-dns","coredns"] verbs: ["list", "get"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cephfs-provisioner subjects: - kind: ServiceAccount name: cephfs-provisioner namespace: cephfs roleRef: kind: ClusterRole name: cephfs-provisioner apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: cephfs-provisioner namespace: cephfs rules: - apiGroups: [""] resources: ["secrets"] verbs: ["create", "get", "delete"] - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: cephfs-provisioner namespace: cephfs roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: cephfs-provisioner subjects: - kind: ServiceAccount name: cephfs-provisioner --- apiVersion: v1 kind: ServiceAccount metadata: name: cephfs-provisioner namespace: cephfsceph-provisioner-deployment.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: cephfs-provisioner namespace: cephfs spec: replicas: 1 selector: matchLabels: app: cephfs-provisioner strategy: type: Recreate template: metadata: labels: app: cephfs-provisioner spec: containers: - name: cephfs-provisioner image: "quay.io/external_storage/cephfs-provisioner:latest" env: - name: PROVISIONER_NAME value: ceph.com/cephfs - name: PROVISIONER_SECRET_NAMESPACE value: cephfs command: - "/usr/local/bin/cephfs-provisioner" args: - "-id=cephfs-provisioner-1" - "-disable-ceph-namespace-isolation" serviceAccount: cephfs-provisioner