Ceph-CSI-Operator Release Installation Guide

---

• Ceph-CSI-Operator Release Installation Guide
• 1. Prerequisites
  • Clone the Repository
• 2. Kubernetes Installation
  • 2.1 All-in-One Installation
  • Install the Operator
  • Verify Installation
  • 2.2 Multi-File Installation
  • Step 1: Install CRDs
  • Step 2: Create RBAC Resources
  • Step 3: Install the Operator
  • Step 4: Verify Installation
• 3. OpenShift Installation
  • Prerequisites
  • 3.1 All-in-One Installation
  • Install the Operator with SCC
  • Verify Installation
  • Verify SCC Resources
  • 3.2 Multi-File Installation
  • Step 1: Install CRDs
  • Step 2: Create RBAC Resources
  • Step 3: Create OpenShift SCC
  • Step 4: Install the Operator
  • Step 5: Verify Installation
• 4. Deploy Ceph-CSI Drivers
  • 4.1 Deploy the RBD Driver
  • 4.2 Deploy the CephFS Driver
  • 4.3 Deploy the Ceph-NFS Driver
• 5. Verify Installation
• 6. Create CephConnection
• 7. Create ClientProfile
• 8. Create Ceph Secrets
• 9. Create StorageClasses
• 10. Create VolumeSnapshotClasses (Optional)
• 11. Verify Storage Provisioning
• 12. Upgrade Ceph-CSI Operator and Drivers
  • Step 1: Fetch and Checkout the Latest Tag
  • Step 2: Apply Updated Manifests
  • For Kubernetes (All-in-One)
  • For OpenShift (All-in-One)
  • For Multi-File Installation
  • Step 3: Verify the Upgrade
• 13. Clean Up Resources
  • Step 1: Delete Custom Resources
  • Step 2: Uninstall the Operator
  • For Kubernetes
  • For OpenShift
  • For Multi-File Installation
  • Step 3: Verify Deletion

1. Prerequisites

Before proceeding with the installation of the Ceph-CSI Operator, ensure the following requirements are met:

• Kubernetes or OpenShift Cluster:
• Kubernetes: A running cluster with a supported version (Kubernetes Release Versions)
• OpenShift: Version 4.19 or later with cluster administrator privileges
• Ceph Cluster: A Ceph cluster with a supported version (Ceph Releases)
• CLI Tools:
• kubectl for Kubernetes clusters
• oc or kubectl for OpenShift clusters

Clone the Repository

For all installation methods, clone the Ceph-CSI-Operator repository and checkout the desired release tag:

git clone https://github.com/ceph/ceph-csi-operator.git
cd ceph-csi-operator
git checkout v0.3.1

Note: Check out the latest tag from Releases.

---

2. Kubernetes Installation

Choose either the All-in-One or Multi-File installation method based on your requirements.

2.1 All-in-One Installation

The All-in-One installation deploys all components (CRDs, RBAC, operator) in a single command.

Install the Operator

kubectl create -f deploy/all-in-one/install.yaml

This creates: - Custom Resource Definitions (CRDs) - RBAC resources (Role-Based Access Control) - Ceph-CSI Operator deployment

Verify Installation

kubectl get pods -n ceph-csi-operator-system

Expected output:

NAME                                                    READY   STATUS    RESTARTS   AGE
ceph-csi-operator-controller-manager-67d45fd9ff-zgst7   2/2     Running   0          40s

2.2 Multi-File Installation

The Multi-File installation provides finer control by deploying components separately.

Step 1: Install CRDs

kubectl create -f deploy/multifile/crd.yaml

This creates the Custom Resource Definitions: CephConnection, ClientProfile, ClientProfileMapping, and Driver.

Step 2: Create RBAC Resources

Create RBAC resources in the namespace where you plan to install the Ceph-CSI drivers:

kubectl create -f deploy/multifile/csi-rbac.yaml -n ceph-csi-operator-system

Step 3: Install the Operator

kubectl create -f deploy/multifile/operator.yaml -n ceph-csi-operator-system

Step 4: Verify Installation

kubectl get pods -n ceph-csi-operator-system

Expected output:

NAME                                                    READY   STATUS    RESTARTS   AGE
ceph-csi-operator-controller-manager-67d45fd9ff-zgst7   2/2     Running   0          40s

---

3. OpenShift Installation

When deploying on OpenShift, additional SecurityContextConstraints (SCC) resources are required to grant necessary permissions for CSI operations.

Prerequisites

• OpenShift 4.x cluster
• Cluster administrator privileges to create SecurityContextConstraints

3.1 All-in-One Installation

The recommended method for OpenShift is the all-in-one installer that includes SCC resources.

Install the Operator with SCC

kubectl create -f deploy/all-in-one/install-openshift.yaml

This creates: - All operator components (CRDs, RBAC, deployment) - SecurityContextConstraint (ceph-csi-scc) with necessary host-level permissions - ClusterRole to use the SCC - ClusterRoleBindings for all CSI service accounts (RBD, CephFS, NFS, NVMe-oF)

Verify Installation

kubectl get pods -n ceph-csi-operator-system

Expected output:

NAME                                                    READY   STATUS    RESTARTS   AGE
ceph-csi-operator-controller-manager-67d45fd9ff-zgst7   2/2     Running   0          40s

Verify SCC Resources

oc get scc | grep ceph-csi

Expected output:

ceph-csi-scc   false   []        RunAsAny   RunAsAny   RunAsAny   RunAsAny   <none>     false

3.2 Multi-File Installation

For more granular control, you can install components separately.

Step 1: Install CRDs

kubectl create -f deploy/multifile/crd.yaml

Step 2: Create RBAC Resources

kubectl create -f deploy/multifile/csi-rbac.yaml -n ceph-csi-operator-system

Step 3: Create OpenShift SCC

kubectl create -f deploy/multifile/openshift-scc.yaml

This creates the SecurityContextConstraints and necessary RBAC for CSI service accounts.

Step 4: Install the Operator

kubectl create -f deploy/multifile/operator.yaml -n ceph-csi-operator-system

Step 5: Verify Installation

kubectl get pods -n ceph-csi-operator-system
oc get scc | grep ceph-csi

---

4. Deploy Ceph-CSI Drivers

Once the operator is installed (on either Kubernetes or OpenShift), deploy the required CSI drivers.

4.1 Deploy the RBD Driver

echo '
apiVersion: csi.ceph.io/v1
kind: Driver
metadata:
  name: rbd.csi.ceph.com
  namespace: ceph-csi-operator-system
' | kubectl create -f -

4.2 Deploy the CephFS Driver

echo '
apiVersion: csi.ceph.io/v1
kind: Driver
metadata:
  name: cephfs.csi.ceph.com
  namespace: ceph-csi-operator-system
' | kubectl create -f -

4.3 Deploy the Ceph-NFS Driver

echo '
apiVersion: csi.ceph.io/v1
kind: Driver
metadata:
  name: nfs.csi.ceph.com
  namespace: ceph-csi-operator-system
' | kubectl create -f -

---

5. Verify Installation

Verify that all CSI driver components are running:

kubectl get pod -n ceph-csi-operator-system

Expected output:

NAME                                                    READY   STATUS    RESTARTS   AGE
ceph-csi-operator-controller-manager-744dc99cb5-scxxh   2/2     Running   0          45s
cephfs.csi.ceph.com-ctrlplugin-5847c998b5-xf85m         5/5     Running   0          27s
cephfs.csi.ceph.com-nodeplugin-r6pkt                    2/2     Running   0          27s
nfs.csi.ceph.com-ctrlplugin-76fd4f5b4c-smk2g            5/5     Running   0          27s
nfs.csi.ceph.com-nodeplugin-kbzms                       2/2     Running   0          27s
rbd.csi.ceph.com-ctrlplugin-6965dcfdb8-w88kn            5/5     Running   0          4m35s
rbd.csi.ceph.com-nodeplugin-lnm4n                       2/2     Running   0          4m35s

---

6. Create CephConnection

Create a CephConnection CR to connect to the Ceph cluster:

echo '
apiVersion: csi.ceph.io/v1
kind: CephConnection
metadata:
  name: ceph-connection
  namespace: ceph-csi-operator-system
spec:
  monitors:
  - 10.98.44.171:6789
' | kubectl create -f -

Replace the monitor IP address with your Ceph cluster's monitor addresses.

---

7. Create ClientProfile

Create a ClientProfile CR to define the client configuration:

echo '
apiVersion: csi.ceph.io/v1
kind: ClientProfile
metadata:
  name: storage
  namespace: ceph-csi-operator-system
spec:
  cephConnectionRef:
    name: ceph-connection
  cephFs:
    subVolumeGroup: csi
' | kubectl create -f -

[!IMPORTANT] The ClientProfile name (storage in this example) will be used as the clusterID parameter in your StorageClass and VolumeSnapshotClass resources.

---

8. Create Ceph Secrets

Before you can provision storage, create Kubernetes Secrets containing Ceph credentials for CSI operations.

For detailed instructions on creating Ceph users and Kubernetes Secrets, refer to the upstream Ceph-CSI documentation:

• Secret Examples:
• RBD Secret Example
• CephFS Secret Example (also used for NFS)
• Ceph Capabilities: Required Ceph Capabilities

[!NOTE] - Create secrets in the namespace where your applications will create PVCs - NFS volumes use the same CephFS secret format since NFS is built on CephFS

9. Create StorageClasses

Create StorageClasses using the upstream Ceph-CSI examples:

• RBD StorageClass Example
• CephFS StorageClass Example
• NFS StorageClass Example

[!IMPORTANT] ClusterID and ClientProfile Mapping

The clusterID parameter must match your ClientProfile CR name:

# In your StorageClass
parameters:
  clusterID: storage  # Must match the ClientProfile name from step 6

This is the key difference from legacy Ceph-CSI deployments where clusterID was arbitrary.

10. Create VolumeSnapshotClasses (Optional)

For snapshot support, use the upstream Ceph-CSI VolumeSnapshotClass examples:

• RBD VolumeSnapshotClass Example
• CephFS VolumeSnapshotClass Example
• NFS VolumeSnapshotClass Example

Ensure the clusterID parameter matches your ClientProfile name:

parameters:
  clusterID: storage  # Must match your ClientProfile name

11. Verify Storage Provisioning

Test your setup using the Ceph-CSI PVC examples:

• RBD PVC Example
• CephFS PVC Example
• NFS PVC Example

The PVC should reach Bound status, indicating successful provisioning.

---

12. Upgrade Ceph-CSI Operator and Drivers

To upgrade to a newer version:

Step 1: Fetch and Checkout the Latest Tag

git fetch --tags
git tag -l                    # List available tags
git checkout v1.0.0           # Replace with desired version

Step 2: Apply Updated Manifests

For Kubernetes (All-in-One)

kubectl apply -f deploy/all-in-one/install.yaml

For OpenShift (All-in-One)

kubectl apply -f deploy/all-in-one/install-openshift.yaml

For Multi-File Installation

kubectl apply -f deploy/multifile/crd.yaml
kubectl apply -f deploy/multifile/operator.yaml -n ceph-csi-operator-system

Step 3: Verify the Upgrade

kubectl get pods -n ceph-csi-operator-system

Ensure all pods are running and using the upgraded version.

---

13. Clean Up Resources

Step 1: Delete Custom Resources

kubectl delete cephconnection ceph-connection -n ceph-csi-operator-system
kubectl delete clientprofile storage -n ceph-csi-operator-system
kubectl delete driver rbd.csi.ceph.com -n ceph-csi-operator-system
kubectl delete driver cephfs.csi.ceph.com -n ceph-csi-operator-system
kubectl delete driver nfs.csi.ceph.com -n ceph-csi-operator-system

Step 2: Uninstall the Operator

For Kubernetes

kubectl delete -f deploy/all-in-one/install.yaml

For OpenShift

kubectl delete -f deploy/all-in-one/install-openshift.yaml

For Multi-File Installation

kubectl delete -f deploy/multifile/operator.yaml -n ceph-csi-operator-system
kubectl delete -f deploy/multifile/csi-rbac.yaml -n ceph-csi-operator-system
kubectl delete -f deploy/multifile/openshift-scc.yaml  # OpenShift only
kubectl delete -f deploy/multifile/crd.yaml

Step 3: Verify Deletion

kubectl get pods -n ceph-csi-operator-system

Expected output:

No resources found in ceph-csi-operator-system namespace.

6. NetworkPolicies

NetworkPolicies are included in all generated manifests by default.

• The operator pod gets a policy that denies all ingress and allows open egress (required for API server access).
• Each controller-plugin pod gets a policy allowing ingress only from the csi-addons controller-manager on the csi-addons gRPC port, and (for RBD) from any pod on port 50051 for snapshot-metadata gRPC.
• Each csi-addons nodeplugin pod (when deployCsiAddons: true) gets a policy allowing ingress only from the csi-addons controller-manager on port 9071.
• The node-plugin DaemonSet runs with hostNetwork: true and is exempt from NetworkPolicies.

Driver pod NetworkPolicies are created by the operator for every reconciled driver.

For design details, see docs/design/network-policy.md.