Velero Backup ignores PVC? (Ceph RBD CSI)

[mame-leadec]

Hello everyone

ive setup a CephCluster via rook/helm and configured the RBD CSI driver.
I created a demo POD and PVC and tried to back it up via Velero. The backup runs succesfully but completly ignores the attached PVC.

./velero backup create nginx-backup5 --snapshot-volumes=true --snapshot-move-data --selector app=csirbd-demo-pod

Can someone point me in the right direction why velero ignores to backup the PVC?

Velero installed with (bucketname redacted):

./velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.2.1,velero/velero-plugin-for-csi:v0.6.0 \
    --bucket <my-bucket-name> \
    --secret-file ../credentials-velero \
    --use-volume-snapshots=true \
    --use-node-agent \
    --features=EnableCSI \
    --backup-location-config region="eu-central-1",s3ForcePathStyle="false"

part of the rook/helm ceph_values.yaml

csi:
  # -- Disable Ceph FS driver
  enableCephfsDriver: false
  # -- Enable Ceph CSI RBD driver
  enableRbdDriver: true
  # -- Enable Snapshotter in NFS provisioner pod
  enableNFSSnapshotter: false
  # -- Enable Snapshotter in RBD provisioner pod
  enableRBDSnapshotter: true
  # -- Enable Snapshotter in CephFS provisioner pod
  enableCephfsSnapshotter: false
  # -- Policy for modifying a volume's ownership or permissions when the RBD PVC is being mounted.
  # supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html
  rbdFSGroupPolicy: "File"

part of the rook/helm ceph_cluster_values.yaml

cephBlockPoolsVolumeSnapshotClass:
  enabled: true
  name: ceph-block
  isDefault: true
  deletionPolicy: Delete
  annotations: {}
  labels:
    velero.io/csi-volumesnapshot-class: "true"
  # see https://rook.io/docs/rook/v1.10/Storage-Configuration/Ceph-CSI/ceph-csi-snapshot/#rbd-snapshots for available configuration
  parameters: {}

installed volumesnapshots CRDs with:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml

POD & PVC

---
apiVersion: v1
kind: Pod
metadata:
  name: csirbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx
      volumeMounts:
        - name: mypvc
          mountPath: /var/lib/www/html
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 2Gi
  storageClassName: ceph-block

output for ./velero backup describe nginx-backup5 --details

Name:         nginx-backup5
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.28.6
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=28

Phase:  Completed


Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  app=csirbd-demo-pod

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  true
Snapshot Move Data:          true
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2024-05-23 17:02:21 +0200 CEST
Completed:  2024-05-23 17:02:25 +0200 CEST

Expiration:  2024-06-22 17:02:20 +0200 CEST

Total items to be backed up:  7
Items backed up:              7

Resource List:
  v1/Namespace:
    - default
    - ingress-nginx
    - kube-node-lease
    - kube-public
    - kube-system
    - rook-ceph
    - velero

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots: <none included>

  Pod Volume Backups: <none included>

HooksAttempted:  0
HooksFailed:     0

log output from backup created with ./velero backup logs nginx-backup5 > debuglog_backup.log
debuglog_backup.log

[sseago]

You specified a backup with a label selector:

Label selector:  app=csirbd-demo-pod

This means that velero will only back up items with a matching label. From what you specified above, it looks like neither your pod nor your PVC have this label. Backup included 7 items -- looks like only the namespaces in the cluster were included -- if you want to limit backups by label selector, then you'll need to make sure everything you want to back up has a matching label. If you don't specify a label selector, then the item list won't be filtered by label, so it will only be filtered based on included/excluded namespaces and resource types.

[mame-leadec]

Thank you so much. I experimented with CephFS CSI before and something got lost during copy&paste...