Velero-Restic backup not restoring VM to Ceph-backed PV

[briantopping]

Hello experts, my first time here! Love the product, thank you for all the hard work on it!

I'm running Velero 1.5 and created a backup of a cluster to Rook/Ceph RGW that I want to restore to a new (different) cluster. The new cluster is able to restore the API objects just fine, but there's some problem restoring the PVs.

What I am seeing is the PV object is created but there is no Ceph RBD volume created in the pool at the CSI volumeHandle in the PV object. I'm guessing this is the volumeHandle that was used in the original backup. There are no additional volumes created after the restore and no errors in the logs or anomaly messages that I can discern. Since I'm new to Velero, I've only gotten a little into the code and @dsu-igeek suggested I post here for some guidance.

What I'm trying to verify first are the subsystems that are involved with the volume creation. It's a new cluster with remote storage and the first time I've built one this way, so I haven't ruled out some nuance of this configuration. I have verified that creating a basic PVC from the new cluster generates the RBD volume just fine, but Velero isn't able to for some reason.

What I am wondering is if Velero needs additional Ceph permissions that aren't required for happy-path PVC generation. Here's what I have now if a reader happens to be a Ceph expert:

	caps: [mgr] allow command config
	caps: [mon] allow r, allow command quorum_status
	caps: [osd] allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow x pool=default.rgw.buckets.index, allow x pool=default.rgw.log

It's a rather complex configuration and am not expecting anyone to know Ceph, so I wanted to discover what Velero is doing to recreate the volumes so I can do the work to find out what's breaking.

If I am able to solve this over the weekend, I'll post here for those that follow.

Thanks!

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: initial
  namespace: velero
spec:
  backupName: test
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  includedNamespaces:
  - vpn
  restorePVs: true

[briantopping]

I forgot that the CSI watchers are supposed to create the backing image when the PV is created (brain fart). As it turns out, Rook depends on storage.kubernetes.io/csiProvisionerIdentity in .spec.csi.volumeAttributes. If this value does not match between the backup cluster and the restore cluster, the PV is ignored, even if the storageClassName matches. I haven't discovered whether it is possible to change this in the target cluster deployment.

Of course, the PV is immutable, so its impossible change the csiProvisionerIdentity without manually deleting it and recreating it. Once I did, the image was created as expected, but there would be no way to do this for Velero while it's doing a restore.

I initially documented above that the target cluster for the migration has subtle differences. That being the case, the right thing seems to be to manually migrate the volumes to this new configuration, then back that up with Velero instead of this patchwork restore. I think what I learned is migration really needs to be on exactly the same foundation so that the restore can run freely, including cloning the Rook foundation such that it is an exact copy and includes keys such as csiProvisionerIdentity so that things work smoothly. Good stuff!