PVC backup/restore only

[s4ndalHat]

Hello everyone,

Wanted to ask, does Velero provide a backup/restore mechanism on PVC only ?

Use case:
I deploy kubernetes clusters and applications using ArgoCD so almost every resources are handled by gitOps.
Sometimes, I don't want the cluster resources design, but I only need the source PVC data to inject into the new deployment.

I currently use fs-backup, but we planned to use CSI snaspshot backup with ceph.

If you have any advice, it would be appreciated 🙏🏾 .

Thank you

[ehemmerlin]

Hello @s4ndalHat, I don't know if you can only backup and restore PVCs using Velero by excluding the cluster resources. Nevertheless, I want to address this other point you are mentioning: I encourage you to move from file system backup to CSI snapshots, as it fixed some issues we faced about consistency data of Kubernetes statefulsets. Velero team did a great job on this recently. More here: https://velero.io/docs/main/csi/

[s4ndalHat]

Hi @ehemmerlin, thanks for your reply.
I'm trying to make it work, but I always face a bunch of warnings during the operation (although it succeed) like below:

Warnings:
  Velero:    message: /VolumeSnapshotContent snapcontent-62c307ad-281d-4c9e-92e9-e4afca44ef53 has error: Failed to create snapshot: failed to take snapshot of the volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9: "rpc error: code = Internal desc = failed to create snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9 on backend blk-ssd-data02: could not find snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for source volume HOMOL_BLK_SDD_DATA02_pvc_a6a45b98_fe9a_4a0c_a8ed_3a49aa8899d9"
             message: /VolumeSnapshotContent snapcontent-62c307ad-281d-4c9e-92e9-e4afca44ef53 has error: Failed to create snapshot: failed to take snapshot of the volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9: "rpc error: code = Internal desc = failed to create snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9 on backend blk-ssd-data02: could not find snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for source volume HOMOL_BLK_SDD_DATA02_pvc_a6a45b98_fe9a_4a0c_a8ed_3a49aa8899d9"
             message: /VolumeSnapshotContent snapcontent-62c307ad-281d-4c9e-92e9-e4afca44ef53 has error: Failed to create snapshot: failed to take snapshot of the volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9: "rpc error: code = Internal desc = failed to create snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9 on backend blk-ssd-data02: could not find snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for source volume HOMOL_BLK_SDD_DATA02_pvc_a6a45b98_fe9a_4a0c_a8ed_3a49aa8899d9"
             message: /VolumeSnapshotContent snapcontent-62c307ad-281d-4c9e-92e9-e4afca44ef53 has error: Failed to create snapshot: failed to take snapshot of the volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9: "rpc error: code = Internal desc = failed to create snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9 on backend blk-ssd-data02: could not find snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for source volume HOMOL_BLK_SDD_DATA02_pvc_a6a45b98_fe9a_4a0c_a8ed_3a49aa8899d9"
             message: /VolumeSnapshotContent snapcontent-62c307ad-281d-4c9e-92e9-e4afca44ef53 has error: Failed to create snapshot: failed to take snapshot of the volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9: "rpc error: code = Internal desc = failed to create snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9 on backend blk-ssd-data02: could not find snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for source volume HOMOL_BLK_SDD_DATA02_pvc_a6a45b98_fe9a_4a0c_a8ed_3a49aa8899d9"
             message: /VolumeSnapshotContent snapcontent-62c307ad-281d-4c9e-92e9-e4afca44ef53 has error: Failed to create snapshot: failed to take snapshot of the volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9: "rpc error: code = Internal desc = failed to create snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9 on backend blk-ssd-data02: could not find snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for source volume HOMOL_BLK_SDD_DATA02_pvc_a6a45b98_fe9a_4a0c_a8ed_3a49aa8899d9"
             message: /VolumeSnapshotContent snapcontent-62c307ad-281d-4c9e-92e9-e4afca44ef53 has error: Failed to create snapshot: failed to take snapshot of the volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9: "rpc error: code = Internal desc = failed to create snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9 on backend blk-ssd-data02: could not find snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for source volume HOMOL_BLK_SDD_DATA02_pvc_a6a45b98_fe9a_4a0c_a8ed_3a49aa8899d9"
             message: /VolumeSnapshotContent snapcontent-62c307ad-281d-4c9e-92e9-e4afca44ef53 has error: Failed to create snapshot: failed to take snapshot of the volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9: "rpc error: code = Internal desc = failed to create snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9 on backend blk-ssd-data02: could not find snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for source volume HOMOL_BLK_SDD_DATA02_pvc_a6a45b98_fe9a_4a0c_a8ed_3a49aa8899d9"
             message: /VolumeSnapshotContent snapcontent-62c307ad-281d-4c9e-92e9-e4afca44ef53 has error: Failed to create snapshot: failed to take snapshot of the volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9: "rpc error: code = Internal desc = failed to create snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9 on backend blk-ssd-data02: could not find snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for source volume HOMOL_BLK_SDD_DATA02_pvc_a6a45b98_fe9a_4a0c_a8ed_3a49aa8899d9"
             message: /VolumeSnapshotContent snapcontent-62c307ad-281d-4c9e-92e9-e4afca44ef53 has error: Failed to create snapshot: failed to take snapshot of the volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9: "rpc error: code = Internal desc = failed to create snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9 on backend blk-ssd-data02: could not find snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for source volume HOMOL_BLK_SDD_DATA02_pvc_a6a45b98_fe9a_4a0c_a8ed_3a49aa8899d9"
             message: /VolumeSnapshotContent snapcontent-62c307ad-281d-4c9e-92e9-e4afca44ef53 has error: Failed to create snapshot: failed to take snapshot of the volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9: "rpc error: code = Internal desc = failed to create snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for volume pvc-a6a45b98-fe9a-4a0c-a8ed-3a49aa8899d9 on backend blk-ssd-data02: could not find snapshot snapshot-62c307ad-281d-4c9e-92e9-e4afca44ef53 for source volume HOMOL_BLK_SDD_DATA02_pvc_a6a45b98_fe9a_4a0c_a8ed_3a49aa8899d9"

and on some environment, this takes too long like 6 hours to backup 100Gi using CSI data mover...

I may misconfigured something ?

Thank you

[ehemmerlin]

You are welcome @s4ndalHat. I'm only a Velero user and not a maintainer, so I'm curious to read an answer to your question as we will soon have to backup 100Gi using CSI data mover too.

[blackpiglet]

@s4ndalHat
Could you help check which part causes the long time to back up the volume data?
In my opinion, the CSI data mover mainly has two parts.
First, take the snapshots of volumes with the CSI VolumeSnapshot, second, create the backup volume and backup pod with the just created snapshot, then upload the backup volume data by the Kopia uploader.

Usually, the first part shouldn't consume too much time.
It's possible the second part may take a long time. There are two ways to accelerate.

• Run multiple DataUploads/DataDownloads in a node-agent
• Support multiple uploads/downloads by Kopia

[s4ndalHat]

@blackpiglet ,

Thank you,

Yes, at the moment, we revert back to using the fs backup, but how can I check which part of the process is taking time ? I mean how can I monitor it ?

[blackpiglet]

If the snapshot creation is time-consuming, please check the VolumeSnapshot's creation time in the PVC namespace.
If the upload is slow, please check the DataUploads created in the Velero namespace's creation time.