PVC restores completed, however data is missing

[niels-s]

Hey folks 👋
We are deploying Velero in our cluster, and it's able to perform backups to an S3 bucket using the Restic provider (We are using Rook/Rook-Ceph for storage). However, when I try to verify everything is backed up with a restore like
velero restore create test --from-schedule daily --include-namespaces blog-production --namespace-mappings blog-production:blog-restore-test --include-resources persistentvolumeclaims,persistentvolumes

We notice the PVC is created successfully, however when verifying the data in use with our monitoring (Prometheus) or when attaching a deployment to the PVC, and it's empty.

When I check the logs of the restore, It too mentions it was successful and completed.

time="2021-01-19T15:14:01Z" level=info msg="Waiting for all restic restores to complete" logSource="pkg/restore/restore.go:470" restore=velero/test
time="2021-01-19T15:14:01Z" level=info msg="Done waiting for all restic restores to complete" logSource="pkg/restore/restore.go:486" restore=velero/test
time="2021-01-19T15:14:01Z" level=info msg="restore completed" logSource="pkg/controller/restore_controller.go:468" restore=velero/test

Any tips or tricks that could be wrong?

We are using Velero v1.4.2.

[dcplaya]

I also recently tried this and got the same exact result, restore "complete" but no actual data. I was also doing a namespace-mapping.

I then had the unfortunate experience of my cluster going down so I got to actually test a backup without namespace-mapping and it worked! I think there is an issue with namespace-mapping in combo with restic restores. The restic-helper init container never started, the backup --details showed the restic restore was "New" but it never actually started the restore.

I also think that even if Restic fails to start the restore, after a period of time, Velero just gives up but doesn't actually check the Restic status. Maybe it does as long as Restic starts but if Restic never starts (like in this case), it can't error so Velero just assumes Restic is happy.

[niels-s]

I tried on another cluster without the namespace mapping but no luck. I did found this line in the logs.

time="2021-02-01T13:52:15Z" level=info msg="Skipping resource persistentvolumes because it's cluster-scoped and only specific namespaces are included in the restore" logSource="pkg/restore/restore.go:697" restore=velero/test-3

Not sure if this has anything to do with it 🤷‍♂️ But the PV itself isn't restored while the PVC is and creates a PV anyway 😕
But I don't want to run a restore for all the cluster so not sure how to fix this 😞

[carlisia]

We will need to investigate this, I opened an issue for it, please subscribe and keep the discussions in the ticket: #3407

[dsu-igeek]

There's no need to specifically include the PVCs and PVs in your backup command (--include-resources persistentvolumeclaims,persistentvolumes) and that may be confusing things. The Restic backup will typically not backup the PV resource itself. Have you annotated all of the PVs to be backed up with Restic? Instruction here: https://velero.io/docs/v1.4/restic/

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Velero version (use velero version):
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):