Documentation Update for OpenShift IBM Cloud for CSI snapshot data movement #8069
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Updates the documentation for CSI snapshot data movement for OpenShift on IBM Cloud. The default hostpath /var/lib/kubelet/pods cannot find PersistentVolumeClaims with volumeMode: Block on host. The correct hostpath for OpenShift on IBM Cloud is /var/data/kubelet/pods. Signed-off-by: Michael Fruchtman <[email protected]>
|
/kind changelog-not-required |
github-actions
bot
added
the
kind/changelog-not-required
Signed-off-by: Michael Fruchtman <[email protected]>
|
nice @msfrucht! I'll make sure we doc this in downstream OADP as well. |
Lyndon-Li
previously approved these changes
Aug 1, 2024
kaovilai
approved these changes
Aug 1, 2024
|
|
||
| ```yaml | ||
| hostPath: | ||
| path: /var/data/kubet/pods |
There was a problem hiding this comment.
@msfrucht Shouldn't this be kubelet ? (s/kubet/kubelet)
There was a problem hiding this comment.
Yes. My error. I'll get that fixed.
There was a problem hiding this comment.
I believe I've resolved the error and looked over the changes again to check for any other issues.
Signed-off-by: Michael Fruchtman <[email protected]>
shubham-pampattiwar
approved these changes
Aug 1, 2024
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #8069 +/- ##
==========================================
+ Coverage 58.79% 58.80% +0.01%
==========================================
Files 357 358 +1
Lines 29841 30085 +244
==========================================
+ Hits 17545 17692 +147
- Misses 10851 10945 +94
- Partials 1445 1448 +3 ☔ View full report in Codecov by Sentry. |
blackpiglet
approved these changes
Aug 2, 2024
msfrucht
added a commit
to msfrucht/openshift-velero
that referenced
this pull request
Aug 2, 2024
I have to admit to some significant error and embarrassment regarding the documentation update about Openshift on IBM Cloud pull request vmware-tanzu#8069. I will correct my error before it gets any further. Just exposing /var/data/kubelet/pods is incorrect and host path /var/lib/kubelet/pods should remain unchanged. The errors with the defaults during csi snapshot data movement were: data path backup failed: Failed to run kopia backup: unable to get local block device entry: resolveSymlink: lstat /var/data/: no such file or directory I suspected this was the same as RancherOS and Nutanix. It is not. The original tested changes changed both /var/lib/kubelet/{pods,plugins} to /var/data/kubelet/{pods,plugins}. The published changes only result in the error: ``` status: completionTimestamp: '2024-08-02T17:12:29Z' message: >- data path backup failed: Failed to run kopia backup: unable to get local block device entry: resolveSymlink: lstat /var/data/kubelet/plugins: no such file or directory node: 10.240.0.5 phase: Failed progress: {} startTimestamp: '2024-08-02T17:12:11Z' ``` After making continued modifications to the daemonset the correct configuration was: ``` volumeMounts: - name: host-pods mountPath: /host_pods mountPropagation: HostToContainer - name: host-plugins mountPath: /var/data/kubelet/plugins mountPropagation: HostToContainer ``` ``` volumes: - name: host-pods hostPath: path: /var/lib/kubelet/pods type: '' - name: host-plugins hostPath: path: /var/data/kubelet/plugins type: '' ``` Only the changes to the plugin path were required. The plugin path changes were required to both the mount path and the host path. Regardless of whether /var/lib/kubelet/pods or /var/data/kubelet/pods host path, backups and restore succeeded provided the plugin path was modified. ``` volumeMounts: - name: host-pods mountPath: /host_pods mountPropagation: HostToContainer - name: host-plugins mountPath: /var/data/kubelet/plugins mountPropagation: HostToContainer ``` ``` volumes: - name: host-pods hostPath: path: /var/data/kubelet/pods type: '' - name: host-plugins hostPath: path: /var/data/kubelet/plugins type: '' ``` After getting on-host access was able to confirm. Pods are at /var/lib/kubelet/pods. ``` ls /var/lib/kubelet/pods 07c0be63-335d-4cfb-b39f-816bc2fb32cd 51f31b3e-4710-4ef0-8626-5f1a78a624b2 a4802fd3-3b62-45a4-8f21-974880b6f92a cccb35c9-b4f9-4ca9-a697-736ae64f09ad 0a5d4366-7fa1-4525-9e45-a43a362b8542 558b0643-0661-4d4a-b03e-aac60c6ad710 a4b106fb-5b7b-48e5-828a-ea7b41ba0e59 ce1290e1-4330-4df6-8166-14784bcce930 ``` On host the volumes are in /var/data/kubelet/plugins. ``` ls /var/data/kubelet/plugins/kubernetes.io/csi/openshift-storage.cephfs.csi.ceph.com/231e04896c4f528efb95d23a3c153db9fc4a7206b7320f74443f30de7228dba5/globalmount/velero/backups/backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c/ backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-csi-volumesnapshotclasses.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-resource-list.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-csi-volumesnapshotcontents.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-results.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-csi-volumesnapshots.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-volumesnapshots.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-itemoperations.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c.tar.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-logs.gz velero-backup.json backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-podvolumebackups.json.gz ``` With the volume config changed to expose /var/data/kubelet/plugins for the plugin hostPath, the DataUploads and DataDownloads succeed for both Filesystem and Block mode PVCs. ``` status: completionTimestamp: '2024-08-02T17:23:33Z' node: 10.240.0.5 path: >- /host_pods/7fcb9d56-7885-437c-acd3-67db6b1ee8ae/volumeDevices/kubernetes.io~csi/pvc-47b91f56-db8c-44bf-9ecc-737170561b4b phase: Completed progress: bytesDone: 5368709120 totalBytes: 5368709120 snapshotID: 8faae36b3592fee4efbfad024f26033e startTimestamp: '2024-08-02T17:21:22Z' ``` ``` status: completionTimestamp: '2024-08-02T18:42:19Z' node: 10.240.0.5 phase: Completed progress: bytesDone: 5368709120 totalBytes: 5368709120 startTimestamp: '2024-08-02T18:41:00Z' ``` My apologies for the error. Signed-off-by: Michael Fruchtman <[email protected]>
3 tasks
sseago
pushed a commit
that referenced
this pull request
Aug 6, 2024
* Correcting Openshift on IBM Documentation Error I have to admit to some significant error and embarrassment regarding the documentation update about Openshift on IBM Cloud pull request #8069. I will correct my error before it gets any further. Just exposing /var/data/kubelet/pods is incorrect and host path /var/lib/kubelet/pods should remain unchanged. The errors with the defaults during csi snapshot data movement were: data path backup failed: Failed to run kopia backup: unable to get local block device entry: resolveSymlink: lstat /var/data/: no such file or directory I suspected this was the same as RancherOS and Nutanix. It is not. The original tested changes changed both /var/lib/kubelet/{pods,plugins} to /var/data/kubelet/{pods,plugins}. The published changes only result in the error: ``` status: completionTimestamp: '2024-08-02T17:12:29Z' message: >- data path backup failed: Failed to run kopia backup: unable to get local block device entry: resolveSymlink: lstat /var/data/kubelet/plugins: no such file or directory node: 10.240.0.5 phase: Failed progress: {} startTimestamp: '2024-08-02T17:12:11Z' ``` After making continued modifications to the daemonset the correct configuration was: ``` volumeMounts: - name: host-pods mountPath: /host_pods mountPropagation: HostToContainer - name: host-plugins mountPath: /var/data/kubelet/plugins mountPropagation: HostToContainer ``` ``` volumes: - name: host-pods hostPath: path: /var/lib/kubelet/pods type: '' - name: host-plugins hostPath: path: /var/data/kubelet/plugins type: '' ``` Only the changes to the plugin path were required. The plugin path changes were required to both the mount path and the host path. Regardless of whether /var/lib/kubelet/pods or /var/data/kubelet/pods host path, backups and restore succeeded provided the plugin path was modified. ``` volumeMounts: - name: host-pods mountPath: /host_pods mountPropagation: HostToContainer - name: host-plugins mountPath: /var/data/kubelet/plugins mountPropagation: HostToContainer ``` ``` volumes: - name: host-pods hostPath: path: /var/data/kubelet/pods type: '' - name: host-plugins hostPath: path: /var/data/kubelet/plugins type: '' ``` After getting on-host access was able to confirm. Pods are at /var/lib/kubelet/pods. ``` ls /var/lib/kubelet/pods 07c0be63-335d-4cfb-b39f-816bc2fb32cd 51f31b3e-4710-4ef0-8626-5f1a78a624b2 a4802fd3-3b62-45a4-8f21-974880b6f92a cccb35c9-b4f9-4ca9-a697-736ae64f09ad 0a5d4366-7fa1-4525-9e45-a43a362b8542 558b0643-0661-4d4a-b03e-aac60c6ad710 a4b106fb-5b7b-48e5-828a-ea7b41ba0e59 ce1290e1-4330-4df6-8166-14784bcce930 ``` On host the volumes are in /var/data/kubelet/plugins. ``` ls /var/data/kubelet/plugins/kubernetes.io/csi/openshift-storage.cephfs.csi.ceph.com/231e04896c4f528efb95d23a3c153db9fc4a7206b7320f74443f30de7228dba5/globalmount/velero/backups/backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c/ backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-csi-volumesnapshotclasses.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-resource-list.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-csi-volumesnapshotcontents.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-results.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-csi-volumesnapshots.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-volumesnapshots.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-itemoperations.json.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c.tar.gz backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-logs.gz velero-backup.json backup-resources-41d84d16-47a7-4ea8-a9cb-6348d01bcb2c-podvolumebackups.json.gz ``` With the volume config changed to expose /var/data/kubelet/plugins for the plugin hostPath, the DataUploads and DataDownloads succeed for both Filesystem and Block mode PVCs. ``` status: completionTimestamp: '2024-08-02T17:23:33Z' node: 10.240.0.5 path: >- /host_pods/7fcb9d56-7885-437c-acd3-67db6b1ee8ae/volumeDevices/kubernetes.io~csi/pvc-47b91f56-db8c-44bf-9ecc-737170561b4b phase: Completed progress: bytesDone: 5368709120 totalBytes: 5368709120 snapshotID: 8faae36b3592fee4efbfad024f26033e startTimestamp: '2024-08-02T17:21:22Z' ``` ``` status: completionTimestamp: '2024-08-02T18:42:19Z' node: 10.240.0.5 phase: Completed progress: bytesDone: 5368709120 totalBytes: 5368709120 startTimestamp: '2024-08-02T18:41:00Z' ``` My apologies for the error. Signed-off-by: Michael Fruchtman <[email protected]> * Add context to plugins mountPath Signed-off-by: Michael Fruchtman <[email protected]> --------- Signed-off-by: Michael Fruchtman <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please add a summary of your change
Updates the documentation for CSI snapshot data movement for OpenShift on IBM Cloud.
The default hostpath /var/lib/kubelet/pods cannot find PersistentVolumeClaims with volumeMode: Block on host.
The correct hostpath for OpenShift on IBM Cloud is /var/data/kubelet/pods
Does your change fix a particular issue?
Fixes #(issue)
Documentation only. Fixes failed CSI snapshot movement for OpenShift on IBM Cloud
Please indicate you've done the following:
/kind changelog-not-requiredas a comment on this pull request.site/content/docs/main.