EKS 1.17: Persistent volume is not a supported volume type for snapshots

[supasteev0]

What steps did you take and what happened:
I recently noticed that all persistent volumes created with the ebs.csi.aws.com driver are not snapshotted anymore.
Indeed I get the following message in Velero: "Persistent volume is not a supported volume type for snapshots, skipping."
Old volumes created with kubernetes.io/aws-ebs are still correctly snasphotted though.
Am I right in thinking that velero-plugin-for-csi plugin and Snapshot k8s feature gate enabled is now mandatory to be able to backup EBS PV in AWS via Velero ? Or I am missing something ?

What did you expect to happen:
New volumes created with latest EBS CSI driver version (v0.6.0) snapshotted correctly by Velero backup.

The output of the following commands will help us better understand what's going on:

Here is the Backup configuration:

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: backup-sandbox
  namespace: velero
spec:
  hooks: {}
  includedNamespaces:
  - default
  includedResources:
  - persistentvolumeclaims
  - persistentvolumes
  includeClusterResources: true
  labelSelector:
    matchLabels:
      app: myapp
  snapshotVolumes: true
  ttl: 168h0m0s

Here is the Velero output when creating a backup for a PV created with ebs.csi.aws.com driver:

2020-08-25 16:58:32 | time="2020-08-25T14:58:32Z" level=info msg="Backup completed" controller=backup logSource="pkg/controller/backup_controller.go:619"
2020-08-25 16:58:32 | time="2020-08-25T14:58:32Z" level=info msg="Backed up a total of 2 items" backup=velero/nuxeo-backup-sandbox logSource="pkg/backup/backup.go:436" progress=
2020-08-25 16:58:32 | time="2020-08-25T14:58:32Z" level=info msg="Backed up 2 items out of an estimated total of 2 (estimate will change throughout the backup)" backup=velero/nuxeo-backup-sandbox logSource="pkg/backup/backup.go:411" name=nuxeo-data namespace=default progress= resource=persistentvolumeclaims
2020-08-25 16:58:32 | time="2020-08-25T14:58:32Z" level=info msg="Persistent volume is not a supported volume type for snapshots, skipping." backup=velero/nuxeo-backup-sandbox logSource="pkg/backup/item_backupper.go:457" name=pvc-2eb8d80e-4e79-45ff-a4a6-a74cd1677dd6 namespace= persistentVolume=pvc-2eb8d80e-4e79-45ff-a4a6-a74cd1677dd6 resource=persistentvolumes
2020-08-25 16:58:32 | time="2020-08-25T14:58:32Z" level=info msg="No volume ID returned by volume snapshotter for persistent volume" backup=velero/nuxeo-backup-sandbox logSource="pkg/backup/item_backupper.go:446" name=pvc-2eb8d80e-4e79-45ff-a4a6-a74cd1677dd6 namespace= persistentVolume=pvc-2eb8d80e-4e79-45ff-a4a6-a74cd1677dd6 resource=persistentvolumes volumeSnapshotLocation=default
2020-08-25 16:58:32 | time="2020-08-25T14:58:32Z" level=info msg="label \"failure-domain.beta.kubernetes.io/zone\" is not present on PersistentVolume" backup=velero/nuxeo-backup-sandbox logSource="pkg/backup/item_backupper.go:423" name=pvc-2eb8d80e-4e79-45ff-a4a6-a74cd1677dd6 namespace= persistentVolume=pvc-2eb8d80e-4e79-45ff-a4a6-a74cd1677dd6 resource=persistentvolumes
2020-08-25 16:58:32 | time="2020-08-25T14:58:32Z" level=info msg="label \"topology.kubernetes.io/zone\" is not present on PersistentVolume, checking deprecated label..." backup=velero/nuxeo-backup-sandbox logSource="pkg/backup/item_backupper.go:420" name=pvc-2eb8d80e-4e79-45ff-a4a6-a74cd1677dd6 namespace= persistentVolume=pvc-2eb8d80e-4e79-45ff-a4a6-a74cd1677dd6 resource=persistentvolumes
2020-08-25 16:58:32 | time="2020-08-25T14:58:32Z" level=info msg="Executing takePVSnapshot" backup=velero/nuxeo-backup-sandbox logSource="pkg/backup/item_backupper.go:393" name=pvc-2eb8d80e-4e79-45ff-a4a6-a74cd1677dd6 namespace= resource=persistentvolumes
2020-08-25 16:58:32 | time="2020-08-25T14:58:32Z" level=info msg="Backing up item" backup=velero/nuxeo-backup-sandbox logSource="pkg/backup/item_backupper.go:119" name=pvc-2eb8d80e-4e79-45ff-a4a6-a74cd1677dd6 namespace= resource=persistentvolumes
2020-08-25 16:58:32 | time="2020-08-25T14:58:32Z" level=info msg="Executing PVCAction" backup=velero/nuxeo-backup-sandbox cmd=/velero logSource="pkg/backup/backup_pv_action.go:49" pluginName=velero
2020-08-25 16:58:32 | time="2020-08-25T14:58:32Z" level=info msg="Executing custom action" backup=velero/nuxeo-backup-sandbox logSource="pkg/backup/item_backupper.go:323" name=nuxeo-data namespace=default resource=persistentvolumeclaims
2020-08-25 16:58:32 | time="2020-08-25T14:58:32Z" level=info msg="Backing up item" backup=velero/nuxeo-backup-sandbox logSource="pkg/backup/item_backupper.go:119" name=nuxeo-data namespace=default resource=persistentvolumeclaims
 2020-08-25 16:58:32 | time="2020-08-25T14:58:32Z" level=info msg="Processing item" backup=velero/nuxeo-backup-sandbox logSource="pkg/backup/backup.go:371" name=nuxeo-data namespace=default progress= resource=persistentvolumeclaims
2020-08-25 16:58:32 | time="2020-08-25T14:58:32Z" level=info msg="Collected 1 items matching the backup spec from the Kubernetes API (actual number of items backed up may be more or less depending on velero.io/exclude-from-backup annotation, plugins returning additional related items to back up, etc.)" backup=velero/nuxeo-backup-sandbox logSource="pkg/backup/backup.go:295" progress=

Environment:

• Velero version (use velero version): 1.4.0
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version): 1.17
• Kubernetes installer & version: EKS
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[ashish-amarnath]

From the logs you posted and the corresponding code, it looks like you don't have a matching VolumeSnapshotLocation.
The loop at https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_backupper.go#L444 isn't getting run causing volumeSnapshotter to be nil, resulting in the error message at https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_backupper.go#L469.

Can you please confirm that you have VolumeSnapshotLocation if you did, can you please share the output of:

kubectl -n velero get backup <BACKUP_NAME> -oyaml
kubectl -n velero get volumesnapshotlocations -oyaml

[supasteev0]

Hi,

I confirm I have a VolumeSnapshotLocation configured.

Here is the detail of the backup:

apiVersion: velero.io/v1
kind: Backup
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"velero.io/v1","kind":"Backup","metadata":{"annotations":{},"labels":{"env":"sandbox","stack":"drb"},"name":"nuxeo-backup-sandbox","namespace":"velero"},"spec":{"hooks":{},"includeClusterResources":true,"includedNamespaces":["default"],"includedResources":["persistentvolumeclaims","persistentvolumes"],"labelSelector":{"matchLabels":{"app":"nuxeo"}},"snapshotVolumes":true,"ttl":"168h0m0s"}}
    velero.io/source-cluster-k8s-gitversion: v1.17.9-eks-4c6976
    velero.io/source-cluster-k8s-major-version: "1"
    velero.io/source-cluster-k8s-minor-version: 17+
  creationTimestamp: "2020-08-25T14:58:31Z"
  generation: 5
  labels:
    env: sandbox
    stack: drb
    velero.io/storage-location: default
  name: nuxeo-backup-sandbox
  namespace: velero
  resourceVersion: "13251539"
  selfLink: /apis/velero.io/v1/namespaces/velero/backups/nuxeo-backup-sandbox
  uid: 78631568-aac5-4d4b-9d8a-58c9b3ec829b
spec:
  hooks: {}
  includeClusterResources: true
  includedNamespaces:
  - default
  includedResources:
  - persistentvolumeclaims
  - persistentvolumes
  labelSelector:
    matchLabels:
      app: nuxeo
  snapshotVolumes: true
  storageLocation: default
  ttl: 168h0m0s
  volumeSnapshotLocations:
  - default
status:
  completionTimestamp: "2020-08-25T14:58:32Z"
  expiration: "2020-09-01T14:58:31Z"
  formatVersion: 1.1.0
  phase: Completed
  progress:
    itemsBackedUp: 2
    totalItems: 2
  startTimestamp: "2020-08-25T14:58:31Z"
  version: 1

And the volumesnapshotlocation:

apiVersion: v1
items:
- apiVersion: velero.io/v1
  kind: VolumeSnapshotLocation
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"velero.io/v1","kind":"VolumeSnapshotLocation","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"velero"},"name":"default","namespace":"velero"},"spec":{"config":{"region":"eu-west-1"},"provider":"aws"}}
    creationTimestamp: "2020-08-25T14:56:07Z"
    generation: 1
    labels:
      app.kubernetes.io/name: velero
    name: default
    namespace: velero
    resourceVersion: "13251022"
    selfLink: /apis/velero.io/v1/namespaces/velero/volumesnapshotlocations/default
    uid: bd8991a0-6162-43fd-9566-861111ce6934
  spec:
    config:
      region: eu-west-1
    provider: aws
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

I am using kube2iam on velero deployment to allow Velero to access S3 backup bucket and create snapshots with the following policy:

resource "aws_iam_policy" "eks-backup-volumes" {
  name = "eks-backup-volumes"
  path = "/"
  description = "Allow access to volumes and snapshots"
  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
            "ec2:DescribeVolumes",
            "ec2:DescribeSnapshots",
            "ec2:CreateTags",
            "ec2:CreateVolume",
            "ec2:CreateSnapshot",
            "ec2:DeleteSnapshot"
        ],
        "Resource": "*"
        }
    ]
}
EOF
}

And here is the AWS plugin configuration in my velero deployment:

      initContainers:
        - image: velero/velero-plugin-for-aws:v1.1.0
          imagePullPolicy: IfNotPresent
          name: velero-plugin-for-aws
          volumeMounts:
          - mountPath: /target
            name: plugins

[carlisia]

This discussion has been moved to an issue: #3341.