This job view page is being replaced by Spyglass soon. Check out the new job view.
PRalice-zheyan-yu: [V2] refactor: combine azdrivernode and node_availability into one controller
ResultABORTED
Tests 0 failed / 0 succeeded
Started2022-11-29 22:02
Elapsed1h20m
Revisionfec06051c4296c3b1fa7923ebdb78ffea8f733f6
Refs 1612

No Test Failures!


Error lines from build-log.txt

... skipping 236 lines ...

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 11345  100 11345    0     0   125k      0 --:--:-- --:--:-- --:--:--  127k
Downloading https://get.helm.sh/helm-v3.10.2-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
docker pull k8sprow.azurecr.io/azuredisk-csi:latest-v2-5fb3153a08fb6a2968d7e7bbe760d033ea35fc7a || make container-all push-manifest
Error response from daemon: manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-5fb3153a08fb6a2968d7e7bbe760d033ea35fc7a not found: manifest unknown: manifest tagged by "latest-v2-5fb3153a08fb6a2968d7e7bbe760d033ea35fc7a" is not found
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=latest-v2-5fb3153a08fb6a2968d7e7bbe760d033ea35fc7a -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=5fb3153a08fb6a2968d7e7bbe760d033ea35fc7a -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2022-11-29T22:06:31Z -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/amd64/azurediskpluginv2.exe ./pkg/azurediskplugin
docker buildx rm container-builder || true
ERROR: no builder "container-builder" found
docker buildx create --use --name=container-builder
container-builder
# enable qemu for arm64 build
# https://github.com/docker/buildx/issues/464#issuecomment-741507760
docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64
Unable to find image 'tonistiigi/binfmt:latest' locally
... skipping 692 lines ...
         }
      }
   ]
}
make[2]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker pull k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-5fb3153a08fb6a2968d7e7bbe760d033ea35fc7a || make azdiskschedulerextender-all push-manifest-azdiskschedulerextender
Error response from daemon: manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-5fb3153a08fb6a2968d7e7bbe760d033ea35fc7a not found: manifest unknown: manifest tagged by "latest-v2-5fb3153a08fb6a2968d7e7bbe760d033ea35fc7a" is not found
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker buildx rm container-builder || true
container-builder removed
docker buildx create --use --name=container-builder
container-builder
# enable qemu for arm64 build
... skipping 1038 lines ...
                    type: string
                type: object
                oneOf:
                - required: ["persistentVolumeClaimName"]
                - required: ["volumeSnapshotContentName"]
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 60 lines ...
                    type: string
                  volumeSnapshotContentName:
                    description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable.
                    type: string
                type: object
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 254 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 108 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 359 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status Required
            properties:
              detail:
                description: Status summarizes the current attachment state of the
                  volume attachment Nil Status indicates that the volume has not yet
                  been attached to the node
                properties:
... skipping 7 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 90 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 13 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 169 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              detail:
                description: Current status detail of the AzVolume Nil detail indicates
                  that the volume has not been created
                properties:
                  accessible_topology:
... skipping 28 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 154 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 34 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 1063 lines ...
          image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0"
          args:
            - "-csi-address=$(ADDRESS)"
            - "-v=2"
            - "-leader-election"
            - "--leader-election-namespace=kube-system"
            - '-handle-volume-inuse-error=false'
            - '-feature-gates=RecoverVolumeExpansionFailure=true'
            - "-timeout=240s"
          env:
            - name: ADDRESS
              value: /csi/csi.sock
          volumeMounts:
... skipping 448 lines ...
Nov 29 22:16:12.999: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comlvmbn] to have phase Bound
Nov 29 22:16:13.108: INFO: PersistentVolumeClaim test.csi.azure.comlvmbn found but phase is Pending instead of Bound.
Nov 29 22:16:15.219: INFO: PersistentVolumeClaim test.csi.azure.comlvmbn found but phase is Pending instead of Bound.
Nov 29 22:16:17.329: INFO: PersistentVolumeClaim test.csi.azure.comlvmbn found and phase=Bound (4.330203277s)
STEP: Creating pod exec-volume-test-dynamicpv-scwr
STEP: Creating a pod to test exec-volume-test
Nov 29 22:16:17.659: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-scwr" in namespace "volume-3353" to be "Succeeded or Failed"
Nov 29 22:16:17.779: INFO: Pod "exec-volume-test-dynamicpv-scwr": Phase="Pending", Reason="", readiness=false. Elapsed: 119.91718ms
Nov 29 22:16:19.890: INFO: Pod "exec-volume-test-dynamicpv-scwr": Phase="Pending", Reason="", readiness=false. Elapsed: 2.230718768s
Nov 29 22:16:22.000: INFO: Pod "exec-volume-test-dynamicpv-scwr": Phase="Pending", Reason="", readiness=false. Elapsed: 4.340819266s
Nov 29 22:16:24.111: INFO: Pod "exec-volume-test-dynamicpv-scwr": Phase="Pending", Reason="", readiness=false. Elapsed: 6.452230556s
Nov 29 22:16:26.221: INFO: Pod "exec-volume-test-dynamicpv-scwr": Phase="Pending", Reason="", readiness=false. Elapsed: 8.561964596s
Nov 29 22:16:28.333: INFO: Pod "exec-volume-test-dynamicpv-scwr": Phase="Pending", Reason="", readiness=false. Elapsed: 10.673986624s
Nov 29 22:16:30.444: INFO: Pod "exec-volume-test-dynamicpv-scwr": Phase="Pending", Reason="", readiness=false. Elapsed: 12.784553305s
Nov 29 22:16:32.554: INFO: Pod "exec-volume-test-dynamicpv-scwr": Phase="Pending", Reason="", readiness=false. Elapsed: 14.895099823s
Nov 29 22:16:34.665: INFO: Pod "exec-volume-test-dynamicpv-scwr": Phase="Pending", Reason="", readiness=false. Elapsed: 17.005709402s
Nov 29 22:16:36.775: INFO: Pod "exec-volume-test-dynamicpv-scwr": Phase="Pending", Reason="", readiness=false. Elapsed: 19.115566429s
Nov 29 22:16:38.885: INFO: Pod "exec-volume-test-dynamicpv-scwr": Phase="Pending", Reason="", readiness=false. Elapsed: 21.226038511s
Nov 29 22:16:40.996: INFO: Pod "exec-volume-test-dynamicpv-scwr": Phase="Succeeded", Reason="", readiness=false. Elapsed: 23.337047342s
STEP: Saw pod success
Nov 29 22:16:40.996: INFO: Pod "exec-volume-test-dynamicpv-scwr" satisfied condition "Succeeded or Failed"
Nov 29 22:16:41.107: INFO: Trying to get logs from node k8s-agentpool1-92105279-vmss000002 pod exec-volume-test-dynamicpv-scwr container exec-container-dynamicpv-scwr: <nil>
STEP: delete the pod
Nov 29 22:16:41.380: INFO: Waiting for pod exec-volume-test-dynamicpv-scwr to disappear
Nov 29 22:16:41.492: INFO: Pod exec-volume-test-dynamicpv-scwr no longer exists
STEP: Deleting pod exec-volume-test-dynamicpv-scwr
Nov 29 22:16:41.492: INFO: Deleting pod "exec-volume-test-dynamicpv-scwr" in namespace "volume-3353"
... skipping 21 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should allow exec of files on the volume
    test/e2e/storage/testsuites/volumes.go:198
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should allow exec of files on the volume","total":31,"completed":1,"skipped":59,"failed":0}

SSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:280

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
... skipping 2 lines ...
STEP: Building a namespace api object, basename provisioning
W1129 22:16:12.100083   39947 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
Nov 29 22:16:12.100: INFO: Found PodSecurityPolicies; testing pod creation to see if PodSecurityPolicy is enabled
Nov 29 22:16:12.211: INFO: No PSP annotation exists on dry run pod; assuming PodSecurityPolicy is disabled
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:280
Nov 29 22:16:12.640: INFO: Creating resource for dynamic PV
Nov 29 22:16:12.640: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-1209-e2e-scccrf7
STEP: creating a claim
Nov 29 22:16:12.752: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 29 22:16:12.863: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comsvptj] to have phase Bound
Nov 29 22:16:12.999: INFO: PersistentVolumeClaim test.csi.azure.comsvptj found but phase is Pending instead of Bound.
Nov 29 22:16:15.127: INFO: PersistentVolumeClaim test.csi.azure.comsvptj found but phase is Pending instead of Bound.
Nov 29 22:16:17.236: INFO: PersistentVolumeClaim test.csi.azure.comsvptj found and phase=Bound (4.372876047s)
STEP: Creating pod pod-subpath-test-dynamicpv-hqvl
STEP: Checking for subpath error in container status
Nov 29 22:16:39.795: INFO: Deleting pod "pod-subpath-test-dynamicpv-hqvl" in namespace "provisioning-1209"
Nov 29 22:16:39.906: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-hqvl" to be fully deleted
STEP: Deleting pod
Nov 29 22:16:42.123: INFO: Deleting pod "pod-subpath-test-dynamicpv-hqvl" in namespace "provisioning-1209"
STEP: Deleting pvc
Nov 29 22:16:42.244: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comsvptj"
... skipping 16 lines ...

• [SLOW TEST:72.266 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:280
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]","total":34,"completed":1,"skipped":19,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode 
  should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299

[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 22:16:13.177: INFO: >>> kubeConfig: /root/tmp1562512509/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename volumemode
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299
Nov 29 22:16:13.947: INFO: Creating resource for dynamic PV
Nov 29 22:16:13.947: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass volumemode-3564-e2e-scl9h85
STEP: creating a claim
Nov 29 22:16:14.173: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comxxx9c] to have phase Bound
Nov 29 22:16:14.282: INFO: PersistentVolumeClaim test.csi.azure.comxxx9c found but phase is Pending instead of Bound.
Nov 29 22:16:16.393: INFO: PersistentVolumeClaim test.csi.azure.comxxx9c found but phase is Pending instead of Bound.
Nov 29 22:16:18.504: INFO: PersistentVolumeClaim test.csi.azure.comxxx9c found and phase=Bound (4.33148962s)
STEP: Creating pod
STEP: Waiting for the pod to fail
Nov 29 22:16:21.185: INFO: Deleting pod "pod-e2ce74b5-04f8-4781-844f-f86c42ffb550" in namespace "volumemode-3564"
Nov 29 22:16:21.301: INFO: Wait up to 5m0s for pod "pod-e2ce74b5-04f8-4781-844f-f86c42ffb550" to be fully deleted
STEP: Deleting pvc
Nov 29 22:16:23.521: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comxxx9c"
Nov 29 22:16:23.636: INFO: Waiting up to 5m0s for PersistentVolume pvc-b78557da-1aba-4d3f-9e6b-e27b06e62564 to get deleted
Nov 29 22:16:23.746: INFO: PersistentVolume pvc-b78557da-1aba-4d3f-9e6b-e27b06e62564 found and phase=Released (109.690743ms)
... skipping 20 lines ...

• [SLOW TEST:82.686 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow]
    test/e2e/storage/testsuites/volumemode.go:299
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should fail to use a volume in a pod with mismatched mode [Slow]","total":31,"completed":1,"skipped":45,"failed":0}

SSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] 
  should concurrently access the single volume from pods on the same node
  test/e2e/storage/testsuites/multivolume.go:298
... skipping 157 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:298
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":39,"completed":1,"skipped":50,"failed":0}

SSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:18:19.048: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 3 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:280

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 116 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:378
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":32,"completed":1,"skipped":99,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:19:49.513: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 3 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:269

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 199 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:138
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":31,"completed":2,"skipped":74,"failed":0}

SSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (ext4)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:20:54.575: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 122 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:378
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":30,"completed":1,"skipped":48,"failed":0}

SSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:21:20.669: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 73 lines ...
[It] should check snapshot fields, check restore correctly works, check deletion (ephemeral)
  test/e2e/storage/testsuites/snapshottable.go:177
Nov 29 22:17:36.643: INFO: Creating resource for dynamic PV
Nov 29 22:17:36.643: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass snapshotting-7598-e2e-sc86qcg
STEP: [init] starting a pod to use the claim
Nov 29 22:17:36.868: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-pg7k6" in namespace "snapshotting-7598" to be "Succeeded or Failed"
Nov 29 22:17:36.977: INFO: Pod "pvc-snapshottable-tester-pg7k6": Phase="Pending", Reason="", readiness=false. Elapsed: 109.264538ms
Nov 29 22:17:39.088: INFO: Pod "pvc-snapshottable-tester-pg7k6": Phase="Pending", Reason="", readiness=false. Elapsed: 2.220185256s
Nov 29 22:17:41.199: INFO: Pod "pvc-snapshottable-tester-pg7k6": Phase="Pending", Reason="", readiness=false. Elapsed: 4.33142324s
Nov 29 22:17:43.313: INFO: Pod "pvc-snapshottable-tester-pg7k6": Phase="Pending", Reason="", readiness=false. Elapsed: 6.445474878s
Nov 29 22:17:45.423: INFO: Pod "pvc-snapshottable-tester-pg7k6": Phase="Pending", Reason="", readiness=false. Elapsed: 8.555656817s
Nov 29 22:17:47.535: INFO: Pod "pvc-snapshottable-tester-pg7k6": Phase="Pending", Reason="", readiness=false. Elapsed: 10.666935422s
... skipping 30 lines ...
Nov 29 22:18:52.990: INFO: Pod "pvc-snapshottable-tester-pg7k6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.122724243s
Nov 29 22:18:55.100: INFO: Pod "pvc-snapshottable-tester-pg7k6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.232407807s
Nov 29 22:18:57.210: INFO: Pod "pvc-snapshottable-tester-pg7k6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.342428027s
Nov 29 22:18:59.335: INFO: Pod "pvc-snapshottable-tester-pg7k6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.466786731s
Nov 29 22:19:01.445: INFO: Pod "pvc-snapshottable-tester-pg7k6": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m24.57719088s
STEP: Saw pod success
Nov 29 22:19:01.445: INFO: Pod "pvc-snapshottable-tester-pg7k6" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
STEP: creating a SnapshotClass
STEP: creating a dynamic VolumeSnapshot
Nov 29 22:19:01.892: INFO: Waiting up to 5m0s for VolumeSnapshot snapshot-w5wnf to become ready
Nov 29 22:19:02.001: INFO: VolumeSnapshot snapshot-w5wnf found but is not ready.
Nov 29 22:19:04.110: INFO: VolumeSnapshot snapshot-w5wnf found but is not ready.
... skipping 40 lines ...
Nov 29 22:20:34.100: INFO: volumesnapshotcontents snapcontent-68936f31-9ef2-43b6-b65a-3ca179d91029 has been found and is not deleted
Nov 29 22:20:35.211: INFO: volumesnapshotcontents snapcontent-68936f31-9ef2-43b6-b65a-3ca179d91029 has been found and is not deleted
Nov 29 22:20:36.321: INFO: volumesnapshotcontents snapcontent-68936f31-9ef2-43b6-b65a-3ca179d91029 has been found and is not deleted
Nov 29 22:20:37.432: INFO: volumesnapshotcontents snapcontent-68936f31-9ef2-43b6-b65a-3ca179d91029 has been found and is not deleted
Nov 29 22:20:38.542: INFO: volumesnapshotcontents snapcontent-68936f31-9ef2-43b6-b65a-3ca179d91029 has been found and is not deleted
Nov 29 22:20:39.652: INFO: volumesnapshotcontents snapcontent-68936f31-9ef2-43b6-b65a-3ca179d91029 has been found and is not deleted
Nov 29 22:20:40.652: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 29 22:20:40.798: INFO: Pod restored-pvc-tester-xq6wp has the following logs: 
Nov 29 22:20:40.799: INFO: Deleting pod "restored-pvc-tester-xq6wp" in namespace "snapshotting-7598"
Nov 29 22:20:40.911: INFO: Wait up to 5m0s for pod "restored-pvc-tester-xq6wp" to be fully deleted
Nov 29 22:21:13.132: INFO: deleting snapshot "snapshotting-7598"/"snapshot-w5wnf"
... skipping 26 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works, check deletion (ephemeral)
        test/e2e/storage/testsuites/snapshottable.go:177
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Ephemeral Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works, check deletion (ephemeral)","total":31,"completed":2,"skipped":58,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral 
  should support two pods which have the same volume definition
  test/e2e/storage/testsuites/ephemeral.go:216
... skipping 85 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should support two pods which have the same volume definition
    test/e2e/storage/testsuites/ephemeral.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support two pods which have the same volume definition","total":33,"completed":1,"skipped":5,"failed":0}

SSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:21:23.068: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 273 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
    test/e2e/storage/testsuites/provisioning.go:208
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":34,"completed":2,"skipped":44,"failed":0}

SSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:21:23.896: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 38 lines ...
Nov 29 22:19:50.529: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com2zg68] to have phase Bound
Nov 29 22:19:50.638: INFO: PersistentVolumeClaim test.csi.azure.com2zg68 found but phase is Pending instead of Bound.
Nov 29 22:19:52.749: INFO: PersistentVolumeClaim test.csi.azure.com2zg68 found but phase is Pending instead of Bound.
Nov 29 22:19:54.859: INFO: PersistentVolumeClaim test.csi.azure.com2zg68 found and phase=Bound (4.32964075s)
STEP: Creating pod pod-subpath-test-dynamicpv-nmjj
STEP: Creating a pod to test subpath
Nov 29 22:19:55.191: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-nmjj" in namespace "provisioning-9199" to be "Succeeded or Failed"
Nov 29 22:19:55.303: INFO: Pod "pod-subpath-test-dynamicpv-nmjj": Phase="Pending", Reason="", readiness=false. Elapsed: 111.960931ms
Nov 29 22:19:57.412: INFO: Pod "pod-subpath-test-dynamicpv-nmjj": Phase="Pending", Reason="", readiness=false. Elapsed: 2.221500068s
Nov 29 22:19:59.523: INFO: Pod "pod-subpath-test-dynamicpv-nmjj": Phase="Pending", Reason="", readiness=false. Elapsed: 4.331783541s
Nov 29 22:20:01.632: INFO: Pod "pod-subpath-test-dynamicpv-nmjj": Phase="Pending", Reason="", readiness=false. Elapsed: 6.441142765s
Nov 29 22:20:03.743: INFO: Pod "pod-subpath-test-dynamicpv-nmjj": Phase="Pending", Reason="", readiness=false. Elapsed: 8.552037942s
Nov 29 22:20:05.853: INFO: Pod "pod-subpath-test-dynamicpv-nmjj": Phase="Pending", Reason="", readiness=false. Elapsed: 10.661993862s
... skipping 22 lines ...
Nov 29 22:20:54.395: INFO: Pod "pod-subpath-test-dynamicpv-nmjj": Phase="Pending", Reason="", readiness=false. Elapsed: 59.203935284s
Nov 29 22:20:56.504: INFO: Pod "pod-subpath-test-dynamicpv-nmjj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.313033768s
Nov 29 22:20:58.613: INFO: Pod "pod-subpath-test-dynamicpv-nmjj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.422430982s
Nov 29 22:21:00.723: INFO: Pod "pod-subpath-test-dynamicpv-nmjj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.532190736s
Nov 29 22:21:02.834: INFO: Pod "pod-subpath-test-dynamicpv-nmjj": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m7.642777923s
STEP: Saw pod success
Nov 29 22:21:02.834: INFO: Pod "pod-subpath-test-dynamicpv-nmjj" satisfied condition "Succeeded or Failed"
Nov 29 22:21:02.942: INFO: Trying to get logs from node k8s-agentpool1-92105279-vmss000001 pod pod-subpath-test-dynamicpv-nmjj container test-container-volume-dynamicpv-nmjj: <nil>
STEP: delete the pod
Nov 29 22:21:03.199: INFO: Waiting for pod pod-subpath-test-dynamicpv-nmjj to disappear
Nov 29 22:21:03.307: INFO: Pod pod-subpath-test-dynamicpv-nmjj no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-nmjj
Nov 29 22:21:03.307: INFO: Deleting pod "pod-subpath-test-dynamicpv-nmjj" in namespace "provisioning-9199"
... skipping 23 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support non-existent path
    test/e2e/storage/testsuites/subpath.go:196
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path","total":32,"completed":2,"skipped":235,"failed":0}

SSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode 
  should not mount / map unused volumes in a pod [LinuxOnly]
  test/e2e/storage/testsuites/volumemode.go:354
... skipping 63 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should not mount / map unused volumes in a pod [LinuxOnly]
    test/e2e/storage/testsuites/volumemode.go:354
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":33,"completed":2,"skipped":264,"failed":0}

SSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:22:26.492: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 58 lines ...
  test/e2e/framework/framework.go:188
Nov 29 22:22:27.892: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "volumelimits-9895" for this suite.

•
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeLimits should verify that all csinodes have volume limits","total":33,"completed":3,"skipped":445,"failed":0}

S
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
... skipping 35 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:258

    Distro debian doesn't support ntfs -- skipping

    test/e2e/storage/framework/testsuite.go:127
------------------------------
... skipping 171 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
    test/e2e/storage/testsuites/provisioning.go:208
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":39,"completed":2,"skipped":116,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should fail if subpath file is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:258

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 22:20:54.620: INFO: >>> kubeConfig: /root/tmp1562512509/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename provisioning
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if subpath file is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:258
Nov 29 22:20:55.385: INFO: Creating resource for dynamic PV
Nov 29 22:20:55.385: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-4150-e2e-scg9jq4
STEP: creating a claim
Nov 29 22:20:55.497: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 29 22:20:55.609: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comqfk2w] to have phase Bound
Nov 29 22:20:55.719: INFO: PersistentVolumeClaim test.csi.azure.comqfk2w found but phase is Pending instead of Bound.
Nov 29 22:20:57.830: INFO: PersistentVolumeClaim test.csi.azure.comqfk2w found but phase is Pending instead of Bound.
Nov 29 22:20:59.940: INFO: PersistentVolumeClaim test.csi.azure.comqfk2w found and phase=Bound (4.33074793s)
STEP: Creating pod pod-subpath-test-dynamicpv-ktxx
STEP: Checking for subpath error in container status
Nov 29 22:21:28.492: INFO: Deleting pod "pod-subpath-test-dynamicpv-ktxx" in namespace "provisioning-4150"
Nov 29 22:21:28.607: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-ktxx" to be fully deleted
STEP: Deleting pod
Nov 29 22:21:30.826: INFO: Deleting pod "pod-subpath-test-dynamicpv-ktxx" in namespace "provisioning-4150"
STEP: Deleting pvc
Nov 29 22:21:30.936: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comqfk2w"
... skipping 22 lines ...

• [SLOW TEST:108.662 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath file is outside the volume [Slow][LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:258
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow][LinuxOnly]","total":31,"completed":3,"skipped":125,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:22:43.320: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 38 lines ...
Nov 29 22:21:22.115: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comw96s9] to have phase Bound
Nov 29 22:21:22.228: INFO: PersistentVolumeClaim test.csi.azure.comw96s9 found but phase is Pending instead of Bound.
Nov 29 22:21:24.355: INFO: PersistentVolumeClaim test.csi.azure.comw96s9 found but phase is Pending instead of Bound.
Nov 29 22:21:26.465: INFO: PersistentVolumeClaim test.csi.azure.comw96s9 found and phase=Bound (4.350504014s)
STEP: Creating pod pod-subpath-test-dynamicpv-kbjt
STEP: Creating a pod to test subpath
Nov 29 22:21:26.797: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-kbjt" in namespace "provisioning-9079" to be "Succeeded or Failed"
Nov 29 22:21:26.914: INFO: Pod "pod-subpath-test-dynamicpv-kbjt": Phase="Pending", Reason="", readiness=false. Elapsed: 116.955752ms
Nov 29 22:21:29.024: INFO: Pod "pod-subpath-test-dynamicpv-kbjt": Phase="Pending", Reason="", readiness=false. Elapsed: 2.227670622s
Nov 29 22:21:31.137: INFO: Pod "pod-subpath-test-dynamicpv-kbjt": Phase="Pending", Reason="", readiness=false. Elapsed: 4.340072665s
Nov 29 22:21:33.246: INFO: Pod "pod-subpath-test-dynamicpv-kbjt": Phase="Pending", Reason="", readiness=false. Elapsed: 6.449597496s
Nov 29 22:21:35.357: INFO: Pod "pod-subpath-test-dynamicpv-kbjt": Phase="Pending", Reason="", readiness=false. Elapsed: 8.560632822s
Nov 29 22:21:37.473: INFO: Pod "pod-subpath-test-dynamicpv-kbjt": Phase="Pending", Reason="", readiness=false. Elapsed: 10.676318817s
... skipping 23 lines ...
Nov 29 22:22:28.137: INFO: Pod "pod-subpath-test-dynamicpv-kbjt": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.34044857s
Nov 29 22:22:30.248: INFO: Pod "pod-subpath-test-dynamicpv-kbjt": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.451140535s
Nov 29 22:22:32.358: INFO: Pod "pod-subpath-test-dynamicpv-kbjt": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.560959157s
Nov 29 22:22:34.477: INFO: Pod "pod-subpath-test-dynamicpv-kbjt": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.680287791s
Nov 29 22:22:36.618: INFO: Pod "pod-subpath-test-dynamicpv-kbjt": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m9.821098126s
STEP: Saw pod success
Nov 29 22:22:36.618: INFO: Pod "pod-subpath-test-dynamicpv-kbjt" satisfied condition "Succeeded or Failed"
Nov 29 22:22:36.736: INFO: Trying to get logs from node k8s-agentpool1-92105279-vmss000000 pod pod-subpath-test-dynamicpv-kbjt container test-container-subpath-dynamicpv-kbjt: <nil>
STEP: delete the pod
Nov 29 22:22:37.003: INFO: Waiting for pod pod-subpath-test-dynamicpv-kbjt to disappear
Nov 29 22:22:37.112: INFO: Pod pod-subpath-test-dynamicpv-kbjt no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-kbjt
Nov 29 22:22:37.112: INFO: Deleting pod "pod-subpath-test-dynamicpv-kbjt" in namespace "provisioning-9079"
... skipping 29 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support existing single file [LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:221
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing single file [LinuxOnly]","total":31,"completed":3,"skipped":159,"failed":0}

SSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral 
  should support multiple inline ephemeral volumes
  test/e2e/storage/testsuites/ephemeral.go:254
... skipping 51 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should support multiple inline ephemeral volumes
    test/e2e/storage/testsuites/ephemeral.go:254
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support multiple inline ephemeral volumes","total":32,"completed":3,"skipped":238,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral 
  should create read/write inline ephemeral volume
  test/e2e/storage/testsuites/ephemeral.go:196
... skipping 45 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should create read/write inline ephemeral volume
    test/e2e/storage/testsuites/ephemeral.go:196
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read/write inline ephemeral volume","total":33,"completed":4,"skipped":506,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:24:08.327: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 38 lines ...
Nov 29 22:22:42.137: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comhcfkb] to have phase Bound
Nov 29 22:22:42.246: INFO: PersistentVolumeClaim test.csi.azure.comhcfkb found but phase is Pending instead of Bound.
Nov 29 22:22:44.359: INFO: PersistentVolumeClaim test.csi.azure.comhcfkb found but phase is Pending instead of Bound.
Nov 29 22:22:46.469: INFO: PersistentVolumeClaim test.csi.azure.comhcfkb found and phase=Bound (4.331959763s)
STEP: Creating pod exec-volume-test-dynamicpv-nvrr
STEP: Creating a pod to test exec-volume-test
Nov 29 22:22:46.799: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-nvrr" in namespace "volume-7610" to be "Succeeded or Failed"
Nov 29 22:22:46.915: INFO: Pod "exec-volume-test-dynamicpv-nvrr": Phase="Pending", Reason="", readiness=false. Elapsed: 115.331252ms
Nov 29 22:22:49.025: INFO: Pod "exec-volume-test-dynamicpv-nvrr": Phase="Pending", Reason="", readiness=false. Elapsed: 2.225500338s
Nov 29 22:22:51.135: INFO: Pod "exec-volume-test-dynamicpv-nvrr": Phase="Pending", Reason="", readiness=false. Elapsed: 4.336006419s
Nov 29 22:22:53.246: INFO: Pod "exec-volume-test-dynamicpv-nvrr": Phase="Pending", Reason="", readiness=false. Elapsed: 6.446469437s
Nov 29 22:22:55.355: INFO: Pod "exec-volume-test-dynamicpv-nvrr": Phase="Pending", Reason="", readiness=false. Elapsed: 8.555773142s
Nov 29 22:22:57.466: INFO: Pod "exec-volume-test-dynamicpv-nvrr": Phase="Pending", Reason="", readiness=false. Elapsed: 10.667167147s
... skipping 43 lines ...
Nov 29 22:24:30.356: INFO: Pod "exec-volume-test-dynamicpv-nvrr": Phase="Pending", Reason="", readiness=false. Elapsed: 1m43.55658275s
Nov 29 22:24:32.467: INFO: Pod "exec-volume-test-dynamicpv-nvrr": Phase="Pending", Reason="", readiness=false. Elapsed: 1m45.667687863s
Nov 29 22:24:34.577: INFO: Pod "exec-volume-test-dynamicpv-nvrr": Phase="Pending", Reason="", readiness=false. Elapsed: 1m47.777561438s
Nov 29 22:24:36.687: INFO: Pod "exec-volume-test-dynamicpv-nvrr": Phase="Pending", Reason="", readiness=false. Elapsed: 1m49.887511732s
Nov 29 22:24:38.797: INFO: Pod "exec-volume-test-dynamicpv-nvrr": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m51.99817077s
STEP: Saw pod success
Nov 29 22:24:38.798: INFO: Pod "exec-volume-test-dynamicpv-nvrr" satisfied condition "Succeeded or Failed"
Nov 29 22:24:38.907: INFO: Trying to get logs from node k8s-agentpool1-92105279-vmss000000 pod exec-volume-test-dynamicpv-nvrr container exec-container-dynamicpv-nvrr: <nil>
STEP: delete the pod
Nov 29 22:24:39.164: INFO: Waiting for pod exec-volume-test-dynamicpv-nvrr to disappear
Nov 29 22:24:39.274: INFO: Pod exec-volume-test-dynamicpv-nvrr no longer exists
STEP: Deleting pod exec-volume-test-dynamicpv-nvrr
Nov 29 22:24:39.274: INFO: Deleting pod "exec-volume-test-dynamicpv-nvrr" in namespace "volume-7610"
... skipping 27 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should allow exec of files on the volume
    test/e2e/storage/testsuites/volumes.go:198
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should allow exec of files on the volume","total":39,"completed":3,"skipped":155,"failed":0}

SSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy 
  (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents
  test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
... skipping 119 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents","total":34,"completed":3,"skipped":73,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:26:04.222: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 37 lines ...
  test/e2e/framework/framework.go:188
Nov 29 22:26:05.485: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "volumelimits-3372" for this suite.

•
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs)] volumeLimits should verify that all csinodes have volume limits","total":34,"completed":4,"skipped":175,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:26:05.762: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 21 lines ...
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 22:26:05.798: INFO: >>> kubeConfig: /root/tmp1562512509/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename topology
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  test/e2e/storage/testsuites/topology.go:194
Nov 29 22:26:06.563: INFO: Driver didn't provide topology keys -- skipping
[AfterEach] [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/framework/framework.go:188
Nov 29 22:26:06.564: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "topology-1181" for this suite.


S [SKIPPING] [0.986 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/storage/framework/testsuite.go:50
    should fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement]
    test/e2e/storage/testsuites/topology.go:194

    Driver didn't provide topology keys -- skipping

    test/e2e/storage/testsuites/topology.go:126
------------------------------
... skipping 36 lines ...
[It] should check snapshot fields, check restore correctly works, check deletion (ephemeral)
  test/e2e/storage/testsuites/snapshottable.go:177
Nov 29 22:23:50.456: INFO: Creating resource for dynamic PV
Nov 29 22:23:50.456: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass snapshotting-1876-e2e-scqlkgw
STEP: [init] starting a pod to use the claim
Nov 29 22:23:50.680: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-2hz7x" in namespace "snapshotting-1876" to be "Succeeded or Failed"
Nov 29 22:23:50.794: INFO: Pod "pvc-snapshottable-tester-2hz7x": Phase="Pending", Reason="", readiness=false. Elapsed: 113.548304ms
Nov 29 22:23:52.907: INFO: Pod "pvc-snapshottable-tester-2hz7x": Phase="Pending", Reason="", readiness=false. Elapsed: 2.227098374s
Nov 29 22:23:55.017: INFO: Pod "pvc-snapshottable-tester-2hz7x": Phase="Pending", Reason="", readiness=false. Elapsed: 4.336609981s
Nov 29 22:23:57.128: INFO: Pod "pvc-snapshottable-tester-2hz7x": Phase="Pending", Reason="", readiness=false. Elapsed: 6.447459289s
Nov 29 22:23:59.237: INFO: Pod "pvc-snapshottable-tester-2hz7x": Phase="Pending", Reason="", readiness=false. Elapsed: 8.556510788s
Nov 29 22:24:01.346: INFO: Pod "pvc-snapshottable-tester-2hz7x": Phase="Pending", Reason="", readiness=false. Elapsed: 10.666111271s
... skipping 18 lines ...
Nov 29 22:24:41.445: INFO: Pod "pvc-snapshottable-tester-2hz7x": Phase="Pending", Reason="", readiness=false. Elapsed: 50.764742902s
Nov 29 22:24:43.556: INFO: Pod "pvc-snapshottable-tester-2hz7x": Phase="Pending", Reason="", readiness=false. Elapsed: 52.875352106s
Nov 29 22:24:45.667: INFO: Pod "pvc-snapshottable-tester-2hz7x": Phase="Pending", Reason="", readiness=false. Elapsed: 54.986934754s
Nov 29 22:24:47.778: INFO: Pod "pvc-snapshottable-tester-2hz7x": Phase="Pending", Reason="", readiness=false. Elapsed: 57.097215863s
Nov 29 22:24:49.889: INFO: Pod "pvc-snapshottable-tester-2hz7x": Phase="Succeeded", Reason="", readiness=false. Elapsed: 59.208444184s
STEP: Saw pod success
Nov 29 22:24:49.889: INFO: Pod "pvc-snapshottable-tester-2hz7x" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
STEP: creating a SnapshotClass
STEP: creating a dynamic VolumeSnapshot
Nov 29 22:24:50.338: INFO: Waiting up to 5m0s for VolumeSnapshot snapshot-rv5tq to become ready
Nov 29 22:24:50.449: INFO: VolumeSnapshot snapshot-rv5tq found but is not ready.
Nov 29 22:24:52.561: INFO: VolumeSnapshot snapshot-rv5tq found but is not ready.
... skipping 49 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works, check deletion (ephemeral)
        test/e2e/storage/testsuites/snapshottable.go:177
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works, check deletion (ephemeral)","total":31,"completed":4,"skipped":175,"failed":0}

SSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (xfs)][Slow] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:26:30.163: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 217 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:138
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":31,"completed":4,"skipped":164,"failed":0}

SSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:26:32.776: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 404 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with pvc data source in parallel [Slow]
    test/e2e/storage/testsuites/provisioning.go:459
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with pvc data source in parallel [Slow]","total":30,"completed":2,"skipped":210,"failed":0}

SSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning
... skipping 323 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should store data
    test/e2e/storage/testsuites/volumes.go:161
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data","total":33,"completed":5,"skipped":660,"failed":0}

SSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
... skipping 124 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support restarting containers using file as subpath [Slow][LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:333
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using file as subpath [Slow][LinuxOnly]","total":34,"completed":5,"skipped":299,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should verify container cannot write to subpath readonly volumes [Slow]
  test/e2e/storage/testsuites/subpath.go:425
... skipping 16 lines ...
Nov 29 22:28:03.741: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 29 22:28:03.855: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comtbvkh] to have phase Bound
Nov 29 22:28:03.966: INFO: PersistentVolumeClaim test.csi.azure.comtbvkh found but phase is Pending instead of Bound.
Nov 29 22:28:06.077: INFO: PersistentVolumeClaim test.csi.azure.comtbvkh found but phase is Pending instead of Bound.
Nov 29 22:28:08.191: INFO: PersistentVolumeClaim test.csi.azure.comtbvkh found and phase=Bound (4.335284613s)
STEP: Creating pod to format volume volume-prep-provisioning-5913
Nov 29 22:28:08.525: INFO: Waiting up to 5m0s for pod "volume-prep-provisioning-5913" in namespace "provisioning-5913" to be "Succeeded or Failed"
Nov 29 22:28:08.636: INFO: Pod "volume-prep-provisioning-5913": Phase="Pending", Reason="", readiness=false. Elapsed: 111.091579ms
Nov 29 22:28:10.748: INFO: Pod "volume-prep-provisioning-5913": Phase="Pending", Reason="", readiness=false. Elapsed: 2.222969634s
Nov 29 22:28:12.867: INFO: Pod "volume-prep-provisioning-5913": Phase="Pending", Reason="", readiness=false. Elapsed: 4.342109888s
Nov 29 22:28:14.978: INFO: Pod "volume-prep-provisioning-5913": Phase="Pending", Reason="", readiness=false. Elapsed: 6.453496808s
Nov 29 22:28:17.090: INFO: Pod "volume-prep-provisioning-5913": Phase="Pending", Reason="", readiness=false. Elapsed: 8.564979013s
Nov 29 22:28:19.204: INFO: Pod "volume-prep-provisioning-5913": Phase="Pending", Reason="", readiness=false. Elapsed: 10.679292901s
Nov 29 22:28:21.318: INFO: Pod "volume-prep-provisioning-5913": Phase="Pending", Reason="", readiness=false. Elapsed: 12.79307213s
Nov 29 22:28:23.431: INFO: Pod "volume-prep-provisioning-5913": Phase="Pending", Reason="", readiness=false. Elapsed: 14.906579087s
Nov 29 22:28:25.545: INFO: Pod "volume-prep-provisioning-5913": Phase="Succeeded", Reason="", readiness=false. Elapsed: 17.020562733s
STEP: Saw pod success
Nov 29 22:28:25.545: INFO: Pod "volume-prep-provisioning-5913" satisfied condition "Succeeded or Failed"
Nov 29 22:28:25.545: INFO: Deleting pod "volume-prep-provisioning-5913" in namespace "provisioning-5913"
Nov 29 22:28:25.663: INFO: Wait up to 5m0s for pod "volume-prep-provisioning-5913" to be fully deleted
STEP: Creating pod pod-subpath-test-dynamicpv-pb7g
STEP: Checking for subpath error in container status
Nov 29 22:28:40.113: INFO: Deleting pod "pod-subpath-test-dynamicpv-pb7g" in namespace "provisioning-5913"
Nov 29 22:28:40.224: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-pb7g" to be fully deleted
STEP: Deleting pod
Nov 29 22:28:42.449: INFO: Deleting pod "pod-subpath-test-dynamicpv-pb7g" in namespace "provisioning-5913"
STEP: Deleting pvc
Nov 29 22:28:42.560: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comtbvkh"
... skipping 19 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should verify container cannot write to subpath readonly volumes [Slow]
    test/e2e/storage/testsuites/subpath.go:425
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should verify container cannot write to subpath readonly volumes [Slow]","total":33,"completed":6,"skipped":718,"failed":0}

SSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy 
  (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents
  test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
... skipping 113 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents","total":39,"completed":4,"skipped":161,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:29:27.665: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 80 lines ...
Nov 29 22:23:59.428: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comr4twh] to have phase Bound
Nov 29 22:23:59.537: INFO: PersistentVolumeClaim test.csi.azure.comr4twh found but phase is Pending instead of Bound.
Nov 29 22:24:01.647: INFO: PersistentVolumeClaim test.csi.azure.comr4twh found but phase is Pending instead of Bound.
Nov 29 22:24:03.756: INFO: PersistentVolumeClaim test.csi.azure.comr4twh found and phase=Bound (4.327549919s)
STEP: [init] starting a pod to use the claim
STEP: [init] check pod success
Nov 29 22:24:04.198: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-975dg" in namespace "snapshotting-3458" to be "Succeeded or Failed"
Nov 29 22:24:04.306: INFO: Pod "pvc-snapshottable-tester-975dg": Phase="Pending", Reason="", readiness=false. Elapsed: 108.160542ms
Nov 29 22:24:06.415: INFO: Pod "pvc-snapshottable-tester-975dg": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217326498s
Nov 29 22:24:08.526: INFO: Pod "pvc-snapshottable-tester-975dg": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327752157s
Nov 29 22:24:10.635: INFO: Pod "pvc-snapshottable-tester-975dg": Phase="Pending", Reason="", readiness=false. Elapsed: 6.436560359s
Nov 29 22:24:12.745: INFO: Pod "pvc-snapshottable-tester-975dg": Phase="Pending", Reason="", readiness=false. Elapsed: 8.546578182s
Nov 29 22:24:14.853: INFO: Pod "pvc-snapshottable-tester-975dg": Phase="Pending", Reason="", readiness=false. Elapsed: 10.655375582s
... skipping 13 lines ...
Nov 29 22:24:44.390: INFO: Pod "pvc-snapshottable-tester-975dg": Phase="Pending", Reason="", readiness=false. Elapsed: 40.192225325s
Nov 29 22:24:46.500: INFO: Pod "pvc-snapshottable-tester-975dg": Phase="Pending", Reason="", readiness=false. Elapsed: 42.302092387s
Nov 29 22:24:48.626: INFO: Pod "pvc-snapshottable-tester-975dg": Phase="Pending", Reason="", readiness=false. Elapsed: 44.427705962s
Nov 29 22:24:50.770: INFO: Pod "pvc-snapshottable-tester-975dg": Phase="Pending", Reason="", readiness=false. Elapsed: 46.571773581s
Nov 29 22:24:52.879: INFO: Pod "pvc-snapshottable-tester-975dg": Phase="Succeeded", Reason="", readiness=false. Elapsed: 48.681161516s
STEP: Saw pod success
Nov 29 22:24:52.879: INFO: Pod "pvc-snapshottable-tester-975dg" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
Nov 29 22:24:52.988: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comr4twh] to have phase Bound
Nov 29 22:24:53.096: INFO: PersistentVolumeClaim test.csi.azure.comr4twh found and phase=Bound (108.08768ms)
STEP: [init] checking the PV
STEP: [init] deleting the pod
Nov 29 22:24:53.462: INFO: Pod pvc-snapshottable-tester-975dg has the following logs: 
... skipping 13 lines ...
Nov 29 22:25:00.895: INFO: received snapshotStatus map[boundVolumeSnapshotContentName:snapcontent-e5ffe0a6-abbc-4f74-ac2c-f3b82d93086e creationTime:2022-11-29T22:24:56Z readyToUse:true restoreSize:5Gi]
Nov 29 22:25:00.895: INFO: snapshotContentName snapcontent-e5ffe0a6-abbc-4f74-ac2c-f3b82d93086e
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Nov 29 22:25:01.338: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-sf7gs" in namespace "snapshotting-3458" to be "Succeeded or Failed"
Nov 29 22:25:01.447: INFO: Pod "pvc-snapshottable-data-tester-sf7gs": Phase="Pending", Reason="", readiness=false. Elapsed: 108.875995ms
Nov 29 22:25:03.561: INFO: Pod "pvc-snapshottable-data-tester-sf7gs": Phase="Pending", Reason="", readiness=false. Elapsed: 2.223049017s
Nov 29 22:25:05.671: INFO: Pod "pvc-snapshottable-data-tester-sf7gs": Phase="Pending", Reason="", readiness=false. Elapsed: 4.332840211s
Nov 29 22:25:07.779: INFO: Pod "pvc-snapshottable-data-tester-sf7gs": Phase="Pending", Reason="", readiness=false. Elapsed: 6.440778483s
Nov 29 22:25:09.888: INFO: Pod "pvc-snapshottable-data-tester-sf7gs": Phase="Pending", Reason="", readiness=false. Elapsed: 8.549689497s
Nov 29 22:25:11.998: INFO: Pod "pvc-snapshottable-data-tester-sf7gs": Phase="Pending", Reason="", readiness=false. Elapsed: 10.659300432s
... skipping 35 lines ...
Nov 29 22:26:27.963: INFO: Pod "pvc-snapshottable-data-tester-sf7gs": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.624332955s
Nov 29 22:26:30.072: INFO: Pod "pvc-snapshottable-data-tester-sf7gs": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.733876617s
Nov 29 22:26:32.182: INFO: Pod "pvc-snapshottable-data-tester-sf7gs": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.843696362s
Nov 29 22:26:34.291: INFO: Pod "pvc-snapshottable-data-tester-sf7gs": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.952683016s
Nov 29 22:26:36.400: INFO: Pod "pvc-snapshottable-data-tester-sf7gs": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m35.062081866s
STEP: Saw pod success
Nov 29 22:26:36.401: INFO: Pod "pvc-snapshottable-data-tester-sf7gs" satisfied condition "Succeeded or Failed"
Nov 29 22:26:36.621: INFO: Pod pvc-snapshottable-data-tester-sf7gs has the following logs: 
Nov 29 22:26:36.621: INFO: Deleting pod "pvc-snapshottable-data-tester-sf7gs" in namespace "snapshotting-3458"
Nov 29 22:26:36.737: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-sf7gs" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the snapshot
Nov 29 22:29:11.290: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-ofo3t2sm.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1562512509/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-3458 exec restored-pvc-tester-mx297 --namespace=snapshotting-3458 -- cat /mnt/test/data'
... skipping 33 lines ...
Nov 29 22:29:37.684: INFO: volumesnapshotcontents snapcontent-e5ffe0a6-abbc-4f74-ac2c-f3b82d93086e has been found and is not deleted
Nov 29 22:29:38.797: INFO: volumesnapshotcontents snapcontent-e5ffe0a6-abbc-4f74-ac2c-f3b82d93086e has been found and is not deleted
Nov 29 22:29:39.907: INFO: volumesnapshotcontents snapcontent-e5ffe0a6-abbc-4f74-ac2c-f3b82d93086e has been found and is not deleted
Nov 29 22:29:41.018: INFO: volumesnapshotcontents snapcontent-e5ffe0a6-abbc-4f74-ac2c-f3b82d93086e has been found and is not deleted
Nov 29 22:29:42.129: INFO: volumesnapshotcontents snapcontent-e5ffe0a6-abbc-4f74-ac2c-f3b82d93086e has been found and is not deleted
Nov 29 22:29:43.238: INFO: volumesnapshotcontents snapcontent-e5ffe0a6-abbc-4f74-ac2c-f3b82d93086e has been found and is not deleted
Nov 29 22:29:44.239: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 29 22:29:44.350: INFO: Error getting logs for pod restored-pvc-tester-mx297: the server could not find the requested resource (get pods restored-pvc-tester-mx297)
Nov 29 22:29:44.350: INFO: Deleting pod "restored-pvc-tester-mx297" in namespace "snapshotting-3458"
Nov 29 22:29:44.459: INFO: deleting claim "snapshotting-3458"/"pvc-fsd9r"
Nov 29 22:29:44.571: INFO: deleting snapshot "snapshotting-3458"/"snapshot-qqjzl"
Nov 29 22:29:44.681: INFO: deleting snapshot content "snapcontent-e5ffe0a6-abbc-4f74-ac2c-f3b82d93086e"
Nov 29 22:29:45.017: INFO: Waiting up to 5m0s for volumesnapshotcontents snapcontent-e5ffe0a6-abbc-4f74-ac2c-f3b82d93086e to be deleted
Nov 29 22:29:45.125: INFO: volumesnapshotcontents snapcontent-e5ffe0a6-abbc-4f74-ac2c-f3b82d93086e has been found and is not deleted
... skipping 27 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        test/e2e/storage/testsuites/snapshottable.go:278
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":32,"completed":4,"skipped":319,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource]
... skipping 41 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach]
    test/e2e/storage/testsuites/volumemode.go:299

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 8 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:280

    Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 57 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should create read/write inline ephemeral volume
    test/e2e/storage/testsuites/ephemeral.go:196
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read/write inline ephemeral volume","total":30,"completed":3,"skipped":533,"failed":0}

SSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
... skipping 101 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should not mount / map unused volumes in a pod [LinuxOnly]
    test/e2e/storage/testsuites/volumemode.go:354
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":39,"completed":5,"skipped":323,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:30:59.627: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 3 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:242

    Distro debian doesn't support ntfs -- skipping

    test/e2e/storage/framework/testsuite.go:127
------------------------------
... skipping 43 lines ...
Nov 29 22:30:13.285: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comp9f8j] to have phase Bound
Nov 29 22:30:13.398: INFO: PersistentVolumeClaim test.csi.azure.comp9f8j found but phase is Pending instead of Bound.
Nov 29 22:30:15.507: INFO: PersistentVolumeClaim test.csi.azure.comp9f8j found but phase is Pending instead of Bound.
Nov 29 22:30:17.617: INFO: PersistentVolumeClaim test.csi.azure.comp9f8j found and phase=Bound (4.332153677s)
STEP: Creating pod pod-subpath-test-dynamicpv-8856
STEP: Creating a pod to test subpath
Nov 29 22:30:17.954: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-8856" in namespace "provisioning-7674" to be "Succeeded or Failed"
Nov 29 22:30:18.063: INFO: Pod "pod-subpath-test-dynamicpv-8856": Phase="Pending", Reason="", readiness=false. Elapsed: 109.379842ms
Nov 29 22:30:20.175: INFO: Pod "pod-subpath-test-dynamicpv-8856": Phase="Pending", Reason="", readiness=false. Elapsed: 2.220809674s
Nov 29 22:30:22.734: INFO: Pod "pod-subpath-test-dynamicpv-8856": Phase="Pending", Reason="", readiness=false. Elapsed: 4.779697676s
Nov 29 22:30:24.852: INFO: Pod "pod-subpath-test-dynamicpv-8856": Phase="Pending", Reason="", readiness=false. Elapsed: 6.897995501s
Nov 29 22:30:26.963: INFO: Pod "pod-subpath-test-dynamicpv-8856": Phase="Pending", Reason="", readiness=false. Elapsed: 9.008787532s
Nov 29 22:30:29.075: INFO: Pod "pod-subpath-test-dynamicpv-8856": Phase="Pending", Reason="", readiness=false. Elapsed: 11.120930016s
... skipping 43 lines ...
Nov 29 22:32:01.965: INFO: Pod "pod-subpath-test-dynamicpv-8856": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.011231412s
Nov 29 22:32:04.075: INFO: Pod "pod-subpath-test-dynamicpv-8856": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.120650043s
Nov 29 22:32:06.188: INFO: Pod "pod-subpath-test-dynamicpv-8856": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.233896388s
Nov 29 22:32:08.299: INFO: Pod "pod-subpath-test-dynamicpv-8856": Phase="Pending", Reason="", readiness=false. Elapsed: 1m50.345098947s
Nov 29 22:32:10.412: INFO: Pod "pod-subpath-test-dynamicpv-8856": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m52.457849468s
STEP: Saw pod success
Nov 29 22:32:10.412: INFO: Pod "pod-subpath-test-dynamicpv-8856" satisfied condition "Succeeded or Failed"
Nov 29 22:32:10.522: INFO: Trying to get logs from node k8s-agentpool1-92105279-vmss000002 pod pod-subpath-test-dynamicpv-8856 container test-container-volume-dynamicpv-8856: <nil>
STEP: delete the pod
Nov 29 22:32:10.785: INFO: Waiting for pod pod-subpath-test-dynamicpv-8856 to disappear
Nov 29 22:32:10.894: INFO: Pod pod-subpath-test-dynamicpv-8856 no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-8856
Nov 29 22:32:10.894: INFO: Deleting pod "pod-subpath-test-dynamicpv-8856" in namespace "provisioning-7674"
... skipping 23 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support existing directory
    test/e2e/storage/testsuites/subpath.go:207
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory","total":30,"completed":4,"skipped":550,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:32:52.980: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 70 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] volumeIO
  test/e2e/storage/framework/testsuite.go:50
    should write files of various sizes, verify size, validate content [Slow]
    test/e2e/storage/testsuites/volume_io.go:149
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumeIO should write files of various sizes, verify size, validate content [Slow]","total":32,"completed":5,"skipped":461,"failed":0}

SS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] 
  should access to two volumes with different volume mode and retain data across pod recreation on the same node
  test/e2e/storage/testsuites/multivolume.go:209
... skipping 207 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:209
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":33,"completed":7,"skipped":735,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] 
  should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
  test/e2e/storage/testsuites/multivolume.go:323
... skipping 121 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:323
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":31,"completed":5,"skipped":199,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] 
  should access to two volumes with the same volume mode and retain data across pod recreation on different node
  test/e2e/storage/testsuites/multivolume.go:168
... skipping 196 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:168
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":34,"completed":6,"skipped":323,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:34:41.848: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 45 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:258

    Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 85 lines ...
Nov 29 22:26:33.907: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.combjhl7] to have phase Bound
Nov 29 22:26:34.016: INFO: PersistentVolumeClaim test.csi.azure.combjhl7 found but phase is Pending instead of Bound.
Nov 29 22:26:36.126: INFO: PersistentVolumeClaim test.csi.azure.combjhl7 found but phase is Pending instead of Bound.
Nov 29 22:26:38.237: INFO: PersistentVolumeClaim test.csi.azure.combjhl7 found and phase=Bound (4.329259734s)
STEP: [init] starting a pod to use the claim
STEP: [init] check pod success
Nov 29 22:26:38.674: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-24dfl" in namespace "snapshotting-2433" to be "Succeeded or Failed"
Nov 29 22:26:38.783: INFO: Pod "pvc-snapshottable-tester-24dfl": Phase="Pending", Reason="", readiness=false. Elapsed: 109.189125ms
Nov 29 22:26:40.892: INFO: Pod "pvc-snapshottable-tester-24dfl": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218507624s
Nov 29 22:26:43.010: INFO: Pod "pvc-snapshottable-tester-24dfl": Phase="Pending", Reason="", readiness=false. Elapsed: 4.335727497s
Nov 29 22:26:45.120: INFO: Pod "pvc-snapshottable-tester-24dfl": Phase="Pending", Reason="", readiness=false. Elapsed: 6.446164651s
Nov 29 22:26:47.229: INFO: Pod "pvc-snapshottable-tester-24dfl": Phase="Pending", Reason="", readiness=false. Elapsed: 8.555496575s
Nov 29 22:26:49.348: INFO: Pod "pvc-snapshottable-tester-24dfl": Phase="Pending", Reason="", readiness=false. Elapsed: 10.674051268s
... skipping 63 lines ...
Nov 29 22:29:04.494: INFO: Pod "pvc-snapshottable-tester-24dfl": Phase="Pending", Reason="", readiness=false. Elapsed: 2m25.819881967s
Nov 29 22:29:06.605: INFO: Pod "pvc-snapshottable-tester-24dfl": Phase="Pending", Reason="", readiness=false. Elapsed: 2m27.930832834s
Nov 29 22:29:08.717: INFO: Pod "pvc-snapshottable-tester-24dfl": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.043441649s
Nov 29 22:29:10.829: INFO: Pod "pvc-snapshottable-tester-24dfl": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.155044406s
Nov 29 22:29:12.939: INFO: Pod "pvc-snapshottable-tester-24dfl": Phase="Succeeded", Reason="", readiness=false. Elapsed: 2m34.264827554s
STEP: Saw pod success
Nov 29 22:29:12.939: INFO: Pod "pvc-snapshottable-tester-24dfl" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
Nov 29 22:29:13.048: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.combjhl7] to have phase Bound
Nov 29 22:29:13.156: INFO: PersistentVolumeClaim test.csi.azure.combjhl7 found and phase=Bound (108.558137ms)
STEP: [init] checking the PV
STEP: [init] deleting the pod
Nov 29 22:29:13.515: INFO: Pod pvc-snapshottable-tester-24dfl has the following logs: 
... skipping 35 lines ...
Nov 29 22:29:26.401: INFO: WaitUntil finished successfully after 112.713371ms
STEP: getting the snapshot and snapshot content
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Nov 29 22:29:26.952: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-49rbq" in namespace "snapshotting-2433" to be "Succeeded or Failed"
Nov 29 22:29:27.061: INFO: Pod "pvc-snapshottable-data-tester-49rbq": Phase="Pending", Reason="", readiness=false. Elapsed: 109.691192ms
Nov 29 22:29:29.174: INFO: Pod "pvc-snapshottable-data-tester-49rbq": Phase="Pending", Reason="", readiness=false. Elapsed: 2.2224262s
Nov 29 22:29:31.285: INFO: Pod "pvc-snapshottable-data-tester-49rbq": Phase="Pending", Reason="", readiness=false. Elapsed: 4.333503652s
Nov 29 22:29:33.397: INFO: Pod "pvc-snapshottable-data-tester-49rbq": Phase="Pending", Reason="", readiness=false. Elapsed: 6.444710148s
Nov 29 22:29:35.506: INFO: Pod "pvc-snapshottable-data-tester-49rbq": Phase="Pending", Reason="", readiness=false. Elapsed: 8.554616859s
Nov 29 22:29:37.617: INFO: Pod "pvc-snapshottable-data-tester-49rbq": Phase="Pending", Reason="", readiness=false. Elapsed: 10.664755133s
... skipping 93 lines ...
Nov 29 22:32:56.059: INFO: Pod "pvc-snapshottable-data-tester-49rbq": Phase="Pending", Reason="", readiness=false. Elapsed: 3m29.106775352s
Nov 29 22:32:58.173: INFO: Pod "pvc-snapshottable-data-tester-49rbq": Phase="Pending", Reason="", readiness=false. Elapsed: 3m31.220860976s
Nov 29 22:33:00.282: INFO: Pod "pvc-snapshottable-data-tester-49rbq": Phase="Pending", Reason="", readiness=false. Elapsed: 3m33.329941249s
Nov 29 22:33:02.391: INFO: Pod "pvc-snapshottable-data-tester-49rbq": Phase="Pending", Reason="", readiness=false. Elapsed: 3m35.439682479s
Nov 29 22:33:04.501: INFO: Pod "pvc-snapshottable-data-tester-49rbq": Phase="Succeeded", Reason="", readiness=false. Elapsed: 3m37.549563461s
STEP: Saw pod success
Nov 29 22:33:04.501: INFO: Pod "pvc-snapshottable-data-tester-49rbq" satisfied condition "Succeeded or Failed"
Nov 29 22:33:04.749: INFO: Pod pvc-snapshottable-data-tester-49rbq has the following logs: 
Nov 29 22:33:04.749: INFO: Deleting pod "pvc-snapshottable-data-tester-49rbq" in namespace "snapshotting-2433"
Nov 29 22:33:04.864: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-49rbq" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the snapshot
Nov 29 22:34:15.439: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-ofo3t2sm.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1562512509/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-2433 exec restored-pvc-tester-5rvgr --namespace=snapshotting-2433 -- cat /mnt/test/data'
... skipping 33 lines ...
Nov 29 22:34:41.727: INFO: volumesnapshotcontents pre-provisioned-snapcontent-47476ed7-3e35-419f-80e6-be424cf56f30 has been found and is not deleted
Nov 29 22:34:42.837: INFO: volumesnapshotcontents pre-provisioned-snapcontent-47476ed7-3e35-419f-80e6-be424cf56f30 has been found and is not deleted
Nov 29 22:34:43.947: INFO: volumesnapshotcontents pre-provisioned-snapcontent-47476ed7-3e35-419f-80e6-be424cf56f30 has been found and is not deleted
Nov 29 22:34:45.058: INFO: volumesnapshotcontents pre-provisioned-snapcontent-47476ed7-3e35-419f-80e6-be424cf56f30 has been found and is not deleted
Nov 29 22:34:46.167: INFO: volumesnapshotcontents pre-provisioned-snapcontent-47476ed7-3e35-419f-80e6-be424cf56f30 has been found and is not deleted
Nov 29 22:34:47.277: INFO: volumesnapshotcontents pre-provisioned-snapcontent-47476ed7-3e35-419f-80e6-be424cf56f30 has been found and is not deleted
Nov 29 22:34:48.278: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 29 22:34:48.387: INFO: Error getting logs for pod restored-pvc-tester-5rvgr: the server could not find the requested resource (get pods restored-pvc-tester-5rvgr)
Nov 29 22:34:48.387: INFO: Deleting pod "restored-pvc-tester-5rvgr" in namespace "snapshotting-2433"
Nov 29 22:34:48.499: INFO: deleting claim "snapshotting-2433"/"pvc-kk99q"
Nov 29 22:34:48.607: INFO: deleting snapshot "snapshotting-2433"/"pre-provisioned-snapshot-47476ed7-3e35-419f-80e6-be424cf56f30"
Nov 29 22:34:48.716: INFO: deleting snapshot content "pre-provisioned-snapcontent-47476ed7-3e35-419f-80e6-be424cf56f30"
Nov 29 22:34:49.066: INFO: Waiting up to 5m0s for volumesnapshotcontents pre-provisioned-snapcontent-47476ed7-3e35-419f-80e6-be424cf56f30 to be deleted
Nov 29 22:34:49.186: INFO: volumesnapshotcontents pre-provisioned-snapcontent-47476ed7-3e35-419f-80e6-be424cf56f30 has been found and is not deleted
... skipping 27 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        test/e2e/storage/testsuites/snapshottable.go:278
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":31,"completed":5,"skipped":315,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:35:06.783: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 216 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:209
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":39,"completed":6,"skipped":364,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral 
  should create read-only inline ephemeral volume
  test/e2e/storage/testsuites/ephemeral.go:175
... skipping 45 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should create read-only inline ephemeral volume
    test/e2e/storage/testsuites/ephemeral.go:175
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read-only inline ephemeral volume","total":31,"completed":6,"skipped":239,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:36:06.175: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 118 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":32,"completed":6,"skipped":463,"failed":0}

SS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:36:07.360: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 118 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":30,"completed":5,"skipped":687,"failed":0}

SSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:36:10.267: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 91 lines ...
Nov 29 22:36:08.358: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comgr697] to have phase Bound
Nov 29 22:36:08.466: INFO: PersistentVolumeClaim test.csi.azure.comgr697 found but phase is Pending instead of Bound.
Nov 29 22:36:10.576: INFO: PersistentVolumeClaim test.csi.azure.comgr697 found but phase is Pending instead of Bound.
Nov 29 22:36:12.692: INFO: PersistentVolumeClaim test.csi.azure.comgr697 found and phase=Bound (4.333926692s)
STEP: Creating pod pod-subpath-test-dynamicpv-cjv4
STEP: Creating a pod to test subpath
Nov 29 22:36:13.028: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-cjv4" in namespace "provisioning-9827" to be "Succeeded or Failed"
Nov 29 22:36:13.136: INFO: Pod "pod-subpath-test-dynamicpv-cjv4": Phase="Pending", Reason="", readiness=false. Elapsed: 108.443001ms
Nov 29 22:36:15.251: INFO: Pod "pod-subpath-test-dynamicpv-cjv4": Phase="Pending", Reason="", readiness=false. Elapsed: 2.223681594s
Nov 29 22:36:17.361: INFO: Pod "pod-subpath-test-dynamicpv-cjv4": Phase="Pending", Reason="", readiness=false. Elapsed: 4.332939008s
Nov 29 22:36:19.473: INFO: Pod "pod-subpath-test-dynamicpv-cjv4": Phase="Pending", Reason="", readiness=false. Elapsed: 6.44492689s
Nov 29 22:36:21.583: INFO: Pod "pod-subpath-test-dynamicpv-cjv4": Phase="Pending", Reason="", readiness=false. Elapsed: 8.554904881s
Nov 29 22:36:23.692: INFO: Pod "pod-subpath-test-dynamicpv-cjv4": Phase="Pending", Reason="", readiness=false. Elapsed: 10.66477092s
Nov 29 22:36:25.803: INFO: Pod "pod-subpath-test-dynamicpv-cjv4": Phase="Pending", Reason="", readiness=false. Elapsed: 12.775154495s
Nov 29 22:36:27.913: INFO: Pod "pod-subpath-test-dynamicpv-cjv4": Phase="Pending", Reason="", readiness=false. Elapsed: 14.885272397s
Nov 29 22:36:30.022: INFO: Pod "pod-subpath-test-dynamicpv-cjv4": Phase="Pending", Reason="", readiness=false. Elapsed: 16.994454332s
Nov 29 22:36:32.132: INFO: Pod "pod-subpath-test-dynamicpv-cjv4": Phase="Succeeded", Reason="", readiness=false. Elapsed: 19.104377389s
STEP: Saw pod success
Nov 29 22:36:32.132: INFO: Pod "pod-subpath-test-dynamicpv-cjv4" satisfied condition "Succeeded or Failed"
Nov 29 22:36:32.241: INFO: Trying to get logs from node k8s-agentpool1-92105279-vmss000001 pod pod-subpath-test-dynamicpv-cjv4 container test-container-subpath-dynamicpv-cjv4: <nil>
STEP: delete the pod
Nov 29 22:36:32.506: INFO: Waiting for pod pod-subpath-test-dynamicpv-cjv4 to disappear
Nov 29 22:36:32.616: INFO: Pod pod-subpath-test-dynamicpv-cjv4 no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-cjv4
Nov 29 22:36:32.616: INFO: Deleting pod "pod-subpath-test-dynamicpv-cjv4" in namespace "provisioning-9827"
... skipping 23 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support readOnly file specified in the volumeMount [LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:382
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly]","total":32,"completed":7,"skipped":483,"failed":0}

SSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] 
  should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
  test/e2e/storage/testsuites/multivolume.go:378
... skipping 83 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:378
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":33,"completed":8,"skipped":772,"failed":0}

SSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:37:21.104: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 38 lines ...
Nov 29 22:35:28.281: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comf68sb] to have phase Bound
Nov 29 22:35:28.390: INFO: PersistentVolumeClaim test.csi.azure.comf68sb found but phase is Pending instead of Bound.
Nov 29 22:35:30.502: INFO: PersistentVolumeClaim test.csi.azure.comf68sb found but phase is Pending instead of Bound.
Nov 29 22:35:32.613: INFO: PersistentVolumeClaim test.csi.azure.comf68sb found and phase=Bound (4.331926417s)
STEP: Creating pod pod-subpath-test-dynamicpv-zvl5
STEP: Creating a pod to test subpath
Nov 29 22:35:32.942: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-zvl5" in namespace "provisioning-8274" to be "Succeeded or Failed"
Nov 29 22:35:33.053: INFO: Pod "pod-subpath-test-dynamicpv-zvl5": Phase="Pending", Reason="", readiness=false. Elapsed: 110.786942ms
Nov 29 22:35:35.170: INFO: Pod "pod-subpath-test-dynamicpv-zvl5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.228407714s
Nov 29 22:35:37.280: INFO: Pod "pod-subpath-test-dynamicpv-zvl5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.338407053s
Nov 29 22:35:39.391: INFO: Pod "pod-subpath-test-dynamicpv-zvl5": Phase="Pending", Reason="", readiness=false. Elapsed: 6.449504918s
Nov 29 22:35:41.502: INFO: Pod "pod-subpath-test-dynamicpv-zvl5": Phase="Pending", Reason="", readiness=false. Elapsed: 8.559956343s
Nov 29 22:35:43.612: INFO: Pod "pod-subpath-test-dynamicpv-zvl5": Phase="Pending", Reason="", readiness=false. Elapsed: 10.670021354s
... skipping 10 lines ...
Nov 29 22:36:06.830: INFO: Pod "pod-subpath-test-dynamicpv-zvl5": Phase="Pending", Reason="", readiness=false. Elapsed: 33.88800496s
Nov 29 22:36:08.941: INFO: Pod "pod-subpath-test-dynamicpv-zvl5": Phase="Pending", Reason="", readiness=false. Elapsed: 35.998872534s
Nov 29 22:36:11.052: INFO: Pod "pod-subpath-test-dynamicpv-zvl5": Phase="Pending", Reason="", readiness=false. Elapsed: 38.1107163s
Nov 29 22:36:13.164: INFO: Pod "pod-subpath-test-dynamicpv-zvl5": Phase="Pending", Reason="", readiness=false. Elapsed: 40.222053926s
Nov 29 22:36:15.277: INFO: Pod "pod-subpath-test-dynamicpv-zvl5": Phase="Succeeded", Reason="", readiness=false. Elapsed: 42.33540556s
STEP: Saw pod success
Nov 29 22:36:15.277: INFO: Pod "pod-subpath-test-dynamicpv-zvl5" satisfied condition "Succeeded or Failed"
Nov 29 22:36:15.387: INFO: Trying to get logs from node k8s-agentpool1-92105279-vmss000002 pod pod-subpath-test-dynamicpv-zvl5 container test-container-subpath-dynamicpv-zvl5: <nil>
STEP: delete the pod
Nov 29 22:36:15.619: INFO: Waiting for pod pod-subpath-test-dynamicpv-zvl5 to disappear
Nov 29 22:36:15.728: INFO: Pod pod-subpath-test-dynamicpv-zvl5 no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-zvl5
Nov 29 22:36:15.728: INFO: Deleting pod "pod-subpath-test-dynamicpv-zvl5" in namespace "provisioning-8274"
... skipping 29 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support readOnly directory specified in the volumeMount
    test/e2e/storage/testsuites/subpath.go:367
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly directory specified in the volumeMount","total":39,"completed":7,"skipped":416,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] 
  should access to two volumes with the same volume mode and retain data across pod recreation on the same node
  test/e2e/storage/testsuites/multivolume.go:138
... skipping 196 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:138
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":34,"completed":7,"skipped":520,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:38:11.563: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 86 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support restarting containers using directory as subpath [Slow]
    test/e2e/storage/testsuites/subpath.go:322
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using directory as subpath [Slow]","total":31,"completed":6,"skipped":368,"failed":0}
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:38:29.974: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
[AfterEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/framework/framework.go:188

... skipping 131 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents","total":31,"completed":7,"skipped":335,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] 
  should concurrently access the single volume from pods on the same node
  test/e2e/storage/testsuites/multivolume.go:298
... skipping 157 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:298
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":30,"completed":6,"skipped":774,"failed":0}

SSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes 
  should allow exec of files on the volume
  test/e2e/storage/testsuites/volumes.go:198
... skipping 17 lines ...
Nov 29 22:38:12.599: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comb6279] to have phase Bound
Nov 29 22:38:12.708: INFO: PersistentVolumeClaim test.csi.azure.comb6279 found but phase is Pending instead of Bound.
Nov 29 22:38:14.817: INFO: PersistentVolumeClaim test.csi.azure.comb6279 found but phase is Pending instead of Bound.
Nov 29 22:38:16.938: INFO: PersistentVolumeClaim test.csi.azure.comb6279 found and phase=Bound (4.339176548s)
STEP: Creating pod exec-volume-test-dynamicpv-mttn
STEP: Creating a pod to test exec-volume-test
Nov 29 22:38:17.282: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-mttn" in namespace "volume-8434" to be "Succeeded or Failed"
Nov 29 22:38:17.392: INFO: Pod "exec-volume-test-dynamicpv-mttn": Phase="Pending", Reason="", readiness=false. Elapsed: 109.930443ms
Nov 29 22:38:19.503: INFO: Pod "exec-volume-test-dynamicpv-mttn": Phase="Pending", Reason="", readiness=false. Elapsed: 2.221230212s
Nov 29 22:38:21.613: INFO: Pod "exec-volume-test-dynamicpv-mttn": Phase="Pending", Reason="", readiness=false. Elapsed: 4.331416428s
Nov 29 22:38:23.723: INFO: Pod "exec-volume-test-dynamicpv-mttn": Phase="Pending", Reason="", readiness=false. Elapsed: 6.441526178s
Nov 29 22:38:25.836: INFO: Pod "exec-volume-test-dynamicpv-mttn": Phase="Pending", Reason="", readiness=false. Elapsed: 8.554667063s
Nov 29 22:38:27.946: INFO: Pod "exec-volume-test-dynamicpv-mttn": Phase="Pending", Reason="", readiness=false. Elapsed: 10.664086253s
... skipping 6 lines ...
Nov 29 22:38:42.719: INFO: Pod "exec-volume-test-dynamicpv-mttn": Phase="Pending", Reason="", readiness=false. Elapsed: 25.437682943s
Nov 29 22:38:44.830: INFO: Pod "exec-volume-test-dynamicpv-mttn": Phase="Pending", Reason="", readiness=false. Elapsed: 27.548797679s
Nov 29 22:38:46.946: INFO: Pod "exec-volume-test-dynamicpv-mttn": Phase="Pending", Reason="", readiness=false. Elapsed: 29.663992969s
Nov 29 22:38:49.055: INFO: Pod "exec-volume-test-dynamicpv-mttn": Phase="Pending", Reason="", readiness=false. Elapsed: 31.77384831s
Nov 29 22:38:51.166: INFO: Pod "exec-volume-test-dynamicpv-mttn": Phase="Succeeded", Reason="", readiness=false. Elapsed: 33.884510164s
STEP: Saw pod success
Nov 29 22:38:51.166: INFO: Pod "exec-volume-test-dynamicpv-mttn" satisfied condition "Succeeded or Failed"
Nov 29 22:38:51.276: INFO: Trying to get logs from node k8s-agentpool1-92105279-vmss000002 pod exec-volume-test-dynamicpv-mttn container exec-container-dynamicpv-mttn: <nil>
STEP: delete the pod
Nov 29 22:38:51.534: INFO: Waiting for pod exec-volume-test-dynamicpv-mttn to disappear
Nov 29 22:38:51.642: INFO: Pod exec-volume-test-dynamicpv-mttn no longer exists
STEP: Deleting pod exec-volume-test-dynamicpv-mttn
Nov 29 22:38:51.642: INFO: Deleting pod "exec-volume-test-dynamicpv-mttn" in namespace "volume-8434"
... skipping 27 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should allow exec of files on the volume
    test/e2e/storage/testsuites/volumes.go:198
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should allow exec of files on the volume","total":34,"completed":8,"skipped":582,"failed":0}

SSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes 
  should store data
  test/e2e/storage/testsuites/volumes.go:161
... skipping 116 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] volumes
  test/e2e/storage/framework/testsuite.go:50
    should store data
    test/e2e/storage/testsuites/volumes.go:161
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes should store data","total":39,"completed":8,"skipped":466,"failed":0}

SSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:40:15.815: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 3 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.002 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:258

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 29 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach]
    test/e2e/storage/testsuites/volumemode.go:299

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 159 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:298
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":31,"completed":7,"skipped":416,"failed":0}

SSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
... skipping 259 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:138
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":32,"completed":8,"skipped":495,"failed":0}

SSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (ntfs)][Feature:Windows] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:40:57.405: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 38 lines ...
Nov 29 22:37:22.121: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com92pw9] to have phase Bound
Nov 29 22:37:22.232: INFO: PersistentVolumeClaim test.csi.azure.com92pw9 found but phase is Pending instead of Bound.
Nov 29 22:37:24.343: INFO: PersistentVolumeClaim test.csi.azure.com92pw9 found but phase is Pending instead of Bound.
Nov 29 22:37:26.455: INFO: PersistentVolumeClaim test.csi.azure.com92pw9 found and phase=Bound (4.333369162s)
STEP: [init] starting a pod to use the claim
STEP: [init] check pod success
Nov 29 22:37:26.919: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-wl4n2" in namespace "snapshotting-6515" to be "Succeeded or Failed"
Nov 29 22:37:27.030: INFO: Pod "pvc-snapshottable-tester-wl4n2": Phase="Pending", Reason="", readiness=false. Elapsed: 110.79358ms
Nov 29 22:37:29.148: INFO: Pod "pvc-snapshottable-tester-wl4n2": Phase="Pending", Reason="", readiness=false. Elapsed: 2.228518632s
Nov 29 22:37:31.260: INFO: Pod "pvc-snapshottable-tester-wl4n2": Phase="Pending", Reason="", readiness=false. Elapsed: 4.340333404s
Nov 29 22:37:33.375: INFO: Pod "pvc-snapshottable-tester-wl4n2": Phase="Pending", Reason="", readiness=false. Elapsed: 6.455453731s
Nov 29 22:37:35.486: INFO: Pod "pvc-snapshottable-tester-wl4n2": Phase="Pending", Reason="", readiness=false. Elapsed: 8.566473153s
Nov 29 22:37:37.597: INFO: Pod "pvc-snapshottable-tester-wl4n2": Phase="Pending", Reason="", readiness=false. Elapsed: 10.678169797s
... skipping 15 lines ...
Nov 29 22:38:11.405: INFO: Pod "pvc-snapshottable-tester-wl4n2": Phase="Pending", Reason="", readiness=false. Elapsed: 44.485555181s
Nov 29 22:38:13.517: INFO: Pod "pvc-snapshottable-tester-wl4n2": Phase="Pending", Reason="", readiness=false. Elapsed: 46.597478558s
Nov 29 22:38:15.629: INFO: Pod "pvc-snapshottable-tester-wl4n2": Phase="Pending", Reason="", readiness=false. Elapsed: 48.709872827s
Nov 29 22:38:17.740: INFO: Pod "pvc-snapshottable-tester-wl4n2": Phase="Pending", Reason="", readiness=false. Elapsed: 50.821199794s
Nov 29 22:38:19.856: INFO: Pod "pvc-snapshottable-tester-wl4n2": Phase="Succeeded", Reason="", readiness=false. Elapsed: 52.937076266s
STEP: Saw pod success
Nov 29 22:38:19.856: INFO: Pod "pvc-snapshottable-tester-wl4n2" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
Nov 29 22:38:19.967: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com92pw9] to have phase Bound
Nov 29 22:38:20.078: INFO: PersistentVolumeClaim test.csi.azure.com92pw9 found and phase=Bound (110.823626ms)
STEP: [init] checking the PV
STEP: [init] deleting the pod
Nov 29 22:38:20.448: INFO: Pod pvc-snapshottable-tester-wl4n2 has the following logs: 
... skipping 33 lines ...
Nov 29 22:38:28.952: INFO: WaitUntil finished successfully after 132.739503ms
STEP: getting the snapshot and snapshot content
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Nov 29 22:38:29.512: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-pdvld" in namespace "snapshotting-6515" to be "Succeeded or Failed"
Nov 29 22:38:29.623: INFO: Pod "pvc-snapshottable-data-tester-pdvld": Phase="Pending", Reason="", readiness=false. Elapsed: 110.97318ms
Nov 29 22:38:31.735: INFO: Pod "pvc-snapshottable-data-tester-pdvld": Phase="Pending", Reason="", readiness=false. Elapsed: 2.223206916s
Nov 29 22:38:33.847: INFO: Pod "pvc-snapshottable-data-tester-pdvld": Phase="Pending", Reason="", readiness=false. Elapsed: 4.334775268s
Nov 29 22:38:35.959: INFO: Pod "pvc-snapshottable-data-tester-pdvld": Phase="Pending", Reason="", readiness=false. Elapsed: 6.446503179s
Nov 29 22:38:38.071: INFO: Pod "pvc-snapshottable-data-tester-pdvld": Phase="Pending", Reason="", readiness=false. Elapsed: 8.559159578s
Nov 29 22:38:40.183: INFO: Pod "pvc-snapshottable-data-tester-pdvld": Phase="Pending", Reason="", readiness=false. Elapsed: 10.671448287s
... skipping 30 lines ...
Nov 29 22:39:45.675: INFO: Pod "pvc-snapshottable-data-tester-pdvld": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.163043327s
Nov 29 22:39:47.795: INFO: Pod "pvc-snapshottable-data-tester-pdvld": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.282705512s
Nov 29 22:39:49.906: INFO: Pod "pvc-snapshottable-data-tester-pdvld": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.394252203s
Nov 29 22:39:52.018: INFO: Pod "pvc-snapshottable-data-tester-pdvld": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.506078736s
Nov 29 22:39:54.129: INFO: Pod "pvc-snapshottable-data-tester-pdvld": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m24.617159238s
STEP: Saw pod success
Nov 29 22:39:54.129: INFO: Pod "pvc-snapshottable-data-tester-pdvld" satisfied condition "Succeeded or Failed"
Nov 29 22:39:54.381: INFO: Pod pvc-snapshottable-data-tester-pdvld has the following logs: 
Nov 29 22:39:54.381: INFO: Deleting pod "pvc-snapshottable-data-tester-pdvld" in namespace "snapshotting-6515"
Nov 29 22:39:54.499: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-pdvld" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the snapshot
Nov 29 22:40:15.074: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-ofo3t2sm.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1562512509/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-6515 exec restored-pvc-tester-kctlj --namespace=snapshotting-6515 -- cat /mnt/test/data'
... skipping 47 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        test/e2e/storage/testsuites/snapshottable.go:278
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":33,"completed":9,"skipped":806,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource]
... skipping 5 lines ...
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should check snapshot fields, check restore correctly works, check deletion (ephemeral)
  test/e2e/storage/testsuites/snapshottable.go:177
Nov 29 22:41:02.206: INFO: volume type "DynamicPV" is not ephemeral
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 29 22:41:02.317: INFO: Error getting logs for pod restored-pvc-tester-kctlj: the server could not find the requested resource (get pods restored-pvc-tester-kctlj)
Nov 29 22:41:02.317: INFO: Deleting pod "restored-pvc-tester-kctlj" in namespace "snapshotting-6515"
Nov 29 22:41:02.430: INFO: deleting claim "snapshotting-6515"/"pvc-t2rxn"
Nov 29 22:41:02.541: INFO: deleting snapshot class "snapshotting-6515p8stj"
Nov 29 22:41:02.652: INFO: Waiting up to 5m0s for volumesnapshotclasses snapshotting-6515p8stj to be deleted
Nov 29 22:41:02.763: INFO: volumesnapshotclasses snapshotting-6515p8stj is not found and has been deleted
Nov 29 22:41:02.763: INFO: WaitUntil finished successfully after 110.599586ms
... skipping 35 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:242

    Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 8 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:269

    Distro debian doesn't support ntfs -- skipping

    test/e2e/storage/framework/testsuite.go:127
------------------------------
... skipping 78 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should create read/write inline ephemeral volume
    test/e2e/storage/testsuites/ephemeral.go:196
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should create read/write inline ephemeral volume","total":30,"completed":7,"skipped":786,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:43:13.828: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 99 lines ...
Nov 29 22:40:58.461: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comgkzng] to have phase Bound
Nov 29 22:40:58.569: INFO: PersistentVolumeClaim test.csi.azure.comgkzng found but phase is Pending instead of Bound.
Nov 29 22:41:00.692: INFO: PersistentVolumeClaim test.csi.azure.comgkzng found but phase is Pending instead of Bound.
Nov 29 22:41:02.801: INFO: PersistentVolumeClaim test.csi.azure.comgkzng found and phase=Bound (4.340626346s)
STEP: Creating pod pod-subpath-test-dynamicpv-9wn9
STEP: Creating a pod to test atomic-volume-subpath
Nov 29 22:41:03.128: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-9wn9" in namespace "provisioning-8617" to be "Succeeded or Failed"
Nov 29 22:41:03.238: INFO: Pod "pod-subpath-test-dynamicpv-9wn9": Phase="Pending", Reason="", readiness=false. Elapsed: 110.046771ms
Nov 29 22:41:05.350: INFO: Pod "pod-subpath-test-dynamicpv-9wn9": Phase="Pending", Reason="", readiness=false. Elapsed: 2.221383802s
Nov 29 22:41:07.460: INFO: Pod "pod-subpath-test-dynamicpv-9wn9": Phase="Pending", Reason="", readiness=false. Elapsed: 4.331714511s
Nov 29 22:41:09.570: INFO: Pod "pod-subpath-test-dynamicpv-9wn9": Phase="Pending", Reason="", readiness=false. Elapsed: 6.441617941s
Nov 29 22:41:11.679: INFO: Pod "pod-subpath-test-dynamicpv-9wn9": Phase="Pending", Reason="", readiness=false. Elapsed: 8.551106224s
Nov 29 22:41:13.790: INFO: Pod "pod-subpath-test-dynamicpv-9wn9": Phase="Pending", Reason="", readiness=false. Elapsed: 10.661677016s
... skipping 56 lines ...
Nov 29 22:43:14.078: INFO: Pod "pod-subpath-test-dynamicpv-9wn9": Phase="Running", Reason="", readiness=true. Elapsed: 2m10.949413293s
Nov 29 22:43:16.186: INFO: Pod "pod-subpath-test-dynamicpv-9wn9": Phase="Running", Reason="", readiness=true. Elapsed: 2m13.058005695s
Nov 29 22:43:18.296: INFO: Pod "pod-subpath-test-dynamicpv-9wn9": Phase="Running", Reason="", readiness=true. Elapsed: 2m15.167691668s
Nov 29 22:43:20.408: INFO: Pod "pod-subpath-test-dynamicpv-9wn9": Phase="Running", Reason="", readiness=false. Elapsed: 2m17.27993011s
Nov 29 22:43:22.517: INFO: Pod "pod-subpath-test-dynamicpv-9wn9": Phase="Succeeded", Reason="", readiness=false. Elapsed: 2m19.389134021s
STEP: Saw pod success
Nov 29 22:43:22.517: INFO: Pod "pod-subpath-test-dynamicpv-9wn9" satisfied condition "Succeeded or Failed"
Nov 29 22:43:22.626: INFO: Trying to get logs from node k8s-agentpool1-92105279-vmss000001 pod pod-subpath-test-dynamicpv-9wn9 container test-container-subpath-dynamicpv-9wn9: <nil>
STEP: delete the pod
Nov 29 22:43:22.889: INFO: Waiting for pod pod-subpath-test-dynamicpv-9wn9 to disappear
Nov 29 22:43:22.997: INFO: Pod pod-subpath-test-dynamicpv-9wn9 no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-9wn9
Nov 29 22:43:22.997: INFO: Deleting pod "pod-subpath-test-dynamicpv-9wn9" in namespace "provisioning-8617"
... skipping 29 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support file as subpath [LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:232
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support file as subpath [LinuxOnly]","total":32,"completed":9,"skipped":571,"failed":0}

SSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:44:35.542: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 55 lines ...

    test/e2e/storage/external/external.go:262
------------------------------
SSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode 
  should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299

[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 22:40:04.068: INFO: >>> kubeConfig: /root/tmp1562512509/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename volumemode
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299
Nov 29 22:40:04.831: INFO: Creating resource for dynamic PV
Nov 29 22:40:04.831: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass volumemode-9248-e2e-scbzzh4
STEP: creating a claim
Nov 29 22:40:05.057: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com6jbjs] to have phase Bound
Nov 29 22:40:05.166: INFO: PersistentVolumeClaim test.csi.azure.com6jbjs found but phase is Pending instead of Bound.
Nov 29 22:40:07.276: INFO: PersistentVolumeClaim test.csi.azure.com6jbjs found but phase is Pending instead of Bound.
Nov 29 22:40:09.395: INFO: PersistentVolumeClaim test.csi.azure.com6jbjs found and phase=Bound (4.338517246s)
STEP: Creating pod
STEP: Waiting for the pod to fail
Nov 29 22:40:12.052: INFO: Deleting pod "pod-16f53040-3d36-498a-84b8-6c0f20a974c3" in namespace "volumemode-9248"
Nov 29 22:40:12.165: INFO: Wait up to 5m0s for pod "pod-16f53040-3d36-498a-84b8-6c0f20a974c3" to be fully deleted
STEP: Deleting pvc
Nov 29 22:40:14.388: INFO: Deleting PersistentVolumeClaim "test.csi.azure.com6jbjs"
Nov 29 22:40:14.498: INFO: Waiting up to 5m0s for PersistentVolume pvc-ae4fd49b-afb3-48cf-8457-e98d8ae11f1b to get deleted
Nov 29 22:40:14.607: INFO: PersistentVolume pvc-ae4fd49b-afb3-48cf-8457-e98d8ae11f1b found and phase=Released (108.7547ms)
... skipping 57 lines ...

• [SLOW TEST:271.708 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow]
    test/e2e/storage/testsuites/volumemode.go:299
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should fail to use a volume in a pod with mismatched mode [Slow]","total":34,"completed":9,"skipped":592,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] 
  should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
  test/e2e/storage/testsuites/multivolume.go:323
... skipping 108 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:323
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":31,"completed":8,"skipped":373,"failed":0}

SS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:45:26.898: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 62 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:280

    Distro debian doesn't support ntfs -- skipping

    test/e2e/storage/framework/testsuite.go:127
------------------------------
... skipping 110 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with pvc data source
    test/e2e/storage/testsuites/provisioning.go:421
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with pvc data source","total":39,"completed":9,"skipped":515,"failed":0}

SSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] 
  should concurrently access the single read-only volume from pods on the same node
  test/e2e/storage/testsuites/multivolume.go:423
... skipping 88 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single read-only volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:423
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":31,"completed":8,"skipped":488,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] 
  should concurrently access the single read-only volume from pods on the same node
  test/e2e/storage/testsuites/multivolume.go:423
... skipping 81 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single read-only volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:423
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":33,"completed":10,"skipped":979,"failed":0}

SSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:46:09.175: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 93 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should support multiple inline ephemeral volumes
    test/e2e/storage/testsuites/ephemeral.go:254
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support multiple inline ephemeral volumes","total":39,"completed":10,"skipped":534,"failed":0}

SSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] 
  should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
  test/e2e/storage/testsuites/multivolume.go:323
... skipping 107 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:323
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":30,"completed":8,"skipped":992,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:48:08.986: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 131 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should store data
    test/e2e/storage/testsuites/volumes.go:161
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data","total":34,"completed":10,"skipped":640,"failed":0}

SSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (default fs)] capacity
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:48:17.328: INFO: Driver test.csi.azure.com doesn't publish storage capacity -- skipping
... skipping 24 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:242

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 124 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents","total":33,"completed":11,"skipped":1021,"failed":0}
Nov 29 22:48:47.350: INFO: Running AfterSuite actions on all nodes
Nov 29 22:48:47.350: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2
Nov 29 22:48:47.350: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2
Nov 29 22:48:47.350: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func8.2
Nov 29 22:48:47.350: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage/vsphere.glob..func17.3
Nov 29 22:48:47.350: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage/vsphere.glob..func9.2
... skipping 106 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single read-only volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:423
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":32,"completed":10,"skipped":657,"failed":0}

SSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] 
  should access to two volumes with different volume mode and retain data across pod recreation on different node
  test/e2e/storage/testsuites/multivolume.go:248
... skipping 195 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:248
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":31,"completed":9,"skipped":461,"failed":0}

SSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] 
  should concurrently access the single volume from pods on the same node
  test/e2e/storage/testsuites/multivolume.go:298
... skipping 152 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:298
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":34,"completed":11,"skipped":806,"failed":0}

SSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:50:17.961: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 158 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should store data
    test/e2e/storage/testsuites/volumes.go:161
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should store data","total":31,"completed":9,"skipped":524,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] 
  should access to two volumes with the same volume mode and retain data across pod recreation on different node
  test/e2e/storage/testsuites/multivolume.go:168
... skipping 190 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:168
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":39,"completed":11,"skipped":550,"failed":0}

SSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:50:58.801: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 3 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:269

    Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 140 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should store data
    test/e2e/storage/testsuites/volumes.go:161
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumes should store data","total":32,"completed":11,"skipped":675,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should support existing directories when readOnly specified in the volumeSource
  test/e2e/storage/testsuites/subpath.go:397
... skipping 17 lines ...
Nov 29 22:50:22.088: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.compzwnl] to have phase Bound
Nov 29 22:50:22.197: INFO: PersistentVolumeClaim test.csi.azure.compzwnl found but phase is Pending instead of Bound.
Nov 29 22:50:24.307: INFO: PersistentVolumeClaim test.csi.azure.compzwnl found but phase is Pending instead of Bound.
Nov 29 22:50:26.416: INFO: PersistentVolumeClaim test.csi.azure.compzwnl found and phase=Bound (4.327190095s)
STEP: Creating pod pod-subpath-test-dynamicpv-9tfq
STEP: Creating a pod to test subpath
Nov 29 22:50:26.747: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-9tfq" in namespace "provisioning-1346" to be "Succeeded or Failed"
Nov 29 22:50:26.860: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Pending", Reason="", readiness=false. Elapsed: 113.119499ms
Nov 29 22:50:28.970: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Pending", Reason="", readiness=false. Elapsed: 2.222928534s
Nov 29 22:50:31.079: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Pending", Reason="", readiness=false. Elapsed: 4.332670559s
Nov 29 22:50:33.189: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Pending", Reason="", readiness=false. Elapsed: 6.442044754s
Nov 29 22:50:35.298: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Pending", Reason="", readiness=false. Elapsed: 8.551259083s
Nov 29 22:50:37.407: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Pending", Reason="", readiness=false. Elapsed: 10.660895299s
... skipping 4 lines ...
Nov 29 22:50:47.960: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Pending", Reason="", readiness=false. Elapsed: 21.213556328s
Nov 29 22:50:50.070: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Pending", Reason="", readiness=false. Elapsed: 23.323585049s
Nov 29 22:50:52.191: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Pending", Reason="", readiness=false. Elapsed: 25.444275172s
Nov 29 22:50:54.301: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Pending", Reason="", readiness=false. Elapsed: 27.554422091s
Nov 29 22:50:56.411: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Succeeded", Reason="", readiness=false. Elapsed: 29.664059245s
STEP: Saw pod success
Nov 29 22:50:56.411: INFO: Pod "pod-subpath-test-dynamicpv-9tfq" satisfied condition "Succeeded or Failed"
Nov 29 22:50:56.520: INFO: Trying to get logs from node k8s-agentpool1-92105279-vmss000001 pod pod-subpath-test-dynamicpv-9tfq container test-container-subpath-dynamicpv-9tfq: <nil>
STEP: delete the pod
Nov 29 22:50:56.794: INFO: Waiting for pod pod-subpath-test-dynamicpv-9tfq to disappear
Nov 29 22:50:56.908: INFO: Pod pod-subpath-test-dynamicpv-9tfq no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-9tfq
Nov 29 22:50:56.908: INFO: Deleting pod "pod-subpath-test-dynamicpv-9tfq" in namespace "provisioning-1346"
STEP: Creating pod pod-subpath-test-dynamicpv-9tfq
STEP: Creating a pod to test subpath
Nov 29 22:50:57.145: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-9tfq" in namespace "provisioning-1346" to be "Succeeded or Failed"
Nov 29 22:50:57.253: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Pending", Reason="", readiness=false. Elapsed: 107.919017ms
Nov 29 22:50:59.363: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218146931s
Nov 29 22:51:01.473: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328038274s
Nov 29 22:51:03.590: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Pending", Reason="", readiness=false. Elapsed: 6.444802114s
Nov 29 22:51:05.698: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Pending", Reason="", readiness=false. Elapsed: 8.55371756s
Nov 29 22:51:07.808: INFO: Pod "pod-subpath-test-dynamicpv-9tfq": Phase="Succeeded", Reason="", readiness=false. Elapsed: 10.663183058s
STEP: Saw pod success
Nov 29 22:51:07.808: INFO: Pod "pod-subpath-test-dynamicpv-9tfq" satisfied condition "Succeeded or Failed"
Nov 29 22:51:07.917: INFO: Trying to get logs from node k8s-agentpool1-92105279-vmss000001 pod pod-subpath-test-dynamicpv-9tfq container test-container-subpath-dynamicpv-9tfq: <nil>
STEP: delete the pod
Nov 29 22:51:08.152: INFO: Waiting for pod pod-subpath-test-dynamicpv-9tfq to disappear
Nov 29 22:51:08.261: INFO: Pod pod-subpath-test-dynamicpv-9tfq no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-9tfq
Nov 29 22:51:08.261: INFO: Deleting pod "pod-subpath-test-dynamicpv-9tfq" in namespace "provisioning-1346"
... skipping 29 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support existing directories when readOnly specified in the volumeSource
    test/e2e/storage/testsuites/subpath.go:397
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource","total":31,"completed":10,"skipped":563,"failed":0}

SSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] provisioning
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] provisioning
... skipping 209 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single read-only volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:423
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":39,"completed":12,"skipped":586,"failed":0}

SSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (default fs)] volume-expand
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:53:28.623: INFO: Driver "test.csi.azure.com" does not support volume expansion - skipping
... skipping 215 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:168
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":31,"completed":10,"skipped":481,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning
  test/e2e/storage/framework/testsuite.go:51
Nov 29 22:54:01.535: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 320 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:168
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":34,"completed":12,"skipped":827,"failed":0}

SSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should fail if subpath directory is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:242

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 22:52:22.970: INFO: >>> kubeConfig: /root/tmp1562512509/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename provisioning
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if subpath directory is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:242
Nov 29 22:52:23.740: INFO: Creating resource for dynamic PV
Nov 29 22:52:23.740: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-444-e2e-sc69rp6
STEP: creating a claim
Nov 29 22:52:23.853: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 29 22:52:23.964: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.commdd8p] to have phase Bound
Nov 29 22:52:24.072: INFO: PersistentVolumeClaim test.csi.azure.commdd8p found but phase is Pending instead of Bound.
Nov 29 22:52:26.183: INFO: PersistentVolumeClaim test.csi.azure.commdd8p found but phase is Pending instead of Bound.
Nov 29 22:52:28.292: INFO: PersistentVolumeClaim test.csi.azure.commdd8p found and phase=Bound (4.328059256s)
STEP: Creating pod pod-subpath-test-dynamicpv-6msj
STEP: Checking for subpath error in container status
Nov 29 22:53:34.843: INFO: Deleting pod "pod-subpath-test-dynamicpv-6msj" in namespace "provisioning-444"
Nov 29 22:53:34.956: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-6msj" to be fully deleted
STEP: Deleting pod
Nov 29 22:53:37.174: INFO: Deleting pod "pod-subpath-test-dynamicpv-6msj" in namespace "provisioning-444"
STEP: Deleting pvc
Nov 29 22:53:37.283: INFO: Deleting PersistentVolumeClaim "test.csi.azure.commdd8p"
... skipping 22 lines ...

• [SLOW TEST:146.612 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:242
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow][LinuxOnly]","total":31,"completed":11,"skipped":654,"failed":0}

SSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
... skipping 109 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should support two pods which have the same volume definition
    test/e2e/storage/testsuites/ephemeral.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should support two pods which have the same volume definition","total":32,"completed":12,"skipped":708,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology
... skipping 71 lines ...
STEP: Deleting pod external-injector in namespace multivolume-7696
Nov 29 22:55:05.952: INFO: Waiting for pod external-injector to disappear
Nov 29 22:55:06.061: INFO: Pod external-injector still exists
Nov 29 22:55:08.062: INFO: Waiting for pod external-injector to disappear
Nov 29 22:55:08.172: INFO: Pod external-injector no longer exists
STEP: Creating pod1 with a volume on {Name: Selector:map[] Affinity:nil}
Nov 29 23:00:08.721: FAIL: Unexpected error:
    <*errors.errorString | 0xc0010bb970>: {
        s: "pod \"pod-fc4fd6b9-31f6-4c5a-92f7-9b34e10b24c7\" is not Running: timed out waiting for the condition",
    }
    pod "pod-fc4fd6b9-31f6-4c5a-92f7-9b34e10b24c7" is not Running: timed out waiting for the condition
occurred

... skipping 73 lines ...
Nov 29 23:06:21.482: INFO: PersistentVolume pvc-12fcc63f-82f3-4441-b355-9d4c1a542d90 found and phase=Released (4m36.089557107s)
Nov 29 23:06:26.591: INFO: PersistentVolume pvc-12fcc63f-82f3-4441-b355-9d4c1a542d90 found and phase=Released (4m41.198510482s)
Nov 29 23:06:31.700: INFO: PersistentVolume pvc-12fcc63f-82f3-4441-b355-9d4c1a542d90 found and phase=Released (4m46.307926763s)
Nov 29 23:06:36.809: INFO: PersistentVolume pvc-12fcc63f-82f3-4441-b355-9d4c1a542d90 found and phase=Released (4m51.417129731s)
Nov 29 23:06:41.919: INFO: PersistentVolume pvc-12fcc63f-82f3-4441-b355-9d4c1a542d90 found and phase=Released (4m56.527019487s)
STEP: Deleting sc
Nov 29 23:06:47.033: FAIL: while cleanup resource
Unexpected error:
    <errors.aggregate | len:1, cap:1>: [
        [
            {
                msg: "persistent Volume pvc-12fcc63f-82f3-4441-b355-9d4c1a542d90 not deleted by dynamic provisioner: PersistentVolume pvc-12fcc63f-82f3-4441-b355-9d4c1a542d90 still exists within 5m0s",
                err: {
                    s: "PersistentVolume pvc-12fcc63f-82f3-4441-b355-9d4c1a542d90 still exists within 5m0s",
... skipping 6 lines ...

Full Stack Trace
k8s.io/kubernetes/test/e2e/storage/testsuites.(*multiVolumeTestSuite).DefineTests.func2()
	test/e2e/storage/testsuites/multivolume.go:129 +0x24f
panic({0x6ca56c0, 0xc002dfc4c0})
	/usr/local/go/src/runtime/panic.go:838 +0x207
k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.Fail.func1()
	test/e2e/framework/ginkgowrapper/wrapper.go:63 +0x7d
panic({0x632c000, 0x7a05090})
	/usr/local/go/src/runtime/panic.go:838 +0x207
k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.Fail({0xc0022e6140, 0x13b}, {0xc001b3ca60?, 0x714b92b?, 0xc001b3ca80?})
	test/e2e/framework/ginkgowrapper/wrapper.go:67 +0x197
k8s.io/kubernetes/test/e2e/framework.Fail({0xc0022e6000, 0x126}, {0xc003680aa0?, 0xc0022e6000?, 0xc001182a90?})
	test/e2e/framework/log.go:63 +0x145
k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).match(0xc001b3cbc8, {0x7a29638, 0xa6b18d8}, 0x0, {0x0, 0x0, 0x0})
	vendor/github.com/onsi/gomega/internal/assertion/assertion.go:79 +0x1bd
k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).NotTo(0xc001b3cbc8, {0x7a29638, 0xa6b18d8}, {0x0, 0x0, 0x0})
	vendor/github.com/onsi/gomega/internal/assertion/assertion.go:48 +0x92
k8s.io/kubernetes/test/e2e/framework.ExpectNoErrorWithOffset(0xc002f7b2a0?, {0x7a08d80?, 0xc0010bb970?}, {0x0?, 0x2?, 0xc000232240?})
... skipping 24 lines ...
Nov 29 23:06:47.249: INFO: At 2022-11-29 22:54:46 +0000 UTC - event for external-injector: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-12fcc63f-82f3-4441-b355-9d4c1a542d90" 
Nov 29 23:06:47.249: INFO: At 2022-11-29 22:54:58 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-92105279-vmss000001} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2"
Nov 29 23:06:47.249: INFO: At 2022-11-29 22:54:58 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-92105279-vmss000001} Created: Created container external-injector
Nov 29 23:06:47.249: INFO: At 2022-11-29 22:54:58 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-92105279-vmss000001} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 336.744176ms
Nov 29 23:06:47.249: INFO: At 2022-11-29 22:54:59 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-92105279-vmss000001} Started: Started container external-injector
Nov 29 23:06:47.249: INFO: At 2022-11-29 22:55:05 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-92105279-vmss000001} Killing: Stopping container external-injector
Nov 29 23:06:47.249: INFO: At 2022-11-29 22:55:08 +0000 UTC - event for pod-fc4fd6b9-31f6-4c5a-92f7-9b34e10b24c7: {attachdetach-controller } FailedAttachVolume: Multi-Attach error for volume "pvc-12fcc63f-82f3-4441-b355-9d4c1a542d90" Volume is already exclusively attached to one node and can't be attached to another
Nov 29 23:06:47.249: INFO: At 2022-11-29 22:55:08 +0000 UTC - event for test.csi.azure.com24pr9-cloned: {test.csi.azure.com_k8s-agentpool1-92105279-vmss000002_83f29e4a-d993-40ff-88f5-42bec00669a6 } Provisioning: External provisioner is provisioning volume for claim "multivolume-7696/test.csi.azure.com24pr9-cloned"
Nov 29 23:06:47.249: INFO: At 2022-11-29 22:55:08 +0000 UTC - event for test.csi.azure.com24pr9-cloned: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "test.csi.azure.com" or manually created by system administrator
Nov 29 23:06:47.249: INFO: At 2022-11-29 22:55:10 +0000 UTC - event for test.csi.azure.com24pr9-cloned: {test.csi.azure.com_k8s-agentpool1-92105279-vmss000002_83f29e4a-d993-40ff-88f5-42bec00669a6 } ProvisioningSucceeded: Successfully provisioned volume pvc-582aa7fe-e6b3-4754-a015-feb618bc2bc8
Nov 29 23:06:47.249: INFO: At 2022-11-29 22:55:56 +0000 UTC - event for pod-fc4fd6b9-31f6-4c5a-92f7-9b34e10b24c7: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-12fcc63f-82f3-4441-b355-9d4c1a542d90" 
Nov 29 23:06:47.249: INFO: At 2022-11-29 22:57:11 +0000 UTC - event for pod-fc4fd6b9-31f6-4c5a-92f7-9b34e10b24c7: {kubelet k8s-agentpool1-92105279-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 kube-api-access-xkltq]: timed out waiting for the condition
Nov 29 23:06:47.249: INFO: At 2022-11-29 22:57:56 +0000 UTC - event for pod-fc4fd6b9-31f6-4c5a-92f7-9b34e10b24c7: {kubelet k8s-agentpool1-92105279-vmss000000} FailedMount: MountVolume.MountDevice failed for volume "pvc-12fcc63f-82f3-4441-b355-9d4c1a542d90" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 29 23:06:47.249: INFO: At 2022-11-29 23:00:54 +0000 UTC - event for pod-fc4fd6b9-31f6-4c5a-92f7-9b34e10b24c7: {kubelet k8s-agentpool1-92105279-vmss000000} FailedMount: MountVolume.MountDevice failed for volume "pvc-12fcc63f-82f3-4441-b355-9d4c1a542d90" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ofo3t2sm/providers/Microsoft.Compute/disks/pvc-12fcc63f-82f3-4441-b355-9d4c1a542d90) to be attached to node (k8s-agentpool1-92105279-vmss000000): rpc error: code = Unknown desc = context deadline exceeded
Nov 29 23:06:47.357: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov 29 23:06:47.357: INFO: 
Nov 29 23:06:47.573: INFO: 
Logging node info for node k8s-agentpool1-92105279-vmss000000
Nov 29 23:06:47.683: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-92105279-vmss000000    e7716b3a-d547-407e-a9ed-2706618be43c 26675 0 2022-11-29 22:04:46 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D8s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westeurope failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-ofo3t2sm kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-92105279-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_D8s_v3 storageprofile:managed storagetier:Premium_LRS topology.kubernetes.io/region:westeurope topology.kubernetes.io/zone:0 topology.test.csi.azure.com/zone:] map[csi.volume.kubernetes.io/nodeid:{"test.csi.azure.com":"k8s-agentpool1-92105279-vmss000000"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2022-11-29 22:04:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{}}}} } {kubectl-label Update v1 2022-11-29 22:04:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {cloud-controller-manager Update v1 2022-11-29 22:05:02 +0000 UTC FieldsV1 {"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.3.0/24\"":{}}}} } {kube-controller-manager Update v1 2022-11-29 22:05:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {cloud-controller-manager Update v1 2022-11-29 22:05:49 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {cloud-node-manager Update v1 2022-11-29 22:06:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2022-11-29 22:49:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.test.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2022-11-29 22:55:56 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ofo3t2sm/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-92105279-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{31025332224 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{33672720384 0} {<nil>} 32883516Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{27922798956 0} {<nil>} 27922798956 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{32886288384 0} {<nil>} 32115516Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-29 22:05:49 +0000 UTC,LastTransitionTime:2022-11-29 22:05:49 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-29 23:03:44 +0000 UTC,LastTransitionTime:2022-11-29 22:04:40 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-29 23:03:44 +0000 UTC,LastTransitionTime:2022-11-29 22:04:40 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-29 23:03:44 +0000 UTC,LastTransitionTime:2022-11-29 22:04:40 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-29 23:03:44 +0000 UTC,LastTransitionTime:2022-11-29 22:05:02 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool1-92105279-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2d155693e7fa4281a08fb77e113c1f09,SystemUUID:badc595a-a00b-e64f-ad71-6b1c80de3c78,BootID:ae2f9bff-8abd-4f93-a225-c6b443284255,KernelVersion:5.4.0-1094-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.14+azure-1,KubeletVersion:v1.23.14,KubeProxyVersion:v1.23.14,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253346057,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:664ec9627d47a4160789ad6994367128b25007b164773c5f1e598028440a158a k8sprow.azurecr.io/azuredisk-csi:latest-v2-5fb3153a08fb6a2968d7e7bbe760d033ea35fc7a],SizeBytes:142859492,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:f5241226198f5a54d22540acf2b3933ea0f49458f90c51fc75833d0c428687b8 k8s.gcr.io/e2e-test-images/agnhost:2.36],SizeBytes:125426639,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:efa8e4b7776fa9fb2279745008780cefec632ee5fdb081e0f6e234ea4b8bbf61 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.14],SizeBytes:112312488,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:949db4eea46ea31f9afa0b49ad0b55f7a31f91aa03b408e0ac9b7720c8ef7d29 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.11],SizeBytes:51376235,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:72fbc9d17947d1a54c09a16671e6810fd7c0d8f09c0502fdbdd212c2192451a7 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.14],SizeBytes:51044459,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:075ea1f8270312350f1396ab6677251e803e61a523822d5abfa5e6acd180cfab mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.11],SizeBytes:50806891,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:eb7d329559d954093f7a234bc0ff322764de48a0b2a1007229032fb53e7bb9eb mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.24.0],SizeBytes:49070187,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:2dfff4dc10f6d25b0a34c01bf4d166540cbf1ff2bccd52635350039138eb23c4 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.18],SizeBytes:48775275,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3e23b9ab6bfee42a1751019c21a60aa474964bd4cba00cbed5055430424242cc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.24.0],SizeBytes:48500843,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:acec9ee6626c2cdea89107ba191ae36e6f60339c9b3b3215b5a59bf0e2e730ba mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.14],SizeBytes:47853675,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:759ac7020e982884b99121583bd577658b5663b1c1ed69746356aa6ce9879da4 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.18],SizeBytes:45850731,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:2fbd1e1a0538a06f2061afd45975df70c942654aa7f86e920720169ee439c2d6 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.5.1],SizeBytes:19743433,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:31547791294872570393470991481c2477a311031d3a03e0ae54eb164347dc34 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0],SizeBytes:17857485,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ofo3t2sm/providers/Microsoft.Compute/disks/pvc-1ac83ddd-a789-425d-9509-b034bcbde378],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ofo3t2sm/providers/Microsoft.Compute/disks/pvc-1ac83ddd-a789-425d-9509-b034bcbde378,DevicePath:,},},Config:nil,},}
Nov 29 23:06:47.684: INFO: 
... skipping 110 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [Measurement]
    test/e2e/storage/testsuites/multivolume.go:378

    Nov 29 23:00:08.721: Unexpected error:
        <*errors.errorString | 0xc0010bb970>: {
            s: "pod \"pod-fc4fd6b9-31f6-4c5a-92f7-9b34e10b24c7\" is not Running: timed out waiting for the condition",
        }
        pod "pod-fc4fd6b9-31f6-4c5a-92f7-9b34e10b24c7" is not Running: timed out waiting for the condition
    occurred

    test/e2e/storage/testsuites/multivolume.go:696
------------------------------
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":34,"completed":12,"skipped":845,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]"]}

SSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
... skipping 236 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should support two pods which have the same volume definition
    test/e2e/storage/testsuites/ephemeral.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support two pods which have the same volume definition","total":31,"completed":12,"skipped":749,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 23:07:46.697: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 84 lines ...
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 23:07:46.814: INFO: >>> kubeConfig: /root/tmp1562512509/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename topology
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  test/e2e/storage/testsuites/topology.go:194
Nov 29 23:07:47.576: INFO: Driver didn't provide topology keys -- skipping
[AfterEach] [Testpattern: Dynamic PV (delayed binding)] topology
  test/e2e/framework/framework.go:188
Nov 29 23:07:47.577: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "topology-9655" for this suite.


S [SKIPPING] [1.088 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (delayed binding)] topology
  test/e2e/storage/framework/testsuite.go:50
    should fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement]
    test/e2e/storage/testsuites/topology.go:194

    Driver didn't provide topology keys -- skipping

    test/e2e/storage/testsuites/topology.go:126
------------------------------
... skipping 95 lines ...
Nov 29 22:55:08.291: INFO: >>> kubeConfig: /root/tmp1562512509/kubeconfig/kubeconfig.westeurope.json
Nov 29 22:55:08.292: INFO: ExecWithOptions: Clientset creation
Nov 29 22:55:08.292: INFO: ExecWithOptions: execute(POST https://kubetest-ofo3t2sm.westeurope.cloudapp.azure.com/api/v1/namespaces/multivolume-2962/pods/pod-64739c7b-38ec-45e2-b2da-230500376278/exec?command=%2Fbin%2Fsh&command=-c&command=dd+if%3D%2Fmnt%2Fvolume2%2Ffile1.txt++bs%3D64+count%3D1+%7C+sha256sum+%7C+grep+-Fq+94ef2fe32af6899f1df1a83e3060f5c116c3f65b7aa39e8600bb4e30d48caf24&container=write-pod&container=write-pod&stderr=true&stdout=true)
Nov 29 22:55:09.201: INFO: Deleting pod "pod-64739c7b-38ec-45e2-b2da-230500376278" in namespace "multivolume-2962"
Nov 29 22:55:09.314: INFO: Wait up to 5m0s for pod "pod-64739c7b-38ec-45e2-b2da-230500376278" to be fully deleted
STEP: Creating pod on {Name: Selector:map[] Affinity:&Affinity{NodeAffinity:&NodeAffinity{RequiredDuringSchedulingIgnoredDuringExecution:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{},MatchFields:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:metadata.name,Operator:NotIn,Values:[k8s-agentpool1-92105279-vmss000000],},},},},},PreferredDuringSchedulingIgnoredDuringExecution:[]PreferredSchedulingTerm{},},PodAffinity:nil,PodAntiAffinity:nil,}} with multiple volumes
Nov 29 23:00:11.990: FAIL: Unexpected error:
    <*errors.errorString | 0xc0017521f0>: {
        s: "pod \"pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e\" is not Running: timed out waiting for the condition",
    }
    pod "pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e" is not Running: timed out waiting for the condition
occurred

... skipping 130 lines ...
Nov 29 23:10:48.588: INFO: PersistentVolume pvc-80afd1e9-64ee-4d84-b81c-9f6e0961ef79 found and phase=Released (4m0.347522576s)
Nov 29 23:10:53.700: INFO: PersistentVolume pvc-80afd1e9-64ee-4d84-b81c-9f6e0961ef79 found and phase=Released (4m5.459541851s)
Nov 29 23:10:58.811: INFO: PersistentVolume pvc-80afd1e9-64ee-4d84-b81c-9f6e0961ef79 found and phase=Released (4m10.570709874s)
Nov 29 23:11:03.926: INFO: PersistentVolume pvc-80afd1e9-64ee-4d84-b81c-9f6e0961ef79 found and phase=Released (4m15.685510461s)
Nov 29 23:11:09.038: INFO: PersistentVolume pvc-80afd1e9-64ee-4d84-b81c-9f6e0961ef79 was removed
STEP: Deleting sc
Nov 29 23:11:09.151: FAIL: while cleanup resource
Unexpected error:
    <errors.aggregate | len:1, cap:1>: [
        [
            {
                msg: "persistent Volume pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4 not deleted by dynamic provisioner: PersistentVolume pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4 still exists within 5m0s",
                err: {
                    s: "PersistentVolume pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4 still exists within 5m0s",
... skipping 6 lines ...

Full Stack Trace
k8s.io/kubernetes/test/e2e/storage/testsuites.(*multiVolumeTestSuite).DefineTests.func2()
	test/e2e/storage/testsuites/multivolume.go:129 +0x24f
panic({0x6ca56c0, 0xc002eec440})
	/usr/local/go/src/runtime/panic.go:838 +0x207
k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.Fail.func1()
	test/e2e/framework/ginkgowrapper/wrapper.go:63 +0x7d
panic({0x632c000, 0x7a05090})
	/usr/local/go/src/runtime/panic.go:838 +0x207
k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.Fail({0xc0006543c0, 0x13b}, {0xc002992a50?, 0x714b92b?, 0xc002992a70?})
	test/e2e/framework/ginkgowrapper/wrapper.go:67 +0x197
k8s.io/kubernetes/test/e2e/framework.Fail({0xc000654280, 0x126}, {0xc0039338e0?, 0xc000654280?, 0xc0014340d0?})
	test/e2e/framework/log.go:63 +0x145
k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).match(0xc002992bb8, {0x7a29638, 0xa6b18d8}, 0x0, {0x0, 0x0, 0x0})
	vendor/github.com/onsi/gomega/internal/assertion/assertion.go:79 +0x1bd
k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).NotTo(0xc002992bb8, {0x7a29638, 0xa6b18d8}, {0x0, 0x0, 0x0})
	vendor/github.com/onsi/gomega/internal/assertion/assertion.go:48 +0x92
k8s.io/kubernetes/test/e2e/framework.ExpectNoErrorWithOffset(0x7a6bf58?, {0x7a08d80?, 0xc0017521f0?}, {0x0?, 0x1?, 0x251cfd0?})
... skipping 32 lines ...
Nov 29 23:11:09.375: INFO: At 2022-11-29 22:54:56 +0000 UTC - event for pod-64739c7b-38ec-45e2-b2da-230500376278: {kubelet k8s-agentpool1-92105279-vmss000000} SuccessfulMountVolume: MapVolume.MapPodDevice succeeded for volume "pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4/dev"
Nov 29 23:11:09.375: INFO: At 2022-11-29 22:54:57 +0000 UTC - event for pod-64739c7b-38ec-45e2-b2da-230500376278: {kubelet k8s-agentpool1-92105279-vmss000000} Started: Started container write-pod
Nov 29 23:11:09.375: INFO: At 2022-11-29 22:54:57 +0000 UTC - event for pod-64739c7b-38ec-45e2-b2da-230500376278: {kubelet k8s-agentpool1-92105279-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2"
Nov 29 23:11:09.375: INFO: At 2022-11-29 22:54:57 +0000 UTC - event for pod-64739c7b-38ec-45e2-b2da-230500376278: {kubelet k8s-agentpool1-92105279-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 388.327666ms
Nov 29 23:11:09.375: INFO: At 2022-11-29 22:54:57 +0000 UTC - event for pod-64739c7b-38ec-45e2-b2da-230500376278: {kubelet k8s-agentpool1-92105279-vmss000000} Created: Created container write-pod
Nov 29 23:11:09.375: INFO: At 2022-11-29 22:55:09 +0000 UTC - event for pod-64739c7b-38ec-45e2-b2da-230500376278: {kubelet k8s-agentpool1-92105279-vmss000000} Killing: Stopping container write-pod
Nov 29 23:11:09.375: INFO: At 2022-11-29 22:55:11 +0000 UTC - event for pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e: {attachdetach-controller } FailedAttachVolume: Multi-Attach error for volume "pvc-80afd1e9-64ee-4d84-b81c-9f6e0961ef79" Volume is already exclusively attached to one node and can't be attached to another
Nov 29 23:11:09.375: INFO: At 2022-11-29 22:55:11 +0000 UTC - event for pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e: {attachdetach-controller } FailedAttachVolume: Multi-Attach error for volume "pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4" Volume is already exclusively attached to one node and can't be attached to another
Nov 29 23:11:09.375: INFO: At 2022-11-29 22:56:29 +0000 UTC - event for pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-80afd1e9-64ee-4d84-b81c-9f6e0961ef79" 
Nov 29 23:11:09.375: INFO: At 2022-11-29 22:56:31 +0000 UTC - event for pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4" 
Nov 29 23:11:09.375: INFO: At 2022-11-29 22:57:14 +0000 UTC - event for pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e: {kubelet k8s-agentpool1-92105279-vmss000001} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume2 volume1], unattached volumes=[volume2 kube-api-access-cr97g volume1]: timed out waiting for the condition
Nov 29 23:11:09.376: INFO: At 2022-11-29 22:58:30 +0000 UTC - event for pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e: {kubelet k8s-agentpool1-92105279-vmss000001} FailedMount: MountVolume.MountDevice failed for volume "pvc-80afd1e9-64ee-4d84-b81c-9f6e0961ef79" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 29 23:11:09.376: INFO: At 2022-11-29 22:58:31 +0000 UTC - event for pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e: {kubelet k8s-agentpool1-92105279-vmss000001} FailedMapVolume: MapVolume.SetUpDevice failed for volume "pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 29 23:11:09.376: INFO: At 2022-11-29 22:59:27 +0000 UTC - event for pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e: {kubelet k8s-agentpool1-92105279-vmss000001} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1 volume2], unattached volumes=[kube-api-access-cr97g volume1 volume2]: timed out waiting for the condition
Nov 29 23:11:09.376: INFO: At 2022-11-29 23:00:32 +0000 UTC - event for pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e: {kubelet k8s-agentpool1-92105279-vmss000001} FailedMapVolume: MapVolume.SetUpDevice failed for volume "pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ofo3t2sm/providers/Microsoft.Compute/disks/pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4) to be attached to node (k8s-agentpool1-92105279-vmss000001): context deadline exceeded
Nov 29 23:11:09.376: INFO: At 2022-11-29 23:00:56 +0000 UTC - event for pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e: {kubelet k8s-agentpool1-92105279-vmss000001} FailedMapVolume: MapVolume.SetUpDevice failed for volume "pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ofo3t2sm/providers/Microsoft.Compute/disks/pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4) to be attached to node (k8s-agentpool1-92105279-vmss000001): rpc error: code = Unknown desc = context deadline exceeded
Nov 29 23:11:09.376: INFO: At 2022-11-29 23:00:57 +0000 UTC - event for pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e: {kubelet k8s-agentpool1-92105279-vmss000001} FailedMount: MountVolume.MountDevice failed for volume "pvc-80afd1e9-64ee-4d84-b81c-9f6e0961ef79" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ofo3t2sm/providers/Microsoft.Compute/disks/pvc-80afd1e9-64ee-4d84-b81c-9f6e0961ef79) to be attached to node (k8s-agentpool1-92105279-vmss000001): rpc error: code = Unknown desc = context deadline exceeded
Nov 29 23:11:09.376: INFO: At 2022-11-29 23:01:57 +0000 UTC - event for pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e: {kubelet k8s-agentpool1-92105279-vmss000001} FailedMount: MountVolume.MountDevice failed while expanding volume for volume "pvc-80afd1e9-64ee-4d84-b81c-9f6e0961ef79" : mountVolume.NodeExpandVolume get PVC failed : persistentvolumeclaims "test.csi.azure.comr7j74" is forbidden: User "system:node:k8s-agentpool1-92105279-vmss000001" cannot get resource "persistentvolumeclaims" in API group "" in the namespace "multivolume-2962": no relationship found between node 'k8s-agentpool1-92105279-vmss000001' and this object
Nov 29 23:11:09.376: INFO: At 2022-11-29 23:01:58 +0000 UTC - event for pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e: {kubelet k8s-agentpool1-92105279-vmss000001} SuccessfulMountVolume: MapVolume.MapPodDevice succeeded for volume "pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4/dev"
Nov 29 23:11:09.376: INFO: At 2022-11-29 23:01:58 +0000 UTC - event for pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e: {kubelet k8s-agentpool1-92105279-vmss000001} SuccessfulMountVolume: MapVolume.MapPodDevice succeeded for volume "pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4" volumeMapPath "/var/lib/kubelet/pods/0fed3334-37ee-4334-aa92-bdb51a7a210f/volumeDevices/kubernetes.io~csi"
Nov 29 23:11:09.376: INFO: At 2022-11-29 23:01:58 +0000 UTC - event for pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e: {kubelet k8s-agentpool1-92105279-vmss000001} FailedMapVolume: MapVolume.MarkVolumeAsMounted failed while expanding volume for volume "pvc-0a89b1f9-8cc7-4409-bc56-7e1a218193a4" : mountVolume.NodeExpandVolume get PVC failed : persistentvolumeclaims "test.csi.azure.comwjzrf" is forbidden: User "system:node:k8s-agentpool1-92105279-vmss000001" cannot get resource "persistentvolumeclaims" in API group "" in the namespace "multivolume-2962": no relationship found between node 'k8s-agentpool1-92105279-vmss000001' and this object
Nov 29 23:11:09.485: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov 29 23:11:09.485: INFO: 
Nov 29 23:11:09.703: INFO: 
Logging node info for node k8s-agentpool1-92105279-vmss000000
Nov 29 23:11:09.813: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-92105279-vmss000000    e7716b3a-d547-407e-a9ed-2706618be43c 28396 0 2022-11-29 22:04:46 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D8s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westeurope failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-ofo3t2sm kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-92105279-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_D8s_v3 storageprofile:managed storagetier:Premium_LRS topology.kubernetes.io/region:westeurope topology.kubernetes.io/zone:0 topology.test.csi.azure.com/zone:] map[csi.volume.kubernetes.io/nodeid:{"test.csi.azure.com":"k8s-agentpool1-92105279-vmss000000"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2022-11-29 22:04:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{}}}} } {kubectl-label Update v1 2022-11-29 22:04:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {cloud-controller-manager Update v1 2022-11-29 22:05:02 +0000 UTC FieldsV1 {"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.3.0/24\"":{}}}} } {kube-controller-manager Update v1 2022-11-29 22:05:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {cloud-controller-manager Update v1 2022-11-29 22:05:49 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {cloud-node-manager Update v1 2022-11-29 22:06:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2022-11-29 23:11:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.test.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ofo3t2sm/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-92105279-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{31025332224 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{33672720384 0} {<nil>} 32883516Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{27922798956 0} {<nil>} 27922798956 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{32886288384 0} {<nil>} 32115516Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-29 22:05:49 +0000 UTC,LastTransitionTime:2022-11-29 22:05:49 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-29 23:11:03 +0000 UTC,LastTransitionTime:2022-11-29 22:04:40 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-29 23:11:03 +0000 UTC,LastTransitionTime:2022-11-29 22:04:40 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-29 23:11:03 +0000 UTC,LastTransitionTime:2022-11-29 22:04:40 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-29 23:11:03 +0000 UTC,LastTransitionTime:2022-11-29 22:05:02 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool1-92105279-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2d155693e7fa4281a08fb77e113c1f09,SystemUUID:badc595a-a00b-e64f-ad71-6b1c80de3c78,BootID:ae2f9bff-8abd-4f93-a225-c6b443284255,KernelVersion:5.4.0-1094-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.14+azure-1,KubeletVersion:v1.23.14,KubeProxyVersion:v1.23.14,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253346057,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:664ec9627d47a4160789ad6994367128b25007b164773c5f1e598028440a158a k8sprow.azurecr.io/azuredisk-csi:latest-v2-5fb3153a08fb6a2968d7e7bbe760d033ea35fc7a],SizeBytes:142859492,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:f5241226198f5a54d22540acf2b3933ea0f49458f90c51fc75833d0c428687b8 k8s.gcr.io/e2e-test-images/agnhost:2.36],SizeBytes:125426639,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:efa8e4b7776fa9fb2279745008780cefec632ee5fdb081e0f6e234ea4b8bbf61 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.14],SizeBytes:112312488,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:949db4eea46ea31f9afa0b49ad0b55f7a31f91aa03b408e0ac9b7720c8ef7d29 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.11],SizeBytes:51376235,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:72fbc9d17947d1a54c09a16671e6810fd7c0d8f09c0502fdbdd212c2192451a7 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.14],SizeBytes:51044459,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:075ea1f8270312350f1396ab6677251e803e61a523822d5abfa5e6acd180cfab mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.11],SizeBytes:50806891,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:eb7d329559d954093f7a234bc0ff322764de48a0b2a1007229032fb53e7bb9eb mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.24.0],SizeBytes:49070187,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:2dfff4dc10f6d25b0a34c01bf4d166540cbf1ff2bccd52635350039138eb23c4 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.18],SizeBytes:48775275,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3e23b9ab6bfee42a1751019c21a60aa474964bd4cba00cbed5055430424242cc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.24.0],SizeBytes:48500843,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:acec9ee6626c2cdea89107ba191ae36e6f60339c9b3b3215b5a59bf0e2e730ba mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.14],SizeBytes:47853675,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:759ac7020e982884b99121583bd577658b5663b1c1ed69746356aa6ce9879da4 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.18],SizeBytes:45850731,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:2fbd1e1a0538a06f2061afd45975df70c942654aa7f86e920720169ee439c2d6 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.5.1],SizeBytes:19743433,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:31547791294872570393470991481c2477a311031d3a03e0ae54eb164347dc34 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0],SizeBytes:17857485,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov 29 23:11:09.813: INFO: 
... skipping 108 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on different node [Measurement]
    test/e2e/storage/testsuites/multivolume.go:248

    Nov 29 23:00:11.990: Unexpected error:
        <*errors.errorString | 0xc0017521f0>: {
            s: "pod \"pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e\" is not Running: timed out waiting for the condition",
        }
        pod "pod-ddc41e69-d5d3-4bbe-a3ac-948cdaf6df8e" is not Running: timed out waiting for the condition
    occurred

    test/e2e/storage/testsuites/multivolume.go:497
------------------------------
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":39,"completed":12,"skipped":615,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] volumeIO
  test/e2e/storage/framework/testsuite.go:51
Nov 29 23:11:12.948: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 182 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with pvc data source
    test/e2e/storage/testsuites/provisioning.go:421
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with pvc data source","total":31,"completed":11,"skipped":730,"failed":0}

SSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 29 23:12:05.751: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 331 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:209
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":34,"completed":13,"skipped":947,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes 
  should allow exec of files on the volume
  test/e2e/storage/testsuites/volumes.go:198
... skipping 17 lines ...
Nov 29 23:11:14.021: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com9hvnv] to have phase Bound
Nov 29 23:11:14.130: INFO: PersistentVolumeClaim test.csi.azure.com9hvnv found but phase is Pending instead of Bound.
Nov 29 23:11:16.241: INFO: PersistentVolumeClaim test.csi.azure.com9hvnv found but phase is Pending instead of Bound.
Nov 29 23:11:18.352: INFO: PersistentVolumeClaim test.csi.azure.com9hvnv found and phase=Bound (4.330331907s)
STEP: Creating pod exec-volume-test-dynamicpv-rtv5
STEP: Creating a pod to test exec-volume-test
Nov 29 23:11:18.686: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-rtv5" in namespace "volume-9374" to be "Succeeded or Failed"
Nov 29 23:11:18.796: INFO: Pod "exec-volume-test-dynamicpv-rtv5": Phase="Pending", Reason="", readiness=false. Elapsed: 109.849917ms
Nov 29 23:11:20.907: INFO: Pod "exec-volume-test-dynamicpv-rtv5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.220220642s
Nov 29 23:11:23.019: INFO: Pod "exec-volume-test-dynamicpv-rtv5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.332960661s
Nov 29 23:11:25.129: INFO: Pod "exec-volume-test-dynamicpv-rtv5": Phase="Pending", Reason="", readiness=false. Elapsed: 6.442924208s
Nov 29 23:11:27.240: INFO: Pod "exec-volume-test-dynamicpv-rtv5": Phase="Pending", Reason="", readiness=false. Elapsed: 8.553288191s
Nov 29 23:11:29.350: INFO: Pod "exec-volume-test-dynamicpv-rtv5": Phase="Pending", Reason="", readiness=false. Elapsed: 10.663917194s
... skipping 9 lines ...
Nov 29 23:11:50.472: INFO: Pod "exec-volume-test-dynamicpv-rtv5": Phase="Pending", Reason="", readiness=false. Elapsed: 31.785631562s
Nov 29 23:11:52.583: INFO: Pod "exec-volume-test-dynamicpv-rtv5": Phase="Pending", Reason="", readiness=false. Elapsed: 33.896744074s
Nov 29 23:11:54.695: INFO: Pod "exec-volume-test-dynamicpv-rtv5": Phase="Pending", Reason="", readiness=false. Elapsed: 36.008467844s
Nov 29 23:11:56.805: INFO: Pod "exec-volume-test-dynamicpv-rtv5": Phase="Pending", Reason="", readiness=false. Elapsed: 38.118697185s
Nov 29 23:11:58.918: INFO: Pod "exec-volume-test-dynamicpv-rtv5": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.231357866s
STEP: Saw pod success
Nov 29 23:11:58.918: INFO: Pod "exec-volume-test-dynamicpv-rtv5" satisfied condition "Succeeded or Failed"
Nov 29 23:11:59.040: INFO: Trying to get logs from node k8s-agentpool1-92105279-vmss000001 pod exec-volume-test-dynamicpv-rtv5 container exec-container-dynamicpv-rtv5: <nil>
STEP: delete the pod
Nov 29 23:11:59.279: INFO: Waiting for pod exec-volume-test-dynamicpv-rtv5 to disappear
Nov 29 23:11:59.387: INFO: Pod exec-volume-test-dynamicpv-rtv5 no longer exists
STEP: Deleting pod exec-volume-test-dynamicpv-rtv5
Nov 29 23:11:59.388: INFO: Deleting pod "exec-volume-test-dynamicpv-rtv5" in namespace "volume-9374"
... skipping 21 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] volumes
  test/e2e/storage/framework/testsuite.go:50
    should allow exec of files on the volume
    test/e2e/storage/testsuites/volumes.go:198
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes should allow exec of files on the volume","total":39,"completed":13,"skipped":725,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

SSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] 
  should access to two volumes with different volume mode and retain data across pod recreation on different node
  test/e2e/storage/testsuites/multivolume.go:248
... skipping 24 lines ...
STEP: creating a claim
Nov 29 22:56:36.250: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com9qk8j] to have phase Bound
Nov 29 22:56:36.358: INFO: PersistentVolumeClaim test.csi.azure.com9qk8j found but phase is Pending instead of Bound.
Nov 29 22:56:38.467: INFO: PersistentVolumeClaim test.csi.azure.com9qk8j found but phase is Pending instead of Bound.
Nov 29 22:56:40.577: INFO: PersistentVolumeClaim test.csi.azure.com9qk8j found and phase=Bound (4.32635794s)
STEP: Creating pod on {Name: Selector:map[] Affinity:nil} with multiple volumes
Nov 29 23:01:41.261: FAIL: Unexpected error:
    <*errors.errorString | 0xc0022482b0>: {
        s: "pod \"pod-d93c7424-44c3-45d1-bec4-966fd5ea5bb9\" is not Running: timed out waiting for the condition",
    }
    pod "pod-d93c7424-44c3-45d1-bec4-966fd5ea5bb9" is not Running: timed out waiting for the condition
occurred

... skipping 130 lines ...
Nov 29 23:12:21.779: INFO: PersistentVolume pvc-0e72e110-b981-447c-adc3-c886d695e36e found and phase=Released (4m0.302429121s)
Nov 29 23:12:26.889: INFO: PersistentVolume pvc-0e72e110-b981-447c-adc3-c886d695e36e found and phase=Released (4m5.413154185s)
Nov 29 23:12:31.999: INFO: PersistentVolume pvc-0e72e110-b981-447c-adc3-c886d695e36e found and phase=Released (4m10.522690719s)
Nov 29 23:12:37.110: INFO: PersistentVolume pvc-0e72e110-b981-447c-adc3-c886d695e36e found and phase=Released (4m15.633775917s)
Nov 29 23:12:42.218: INFO: PersistentVolume pvc-0e72e110-b981-447c-adc3-c886d695e36e was removed
STEP: Deleting sc
Nov 29 23:12:42.329: FAIL: while cleanup resource
Unexpected error:
    <errors.aggregate | len:1, cap:1>: [
        [
            {
                msg: "persistent Volume pvc-474611e1-0421-4ef4-9037-0e3d28c5d897 not deleted by dynamic provisioner: PersistentVolume pvc-474611e1-0421-4ef4-9037-0e3d28c5d897 still exists within 5m0s",
                err: {
                    s: "PersistentVolume pvc-474611e1-0421-4ef4-9037-0e3d28c5d897 still exists within 5m0s",
... skipping 6 lines ...

Full Stack Trace
k8s.io/kubernetes/test/e2e/storage/testsuites.(*multiVolumeTestSuite).DefineTests.func2()
	test/e2e/storage/testsuites/multivolume.go:129 +0x24f
panic({0x6ca56c0, 0xc001ed4c80})
	/usr/local/go/src/runtime/panic.go:838 +0x207
k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.Fail.func1()
	test/e2e/framework/ginkgowrapper/wrapper.go:63 +0x7d
panic({0x632c000, 0x7a05090})
	/usr/local/go/src/runtime/panic.go:838 +0x207
k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.Fail({0xc0004ae3c0, 0x13b}, {0xc0013b8a50?, 0x714b92b?, 0xc0013b8a70?})
	test/e2e/framework/ginkgowrapper/wrapper.go:67 +0x197
k8s.io/kubernetes/test/e2e/framework.Fail({0xc0004ae280, 0x126}, {0xc003ac3f30?, 0xc0004ae280?, 0xc002aea1a0?})
	test/e2e/framework/log.go:63 +0x145
k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).match(0xc0013b8bb8, {0x7a29638, 0xa6b18d8}, 0x0, {0x0, 0x0, 0x0})
	vendor/github.com/onsi/gomega/internal/assertion/assertion.go:79 +0x1bd
k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).NotTo(0xc0013b8bb8, {0x7a29638, 0xa6b18d8}, {0x0, 0x0, 0x0})
	vendor/github.com/onsi/gomega/internal/assertion/assertion.go:48 +0x92
k8s.io/kubernetes/test/e2e/framework.ExpectNoErrorWithOffset(0x7a6bf58?, {0x7a08d80?, 0xc0022482b0?}, {0x0?, 0x1?, 0x7a3dbc8?})
... skipping 22 lines ...
Nov 29 23:12:42.438: INFO: At 2022-11-29 22:56:31 +0000 UTC - event for test.csi.azure.com7cbcr: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "test.csi.azure.com" or manually created by system administrator
Nov 29 23:12:42.438: INFO: At 2022-11-29 22:56:31 +0000 UTC - event for test.csi.azure.com7cbcr: {test.csi.azure.com_k8s-agentpool1-92105279-vmss000002_83f29e4a-d993-40ff-88f5-42bec00669a6 } Provisioning: External provisioner is provisioning volume for claim "multivolume-4365/test.csi.azure.com7cbcr"
Nov 29 23:12:42.438: INFO: At 2022-11-29 22:56:33 +0000 UTC - event for test.csi.azure.com7cbcr: {test.csi.azure.com_k8s-agentpool1-92105279-vmss000002_83f29e4a-d993-40ff-88f5-42bec00669a6 } ProvisioningSucceeded: Successfully provisioned volume pvc-474611e1-0421-4ef4-9037-0e3d28c5d897
Nov 29 23:12:42.438: INFO: At 2022-11-29 22:56:36 +0000 UTC - event for test.csi.azure.com9qk8j: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "test.csi.azure.com" or manually created by system administrator
Nov 29 23:12:42.438: INFO: At 2022-11-29 22:56:36 +0000 UTC - event for test.csi.azure.com9qk8j: {test.csi.azure.com_k8s-agentpool1-92105279-vmss000002_83f29e4a-d993-40ff-88f5-42bec00669a6 } Provisioning: External provisioner is provisioning volume for claim "multivolume-4365/test.csi.azure.com9qk8j"
Nov 29 23:12:42.438: INFO: At 2022-11-29 22:56:38 +0000 UTC - event for test.csi.azure.com9qk8j: {test.csi.azure.com_k8s-agentpool1-92105279-vmss000002_83f29e4a-d993-40ff-88f5-42bec00669a6 } ProvisioningSucceeded: Successfully provisioned volume pvc-0e72e110-b981-447c-adc3-c886d695e36e
Nov 29 23:12:42.438: INFO: At 2022-11-29 22:58:40 +0000 UTC - event for pod-d93c7424-44c3-45d1-bec4-966fd5ea5bb9: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc-0e72e110-b981-447c-adc3-c886d695e36e" : Attach timeout for volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ofo3t2sm/providers/Microsoft.Compute/disks/pvc-0e72e110-b981-447c-adc3-c886d695e36e
Nov 29 23:12:42.438: INFO: At 2022-11-29 22:58:40 +0000 UTC - event for pod-d93c7424-44c3-45d1-bec4-966fd5ea5bb9: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc-474611e1-0421-4ef4-9037-0e3d28c5d897" : Attach timeout for volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ofo3t2sm/providers/Microsoft.Compute/disks/pvc-474611e1-0421-4ef4-9037-0e3d28c5d897
Nov 29 23:12:42.439: INFO: At 2022-11-29 22:58:43 +0000 UTC - event for pod-d93c7424-44c3-45d1-bec4-966fd5ea5bb9: {kubelet k8s-agentpool1-92105279-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume2 volume1], unattached volumes=[volume2 kube-api-access-ltwd7 volume1]: timed out waiting for the condition
Nov 29 23:12:42.439: INFO: At 2022-11-29 23:00:54 +0000 UTC - event for pod-d93c7424-44c3-45d1-bec4-966fd5ea5bb9: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-0e72e110-b981-447c-adc3-c886d695e36e" 
Nov 29 23:12:42.439: INFO: At 2022-11-29 23:00:54 +0000 UTC - event for pod-d93c7424-44c3-45d1-bec4-966fd5ea5bb9: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-474611e1-0421-4ef4-9037-0e3d28c5d897" 
Nov 29 23:12:42.439: INFO: At 2022-11-29 23:01:41 +0000 UTC - event for pod-d93c7424-44c3-45d1-bec4-966fd5ea5bb9: {kubelet k8s-agentpool1-92105279-vmss000000} FailedMount: MountVolume.MountDevice failed for volume "pvc-0e72e110-b981-447c-adc3-c886d695e36e" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ofo3t2sm/providers/Microsoft.Compute/disks/pvc-0e72e110-b981-447c-adc3-c886d695e36e) to be attached to node (k8s-agentpool1-92105279-vmss000000): rpc error: code = Unknown desc = context deadline exceeded
Nov 29 23:12:42.439: INFO: At 2022-11-29 23:01:41 +0000 UTC - event for pod-d93c7424-44c3-45d1-bec4-966fd5ea5bb9: {kubelet k8s-agentpool1-92105279-vmss000000} FailedMount: MountVolume.MountDevice failed for volume "pvc-474611e1-0421-4ef4-9037-0e3d28c5d897" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ofo3t2sm/providers/Microsoft.Compute/disks/pvc-474611e1-0421-4ef4-9037-0e3d28c5d897) to be attached to node (k8s-agentpool1-92105279-vmss000000): rpc error: code = Unknown desc = context deadline exceeded
Nov 29 23:12:42.439: INFO: At 2022-11-29 23:03:41 +0000 UTC - event for pod-d93c7424-44c3-45d1-bec4-966fd5ea5bb9: {kubelet k8s-agentpool1-92105279-vmss000000} FailedMount: MountVolume.MountDevice failed for volume "pvc-474611e1-0421-4ef4-9037-0e3d28c5d897" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 29 23:12:42.439: INFO: At 2022-11-29 23:03:41 +0000 UTC - event for pod-d93c7424-44c3-45d1-bec4-966fd5ea5bb9: {kubelet k8s-agentpool1-92105279-vmss000000} FailedMount: MountVolume.MountDevice failed for volume "pvc-0e72e110-b981-447c-adc3-c886d695e36e" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 29 23:12:42.547: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov 29 23:12:42.547: INFO: 
Nov 29 23:12:42.762: INFO: 
Logging node info for node k8s-agentpool1-92105279-vmss000000
Nov 29 23:12:42.870: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-92105279-vmss000000    e7716b3a-d547-407e-a9ed-2706618be43c 29065 0 2022-11-29 22:04:46 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D8s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westeurope failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-ofo3t2sm kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-92105279-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_D8s_v3 storageprofile:managed storagetier:Premium_LRS topology.kubernetes.io/region:westeurope topology.kubernetes.io/zone:0 topology.test.csi.azure.com/zone:] map[csi.volume.kubernetes.io/nodeid:{"test.csi.azure.com":"k8s-agentpool1-92105279-vmss000000"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2022-11-29 22:04:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{}}}} } {kubectl-label Update v1 2022-11-29 22:04:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {cloud-controller-manager Update v1 2022-11-29 22:05:02 +0000 UTC FieldsV1 {"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.3.0/24\"":{}}}} } {kube-controller-manager Update v1 2022-11-29 22:05:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {cloud-controller-manager Update v1 2022-11-29 22:05:49 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {cloud-node-manager Update v1 2022-11-29 22:06:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-29 23:12:32 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-29 23:12:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.test.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ofo3t2sm/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-92105279-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{31025332224 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{33672720384 0} {<nil>} 32883516Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{27922798956 0} {<nil>} 27922798956 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{32886288384 0} {<nil>} 32115516Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-29 22:05:49 +0000 UTC,LastTransitionTime:2022-11-29 22:05:49 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-29 23:12:35 +0000 UTC,LastTransitionTime:2022-11-29 22:04:40 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-29 23:12:35 +0000 UTC,LastTransitionTime:2022-11-29 22:04:40 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-29 23:12:35 +0000 UTC,LastTransitionTime:2022-11-29 22:04:40 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-29 23:12:35 +0000 UTC,LastTransitionTime:2022-11-29 22:05:02 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool1-92105279-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2d155693e7fa4281a08fb77e113c1f09,SystemUUID:badc595a-a00b-e64f-ad71-6b1c80de3c78,BootID:ae2f9bff-8abd-4f93-a225-c6b443284255,KernelVersion:5.4.0-1094-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.14+azure-1,KubeletVersion:v1.23.14,KubeProxyVersion:v1.23.14,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253346057,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:664ec9627d47a4160789ad6994367128b25007b164773c5f1e598028440a158a k8sprow.azurecr.io/azuredisk-csi:latest-v2-5fb3153a08fb6a2968d7e7bbe760d033ea35fc7a],SizeBytes:142859492,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:f5241226198f5a54d22540acf2b3933ea0f49458f90c51fc75833d0c428687b8 k8s.gcr.io/e2e-test-images/agnhost:2.36],SizeBytes:125426639,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:efa8e4b7776fa9fb2279745008780cefec632ee5fdb081e0f6e234ea4b8bbf61 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.14],SizeBytes:112312488,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:949db4eea46ea31f9afa0b49ad0b55f7a31f91aa03b408e0ac9b7720c8ef7d29 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.11],SizeBytes:51376235,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:72fbc9d17947d1a54c09a16671e6810fd7c0d8f09c0502fdbdd212c2192451a7 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.14],SizeBytes:51044459,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:075ea1f8270312350f1396ab6677251e803e61a523822d5abfa5e6acd180cfab mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.11],SizeBytes:50806891,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:eb7d329559d954093f7a234bc0ff322764de48a0b2a1007229032fb53e7bb9eb mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.24.0],SizeBytes:49070187,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:2dfff4dc10f6d25b0a34c01bf4d166540cbf1ff2bccd52635350039138eb23c4 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.18],SizeBytes:48775275,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3e23b9ab6bfee42a1751019c21a60aa474964bd4cba00cbed5055430424242cc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.24.0],SizeBytes:48500843,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:acec9ee6626c2cdea89107ba191ae36e6f60339c9b3b3215b5a59bf0e2e730ba mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.14],SizeBytes:47853675,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:759ac7020e982884b99121583bd577658b5663b1c1ed69746356aa6ce9879da4 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.18],SizeBytes:45850731,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:2fbd1e1a0538a06f2061afd45975df70c942654aa7f86e920720169ee439c2d6 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.5.1],SizeBytes:19743433,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:31547791294872570393470991481c2477a311031d3a03e0ae54eb164347dc34 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0],SizeBytes:17857485,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ofo3t2sm/providers/Microsoft.Compute/disks/pvc-2a67c84d-6aa7-4d49-a2ca-90b39f0e1b1f],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-ofo3t2sm/providers/Microsoft.Compute/disks/pvc-2a67c84d-6aa7-4d49-a2ca-90b39f0e1b1f,DevicePath:,},},Config:nil,},}
Nov 29 23:12:42.871: INFO: 
... skipping 110 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on different node [Measurement]
    test/e2e/storage/testsuites/multivolume.go:248

    Nov 29 23:01:41.261: Unexpected error:
        <*errors.errorString | 0xc0022482b0>: {
            s: "pod \"pod-d93c7424-44c3-45d1-bec4-966fd5ea5bb9\" is not Running: timed out waiting for the condition",
        }
        pod "pod-d93c7424-44c3-45d1-bec4-966fd5ea5bb9" is not Running: timed out waiting for the condition
    occurred

    test/e2e/storage/testsuites/multivolume.go:497
------------------------------
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":32,"completed":12,"skipped":775,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand
  test/e2e/storage/framework/testsuite.go:51
Nov 29 23:12:46.016: INFO: Driver "test.csi.azure.com" does not support volume expansion - skipping
... skipping 34 lines ...

    test/e2e/storage/external/external.go:262
------------------------------
SSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:269

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 23:12:46.032: INFO: >>> kubeConfig: /root/tmp1562512509/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename provisioning
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:269
Nov 29 23:12:46.791: INFO: Creating resource for dynamic PV
Nov 29 23:12:46.791: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-7276-e2e-scmw5jp
STEP: creating a claim
Nov 29 23:12:46.903: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 29 23:12:47.017: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com7vv29] to have phase Bound
Nov 29 23:12:47.126: INFO: PersistentVolumeClaim test.csi.azure.com7vv29 found but phase is Pending instead of Bound.
Nov 29 23:12:49.235: INFO: PersistentVolumeClaim test.csi.azure.com7vv29 found but phase is Pending instead of Bound.
Nov 29 23:12:51.344: INFO: PersistentVolumeClaim test.csi.azure.com7vv29 found and phase=Bound (4.326569598s)
STEP: Creating pod pod-subpath-test-dynamicpv-jj5j
STEP: Checking for subpath error in container status
Nov 29 23:14:21.894: INFO: Deleting pod "pod-subpath-test-dynamicpv-jj5j" in namespace "provisioning-7276"
Nov 29 23:14:22.007: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-jj5j" to be fully deleted
STEP: Deleting pod
Nov 29 23:14:24.229: INFO: Deleting pod "pod-subpath-test-dynamicpv-jj5j" in namespace "provisioning-7276"
STEP: Deleting pvc
Nov 29 23:14:24.350: INFO: Deleting PersistentVolumeClaim "test.csi.azure.com7vv29"
... skipping 22 lines ...

• [SLOW TEST:170.653 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:269
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]","total":32,"completed":13,"skipped":872,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

SSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 23:15:36.693: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 112 lines ...
Nov 29 23:12:26.528: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comf7x8h] to have phase Bound
Nov 29 23:12:26.636: INFO: PersistentVolumeClaim test.csi.azure.comf7x8h found but phase is Pending instead of Bound.
Nov 29 23:12:28.746: INFO: PersistentVolumeClaim test.csi.azure.comf7x8h found but phase is Pending instead of Bound.
Nov 29 23:12:30.856: INFO: PersistentVolumeClaim test.csi.azure.comf7x8h found and phase=Bound (4.328283051s)
STEP: [init] starting a pod to use the claim
STEP: [init] check pod success
Nov 29 23:12:31.295: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-lpft6" in namespace "snapshotting-407" to be "Succeeded or Failed"
Nov 29 23:12:31.403: INFO: Pod "pvc-snapshottable-tester-lpft6": Phase="Pending", Reason="", readiness=false. Elapsed: 108.832737ms
Nov 29 23:12:33.514: INFO: Pod "pvc-snapshottable-tester-lpft6": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219054366s
Nov 29 23:12:35.623: INFO: Pod "pvc-snapshottable-tester-lpft6": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328750492s
Nov 29 23:12:37.734: INFO: Pod "pvc-snapshottable-tester-lpft6": Phase="Pending", Reason="", readiness=false. Elapsed: 6.439525481s
Nov 29 23:12:39.844: INFO: Pod "pvc-snapshottable-tester-lpft6": Phase="Pending", Reason="", readiness=false. Elapsed: 8.548963606s
Nov 29 23:12:41.955: INFO: Pod "pvc-snapshottable-tester-lpft6": Phase="Pending", Reason="", readiness=false. Elapsed: 10.660830475s
Nov 29 23:12:44.066: INFO: Pod "pvc-snapshottable-tester-lpft6": Phase="Pending", Reason="", readiness=false. Elapsed: 12.771101359s
Nov 29 23:12:46.176: INFO: Pod "pvc-snapshottable-tester-lpft6": Phase="Pending", Reason="", readiness=false. Elapsed: 14.881171214s
Nov 29 23:12:48.285: INFO: Pod "pvc-snapshottable-tester-lpft6": Phase="Pending", Reason="", readiness=false. Elapsed: 16.990656853s
Nov 29 23:12:50.404: INFO: Pod "pvc-snapshottable-tester-lpft6": Phase="Succeeded", Reason="", readiness=false. Elapsed: 19.109312711s
STEP: Saw pod success
Nov 29 23:12:50.404: INFO: Pod "pvc-snapshottable-tester-lpft6" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
Nov 29 23:12:50.515: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comf7x8h] to have phase Bound
Nov 29 23:12:50.638: INFO: PersistentVolumeClaim test.csi.azure.comf7x8h found and phase=Bound (122.222749ms)
STEP: [init] checking the PV
STEP: [init] deleting the pod
Nov 29 23:12:50.982: INFO: Pod pvc-snapshottable-tester-lpft6 has the following logs: 
... skipping 15 lines ...
Nov 29 23:13:02.847: INFO: received snapshotStatus map[boundVolumeSnapshotContentName:snapcontent-cefcf59e-5f28-4790-816e-7437757ec5ee creationTime:2022-11-29T23:12:58Z readyToUse:true restoreSize:5Gi]
Nov 29 23:13:02.848: INFO: snapshotContentName snapcontent-cefcf59e-5f28-4790-816e-7437757ec5ee
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Nov 29 23:13:03.288: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-68f2w" in namespace "snapshotting-407" to be "Succeeded or Failed"
Nov 29 23:13:03.397: INFO: Pod "pvc-snapshottable-data-tester-68f2w": Phase="Pending", Reason="", readiness=false. Elapsed: 108.85461ms
Nov 29 23:13:05.507: INFO: Pod "pvc-snapshottable-data-tester-68f2w": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218993374s
Nov 29 23:13:07.616: INFO: Pod "pvc-snapshottable-data-tester-68f2w": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328305457s
Nov 29 23:13:09.729: INFO: Pod "pvc-snapshottable-data-tester-68f2w": Phase="Pending", Reason="", readiness=false. Elapsed: 6.440992284s
Nov 29 23:13:11.839: INFO: Pod "pvc-snapshottable-data-tester-68f2w": Phase="Pending", Reason="", readiness=false. Elapsed: 8.550984124s
Nov 29 23:13:13.948: INFO: Pod "pvc-snapshottable-data-tester-68f2w": Phase="Pending", Reason="", readiness=false. Elapsed: 10.66050821s
... skipping 16 lines ...
Nov 29 23:13:49.831: INFO: Pod "pvc-snapshottable-data-tester-68f2w": Phase="Pending", Reason="", readiness=false. Elapsed: 46.543131962s
Nov 29 23:13:51.941: INFO: Pod "pvc-snapshottable-data-tester-68f2w": Phase="Pending", Reason="", readiness=false. Elapsed: 48.652656782s
Nov 29 23:13:54.051: INFO: Pod "pvc-snapshottable-data-tester-68f2w": Phase="Pending", Reason="", readiness=false. Elapsed: 50.762897379s
Nov 29 23:13:56.162: INFO: Pod "pvc-snapshottable-data-tester-68f2w": Phase="Pending", Reason="", readiness=false. Elapsed: 52.874247835s
Nov 29 23:13:58.284: INFO: Pod "pvc-snapshottable-data-tester-68f2w": Phase="Succeeded", Reason="", readiness=false. Elapsed: 54.996196582s
STEP: Saw pod success
Nov 29 23:13:58.284: INFO: Pod "pvc-snapshottable-data-tester-68f2w" satisfied condition "Succeeded or Failed"
Nov 29 23:13:58.510: INFO: Pod pvc-snapshottable-data-tester-68f2w has the following logs: 
Nov 29 23:13:58.510: INFO: Deleting pod "pvc-snapshottable-data-tester-68f2w" in namespace "snapshotting-407"
Nov 29 23:13:58.625: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-68f2w" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the snapshot
Nov 29 23:15:03.182: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-ofo3t2sm.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1562512509/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-407 exec restored-pvc-tester-w24g2 --namespace=snapshotting-407 -- cat /mnt/test/data'
... skipping 47 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        test/e2e/storage/testsuites/snapshottable.go:278
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":34,"completed":14,"skipped":1005,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]"]}

SSSSSSSSSSSSS
------------------------------
Nov 29 23:15:49.419: INFO: Running AfterSuite actions on all nodes
Nov 29 23:15:49.419: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2
Nov 29 23:15:49.419: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2
... skipping 338 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with pvc data source in parallel [Slow]
    test/e2e/storage/testsuites/provisioning.go:459
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with pvc data source in parallel [Slow]","total":39,"completed":14,"skipped":737,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

SSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 23:15:53.353: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 59 lines ...
Nov 29 23:15:38.778: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com9bdrm] to have phase Bound
Nov 29 23:15:38.887: INFO: PersistentVolumeClaim test.csi.azure.com9bdrm found but phase is Pending instead of Bound.
Nov 29 23:15:40.996: INFO: PersistentVolumeClaim test.csi.azure.com9bdrm found but phase is Pending instead of Bound.
Nov 29 23:15:43.107: INFO: PersistentVolumeClaim test.csi.azure.com9bdrm found and phase=Bound (4.329065198s)
STEP: Creating pod pod-subpath-test-dynamicpv-chbr
STEP: Creating a pod to test multi_subpath
Nov 29 23:15:43.434: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-chbr" in namespace "provisioning-7066" to be "Succeeded or Failed"
Nov 29 23:15:43.547: INFO: Pod "pod-subpath-test-dynamicpv-chbr": Phase="Pending", Reason="", readiness=false. Elapsed: 112.156371ms
Nov 29 23:15:45.656: INFO: Pod "pod-subpath-test-dynamicpv-chbr": Phase="Pending", Reason="", readiness=false. Elapsed: 2.221255746s
Nov 29 23:15:47.779: INFO: Pod "pod-subpath-test-dynamicpv-chbr": Phase="Pending", Reason="", readiness=false. Elapsed: 4.34473311s
Nov 29 23:15:49.889: INFO: Pod "pod-subpath-test-dynamicpv-chbr": Phase="Pending", Reason="", readiness=false. Elapsed: 6.454978811s
Nov 29 23:15:51.999: INFO: Pod "pod-subpath-test-dynamicpv-chbr": Phase="Pending", Reason="", readiness=false. Elapsed: 8.564809874s
Nov 29 23:15:54.109: INFO: Pod "pod-subpath-test-dynamicpv-chbr": Phase="Pending", Reason="", readiness=false. Elapsed: 10.674970453s
... skipping 4 lines ...
Nov 29 23:16:04.658: INFO: Pod "pod-subpath-test-dynamicpv-chbr": Phase="Pending", Reason="", readiness=false. Elapsed: 21.224042915s
Nov 29 23:16:06.769: INFO: Pod "pod-subpath-test-dynamicpv-chbr": Phase="Pending", Reason="", readiness=false. Elapsed: 23.335084115s
Nov 29 23:16:08.882: INFO: Pod "pod-subpath-test-dynamicpv-chbr": Phase="Pending", Reason="", readiness=false. Elapsed: 25.447116288s
Nov 29 23:16:10.993: INFO: Pod "pod-subpath-test-dynamicpv-chbr": Phase="Pending", Reason="", readiness=false. Elapsed: 27.559102644s
Nov 29 23:16:13.103: INFO: Pod "pod-subpath-test-dynamicpv-chbr": Phase="Succeeded", Reason="", readiness=false. Elapsed: 29.668726713s
STEP: Saw pod success
Nov 29 23:16:13.103: INFO: Pod "pod-subpath-test-dynamicpv-chbr" satisfied condition "Succeeded or Failed"
Nov 29 23:16:13.212: INFO: Trying to get logs from node k8s-agentpool1-92105279-vmss000001 pod pod-subpath-test-dynamicpv-chbr container test-container-subpath-dynamicpv-chbr: <nil>
STEP: delete the pod
Nov 29 23:16:13.478: INFO: Waiting for pod pod-subpath-test-dynamicpv-chbr to disappear
Nov 29 23:16:13.587: INFO: Pod pod-subpath-test-dynamicpv-chbr no longer exists
STEP: Deleting pod
Nov 29 23:16:13.587: INFO: Deleting pod "pod-subpath-test-dynamicpv-chbr" in namespace "provisioning-7066"
... skipping 21 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support creating multiple subpath from same volumes [Slow]
    test/e2e/storage/testsuites/subpath.go:296
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support creating multiple subpath from same volumes [Slow]","total":32,"completed":14,"skipped":997,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

SSSSSSSSSSSSSSSSSSSS
------------------------------
Nov 29 23:16:55.363: INFO: Running AfterSuite actions on all nodes
Nov 29 23:16:55.363: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2
Nov 29 23:16:55.363: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2
... skipping 55 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should create read-only inline ephemeral volume
    test/e2e/storage/testsuites/ephemeral.go:175
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read-only inline ephemeral volume","total":39,"completed":15,"skipped":765,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 23:17:25.339: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 114 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should be able to unmount after the subpath directory is deleted [LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:447
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [LinuxOnly]","total":39,"completed":16,"skipped":825,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volume-expand
  test/e2e/storage/framework/testsuite.go:51
Nov 29 23:18:38.756: INFO: Driver "test.csi.azure.com" does not support volume expansion - skipping
... skipping 47 lines ...
Nov 29 23:18:38.837: INFO: Running AfterSuite actions on node 1



Summarizing 3 Failures:

[Fail] External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] [Measurement] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] 
test/e2e/storage/testsuites/multivolume.go:696

[Fail] External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [Measurement] should access to two volumes with different volume mode and retain data across pod recreation on different node 
test/e2e/storage/testsuites/multivolume.go:497

[Fail] External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [Measurement] should access to two volumes with different volume mode and retain data across pod recreation on different node 
test/e2e/storage/testsuites/multivolume.go:497

Ran 89 of 7227 Specs in 3748.991 seconds
FAIL! -- 86 Passed | 3 Failed | 0 Pending | 7138 Skipped 

Ginkgo ran 1 suite in 1h2m32.912682661s
Test Suite Failed
+ print_logs
+ sed -i s/disk.csi.azure.com/test.csi.azure.com/g deploy/example/storageclass-azuredisk-csi.yaml
+ bash ./hack/verify-examples.sh linux azurepubliccloud ephemeral test
begin to create deployment examples ...
storageclass.storage.k8s.io/managed-csi created
Applying config "deploy/example/deployment.yaml"
... skipping 20 lines ...