pull-azuredisk-csi-driver-external-e2e-single-az-mainv2 #1593038299914571776 Results

Error lines from build-log.txt

Raw build-log.txt
... skipping 236 lines ...

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 11156  100 11156    0     0   162k      0 --:--:-- --:--:-- --:--:--  162k
Downloading https://get.helm.sh/helm-v3.10.2-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
docker pull k8sprow.azurecr.io/azuredisk-csi:latest-v2-c551e0323ad732edee8d00639b70d4fc25a9b407 || make container-all push-manifest
Error response from daemon: manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-c551e0323ad732edee8d00639b70d4fc25a9b407 not found: manifest unknown: manifest tagged by "latest-v2-c551e0323ad732edee8d00639b70d4fc25a9b407" is not found
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=latest-v2-c551e0323ad732edee8d00639b70d4fc25a9b407 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=c551e0323ad732edee8d00639b70d4fc25a9b407 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2022-11-17T00:35:02Z -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/amd64/azurediskpluginv2.exe ./pkg/azurediskplugin
docker buildx rm container-builder || true
ERROR: no builder "container-builder" found
docker buildx create --use --name=container-builder
container-builder
# enable qemu for arm64 build
# https://github.com/docker/buildx/issues/464#issuecomment-741507760
docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64
Unable to find image 'tonistiigi/binfmt:latest' locally
... skipping 677 lines ...
         }
      }
   ]
}
make[2]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker pull k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-c551e0323ad732edee8d00639b70d4fc25a9b407 || make azdiskschedulerextender-all push-manifest-azdiskschedulerextender
Error response from daemon: manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-c551e0323ad732edee8d00639b70d4fc25a9b407 not found: manifest unknown: manifest tagged by "latest-v2-c551e0323ad732edee8d00639b70d4fc25a9b407" is not found
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker buildx rm container-builder || true
container-builder removed
docker buildx create --use --name=container-builder
container-builder
# enable qemu for arm64 build
... skipping 1039 lines ...
                    type: string
                type: object
                oneOf:
                - required: ["persistentVolumeClaimName"]
                - required: ["volumeSnapshotContentName"]
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 60 lines ...
                    type: string
                  volumeSnapshotContentName:
                    description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable.
                    type: string
                type: object
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 254 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 108 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 359 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status Required
            properties:
              detail:
                description: Status summarizes the current attachment state of the
                  volume attachment Nil Status indicates that the volume has not yet
                  been attached to the node
                properties:
... skipping 7 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 90 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 13 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 169 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              detail:
                description: Current status detail of the AzVolume Nil detail indicates
                  that the volume has not been created
                properties:
                  accessible_topology:
... skipping 28 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 154 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 34 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 1063 lines ...
          image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0"
          args:
            - "-csi-address=$(ADDRESS)"
            - "-v=2"
            - "-leader-election"
            - "--leader-election-namespace=kube-system"
            - '-handle-volume-inuse-error=false'
            - '-feature-gates=RecoverVolumeExpansionFailure=true'
            - "-timeout=240s"
          env:
            - name: ADDRESS
              value: /csi/csi.sock
          volumeMounts:
... skipping 785 lines ...
  test/e2e/framework/framework.go:188
Nov 17 00:45:40.693: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[1mSTEP[0m: Destroying namespace "volumelimits-358" for this suite.

[32m•[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeLimits should verify that all csinodes have volume limits","total":43,"completed":1,"skipped":38,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 00:45:40.954: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 3 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:269[0m

    [36mDistro debian doesn't support ntfs -- skipping[0m

    test/e2e/storage/framework/testsuite.go:127
[90m------------------------------[0m
... skipping 42 lines ...
  test/e2e/framework/framework.go:188
Nov 17 00:45:40.804: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[1mSTEP[0m: Destroying namespace "volumelimits-8098" for this suite.

[32m•[0m[36mS[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs)] volumeLimits should verify that all csinodes have volume limits","total":34,"completed":1,"skipped":115,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 00:45:41.053: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 41 lines ...
Nov 17 00:45:39.238: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comqxj5l] to have phase Bound
Nov 17 00:45:39.346: INFO: PersistentVolumeClaim test.csi.azure.comqxj5l found but phase is Pending instead of Bound.
Nov 17 00:45:41.455: INFO: PersistentVolumeClaim test.csi.azure.comqxj5l found but phase is Pending instead of Bound.
Nov 17 00:45:43.564: INFO: PersistentVolumeClaim test.csi.azure.comqxj5l found and phase=Bound (4.32583816s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-nlx6
[1mSTEP[0m: Creating a pod to test subpath
Nov 17 00:45:43.889: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-nlx6" in namespace "provisioning-681" to be "Succeeded or Failed"
Nov 17 00:45:44.001: INFO: Pod "pod-subpath-test-dynamicpv-nlx6": Phase="Pending", Reason="", readiness=false. Elapsed: 110.97544ms
Nov 17 00:45:46.111: INFO: Pod "pod-subpath-test-dynamicpv-nlx6": Phase="Pending", Reason="", readiness=false. Elapsed: 2.221603144s
Nov 17 00:45:48.220: INFO: Pod "pod-subpath-test-dynamicpv-nlx6": Phase="Pending", Reason="", readiness=false. Elapsed: 4.330744293s
Nov 17 00:45:50.330: INFO: Pod "pod-subpath-test-dynamicpv-nlx6": Phase="Pending", Reason="", readiness=false. Elapsed: 6.440691338s
Nov 17 00:45:52.439: INFO: Pod "pod-subpath-test-dynamicpv-nlx6": Phase="Pending", Reason="", readiness=false. Elapsed: 8.549902487s
Nov 17 00:45:54.549: INFO: Pod "pod-subpath-test-dynamicpv-nlx6": Phase="Pending", Reason="", readiness=false. Elapsed: 10.659625533s
... skipping 14 lines ...
Nov 17 00:46:26.208: INFO: Pod "pod-subpath-test-dynamicpv-nlx6": Phase="Pending", Reason="", readiness=false. Elapsed: 42.31822242s
Nov 17 00:46:28.317: INFO: Pod "pod-subpath-test-dynamicpv-nlx6": Phase="Pending", Reason="", readiness=false. Elapsed: 44.427328523s
Nov 17 00:46:30.431: INFO: Pod "pod-subpath-test-dynamicpv-nlx6": Phase="Pending", Reason="", readiness=false. Elapsed: 46.541611459s
Nov 17 00:46:32.541: INFO: Pod "pod-subpath-test-dynamicpv-nlx6": Phase="Pending", Reason="", readiness=false. Elapsed: 48.651312241s
Nov 17 00:46:34.653: INFO: Pod "pod-subpath-test-dynamicpv-nlx6": Phase="Succeeded", Reason="", readiness=false. Elapsed: 50.763447961s
[1mSTEP[0m: Saw pod success
Nov 17 00:46:34.655: INFO: Pod "pod-subpath-test-dynamicpv-nlx6" satisfied condition "Succeeded or Failed"
Nov 17 00:46:34.763: INFO: Trying to get logs from node k8s-agentpool1-18975173-vmss000000 pod pod-subpath-test-dynamicpv-nlx6 container test-container-volume-dynamicpv-nlx6: <nil>
[1mSTEP[0m: delete the pod
Nov 17 00:46:35.013: INFO: Waiting for pod pod-subpath-test-dynamicpv-nlx6 to disappear
Nov 17 00:46:35.121: INFO: Pod pod-subpath-test-dynamicpv-nlx6 no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-nlx6
Nov 17 00:46:35.121: INFO: Deleting pod "pod-subpath-test-dynamicpv-nlx6" in namespace "provisioning-681"
... skipping 28 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support existing directory
    [90mtest/e2e/storage/testsuites/subpath.go:207[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory","total":27,"completed":1,"skipped":9,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (xfs)][Slow] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 17 00:47:42.612: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 138 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should store data
    [90mtest/e2e/storage/testsuites/volumes.go:161[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should store data","total":27,"completed":1,"skipped":77,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext4)] multiVolume [Slow][0m 
  [1mshould concurrently access the single volume from pods on the same node[0m
  [37mtest/e2e/storage/testsuites/multivolume.go:298[0m
... skipping 157 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:298[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":36,"completed":1,"skipped":101,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext4)] multiVolume [Slow][0m 
  [1mshould concurrently access the single read-only volume from pods on the same node[0m
  [37mtest/e2e/storage/testsuites/multivolume.go:423[0m
... skipping 85 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single read-only volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:423[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":26,"completed":1,"skipped":36,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 17 00:49:04.897: INFO: >>> kubeConfig: /root/tmp1926242230/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename topology
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  test/e2e/storage/testsuites/topology.go:194
Nov 17 00:49:05.659: INFO: Driver didn't provide topology keys -- skipping
[AfterEach] [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/framework/framework.go:188
Nov 17 00:49:05.660: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[1mSTEP[0m: Destroying namespace "topology-6729" for this suite.


[36m[1mS [SKIPPING] [0.985 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (immediate binding)] topology
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement][0m
    [90mtest/e2e/storage/testsuites/topology.go:194[0m

    [36mDriver didn't provide topology keys -- skipping[0m

    test/e2e/storage/testsuites/topology.go:126
[90m------------------------------[0m
... skipping 139 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents
    [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents","total":27,"completed":2,"skipped":80,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 00:49:08.459: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 31 lines ...
[It] should check snapshot fields, check restore correctly works, check deletion (ephemeral)
  test/e2e/storage/testsuites/snapshottable.go:177
Nov 17 00:48:07.889: INFO: Creating resource for dynamic PV
Nov 17 00:48:07.889: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass snapshotting-3943-e2e-scv658f
[1mSTEP[0m: [init] starting a pod to use the claim
Nov 17 00:48:08.112: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-szsmt" in namespace "snapshotting-3943" to be "Succeeded or Failed"
Nov 17 00:48:08.221: INFO: Pod "pvc-snapshottable-tester-szsmt": Phase="Pending", Reason="", readiness=false. Elapsed: 109.229184ms
Nov 17 00:48:10.332: INFO: Pod "pvc-snapshottable-tester-szsmt": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219690007s
Nov 17 00:48:12.443: INFO: Pod "pvc-snapshottable-tester-szsmt": Phase="Pending", Reason="", readiness=false. Elapsed: 4.33063888s
Nov 17 00:48:14.553: INFO: Pod "pvc-snapshottable-tester-szsmt": Phase="Pending", Reason="", readiness=false. Elapsed: 6.441188733s
Nov 17 00:48:16.665: INFO: Pod "pvc-snapshottable-tester-szsmt": Phase="Pending", Reason="", readiness=false. Elapsed: 8.552783038s
Nov 17 00:48:18.776: INFO: Pod "pvc-snapshottable-tester-szsmt": Phase="Pending", Reason="", readiness=false. Elapsed: 10.663612703s
Nov 17 00:48:20.887: INFO: Pod "pvc-snapshottable-tester-szsmt": Phase="Pending", Reason="", readiness=false. Elapsed: 12.77463998s
Nov 17 00:48:22.998: INFO: Pod "pvc-snapshottable-tester-szsmt": Phase="Pending", Reason="", readiness=false. Elapsed: 14.885432652s
Nov 17 00:48:25.108: INFO: Pod "pvc-snapshottable-tester-szsmt": Phase="Pending", Reason="", readiness=false. Elapsed: 16.995966745s
Nov 17 00:48:27.220: INFO: Pod "pvc-snapshottable-tester-szsmt": Phase="Pending", Reason="", readiness=false. Elapsed: 19.107947556s
Nov 17 00:48:29.331: INFO: Pod "pvc-snapshottable-tester-szsmt": Phase="Succeeded", Reason="", readiness=false. Elapsed: 21.218644436s
[1mSTEP[0m: Saw pod success
Nov 17 00:48:29.331: INFO: Pod "pvc-snapshottable-tester-szsmt" satisfied condition "Succeeded or Failed"
[1mSTEP[0m: [init] checking the claim
[1mSTEP[0m: creating a SnapshotClass
[1mSTEP[0m: creating a dynamic VolumeSnapshot
Nov 17 00:48:29.775: INFO: Waiting up to 5m0s for VolumeSnapshot snapshot-w6v7m to become ready
Nov 17 00:48:29.885: INFO: VolumeSnapshot snapshot-w6v7m found but is not ready.
Nov 17 00:48:31.996: INFO: VolumeSnapshot snapshot-w6v7m found but is not ready.
... skipping 49 lines ...
    [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m
      
      [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m
        should check snapshot fields, check restore correctly works, check deletion (ephemeral)
        [90mtest/e2e/storage/testsuites/snapshottable.go:177[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works, check deletion (ephemeral)","total":27,"completed":2,"skipped":121,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow][0m 
  [1mshould access to two volumes with the same volume mode and retain data across pod recreation on different node[0m
  [37mtest/e2e/storage/testsuites/multivolume.go:168[0m
... skipping 196 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    [90mtest/e2e/storage/testsuites/multivolume.go:168[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":34,"completed":2,"skipped":193,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand
  test/e2e/storage/framework/testsuite.go:51
Nov 17 00:50:06.286: INFO: Driver "test.csi.azure.com" does not support volume expansion - skipping
... skipping 24 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:242[0m

    [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 22 lines ...
Nov 17 00:45:41.639: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comd6f66] to have phase Bound
Nov 17 00:45:41.748: INFO: PersistentVolumeClaim test.csi.azure.comd6f66 found but phase is Pending instead of Bound.
Nov 17 00:45:43.857: INFO: PersistentVolumeClaim test.csi.azure.comd6f66 found but phase is Pending instead of Bound.
Nov 17 00:45:45.967: INFO: PersistentVolumeClaim test.csi.azure.comd6f66 found and phase=Bound (4.328054615s)
[1mSTEP[0m: [init] starting a pod to use the claim
[1mSTEP[0m: [init] check pod success
Nov 17 00:45:46.408: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-c8kjm" in namespace "snapshotting-2139" to be "Succeeded or Failed"
Nov 17 00:45:46.516: INFO: Pod "pvc-snapshottable-tester-c8kjm": Phase="Pending", Reason="", readiness=false. Elapsed: 108.481761ms
Nov 17 00:45:48.627: INFO: Pod "pvc-snapshottable-tester-c8kjm": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218977029s
Nov 17 00:45:50.737: INFO: Pod "pvc-snapshottable-tester-c8kjm": Phase="Pending", Reason="", readiness=false. Elapsed: 4.329268527s
Nov 17 00:45:52.847: INFO: Pod "pvc-snapshottable-tester-c8kjm": Phase="Pending", Reason="", readiness=false. Elapsed: 6.4391664s
Nov 17 00:45:54.958: INFO: Pod "pvc-snapshottable-tester-c8kjm": Phase="Pending", Reason="", readiness=false. Elapsed: 8.549939689s
Nov 17 00:45:57.068: INFO: Pod "pvc-snapshottable-tester-c8kjm": Phase="Pending", Reason="", readiness=false. Elapsed: 10.66013038s
... skipping 14 lines ...
Nov 17 00:46:28.732: INFO: Pod "pvc-snapshottable-tester-c8kjm": Phase="Pending", Reason="", readiness=false. Elapsed: 42.323892314s
Nov 17 00:46:30.842: INFO: Pod "pvc-snapshottable-tester-c8kjm": Phase="Pending", Reason="", readiness=false. Elapsed: 44.434158514s
Nov 17 00:46:32.952: INFO: Pod "pvc-snapshottable-tester-c8kjm": Phase="Pending", Reason="", readiness=false. Elapsed: 46.544392845s
Nov 17 00:46:35.070: INFO: Pod "pvc-snapshottable-tester-c8kjm": Phase="Pending", Reason="", readiness=false. Elapsed: 48.661891423s
Nov 17 00:46:37.180: INFO: Pod "pvc-snapshottable-tester-c8kjm": Phase="Succeeded", Reason="", readiness=false. Elapsed: 50.772369183s
[1mSTEP[0m: Saw pod success
Nov 17 00:46:37.180: INFO: Pod "pvc-snapshottable-tester-c8kjm" satisfied condition "Succeeded or Failed"
[1mSTEP[0m: [init] checking the claim
Nov 17 00:46:37.289: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comd6f66] to have phase Bound
Nov 17 00:46:37.398: INFO: PersistentVolumeClaim test.csi.azure.comd6f66 found and phase=Bound (109.411408ms)
[1mSTEP[0m: [init] checking the PV
[1mSTEP[0m: [init] deleting the pod
Nov 17 00:46:37.752: INFO: Pod pvc-snapshottable-tester-c8kjm has the following logs: 
... skipping 13 lines ...
Nov 17 00:46:45.181: INFO: received snapshotStatus map[boundVolumeSnapshotContentName:snapcontent-d21f77d3-f79e-4a05-9fff-01ff32eefba1 creationTime:2022-11-17T00:46:40Z readyToUse:true restoreSize:5Gi]
Nov 17 00:46:45.181: INFO: snapshotContentName snapcontent-d21f77d3-f79e-4a05-9fff-01ff32eefba1
[1mSTEP[0m: checking the snapshot
[1mSTEP[0m: checking the SnapshotContent
[1mSTEP[0m: Modifying source data test
[1mSTEP[0m: modifying the data in the source PVC
Nov 17 00:46:45.630: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-zrn9t" in namespace "snapshotting-2139" to be "Succeeded or Failed"
Nov 17 00:46:45.748: INFO: Pod "pvc-snapshottable-data-tester-zrn9t": Phase="Pending", Reason="", readiness=false. Elapsed: 117.53238ms
Nov 17 00:46:47.861: INFO: Pod "pvc-snapshottable-data-tester-zrn9t": Phase="Pending", Reason="", readiness=false. Elapsed: 2.230474355s
Nov 17 00:46:49.970: INFO: Pod "pvc-snapshottable-data-tester-zrn9t": Phase="Pending", Reason="", readiness=false. Elapsed: 4.340155703s
Nov 17 00:46:52.082: INFO: Pod "pvc-snapshottable-data-tester-zrn9t": Phase="Pending", Reason="", readiness=false. Elapsed: 6.451240304s
Nov 17 00:46:54.191: INFO: Pod "pvc-snapshottable-data-tester-zrn9t": Phase="Pending", Reason="", readiness=false. Elapsed: 8.560952154s
Nov 17 00:46:56.301: INFO: Pod "pvc-snapshottable-data-tester-zrn9t": Phase="Pending", Reason="", readiness=false. Elapsed: 10.671135716s
... skipping 40 lines ...
Nov 17 00:48:22.843: INFO: Pod "pvc-snapshottable-data-tester-zrn9t": Phase="Pending", Reason="", readiness=false. Elapsed: 1m37.212671715s
Nov 17 00:48:24.954: INFO: Pod "pvc-snapshottable-data-tester-zrn9t": Phase="Pending", Reason="", readiness=false. Elapsed: 1m39.323301764s
Nov 17 00:48:27.064: INFO: Pod "pvc-snapshottable-data-tester-zrn9t": Phase="Pending", Reason="", readiness=false. Elapsed: 1m41.433581032s
Nov 17 00:48:29.173: INFO: Pod "pvc-snapshottable-data-tester-zrn9t": Phase="Pending", Reason="", readiness=false. Elapsed: 1m43.542707132s
Nov 17 00:48:31.283: INFO: Pod "pvc-snapshottable-data-tester-zrn9t": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m45.652319952s
[1mSTEP[0m: Saw pod success
Nov 17 00:48:31.283: INFO: Pod "pvc-snapshottable-data-tester-zrn9t" satisfied condition "Succeeded or Failed"
Nov 17 00:48:31.527: INFO: Pod pvc-snapshottable-data-tester-zrn9t has the following logs: 
Nov 17 00:48:31.527: INFO: Deleting pod "pvc-snapshottable-data-tester-zrn9t" in namespace "snapshotting-2139"
Nov 17 00:48:31.643: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-zrn9t" to be fully deleted
[1mSTEP[0m: creating a pvc from the snapshot
[1mSTEP[0m: starting a pod to use the snapshot
Nov 17 00:49:46.201: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-qb0hkalz.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1926242230/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-2139 exec restored-pvc-tester-cwgst --namespace=snapshotting-2139 -- cat /mnt/test/data'
... skipping 32 lines ...
Nov 17 00:50:11.369: INFO: volumesnapshotcontents snapcontent-d21f77d3-f79e-4a05-9fff-01ff32eefba1 has been found and is not deleted
Nov 17 00:50:12.481: INFO: volumesnapshotcontents snapcontent-d21f77d3-f79e-4a05-9fff-01ff32eefba1 has been found and is not deleted
Nov 17 00:50:13.593: INFO: volumesnapshotcontents snapcontent-d21f77d3-f79e-4a05-9fff-01ff32eefba1 has been found and is not deleted
Nov 17 00:50:14.705: INFO: volumesnapshotcontents snapcontent-d21f77d3-f79e-4a05-9fff-01ff32eefba1 has been found and is not deleted
Nov 17 00:50:15.815: INFO: volumesnapshotcontents snapcontent-d21f77d3-f79e-4a05-9fff-01ff32eefba1 has been found and is not deleted
Nov 17 00:50:16.925: INFO: volumesnapshotcontents snapcontent-d21f77d3-f79e-4a05-9fff-01ff32eefba1 has been found and is not deleted
Nov 17 00:50:17.927: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 17 00:50:18.048: INFO: Pod restored-pvc-tester-cwgst has the following logs: 
Nov 17 00:50:18.048: INFO: Deleting pod "restored-pvc-tester-cwgst" in namespace "snapshotting-2139"
Nov 17 00:50:18.158: INFO: Wait up to 5m0s for pod "restored-pvc-tester-cwgst" to be fully deleted
Nov 17 00:50:20.413: INFO: deleting claim "snapshotting-2139"/"pvc-gccvt"
... skipping 31 lines ...
    [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m
      
      [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":37,"completed":1,"skipped":175,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (ntfs)][Feature:Windows] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 17 00:50:38.723: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 167 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support multiple inline ephemeral volumes
    [90mtest/e2e/storage/testsuites/ephemeral.go:254[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support multiple inline ephemeral volumes","total":36,"completed":2,"skipped":122,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 00:50:40.988: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 3 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:242[0m

    [36mDistro debian doesn't support ntfs -- skipping[0m

    test/e2e/storage/framework/testsuite.go:127
[90m------------------------------[0m
... skipping 75 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support two pods which have the same volume definition
    [90mtest/e2e/storage/testsuites/ephemeral.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should support two pods which have the same volume definition","total":43,"completed":2,"skipped":210,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
... skipping 49 lines ...
Nov 17 00:50:07.295: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comjmlx2] to have phase Bound
Nov 17 00:50:07.405: INFO: PersistentVolumeClaim test.csi.azure.comjmlx2 found but phase is Pending instead of Bound.
Nov 17 00:50:09.515: INFO: PersistentVolumeClaim test.csi.azure.comjmlx2 found but phase is Pending instead of Bound.
Nov 17 00:50:11.626: INFO: PersistentVolumeClaim test.csi.azure.comjmlx2 found and phase=Bound (4.330642444s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-pstj
[1mSTEP[0m: Creating a pod to test subpath
Nov 17 00:50:11.957: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-pstj" in namespace "provisioning-1594" to be "Succeeded or Failed"
Nov 17 00:50:12.069: INFO: Pod "pod-subpath-test-dynamicpv-pstj": Phase="Pending", Reason="", readiness=false. Elapsed: 111.80745ms
Nov 17 00:50:14.180: INFO: Pod "pod-subpath-test-dynamicpv-pstj": Phase="Pending", Reason="", readiness=false. Elapsed: 2.223237997s
Nov 17 00:50:16.291: INFO: Pod "pod-subpath-test-dynamicpv-pstj": Phase="Pending", Reason="", readiness=false. Elapsed: 4.334498034s
Nov 17 00:50:18.401: INFO: Pod "pod-subpath-test-dynamicpv-pstj": Phase="Pending", Reason="", readiness=false. Elapsed: 6.444355662s
Nov 17 00:50:20.513: INFO: Pod "pod-subpath-test-dynamicpv-pstj": Phase="Pending", Reason="", readiness=false. Elapsed: 8.556384448s
Nov 17 00:50:22.625: INFO: Pod "pod-subpath-test-dynamicpv-pstj": Phase="Pending", Reason="", readiness=false. Elapsed: 10.667645004s
... skipping 13 lines ...
Nov 17 00:50:52.188: INFO: Pod "pod-subpath-test-dynamicpv-pstj": Phase="Pending", Reason="", readiness=false. Elapsed: 40.230688452s
Nov 17 00:50:54.299: INFO: Pod "pod-subpath-test-dynamicpv-pstj": Phase="Pending", Reason="", readiness=false. Elapsed: 42.34175616s
Nov 17 00:50:56.410: INFO: Pod "pod-subpath-test-dynamicpv-pstj": Phase="Pending", Reason="", readiness=false. Elapsed: 44.453264018s
Nov 17 00:50:58.522: INFO: Pod "pod-subpath-test-dynamicpv-pstj": Phase="Pending", Reason="", readiness=false. Elapsed: 46.564733206s
Nov 17 00:51:00.632: INFO: Pod "pod-subpath-test-dynamicpv-pstj": Phase="Succeeded", Reason="", readiness=false. Elapsed: 48.67488915s
[1mSTEP[0m: Saw pod success
Nov 17 00:51:00.632: INFO: Pod "pod-subpath-test-dynamicpv-pstj" satisfied condition "Succeeded or Failed"
Nov 17 00:51:00.741: INFO: Trying to get logs from node k8s-agentpool1-18975173-vmss000001 pod pod-subpath-test-dynamicpv-pstj container test-container-subpath-dynamicpv-pstj: <nil>
[1mSTEP[0m: delete the pod
Nov 17 00:51:00.971: INFO: Waiting for pod pod-subpath-test-dynamicpv-pstj to disappear
Nov 17 00:51:01.087: INFO: Pod pod-subpath-test-dynamicpv-pstj no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-pstj
Nov 17 00:51:01.087: INFO: Deleting pod "pod-subpath-test-dynamicpv-pstj" in namespace "provisioning-1594"
... skipping 23 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support readOnly directory specified in the volumeMount
    [90mtest/e2e/storage/testsuites/subpath.go:367[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly directory specified in the volumeMount","total":34,"completed":3,"skipped":228,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (xfs)][Slow] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 17 00:51:42.989: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 21 lines ...
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 17 00:51:43.020: INFO: >>> kubeConfig: /root/tmp1926242230/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename topology
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  test/e2e/storage/testsuites/topology.go:194
Nov 17 00:51:43.789: INFO: Driver didn't provide topology keys -- skipping
[AfterEach] [Testpattern: Dynamic PV (delayed binding)] topology
  test/e2e/framework/framework.go:188
Nov 17 00:51:43.791: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[1mSTEP[0m: Destroying namespace "topology-6219" for this suite.


[36m[1mS [SKIPPING] [1.100 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (delayed binding)] topology
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement][0m
    [90mtest/e2e/storage/testsuites/topology.go:194[0m

    [36mDriver didn't provide topology keys -- skipping[0m

    test/e2e/storage/testsuites/topology.go:126
[90m------------------------------[0m
... skipping 29 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:280[0m

    [36mDistro debian doesn't support ntfs -- skipping[0m

    test/e2e/storage/framework/testsuite.go:127
[90m------------------------------[0m
... skipping 161 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents
    [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents","total":26,"completed":2,"skipped":90,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] multiVolume [Slow][0m 
  [1mshould concurrently access the single read-only volume from pods on the same node[0m
  [37mtest/e2e/storage/testsuites/multivolume.go:423[0m
... skipping 124 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single read-only volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:423[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":36,"completed":3,"skipped":202,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] provisioning[0m 
  [1mshould provision storage with snapshot data source [Feature:VolumeSnapshotDataSource][0m
  [37mtest/e2e/storage/testsuites/provisioning.go:208[0m
... skipping 167 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] provisioning
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
    [90mtest/e2e/storage/testsuites/provisioning.go:208[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":27,"completed":3,"skipped":117,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
... skipping 65 lines ...
Nov 17 00:55:55.951: INFO: Waiting for pod external-injector to disappear
Nov 17 00:55:56.061: INFO: Pod external-injector still exists
Nov 17 00:55:57.950: INFO: Waiting for pod external-injector to disappear
Nov 17 00:55:58.059: INFO: Pod external-injector still exists
Nov 17 00:55:59.950: INFO: Waiting for pod external-injector to disappear
Nov 17 00:56:00.061: INFO: Pod external-injector no longer exists
Nov 17 00:56:00.061: FAIL: Failed to create injector pod: timed out waiting for the condition

Full Stack Trace
k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumesTestSuite).DefineTests.func3()
	test/e2e/storage/testsuites/volumes.go:188 +0x485
k8s.io/kubernetes/test/e2e.RunE2ETests(0x257e5b7?)
	test/e2e/e2e.go:130 +0x686
... skipping 42 lines ...
Nov 17 00:58:13.365: INFO: At 2022-11-17 00:50:46 +0000 UTC - event for test.csi.azure.comndkm6: {test.csi.azure.com_k8s-agentpool1-18975173-vmss000001_8455a393-7015-4ee9-b20f-0a0e1dc5ed82 } Provisioning: External provisioner is provisioning volume for claim "volume-1678/test.csi.azure.comndkm6"
Nov 17 00:58:13.365: INFO: At 2022-11-17 00:50:46 +0000 UTC - event for test.csi.azure.comndkm6: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "test.csi.azure.com" or manually created by system administrator
Nov 17 00:58:13.365: INFO: At 2022-11-17 00:50:49 +0000 UTC - event for test.csi.azure.comndkm6: {test.csi.azure.com_k8s-agentpool1-18975173-vmss000001_8455a393-7015-4ee9-b20f-0a0e1dc5ed82 } ProvisioningSucceeded: Successfully provisioned volume pvc-6984cc32-fbea-4cf2-951d-50bfc2655c9c
Nov 17 00:58:13.365: INFO: At 2022-11-17 00:50:51 +0000 UTC - event for external-injector: {default-scheduler } Scheduled: Successfully assigned volume-1678/external-injector to k8s-agentpool1-18975173-vmss000000
Nov 17 00:58:13.365: INFO: At 2022-11-17 00:50:58 +0000 UTC - event for external-injector: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-6984cc32-fbea-4cf2-951d-50bfc2655c9c" 
Nov 17 00:58:13.365: INFO: At 2022-11-17 00:52:54 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-18975173-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[kube-api-access-pw7mt external-volume-0]: timed out waiting for the condition
Nov 17 00:58:13.365: INFO: At 2022-11-17 00:52:59 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-18975173-vmss000000} FailedMount: MountVolume.MountDevice failed for volume "pvc-6984cc32-fbea-4cf2-951d-50bfc2655c9c" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 17 00:58:13.365: INFO: At 2022-11-17 00:55:11 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-18975173-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[external-volume-0 kube-api-access-pw7mt]: timed out waiting for the condition
Nov 17 00:58:13.365: INFO: At 2022-11-17 00:55:51 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-18975173-vmss000000} FailedMount: MountVolume.MountDevice failed for volume "pvc-6984cc32-fbea-4cf2-951d-50bfc2655c9c" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-6984cc32-fbea-4cf2-951d-50bfc2655c9c) to be attached to node (k8s-agentpool1-18975173-vmss000000): rpc error: code = Unknown desc = context deadline exceeded
Nov 17 00:58:13.365: INFO: At 2022-11-17 00:55:56 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-18975173-vmss000000} Created: Created container external-injector
Nov 17 00:58:13.365: INFO: At 2022-11-17 00:55:56 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-18975173-vmss000000} Started: Started container external-injector
Nov 17 00:58:13.365: INFO: At 2022-11-17 00:55:56 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-18975173-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2"
Nov 17 00:58:13.365: INFO: At 2022-11-17 00:55:56 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-18975173-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 339.645429ms
Nov 17 00:58:13.365: INFO: At 2022-11-17 00:55:57 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-18975173-vmss000000} Killing: Stopping container external-injector
Nov 17 00:58:13.365: INFO: At 2022-11-17 00:56:00 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-18975173-vmss000000} FailedKillPod: error killing pod: failed to "KillContainer" for "external-injector" with KillContainerError: "rpc error: code = Unknown desc = Error response from daemon: No such container: 96a72afc65feaa5e2019afe85c26e011b6fcacac9bf0c07c8f9e92e564744cca"
Nov 17 00:58:13.477: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov 17 00:58:13.477: INFO: 
Nov 17 00:58:13.693: INFO: 
Logging node info for node k8s-agentpool1-18975173-vmss000000
Nov 17 00:58:13.802: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-18975173-vmss000000    6bbe8ef7-c343-4d8d-bfad-71482956f38d 8561 0 2022-11-17 00:33:10 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D8s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westeurope failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-qb0hkalz kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-18975173-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_D8s_v3 storageprofile:managed storagetier:Premium_LRS topology.kubernetes.io/region:westeurope topology.kubernetes.io/zone:0 topology.test.csi.azure.com/zone:] map[csi.volume.kubernetes.io/nodeid:{"test.csi.azure.com":"k8s-agentpool1-18975173-vmss000000"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2022-11-17 00:33:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{}}}} } {kubectl-label Update v1 2022-11-17 00:33:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kube-controller-manager Update v1 2022-11-17 00:33:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.3.0/24\"":{}}}} } {cloud-controller-manager Update v1 2022-11-17 00:34:25 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {cloud-node-manager Update v1 2022-11-17 00:34:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2022-11-17 00:57:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.test.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2022-11-17 00:58:04 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-18975173-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{31025332224 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{33672720384 0} {<nil>} 32883516Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{27922798956 0} {<nil>} 27922798956 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{32886288384 0} {<nil>} 32115516Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-17 00:34:25 +0000 UTC,LastTransitionTime:2022-11-17 00:34:25 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-17 00:57:41 +0000 UTC,LastTransitionTime:2022-11-17 00:33:09 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-17 00:57:41 +0000 UTC,LastTransitionTime:2022-11-17 00:33:09 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-17 00:57:41 +0000 UTC,LastTransitionTime:2022-11-17 00:33:09 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-17 00:57:41 +0000 UTC,LastTransitionTime:2022-11-17 00:33:31 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool1-18975173-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:868dc48bcc3546e6983f3183d46a3bf9,SystemUUID:3ae78ac1-15da-624c-b4a4-d4535e608c88,BootID:d5e21401-c116-44ad-aaa1-f7439638ebd4,KernelVersion:5.4.0-1094-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.14+azure-1,KubeletVersion:v1.23.13,KubeProxyVersion:v1.23.13,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:5caa0cfd48503b664488aec8c33484a6813c7cd69ecf2ab8655353b18ac91a1c k8sprow.azurecr.io/azuredisk-csi:latest-v2-c551e0323ad732edee8d00639b70d4fc25a9b407],SizeBytes:142868444,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver@sha256:46c1541fe63e34dfc697e3acbf8d40661ba8af217bb5457b8ff5adb9af4b248e mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.23.13],SizeBytes:135172310,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:f5241226198f5a54d22540acf2b3933ea0f49458f90c51fc75833d0c428687b8 k8s.gcr.io/e2e-test-images/agnhost:2.36],SizeBytes:125426639,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager@sha256:041c9d6da46acbdbaaa937ef2f370a341cf86c77654304ce3b04ab82f28b4952 mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.23.13],SizeBytes:124990264,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:9aeab889b5d0623cd5ca7aceb506ed2bc34836ee99fec603bfd0e0abe95e983e mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.13],SizeBytes:112304283,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:95d70317663e1da01ef37d42cce2e35241292b777e934ec4e58089e3d69a7e10 mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.23.13],SizeBytes:53486364,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:949db4eea46ea31f9afa0b49ad0b55f7a31f91aa03b408e0ac9b7720c8ef7d29 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.11],SizeBytes:51376235,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:72fbc9d17947d1a54c09a16671e6810fd7c0d8f09c0502fdbdd212c2192451a7 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.14],SizeBytes:51044459,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:075ea1f8270312350f1396ab6677251e803e61a523822d5abfa5e6acd180cfab mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.11],SizeBytes:50806891,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:eb7d329559d954093f7a234bc0ff322764de48a0b2a1007229032fb53e7bb9eb mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.24.0],SizeBytes:49070187,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:2dfff4dc10f6d25b0a34c01bf4d166540cbf1ff2bccd52635350039138eb23c4 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.18],SizeBytes:48775275,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3e23b9ab6bfee42a1751019c21a60aa474964bd4cba00cbed5055430424242cc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.24.0],SizeBytes:48500843,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:acec9ee6626c2cdea89107ba191ae36e6f60339c9b3b3215b5a59bf0e2e730ba mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.14],SizeBytes:47853675,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:759ac7020e982884b99121583bd577658b5663b1c1ed69746356aa6ce9879da4 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.18],SizeBytes:45850731,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:2fbd1e1a0538a06f2061afd45975df70c942654aa7f86e920720169ee439c2d6 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.5.1],SizeBytes:19743433,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:31547791294872570393470991481c2477a311031d3a03e0ae54eb164347dc34 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0],SizeBytes:17857485,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-63911cdd-1095-4b04-a2e8-d47057864824],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-63911cdd-1095-4b04-a2e8-d47057864824,DevicePath:,},},Config:nil,},}
Nov 17 00:58:13.803: INFO: 
... skipping 110 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext3)] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [91m[1mshould store data [Measurement][0m
    [90mtest/e2e/storage/testsuites/volumes.go:161[0m

    [91mNov 17 00:56:00.061: Failed to create injector pod: timed out waiting for the condition[0m

    test/e2e/storage/testsuites/volumes.go:188
[90m------------------------------[0m
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data","total":43,"completed":2,"skipped":235,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 00:58:16.842: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 61 lines ...
[1mSTEP[0m: Deleting pod external-injector in namespace multivolume-9729
Nov 17 00:50:36.105: INFO: Waiting for pod external-injector to disappear
Nov 17 00:50:36.216: INFO: Pod external-injector still exists
Nov 17 00:50:38.216: INFO: Waiting for pod external-injector to disappear
Nov 17 00:50:38.327: INFO: Pod external-injector no longer exists
[1mSTEP[0m: Creating pod1 with a volume on {Name: Selector:map[] Affinity:nil}
Nov 17 00:55:38.886: FAIL: Unexpected error:
    <*errors.errorString | 0xc0030f80d0>: {
        s: "pod \"pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015\" is not Running: timed out waiting for the condition",
    }
    pod "pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015" is not Running: timed out waiting for the condition
occurred

... skipping 44 lines ...
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:50:29 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-18975173-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2"
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:50:30 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-18975173-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 375.618468ms
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:50:30 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-18975173-vmss000000} Created: Created container external-injector
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:50:30 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-18975173-vmss000000} Started: Started container external-injector
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:50:36 +0000 UTC - event for external-injector: {kubelet k8s-agentpool1-18975173-vmss000000} Killing: Stopping container external-injector
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:50:38 +0000 UTC - event for pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015: {default-scheduler } Scheduled: Successfully assigned multivolume-9729/pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015 to k8s-agentpool1-18975173-vmss000002
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:50:38 +0000 UTC - event for pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015: {attachdetach-controller } FailedAttachVolume: Multi-Attach error for volume "pvc-29f538b3-3d88-41cd-848c-96e68a027152" Volume is already exclusively attached to one node and can't be attached to another
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:50:38 +0000 UTC - event for test.csi.azure.comfmfst-cloned: {test.csi.azure.com_k8s-agentpool1-18975173-vmss000001_8455a393-7015-4ee9-b20f-0a0e1dc5ed82 } Provisioning: External provisioner is provisioning volume for claim "multivolume-9729/test.csi.azure.comfmfst-cloned"
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:50:38 +0000 UTC - event for test.csi.azure.comfmfst-cloned: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "test.csi.azure.com" or manually created by system administrator
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:50:40 +0000 UTC - event for test.csi.azure.comfmfst-cloned: {test.csi.azure.com_k8s-agentpool1-18975173-vmss000001_8455a393-7015-4ee9-b20f-0a0e1dc5ed82 } ProvisioningSucceeded: Successfully provisioned volume pvc-92232f29-47f7-43b8-94a4-c29395e02cbd
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:51:21 +0000 UTC - event for pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-29f538b3-3d88-41cd-848c-96e68a027152" 
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:52:41 +0000 UTC - event for pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015: {kubelet k8s-agentpool1-18975173-vmss000002} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 kube-api-access-wl5rg]: timed out waiting for the condition
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:53:21 +0000 UTC - event for pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015: {kubelet k8s-agentpool1-18975173-vmss000002} FailedMount: MountVolume.MountDevice failed for volume "pvc-29f538b3-3d88-41cd-848c-96e68a027152" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:54:56 +0000 UTC - event for pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015: {kubelet k8s-agentpool1-18975173-vmss000002} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[kube-api-access-wl5rg volume1]: timed out waiting for the condition
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:56:20 +0000 UTC - event for pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015: {kubelet k8s-agentpool1-18975173-vmss000002} FailedMount: MountVolume.MountDevice failed for volume "pvc-29f538b3-3d88-41cd-848c-96e68a027152" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-29f538b3-3d88-41cd-848c-96e68a027152) to be attached to node (k8s-agentpool1-18975173-vmss000002): rpc error: code = Unknown desc = context deadline exceeded
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:57:00 +0000 UTC - event for pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015: {kubelet k8s-agentpool1-18975173-vmss000002} Started: Started container write-pod
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:57:00 +0000 UTC - event for pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015: {kubelet k8s-agentpool1-18975173-vmss000002} Created: Created container write-pod
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:57:00 +0000 UTC - event for pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015: {kubelet k8s-agentpool1-18975173-vmss000002} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5" in 383.203469ms
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:57:00 +0000 UTC - event for pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015: {kubelet k8s-agentpool1-18975173-vmss000002} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5"
Nov 17 00:58:15.553: INFO: At 2022-11-17 00:57:01 +0000 UTC - event for pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015: {kubelet k8s-agentpool1-18975173-vmss000002} Killing: Stopping container write-pod
Nov 17 00:58:15.663: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
... skipping 115 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [91m[1mshould concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [Measurement][0m
    [90mtest/e2e/storage/testsuites/multivolume.go:378[0m

    [91mNov 17 00:55:38.886: Unexpected error:
        <*errors.errorString | 0xc0030f80d0>: {
            s: "pod \"pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015\" is not Running: timed out waiting for the condition",
        }
        pod "pod-fdbe3c23-881c-48c1-9c9a-6d03fd0f8015" is not Running: timed out waiting for the condition
    occurred[0m

    test/e2e/storage/testsuites/multivolume.go:696
[90m------------------------------[0m
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":27,"completed":2,"skipped":132,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 00:58:18.994: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 3 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:280[0m

    [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 91 lines ...
[1mSTEP[0m: creating a claim
Nov 17 00:50:46.052: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com69hfz] to have phase Bound
Nov 17 00:50:46.162: INFO: PersistentVolumeClaim test.csi.azure.com69hfz found but phase is Pending instead of Bound.
Nov 17 00:50:48.272: INFO: PersistentVolumeClaim test.csi.azure.com69hfz found but phase is Pending instead of Bound.
Nov 17 00:50:50.383: INFO: PersistentVolumeClaim test.csi.azure.com69hfz found and phase=Bound (4.330907236s)
[1mSTEP[0m: Creating pod on {Name: Selector:map[] Affinity:nil} with multiple volumes
Nov 17 00:55:51.053: FAIL: Unexpected error:
    <*errors.errorString | 0xc003232420>: {
        s: "pod \"pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac\" is not Running: timed out waiting for the condition",
    }
    pod "pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac" is not Running: timed out waiting for the condition
occurred

... skipping 63 lines ...
Nov 17 00:58:23.388: INFO: At 2022-11-17 00:50:46 +0000 UTC - event for test.csi.azure.com69hfz: {test.csi.azure.com_k8s-agentpool1-18975173-vmss000001_8455a393-7015-4ee9-b20f-0a0e1dc5ed82 } Provisioning: External provisioner is provisioning volume for claim "multivolume-5284/test.csi.azure.com69hfz"
Nov 17 00:58:23.388: INFO: At 2022-11-17 00:50:48 +0000 UTC - event for test.csi.azure.com69hfz: {test.csi.azure.com_k8s-agentpool1-18975173-vmss000001_8455a393-7015-4ee9-b20f-0a0e1dc5ed82 } ProvisioningSucceeded: Successfully provisioned volume pvc-d8992cc6-390e-4dc5-8aa1-0cca83be09cf
Nov 17 00:58:23.388: INFO: At 2022-11-17 00:50:50 +0000 UTC - event for pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac: {default-scheduler } Scheduled: Successfully assigned multivolume-5284/pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac to k8s-agentpool1-18975173-vmss000000
Nov 17 00:58:23.388: INFO: At 2022-11-17 00:50:59 +0000 UTC - event for pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-d8992cc6-390e-4dc5-8aa1-0cca83be09cf" 
Nov 17 00:58:23.388: INFO: At 2022-11-17 00:50:59 +0000 UTC - event for pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-6391970f-404d-4b8c-b710-62c46f80ed2e" 
Nov 17 00:58:23.388: INFO: At 2022-11-17 00:52:53 +0000 UTC - event for pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac: {kubelet k8s-agentpool1-18975173-vmss000000} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1 volume2], unattached volumes=[kube-api-access-6lnx6 volume1 volume2]: timed out waiting for the condition
Nov 17 00:58:23.388: INFO: At 2022-11-17 00:52:59 +0000 UTC - event for pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac: {kubelet k8s-agentpool1-18975173-vmss000000} FailedMapVolume: MapVolume.SetUpDevice failed for volume "pvc-d8992cc6-390e-4dc5-8aa1-0cca83be09cf" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 17 00:58:23.388: INFO: At 2022-11-17 00:52:59 +0000 UTC - event for pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac: {kubelet k8s-agentpool1-18975173-vmss000000} FailedMapVolume: MapVolume.SetUpDevice failed for volume "pvc-6391970f-404d-4b8c-b710-62c46f80ed2e" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-6391970f-404d-4b8c-b710-62c46f80ed2e) to be attached to node (k8s-agentpool1-18975173-vmss000000): context deadline exceeded
Nov 17 00:58:23.388: INFO: At 2022-11-17 00:54:59 +0000 UTC - event for pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac: {kubelet k8s-agentpool1-18975173-vmss000000} FailedMapVolume: MapVolume.SetUpDevice failed for volume "pvc-6391970f-404d-4b8c-b710-62c46f80ed2e" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 17 00:58:23.388: INFO: At 2022-11-17 00:55:50 +0000 UTC - event for pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac: {kubelet k8s-agentpool1-18975173-vmss000000} FailedMapVolume: MapVolume.SetUpDevice failed for volume "pvc-6391970f-404d-4b8c-b710-62c46f80ed2e" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-6391970f-404d-4b8c-b710-62c46f80ed2e) to be attached to node (k8s-agentpool1-18975173-vmss000000): rpc error: code = Unknown desc = context deadline exceeded
Nov 17 00:58:23.388: INFO: At 2022-11-17 00:55:50 +0000 UTC - event for pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac: {kubelet k8s-agentpool1-18975173-vmss000000} FailedMapVolume: MapVolume.SetUpDevice failed for volume "pvc-d8992cc6-390e-4dc5-8aa1-0cca83be09cf" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-d8992cc6-390e-4dc5-8aa1-0cca83be09cf) to be attached to node (k8s-agentpool1-18975173-vmss000000): rpc error: code = Unknown desc = context deadline exceeded
Nov 17 00:58:23.388: INFO: At 2022-11-17 00:55:54 +0000 UTC - event for pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac: {kubelet k8s-agentpool1-18975173-vmss000000} SuccessfulMountVolume: MapVolume.MapPodDevice succeeded for volume "pvc-6391970f-404d-4b8c-b710-62c46f80ed2e" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-6391970f-404d-4b8c-b710-62c46f80ed2e/dev"
Nov 17 00:58:23.389: INFO: At 2022-11-17 00:55:54 +0000 UTC - event for pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac: {kubelet k8s-agentpool1-18975173-vmss000000} SuccessfulMountVolume: MapVolume.MapPodDevice succeeded for volume "pvc-6391970f-404d-4b8c-b710-62c46f80ed2e" volumeMapPath "/var/lib/kubelet/pods/aace6931-9488-4e29-a7c1-60a9f13feb1f/volumeDevices/kubernetes.io~csi"
Nov 17 00:58:23.389: INFO: At 2022-11-17 00:55:54 +0000 UTC - event for pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac: {kubelet k8s-agentpool1-18975173-vmss000000} SuccessfulMountVolume: MapVolume.MapPodDevice succeeded for volume "pvc-d8992cc6-390e-4dc5-8aa1-0cca83be09cf" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-d8992cc6-390e-4dc5-8aa1-0cca83be09cf/dev"
Nov 17 00:58:23.389: INFO: At 2022-11-17 00:55:54 +0000 UTC - event for pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac: {kubelet k8s-agentpool1-18975173-vmss000000} SuccessfulMountVolume: MapVolume.MapPodDevice succeeded for volume "pvc-d8992cc6-390e-4dc5-8aa1-0cca83be09cf" volumeMapPath "/var/lib/kubelet/pods/aace6931-9488-4e29-a7c1-60a9f13feb1f/volumeDevices/kubernetes.io~csi"
Nov 17 00:58:23.389: INFO: At 2022-11-17 00:55:55 +0000 UTC - event for pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac: {kubelet k8s-agentpool1-18975173-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2"
Nov 17 00:58:23.389: INFO: At 2022-11-17 00:55:56 +0000 UTC - event for pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac: {kubelet k8s-agentpool1-18975173-vmss000000} Killing: Stopping container write-pod
... skipping 123 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [91m[1mshould access to two volumes with the same volume mode and retain data across pod recreation on different node [Measurement][0m
    [90mtest/e2e/storage/testsuites/multivolume.go:168[0m

    [91mNov 17 00:55:51.053: Unexpected error:
        <*errors.errorString | 0xc003232420>: {
            s: "pod \"pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac\" is not Running: timed out waiting for the condition",
        }
        pod "pod-006991f9-5588-4dfd-9a2e-96e09d01e2ac" is not Running: timed out waiting for the condition
    occurred[0m

    test/e2e/storage/testsuites/multivolume.go:497
[90m------------------------------[0m
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":37,"completed":1,"skipped":428,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node"]}

[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning
  test/e2e/storage/framework/testsuite.go:51
Nov 17 00:58:26.750: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 87 lines ...
[1mSTEP[0m: creating a claim
Nov 17 00:51:50.162: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comtk6cb] to have phase Bound
Nov 17 00:51:50.318: INFO: PersistentVolumeClaim test.csi.azure.comtk6cb found but phase is Pending instead of Bound.
Nov 17 00:51:52.433: INFO: PersistentVolumeClaim test.csi.azure.comtk6cb found but phase is Pending instead of Bound.
Nov 17 00:51:54.543: INFO: PersistentVolumeClaim test.csi.azure.comtk6cb found and phase=Bound (4.380355734s)
[1mSTEP[0m: Creating pod on {Name: Selector:map[] Affinity:nil} with multiple volumes
Nov 17 00:56:55.210: FAIL: Unexpected error:
    <*errors.errorString | 0xc0028f06a0>: {
        s: "pod \"pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98\" is not Running: timed out waiting for the condition",
    }
    pod "pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98" is not Running: timed out waiting for the condition
occurred

... skipping 51 lines ...
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:51:50 +0000 UTC - event for test.csi.azure.comtk6cb: {test.csi.azure.com_k8s-agentpool1-18975173-vmss000001_8455a393-7015-4ee9-b20f-0a0e1dc5ed82 } Provisioning: External provisioner is provisioning volume for claim "multivolume-6251/test.csi.azure.comtk6cb"
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:51:52 +0000 UTC - event for test.csi.azure.comtk6cb: {test.csi.azure.com_k8s-agentpool1-18975173-vmss000001_8455a393-7015-4ee9-b20f-0a0e1dc5ed82 } ProvisioningSucceeded: Successfully provisioned volume pvc-81485d95-fe0b-49b7-aef7-7edeb434a4cc
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:51:54 +0000 UTC - event for pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98: {default-scheduler } Scheduled: Successfully assigned multivolume-6251/pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98 to k8s-agentpool1-18975173-vmss000001
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:51:56 +0000 UTC - event for pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-e0696c48-3947-43cf-a288-522461947b6b" 
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:51:56 +0000 UTC - event for pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-81485d95-fe0b-49b7-aef7-7edeb434a4cc" 
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:53:57 +0000 UTC - event for pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98: {kubelet k8s-agentpool1-18975173-vmss000001} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1 volume2], unattached volumes=[kube-api-access-klsjq volume1 volume2]: timed out waiting for the condition
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:54:05 +0000 UTC - event for pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98: {kubelet k8s-agentpool1-18975173-vmss000001} FailedMount: MountVolume.MountDevice failed for volume "pvc-e0696c48-3947-43cf-a288-522461947b6b" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:54:05 +0000 UTC - event for pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98: {kubelet k8s-agentpool1-18975173-vmss000001} FailedMount: MountVolume.MountDevice failed for volume "pvc-81485d95-fe0b-49b7-aef7-7edeb434a4cc" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:56:12 +0000 UTC - event for pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98: {kubelet k8s-agentpool1-18975173-vmss000001} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1 volume2], unattached volumes=[volume1 volume2 kube-api-access-klsjq]: timed out waiting for the condition
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:56:54 +0000 UTC - event for pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98: {kubelet k8s-agentpool1-18975173-vmss000001} FailedMount: MountVolume.MountDevice failed for volume "pvc-81485d95-fe0b-49b7-aef7-7edeb434a4cc" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-81485d95-fe0b-49b7-aef7-7edeb434a4cc) to be attached to node (k8s-agentpool1-18975173-vmss000001): rpc error: code = Unknown desc = context deadline exceeded
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:56:54 +0000 UTC - event for pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98: {kubelet k8s-agentpool1-18975173-vmss000001} FailedMount: MountVolume.MountDevice failed for volume "pvc-e0696c48-3947-43cf-a288-522461947b6b" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-e0696c48-3947-43cf-a288-522461947b6b) to be attached to node (k8s-agentpool1-18975173-vmss000001): rpc error: code = Unknown desc = context deadline exceeded
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:56:58 +0000 UTC - event for pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98: {kubelet k8s-agentpool1-18975173-vmss000001} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2"
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:56:59 +0000 UTC - event for pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98: {kubelet k8s-agentpool1-18975173-vmss000001} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 368.329914ms
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:56:59 +0000 UTC - event for pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98: {kubelet k8s-agentpool1-18975173-vmss000001} Started: Started container write-pod
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:56:59 +0000 UTC - event for pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98: {kubelet k8s-agentpool1-18975173-vmss000001} Created: Created container write-pod
Nov 17 00:58:24.195: INFO: At 2022-11-17 00:57:00 +0000 UTC - event for pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98: {kubelet k8s-agentpool1-18975173-vmss000001} Killing: Stopping container write-pod
Nov 17 00:58:24.303: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
... skipping 119 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [91m[1mshould access to two volumes with different volume mode and retain data across pod recreation on different node [Measurement][0m
    [90mtest/e2e/storage/testsuites/multivolume.go:248[0m

    [91mNov 17 00:56:55.210: Unexpected error:
        <*errors.errorString | 0xc0028f06a0>: {
            s: "pod \"pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98\" is not Running: timed out waiting for the condition",
        }
        pod "pod-6e260818-8cf0-4dd1-af0e-f0fe32b8ed98" is not Running: timed out waiting for the condition
    occurred[0m

    test/e2e/storage/testsuites/multivolume.go:497
[90m------------------------------[0m
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":34,"completed":3,"skipped":335,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] volume-expand
  test/e2e/storage/framework/testsuite.go:51
Nov 17 00:58:27.570: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 34 lines ...

    test/e2e/storage/framework/testsuite.go:127
[90m------------------------------[0m
[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould fail if subpath file is outside the volume [Slow][LinuxOnly][0m
  [37mtest/e2e/storage/testsuites/subpath.go:258[0m

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 17 00:55:43.442: INFO: >>> kubeConfig: /root/tmp1926242230/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename provisioning
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if subpath file is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:258
Nov 17 00:55:44.206: INFO: Creating resource for dynamic PV
Nov 17 00:55:44.206: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass provisioning-1227-e2e-sczllxk
[1mSTEP[0m: creating a claim
Nov 17 00:55:44.316: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 17 00:55:44.428: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comt4fsj] to have phase Bound
Nov 17 00:55:44.537: INFO: PersistentVolumeClaim test.csi.azure.comt4fsj found but phase is Pending instead of Bound.
Nov 17 00:55:46.647: INFO: PersistentVolumeClaim test.csi.azure.comt4fsj found but phase is Pending instead of Bound.
Nov 17 00:55:48.756: INFO: PersistentVolumeClaim test.csi.azure.comt4fsj found and phase=Bound (4.328214932s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-f6zb
[1mSTEP[0m: Checking for subpath error in container status
Nov 17 00:57:27.320: INFO: Deleting pod "pod-subpath-test-dynamicpv-f6zb" in namespace "provisioning-1227"
Nov 17 00:57:27.431: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-f6zb" to be fully deleted
[1mSTEP[0m: Deleting pod
Nov 17 00:57:29.649: INFO: Deleting pod "pod-subpath-test-dynamicpv-f6zb" in namespace "provisioning-1227"
[1mSTEP[0m: Deleting pvc
Nov 17 00:57:29.758: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comt4fsj"
... skipping 22 lines ...

[32m• [SLOW TEST:178.645 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should fail if subpath file is outside the volume [Slow][LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:258[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow][LinuxOnly]","total":27,"completed":4,"skipped":276,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 00:58:42.120: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 112 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
    [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":26,"completed":3,"skipped":141,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning
  test/e2e/storage/framework/testsuite.go:51
Nov 17 00:59:09.904: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 148 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should not mount / map unused volumes in a pod [LinuxOnly]
    [90mtest/e2e/storage/testsuites/volumemode.go:354[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":36,"completed":4,"skipped":236,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould support creating multiple subpath from same volumes [Slow][0m
  [37mtest/e2e/storage/testsuites/subpath.go:296[0m
... skipping 17 lines ...
Nov 17 00:58:43.123: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.compb8nb] to have phase Bound
Nov 17 00:58:43.232: INFO: PersistentVolumeClaim test.csi.azure.compb8nb found but phase is Pending instead of Bound.
Nov 17 00:58:45.343: INFO: PersistentVolumeClaim test.csi.azure.compb8nb found but phase is Pending instead of Bound.
Nov 17 00:58:47.453: INFO: PersistentVolumeClaim test.csi.azure.compb8nb found and phase=Bound (4.329746917s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-p5cj
[1mSTEP[0m: Creating a pod to test multi_subpath
Nov 17 00:58:47.785: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-p5cj" in namespace "provisioning-2832" to be "Succeeded or Failed"
Nov 17 00:58:47.899: INFO: Pod "pod-subpath-test-dynamicpv-p5cj": Phase="Pending", Reason="", readiness=false. Elapsed: 114.11095ms
Nov 17 00:58:50.009: INFO: Pod "pod-subpath-test-dynamicpv-p5cj": Phase="Pending", Reason="", readiness=false. Elapsed: 2.224182438s
Nov 17 00:58:52.119: INFO: Pod "pod-subpath-test-dynamicpv-p5cj": Phase="Pending", Reason="", readiness=false. Elapsed: 4.33455507s
Nov 17 00:58:54.230: INFO: Pod "pod-subpath-test-dynamicpv-p5cj": Phase="Pending", Reason="", readiness=false. Elapsed: 6.44529653s
Nov 17 00:58:56.342: INFO: Pod "pod-subpath-test-dynamicpv-p5cj": Phase="Pending", Reason="", readiness=false. Elapsed: 8.556666561s
Nov 17 00:58:58.452: INFO: Pod "pod-subpath-test-dynamicpv-p5cj": Phase="Pending", Reason="", readiness=false. Elapsed: 10.666674528s
Nov 17 00:59:00.561: INFO: Pod "pod-subpath-test-dynamicpv-p5cj": Phase="Pending", Reason="", readiness=false. Elapsed: 12.776002459s
Nov 17 00:59:02.670: INFO: Pod "pod-subpath-test-dynamicpv-p5cj": Phase="Pending", Reason="", readiness=false. Elapsed: 14.885486384s
Nov 17 00:59:04.781: INFO: Pod "pod-subpath-test-dynamicpv-p5cj": Phase="Pending", Reason="", readiness=false. Elapsed: 16.995665587s
Nov 17 00:59:06.892: INFO: Pod "pod-subpath-test-dynamicpv-p5cj": Phase="Succeeded", Reason="", readiness=false. Elapsed: 19.1070468s
[1mSTEP[0m: Saw pod success
Nov 17 00:59:06.892: INFO: Pod "pod-subpath-test-dynamicpv-p5cj" satisfied condition "Succeeded or Failed"
Nov 17 00:59:07.002: INFO: Trying to get logs from node k8s-agentpool1-18975173-vmss000001 pod pod-subpath-test-dynamicpv-p5cj container test-container-subpath-dynamicpv-p5cj: <nil>
[1mSTEP[0m: delete the pod
Nov 17 00:59:07.232: INFO: Waiting for pod pod-subpath-test-dynamicpv-p5cj to disappear
Nov 17 00:59:07.341: INFO: Pod pod-subpath-test-dynamicpv-p5cj no longer exists
[1mSTEP[0m: Deleting pod
Nov 17 00:59:07.341: INFO: Deleting pod "pod-subpath-test-dynamicpv-p5cj" in namespace "provisioning-2832"
... skipping 27 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support creating multiple subpath from same volumes [Slow]
    [90mtest/e2e/storage/testsuites/subpath.go:296[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support creating multiple subpath from same volumes [Slow]","total":27,"completed":5,"skipped":317,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)(allowExpansion)][Feature:Windows] volume-expand
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:00:19.778: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 124 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should store data
    [90mtest/e2e/storage/testsuites/volumes.go:161[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumes should store data","total":43,"completed":3,"skipped":274,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:00:20.441: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 130 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single read-only volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:423[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":27,"completed":3,"skipped":333,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:00:24.522: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 3 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:258[0m

    [36mDistro debian doesn't support ntfs -- skipping[0m

    test/e2e/storage/framework/testsuite.go:127
[90m------------------------------[0m
... skipping 235 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:138[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":37,"completed":2,"skipped":522,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource][0m [0mvolume snapshot controller[0m [90m[0m 
  [1mshould check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)[0m
  [37mtest/e2e/storage/testsuites/snapshottable.go:278[0m
... skipping 17 lines ...
Nov 17 00:58:28.657: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comg9k5g] to have phase Bound
Nov 17 00:58:28.767: INFO: PersistentVolumeClaim test.csi.azure.comg9k5g found but phase is Pending instead of Bound.
Nov 17 00:58:30.878: INFO: PersistentVolumeClaim test.csi.azure.comg9k5g found but phase is Pending instead of Bound.
Nov 17 00:58:32.988: INFO: PersistentVolumeClaim test.csi.azure.comg9k5g found and phase=Bound (4.330487203s)
[1mSTEP[0m: [init] starting a pod to use the claim
[1mSTEP[0m: [init] check pod success
Nov 17 00:58:33.426: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-nm8hn" in namespace "snapshotting-6503" to be "Succeeded or Failed"
Nov 17 00:58:33.536: INFO: Pod "pvc-snapshottable-tester-nm8hn": Phase="Pending", Reason="", readiness=false. Elapsed: 109.191913ms
Nov 17 00:58:35.646: INFO: Pod "pvc-snapshottable-tester-nm8hn": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219210137s
Nov 17 00:58:37.756: INFO: Pod "pvc-snapshottable-tester-nm8hn": Phase="Pending", Reason="", readiness=false. Elapsed: 4.329437625s
Nov 17 00:58:39.865: INFO: Pod "pvc-snapshottable-tester-nm8hn": Phase="Pending", Reason="", readiness=false. Elapsed: 6.439067718s
Nov 17 00:58:41.977: INFO: Pod "pvc-snapshottable-tester-nm8hn": Phase="Pending", Reason="", readiness=false. Elapsed: 8.550160384s
Nov 17 00:58:44.087: INFO: Pod "pvc-snapshottable-tester-nm8hn": Phase="Pending", Reason="", readiness=false. Elapsed: 10.660916784s
... skipping 6 lines ...
Nov 17 00:58:58.873: INFO: Pod "pvc-snapshottable-tester-nm8hn": Phase="Pending", Reason="", readiness=false. Elapsed: 25.446469104s
Nov 17 00:59:00.984: INFO: Pod "pvc-snapshottable-tester-nm8hn": Phase="Pending", Reason="", readiness=false. Elapsed: 27.557712347s
Nov 17 00:59:03.098: INFO: Pod "pvc-snapshottable-tester-nm8hn": Phase="Pending", Reason="", readiness=false. Elapsed: 29.671262243s
Nov 17 00:59:05.208: INFO: Pod "pvc-snapshottable-tester-nm8hn": Phase="Pending", Reason="", readiness=false. Elapsed: 31.781982202s
Nov 17 00:59:07.319: INFO: Pod "pvc-snapshottable-tester-nm8hn": Phase="Succeeded", Reason="", readiness=false. Elapsed: 33.892412509s
[1mSTEP[0m: Saw pod success
Nov 17 00:59:07.319: INFO: Pod "pvc-snapshottable-tester-nm8hn" satisfied condition "Succeeded or Failed"
[1mSTEP[0m: [init] checking the claim
Nov 17 00:59:07.429: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comg9k5g] to have phase Bound
Nov 17 00:59:07.540: INFO: PersistentVolumeClaim test.csi.azure.comg9k5g found and phase=Bound (110.837221ms)
[1mSTEP[0m: [init] checking the PV
[1mSTEP[0m: [init] deleting the pod
Nov 17 00:59:07.878: INFO: Pod pvc-snapshottable-tester-nm8hn has the following logs: 
... skipping 35 lines ...
Nov 17 00:59:20.748: INFO: WaitUntil finished successfully after 109.918843ms
[1mSTEP[0m: getting the snapshot and snapshot content
[1mSTEP[0m: checking the snapshot
[1mSTEP[0m: checking the SnapshotContent
[1mSTEP[0m: Modifying source data test
[1mSTEP[0m: modifying the data in the source PVC
Nov 17 00:59:21.300: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-cx8pd" in namespace "snapshotting-6503" to be "Succeeded or Failed"
Nov 17 00:59:21.409: INFO: Pod "pvc-snapshottable-data-tester-cx8pd": Phase="Pending", Reason="", readiness=false. Elapsed: 109.447757ms
Nov 17 00:59:23.521: INFO: Pod "pvc-snapshottable-data-tester-cx8pd": Phase="Pending", Reason="", readiness=false. Elapsed: 2.220803846s
Nov 17 00:59:25.633: INFO: Pod "pvc-snapshottable-data-tester-cx8pd": Phase="Pending", Reason="", readiness=false. Elapsed: 4.333012287s
Nov 17 00:59:27.743: INFO: Pod "pvc-snapshottable-data-tester-cx8pd": Phase="Pending", Reason="", readiness=false. Elapsed: 6.443039755s
Nov 17 00:59:29.857: INFO: Pod "pvc-snapshottable-data-tester-cx8pd": Phase="Pending", Reason="", readiness=false. Elapsed: 8.55659882s
Nov 17 00:59:31.969: INFO: Pod "pvc-snapshottable-data-tester-cx8pd": Phase="Pending", Reason="", readiness=false. Elapsed: 10.669200488s
... skipping 22 lines ...
Nov 17 01:00:20.523: INFO: Pod "pvc-snapshottable-data-tester-cx8pd": Phase="Pending", Reason="", readiness=false. Elapsed: 59.223392948s
Nov 17 01:00:22.634: INFO: Pod "pvc-snapshottable-data-tester-cx8pd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.334410112s
Nov 17 01:00:24.745: INFO: Pod "pvc-snapshottable-data-tester-cx8pd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.445335034s
Nov 17 01:00:26.857: INFO: Pod "pvc-snapshottable-data-tester-cx8pd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.556795355s
Nov 17 01:00:28.969: INFO: Pod "pvc-snapshottable-data-tester-cx8pd": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m7.668951554s
[1mSTEP[0m: Saw pod success
Nov 17 01:00:28.969: INFO: Pod "pvc-snapshottable-data-tester-cx8pd" satisfied condition "Succeeded or Failed"
Nov 17 01:00:29.193: INFO: Pod pvc-snapshottable-data-tester-cx8pd has the following logs: 
Nov 17 01:00:29.193: INFO: Deleting pod "pvc-snapshottable-data-tester-cx8pd" in namespace "snapshotting-6503"
Nov 17 01:00:29.307: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-cx8pd" to be fully deleted
[1mSTEP[0m: creating a pvc from the snapshot
[1mSTEP[0m: starting a pod to use the snapshot
Nov 17 01:00:49.864: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-qb0hkalz.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1926242230/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-6503 exec restored-pvc-tester-rxnbd --namespace=snapshotting-6503 -- cat /mnt/test/data'
... skipping 33 lines ...
Nov 17 01:01:16.076: INFO: volumesnapshotcontents pre-provisioned-snapcontent-9bd537cc-4370-428c-b762-fd94b309b60e has been found and is not deleted
Nov 17 01:01:17.187: INFO: volumesnapshotcontents pre-provisioned-snapcontent-9bd537cc-4370-428c-b762-fd94b309b60e has been found and is not deleted
Nov 17 01:01:18.306: INFO: volumesnapshotcontents pre-provisioned-snapcontent-9bd537cc-4370-428c-b762-fd94b309b60e has been found and is not deleted
Nov 17 01:01:19.417: INFO: volumesnapshotcontents pre-provisioned-snapcontent-9bd537cc-4370-428c-b762-fd94b309b60e has been found and is not deleted
Nov 17 01:01:20.527: INFO: volumesnapshotcontents pre-provisioned-snapcontent-9bd537cc-4370-428c-b762-fd94b309b60e has been found and is not deleted
Nov 17 01:01:21.638: INFO: volumesnapshotcontents pre-provisioned-snapcontent-9bd537cc-4370-428c-b762-fd94b309b60e has been found and is not deleted
Nov 17 01:01:22.639: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 17 01:01:22.748: INFO: Error getting logs for pod restored-pvc-tester-rxnbd: the server could not find the requested resource (get pods restored-pvc-tester-rxnbd)
Nov 17 01:01:22.748: INFO: Deleting pod "restored-pvc-tester-rxnbd" in namespace "snapshotting-6503"
Nov 17 01:01:22.858: INFO: deleting claim "snapshotting-6503"/"pvc-jdg27"
Nov 17 01:01:22.967: INFO: deleting snapshot "snapshotting-6503"/"pre-provisioned-snapshot-9bd537cc-4370-428c-b762-fd94b309b60e"
Nov 17 01:01:23.077: INFO: deleting snapshot content "pre-provisioned-snapcontent-9bd537cc-4370-428c-b762-fd94b309b60e"
Nov 17 01:01:23.414: INFO: Waiting up to 5m0s for volumesnapshotcontents pre-provisioned-snapcontent-9bd537cc-4370-428c-b762-fd94b309b60e to be deleted
Nov 17 01:01:23.524: INFO: volumesnapshotcontents pre-provisioned-snapcontent-9bd537cc-4370-428c-b762-fd94b309b60e has been found and is not deleted
... skipping 26 lines ...
    [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m
      
      [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":34,"completed":4,"skipped":446,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Ephemeral Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Ephemeral Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
... skipping 176 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should create read/write inline ephemeral volume
    [90mtest/e2e/storage/testsuites/ephemeral.go:196[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should create read/write inline ephemeral volume","total":26,"completed":4,"skipped":295,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould verify container cannot write to subpath readonly volumes [Slow][0m
  [37mtest/e2e/storage/testsuites/subpath.go:425[0m
... skipping 16 lines ...
Nov 17 01:00:25.491: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 17 01:00:25.605: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com46pm9] to have phase Bound
Nov 17 01:00:25.715: INFO: PersistentVolumeClaim test.csi.azure.com46pm9 found but phase is Pending instead of Bound.
Nov 17 01:00:27.825: INFO: PersistentVolumeClaim test.csi.azure.com46pm9 found but phase is Pending instead of Bound.
Nov 17 01:00:29.934: INFO: PersistentVolumeClaim test.csi.azure.com46pm9 found and phase=Bound (4.329866588s)
[1mSTEP[0m: Creating pod to format volume volume-prep-provisioning-3527
Nov 17 01:00:30.265: INFO: Waiting up to 5m0s for pod "volume-prep-provisioning-3527" in namespace "provisioning-3527" to be "Succeeded or Failed"
Nov 17 01:00:30.381: INFO: Pod "volume-prep-provisioning-3527": Phase="Pending", Reason="", readiness=false. Elapsed: 115.960675ms
Nov 17 01:00:32.491: INFO: Pod "volume-prep-provisioning-3527": Phase="Pending", Reason="", readiness=false. Elapsed: 2.225824326s
Nov 17 01:00:34.601: INFO: Pod "volume-prep-provisioning-3527": Phase="Pending", Reason="", readiness=false. Elapsed: 4.335792089s
Nov 17 01:00:36.712: INFO: Pod "volume-prep-provisioning-3527": Phase="Pending", Reason="", readiness=false. Elapsed: 6.44666652s
Nov 17 01:00:38.822: INFO: Pod "volume-prep-provisioning-3527": Phase="Pending", Reason="", readiness=false. Elapsed: 8.556952252s
Nov 17 01:00:40.934: INFO: Pod "volume-prep-provisioning-3527": Phase="Pending", Reason="", readiness=false. Elapsed: 10.668847558s
Nov 17 01:00:43.046: INFO: Pod "volume-prep-provisioning-3527": Phase="Pending", Reason="", readiness=false. Elapsed: 12.780107637s
Nov 17 01:00:45.157: INFO: Pod "volume-prep-provisioning-3527": Phase="Pending", Reason="", readiness=false. Elapsed: 14.891353302s
Nov 17 01:00:47.269: INFO: Pod "volume-prep-provisioning-3527": Phase="Pending", Reason="", readiness=false. Elapsed: 17.00314266s
Nov 17 01:00:49.379: INFO: Pod "volume-prep-provisioning-3527": Phase="Pending", Reason="", readiness=false. Elapsed: 19.11359483s
Nov 17 01:00:51.490: INFO: Pod "volume-prep-provisioning-3527": Phase="Pending", Reason="", readiness=false. Elapsed: 21.224498799s
Nov 17 01:00:53.601: INFO: Pod "volume-prep-provisioning-3527": Phase="Succeeded", Reason="", readiness=false. Elapsed: 23.335146833s
[1mSTEP[0m: Saw pod success
Nov 17 01:00:53.601: INFO: Pod "volume-prep-provisioning-3527" satisfied condition "Succeeded or Failed"
Nov 17 01:00:53.601: INFO: Deleting pod "volume-prep-provisioning-3527" in namespace "provisioning-3527"
Nov 17 01:00:53.725: INFO: Wait up to 5m0s for pod "volume-prep-provisioning-3527" to be fully deleted
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-sfcw
[1mSTEP[0m: Checking for subpath error in container status
Nov 17 01:01:00.169: INFO: Deleting pod "pod-subpath-test-dynamicpv-sfcw" in namespace "provisioning-3527"
Nov 17 01:01:00.282: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-sfcw" to be fully deleted
[1mSTEP[0m: Deleting pod
Nov 17 01:01:02.502: INFO: Deleting pod "pod-subpath-test-dynamicpv-sfcw" in namespace "provisioning-3527"
[1mSTEP[0m: Deleting pvc
Nov 17 01:01:02.612: INFO: Deleting PersistentVolumeClaim "test.csi.azure.com46pm9"
... skipping 19 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should verify container cannot write to subpath readonly volumes [Slow]
    [90mtest/e2e/storage/testsuites/subpath.go:425[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should verify container cannot write to subpath readonly volumes [Slow]","total":27,"completed":4,"skipped":445,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Ephemeral Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource][0m [0mvolume snapshot controller[0m [90m[0m 
  [1mshould check snapshot fields, check restore correctly works, check deletion (ephemeral)[0m
  [37mtest/e2e/storage/testsuites/snapshottable.go:177[0m
... skipping 10 lines ...
[It] should check snapshot fields, check restore correctly works, check deletion (ephemeral)
  test/e2e/storage/testsuites/snapshottable.go:177
Nov 17 01:00:16.909: INFO: Creating resource for dynamic PV
Nov 17 01:00:16.909: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass snapshotting-6941-e2e-sctkhdn
[1mSTEP[0m: [init] starting a pod to use the claim
Nov 17 01:00:17.141: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-5n6dx" in namespace "snapshotting-6941" to be "Succeeded or Failed"
Nov 17 01:00:17.252: INFO: Pod "pvc-snapshottable-tester-5n6dx": Phase="Pending", Reason="", readiness=false. Elapsed: 111.40333ms
Nov 17 01:00:19.370: INFO: Pod "pvc-snapshottable-tester-5n6dx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.229107877s
Nov 17 01:00:21.484: INFO: Pod "pvc-snapshottable-tester-5n6dx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.34291568s
Nov 17 01:00:23.597: INFO: Pod "pvc-snapshottable-tester-5n6dx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.45560393s
Nov 17 01:00:25.709: INFO: Pod "pvc-snapshottable-tester-5n6dx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.567737336s
Nov 17 01:00:27.821: INFO: Pod "pvc-snapshottable-tester-5n6dx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.680306913s
... skipping 2 lines ...
Nov 17 01:00:34.157: INFO: Pod "pvc-snapshottable-tester-5n6dx": Phase="Pending", Reason="", readiness=false. Elapsed: 17.015921677s
Nov 17 01:00:36.269: INFO: Pod "pvc-snapshottable-tester-5n6dx": Phase="Pending", Reason="", readiness=false. Elapsed: 19.127872167s
Nov 17 01:00:38.382: INFO: Pod "pvc-snapshottable-tester-5n6dx": Phase="Pending", Reason="", readiness=false. Elapsed: 21.241014062s
Nov 17 01:00:40.495: INFO: Pod "pvc-snapshottable-tester-5n6dx": Phase="Pending", Reason="", readiness=false. Elapsed: 23.353973125s
Nov 17 01:00:42.607: INFO: Pod "pvc-snapshottable-tester-5n6dx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 25.466248519s
[1mSTEP[0m: Saw pod success
Nov 17 01:00:42.607: INFO: Pod "pvc-snapshottable-tester-5n6dx" satisfied condition "Succeeded or Failed"
[1mSTEP[0m: [init] checking the claim
[1mSTEP[0m: creating a SnapshotClass
[1mSTEP[0m: creating a dynamic VolumeSnapshot
Nov 17 01:00:43.055: INFO: Waiting up to 5m0s for VolumeSnapshot snapshot-2b2c6 to become ready
Nov 17 01:00:43.167: INFO: VolumeSnapshot snapshot-2b2c6 found but is not ready.
Nov 17 01:00:45.290: INFO: VolumeSnapshot snapshot-2b2c6 found but is not ready.
... skipping 39 lines ...
Nov 17 01:01:54.247: INFO: volumesnapshotcontents snapcontent-dd037f56-2ce6-446d-9e6a-0c7001d97ede has been found and is not deleted
Nov 17 01:01:55.360: INFO: volumesnapshotcontents snapcontent-dd037f56-2ce6-446d-9e6a-0c7001d97ede has been found and is not deleted
Nov 17 01:01:56.472: INFO: volumesnapshotcontents snapcontent-dd037f56-2ce6-446d-9e6a-0c7001d97ede has been found and is not deleted
Nov 17 01:01:57.585: INFO: volumesnapshotcontents snapcontent-dd037f56-2ce6-446d-9e6a-0c7001d97ede has been found and is not deleted
Nov 17 01:01:58.696: INFO: volumesnapshotcontents snapcontent-dd037f56-2ce6-446d-9e6a-0c7001d97ede has been found and is not deleted
Nov 17 01:01:59.810: INFO: volumesnapshotcontents snapcontent-dd037f56-2ce6-446d-9e6a-0c7001d97ede has been found and is not deleted
Nov 17 01:02:00.810: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 17 01:02:00.946: INFO: Pod restored-pvc-tester-xnsdq has the following logs: 
Nov 17 01:02:00.946: INFO: Deleting pod "restored-pvc-tester-xnsdq" in namespace "snapshotting-6941"
Nov 17 01:02:01.061: INFO: Wait up to 5m0s for pod "restored-pvc-tester-xnsdq" to be fully deleted
Nov 17 01:02:33.291: INFO: deleting snapshot "snapshotting-6941"/"snapshot-2b2c6"
... skipping 26 lines ...
    [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m
      
      [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m
        should check snapshot fields, check restore correctly works, check deletion (ephemeral)
        [90mtest/e2e/storage/testsuites/snapshottable.go:177[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Ephemeral Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works, check deletion (ephemeral)","total":36,"completed":5,"skipped":267,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] provisioning[0m 
  [1mshould provision storage with snapshot data source [Feature:VolumeSnapshotDataSource][0m
  [37mtest/e2e/storage/testsuites/provisioning.go:208[0m
... skipping 126 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] provisioning
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
    [90mtest/e2e/storage/testsuites/provisioning.go:208[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":43,"completed":4,"skipped":323,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:02:53.947: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 228 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:298[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":27,"completed":6,"skipped":335,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (ext4)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:03:20.508: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 73 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should create read/write inline ephemeral volume
    [90mtest/e2e/storage/testsuites/ephemeral.go:196[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read/write inline ephemeral volume","total":27,"completed":5,"skipped":497,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral[0m 
  [1mshould create read/write inline ephemeral volume[0m
  [37mtest/e2e/storage/testsuites/ephemeral.go:196[0m
... skipping 45 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should create read/write inline ephemeral volume
    [90mtest/e2e/storage/testsuites/ephemeral.go:196[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read/write inline ephemeral volume","total":34,"completed":5,"skipped":675,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:04:14.194: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 331 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] provisioning
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should provision storage with pvc data source in parallel [Slow]
    [90mtest/e2e/storage/testsuites/provisioning.go:459[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with pvc data source in parallel [Slow]","total":37,"completed":3,"skipped":528,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node"]}
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:04:15.651: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
[AfterEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/framework/framework.go:188


[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Inline-volume (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:269[0m

    [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 142 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    [90mtest/e2e/storage/testsuites/multivolume.go:378[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":26,"completed":5,"skipped":309,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:04:37.217: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 59 lines ...
Nov 17 01:04:17.654: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com86jln] to have phase Bound
Nov 17 01:04:17.762: INFO: PersistentVolumeClaim test.csi.azure.com86jln found but phase is Pending instead of Bound.
Nov 17 01:04:19.870: INFO: PersistentVolumeClaim test.csi.azure.com86jln found but phase is Pending instead of Bound.
Nov 17 01:04:21.987: INFO: PersistentVolumeClaim test.csi.azure.com86jln found and phase=Bound (4.332905905s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-q6sc
[1mSTEP[0m: Creating a pod to test subpath
Nov 17 01:04:22.314: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-q6sc" in namespace "provisioning-4291" to be "Succeeded or Failed"
Nov 17 01:04:22.424: INFO: Pod "pod-subpath-test-dynamicpv-q6sc": Phase="Pending", Reason="", readiness=false. Elapsed: 109.867688ms
Nov 17 01:04:24.533: INFO: Pod "pod-subpath-test-dynamicpv-q6sc": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218484109s
Nov 17 01:04:26.642: INFO: Pod "pod-subpath-test-dynamicpv-q6sc": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327576498s
Nov 17 01:04:28.752: INFO: Pod "pod-subpath-test-dynamicpv-q6sc": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437404504s
Nov 17 01:04:30.861: INFO: Pod "pod-subpath-test-dynamicpv-q6sc": Phase="Pending", Reason="", readiness=false. Elapsed: 8.546386922s
Nov 17 01:04:32.970: INFO: Pod "pod-subpath-test-dynamicpv-q6sc": Phase="Pending", Reason="", readiness=false. Elapsed: 10.655851082s
... skipping 2 lines ...
Nov 17 01:04:39.300: INFO: Pod "pod-subpath-test-dynamicpv-q6sc": Phase="Pending", Reason="", readiness=false. Elapsed: 16.985788626s
Nov 17 01:04:41.409: INFO: Pod "pod-subpath-test-dynamicpv-q6sc": Phase="Pending", Reason="", readiness=false. Elapsed: 19.094941601s
Nov 17 01:04:43.518: INFO: Pod "pod-subpath-test-dynamicpv-q6sc": Phase="Pending", Reason="", readiness=false. Elapsed: 21.204072734s
Nov 17 01:04:45.627: INFO: Pod "pod-subpath-test-dynamicpv-q6sc": Phase="Pending", Reason="", readiness=false. Elapsed: 23.313024306s
Nov 17 01:04:47.738: INFO: Pod "pod-subpath-test-dynamicpv-q6sc": Phase="Succeeded", Reason="", readiness=false. Elapsed: 25.423410777s
[1mSTEP[0m: Saw pod success
Nov 17 01:04:47.738: INFO: Pod "pod-subpath-test-dynamicpv-q6sc" satisfied condition "Succeeded or Failed"
Nov 17 01:04:47.846: INFO: Trying to get logs from node k8s-agentpool1-18975173-vmss000002 pod pod-subpath-test-dynamicpv-q6sc container test-container-subpath-dynamicpv-q6sc: <nil>
[1mSTEP[0m: delete the pod
Nov 17 01:04:48.118: INFO: Waiting for pod pod-subpath-test-dynamicpv-q6sc to disappear
Nov 17 01:04:48.226: INFO: Pod pod-subpath-test-dynamicpv-q6sc no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-q6sc
Nov 17 01:04:48.226: INFO: Deleting pod "pod-subpath-test-dynamicpv-q6sc" in namespace "provisioning-4291"
... skipping 29 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support existing single file [LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:221[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing single file [LinuxOnly]","total":37,"completed":4,"skipped":564,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow][0m 
  [1mshould access to two volumes with the same volume mode and retain data across pod recreation on the same node[0m
  [37mtest/e2e/storage/testsuites/multivolume.go:138[0m
... skipping 192 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:138[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":34,"completed":6,"skipped":698,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext3)] volumes[0m 
  [1mshould allow exec of files on the volume[0m
  [37mtest/e2e/storage/testsuites/volumes.go:198[0m
... skipping 17 lines ...
Nov 17 01:03:21.564: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comqcbw5] to have phase Bound
Nov 17 01:03:21.673: INFO: PersistentVolumeClaim test.csi.azure.comqcbw5 found but phase is Pending instead of Bound.
Nov 17 01:03:23.784: INFO: PersistentVolumeClaim test.csi.azure.comqcbw5 found but phase is Pending instead of Bound.
Nov 17 01:03:25.894: INFO: PersistentVolumeClaim test.csi.azure.comqcbw5 found and phase=Bound (4.32991521s)
[1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-lhfw
[1mSTEP[0m: Creating a pod to test exec-volume-test
Nov 17 01:03:26.225: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-lhfw" in namespace "volume-3064" to be "Succeeded or Failed"
Nov 17 01:03:26.334: INFO: Pod "exec-volume-test-dynamicpv-lhfw": Phase="Pending", Reason="", readiness=false. Elapsed: 108.995073ms
Nov 17 01:03:28.446: INFO: Pod "exec-volume-test-dynamicpv-lhfw": Phase="Pending", Reason="", readiness=false. Elapsed: 2.221039159s
Nov 17 01:03:30.557: INFO: Pod "exec-volume-test-dynamicpv-lhfw": Phase="Pending", Reason="", readiness=false. Elapsed: 4.331599082s
Nov 17 01:03:32.673: INFO: Pod "exec-volume-test-dynamicpv-lhfw": Phase="Pending", Reason="", readiness=false. Elapsed: 6.447394534s
Nov 17 01:03:34.794: INFO: Pod "exec-volume-test-dynamicpv-lhfw": Phase="Pending", Reason="", readiness=false. Elapsed: 8.568100106s
Nov 17 01:03:36.904: INFO: Pod "exec-volume-test-dynamicpv-lhfw": Phase="Pending", Reason="", readiness=false. Elapsed: 10.67895792s
... skipping 28 lines ...
Nov 17 01:04:38.114: INFO: Pod "exec-volume-test-dynamicpv-lhfw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m11.888605474s
Nov 17 01:04:40.224: INFO: Pod "exec-volume-test-dynamicpv-lhfw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m13.998092928s
Nov 17 01:04:42.333: INFO: Pod "exec-volume-test-dynamicpv-lhfw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.107691016s
Nov 17 01:04:44.443: INFO: Pod "exec-volume-test-dynamicpv-lhfw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.217714642s
Nov 17 01:04:46.554: INFO: Pod "exec-volume-test-dynamicpv-lhfw": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m20.328448249s
[1mSTEP[0m: Saw pod success
Nov 17 01:04:46.554: INFO: Pod "exec-volume-test-dynamicpv-lhfw" satisfied condition "Succeeded or Failed"
Nov 17 01:04:46.663: INFO: Trying to get logs from node k8s-agentpool1-18975173-vmss000001 pod exec-volume-test-dynamicpv-lhfw container exec-container-dynamicpv-lhfw: <nil>
[1mSTEP[0m: delete the pod
Nov 17 01:04:46.928: INFO: Waiting for pod exec-volume-test-dynamicpv-lhfw to disappear
Nov 17 01:04:47.037: INFO: Pod exec-volume-test-dynamicpv-lhfw no longer exists
[1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-lhfw
Nov 17 01:04:47.037: INFO: Deleting pod "exec-volume-test-dynamicpv-lhfw" in namespace "volume-3064"
... skipping 38 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext3)] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should allow exec of files on the volume
    [90mtest/e2e/storage/testsuites/volumes.go:198[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should allow exec of files on the volume","total":27,"completed":7,"skipped":461,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:06:55.800: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 36 lines ...
Nov 17 01:02:42.283: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comkzwpj] to have phase Bound
Nov 17 01:02:42.400: INFO: PersistentVolumeClaim test.csi.azure.comkzwpj found but phase is Pending instead of Bound.
Nov 17 01:02:44.511: INFO: PersistentVolumeClaim test.csi.azure.comkzwpj found but phase is Pending instead of Bound.
Nov 17 01:02:46.624: INFO: PersistentVolumeClaim test.csi.azure.comkzwpj found and phase=Bound (4.340863579s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-6hl4
[1mSTEP[0m: Creating a pod to test subpath
Nov 17 01:02:46.961: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-6hl4" in namespace "provisioning-1758" to be "Succeeded or Failed"
Nov 17 01:02:47.073: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 111.777425ms
Nov 17 01:02:49.185: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 2.223843365s
Nov 17 01:02:51.299: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 4.33776502s
Nov 17 01:02:53.412: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 6.450446317s
Nov 17 01:02:55.524: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 8.562469398s
Nov 17 01:02:57.636: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 10.674453611s
... skipping 48 lines ...
Nov 17 01:04:41.168: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 1m54.206513239s
Nov 17 01:04:43.280: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 1m56.318639068s
Nov 17 01:04:45.393: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 1m58.431438074s
Nov 17 01:04:47.505: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 2m0.543468974s
Nov 17 01:04:49.619: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Succeeded", Reason="", readiness=false. Elapsed: 2m2.657603046s
[1mSTEP[0m: Saw pod success
Nov 17 01:04:49.619: INFO: Pod "pod-subpath-test-dynamicpv-6hl4" satisfied condition "Succeeded or Failed"
Nov 17 01:04:49.732: INFO: Trying to get logs from node k8s-agentpool1-18975173-vmss000000 pod pod-subpath-test-dynamicpv-6hl4 container test-container-subpath-dynamicpv-6hl4: <nil>
[1mSTEP[0m: delete the pod
Nov 17 01:04:49.993: INFO: Waiting for pod pod-subpath-test-dynamicpv-6hl4 to disappear
Nov 17 01:04:50.105: INFO: Pod pod-subpath-test-dynamicpv-6hl4 no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-6hl4
Nov 17 01:04:50.105: INFO: Deleting pod "pod-subpath-test-dynamicpv-6hl4" in namespace "provisioning-1758"
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-6hl4
[1mSTEP[0m: Creating a pod to test subpath
Nov 17 01:04:50.332: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-6hl4" in namespace "provisioning-1758" to be "Succeeded or Failed"
Nov 17 01:04:50.444: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 111.780136ms
Nov 17 01:04:52.557: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 2.224321869s
Nov 17 01:04:54.670: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 4.337778222s
Nov 17 01:04:56.783: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 6.450642033s
Nov 17 01:04:58.895: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 8.562289182s
Nov 17 01:05:01.008: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 10.675330448s
... skipping 46 lines ...
Nov 17 01:06:40.330: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 1m49.997828554s
Nov 17 01:06:42.443: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 1m52.110766538s
Nov 17 01:06:44.556: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 1m54.223785511s
Nov 17 01:06:46.669: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Pending", Reason="", readiness=false. Elapsed: 1m56.336642636s
Nov 17 01:06:48.781: INFO: Pod "pod-subpath-test-dynamicpv-6hl4": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m58.44889071s
[1mSTEP[0m: Saw pod success
Nov 17 01:06:48.781: INFO: Pod "pod-subpath-test-dynamicpv-6hl4" satisfied condition "Succeeded or Failed"
Nov 17 01:06:48.893: INFO: Trying to get logs from node k8s-agentpool1-18975173-vmss000000 pod pod-subpath-test-dynamicpv-6hl4 container test-container-subpath-dynamicpv-6hl4: <nil>
[1mSTEP[0m: delete the pod
Nov 17 01:06:49.151: INFO: Waiting for pod pod-subpath-test-dynamicpv-6hl4 to disappear
Nov 17 01:06:49.261: INFO: Pod pod-subpath-test-dynamicpv-6hl4 no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-6hl4
Nov 17 01:06:49.262: INFO: Deleting pod "pod-subpath-test-dynamicpv-6hl4" in namespace "provisioning-1758"
... skipping 29 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support existing directories when readOnly specified in the volumeSource
    [90mtest/e2e/storage/testsuites/subpath.go:397[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource","total":36,"completed":6,"skipped":299,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral[0m 
  [1mshould create read-only inline ephemeral volume[0m
  [37mtest/e2e/storage/testsuites/ephemeral.go:175[0m
... skipping 64 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should create read-only inline ephemeral volume
    [90mtest/e2e/storage/testsuites/ephemeral.go:175[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read-only inline ephemeral volume","total":26,"completed":6,"skipped":424,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:08:14.410: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 108 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:269[0m

    [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 274 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with different volume mode and retain data across pod recreation on different node
    [90mtest/e2e/storage/testsuites/multivolume.go:248[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":43,"completed":5,"skipped":390,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] multiVolume [Slow][0m 
  [1mshould access to two volumes with different volume mode and retain data across pod recreation on different node[0m
  [37mtest/e2e/storage/testsuites/multivolume.go:248[0m
... skipping 188 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with different volume mode and retain data across pod recreation on different node
    [90mtest/e2e/storage/testsuites/multivolume.go:248[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":27,"completed":6,"skipped":521,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:08:32.774: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 13 lines ...

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly][0m
  [37mtest/e2e/storage/testsuites/subpath.go:269[0m

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 17 01:06:41.234: INFO: >>> kubeConfig: /root/tmp1926242230/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename provisioning
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:269
Nov 17 01:06:41.994: INFO: Creating resource for dynamic PV
Nov 17 01:06:41.994: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass provisioning-382-e2e-scfw5v4
[1mSTEP[0m: creating a claim
Nov 17 01:06:42.104: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 17 01:06:42.214: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.coms8hh4] to have phase Bound
Nov 17 01:06:42.323: INFO: PersistentVolumeClaim test.csi.azure.coms8hh4 found but phase is Pending instead of Bound.
Nov 17 01:06:44.434: INFO: PersistentVolumeClaim test.csi.azure.coms8hh4 found but phase is Pending instead of Bound.
Nov 17 01:06:46.543: INFO: PersistentVolumeClaim test.csi.azure.coms8hh4 found and phase=Bound (4.328649048s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-fw48
[1mSTEP[0m: Checking for subpath error in container status
Nov 17 01:07:19.092: INFO: Deleting pod "pod-subpath-test-dynamicpv-fw48" in namespace "provisioning-382"
Nov 17 01:07:19.207: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-fw48" to be fully deleted
[1mSTEP[0m: Deleting pod
Nov 17 01:07:21.426: INFO: Deleting pod "pod-subpath-test-dynamicpv-fw48" in namespace "provisioning-382"
[1mSTEP[0m: Deleting pvc
Nov 17 01:07:21.534: INFO: Deleting PersistentVolumeClaim "test.csi.azure.coms8hh4"
... skipping 22 lines ...

[32m• [SLOW TEST:112.634 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:269[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]","total":34,"completed":7,"skipped":699,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:08:33.922: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 59 lines ...
Nov 17 01:08:02.885: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comgrwqt] to have phase Bound
Nov 17 01:08:02.996: INFO: PersistentVolumeClaim test.csi.azure.comgrwqt found but phase is Pending instead of Bound.
Nov 17 01:08:05.108: INFO: PersistentVolumeClaim test.csi.azure.comgrwqt found but phase is Pending instead of Bound.
Nov 17 01:08:07.221: INFO: PersistentVolumeClaim test.csi.azure.comgrwqt found and phase=Bound (4.336334847s)
[1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-nh6v
[1mSTEP[0m: Creating a pod to test exec-volume-test
Nov 17 01:08:07.556: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-nh6v" in namespace "volume-3085" to be "Succeeded or Failed"
Nov 17 01:08:07.668: INFO: Pod "exec-volume-test-dynamicpv-nh6v": Phase="Pending", Reason="", readiness=false. Elapsed: 111.375003ms
Nov 17 01:08:09.780: INFO: Pod "exec-volume-test-dynamicpv-nh6v": Phase="Pending", Reason="", readiness=false. Elapsed: 2.223361202s
Nov 17 01:08:11.893: INFO: Pod "exec-volume-test-dynamicpv-nh6v": Phase="Pending", Reason="", readiness=false. Elapsed: 4.336477654s
Nov 17 01:08:14.004: INFO: Pod "exec-volume-test-dynamicpv-nh6v": Phase="Pending", Reason="", readiness=false. Elapsed: 6.447738016s
Nov 17 01:08:16.117: INFO: Pod "exec-volume-test-dynamicpv-nh6v": Phase="Pending", Reason="", readiness=false. Elapsed: 8.56060134s
Nov 17 01:08:18.230: INFO: Pod "exec-volume-test-dynamicpv-nh6v": Phase="Pending", Reason="", readiness=false. Elapsed: 10.673808341s
Nov 17 01:08:20.344: INFO: Pod "exec-volume-test-dynamicpv-nh6v": Phase="Pending", Reason="", readiness=false. Elapsed: 12.787161864s
Nov 17 01:08:22.456: INFO: Pod "exec-volume-test-dynamicpv-nh6v": Phase="Pending", Reason="", readiness=false. Elapsed: 14.899215497s
Nov 17 01:08:24.569: INFO: Pod "exec-volume-test-dynamicpv-nh6v": Phase="Pending", Reason="", readiness=false. Elapsed: 17.012368205s
Nov 17 01:08:26.681: INFO: Pod "exec-volume-test-dynamicpv-nh6v": Phase="Pending", Reason="", readiness=false. Elapsed: 19.12481866s
Nov 17 01:08:28.793: INFO: Pod "exec-volume-test-dynamicpv-nh6v": Phase="Pending", Reason="", readiness=false. Elapsed: 21.23659703s
Nov 17 01:08:30.906: INFO: Pod "exec-volume-test-dynamicpv-nh6v": Phase="Succeeded", Reason="", readiness=false. Elapsed: 23.349362466s
[1mSTEP[0m: Saw pod success
Nov 17 01:08:30.906: INFO: Pod "exec-volume-test-dynamicpv-nh6v" satisfied condition "Succeeded or Failed"
Nov 17 01:08:31.017: INFO: Trying to get logs from node k8s-agentpool1-18975173-vmss000001 pod exec-volume-test-dynamicpv-nh6v container exec-container-dynamicpv-nh6v: <nil>
[1mSTEP[0m: delete the pod
Nov 17 01:08:31.270: INFO: Waiting for pod exec-volume-test-dynamicpv-nh6v to disappear
Nov 17 01:08:31.381: INFO: Pod exec-volume-test-dynamicpv-nh6v no longer exists
[1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-nh6v
Nov 17 01:08:31.381: INFO: Deleting pod "exec-volume-test-dynamicpv-nh6v" in namespace "volume-3085"
... skipping 27 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should allow exec of files on the volume
    [90mtest/e2e/storage/testsuites/volumes.go:198[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should allow exec of files on the volume","total":36,"completed":7,"skipped":321,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext4)] multiVolume [Slow][0m 
  [1mshould access to two volumes with the same volume mode and retain data across pod recreation on different node[0m
  [37mtest/e2e/storage/testsuites/multivolume.go:168[0m
... skipping 196 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    [90mtest/e2e/storage/testsuites/multivolume.go:168[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":37,"completed":5,"skipped":585,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
... skipping 331 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:138[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":27,"completed":8,"skipped":591,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] volumeIO
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:10:26.691: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 38 lines ...
Nov 17 01:08:35.047: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comwd8gb] to have phase Bound
Nov 17 01:08:35.155: INFO: PersistentVolumeClaim test.csi.azure.comwd8gb found but phase is Pending instead of Bound.
Nov 17 01:08:37.265: INFO: PersistentVolumeClaim test.csi.azure.comwd8gb found but phase is Pending instead of Bound.
Nov 17 01:08:39.375: INFO: PersistentVolumeClaim test.csi.azure.comwd8gb found and phase=Bound (4.328547008s)
[1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-4zrn
[1mSTEP[0m: Creating a pod to test exec-volume-test
Nov 17 01:08:39.704: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-4zrn" in namespace "volume-4642" to be "Succeeded or Failed"
Nov 17 01:08:39.816: INFO: Pod "exec-volume-test-dynamicpv-4zrn": Phase="Pending", Reason="", readiness=false. Elapsed: 111.992751ms
Nov 17 01:08:41.927: INFO: Pod "exec-volume-test-dynamicpv-4zrn": Phase="Pending", Reason="", readiness=false. Elapsed: 2.222575458s
Nov 17 01:08:44.037: INFO: Pod "exec-volume-test-dynamicpv-4zrn": Phase="Pending", Reason="", readiness=false. Elapsed: 4.333036793s
Nov 17 01:08:46.147: INFO: Pod "exec-volume-test-dynamicpv-4zrn": Phase="Pending", Reason="", readiness=false. Elapsed: 6.442964765s
Nov 17 01:08:48.256: INFO: Pod "exec-volume-test-dynamicpv-4zrn": Phase="Pending", Reason="", readiness=false. Elapsed: 8.552125759s
Nov 17 01:08:50.366: INFO: Pod "exec-volume-test-dynamicpv-4zrn": Phase="Pending", Reason="", readiness=false. Elapsed: 10.662152301s
... skipping 11 lines ...
Nov 17 01:09:15.688: INFO: Pod "exec-volume-test-dynamicpv-4zrn": Phase="Pending", Reason="", readiness=false. Elapsed: 35.984436748s
Nov 17 01:09:17.798: INFO: Pod "exec-volume-test-dynamicpv-4zrn": Phase="Pending", Reason="", readiness=false. Elapsed: 38.094441293s
Nov 17 01:09:19.908: INFO: Pod "exec-volume-test-dynamicpv-4zrn": Phase="Pending", Reason="", readiness=false. Elapsed: 40.203604947s
Nov 17 01:09:22.018: INFO: Pod "exec-volume-test-dynamicpv-4zrn": Phase="Pending", Reason="", readiness=false. Elapsed: 42.313554962s
Nov 17 01:09:24.128: INFO: Pod "exec-volume-test-dynamicpv-4zrn": Phase="Succeeded", Reason="", readiness=false. Elapsed: 44.423702209s
[1mSTEP[0m: Saw pod success
Nov 17 01:09:24.128: INFO: Pod "exec-volume-test-dynamicpv-4zrn" satisfied condition "Succeeded or Failed"
Nov 17 01:09:24.237: INFO: Trying to get logs from node k8s-agentpool1-18975173-vmss000002 pod exec-volume-test-dynamicpv-4zrn container exec-container-dynamicpv-4zrn: <nil>
[1mSTEP[0m: delete the pod
Nov 17 01:09:24.486: INFO: Waiting for pod exec-volume-test-dynamicpv-4zrn to disappear
Nov 17 01:09:24.594: INFO: Pod exec-volume-test-dynamicpv-4zrn no longer exists
[1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-4zrn
Nov 17 01:09:24.594: INFO: Deleting pod "exec-volume-test-dynamicpv-4zrn" in namespace "volume-4642"
... skipping 26 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should allow exec of files on the volume
    [90mtest/e2e/storage/testsuites/volumes.go:198[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should allow exec of files on the volume","total":34,"completed":8,"skipped":903,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:10:31.922: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 3 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:258[0m

    [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 16 lines ...

    [36mDistro debian doesn't support ntfs -- skipping[0m

    test/e2e/storage/framework/testsuite.go:127
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly][0m
  [37mtest/e2e/storage/testsuites/subpath.go:280[0m

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 17 01:08:32.800: INFO: >>> kubeConfig: /root/tmp1926242230/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename provisioning
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:280
Nov 17 01:08:33.564: INFO: Creating resource for dynamic PV
Nov 17 01:08:33.564: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass provisioning-1355-e2e-sccsqs8
[1mSTEP[0m: creating a claim
Nov 17 01:08:33.674: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 17 01:08:33.786: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comgp5qx] to have phase Bound
Nov 17 01:08:33.894: INFO: PersistentVolumeClaim test.csi.azure.comgp5qx found but phase is Pending instead of Bound.
Nov 17 01:08:36.017: INFO: PersistentVolumeClaim test.csi.azure.comgp5qx found but phase is Pending instead of Bound.
Nov 17 01:08:38.127: INFO: PersistentVolumeClaim test.csi.azure.comgp5qx found and phase=Bound (4.340538031s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-4drx
[1mSTEP[0m: Checking for subpath error in container status
Nov 17 01:09:54.680: INFO: Deleting pod "pod-subpath-test-dynamicpv-4drx" in namespace "provisioning-1355"
Nov 17 01:09:54.793: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-4drx" to be fully deleted
[1mSTEP[0m: Deleting pod
Nov 17 01:09:57.011: INFO: Deleting pod "pod-subpath-test-dynamicpv-4drx" in namespace "provisioning-1355"
[1mSTEP[0m: Deleting pvc
Nov 17 01:09:57.120: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comgp5qx"
... skipping 22 lines ...

[32m• [SLOW TEST:156.628 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:280[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]","total":27,"completed":7,"skipped":577,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:11:09.449: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 153 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents
    [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents","total":43,"completed":6,"skipped":408,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:11:34.991: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 72 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should be able to unmount after the subpath directory is deleted [LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:447[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [LinuxOnly]","total":36,"completed":8,"skipped":365,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly][0m
  [37mtest/e2e/storage/testsuites/subpath.go:242[0m

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 17 01:10:31.929: INFO: >>> kubeConfig: /root/tmp1926242230/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename provisioning
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if subpath directory is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:242
Nov 17 01:10:32.694: INFO: Creating resource for dynamic PV
Nov 17 01:10:32.694: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass provisioning-2125-e2e-scfj4f2
[1mSTEP[0m: creating a claim
Nov 17 01:10:32.804: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 17 01:10:32.917: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comx6pvg] to have phase Bound
Nov 17 01:10:33.026: INFO: PersistentVolumeClaim test.csi.azure.comx6pvg found but phase is Pending instead of Bound.
Nov 17 01:10:35.136: INFO: PersistentVolumeClaim test.csi.azure.comx6pvg found but phase is Pending instead of Bound.
Nov 17 01:10:37.247: INFO: PersistentVolumeClaim test.csi.azure.comx6pvg found and phase=Bound (4.329718545s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-wmmr
[1mSTEP[0m: Checking for subpath error in container status
Nov 17 01:11:03.797: INFO: Deleting pod "pod-subpath-test-dynamicpv-wmmr" in namespace "provisioning-2125"
Nov 17 01:11:03.909: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-wmmr" to be fully deleted
[1mSTEP[0m: Deleting pod
Nov 17 01:11:06.128: INFO: Deleting pod "pod-subpath-test-dynamicpv-wmmr" in namespace "provisioning-2125"
[1mSTEP[0m: Deleting pvc
Nov 17 01:11:06.238: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comx6pvg"
... skipping 22 lines ...

[32m• [SLOW TEST:106.634 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should fail if subpath directory is outside the volume [Slow][LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:242[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow][LinuxOnly]","total":34,"completed":9,"skipped":909,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:12:18.576: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 3 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Pre-provisioned PV (block volmode)] volumeMode
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/volumemode.go:299[0m

    [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 151 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    [90mtest/e2e/storage/testsuites/multivolume.go:378[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":26,"completed":7,"skipped":897,"failed":0}

[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] multiVolume [Slow][0m 
  [1mshould access to two volumes with different volume mode and retain data across pod recreation on the same node[0m
  [37mtest/e2e/storage/testsuites/multivolume.go:209[0m
... skipping 187 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with different volume mode and retain data across pod recreation on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:209[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":37,"completed":6,"skipped":817,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource][0m [0mvolume snapshot controller[0m [90m[0m 
  [1mshould check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)[0m
  [37mtest/e2e/storage/testsuites/snapshottable.go:278[0m
... skipping 18 lines ...
Nov 17 01:10:27.905: INFO: PersistentVolumeClaim test.csi.azure.comdp2cx found but phase is Pending instead of Bound.
Nov 17 01:10:30.015: INFO: PersistentVolumeClaim test.csi.azure.comdp2cx found but phase is Pending instead of Bound.
Nov 17 01:10:32.125: INFO: PersistentVolumeClaim test.csi.azure.comdp2cx found but phase is Pending instead of Bound.
Nov 17 01:10:34.234: INFO: PersistentVolumeClaim test.csi.azure.comdp2cx found and phase=Bound (6.437017153s)
[1mSTEP[0m: [init] starting a pod to use the claim
[1mSTEP[0m: [init] check pod success
Nov 17 01:10:34.669: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-q79v5" in namespace "snapshotting-7700" to be "Succeeded or Failed"
Nov 17 01:10:34.777: INFO: Pod "pvc-snapshottable-tester-q79v5": Phase="Pending", Reason="", readiness=false. Elapsed: 108.113188ms
Nov 17 01:10:36.886: INFO: Pod "pvc-snapshottable-tester-q79v5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217458643s
Nov 17 01:10:38.995: INFO: Pod "pvc-snapshottable-tester-q79v5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326839749s
Nov 17 01:10:41.107: INFO: Pod "pvc-snapshottable-tester-q79v5": Phase="Pending", Reason="", readiness=false. Elapsed: 6.438169516s
Nov 17 01:10:43.217: INFO: Pod "pvc-snapshottable-tester-q79v5": Phase="Pending", Reason="", readiness=false. Elapsed: 8.548016028s
Nov 17 01:10:45.327: INFO: Pod "pvc-snapshottable-tester-q79v5": Phase="Pending", Reason="", readiness=false. Elapsed: 10.658697263s
... skipping 2 lines ...
Nov 17 01:10:51.655: INFO: Pod "pvc-snapshottable-tester-q79v5": Phase="Pending", Reason="", readiness=false. Elapsed: 16.985996572s
Nov 17 01:10:53.764: INFO: Pod "pvc-snapshottable-tester-q79v5": Phase="Pending", Reason="", readiness=false. Elapsed: 19.094996544s
Nov 17 01:10:55.875: INFO: Pod "pvc-snapshottable-tester-q79v5": Phase="Pending", Reason="", readiness=false. Elapsed: 21.206567861s
Nov 17 01:10:57.984: INFO: Pod "pvc-snapshottable-tester-q79v5": Phase="Pending", Reason="", readiness=false. Elapsed: 23.315742923s
Nov 17 01:11:00.094: INFO: Pod "pvc-snapshottable-tester-q79v5": Phase="Succeeded", Reason="", readiness=false. Elapsed: 25.425065851s
[1mSTEP[0m: Saw pod success
Nov 17 01:11:00.094: INFO: Pod "pvc-snapshottable-tester-q79v5" satisfied condition "Succeeded or Failed"
[1mSTEP[0m: [init] checking the claim
Nov 17 01:11:00.203: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comdp2cx] to have phase Bound
Nov 17 01:11:00.311: INFO: PersistentVolumeClaim test.csi.azure.comdp2cx found and phase=Bound (108.257179ms)
[1mSTEP[0m: [init] checking the PV
[1mSTEP[0m: [init] deleting the pod
Nov 17 01:11:00.662: INFO: Pod pvc-snapshottable-tester-q79v5 has the following logs: 
... skipping 16 lines ...
Nov 17 01:11:14.735: INFO: received snapshotStatus map[boundVolumeSnapshotContentName:snapcontent-051df92f-aa27-4882-be91-136d0cede92b creationTime:2022-11-17T01:11:10Z readyToUse:true restoreSize:5Gi]
Nov 17 01:11:14.735: INFO: snapshotContentName snapcontent-051df92f-aa27-4882-be91-136d0cede92b
[1mSTEP[0m: checking the snapshot
[1mSTEP[0m: checking the SnapshotContent
[1mSTEP[0m: Modifying source data test
[1mSTEP[0m: modifying the data in the source PVC
Nov 17 01:11:15.172: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-xlwvq" in namespace "snapshotting-7700" to be "Succeeded or Failed"
Nov 17 01:11:15.281: INFO: Pod "pvc-snapshottable-data-tester-xlwvq": Phase="Pending", Reason="", readiness=false. Elapsed: 108.74672ms
Nov 17 01:11:17.391: INFO: Pod "pvc-snapshottable-data-tester-xlwvq": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218219121s
Nov 17 01:11:19.500: INFO: Pod "pvc-snapshottable-data-tester-xlwvq": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327913701s
Nov 17 01:11:21.610: INFO: Pod "pvc-snapshottable-data-tester-xlwvq": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437123439s
Nov 17 01:11:23.719: INFO: Pod "pvc-snapshottable-data-tester-xlwvq": Phase="Pending", Reason="", readiness=false. Elapsed: 8.545989406s
Nov 17 01:11:25.827: INFO: Pod "pvc-snapshottable-data-tester-xlwvq": Phase="Pending", Reason="", readiness=false. Elapsed: 10.654815216s
... skipping 12 lines ...
Nov 17 01:11:53.255: INFO: Pod "pvc-snapshottable-data-tester-xlwvq": Phase="Pending", Reason="", readiness=false. Elapsed: 38.082540311s
Nov 17 01:11:55.365: INFO: Pod "pvc-snapshottable-data-tester-xlwvq": Phase="Pending", Reason="", readiness=false. Elapsed: 40.192443455s
Nov 17 01:11:57.474: INFO: Pod "pvc-snapshottable-data-tester-xlwvq": Phase="Pending", Reason="", readiness=false. Elapsed: 42.301955281s
Nov 17 01:11:59.585: INFO: Pod "pvc-snapshottable-data-tester-xlwvq": Phase="Pending", Reason="", readiness=false. Elapsed: 44.412210202s
Nov 17 01:12:01.694: INFO: Pod "pvc-snapshottable-data-tester-xlwvq": Phase="Succeeded", Reason="", readiness=false. Elapsed: 46.521461988s
[1mSTEP[0m: Saw pod success
Nov 17 01:12:01.694: INFO: Pod "pvc-snapshottable-data-tester-xlwvq" satisfied condition "Succeeded or Failed"
Nov 17 01:12:01.915: INFO: Pod pvc-snapshottable-data-tester-xlwvq has the following logs: 
Nov 17 01:12:01.915: INFO: Deleting pod "pvc-snapshottable-data-tester-xlwvq" in namespace "snapshotting-7700"
Nov 17 01:12:02.027: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-xlwvq" to be fully deleted
[1mSTEP[0m: creating a pvc from the snapshot
[1mSTEP[0m: starting a pod to use the snapshot
Nov 17 01:12:26.576: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-qb0hkalz.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1926242230/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-7700 exec restored-pvc-tester-zphnj --namespace=snapshotting-7700 -- cat /mnt/test/data'
... skipping 47 lines ...
    [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m
      
      [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":27,"completed":9,"skipped":743,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:13:12.745: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 270 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:298[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":43,"completed":7,"skipped":445,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] volumeIO[0m 
  [1mshould write files of various sizes, verify size, validate content [Slow][0m
  [37mtest/e2e/storage/testsuites/volume_io.go:149[0m
... skipping 46 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] volumeIO
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should write files of various sizes, verify size, validate content [Slow]
    [90mtest/e2e/storage/testsuites/volume_io.go:149[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumeIO should write files of various sizes, verify size, validate content [Slow]","total":26,"completed":8,"skipped":900,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:13:54.310: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 248 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:138[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":36,"completed":9,"skipped":386,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy[0m 
  [1m(Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents[0m
  [37mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m
... skipping 91 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
    [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":43,"completed":8,"skipped":452,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral[0m 
  [1mshould support two pods which have the same volume definition[0m
  [37mtest/e2e/storage/testsuites/ephemeral.go:216[0m
... skipping 69 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support two pods which have the same volume definition
    [90mtest/e2e/storage/testsuites/ephemeral.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support two pods which have the same volume definition","total":27,"completed":8,"skipped":670,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:15:34.150: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 314 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    [90mtest/e2e/storage/testsuites/multivolume.go:168[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":27,"completed":10,"skipped":838,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:16:21.366: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 24 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Inline-volume (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:258[0m

    [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 40 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Inline-volume (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:242[0m

    [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 32 lines ...
Nov 17 01:15:37.257: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com4ljzx] to have phase Bound
Nov 17 01:15:37.366: INFO: PersistentVolumeClaim test.csi.azure.com4ljzx found but phase is Pending instead of Bound.
Nov 17 01:15:39.474: INFO: PersistentVolumeClaim test.csi.azure.com4ljzx found but phase is Pending instead of Bound.
Nov 17 01:15:41.584: INFO: PersistentVolumeClaim test.csi.azure.com4ljzx found and phase=Bound (4.327540721s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-zbxt
[1mSTEP[0m: Creating a pod to test subpath
Nov 17 01:15:41.913: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-zbxt" in namespace "provisioning-5056" to be "Succeeded or Failed"
Nov 17 01:15:42.023: INFO: Pod "pod-subpath-test-dynamicpv-zbxt": Phase="Pending", Reason="", readiness=false. Elapsed: 109.838228ms
Nov 17 01:15:44.133: INFO: Pod "pod-subpath-test-dynamicpv-zbxt": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21944798s
Nov 17 01:15:46.244: INFO: Pod "pod-subpath-test-dynamicpv-zbxt": Phase="Pending", Reason="", readiness=false. Elapsed: 4.330020855s
Nov 17 01:15:48.353: INFO: Pod "pod-subpath-test-dynamicpv-zbxt": Phase="Pending", Reason="", readiness=false. Elapsed: 6.439435046s
Nov 17 01:15:50.462: INFO: Pod "pod-subpath-test-dynamicpv-zbxt": Phase="Pending", Reason="", readiness=false. Elapsed: 8.548718849s
Nov 17 01:15:52.571: INFO: Pod "pod-subpath-test-dynamicpv-zbxt": Phase="Pending", Reason="", readiness=false. Elapsed: 10.657783014s
... skipping 6 lines ...
Nov 17 01:16:07.343: INFO: Pod "pod-subpath-test-dynamicpv-zbxt": Phase="Pending", Reason="", readiness=false. Elapsed: 25.429477883s
Nov 17 01:16:09.455: INFO: Pod "pod-subpath-test-dynamicpv-zbxt": Phase="Pending", Reason="", readiness=false. Elapsed: 27.54123733s
Nov 17 01:16:11.565: INFO: Pod "pod-subpath-test-dynamicpv-zbxt": Phase="Pending", Reason="", readiness=false. Elapsed: 29.651782646s
Nov 17 01:16:13.675: INFO: Pod "pod-subpath-test-dynamicpv-zbxt": Phase="Pending", Reason="", readiness=false. Elapsed: 31.761816845s
Nov 17 01:16:15.787: INFO: Pod "pod-subpath-test-dynamicpv-zbxt": Phase="Succeeded", Reason="", readiness=false. Elapsed: 33.873461009s
[1mSTEP[0m: Saw pod success
Nov 17 01:16:15.787: INFO: Pod "pod-subpath-test-dynamicpv-zbxt" satisfied condition "Succeeded or Failed"
Nov 17 01:16:15.896: INFO: Trying to get logs from node k8s-agentpool1-18975173-vmss000001 pod pod-subpath-test-dynamicpv-zbxt container test-container-volume-dynamicpv-zbxt: <nil>
[1mSTEP[0m: delete the pod
Nov 17 01:16:16.144: INFO: Waiting for pod pod-subpath-test-dynamicpv-zbxt to disappear
Nov 17 01:16:16.252: INFO: Pod pod-subpath-test-dynamicpv-zbxt no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-zbxt
Nov 17 01:16:16.252: INFO: Deleting pod "pod-subpath-test-dynamicpv-zbxt" in namespace "provisioning-5056"
... skipping 23 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support non-existent path
    [90mtest/e2e/storage/testsuites/subpath.go:196[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path","total":27,"completed":9,"skipped":879,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:16:58.135: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 72 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support multiple inline ephemeral volumes
    [90mtest/e2e/storage/testsuites/ephemeral.go:254[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support multiple inline ephemeral volumes","total":43,"completed":9,"skipped":499,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] provisioning[0m 
  [1mshould provision storage with pvc data source[0m
  [37mtest/e2e/storage/testsuites/provisioning.go:421[0m
... skipping 98 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] provisioning
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should provision storage with pvc data source
    [90mtest/e2e/storage/testsuites/provisioning.go:421[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with pvc data source","total":36,"completed":10,"skipped":489,"failed":0}
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:17:23.662: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
[AfterEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/framework/framework.go:188

... skipping 129 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    [90mtest/e2e/storage/testsuites/multivolume.go:378[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":37,"completed":7,"skipped":846,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:18:09.887: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 119 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should store data
    [90mtest/e2e/storage/testsuites/volumes.go:161[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data","total":43,"completed":10,"skipped":517,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:18:45.619: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 80 lines ...
Nov 17 01:17:24.712: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comq6bmp] to have phase Bound
Nov 17 01:17:24.823: INFO: PersistentVolumeClaim test.csi.azure.comq6bmp found but phase is Pending instead of Bound.
Nov 17 01:17:26.934: INFO: PersistentVolumeClaim test.csi.azure.comq6bmp found but phase is Pending instead of Bound.
Nov 17 01:17:29.045: INFO: PersistentVolumeClaim test.csi.azure.comq6bmp found and phase=Bound (4.3324429s)
[1mSTEP[0m: [init] starting a pod to use the claim
[1mSTEP[0m: [init] check pod success
Nov 17 01:17:29.490: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-lnvh9" in namespace "snapshotting-8406" to be "Succeeded or Failed"
Nov 17 01:17:29.601: INFO: Pod "pvc-snapshottable-tester-lnvh9": Phase="Pending", Reason="", readiness=false. Elapsed: 110.595502ms
Nov 17 01:17:31.712: INFO: Pod "pvc-snapshottable-tester-lnvh9": Phase="Pending", Reason="", readiness=false. Elapsed: 2.22191961s
Nov 17 01:17:33.825: INFO: Pod "pvc-snapshottable-tester-lnvh9": Phase="Pending", Reason="", readiness=false. Elapsed: 4.334656262s
Nov 17 01:17:35.936: INFO: Pod "pvc-snapshottable-tester-lnvh9": Phase="Pending", Reason="", readiness=false. Elapsed: 6.445491494s
Nov 17 01:17:38.047: INFO: Pod "pvc-snapshottable-tester-lnvh9": Phase="Pending", Reason="", readiness=false. Elapsed: 8.556372515s
Nov 17 01:17:40.158: INFO: Pod "pvc-snapshottable-tester-lnvh9": Phase="Pending", Reason="", readiness=false. Elapsed: 10.667918964s
Nov 17 01:17:42.270: INFO: Pod "pvc-snapshottable-tester-lnvh9": Phase="Pending", Reason="", readiness=false. Elapsed: 12.77967917s
Nov 17 01:17:44.382: INFO: Pod "pvc-snapshottable-tester-lnvh9": Phase="Pending", Reason="", readiness=false. Elapsed: 14.892089587s
Nov 17 01:17:46.494: INFO: Pod "pvc-snapshottable-tester-lnvh9": Phase="Pending", Reason="", readiness=false. Elapsed: 17.003794891s
Nov 17 01:17:48.605: INFO: Pod "pvc-snapshottable-tester-lnvh9": Phase="Pending", Reason="", readiness=false. Elapsed: 19.114518138s
Nov 17 01:17:50.716: INFO: Pod "pvc-snapshottable-tester-lnvh9": Phase="Pending", Reason="", readiness=false. Elapsed: 21.225689353s
Nov 17 01:17:52.828: INFO: Pod "pvc-snapshottable-tester-lnvh9": Phase="Succeeded", Reason="", readiness=false. Elapsed: 23.3376001s
[1mSTEP[0m: Saw pod success
Nov 17 01:17:52.828: INFO: Pod "pvc-snapshottable-tester-lnvh9" satisfied condition "Succeeded or Failed"
[1mSTEP[0m: [init] checking the claim
Nov 17 01:17:52.938: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comq6bmp] to have phase Bound
Nov 17 01:17:53.049: INFO: PersistentVolumeClaim test.csi.azure.comq6bmp found and phase=Bound (110.411071ms)
[1mSTEP[0m: [init] checking the PV
[1mSTEP[0m: [init] deleting the pod
Nov 17 01:17:53.404: INFO: Pod pvc-snapshottable-tester-lnvh9 has the following logs: 
... skipping 34 lines ...
Nov 17 01:18:04.083: INFO: WaitUntil finished successfully after 109.691194ms
[1mSTEP[0m: getting the snapshot and snapshot content
[1mSTEP[0m: checking the snapshot
[1mSTEP[0m: checking the SnapshotContent
[1mSTEP[0m: Modifying source data test
[1mSTEP[0m: modifying the data in the source PVC
Nov 17 01:18:04.636: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-9jpwx" in namespace "snapshotting-8406" to be "Succeeded or Failed"
Nov 17 01:18:04.747: INFO: Pod "pvc-snapshottable-data-tester-9jpwx": Phase="Pending", Reason="", readiness=false. Elapsed: 110.344007ms
Nov 17 01:18:06.859: INFO: Pod "pvc-snapshottable-data-tester-9jpwx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.222465568s
Nov 17 01:18:08.970: INFO: Pod "pvc-snapshottable-data-tester-9jpwx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.333414852s
Nov 17 01:18:11.082: INFO: Pod "pvc-snapshottable-data-tester-9jpwx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.445208013s
Nov 17 01:18:13.192: INFO: Pod "pvc-snapshottable-data-tester-9jpwx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.556049642s
Nov 17 01:18:15.304: INFO: Pod "pvc-snapshottable-data-tester-9jpwx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.66718594s
... skipping 11 lines ...
Nov 17 01:18:40.647: INFO: Pod "pvc-snapshottable-data-tester-9jpwx": Phase="Pending", Reason="", readiness=false. Elapsed: 36.010769522s
Nov 17 01:18:42.759: INFO: Pod "pvc-snapshottable-data-tester-9jpwx": Phase="Pending", Reason="", readiness=false. Elapsed: 38.122877953s
Nov 17 01:18:44.870: INFO: Pod "pvc-snapshottable-data-tester-9jpwx": Phase="Pending", Reason="", readiness=false. Elapsed: 40.233845849s
Nov 17 01:18:46.981: INFO: Pod "pvc-snapshottable-data-tester-9jpwx": Phase="Pending", Reason="", readiness=false. Elapsed: 42.345035069s
Nov 17 01:18:49.120: INFO: Pod "pvc-snapshottable-data-tester-9jpwx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 44.483262775s
[1mSTEP[0m: Saw pod success
Nov 17 01:18:49.120: INFO: Pod "pvc-snapshottable-data-tester-9jpwx" satisfied condition "Succeeded or Failed"
Nov 17 01:18:49.368: INFO: Pod pvc-snapshottable-data-tester-9jpwx has the following logs: 
Nov 17 01:18:49.368: INFO: Deleting pod "pvc-snapshottable-data-tester-9jpwx" in namespace "snapshotting-8406"
Nov 17 01:18:49.490: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-9jpwx" to be fully deleted
[1mSTEP[0m: creating a pvc from the snapshot
[1mSTEP[0m: starting a pod to use the snapshot
Nov 17 01:19:36.058: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-qb0hkalz.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1926242230/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-8406 exec restored-pvc-tester-s595c --namespace=snapshotting-8406 -- cat /mnt/test/data'
... skipping 47 lines ...
    [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m
      
      [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":36,"completed":11,"skipped":545,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] volumeMode[0m 
  [1mshould fail to use a volume in a pod with mismatched mode [Slow][0m
  [37mtest/e2e/storage/testsuites/volumemode.go:299[0m

[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 17 01:18:45.757: INFO: >>> kubeConfig: /root/tmp1926242230/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename volumemode
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299
Nov 17 01:18:46.519: INFO: Creating resource for dynamic PV
Nov 17 01:18:46.519: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass volumemode-246-e2e-sc9f2sn
[1mSTEP[0m: creating a claim
Nov 17 01:18:46.740: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comfjn9n] to have phase Bound
Nov 17 01:18:46.849: INFO: PersistentVolumeClaim test.csi.azure.comfjn9n found but phase is Pending instead of Bound.
Nov 17 01:18:48.958: INFO: PersistentVolumeClaim test.csi.azure.comfjn9n found but phase is Pending instead of Bound.
Nov 17 01:18:51.068: INFO: PersistentVolumeClaim test.csi.azure.comfjn9n found and phase=Bound (4.327611375s)
[1mSTEP[0m: Creating pod
[1mSTEP[0m: Waiting for the pod to fail
Nov 17 01:18:53.725: INFO: Deleting pod "pod-51f8d4f5-a003-4015-baae-4b8a7f21e3e0" in namespace "volumemode-246"
Nov 17 01:18:53.836: INFO: Wait up to 5m0s for pod "pod-51f8d4f5-a003-4015-baae-4b8a7f21e3e0" to be fully deleted
[1mSTEP[0m: Deleting pvc
Nov 17 01:18:56.055: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comfjn9n"
Nov 17 01:18:56.165: INFO: Waiting up to 5m0s for PersistentVolume pvc-203f03b1-e867-4e46-a476-242640f74a77 to get deleted
Nov 17 01:18:56.276: INFO: PersistentVolume pvc-203f03b1-e867-4e46-a476-242640f74a77 found and phase=Released (110.744552ms)
... skipping 32 lines ...

[32m• [SLOW TEST:143.929 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] volumeMode
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should fail to use a volume in a pod with mismatched mode [Slow]
    [90mtest/e2e/storage/testsuites/volumemode.go:299[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should fail to use a volume in a pod with mismatched mode [Slow]","total":43,"completed":11,"skipped":691,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] provisioning[0m 
  [1mshould provision storage with pvc data source in parallel [Slow][0m
  [37mtest/e2e/storage/testsuites/provisioning.go:459[0m
... skipping 326 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] provisioning
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should provision storage with pvc data source in parallel [Slow]
    [90mtest/e2e/storage/testsuites/provisioning.go:459[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with pvc data source in parallel [Slow]","total":37,"completed":8,"skipped":924,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning
... skipping 87 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should create read-only inline ephemeral volume
    [90mtest/e2e/storage/testsuites/ephemeral.go:175[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read-only inline ephemeral volume","total":36,"completed":12,"skipped":553,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:22:06.324: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 252 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:298[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":36,"completed":13,"skipped":596,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:24:10.598: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 37 lines ...
[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext4)] multiVolume [Slow][0m 
  [1mshould concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS][0m
  [37mtest/e2e/storage/testsuites/multivolume.go:323[0m

{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using file as subpath [Slow][LinuxOnly]","total":43,"completed":12,"skipped":699,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}
[BeforeEach] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 17 01:23:52.808: INFO: >>> kubeConfig: /root/tmp1926242230/kubeconfig/kubeconfig.westeurope.json
... skipping 103 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    [90mtest/e2e/storage/testsuites/multivolume.go:323[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":43,"completed":13,"skipped":699,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral[0m 
  [1mshould support two pods which have the same volume definition[0m
  [37mtest/e2e/storage/testsuites/ephemeral.go:216[0m
... skipping 53 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support two pods which have the same volume definition
    [90mtest/e2e/storage/testsuites/ephemeral.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support two pods which have the same volume definition","total":36,"completed":14,"skipped":683,"failed":0}

[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] multiVolume [Slow][0m 
  [1mshould concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS][0m
  [37mtest/e2e/storage/testsuites/multivolume.go:323[0m
... skipping 107 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    [90mtest/e2e/storage/testsuites/multivolume.go:323[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":43,"completed":14,"skipped":724,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] volumeIO
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:29:38.754: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 45 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/volumemode.go:299[0m

    [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 119 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should store data
    [90mtest/e2e/storage/testsuites/volumes.go:161[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes should store data","total":36,"completed":15,"skipped":685,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (xfs)][Slow] volumes[0m 
  [1mshould allow exec of files on the volume[0m
  [37mtest/e2e/storage/testsuites/volumes.go:198[0m
... skipping 17 lines ...
Nov 17 01:29:39.841: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comrrnmh] to have phase Bound
Nov 17 01:29:39.949: INFO: PersistentVolumeClaim test.csi.azure.comrrnmh found but phase is Pending instead of Bound.
Nov 17 01:29:42.059: INFO: PersistentVolumeClaim test.csi.azure.comrrnmh found but phase is Pending instead of Bound.
Nov 17 01:29:44.167: INFO: PersistentVolumeClaim test.csi.azure.comrrnmh found and phase=Bound (4.326303743s)
[1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-fqw5
[1mSTEP[0m: Creating a pod to test exec-volume-test
Nov 17 01:29:44.496: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-fqw5" in namespace "volume-2038" to be "Succeeded or Failed"
Nov 17 01:29:44.604: INFO: Pod "exec-volume-test-dynamicpv-fqw5": Phase="Pending", Reason="", readiness=false. Elapsed: 108.388539ms
Nov 17 01:29:46.713: INFO: Pod "exec-volume-test-dynamicpv-fqw5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216868972s
Nov 17 01:29:48.823: INFO: Pod "exec-volume-test-dynamicpv-fqw5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327030458s
Nov 17 01:29:50.932: INFO: Pod "exec-volume-test-dynamicpv-fqw5": Phase="Pending", Reason="", readiness=false. Elapsed: 6.435933156s
Nov 17 01:29:53.045: INFO: Pod "exec-volume-test-dynamicpv-fqw5": Phase="Pending", Reason="", readiness=false. Elapsed: 8.549229857s
Nov 17 01:29:55.155: INFO: Pod "exec-volume-test-dynamicpv-fqw5": Phase="Pending", Reason="", readiness=false. Elapsed: 10.659513667s
Nov 17 01:29:57.265: INFO: Pod "exec-volume-test-dynamicpv-fqw5": Phase="Pending", Reason="", readiness=false. Elapsed: 12.768850722s
Nov 17 01:29:59.374: INFO: Pod "exec-volume-test-dynamicpv-fqw5": Phase="Pending", Reason="", readiness=false. Elapsed: 14.878223344s
Nov 17 01:30:01.484: INFO: Pod "exec-volume-test-dynamicpv-fqw5": Phase="Pending", Reason="", readiness=false. Elapsed: 16.988211787s
Nov 17 01:30:03.594: INFO: Pod "exec-volume-test-dynamicpv-fqw5": Phase="Succeeded", Reason="", readiness=false. Elapsed: 19.098008247s
[1mSTEP[0m: Saw pod success
Nov 17 01:30:03.594: INFO: Pod "exec-volume-test-dynamicpv-fqw5" satisfied condition "Succeeded or Failed"
Nov 17 01:30:03.703: INFO: Trying to get logs from node k8s-agentpool1-18975173-vmss000001 pod exec-volume-test-dynamicpv-fqw5 container exec-container-dynamicpv-fqw5: <nil>
[1mSTEP[0m: delete the pod
Nov 17 01:30:03.953: INFO: Waiting for pod exec-volume-test-dynamicpv-fqw5 to disappear
Nov 17 01:30:04.061: INFO: Pod exec-volume-test-dynamicpv-fqw5 no longer exists
[1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-fqw5
Nov 17 01:30:04.061: INFO: Deleting pod "exec-volume-test-dynamicpv-fqw5" in namespace "volume-2038"
... skipping 19 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should allow exec of files on the volume
    [90mtest/e2e/storage/testsuites/volumes.go:198[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes should allow exec of files on the volume","total":43,"completed":15,"skipped":871,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volume-expand
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:30:35.614: INFO: Driver "test.csi.azure.com" does not support volume expansion - skipping
... skipping 38 lines ...
Nov 17 01:29:40.005: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com6klf4] to have phase Bound
Nov 17 01:29:40.116: INFO: PersistentVolumeClaim test.csi.azure.com6klf4 found but phase is Pending instead of Bound.
Nov 17 01:29:42.229: INFO: PersistentVolumeClaim test.csi.azure.com6klf4 found but phase is Pending instead of Bound.
Nov 17 01:29:44.342: INFO: PersistentVolumeClaim test.csi.azure.com6klf4 found and phase=Bound (4.336621134s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-ffqq
[1mSTEP[0m: Creating a pod to test atomic-volume-subpath
Nov 17 01:29:44.675: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-ffqq" in namespace "provisioning-192" to be "Succeeded or Failed"
Nov 17 01:29:44.786: INFO: Pod "pod-subpath-test-dynamicpv-ffqq": Phase="Pending", Reason="", readiness=false. Elapsed: 110.528479ms
Nov 17 01:29:46.898: INFO: Pod "pod-subpath-test-dynamicpv-ffqq": Phase="Pending", Reason="", readiness=false. Elapsed: 2.222636704s
Nov 17 01:29:49.010: INFO: Pod "pod-subpath-test-dynamicpv-ffqq": Phase="Pending", Reason="", readiness=false. Elapsed: 4.334640778s
Nov 17 01:29:51.122: INFO: Pod "pod-subpath-test-dynamicpv-ffqq": Phase="Pending", Reason="", readiness=false. Elapsed: 6.446326784s
Nov 17 01:29:53.235: INFO: Pod "pod-subpath-test-dynamicpv-ffqq": Phase="Pending", Reason="", readiness=false. Elapsed: 8.559130942s
Nov 17 01:29:55.348: INFO: Pod "pod-subpath-test-dynamicpv-ffqq": Phase="Pending", Reason="", readiness=false. Elapsed: 10.672104365s
... skipping 8 lines ...
Nov 17 01:30:14.358: INFO: Pod "pod-subpath-test-dynamicpv-ffqq": Phase="Running", Reason="", readiness=true. Elapsed: 29.68237477s
Nov 17 01:30:16.470: INFO: Pod "pod-subpath-test-dynamicpv-ffqq": Phase="Running", Reason="", readiness=true. Elapsed: 31.79429826s
Nov 17 01:30:18.581: INFO: Pod "pod-subpath-test-dynamicpv-ffqq": Phase="Running", Reason="", readiness=true. Elapsed: 33.9058065s
Nov 17 01:30:20.693: INFO: Pod "pod-subpath-test-dynamicpv-ffqq": Phase="Running", Reason="", readiness=true. Elapsed: 36.017415486s
Nov 17 01:30:22.805: INFO: Pod "pod-subpath-test-dynamicpv-ffqq": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.130029694s
[1mSTEP[0m: Saw pod success
Nov 17 01:30:22.806: INFO: Pod "pod-subpath-test-dynamicpv-ffqq" satisfied condition "Succeeded or Failed"
Nov 17 01:30:22.917: INFO: Trying to get logs from node k8s-agentpool1-18975173-vmss000000 pod pod-subpath-test-dynamicpv-ffqq container test-container-subpath-dynamicpv-ffqq: <nil>
[1mSTEP[0m: delete the pod
Nov 17 01:30:23.172: INFO: Waiting for pod pod-subpath-test-dynamicpv-ffqq to disappear
Nov 17 01:30:23.283: INFO: Pod pod-subpath-test-dynamicpv-ffqq no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-ffqq
Nov 17 01:30:23.283: INFO: Deleting pod "pod-subpath-test-dynamicpv-ffqq" in namespace "provisioning-192"
... skipping 28 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support file as subpath [LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:232[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support file as subpath [LinuxOnly]","total":36,"completed":16,"skipped":771,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] provisioning[0m 
  [1mshould provision storage with pvc data source[0m
  [37mtest/e2e/storage/testsuites/provisioning.go:421[0m
... skipping 97 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] provisioning
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should provision storage with pvc data source
    [90mtest/e2e/storage/testsuites/provisioning.go:421[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with pvc data source","total":43,"completed":16,"skipped":896,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:32:57.622: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 210 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with different volume mode and retain data across pod recreation on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:209[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":36,"completed":17,"skipped":785,"failed":0}

[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:33:37.385: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 144 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    [90mtest/e2e/storage/testsuites/multivolume.go:323[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":43,"completed":17,"skipped":935,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould support readOnly file specified in the volumeMount [LinuxOnly][0m
  [37mtest/e2e/storage/testsuites/subpath.go:382[0m
... skipping 17 lines ...
Nov 17 01:35:04.841: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.coms2m9d] to have phase Bound
Nov 17 01:35:04.949: INFO: PersistentVolumeClaim test.csi.azure.coms2m9d found but phase is Pending instead of Bound.
Nov 17 01:35:07.060: INFO: PersistentVolumeClaim test.csi.azure.coms2m9d found but phase is Pending instead of Bound.
Nov 17 01:35:09.170: INFO: PersistentVolumeClaim test.csi.azure.coms2m9d found and phase=Bound (4.329143627s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-2dlx
[1mSTEP[0m: Creating a pod to test subpath
Nov 17 01:35:09.502: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-2dlx" in namespace "provisioning-7668" to be "Succeeded or Failed"
Nov 17 01:35:09.611: INFO: Pod "pod-subpath-test-dynamicpv-2dlx": Phase="Pending", Reason="", readiness=false. Elapsed: 108.954921ms
Nov 17 01:35:11.720: INFO: Pod "pod-subpath-test-dynamicpv-2dlx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218018599s
Nov 17 01:35:13.829: INFO: Pod "pod-subpath-test-dynamicpv-2dlx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326438555s
Nov 17 01:35:15.937: INFO: Pod "pod-subpath-test-dynamicpv-2dlx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.435209102s
Nov 17 01:35:18.046: INFO: Pod "pod-subpath-test-dynamicpv-2dlx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.544357999s
Nov 17 01:35:20.155: INFO: Pod "pod-subpath-test-dynamicpv-2dlx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.653306463s
Nov 17 01:35:22.265: INFO: Pod "pod-subpath-test-dynamicpv-2dlx": Phase="Pending", Reason="", readiness=false. Elapsed: 12.762770467s
Nov 17 01:35:24.377: INFO: Pod "pod-subpath-test-dynamicpv-2dlx": Phase="Pending", Reason="", readiness=false. Elapsed: 14.874640181s
Nov 17 01:35:26.489: INFO: Pod "pod-subpath-test-dynamicpv-2dlx": Phase="Pending", Reason="", readiness=false. Elapsed: 16.987100951s
Nov 17 01:35:28.598: INFO: Pod "pod-subpath-test-dynamicpv-2dlx": Phase="Pending", Reason="", readiness=false. Elapsed: 19.096354377s
Nov 17 01:35:30.708: INFO: Pod "pod-subpath-test-dynamicpv-2dlx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 21.205841895s
[1mSTEP[0m: Saw pod success
Nov 17 01:35:30.708: INFO: Pod "pod-subpath-test-dynamicpv-2dlx" satisfied condition "Succeeded or Failed"
Nov 17 01:35:30.816: INFO: Trying to get logs from node k8s-agentpool1-18975173-vmss000001 pod pod-subpath-test-dynamicpv-2dlx container test-container-subpath-dynamicpv-2dlx: <nil>
[1mSTEP[0m: delete the pod
Nov 17 01:35:31.076: INFO: Waiting for pod pod-subpath-test-dynamicpv-2dlx to disappear
Nov 17 01:35:31.184: INFO: Pod pod-subpath-test-dynamicpv-2dlx no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-2dlx
Nov 17 01:35:31.184: INFO: Deleting pod "pod-subpath-test-dynamicpv-2dlx" in namespace "provisioning-7668"
... skipping 23 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support readOnly file specified in the volumeMount [LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:382[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly]","total":43,"completed":18,"skipped":942,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (ext4)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:36:13.105: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 143 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support restarting containers using directory as subpath [Slow]
    [90mtest/e2e/storage/testsuites/subpath.go:322[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using directory as subpath [Slow]","total":36,"completed":18,"skipped":825,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:36:26.553: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping
... skipping 84 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] volumeMode
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should not mount / map unused volumes in a pod [LinuxOnly]
    [90mtest/e2e/storage/testsuites/volumemode.go:354[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":36,"completed":19,"skipped":834,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 17 01:37:25.634: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 45 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Inline-volume (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:280[0m

    [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 118 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents
    [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents","total":43,"completed":19,"skipped":1011,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data"]}

[36mS[0m
[90m------------------------------[0m
Nov 17 01:37:41.657: INFO: Running AfterSuite actions on all nodes
Nov 17 01:37:41.657: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2
Nov 17 01:37:41.657: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2
... skipping 91 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single read-only volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:423[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":36,"completed":20,"skipped":976,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext4)] multiVolume [Slow][0m 
  [1mshould access to two volumes with different volume mode and retain data across pod recreation on the same node[0m
  [37mtest/e2e/storage/testsuites/multivolume.go:209[0m
... skipping 189 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with different volume mode and retain data across pod recreation on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:209[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":36,"completed":21,"skipped":981,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (filesystem volmode)] volumeMode[0m 
  [1mshould fail to use a volume in a pod with mismatched mode [Slow][0m
  [37mtest/e2e/storage/testsuites/volumemode.go:299[0m

[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 17 01:41:15.178: INFO: >>> kubeConfig: /root/tmp1926242230/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename volumemode
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299
Nov 17 01:41:15.948: INFO: Creating resource for dynamic PV
Nov 17 01:41:15.948: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass volumemode-4453-e2e-scg5zv6
[1mSTEP[0m: creating a claim
Nov 17 01:41:16.171: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comq8q4p] to have phase Bound
Nov 17 01:41:16.281: INFO: PersistentVolumeClaim test.csi.azure.comq8q4p found but phase is Pending instead of Bound.
Nov 17 01:41:18.393: INFO: PersistentVolumeClaim test.csi.azure.comq8q4p found but phase is Pending instead of Bound.
Nov 17 01:41:20.505: INFO: PersistentVolumeClaim test.csi.azure.comq8q4p found and phase=Bound (4.333291321s)
[1mSTEP[0m: Creating pod
[1mSTEP[0m: Waiting for the pod to fail
Nov 17 01:41:23.169: INFO: Deleting pod "pod-45bee9d4-dac0-4614-86ce-ed65821de5be" in namespace "volumemode-4453"
Nov 17 01:41:23.282: INFO: Wait up to 5m0s for pod "pod-45bee9d4-dac0-4614-86ce-ed65821de5be" to be fully deleted
[1mSTEP[0m: Deleting pvc
Nov 17 01:41:25.503: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comq8q4p"
Nov 17 01:41:25.615: INFO: Waiting up to 5m0s for PersistentVolume pvc-e362afcd-05b8-4d75-bc7c-90bfdd715d06 to get deleted
Nov 17 01:41:25.726: INFO: PersistentVolume pvc-e362afcd-05b8-4d75-bc7c-90bfdd715d06 found and phase=Released (110.205396ms)
... skipping 14 lines ...

[32m• [SLOW TEST:51.990 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should fail to use a volume in a pod with mismatched mode [Slow]
    [90mtest/e2e/storage/testsuites/volumemode.go:299[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should fail to use a volume in a pod with mismatched mode [Slow]","total":36,"completed":22,"skipped":1007,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
Nov 17 01:42:07.172: INFO: Running AfterSuite actions on all nodes
Nov 17 01:42:07.172: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2
Nov 17 01:42:07.173: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2
... skipping 15 lines ...
Nov 17 01:42:07.221: INFO: Running AfterSuite actions on node 1



[91m[1mSummarizing 4 Failures:[0m

[91m[1m[Fail] [0m[90mExternal Storage [Driver: test.csi.azure.com] [0m[0m[Testpattern: Dynamic PV (ext3)] volumes [0m[91m[1m[Measurement] should store data [0m
[37mtest/e2e/storage/testsuites/volumes.go:188[0m

[91m[1m[Fail] [0m[90mExternal Storage [Driver: test.csi.azure.com] [0m[0m[Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [0m[91m[1m[Measurement] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [0m
[37mtest/e2e/storage/testsuites/multivolume.go:696[0m

[91m[1m[Fail] [0m[90mExternal Storage [Driver: test.csi.azure.com] [0m[0m[Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [0m[91m[1m[Measurement] should access to two volumes with the same volume mode and retain data across pod recreation on different node [0m
[37mtest/e2e/storage/testsuites/multivolume.go:497[0m

[91m[1m[Fail] [0m[90mExternal Storage [Driver: test.csi.azure.com] [0m[0m[Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [0m[91m[1m[Measurement] should access to two volumes with different volume mode and retain data across pod recreation on different node [0m
[37mtest/e2e/storage/testsuites/multivolume.go:497[0m

[1m[91mRan 89 of 7227 Specs in 3391.045 seconds[0m
[1m[91mFAIL![0m -- [32m[1m85 Passed[0m | [91m[1m4 Failed[0m | [33m[1m0 Pending[0m | [36m[1m7138 Skipped[0m 

Ginkgo ran 1 suite in 56m34.778103868s
Test Suite Failed
+ print_logs
+ sed -i s/disk.csi.azure.com/test.csi.azure.com/g deploy/example/storageclass-azuredisk-csi.yaml
+ bash ./hack/verify-examples.sh linux azurepubliccloud ephemeral test
begin to create deployment examples ...
storageclass.storage.k8s.io/managed-csi created
Applying config "deploy/example/deployment.yaml"
... skipping 100 lines ...
I1117 00:45:25.268668       1 reflector.go:257] Listing and watching *v1beta2.AzVolumeAttachment from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117
I1117 00:45:25.268629       1 reflector.go:221] Starting reflector *v1beta2.AzVolume (30s) from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117
I1117 00:45:25.268765       1 reflector.go:257] Listing and watching *v1beta2.AzVolume from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117
I1117 00:45:25.369112       1 shared_informer.go:285] caches populated
I1117 00:45:25.369143       1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-c551e0323ad732edee8d00639b70d4fc25a9b407 e2e-test
I1117 00:45:25.369155       1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider
I1117 00:45:25.372235       1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I1117 00:45:25.372256       1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider
I1117 00:45:25.372262       1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I1117 00:45:25.372284       1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully
I1117 00:45:25.372918       1 azure_auth.go:245] Using AzurePublicCloud environment
I1117 00:45:25.372937       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I1117 00:45:25.372942       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
... skipping 29 lines ...
I1117 00:45:25.373263       1 azure_vmasclient.go:73] Azure AvailabilitySetsClient  (write ops) using rate limit config: QPS=100, bucket=1000
I1117 00:45:25.373308       1 azure.go:1039] attach/detach disk operation rate limiter configuration - Enabled: true QPS: 1.333000, Bucket: 240
I1117 00:45:25.373350       1 azuredisk_v2.go:215] disable UseInstanceMetadata for controller
I1117 00:45:25.373357       1 azuredisk_v2.go:225] DisableAvailabilitySetNodes for controller since current VMType is vmss
I1117 00:45:25.373364       1 azuredisk_v2.go:231] cloud: AzurePublicCloud, location: westeurope, rg: kubetest-qb0hkalz, VMType: vmss, PrimaryScaleSetName: k8s-agentpool1-18975173-vmss, PrimaryAvailabilitySetName: , DisableAvailabilitySetNodes: true
I1117 00:45:25.373370       1 skus.go:123] NewNodeInfo: Starting to populate node and disk sku information.
E1117 00:45:25.373381       1 azuredisk_v2.go:239] Failed to get node info. Error: NewNodeInfo: Failed to get instance type from Azure cloud provider, nodeName: , error: not a vmss instance
I1117 00:45:25.373474       1 mount_linux.go:208] Detected OS without systemd
I1117 00:45:25.373491       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I1117 00:45:25.373496       1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I1117 00:45:25.373499       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I1117 00:45:25.373503       1 driver.go:81] Enabling controller service capability: CLONE_VOLUME
I1117 00:45:25.373507       1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME
... skipping 260 lines ...
I1117 00:45:25.553338       1 shared_state.go:483]  "msg"="Storing pod kube-addon-manager-k8s-master-18975173-0 and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="1ff54b79-6611-11ed-bda2-000d3a2cfc6c"
I1117 00:45:25.553357       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/kube-addon-manager-k8s-master-18975173-0." "disk.csi.azure.com/request-id"="1ff54b79-6611-11ed-bda2-000d3a2cfc6c" "disk.csi/azure.com/pod-name"="kube-system/kube-addon-manager-k8s-master-18975173-0"
I1117 00:45:25.553378       1 shared_state.go:307]  "msg"="Getting requested volumes for pod (kube-system/kube-addon-manager-k8s-master-18975173-0)." "disk.csi.azure.com/request-id"="1ff54b79-6611-11ed-bda2-000d3a2cfc6c" "disk.csi/azure.com/pod-name"="kube-system/kube-addon-manager-k8s-master-18975173-0"
I1117 00:45:25.553390       1 pod.go:99]  "msg"="Pod kube-system/kube-addon-manager-k8s-master-18975173-0 has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="1ff54b79-6611-11ed-bda2-000d3a2cfc6c" "disk.csi/azure.com/pod-name"="kube-system/kube-addon-manager-k8s-master-18975173-0"
I1117 00:45:25.553406       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="1ff54b79-6611-11ed-bda2-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/kube-addon-manager-k8s-master-18975173-0" "latency"=49901
I1117 00:45:25.553418       1 shared_state.go:413]  "msg"="Adding pod csi-test-controller-6f4f8f4478-8jrqp to shared map with keyName kube-system/csi-test-controller-6f4f8f4478-8jrqp." "disk.csi.azure.com/request-id"="1ff54b79-6611-11ed-bda2-000d3a2cfc6c"
I1117 00:45:25.553671       1 shared_state.go:419]  "msg"="Pod spec of pod csi-test-controller-6f4f8f4478-8jrqp is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-hktbm VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-hktbm ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-hktbm ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-hktbm ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-hktbm ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-hktbm ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-c551e0323ad732edee8d00639b70d4fc25a9b407 Command:[] Args:[--v=5 --config=/etc/csi-test-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-test-controller-config ReadOnly:false MountPath:/etc/csi-test-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-hktbm ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc00093c4e8 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-agentpool1-18975173-vmss000001 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc00093c4f0} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc00093c4f8}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc00093c500 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc00093c504 PreemptionPolicy:0xc000a370c0 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil>}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-hktbm VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" "disk.csi.azure.com/request-id"="1ff54b79-6611-11ed-bda2-000d3a2cfc6c"
I1117 00:45:25.553706       1 shared_state.go:457]  "msg"="Pod csi-test-controller-6f4f8f4478-8jrqp: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="1ff54b79-6611-11ed-bda2-000d3a2cfc6c"
I1117 00:45:25.553725       1 shared_state.go:457]  "msg"="Pod csi-test-controller-6f4f8f4478-8jrqp: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="1ff54b79-6611-11ed-bda2-000d3a2cfc6c"
I1117 00:45:25.553743       1 shared_state.go:457]  "msg"="Pod csi-test-controller-6f4f8f4478-8jrqp: Skipping Volume {csi-test-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="1ff54b79-6611-11ed-bda2-000d3a2cfc6c"
I1117 00:45:25.553770       1 shared_state.go:457]  "msg"="Pod csi-test-controller-6f4f8f4478-8jrqp: Skipping Volume {kube-api-access-hktbm {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="1ff54b79-6611-11ed-bda2-000d3a2cfc6c"
I1117 00:45:25.553782       1 shared_state.go:483]  "msg"="Storing pod csi-test-controller-6f4f8f4478-8jrqp and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="1ff54b79-6611-11ed-bda2-000d3a2cfc6c"
I1117 00:45:25.553795       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-test-controller-6f4f8f4478-8jrqp." "disk.csi.azure.com/request-id"="1ff54b79-6611-11ed-bda2-000d3a2cfc6c" "disk.csi/azure.com/pod-name"="kube-system/csi-test-controller-6f4f8f4478-8jrqp"
... skipping 31815 lines ...




rmers/externalversions/factory.go:117: forcing resync
I1117 00:46:18.956242       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:46:18.956340       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:46:18.956417       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:46:19.081320       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:46:20.940627       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:46:20.940718       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="2c161ab5-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-dc7454b5-b461-4c40-a7f5-5acfb48c26fb" "latency"=35092704784
I1117 00:46:20.940753       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2c161ab5-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=35093064590
I1117 00:46:20.944031       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:46:20.944089       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="2c161bcb-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6" "latency"=35096052143
I1117 00:46:20.944123       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2c161bcb-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=35096407950
I1117 00:46:21.427260       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 00:46:21.427283       1 utils.go:79] GRPC request: {}
I1117 00:46:21.427334       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 00:46:21.431565       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 83 lines ...
I1117 00:46:30.677652       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-dc7454b5-b461-4c40-a7f5-5acfb48c26fb/globalmount","target_path":"/var/lib/kubelet/pods/79e9ba07-5d49-4498-9ce0-62e77ccbb98a/volumes/kubernetes.io~csi/pvc-dc7454b5-b461-4c40-a7f5-5acfb48c26fb/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-dc7454b5-b461-4c40-a7f5-5acfb48c26fb","csi.storage.k8s.io/pvc/name":"test.csi.azure.comtq4fn","csi.storage.k8s.io/pvc/namespace":"multivolume-8097","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-dc7454b5-b461-4c40-a7f5-5acfb48c26fb"}
I1117 00:46:30.677829       1 nodeserver_v2.go:348] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-dc7454b5-b461-4c40-a7f5-5acfb48c26fb/globalmount at /var/lib/kubelet/pods/79e9ba07-5d49-4498-9ce0-62e77ccbb98a/volumes/kubernetes.io~csi/pvc-dc7454b5-b461-4c40-a7f5-5acfb48c26fb/mount
I1117 00:46:30.677870       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-dc7454b5-b461-4c40-a7f5-5acfb48c26fb/globalmount /var/lib/kubelet/pods/79e9ba07-5d49-4498-9ce0-62e77ccbb98a/volumes/kubernetes.io~csi/pvc-dc7454b5-b461-4c40-a7f5-5acfb48c26fb/mount)
I1117 00:46:30.678746       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-dc7454b5-b461-4c40-a7f5-5acfb48c26fb/globalmount /var/lib/kubelet/pods/79e9ba07-5d49-4498-9ce0-62e77ccbb98a/volumes/kubernetes.io~csi/pvc-dc7454b5-b461-4c40-a7f5-5acfb48c26fb/mount)
I1117 00:46:30.679519       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-dc7454b5-b461-4c40-a7f5-5acfb48c26fb/globalmount at /var/lib/kubelet/pods/79e9ba07-5d49-4498-9ce0-62e77ccbb98a/volumes/kubernetes.io~csi/pvc-dc7454b5-b461-4c40-a7f5-5acfb48c26fb/mount successfully
I1117 00:46:30.679537       1 utils.go:85] GRPC response: {}
I1117 00:46:31.178382       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:46:31.178446       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="414c3405-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c30fdae7-5de7-469b-bb08-7e06de596848" "latency"=9743816599
I1117 00:46:31.178484       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="414c3405-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9744097804
I1117 00:46:31.801920       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1117 00:46:31.801964       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1117 00:46:31.801993       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c30fdae7-5de7-469b-bb08-7e06de596848/globalmount with mount options([])
I1117 00:46:31.802005       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 117 lines ...
I1117 00:47:50.354225       1 utils.go:79] GRPC request: {}
I1117 00:47:50.354257       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 00:47:50.354928       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 00:47:50.354942       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5","csi.storage.k8s.io/pvc/name":"test.csi.azure.compmvcc","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-7072","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5"}
I1117 00:47:50.355177       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="764c6cc2-6611-11ed-b53d-000d3a2cfc6c"
I1117 00:47:50.355232       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 00:48:00.170611       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:48:00.170682       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="764c6cc2-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5" "latency"=9815391823
I1117 00:48:00.170713       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="764c6cc2-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9815629427
I1117 00:48:00.766126       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 00:48:00.766170       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 00:48:00.766201       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5/globalmount with mount options([])
I1117 00:48:00.766218       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 106 lines ...
I1117 00:48:20.211175       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5/globalmount","target_path":"/var/lib/kubelet/pods/9cb339eb-2dde-439d-8c11-19ad0129c220/volumes/kubernetes.io~csi/pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5","csi.storage.k8s.io/pvc/name":"test.csi.azure.compmvcc","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-7072","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5"}
I1117 00:48:20.211309       1 nodeserver_v2.go:348] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5/globalmount at /var/lib/kubelet/pods/9cb339eb-2dde-439d-8c11-19ad0129c220/volumes/kubernetes.io~csi/pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5/mount
I1117 00:48:20.211326       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5/globalmount /var/lib/kubelet/pods/9cb339eb-2dde-439d-8c11-19ad0129c220/volumes/kubernetes.io~csi/pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5/mount)
I1117 00:48:20.212105       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5/globalmount /var/lib/kubelet/pods/9cb339eb-2dde-439d-8c11-19ad0129c220/volumes/kubernetes.io~csi/pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5/mount)
I1117 00:48:20.212828       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5/globalmount at /var/lib/kubelet/pods/9cb339eb-2dde-439d-8c11-19ad0129c220/volumes/kubernetes.io~csi/pvc-0113bebf-770c-433c-8d96-ca8f9286d1d5/mount successfully
I1117 00:48:20.212840       1 utils.go:85] GRPC response: {}
I1117 00:48:22.960023       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:48:22.960090       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="87b672a3-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d108a6fd-cb53-4512-9cb8-46d2c133c8d8" "latency"=3388434374
I1117 00:48:22.960123       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="87b672a3-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3388940482
I1117 00:48:23.559895       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1117 00:48:23.559941       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 00:48:23.559969       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d108a6fd-cb53-4512-9cb8-46d2c133c8d8/globalmount with mount options([])
I1117 00:48:23.559979       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 105 lines ...
I1117 00:49:33.686669       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:49:33.686694       1 utils.go:79] GRPC request: {}
I1117 00:49:33.686734       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:49:37.181247       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:49:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="b5f8cc92-6611-11ed-b53d-000d3a2cfc6c"
I1117 00:49:37.189901       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 8 milliseconds
I1117 00:49:37.190265       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="b5f8cc92-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9043160
I1117 00:49:37.963677       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:49:37.963739       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="ac366c26-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d04cd0fb-eedd-4a63-b56b-3d0cb49a3786" "latency"=17155654998
I1117 00:49:37.963764       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ac366c26-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=17155913003
I1117 00:49:38.556400       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 00:49:38.556441       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 00:49:38.556465       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d04cd0fb-eedd-4a63-b56b-3d0cb49a3786/globalmount with mount options([])
I1117 00:49:38.556475       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 103 lines ...
I1117 00:50:33.686772       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:50:37.180664       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:50:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="d9bbfad6-6611-11ed-b53d-000d3a2cfc6c"
I1117 00:50:37.187869       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 6 milliseconds
I1117 00:50:37.188121       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="d9bbfad6-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7502631
I1117 00:50:37.956779       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 184 items received
I1117 00:50:37.959468       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=5534&timeout=7m8s&timeoutSeconds=428&watch=true 200 OK in 2 milliseconds
I1117 00:50:48.693731       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:50:48.693777       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="d0b71e7f-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-baea0f45-5ca1-4e5c-8165-915e6848898b" "latency"=26644324133
I1117 00:50:48.693798       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d0b71e7f-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=26644544137
I1117 00:50:48.959015       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:50:48.960160       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:50:48.961307       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:50:49.086486       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 55 lines ...
I1117 00:50:58.970347       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-baea0f45-5ca1-4e5c-8165-915e6848898b/globalmount
I1117 00:50:58.970376       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-baea0f45-5ca1-4e5c-8165-915e6848898b/globalmount" is a mountpoint, unmounting
I1117 00:50:58.970395       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-baea0f45-5ca1-4e5c-8165-915e6848898b/globalmount
W1117 00:50:58.982348       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-baea0f45-5ca1-4e5c-8165-915e6848898b/globalmount" is not a mountpoint, deleting
I1117 00:50:58.982426       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-baea0f45-5ca1-4e5c-8165-915e6848898b/globalmount successfully
I1117 00:50:58.982437       1 utils.go:85] GRPC response: {}
I1117 00:50:59.012652       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:50:59.012709       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="e343f812-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7f4e0235-e877-4a4a-ad38-865dac20f532" "latency"=5841169335
I1117 00:50:59.012735       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e343f812-6611-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5841417738
I1117 00:50:59.617245       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1117 00:50:59.617294       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 00:50:59.617308       1 utils.go:85] GRPC response: {}
I1117 00:50:59.621222       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 112 lines ...
I1117 00:52:07.180843       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:52:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="0f60eb1c-6612-11ed-b53d-000d3a2cfc6c"
I1117 00:52:07.191043       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 10 milliseconds
I1117 00:52:07.191706       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="0f60eb1c-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10896692
I1117 00:52:18.961220       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:52:18.961389       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:52:18.963384       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:52:18.963468       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:52:18.963489       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:52:19.087853       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:52:33.686708       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:52:33.686730       1 utils.go:79] GRPC request: {}
I1117 00:52:33.686776       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:52:37.181164       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:52:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="21429aa5-6612-11ed-b53d-000d3a2cfc6c"
I1117 00:52:37.188220       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 6 milliseconds
I1117 00:52:37.188585       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="21429aa5-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7458131
I1117 00:52:48.962123       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:52:48.962162       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:52:48.963449       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:52:48.963529       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:52:48.963561       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:52:49.088806       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:53:02.954459       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 154 items received
I1117 00:53:02.957488       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=6416&timeout=9m23s&timeoutSeconds=563&watch=true 200 OK in 2 milliseconds
I1117 00:53:03.686529       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:53:03.686556       1 utils.go:79] GRPC request: {}
I1117 00:53:03.686594       1 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 2 lines ...
I1117 00:53:07.189449       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="33244df6-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7909640
I1117 00:53:12.081600       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 27 items received
I1117 00:53:12.087437       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-18975173-vmss000001&resourceVersion=6759&timeout=7m45s&timeoutSeconds=465&watch=true 200 OK in 5 milliseconds
I1117 00:53:18.962423       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:53:18.962449       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:53:18.963608       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:53:18.963692       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:53:18.963725       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:53:19.089940       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:53:33.687070       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:53:33.687099       1 utils.go:79] GRPC request: {}
I1117 00:53:33.687147       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:53:37.181104       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:53:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="4505ddf7-6612-11ed-b53d-000d3a2cfc6c"
I1117 00:53:37.192019       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 10 milliseconds
I1117 00:53:37.192218       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="4505ddf7-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11153296
I1117 00:53:48.962587       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:53:48.962701       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:53:48.963747       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:53:48.963836       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:53:48.963879       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:53:49.090055       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:03.686655       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:54:03.686675       1 utils.go:79] GRPC request: {}
I1117 00:54:03.686714       1 utils.go:85] GRPC response: {"ready":{"value":true}}
E1117 00:54:05.356251       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="0e4b5014-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-81485d95-fe0b-49b7-aef7-7edeb434a4cc" "latency"=119994493542
E1117 00:54:05.356287       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="0e4b57ff-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e0696c48-3947-43cf-a288-522461947b6b" "latency"=119994312139
E1117 00:54:05.356323       1 crdprovisioner.go:699]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0e4b5014-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119994835947
E1117 00:54:05.356338       1 crdprovisioner.go:699]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0e4b57ff-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119994648645
E1117 00:54:05.356345       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-81485d95-fe0b-49b7-aef7-7edeb434a4cc) to be attached to node (k8s-agentpool1-18975173-vmss000001): context canceled
E1117 00:54:05.356358       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-e0696c48-3947-43cf-a288-522461947b6b) to be attached to node (k8s-agentpool1-18975173-vmss000001): context deadline exceeded
I1117 00:54:05.866831       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 00:54:05.866885       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 00:54:05.866860       1 utils.go:79] GRPC request: {}
I1117 00:54:05.866901       1 utils.go:79] GRPC request: {}
I1117 00:54:05.866915       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 00:54:05.866955       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 20 lines ...
I1117 00:54:07.180181       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:54:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="56e75d5d-6612-11ed-b53d-000d3a2cfc6c"
I1117 00:54:07.189795       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 9 milliseconds
I1117 00:54:07.190023       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="56e75d5d-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9883372
I1117 00:54:18.963382       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:18.963502       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:18.964580       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:18.964711       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:54:18.964762       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:54:19.090939       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:33.687148       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:54:33.687177       1 utils.go:79] GRPC request: {}
I1117 00:54:33.687218       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:54:37.180594       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:54:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="68c9104f-6612-11ed-b53d-000d3a2cfc6c"
I1117 00:54:37.190877       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 10 milliseconds
I1117 00:54:37.191113       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="68c9104f-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10550386
I1117 00:54:48.963936       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:48.963975       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:48.965115       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:48.965196       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:54:48.965223       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:54:49.091461       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:50.954446       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 91 items received
I1117 00:54:50.958417       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=7142&timeout=8m36s&timeoutSeconds=516&watch=true 200 OK in 3 milliseconds
I1117 00:55:03.686729       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:55:03.686751       1 utils.go:79] GRPC request: {}
I1117 00:55:03.686799       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:55:07.180178       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:55:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="7aaaa39a-6612-11ed-b53d-000d3a2cfc6c"
I1117 00:55:07.190208       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 9 milliseconds
I1117 00:55:07.190468       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="7aaaa39a-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10326382
I1117 00:55:18.965009       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:55:18.965097       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:55:18.966234       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:55:18.966316       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:55:18.966370       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:55:19.092561       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:55:33.687494       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:55:33.687519       1 utils.go:79] GRPC request: {}
I1117 00:55:33.687567       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:55:37.180211       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:55:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="8c8c46f0-6612-11ed-b53d-000d3a2cfc6c"
I1117 00:55:37.188922       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 8 milliseconds
I1117 00:55:37.189179       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="8c8c46f0-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9012559
I1117 00:55:48.965184       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:55:48.965266       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:55:48.966548       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:55:48.966622       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:55:48.966657       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:55:49.092786       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:55:58.936105       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 45 items received
I1117 00:55:58.941347       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=7713&timeout=7m7s&timeoutSeconds=427&watch=true 200 OK in 5 milliseconds
I1117 00:56:03.687627       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:56:03.687653       1 utils.go:79] GRPC request: {}
I1117 00:56:03.687701       1 utils.go:85] GRPC response: {"ready":{"value":true}}
E1117 00:56:05.866707       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="561fd71f-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-81485d95-fe0b-49b7-aef7-7edeb434a4cc" "latency"=119993946048
E1117 00:56:05.866776       1 crdprovisioner.go:699]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="561fd71f-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119994234753
E1117 00:56:05.866796       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-81485d95-fe0b-49b7-aef7-7edeb434a4cc) to be attached to node (k8s-agentpool1-18975173-vmss000001): context canceled
E1117 00:56:05.866723       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="561fc274-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e0696c48-3947-43cf-a288-522461947b6b" "latency"=119994406956
E1117 00:56:05.866849       1 crdprovisioner.go:699]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="561fc274-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119994841464
E1117 00:56:05.866871       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-e0696c48-3947-43cf-a288-522461947b6b) to be attached to node (k8s-agentpool1-18975173-vmss000001): context deadline exceeded
I1117 00:56:06.937334       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 00:56:06.937359       1 utils.go:79] GRPC request: {}
I1117 00:56:06.937416       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 00:56:06.937436       1 utils.go:79] GRPC request: {}
I1117 00:56:06.937403       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 00:56:06.937483       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 18 lines ...
I1117 00:56:06.943230       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-e0696c48-3947-43cf-a288-522461947b6b." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9e49bfd8-6612-11ed-b53d-000d3a2cfc6c"
I1117 00:56:06.943277       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e0696c48-3947-43cf-a288-522461947b6b-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 00:56:07.180468       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:56:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="9e6df4e0-6612-11ed-b53d-000d3a2cfc6c"
I1117 00:56:07.185235       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 4 milliseconds
I1117 00:56:07.185916       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="9e6df4e0-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5482397
I1117 00:56:18.968605       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:56:18.968721       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:56:18.968742       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:56:18.968754       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:56:18.968772       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:56:19.093070       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:56:33.686257       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:56:33.686276       1 utils.go:79] GRPC request: {}
I1117 00:56:33.686319       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:56:37.180817       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:56:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="b04fa4f1-6612-11ed-b53d-000d3a2cfc6c"
I1117 00:56:37.189742       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 8 milliseconds
I1117 00:56:37.189988       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="b04fa4f1-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9214164
I1117 00:56:48.978435       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:56:48.978509       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:56:48.978587       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:56:48.978604       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:56:48.978615       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:56:49.093879       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:56:54.955976       1 conditionwatcher.go:187] condition result: succeeded: false, error: rpc error: code = Unknown desc = context deadline exceeded
E1117 00:56:54.956087       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [rpc error: code = Unknown desc = context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="9e49bb46-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-81485d95-fe0b-49b7-aef7-7edeb434a4cc" "latency"=48012789546
E1117 00:56:54.956131       1 crdprovisioner.go:699]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [rpc error: code = Unknown desc = context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9e49bb46-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=48013099252
E1117 00:56:54.956157       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-81485d95-fe0b-49b7-aef7-7edeb434a4cc) to be attached to node (k8s-agentpool1-18975173-vmss000001): rpc error: code = Unknown desc = context deadline exceeded
I1117 00:56:54.963808       1 conditionwatcher.go:187] condition result: succeeded: false, error: rpc error: code = Unknown desc = context deadline exceeded
E1117 00:56:54.963882       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [rpc error: code = Unknown desc = context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="9e49bfd8-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e0696c48-3947-43cf-a288-522461947b6b" "latency"=48020542732
E1117 00:56:54.963923       1 crdprovisioner.go:699]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [rpc error: code = Unknown desc = context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9e49bfd8-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=48020773336
E1117 00:56:54.963952       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-e0696c48-3947-43cf-a288-522461947b6b) to be attached to node (k8s-agentpool1-18975173-vmss000001): rpc error: code = Unknown desc = context deadline exceeded
I1117 00:56:56.124808       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 00:56:56.124836       1 utils.go:79] GRPC request: {}
I1117 00:56:56.124874       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 00:56:56.128378       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 00:56:56.128396       1 utils.go:79] GRPC request: {}
I1117 00:56:56.128422       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 31 lines ...
I1117 00:56:57.049804       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-81485d95-fe0b-49b7-aef7-7edeb434a4cc-k8s-agentpool1-18975173-vmss000001-attachment/status 200 OK in 12 milliseconds
I1117 00:56:57.050101       1 azure_disk_utils.go:1084]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/request-id"="bc2561f0-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "latency"=13478239
I1117 00:56:57.050123       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-81485d95-fe0b-49b7-aef7-7edeb434a4cc-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 00:56:57.056236       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-e0696c48-3947-43cf-a288-522461947b6b-k8s-agentpool1-18975173-vmss000001-attachment/status 200 OK in 19 milliseconds
I1117 00:56:57.058063       1 azure_disk_utils.go:1084]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/request-id"="bc256051-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "latency"=21462280
I1117 00:56:57.058083       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e0696c48-3947-43cf-a288-522461947b6b-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 00:56:57.095561       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:56:57.103951       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:56:57.191996       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:56:57.194684       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:56:57.202068       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:56:57.202135       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="bc256051-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e0696c48-3947-43cf-a288-522461947b6b" "latency"=143989746
I1117 00:56:57.202167       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bc256051-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=165750131
I1117 00:56:57.205508       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:56:57.205552       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="bc2561f0-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-81485d95-fe0b-49b7-aef7-7edeb434a4cc" "latency"=155377048
I1117 00:56:57.205580       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bc2561f0-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=169124591
I1117 00:56:57.802302       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1117 00:56:57.802364       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 00:56:57.802401       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-81485d95-fe0b-49b7-aef7-7edeb434a4cc/globalmount with mount options([])
I1117 00:56:57.802412       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 114 lines ...
I1117 00:57:07.180470       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:57:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="c2313a0c-6612-11ed-b53d-000d3a2cfc6c"
I1117 00:57:07.186807       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 6 milliseconds
I1117 00:57:07.187010       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="c2313a0c-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6596417
I1117 00:57:18.984669       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:57:18.984732       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:57:18.984757       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:57:18.984832       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:57:18.984854       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:57:19.094011       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:57:21.621702       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:57:21.621756       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="bd67e2d6-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-63911cdd-1095-4b04-a2e8-d47057864824" "latency"=22471503548
I1117 00:57:21.621782       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bd67e2d6-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=22471770452
I1117 00:57:21.630784       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:57:21.630843       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="bb9b0d6d-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6a8cbafd-b969-4fb3-bc73-a44bd73dfa61" "latency"=25500712898
I1117 00:57:21.630875       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bb9b0d6d-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=25500980404
I1117 00:57:22.237384       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdf under /dev/disk/azure/scsi1/
I1117 00:57:22.237439       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 00:57:22.237385       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sde under /dev/disk/azure/scsi1/
I1117 00:57:22.237468       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-63911cdd-1095-4b04-a2e8-d47057864824/globalmount with mount options([])
... skipping 127 lines ...
I1117 00:58:24.347104       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-ea276d92-55ec-4e0d-9529-3739d4188f24","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ea276d92-55ec-4e0d-9529-3739d4188f24","csi.storage.k8s.io/pvc/name":"test.csi.azure.coml6h7p","csi.storage.k8s.io/pvc/namespace":"volume-1924","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-ea276d92-55ec-4e0d-9529-3739d4188f24"}
I1117 00:58:24.347364       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-ea276d92-55ec-4e0d-9529-3739d4188f24." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f02ff482-6612-11ed-b53d-000d3a2cfc6c"
I1117 00:58:24.347416       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-ea276d92-55ec-4e0d-9529-3739d4188f24-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 00:58:33.686287       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:58:33.686312       1 utils.go:79] GRPC request: {}
I1117 00:58:33.686357       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:58:33.867195       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:58:33.867267       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="f02ff482-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ea276d92-55ec-4e0d-9529-3739d4188f24" "latency"=9519779508
I1117 00:58:33.867298       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f02ff482-6612-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9520029213
I1117 00:58:35.689637       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 00:58:35.689691       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 00:58:35.689705       1 utils.go:85] GRPC response: {}
I1117 00:58:35.697253       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 67 lines ...
I1117 00:58:55.635628       1 nodeprovisioner.go:164] EnsureBlockTargetReady [block]: making target file /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-ea276d92-55ec-4e0d-9529-3739d4188f24/5405b0a1-4180-493a-af9b-c77a0c1ac108
I1117 00:58:55.635686       1 nodeserver_v2.go:348] NodePublishVolume: mounting /dev/disk/azure/scsi1/lun0 at /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-ea276d92-55ec-4e0d-9529-3739d4188f24/5405b0a1-4180-493a-af9b-c77a0c1ac108
I1117 00:58:55.635707       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-ea276d92-55ec-4e0d-9529-3739d4188f24/5405b0a1-4180-493a-af9b-c77a0c1ac108)
I1117 00:58:55.636710       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-ea276d92-55ec-4e0d-9529-3739d4188f24/5405b0a1-4180-493a-af9b-c77a0c1ac108)
I1117 00:58:55.637569       1 nodeserver_v2.go:353] NodePublishVolume: mount /dev/disk/azure/scsi1/lun0 at /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-ea276d92-55ec-4e0d-9529-3739d4188f24/5405b0a1-4180-493a-af9b-c77a0c1ac108 successfully
I1117 00:58:55.637589       1 utils.go:85] GRPC response: {}
I1117 00:58:59.390143       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:58:59.390239       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="02174523-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9485aec4-9d97-4401-82a5-71cc411343a0" "latency"=5005555920
I1117 00:58:59.390273       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="02174523-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5005788924
I1117 00:59:00.012181       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1117 00:59:00.012236       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 00:59:00.012277       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9485aec4-9d97-4401-82a5-71cc411343a0/globalmount with mount options([])
I1117 00:59:00.012292       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 100 lines ...
I1117 01:00:07.180874       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:00:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="2d7b1bce-6613-11ed-b53d-000d3a2cfc6c"
I1117 01:00:07.190700       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 9 milliseconds
I1117 01:00:07.190910       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="2d7b1bce-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10093780
I1117 01:00:18.993941       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:00:18.993991       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:00:18.994027       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:00:18.994085       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:00:19.098431       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:00:22.112638       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:00:22.112704       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="24b89e4b-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8b57b53e-4295-4e7b-a0e8-830f441afbef" "latency"=29628067488
I1117 01:00:22.112733       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="24b89e4b-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29628305992
I1117 01:00:23.897412       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:00:23.897462       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:00:23.897490       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8b57b53e-4295-4e7b-a0e8-830f441afbef/globalmount with mount options([])
I1117 01:00:23.897499       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 56 lines ...
I1117 01:00:33.686772       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:00:33.686796       1 utils.go:79] GRPC request: {}
I1117 01:00:33.686847       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:00:37.180097       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:00:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="3f5ca012-6613-11ed-b53d-000d3a2cfc6c"
I1117 01:00:37.188495       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 8 milliseconds
I1117 01:00:37.188698       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="3f5ca012-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8665453
I1117 01:00:37.279429       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:00:37.279507       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="396d8421-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-aabaec3f-e310-461b-b57a-939d79cf2663" "latency"=10054916731
I1117 01:00:37.279548       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="396d8421-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10055152835
I1117 01:00:37.878556       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1117 01:00:37.878608       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 01:00:37.878623       1 utils.go:85] GRPC response: {}
I1117 01:00:37.885408       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 19 lines ...
I1117 01:00:38.482981       1 nodeprovisioner.go:164] EnsureBlockTargetReady [block]: making target file /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-aabaec3f-e310-461b-b57a-939d79cf2663/fd117469-5940-4656-af09-b599c257888f
I1117 01:00:38.483048       1 nodeserver_v2.go:348] NodePublishVolume: mounting /dev/disk/azure/scsi1/lun1 at /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-aabaec3f-e310-461b-b57a-939d79cf2663/fd117469-5940-4656-af09-b599c257888f
I1117 01:00:38.483062       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /dev/disk/azure/scsi1/lun1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-aabaec3f-e310-461b-b57a-939d79cf2663/fd117469-5940-4656-af09-b599c257888f)
I1117 01:00:38.484087       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /dev/disk/azure/scsi1/lun1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-aabaec3f-e310-461b-b57a-939d79cf2663/fd117469-5940-4656-af09-b599c257888f)
I1117 01:00:38.485020       1 nodeserver_v2.go:353] NodePublishVolume: mount /dev/disk/azure/scsi1/lun1 at /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-aabaec3f-e310-461b-b57a-939d79cf2663/fd117469-5940-4656-af09-b599c257888f successfully
I1117 01:00:38.485039       1 utils.go:85] GRPC response: {}
I1117 01:00:47.541997       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:00:47.542045       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="3ff03b53-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d4bfa70c-95fc-46a1-badd-932dbb5861ab" "latency"=9394446366
I1117 01:00:47.542065       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3ff03b53-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9394692169
I1117 01:00:47.747521       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1117 01:00:47.747541       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-aabaec3f-e310-461b-b57a-939d79cf2663/fd117469-5940-4656-af09-b599c257888f","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-aabaec3f-e310-461b-b57a-939d79cf2663"}
I1117 01:00:47.747592       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-aabaec3f-e310-461b-b57a-939d79cf2663 on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-aabaec3f-e310-461b-b57a-939d79cf2663/fd117469-5940-4656-af09-b599c257888f
I1117 01:00:47.747626       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-aabaec3f-e310-461b-b57a-939d79cf2663/fd117469-5940-4656-af09-b599c257888f" is a mountpoint, unmounting
... skipping 144 lines ...
I1117 01:01:07.180658       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:01:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="513e59cc-6613-11ed-b53d-000d3a2cfc6c"
I1117 01:01:07.188397       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 7 milliseconds
I1117 01:01:07.188825       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="513e59cc-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8185145
I1117 01:01:18.997565       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:01:18.997731       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:01:18.997799       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:01:18.997937       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:01:19.100204       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:01:33.686562       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:01:33.686587       1 utils.go:79] GRPC request: {}
I1117 01:01:33.686625       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:01:37.180235       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:01:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="631fec80-6613-11ed-b53d-000d3a2cfc6c"
I1117 01:01:37.187702       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 7 milliseconds
I1117 01:01:37.188596       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="631fec80-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8270046
I1117 01:01:43.573546       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:01:43.573609       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="4b57dbcc-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-81635160-7b73-47f5-955f-e345e0021cc9" "latency"=46291832728
I1117 01:01:43.573637       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4b57dbcc-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=46292193333
I1117 01:01:45.488176       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/
I1117 01:01:45.488223       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1117 01:01:45.488235       1 utils.go:85] GRPC response: {}
I1117 01:01:45.513092       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 55 lines ...
I1117 01:02:07.180253       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:02:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="75018ff1-6613-11ed-b53d-000d3a2cfc6c"
I1117 01:02:07.187472       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 7 milliseconds
I1117 01:02:07.187754       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="75018ff1-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7540233
I1117 01:02:18.998290       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:02:18.998419       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:02:18.998488       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:02:18.998624       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:02:19.101951       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:02:19.345881       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:02:19.345935       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="6a901732-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1107a484-ce48-4a81-b6a5-daba8a289021" "latency"=29686387773
I1117 01:02:19.345955       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6a901732-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29686624877
I1117 01:02:19.940501       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1117 01:02:19.940548       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:02:19.940582       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1107a484-ce48-4a81-b6a5-daba8a289021/globalmount with mount options([])
I1117 01:02:19.940594       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 121 lines ...
I1117 01:02:50.212753       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-af35610d-01f3-4811-8c1f-e2249d331d1c","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-af35610d-01f3-4811-8c1f-e2249d331d1c","csi.storage.k8s.io/pvc/name":"pvc-rpvnc","csi.storage.k8s.io/pvc/namespace":"provisioning-8413","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-af35610d-01f3-4811-8c1f-e2249d331d1c"}
I1117 01:02:50.212837       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-bb820651-51a7-4d74-b076-d7a36fafb444","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-bb820651-51a7-4d74-b076-d7a36fafb444","csi.storage.k8s.io/pvc/name":"pvc-k8hd7","csi.storage.k8s.io/pvc/namespace":"provisioning-8413","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-bb820651-51a7-4d74-b076-d7a36fafb444"}
I1117 01:02:50.213027       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-af35610d-01f3-4811-8c1f-e2249d331d1c." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8ea7d5cb-6613-11ed-b53d-000d3a2cfc6c"
I1117 01:02:50.213092       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-af35610d-01f3-4811-8c1f-e2249d331d1c-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:02:50.213094       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-bb820651-51a7-4d74-b076-d7a36fafb444." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8ea7d860-6613-11ed-b53d-000d3a2cfc6c"
I1117 01:02:50.213171       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-bb820651-51a7-4d74-b076-d7a36fafb444-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:02:58.611016       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:02:58.611094       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="8ea7d5cb-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-af35610d-01f3-4811-8c1f-e2249d331d1c" "latency"=8397924674
I1117 01:02:58.611126       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8ea7d5cb-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8398199579
I1117 01:02:58.611246       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:02:58.611298       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="8ea7d860-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-bb820651-51a7-4d74-b076-d7a36fafb444" "latency"=8398074577
I1117 01:02:58.611339       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8ea7d860-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8398344982
I1117 01:02:59.099698       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 01:02:59.099720       1 utils.go:79] GRPC request: {}
I1117 01:02:59.099765       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:02:59.103170       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 77 lines ...
W1117 01:03:12.903132       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-af35610d-01f3-4811-8c1f-e2249d331d1c" is not a mountpoint, deleting
I1117 01:03:12.903187       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-af35610d-01f3-4811-8c1f-e2249d331d1c successfully
I1117 01:03:12.903197       1 utils.go:85] GRPC response: {}
W1117 01:03:12.903218       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-bb820651-51a7-4d74-b076-d7a36fafb444" is not a mountpoint, deleting
I1117 01:03:12.903265       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-bb820651-51a7-4d74-b076-d7a36fafb444 successfully
I1117 01:03:12.903284       1 utils.go:85] GRPC response: {}
I1117 01:03:14.405605       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:03:14.405678       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="93f4a418-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e49b6dac-8143-4696-a1aa-ef7e8258f27f" "latency"=15300546030
I1117 01:03:14.405724       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="93f4a418-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15300833535
I1117 01:03:15.000397       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/
I1117 01:03:15.000449       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1117 01:03:15.000482       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e49b6dac-8143-4696-a1aa-ef7e8258f27f/globalmount with mount options([])
I1117 01:03:15.000491       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3])
... skipping 87 lines ...
I1117 01:03:37.180447       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:03:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="aaa6806c-6613-11ed-b53d-000d3a2cfc6c"
I1117 01:03:37.186668       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 6 milliseconds
I1117 01:03:37.187442       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="aaa6806c-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7039826
I1117 01:03:49.003208       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:03:49.003233       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:03:49.003209       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:03:49.003423       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:03:49.103472       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:04:03.686832       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:04:03.686854       1 utils.go:79] GRPC request: {}
I1117 01:04:03.686901       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:04:07.180569       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:04:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="bc88283e-6613-11ed-b53d-000d3a2cfc6c"
I1117 01:04:07.187569       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 6 milliseconds
I1117 01:04:07.187764       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="bc88283e-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7243029
I1117 01:04:19.003416       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:04:19.003457       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:04:19.003513       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:04:19.003544       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:04:19.103915       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:04:33.687085       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:04:33.687111       1 utils.go:79] GRPC request: {}
I1117 01:04:33.687161       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:04:37.180839       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:04:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="ce69d677-6613-11ed-b53d-000d3a2cfc6c"
I1117 01:04:37.190038       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 8 milliseconds
I1117 01:04:37.190706       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="ce69d677-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9890376
I1117 01:04:37.905976       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:04:37.906049       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="a6c11035-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c125a767-03ae-4232-9b75-8658d19929aa" "latency"=67262234124
I1117 01:04:37.906085       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a6c11035-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=67262505629
I1117 01:04:38.860575       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 01:04:38.860603       1 utils.go:79] GRPC request: {}
I1117 01:04:38.860641       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:04:38.861572       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 64 lines ...
I1117 01:04:44.899399       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c125a767-03ae-4232-9b75-8658d19929aa/globalmount
I1117 01:04:44.899426       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c125a767-03ae-4232-9b75-8658d19929aa/globalmount" is a mountpoint, unmounting
I1117 01:04:44.899439       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c125a767-03ae-4232-9b75-8658d19929aa/globalmount
W1117 01:04:44.910029       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c125a767-03ae-4232-9b75-8658d19929aa/globalmount" is not a mountpoint, deleting
I1117 01:04:44.910099       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c125a767-03ae-4232-9b75-8658d19929aa/globalmount successfully
I1117 01:04:44.910114       1 utils.go:85] GRPC response: {}
I1117 01:04:48.130950       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:04:48.131027       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="cf85e82c-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7e6362bf-1cf1-4db9-9fa4-ed06c216c71c" "latency"=9088198660
I1117 01:04:48.131065       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cf85e82c-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9088586267
I1117 01:04:48.141035       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:04:48.141103       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="cf6a72d2-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8796a13f-ff41-4166-bc27-4ddf40b17d40" "latency"=9278361466
I1117 01:04:48.141141       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cf6a72d2-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9278613770
I1117 01:04:48.740075       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/
I1117 01:04:48.740130       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:04:48.740135       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1117 01:04:48.740160       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
... skipping 100 lines ...
I1117 01:05:07.179951       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:05:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="e04b55f1-6613-11ed-b53d-000d3a2cfc6c"
I1117 01:05:07.194402       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 14 milliseconds
I1117 01:05:07.194966       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="e04b55f1-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15064167
I1117 01:05:19.010290       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:05:19.010361       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:05:19.010389       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:05:19.010454       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:05:19.105686       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:05:23.562190       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:05:23.562262       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="d759334a-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-cf17b189-e0d6-416e-a98c-7546694304d0" "latency"=31390748221
I1117 01:05:23.562294       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d759334a-6613-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31391029726
I1117 01:05:24.148795       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdf under /dev/disk/azure/scsi1/
I1117 01:05:24.148853       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1117 01:05:24.148892       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cf17b189-e0d6-416e-a98c-7546694304d0/globalmount with mount options([])
I1117 01:05:24.148908       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 87 lines ...
I1117 01:06:13.233013       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:06:13.233028       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"3"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7449fae9-8728-4a63-8337-425e114e31a8/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7449fae9-8728-4a63-8337-425e114e31a8","csi.storage.k8s.io/pvc/name":"test.csi.azure.comndj6z","csi.storage.k8s.io/pvc/namespace":"multivolume-6358","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-7449fae9-8728-4a63-8337-425e114e31a8"}
I1117 01:06:13.233283       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-7449fae9-8728-4a63-8337-425e114e31a8." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="07aa4468-6614-11ed-b53d-000d3a2cfc6c"
I1117 01:06:13.233353       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7449fae9-8728-4a63-8337-425e114e31a8-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:06:19.018937       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:06:19.019009       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:06:19.019072       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:06:19.019087       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:06:19.019094       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:06:19.106421       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:06:33.686508       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:06:33.686536       1 utils.go:79] GRPC request: {}
I1117 01:06:33.686585       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:06:37.180719       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:06:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="15f05c89-6614-11ed-b53d-000d3a2cfc6c"
I1117 01:06:37.192259       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 11 milliseconds
I1117 01:06:37.192482       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="15f05c89-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11826109
I1117 01:06:40.092025       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 19 items received
I1117 01:06:40.095609       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-18975173-vmss000001&resourceVersion=14067&timeout=9m57s&timeoutSeconds=597&watch=true 200 OK in 3 milliseconds
I1117 01:06:49.019236       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:06:49.019328       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:06:49.019348       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:06:49.019434       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:06:49.019461       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:06:49.106555       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:07:03.687059       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:07:03.687086       1 utils.go:79] GRPC request: {}
I1117 01:07:03.687134       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:07:07.180500       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:07:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="27d1f779-6614-11ed-b53d-000d3a2cfc6c"
I1117 01:07:07.189626       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 8 milliseconds
I1117 01:07:07.189918       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="27d1f779-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9463467
I1117 01:07:10.270687       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:07:10.270758       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="079b83c8-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-aa6233ad-24c6-40af-908c-228072860561" "latency"=57134011582
I1117 01:07:10.270789       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="079b83c8-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=57134291187
I1117 01:07:10.277533       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:07:10.277589       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="07aa4468-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7449fae9-8728-4a63-8337-425e114e31a8" "latency"=57044181198
I1117 01:07:10.277609       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="07aa4468-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=57044437903
I1117 01:07:10.864941       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdc under /dev/disk/azure/scsi1/
I1117 01:07:10.864989       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType 
I1117 01:07:10.865021       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun4 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-aa6233ad-24c6-40af-908c-228072860561/globalmount with mount options([])
I1117 01:07:10.865031       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun4" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun4])
... skipping 129 lines ...
I1117 01:08:15.641174       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-c257070a-5353-4cdb-80e7-4fc48a46854a." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="50a03b00-6614-11ed-b53d-000d3a2cfc6c"
I1117 01:08:15.641216       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-c257070a-5353-4cdb-80e7-4fc48a46854a-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:08:15.946659       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 81 items received
I1117 01:08:15.948118       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=14939&timeout=5m34s&timeoutSeconds=334&watch=true 200 OK in 1 milliseconds
I1117 01:08:19.027776       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:19.027855       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:19.027882       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:08:19.027902       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:19.108250       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:23.946293       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:08:23.946356       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="50a03b00-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c257070a-5353-4cdb-80e7-4fc48a46854a" "latency"=8305075568
I1117 01:08:23.946387       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="50a03b00-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8305308772
I1117 01:08:25.721808       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:08:25.721874       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:08:25.721905       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c257070a-5353-4cdb-80e7-4fc48a46854a/globalmount with mount options([])
I1117 01:08:25.721914       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 60 lines ...
I1117 01:08:36.088991       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3755adb7-3dee-47ab-a298-13fb563d8ee4/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3755adb7-3dee-47ab-a298-13fb563d8ee4","csi.storage.k8s.io/pvc/name":"test.csi.azure.comf4ff9","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-4878","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-3755adb7-3dee-47ab-a298-13fb563d8ee4"}
I1117 01:08:36.089243       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-3755adb7-3dee-47ab-a298-13fb563d8ee4." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5cd05b66-6614-11ed-b53d-000d3a2cfc6c"
I1117 01:08:36.089303       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-3755adb7-3dee-47ab-a298-13fb563d8ee4-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:08:37.180733       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:08:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="5d76ea2e-6614-11ed-b53d-000d3a2cfc6c"
I1117 01:08:37.193447       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 12 milliseconds
I1117 01:08:37.193623       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="5d76ea2e-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12923228
I1117 01:08:43.713349       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:08:43.713400       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="5cd05b66-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3755adb7-3dee-47ab-a298-13fb563d8ee4" "latency"=7624035971
I1117 01:08:43.713420       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5cd05b66-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7624286275
I1117 01:08:44.330030       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1117 01:08:44.330079       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 01:08:44.330114       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3755adb7-3dee-47ab-a298-13fb563d8ee4/globalmount with mount options([])
I1117 01:08:44.330130       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 44 lines ...
I1117 01:08:48.193320       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1117 01:08:48.193334       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-3755adb7-3dee-47ab-a298-13fb563d8ee4","volume_path":"/var/lib/kubelet/pods/7ebaf141-7d7a-48ad-b1a9-d4e50ef0b793/volumes/kubernetes.io~csi/pvc-3755adb7-3dee-47ab-a298-13fb563d8ee4/mount"}
I1117 01:08:48.193411       1 utils.go:85] GRPC response: {"usage":[{"available":5179580416,"total":5196382208,"unit":1,"used":24576},{"available":327669,"total":327680,"unit":2,"used":11}]}
I1117 01:08:49.028669       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:49.028704       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:49.028670       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:49.028780       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:08:49.109170       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:55.413800       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1117 01:08:55.413823       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/7ebaf141-7d7a-48ad-b1a9-d4e50ef0b793/volumes/kubernetes.io~csi/pvc-3755adb7-3dee-47ab-a298-13fb563d8ee4/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-3755adb7-3dee-47ab-a298-13fb563d8ee4"}
I1117 01:08:55.413910       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-3755adb7-3dee-47ab-a298-13fb563d8ee4 on /var/lib/kubelet/pods/7ebaf141-7d7a-48ad-b1a9-d4e50ef0b793/volumes/kubernetes.io~csi/pvc-3755adb7-3dee-47ab-a298-13fb563d8ee4/mount
I1117 01:08:55.413953       1 mount_helper_common.go:99] "/var/lib/kubelet/pods/7ebaf141-7d7a-48ad-b1a9-d4e50ef0b793/volumes/kubernetes.io~csi/pvc-3755adb7-3dee-47ab-a298-13fb563d8ee4/mount" is a mountpoint, unmounting
I1117 01:08:55.413963       1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/7ebaf141-7d7a-48ad-b1a9-d4e50ef0b793/volumes/kubernetes.io~csi/pvc-3755adb7-3dee-47ab-a298-13fb563d8ee4/mount
... skipping 17 lines ...
I1117 01:09:07.180640       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:09:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="6f58898d-6614-11ed-b53d-000d3a2cfc6c"
I1117 01:09:07.188295       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 7 milliseconds
I1117 01:09:07.188952       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="6f58898d-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8357147
I1117 01:09:19.031585       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:09:19.031618       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:09:19.031687       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:09:19.031755       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:09:19.110007       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:09:33.687139       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:09:33.687160       1 utils.go:79] GRPC request: {}
I1117 01:09:33.687197       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:09:37.180062       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:09:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="813a14fc-6614-11ed-b53d-000d3a2cfc6c"
I1117 01:09:37.192558       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 12 milliseconds
I1117 01:09:37.193349       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="813a14fc-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13342335
I1117 01:09:37.964206       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 58 items received
I1117 01:09:37.967958       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=15736&timeout=9m38s&timeoutSeconds=578&watch=true 200 OK in 3 milliseconds
I1117 01:09:49.032039       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:09:49.032099       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:09:49.032161       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:09:49.032164       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:09:49.110458       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:09:49.516370       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:09:49.516443       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="62ff7c3a-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fc1496a1-11d7-44bb-bb2e-5381b8b7c602" "latency"=63051814619
I1117 01:09:49.516485       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="62ff7c3a-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=63052154725
I1117 01:09:50.014146       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 01:09:50.014169       1 utils.go:79] GRPC request: {}
I1117 01:09:50.014213       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:09:50.017639       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 63 lines ...
I1117 01:10:07.180254       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:10:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="931bc08a-6614-11ed-b53d-000d3a2cfc6c"
I1117 01:10:07.187669       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 7 milliseconds
I1117 01:10:07.188490       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="931bc08a-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8269145
I1117 01:10:19.032153       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:19.032256       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:19.032280       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:19.032264       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:10:19.111668       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:33.687102       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:10:33.687129       1 utils.go:79] GRPC request: {}
I1117 01:10:33.687178       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:10:37.180270       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:10:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="a4fd63be-6614-11ed-b53d-000d3a2cfc6c"
I1117 01:10:37.186138       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 5 milliseconds
I1117 01:10:37.186342       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="a4fd63be-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6114208
I1117 01:10:49.033071       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:49.033117       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:49.033217       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:10:49.033235       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:49.112514       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:55.835027       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:10:55.835084       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="88e13266-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8e611c76-c4e1-420e-8ffd-ceebf9205d3d" "latency"=65815590903
I1117 01:10:55.835108       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="88e13266-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=65815860808
I1117 01:10:57.537521       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:10:57.537564       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:10:57.537594       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8e611c76-c4e1-420e-8ffd-ceebf9205d3d/globalmount with mount options([nouuid])
I1117 01:10:57.537613       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 50 lines ...
I1117 01:11:03.686319       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:11:03.686357       1 utils.go:79] GRPC request: {}
I1117 01:11:03.686406       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:11:07.180163       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:11:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="b6df0304-6614-11ed-b53d-000d3a2cfc6c"
I1117 01:11:07.191334       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 11 milliseconds
I1117 01:11:07.191553       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="b6df0304-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11429405
I1117 01:11:16.302031       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:11:16.302095       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="b1a8d90e-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d5c8f003-0658-4ab3-80ec-c3ca04e9b0ab" "latency"=17865328941
I1117 01:11:16.302126       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b1a8d90e-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=17865576045
I1117 01:11:16.302483       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:11:16.302529       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="b16b1523-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9eede198-69b2-40e2-87e4-433451e224d9" "latency"=18270537588
I1117 01:11:16.302562       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b16b1523-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=18270800292
I1117 01:11:16.909356       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1117 01:11:16.909356       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1117 01:11:16.909417       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1117 01:11:16.909432       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
... skipping 165 lines ...
I1117 01:11:41.743607       1 nodeprovisioner.go:164] EnsureBlockTargetReady [block]: making target file /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-9eede198-69b2-40e2-87e4-433451e224d9/d90de220-7c30-4f1d-b28f-8d48a06a3819
I1117 01:11:41.743672       1 nodeserver_v2.go:348] NodePublishVolume: mounting /dev/disk/azure/scsi1/lun1 at /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-9eede198-69b2-40e2-87e4-433451e224d9/d90de220-7c30-4f1d-b28f-8d48a06a3819
I1117 01:11:41.743685       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /dev/disk/azure/scsi1/lun1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-9eede198-69b2-40e2-87e4-433451e224d9/d90de220-7c30-4f1d-b28f-8d48a06a3819)
I1117 01:11:41.744781       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /dev/disk/azure/scsi1/lun1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-9eede198-69b2-40e2-87e4-433451e224d9/d90de220-7c30-4f1d-b28f-8d48a06a3819)
I1117 01:11:41.745666       1 nodeserver_v2.go:353] NodePublishVolume: mount /dev/disk/azure/scsi1/lun1 at /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-9eede198-69b2-40e2-87e4-433451e224d9/d90de220-7c30-4f1d-b28f-8d48a06a3819 successfully
I1117 01:11:41.745691       1 utils.go:85] GRPC response: {}
I1117 01:11:46.783018       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:11:46.783073       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="be6b0371-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-38c9776a-931d-43ba-a74e-6e93a02526bd" "latency"=26941122744
I1117 01:11:46.783093       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="be6b0371-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=26941407750
I1117 01:11:47.389459       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/
I1117 01:11:47.389503       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1117 01:11:47.389531       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-38c9776a-931d-43ba-a74e-6e93a02526bd/globalmount with mount options([nouuid])
I1117 01:11:47.389542       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3])
... skipping 139 lines ...
I1117 01:12:10.998429       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"4"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-79376339-90c7-45db-adfe-fd37cf2f4855/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-79376339-90c7-45db-adfe-fd37cf2f4855","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-fmb58-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-1296","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-79376339-90c7-45db-adfe-fd37cf2f4855"}
I1117 01:12:10.998713       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-79376339-90c7-45db-adfe-fd37cf2f4855." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="dce8f03a-6614-11ed-b53d-000d3a2cfc6c"
I1117 01:12:10.998773       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-79376339-90c7-45db-adfe-fd37cf2f4855-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:12:19.036895       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:12:19.036926       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:12:19.036974       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:12:19.037024       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:12:19.114346       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:12:33.686581       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:12:33.686600       1 utils.go:79] GRPC request: {}
I1117 01:12:33.686650       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:12:37.180968       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:12:37Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="ec840ace-6614-11ed-b53d-000d3a2cfc6c"
I1117 01:12:37.190002       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 8 milliseconds
I1117 01:12:37.190221       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="ec840ace-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9300864
I1117 01:12:37.681953       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:12:37.682016       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="dce8f03a-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-79376339-90c7-45db-adfe-fd37cf2f4855" "latency"=26683190460
I1117 01:12:37.682039       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="dce8f03a-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=26683441164
I1117 01:12:39.453057       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdc under /dev/disk/azure/scsi1/
I1117 01:12:39.453107       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType 
I1117 01:12:39.453135       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun4 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-79376339-90c7-45db-adfe-fd37cf2f4855/globalmount with mount options([])
I1117 01:12:39.453145       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun4" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun4])
... skipping 51 lines ...
I1117 01:13:03.686614       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:13:03.686639       1 utils.go:79] GRPC request: {}
I1117 01:13:03.686687       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:13:07.180074       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:13:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="fe658ada-6614-11ed-b53d-000d3a2cfc6c"
I1117 01:13:07.187917       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 7 milliseconds
I1117 01:13:07.188046       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="fe658ada-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8025341
I1117 01:13:13.114880       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:13:13.114937       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="f50cb694-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5f8afb0d-719e-49d9-8c1b-6778fd9abd5b" "latency"=21616333139
I1117 01:13:13.114958       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f50cb694-6614-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=21616591144
I1117 01:13:13.723769       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1117 01:13:13.723819       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:13:13.723847       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5f8afb0d-719e-49d9-8c1b-6778fd9abd5b/globalmount with mount options([])
I1117 01:13:13.723863       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 83 lines ...
I1117 01:13:42.541474       1 utils.go:79] GRPC request: {}
I1117 01:13:42.541508       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:13:42.542354       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:13:42.542369       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-dcab4389-64c3-4944-abbc-96ccaa625bcf/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-dcab4389-64c3-4944-abbc-96ccaa625bcf","csi.storage.k8s.io/pvc/name":"test.csi.azure.comlbk4f","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-4887","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-dcab4389-64c3-4944-abbc-96ccaa625bcf"}
I1117 01:13:42.542645       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-dcab4389-64c3-4944-abbc-96ccaa625bcf." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="13796f5f-6615-11ed-b53d-000d3a2cfc6c"
I1117 01:13:42.542694       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-dcab4389-64c3-4944-abbc-96ccaa625bcf-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:13:48.957163       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:13:48.957215       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="13796f5f-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-dcab4389-64c3-4944-abbc-96ccaa625bcf" "latency"=6414469788
I1117 01:13:48.957236       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="13796f5f-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6414699896
I1117 01:13:49.038868       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:13:49.038879       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:13:49.038892       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:13:49.116416       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 151 lines ...
I1117 01:14:41.087931       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:14:41.088691       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:14:41.088706       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a8709542-4b11-4779-8c79-2e717c4c4cad/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a8709542-4b11-4779-8c79-2e717c4c4cad","csi.storage.k8s.io/pvc/name":"test.csi.azure.commgvdn","csi.storage.k8s.io/pvc/namespace":"multivolume-3404","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-a8709542-4b11-4779-8c79-2e717c4c4cad"}
I1117 01:14:41.088970       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-a8709542-4b11-4779-8c79-2e717c4c4cad." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="365ee54c-6615-11ed-b53d-000d3a2cfc6c"
I1117 01:14:41.089028       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a8709542-4b11-4779-8c79-2e717c4c4cad-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:14:49.044508       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:14:49.044623       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:14:49.044639       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:14:49.044653       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:14:49.044659       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:14:49.117983       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:15:03.687147       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:15:03.687177       1 utils.go:79] GRPC request: {}
I1117 01:15:03.687231       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:15:07.180912       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:15:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="45ec37d2-6615-11ed-b53d-000d3a2cfc6c"
I1117 01:15:07.188360       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 7 milliseconds
I1117 01:15:07.188581       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="45ec37d2-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7711926
I1117 01:15:15.820976       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:15:15.821058       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="36122e09-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-863fca1d-68f0-45f2-931d-e9ecdd0fa10e" "latency"=35234689536
I1117 01:15:15.821088       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="36122e09-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=35234984641
I1117 01:15:15.825501       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:15:15.825554       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="365ee54c-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a8709542-4b11-4779-8c79-2e717c4c4cad" "latency"=34736480156
I1117 01:15:15.825575       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="365ee54c-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=34736711460
I1117 01:15:16.407600       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1117 01:15:16.407653       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 01:15:16.407680       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-863fca1d-68f0-45f2-931d-e9ecdd0fa10e/globalmount with mount options([])
I1117 01:15:16.407689       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 118 lines ...
I1117 01:15:45.066646       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9f2f9323-c6f6-4ec7-8e19-56385038a97a/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9f2f9323-c6f6-4ec7-8e19-56385038a97a","csi.storage.k8s.io/pvc/name":"test.csi.azure.com4ljzx","csi.storage.k8s.io/pvc/namespace":"provisioning-5056","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-9f2f9323-c6f6-4ec7-8e19-56385038a97a"}
I1117 01:15:45.066875       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-9f2f9323-c6f6-4ec7-8e19-56385038a97a." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5c8126ad-6615-11ed-b53d-000d3a2cfc6c"
I1117 01:15:45.066930       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-9f2f9323-c6f6-4ec7-8e19-56385038a97a-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:15:49.050716       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:15:49.050818       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:15:49.050847       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:15:49.050898       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:15:49.119179       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:16:03.686979       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:16:03.687000       1 utils.go:79] GRPC request: {}
I1117 01:16:03.687042       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:16:06.826645       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:16:06.826717       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="5c8126ad-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9f2f9323-c6f6-4ec7-8e19-56385038a97a" "latency"=21759720900
I1117 01:16:06.826754       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5c8126ad-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=21759969204
I1117 01:16:07.180923       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:16:07Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="69af7e3b-6615-11ed-b53d-000d3a2cfc6c"
I1117 01:16:07.188475       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000001/status 200 OK in 7 milliseconds
I1117 01:16:07.188612       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="69af7e3b-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7732933
I1117 01:16:08.643773       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/
... skipping 100 lines ...
I1117 01:18:18.558660       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:18:18.559372       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:18:18.559389       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-88484516-774c-498b-b6b2-e3c3a26d0a28/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-88484516-774c-498b-b6b2-e3c3a26d0a28","csi.storage.k8s.io/pvc/name":"pvc-q59rt","csi.storage.k8s.io/pvc/namespace":"provisioning-5631","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-88484516-774c-498b-b6b2-e3c3a26d0a28"}
I1117 01:18:18.559630       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-88484516-774c-498b-b6b2-e3c3a26d0a28." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b7fe4925-6615-11ed-b53d-000d3a2cfc6c"
I1117 01:18:18.559670       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-88484516-774c-498b-b6b2-e3c3a26d0a28-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:18:19.056176       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:18:19.056256       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:18:19.056270       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:18:19.056303       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:18:19.123772       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:18:28.047500       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:18:28.047598       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="b7fe4925-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-88484516-774c-498b-b6b2-e3c3a26d0a28" "latency"=9487834301
I1117 01:18:28.047642       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b7fe4925-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9488105706
I1117 01:18:29.790637       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:18:29.790686       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:18:29.790727       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-88484516-774c-498b-b6b2-e3c3a26d0a28/globalmount with mount options([])
I1117 01:18:29.790741       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 77 lines ...
I1117 01:19:07.192759       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="d4f95e7e-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11505496
I1117 01:19:15.968796       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 86 items received
I1117 01:19:15.971424       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=20955&timeout=7m49s&timeoutSeconds=469&watch=true 200 OK in 2 milliseconds
I1117 01:19:19.057308       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:19:19.057409       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:19:19.057502       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:19:19.057530       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:19:19.124912       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:19:30.870236       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:19:30.870319       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="d04db94a-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-40caeb3b-90af-44a6-82ec-72c7c4cd1510" "latency"=31524599071
I1117 01:19:30.870353       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d04db94a-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31524893976
I1117 01:19:31.561774       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 01:19:31.561793       1 utils.go:79] GRPC request: {}
I1117 01:19:31.561833       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:19:31.565361       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 49 lines ...
I1117 01:19:48.200226       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1117 01:19:48.200243       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-40caeb3b-90af-44a6-82ec-72c7c4cd1510","volume_path":"/var/lib/kubelet/pods/a7e40631-a603-4a5a-b6fd-62114d1d5553/volumes/kubernetes.io~csi/pvc-40caeb3b-90af-44a6-82ec-72c7c4cd1510/mount"}
I1117 01:19:48.200315       1 utils.go:85] GRPC response: {"usage":[{"available":5179576320,"total":5196382208,"unit":1,"used":28672},{"available":327668,"total":327680,"unit":2,"used":12}]}
I1117 01:19:49.057713       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:19:49.057791       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:19:49.057801       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:19:49.057897       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:19:49.125219       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:20:03.686467       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:20:03.686491       1 utils.go:79] GRPC request: {}
I1117 01:20:03.686532       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:20:06.335867       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:20:06.335932       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="e3825ca2-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-172557d5-c972-40f5-a23a-a5a240b03379" "latency"=34768549941
I1117 01:20:06.335963       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e3825ca2-6615-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=34768825046
I1117 01:20:06.921216       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1117 01:20:06.921261       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:20:06.921288       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-172557d5-c972-40f5-a23a-a5a240b03379/globalmount with mount options([])
I1117 01:20:06.921297       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 99 lines ...
I1117 01:21:22.379536       1 utils.go:79] GRPC request: {}
I1117 01:21:22.379566       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:21:22.380309       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:21:22.380326       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b19dcf32-ae32-498b-bbea-52805047ba95/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b19dcf32-ae32-498b-bbea-52805047ba95","csi.storage.k8s.io/pvc/name":"test.csi.azure.com7sw2f","csi.storage.k8s.io/pvc/namespace":"provisioning-6921","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-b19dcf32-ae32-498b-bbea-52805047ba95"}
I1117 01:21:22.380611       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-b19dcf32-ae32-498b-bbea-52805047ba95." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="258f23f5-6616-11ed-b53d-000d3a2cfc6c"
I1117 01:21:22.380671       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-b19dcf32-ae32-498b-bbea-52805047ba95-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:21:26.807710       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:21:26.807773       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="258f23f5-6616-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b19dcf32-ae32-498b-bbea-52805047ba95" "latency"=4427029988
I1117 01:21:26.807796       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="258f23f5-6616-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4427293393
I1117 01:21:28.482160       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:21:28.482198       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:21:28.482224       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b19dcf32-ae32-498b-bbea-52805047ba95/globalmount with mount options([])
I1117 01:21:28.482232       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 129 lines ...
I1117 01:24:26.370516       1 utils.go:79] GRPC request: {}
I1117 01:24:26.370555       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:24:26.371373       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:24:26.371394       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-65faf254-f03d-4f3b-b1b6-8c1c94e88d93","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-65faf254-f03d-4f3b-b1b6-8c1c94e88d93","csi.storage.k8s.io/pvc/name":"inline-volume-tester-4wsj4-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-4725","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-65faf254-f03d-4f3b-b1b6-8c1c94e88d93"}
I1117 01:24:26.371656       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-65faf254-f03d-4f3b-b1b6-8c1c94e88d93." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9339f226-6616-11ed-b53d-000d3a2cfc6c"
I1117 01:24:26.371700       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-65faf254-f03d-4f3b-b1b6-8c1c94e88d93-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:24:28.497036       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:24:28.497101       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="9339f226-6616-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-65faf254-f03d-4f3b-b1b6-8c1c94e88d93" "latency"=2125328312
I1117 01:24:28.497134       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9339f226-6616-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2125578816
I1117 01:24:30.310596       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:24:30.310652       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:24:30.310668       1 utils.go:85] GRPC response: {}
I1117 01:24:30.317776       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 164 lines ...
I1117 01:29:53.677230       1 utils.go:79] GRPC request: {}
I1117 01:29:53.677251       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:29:53.677973       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:29:53.677991       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4fc7d2a8-31aa-4da1-99ed-c5f0a5d1d199/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4fc7d2a8-31aa-4da1-99ed-c5f0a5d1d199","csi.storage.k8s.io/pvc/name":"test.csi.azure.comrrnmh","csi.storage.k8s.io/pvc/namespace":"volume-2038","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-4fc7d2a8-31aa-4da1-99ed-c5f0a5d1d199"}
I1117 01:29:53.678250       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-4fc7d2a8-31aa-4da1-99ed-c5f0a5d1d199." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5650f876-6617-11ed-b53d-000d3a2cfc6c"
I1117 01:29:53.678295       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-4fc7d2a8-31aa-4da1-99ed-c5f0a5d1d199-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:29:55.935007       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:29:55.935076       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="5650f876-6617-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4fc7d2a8-31aa-4da1-99ed-c5f0a5d1d199" "latency"=2256708155
I1117 01:29:55.935120       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5650f876-6617-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2256959860
I1117 01:29:57.637160       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:29:57.637207       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:29:57.637239       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4fc7d2a8-31aa-4da1-99ed-c5f0a5d1d199/globalmount with mount options([nouuid])
I1117 01:29:57.637255       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 70 lines ...
I1117 01:30:44.667613       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:30:44.668397       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:30:44.668413       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e8166beb-9f47-4083-a611-074e69ecf332","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e8166beb-9f47-4083-a611-074e69ecf332","csi.storage.k8s.io/pvc/name":"pvc-88gpl","csi.storage.k8s.io/pvc/namespace":"provisioning-8900","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-e8166beb-9f47-4083-a611-074e69ecf332"}
I1117 01:30:44.668683       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-e8166beb-9f47-4083-a611-074e69ecf332." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="74b57d4a-6617-11ed-b53d-000d3a2cfc6c"
I1117 01:30:44.668725       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e8166beb-9f47-4083-a611-074e69ecf332-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:30:49.100973       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:30:49.101057       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:30:49.101062       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:30:49.101073       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:30:49.139551       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:30:53.683398       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:30:53.683470       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="74b57d4a-6617-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e8166beb-9f47-4083-a611-074e69ecf332" "latency"=9014683816
I1117 01:30:53.683500       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="74b57d4a-6617-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9014932720
I1117 01:30:55.471872       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:30:55.471917       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:30:55.471928       1 utils.go:85] GRPC response: {}
I1117 01:30:55.479811       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 100 lines ...
I1117 01:33:50.007319       1 utils.go:79] GRPC request: {}
I1117 01:33:50.007355       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:33:50.008087       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:33:50.008101       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9ac93ab1-0ca7-424c-a4a8-0ad46902a107/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9ac93ab1-0ca7-424c-a4a8-0ad46902a107","csi.storage.k8s.io/pvc/name":"test.csi.azure.comlwlhz","csi.storage.k8s.io/pvc/namespace":"multivolume-7146","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-9ac93ab1-0ca7-424c-a4a8-0ad46902a107"}
I1117 01:33:50.008422       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-9ac93ab1-0ca7-424c-a4a8-0ad46902a107." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e32e1714-6617-11ed-b53d-000d3a2cfc6c"
I1117 01:33:50.008492       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-9ac93ab1-0ca7-424c-a4a8-0ad46902a107-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:33:59.481091       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:33:59.481171       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="e32e1714-6617-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9ac93ab1-0ca7-424c-a4a8-0ad46902a107" "latency"=9472579394
I1117 01:33:59.481207       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e32e1714-6617-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9472891099
I1117 01:34:01.221587       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:34:01.221636       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:34:01.221664       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9ac93ab1-0ca7-424c-a4a8-0ad46902a107/globalmount with mount options([nouuid])
I1117 01:34:01.221677       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 40 lines ...
I1117 01:34:08.899747       1 utils.go:79] GRPC request: {}
I1117 01:34:08.899789       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:34:08.900739       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:34:08.900760       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1581c112-4dd5-452b-b773-7bd7253e903a/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1581c112-4dd5-452b-b773-7bd7253e903a","csi.storage.k8s.io/pvc/name":"test.csi.azure.comlwlhz-restored","csi.storage.k8s.io/pvc/namespace":"multivolume-7146","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-1581c112-4dd5-452b-b773-7bd7253e903a"}
I1117 01:34:08.901106       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-1581c112-4dd5-452b-b773-7bd7253e903a." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ee70e116-6617-11ed-b53d-000d3a2cfc6c"
I1117 01:34:08.901180       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-1581c112-4dd5-452b-b773-7bd7253e903a-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:34:15.043866       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:34:15.043939       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="ee70e116-6617-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1581c112-4dd5-452b-b773-7bd7253e903a" "latency"=6142699978
I1117 01:34:15.043978       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ee70e116-6617-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6143018484
I1117 01:34:15.637127       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1117 01:34:15.637174       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 01:34:15.637204       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1581c112-4dd5-452b-b773-7bd7253e903a/globalmount with mount options([nouuid])
I1117 01:34:15.637216       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 96 lines ...
I1117 01:35:19.695894       1 utils.go:79] GRPC request: {}
I1117 01:35:19.695922       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:35:19.696635       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:35:19.696653       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cd237ebf-c6e4-442c-8aea-714def328cb2/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-cd237ebf-c6e4-442c-8aea-714def328cb2","csi.storage.k8s.io/pvc/name":"test.csi.azure.coms2m9d","csi.storage.k8s.io/pvc/namespace":"provisioning-7668","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-cd237ebf-c6e4-442c-8aea-714def328cb2"}
I1117 01:35:19.696886       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-cd237ebf-c6e4-442c-8aea-714def328cb2." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="18a3772b-6618-11ed-b53d-000d3a2cfc6c"
I1117 01:35:19.696927       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-cd237ebf-c6e4-442c-8aea-714def328cb2-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:35:20.968559       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:35:20.968636       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="18a3772b-6618-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-cd237ebf-c6e4-442c-8aea-714def328cb2" "latency"=1271631182
I1117 01:35:20.968676       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="18a3772b-6618-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1271887487
I1117 01:35:22.759336       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:35:22.759381       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:35:22.759408       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cd237ebf-c6e4-442c-8aea-714def328cb2/globalmount with mount options([])
I1117 01:35:22.759417       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 119 lines ...
I1117 01:38:10.813127       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8a2e475f-39d2-46e8-92dc-7f57b6b3f9e3/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8a2e475f-39d2-46e8-92dc-7f57b6b3f9e3","csi.storage.k8s.io/pvc/name":"test.csi.azure.com5h8md","csi.storage.k8s.io/pvc/namespace":"multivolume-2002","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-8a2e475f-39d2-46e8-92dc-7f57b6b3f9e3"}
I1117 01:38:10.813379       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-8a2e475f-39d2-46e8-92dc-7f57b6b3f9e3." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7ea1c506-6618-11ed-b53d-000d3a2cfc6c"
I1117 01:38:10.813425       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-8a2e475f-39d2-46e8-92dc-7f57b6b3f9e3-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:38:19.127704       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:38:19.127773       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:38:19.127802       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:38:19.127859       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:38:19.149098       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:38:21.516152       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:38:21.516225       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="7ea1c506-6618-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8a2e475f-39d2-46e8-92dc-7f57b6b3f9e3" "latency"=10702712908
I1117 01:38:21.516250       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7ea1c506-6618-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10702976813
I1117 01:38:23.230962       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:38:23.231025       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:38:23.231067       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8a2e475f-39d2-46e8-92dc-7f57b6b3f9e3/globalmount with mount options([])
I1117 01:38:23.231086       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 161 lines ...
I1117 01:42:19.165563       1 utils.go:79] GRPC request: {}
I1117 01:42:19.165607       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:42:19.166370       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:42:19.166385       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-271bb09c-910a-44de-bb6a-1d0fb19c61e1/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-271bb09c-910a-44de-bb6a-1d0fb19c61e1","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-271bb09c-910a-44de-bb6a-1d0fb19c61e1"}
I1117 01:42:19.166647       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-271bb09c-910a-44de-bb6a-1d0fb19c61e1." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="12a9789b-6619-11ed-b53d-000d3a2cfc6c"
I1117 01:42:19.166695       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-271bb09c-910a-44de-bb6a-1d0fb19c61e1-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:42:24.044715       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:42:24.044786       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="12a9789b-6619-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-271bb09c-910a-44de-bb6a-1d0fb19c61e1" "latency"=4878004420
I1117 01:42:24.044811       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="12a9789b-6619-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4878269024
I1117 01:42:25.761089       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:42:25.761150       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS
I1117 01:42:25.761185       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-271bb09c-910a-44de-bb6a-1d0fb19c61e1/globalmount with mount options([])
I1117 01:42:25.761205       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 72 lines ...
I1117 01:43:41.046749       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6b0b1726-6e17-44c1-af12-bd309ec1f89f/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6b0b1726-6e17-44c1-af12-bd309ec1f89f","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-9bcd7-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-6b0b1726-6e17-44c1-af12-bd309ec1f89f"}
I1117 01:43:41.047035       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000001,disk.csi.azure.com/volume-name!=pvc-6b0b1726-6e17-44c1-af12-bd309ec1f89f." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="43776d2b-6619-11ed-b53d-000d3a2cfc6c"
I1117 01:43:41.047106       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-6b0b1726-6e17-44c1-af12-bd309ec1f89f-k8s-agentpool1-18975173-vmss000001-attachment)
I1117 01:43:49.134954       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:43:49.134984       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:43:49.135022       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:43:49.135064       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:43:49.154311       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:43:55.918640       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:43:55.918721       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000001" "disk.csi.azure.com/request-id"="43776d2b-6619-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6b0b1726-6e17-44c1-af12-bd309ec1f89f" "latency"=14871512085
I1117 01:43:55.918755       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="43776d2b-6619-11ed-b53d-000d3a2cfc6c" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14871820290
I1117 01:43:56.512680       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1117 01:43:56.512727       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS
I1117 01:43:56.512757       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6b0b1726-6e17-44c1-af12-bd309ec1f89f/globalmount with mount options([])
I1117 01:43:56.512765       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 107 lines ...
I1117 00:45:18.605405       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=2449&timeout=7m23s&timeoutSeconds=443&watch=true 200 OK in 1 milliseconds
I1117 00:45:18.606450       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=2451&timeout=6m0s&timeoutSeconds=360&watch=true 200 OK in 2 milliseconds
I1117 00:45:18.694653       1 shared_informer.go:285] caches populated
I1117 00:45:18.694927       1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-c551e0323ad732edee8d00639b70d4fc25a9b407 e2e-test
I1117 00:45:18.695034       1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider
I1117 00:45:18.700772       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 5 milliseconds
I1117 00:45:18.701590       1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I1117 00:45:18.701758       1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider
I1117 00:45:18.701844       1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I1117 00:45:18.701998       1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully
I1117 00:45:18.703582       1 azure_auth.go:245] Using AzurePublicCloud environment
I1117 00:45:18.703835       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I1117 00:45:18.710064       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
... skipping 1371 lines ...
I1117 00:45:19.400577       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=2477&timeout=5m44s&timeoutSeconds=344&watch=true 200 OK in 0 milliseconds
I1117 00:45:19.400734       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=2451&timeout=8m10s&timeoutSeconds=490&watch=true 200 OK in 0 milliseconds
I1117 00:45:19.496031       1 shared_informer.go:285] caches populated
I1117 00:45:19.496070       1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-c551e0323ad732edee8d00639b70d4fc25a9b407 e2e-test
I1117 00:45:19.496083       1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider
I1117 00:45:19.499088       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I1117 00:45:19.499331       1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I1117 00:45:19.499349       1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider
I1117 00:45:19.499354       1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I1117 00:45:19.499375       1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully
I1117 00:45:19.500022       1 azure_auth.go:245] Using AzurePublicCloud environment
I1117 00:45:19.500041       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I1117 00:45:19.500046       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
... skipping 53 lines ...
I1117 00:45:19.626580       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes 201 Created in 6 milliseconds
I1117 00:45:19.630945       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 4 milliseconds
I1117 00:45:19.631233       1 azuredisk_v2.go:532]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="1c73e869-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).registerAzDriverNode" "latency"=12298833
I1117 00:45:19.631335       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:45:19Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="1c75cce9-6611-11ed-ae03-000d3a2cfaa5"
I1117 00:45:19.631457       1 server.go:117] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I1117 00:45:19.634679       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 409 Conflict in 3 milliseconds
E1117 00:45:19.634789       1 azuredisk_v2.go:591]  "msg"="Failed to update AzDriverNode status after creation" "error"="Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-agentpool1-18975173-vmss000002\": the object has been modified; please apply your changes to the latest version and try again" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="1c75cce9-6611-11ed-ae03-000d3a2cfaa5"
E1117 00:45:19.634843       1 azuredisk_v2.go:543]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-agentpool1-18975173-vmss000002\": the object has been modified; please apply your changes to the latest version and try again]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="1c75cce9-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=3524538
I1117 00:45:19.634859       1 azuredisk_v2.go:506] Starting heartbeat loop with initial delay 13.069s and frequency 30s
I1117 00:45:19.817680       1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I1117 00:45:19.817701       1 utils.go:79] GRPC request: {}
I1117 00:45:19.818843       1 utils.go:85] GRPC response: {"name":"test.csi.azure.com","vendor_version":"latest-v2-c551e0323ad732edee8d00639b70d4fc25a9b407"}
I1117 00:45:20.061214       1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I1117 00:45:20.061236       1 utils.go:79] GRPC request: {}
... skipping 17 lines ...
I1117 00:45:47.721569       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1c1d324e-f316-42e9-b7d6-082aa150705d/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1c1d324e-f316-42e9-b7d6-082aa150705d","csi.storage.k8s.io/pvc/name":"test.csi.azure.comxhp6r","csi.storage.k8s.io/pvc/namespace":"volume-8965","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-1c1d324e-f316-42e9-b7d6-082aa150705d"}
I1117 00:45:47.721905       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-1c1d324e-f316-42e9-b7d6-082aa150705d." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2d3412a3-6611-11ed-ae03-000d3a2cfaa5"
I1117 00:45:47.721960       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-1c1d324e-f316-42e9-b7d6-082aa150705d-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 00:45:49.399691       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:45:49.399785       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:45:49.399838       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:45:49.399842       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:45:49.521684       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:46:02.713477       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:46:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="36239c69-6611-11ed-ae03-000d3a2cfaa5"
I1117 00:46:02.721550       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 7 milliseconds
I1117 00:46:02.721744       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="36239c69-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8316990
I1117 00:46:03.357109       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:46:03.357137       1 utils.go:79] GRPC request: {}
I1117 00:46:03.357200       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:46:19.400551       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:46:19.400617       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:46:19.400640       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:46:19.400704       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:46:19.521908       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:46:20.931310       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:46:20.931404       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="2d3412a3-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1c1d324e-f316-42e9-b7d6-082aa150705d" "latency"=33209364452
I1117 00:46:20.931435       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2d3412a3-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=33209626954
I1117 00:46:21.570509       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 00:46:21.570538       1 utils.go:79] GRPC request: {}
I1117 00:46:21.570588       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 00:46:21.573629       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 35 lines ...
I1117 00:46:23.151660       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1c1d324e-f316-42e9-b7d6-082aa150705d/globalmount","target_path":"/var/lib/kubelet/pods/bb485aff-fa93-4618-8169-a0faeb9ea3a9/volumes/kubernetes.io~csi/pvc-1c1d324e-f316-42e9-b7d6-082aa150705d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1c1d324e-f316-42e9-b7d6-082aa150705d","csi.storage.k8s.io/pvc/name":"test.csi.azure.comxhp6r","csi.storage.k8s.io/pvc/namespace":"volume-8965","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-1c1d324e-f316-42e9-b7d6-082aa150705d"}
I1117 00:46:23.151839       1 nodeserver_v2.go:348] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1c1d324e-f316-42e9-b7d6-082aa150705d/globalmount at /var/lib/kubelet/pods/bb485aff-fa93-4618-8169-a0faeb9ea3a9/volumes/kubernetes.io~csi/pvc-1c1d324e-f316-42e9-b7d6-082aa150705d/mount
I1117 00:46:23.151853       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1c1d324e-f316-42e9-b7d6-082aa150705d/globalmount /var/lib/kubelet/pods/bb485aff-fa93-4618-8169-a0faeb9ea3a9/volumes/kubernetes.io~csi/pvc-1c1d324e-f316-42e9-b7d6-082aa150705d/mount)
I1117 00:46:23.152706       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1c1d324e-f316-42e9-b7d6-082aa150705d/globalmount /var/lib/kubelet/pods/bb485aff-fa93-4618-8169-a0faeb9ea3a9/volumes/kubernetes.io~csi/pvc-1c1d324e-f316-42e9-b7d6-082aa150705d/mount)
I1117 00:46:23.153508       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1c1d324e-f316-42e9-b7d6-082aa150705d/globalmount at /var/lib/kubelet/pods/bb485aff-fa93-4618-8169-a0faeb9ea3a9/volumes/kubernetes.io~csi/pvc-1c1d324e-f316-42e9-b7d6-082aa150705d/mount successfully
I1117 00:46:23.153523       1 utils.go:85] GRPC response: {}
I1117 00:46:31.190699       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:46:31.190759       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="4161b0d0-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ef74f3fd-7019-4e05-9136-69f1a0204d9a" "latency"=9615356786
I1117 00:46:31.190781       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4161b0d0-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9615582188
I1117 00:46:31.826499       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1117 00:46:31.826555       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 00:46:31.826584       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ef74f3fd-7019-4e05-9136-69f1a0204d9a/globalmount with mount options([])
I1117 00:46:31.826597       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 159 lines ...
I1117 00:47:15.176120       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6","csi.storage.k8s.io/pvc/name":"test.csi.azure.comhmgpk","csi.storage.k8s.io/pvc/namespace":"multivolume-308","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6"}
I1117 00:47:15.176770       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="61549156-6611-11ed-ae03-000d3a2cfaa5"
I1117 00:47:15.176896       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 00:47:19.401604       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:47:19.401680       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:47:19.401718       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:47:19.401806       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:47:19.523027       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:47:32.718563       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:47:32Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="6bc94cb5-6611-11ed-ae03-000d3a2cfaa5"
I1117 00:47:32.725390       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 6 milliseconds
I1117 00:47:32.725933       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="6bc94cb5-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7404780
I1117 00:47:33.356831       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:47:33.356858       1 utils.go:79] GRPC request: {}
I1117 00:47:33.356909       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:47:49.402654       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:47:49.402765       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:47:49.402815       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:47:49.402861       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:47:49.523266       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:48:00.793622       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:48:00.793693       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="61549156-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6" "latency"=45616719112
I1117 00:48:00.793722       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="61549156-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=45617427419
I1117 00:48:02.580129       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1117 00:48:02.580177       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 00:48:02.580204       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6/globalmount with mount options([])
I1117 00:48:02.580213       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 80 lines ...
W1117 00:48:17.860317       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/pods/475065e3-cd95-4593-a33d-1537ad2b2e42/volumes/kubernetes.io~csi/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6/mount" is not a mountpoint, deleting
I1117 00:48:17.860408       1 nodeserver_v2.go:375] NodeUnpublishVolume: unmount volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6 on /var/lib/kubelet/pods/475065e3-cd95-4593-a33d-1537ad2b2e42/volumes/kubernetes.io~csi/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6/mount successfully
I1117 00:48:17.860427       1 utils.go:85] GRPC response: {}
I1117 00:48:19.403101       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:48:19.403148       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:48:19.403114       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:48:19.403228       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:48:19.403283       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:48:19.524168       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:48:21.789241       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1117 00:48:21.789264       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/3d682f9d-6718-43f0-af2c-2ddd0354f054/volumes/kubernetes.io~csi/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6"}
I1117 00:48:21.789326       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6 on /var/lib/kubelet/pods/3d682f9d-6718-43f0-af2c-2ddd0354f054/volumes/kubernetes.io~csi/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6/mount
I1117 00:48:21.789355       1 mount_helper_common.go:99] "/var/lib/kubelet/pods/3d682f9d-6718-43f0-af2c-2ddd0354f054/volumes/kubernetes.io~csi/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6/mount" is a mountpoint, unmounting
I1117 00:48:21.789377       1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/3d682f9d-6718-43f0-af2c-2ddd0354f054/volumes/kubernetes.io~csi/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6/mount
... skipping 8 lines ...
I1117 00:48:21.891862       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6/globalmount
I1117 00:48:21.891897       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6/globalmount" is a mountpoint, unmounting
I1117 00:48:21.891909       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6/globalmount
W1117 00:48:21.894906       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6/globalmount" is not a mountpoint, deleting
I1117 00:48:21.894976       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-91b83672-8a00-4fdb-877f-ab5da1aa59c6/globalmount successfully
I1117 00:48:21.894990       1 utils.go:85] GRPC response: {}
I1117 00:48:25.150984       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:48:25.151047       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="7ef407c6-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c084ab74-6510-4c34-a53b-6bcab40eed77" "latency"=20275540621
I1117 00:48:25.151071       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7ef407c6-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20275800524
I1117 00:48:25.153941       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:48:25.154001       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="7f5027ef-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4ffe1453-84ac-4539-aabf-30311ae5d0e8" "latency"=19674791029
I1117 00:48:25.154034       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7f5027ef-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19675011031
I1117 00:48:25.748200       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1117 00:48:25.748249       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 00:48:25.748289       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4ffe1453-84ac-4539-aabf-30311ae5d0e8/globalmount with mount options([nouuid])
I1117 00:48:25.748312       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 112 lines ...
W1117 00:48:42.544851       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4ffe1453-84ac-4539-aabf-30311ae5d0e8/globalmount" is not a mountpoint, deleting
I1117 00:48:42.544906       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4ffe1453-84ac-4539-aabf-30311ae5d0e8/globalmount successfully
I1117 00:48:42.544916       1 utils.go:85] GRPC response: {}
I1117 00:48:49.403587       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:48:49.403667       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:48:49.403695       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:48:49.403798       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:48:49.525211       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:49:02.711104       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:49:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="a16d116d-6611-11ed-ae03-000d3a2cfaa5"
I1117 00:49:02.717818       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 6 milliseconds
I1117 00:49:02.718564       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="a16d116d-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7513877
I1117 00:49:03.357490       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:49:03.357512       1 utils.go:79] GRPC request: {}
I1117 00:49:03.357559       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:49:07.667506       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:49:07.667589       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="94811462-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3fae0f4f-fa5d-4d01-918b-f7a5bfbcf273" "latency"=26635547827
I1117 00:49:07.667622       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="94811462-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=26635811729
I1117 00:49:08.269796       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/
I1117 00:49:08.269850       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1117 00:49:08.269889       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3fae0f4f-fa5d-4d01-918b-f7a5bfbcf273/globalmount with mount options([])
I1117 00:49:08.269909       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3])
... skipping 66 lines ...
W1117 00:49:48.726610       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3fae0f4f-fa5d-4d01-918b-f7a5bfbcf273/globalmount" is not a mountpoint, deleting
I1117 00:49:48.726674       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3fae0f4f-fa5d-4d01-918b-f7a5bfbcf273/globalmount successfully
I1117 00:49:48.726690       1 utils.go:85] GRPC response: {}
I1117 00:49:49.405004       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:49:49.405045       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:49:49.405082       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:49:49.405159       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:49:49.526497       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:50:02.709290       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:50:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="c53010aa-6611-11ed-ae03-000d3a2cfaa5"
I1117 00:50:02.716883       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 7 milliseconds
I1117 00:50:02.717073       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="c53010aa-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7834083
I1117 00:50:03.356518       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:50:03.356545       1 utils.go:79] GRPC request: {}
I1117 00:50:03.356588       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:50:08.450671       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:50:08.450752       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="acf08269-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ef537bac-ad92-4df5-831b-bcae3bb40adb" "latency"=46423109412
I1117 00:50:08.450778       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="acf08269-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=46423389914
I1117 00:50:08.975832       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 00:50:08.975859       1 utils.go:79] GRPC request: {}
I1117 00:50:08.975905       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 00:50:08.978991       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 57 lines ...
W1117 00:50:17.851367       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ef537bac-ad92-4df5-831b-bcae3bb40adb/globalmount" is not a mountpoint, deleting
I1117 00:50:17.851430       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ef537bac-ad92-4df5-831b-bcae3bb40adb/globalmount successfully
I1117 00:50:17.851450       1 utils.go:85] GRPC response: {}
I1117 00:50:19.405559       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:50:19.405643       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:50:19.405674       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:50:19.405751       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:50:19.527201       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:50:24.400719       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 175 items received
I1117 00:50:24.405103       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=5445&timeout=7m14s&timeoutSeconds=434&watch=true 200 OK in 4 milliseconds
I1117 00:50:32.708669       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:50:32Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="d7119c39-6611-11ed-ae03-000d3a2cfaa5"
I1117 00:50:32.719028       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 10 milliseconds
I1117 00:50:32.719716       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="d7119c39-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11065721
I1117 00:50:33.357581       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:50:33.357606       1 utils.go:79] GRPC request: {}
I1117 00:50:33.357644       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:50:49.406318       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:50:49.406358       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:50:49.406396       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:50:49.406441       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:50:49.527909       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:50:59.631333       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:50:59.631404       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="c8ed0605-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d04cd0fb-eedd-4a63-b56b-3d0cb49a3786" "latency"=50650378798
I1117 00:50:59.631426       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c8ed0605-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=50650678601
I1117 00:51:01.368484       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 00:51:01.368537       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 00:51:01.368578       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d04cd0fb-eedd-4a63-b56b-3d0cb49a3786/globalmount with mount options([])
I1117 00:51:01.368595       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 74 lines ...
I1117 00:51:33.357973       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:51:33.358322       1 utils.go:79] GRPC request: {}
I1117 00:51:33.358390       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:51:49.408612       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:51:49.408653       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:51:49.408706       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:51:49.408772       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:51:49.529086       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:52:02.712025       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:52:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="0cb707b8-6612-11ed-ae03-000d3a2cfaa5"
I1117 00:52:02.718900       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 6 milliseconds
I1117 00:52:02.720319       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="0cb707b8-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8344191
I1117 00:52:03.357110       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:52:03.357135       1 utils.go:79] GRPC request: {}
I1117 00:52:03.357182       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:52:13.523087       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 23 items received
I1117 00:52:13.527650       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-18975173-vmss000002&resourceVersion=6501&timeout=7m27s&timeoutSeconds=447&watch=true 200 OK in 4 milliseconds
I1117 00:52:19.409631       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:52:19.409721       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:52:19.409754       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:52:19.409851       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:52:19.530060       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:52:32.716662       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:52:32Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="1e995e62-6612-11ed-ae03-000d3a2cfaa5"
I1117 00:52:32.724209       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 7 milliseconds
I1117 00:52:32.724422       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="1e995e62-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7843211
I1117 00:52:33.357105       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:52:33.357129       1 utils.go:79] GRPC request: {}
I1117 00:52:33.357171       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:52:49.410498       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:52:49.410526       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:52:49.410558       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:52:49.410613       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:52:49.530905       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:53:02.718633       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:53:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="307b4f99-6612-11ed-ae03-000d3a2cfaa5"
I1117 00:53:02.726529       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 7 milliseconds
I1117 00:53:02.727186       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="307b4f99-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8605791
I1117 00:53:03.357211       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:53:03.357238       1 utils.go:79] GRPC request: {}
I1117 00:53:03.357285       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:53:19.411258       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:53:19.411369       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:53:19.411371       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:53:19.411394       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:53:19.531721       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
E1117 00:53:21.816052       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="f457803a-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-29f538b3-3d88-41cd-848c-96e68a027152" "latency"=119995220178
E1117 00:53:21.816102       1 crdprovisioner.go:699]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f457803a-6611-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119995511181
E1117 00:53:21.816119       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-29f538b3-3d88-41cd-848c-96e68a027152) to be attached to node (k8s-agentpool1-18975173-vmss000002): context canceled
I1117 00:53:22.328753       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 00:53:22.328773       1 utils.go:79] GRPC request: {}
I1117 00:53:22.328809       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 00:53:22.332361       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 00:53:22.332381       1 utils.go:79] GRPC request: {}
I1117 00:53:22.332422       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 12 lines ...
I1117 00:53:33.357088       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:53:33.357111       1 utils.go:79] GRPC request: {}
I1117 00:53:33.357161       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:53:49.411861       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:53:49.411921       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:53:49.411946       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:53:49.412019       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:53:49.532294       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:02.718252       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:54:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="543e876c-6612-11ed-ae03-000d3a2cfaa5"
I1117 00:54:02.725801       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 7 milliseconds
I1117 00:54:02.726084       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="543e876c-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7865884
I1117 00:54:03.357177       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:54:03.357200       1 utils.go:79] GRPC request: {}
I1117 00:54:03.357244       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:54:19.412018       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:19.412113       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:19.412264       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:19.412514       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:54:19.532694       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:31.320899       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 123 items received
I1117 00:54:31.326735       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=6840&timeout=7m1s&timeoutSeconds=421&watch=true 200 OK in 5 milliseconds
I1117 00:54:32.709242       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:54:32Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="661ecaac-6612-11ed-ae03-000d3a2cfaa5"
I1117 00:54:32.714582       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 5 milliseconds
I1117 00:54:32.714778       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="661ecaac-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5566160
I1117 00:54:33.357289       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:54:33.357316       1 utils.go:79] GRPC request: {}
I1117 00:54:33.357370       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:54:49.413105       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:49.413180       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:49.413203       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:54:49.413273       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:54:49.533681       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:55:02.717777       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:55:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="7801bad5-6612-11ed-ae03-000d3a2cfaa5"
I1117 00:55:02.729159       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 11 milliseconds
I1117 00:55:02.729337       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="7801bad5-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11592027
I1117 00:55:03.357502       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:55:03.357527       1 utils.go:79] GRPC request: {}
I1117 00:55:03.357574       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:55:19.414059       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:55:19.414140       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:55:19.414165       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:55:19.414238       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:55:19.534485       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
E1117 00:55:22.329173       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="3c2c5fb3-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-29f538b3-3d88-41cd-848c-96e68a027152" "latency"=119994983160
E1117 00:55:22.329224       1 crdprovisioner.go:699]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3c2c5fb3-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119995314264
E1117 00:55:22.329239       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-29f538b3-3d88-41cd-848c-96e68a027152) to be attached to node (k8s-agentpool1-18975173-vmss000002): context deadline exceeded
I1117 00:55:23.411878       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 00:55:23.411899       1 utils.go:79] GRPC request: {}
I1117 00:55:23.411934       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 00:55:23.415550       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 00:55:23.415570       1 utils.go:79] GRPC request: {}
I1117 00:55:23.415613       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 10 lines ...
I1117 00:55:33.357447       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:55:33.357472       1 utils.go:79] GRPC request: {}
I1117 00:55:33.357507       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:55:49.414575       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:55:49.414682       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:55:49.414709       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:55:49.414835       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:55:49.535125       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:56:02.711860       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:56:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="9bc4190b-6612-11ed-ae03-000d3a2cfaa5"
I1117 00:56:02.718582       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 6 milliseconds
I1117 00:56:02.719539       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="9bc4190b-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7730683
I1117 00:56:03.356636       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:56:03.356664       1 utils.go:79] GRPC request: {}
I1117 00:56:03.356710       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:56:19.415185       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:56:19.415242       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:56:19.415301       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:56:19.415361       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:56:19.535689       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:56:20.571516       1 conditionwatcher.go:187] condition result: succeeded: false, error: rpc error: code = Unknown desc = context deadline exceeded
E1117 00:56:20.571639       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [rpc error: code = Unknown desc = context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="845841bc-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-29f538b3-3d88-41cd-848c-96e68a027152" "latency"=57153932772
E1117 00:56:20.571675       1 crdprovisioner.go:699]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [rpc error: code = Unknown desc = context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="845841bc-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=57154219575
E1117 00:56:20.571699       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-29f538b3-3d88-41cd-848c-96e68a027152) to be attached to node (k8s-agentpool1-18975173-vmss000002): rpc error: code = Unknown desc = context deadline exceeded
I1117 00:56:22.645109       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 00:56:22.645129       1 utils.go:79] GRPC request: {}
I1117 00:56:22.645172       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 00:56:22.651164       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 00:56:22.651180       1 utils.go:79] GRPC request: {}
I1117 00:56:22.651211       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 3 lines ...
I1117 00:56:22.652645       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 00:56:22.652661       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-29f538b3-3d88-41cd-848c-96e68a027152/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-29f538b3-3d88-41cd-848c-96e68a027152","csi.storage.k8s.io/pvc/name":"test.csi.azure.comfmfst","csi.storage.k8s.io/pvc/namespace":"multivolume-9729","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-29f538b3-3d88-41cd-848c-96e68a027152"}
I1117 00:56:22.652934       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-29f538b3-3d88-41cd-848c-96e68a027152." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a7a6db42-6612-11ed-ae03-000d3a2cfaa5"
I1117 00:56:22.661238       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-29f538b3-3d88-41cd-848c-96e68a027152-k8s-agentpool1-18975173-vmss000002-attachment/status 200 OK in 7 milliseconds
I1117 00:56:22.661424       1 azure_disk_utils.go:1084]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/request-id"="a7a6db42-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "latency"=8412691
I1117 00:56:22.661440       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-29f538b3-3d88-41cd-848c-96e68a027152-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 00:56:22.666542       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:56:22.772120       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:56:23.787804       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:56:23.787874       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="a7a6db42-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-29f538b3-3d88-41cd-848c-96e68a027152" "latency"=1126368422
I1117 00:56:23.787905       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a7a6db42-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1135081916
I1117 00:56:32.709264       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:56:32Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="ada5554c-6612-11ed-ae03-000d3a2cfaa5"
I1117 00:56:32.714725       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 5 milliseconds
I1117 00:56:32.715535       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="ada5554c-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6364569
I1117 00:56:33.357050       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
... skipping 103 lines ...
I1117 00:58:32.708998       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T00:58:32Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="f52bd904-6612-11ed-ae03-000d3a2cfaa5"
I1117 00:58:32.715766       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 6 milliseconds
I1117 00:58:32.715928       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="f52bd904-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6963075
I1117 00:58:33.356872       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 00:58:33.356897       1 utils.go:79] GRPC request: {}
I1117 00:58:33.356941       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:58:36.340232       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 00:58:36.340292       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="f517e856-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e5f14ac0-f8f6-47ce-909d-dc6a7348945d" "latency"=3761777593
I1117 00:58:36.340315       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f517e856-6612-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3762040695
I1117 00:58:38.065156       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 00:58:38.065207       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 00:58:38.065243       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e5f14ac0-f8f6-47ce-909d-dc6a7348945d/globalmount with mount options([nouuid])
I1117 00:58:38.065264       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 161 lines ...
I1117 00:59:33.356845       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 00:59:40.527844       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 23 items received
I1117 00:59:40.531605       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-18975173-vmss000002&resourceVersion=9634&timeout=6m52s&timeoutSeconds=412&watch=true 200 OK in 3 milliseconds
I1117 00:59:49.420862       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:59:49.420894       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:59:49.420924       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 00:59:49.420982       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 00:59:49.541205       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:00:00.405615       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 88 items received
I1117 01:00:00.408240       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=9484&timeout=9m44s&timeoutSeconds=584&watch=true 200 OK in 2 milliseconds
I1117 01:00:02.710686       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:00:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="2ad1040b-6613-11ed-ae03-000d3a2cfaa5"
I1117 01:00:02.716791       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 5 milliseconds
I1117 01:00:02.716934       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="2ad1040b-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6277066
I1117 01:00:03.357037       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:00:03.357065       1 utils.go:79] GRPC request: {}
I1117 01:00:03.357103       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:00:19.421689       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:00:19.421790       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:00:19.421812       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:00:19.421827       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:00:19.541458       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:00:25.289759       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:00:25.289844       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="138577e7-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-68cb52cc-7f8e-45a4-b8fb-bc23039b9522" "latency"=61661709580
I1117 01:00:25.289869       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="138577e7-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=61661929783
I1117 01:00:27.151500       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1117 01:00:27.151549       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 01:00:27.151561       1 utils.go:85] GRPC response: {}
I1117 01:00:27.157349       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 25 lines ...
I1117 01:00:35.149095       1 utils.go:79] GRPC request: {}
I1117 01:00:35.149128       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:00:35.149839       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:00:35.149856       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b8eb8c28-11b8-4928-a473-2c1403f2bc55/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b8eb8c28-11b8-4928-a473-2c1403f2bc55","csi.storage.k8s.io/pvc/name":"test.csi.azure.commplpt","csi.storage.k8s.io/pvc/namespace":"multivolume-288","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-b8eb8c28-11b8-4928-a473-2c1403f2bc55"}
I1117 01:00:35.150146       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-b8eb8c28-11b8-4928-a473-2c1403f2bc55." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3e26df34-6613-11ed-ae03-000d3a2cfaa5"
I1117 01:00:35.150212       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-b8eb8c28-11b8-4928-a473-2c1403f2bc55-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:00:36.998204       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:00:36.998298       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="3e26df34-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b8eb8c28-11b8-4928-a473-2c1403f2bc55" "latency"=1848006060
I1117 01:00:36.998378       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3e26df34-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1848348064
I1117 01:00:37.613162       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1117 01:00:37.613207       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:00:37.613233       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b8eb8c28-11b8-4928-a473-2c1403f2bc55/globalmount with mount options([])
I1117 01:00:37.613243       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 117 lines ...
I1117 01:01:16.032331       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-ef61a4b7-d070-4b4d-b3af-e97c74b47738","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ef61a4b7-d070-4b4d-b3af-e97c74b47738","csi.storage.k8s.io/pvc/name":"pvc-29pv9","csi.storage.k8s.io/pvc/namespace":"provisioning-4016","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-ef61a4b7-d070-4b4d-b3af-e97c74b47738"}
I1117 01:01:16.032643       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-ef61a4b7-d070-4b4d-b3af-e97c74b47738." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="56850bb6-6613-11ed-ae03-000d3a2cfaa5"
I1117 01:01:16.032711       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-ef61a4b7-d070-4b4d-b3af-e97c74b47738-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:01:19.423252       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:01:19.423293       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:01:19.423370       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:01:19.423409       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:01:19.541656       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:01:32.328201       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 90 items received
I1117 01:01:32.332719       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=10803&timeout=9m9s&timeoutSeconds=549&watch=true 200 OK in 4 milliseconds
I1117 01:01:32.709403       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:01:32Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="6075b841-6613-11ed-ae03-000d3a2cfaa5"
I1117 01:01:32.714795       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 5 milliseconds
I1117 01:01:32.715465       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="6075b841-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6165362
I1117 01:01:33.357611       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:01:33.357639       1 utils.go:79] GRPC request: {}
I1117 01:01:33.357686       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:01:47.580982       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:01:47.581047       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="56850bb6-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ef61a4b7-d070-4b4d-b3af-e97c74b47738" "latency"=31548271233
I1117 01:01:47.581069       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="56850bb6-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31548554337
I1117 01:01:48.208961       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/
I1117 01:01:48.209004       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1117 01:01:48.209015       1 utils.go:85] GRPC response: {}
I1117 01:01:48.212767       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 22 lines ...
I1117 01:01:48.837926       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /dev/disk/azure/scsi1/lun2 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-ef61a4b7-d070-4b4d-b3af-e97c74b47738/7e648ef6-c38f-45b4-9f17-5bab80e33baf)
I1117 01:01:48.838812       1 nodeserver_v2.go:353] NodePublishVolume: mount /dev/disk/azure/scsi1/lun2 at /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-ef61a4b7-d070-4b4d-b3af-e97c74b47738/7e648ef6-c38f-45b4-9f17-5bab80e33baf successfully
I1117 01:01:48.838832       1 utils.go:85] GRPC response: {}
I1117 01:01:49.424097       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:01:49.424121       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:01:49.424110       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:01:49.424279       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:01:49.541938       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:02:01.839408       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1117 01:02:01.839431       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-ef61a4b7-d070-4b4d-b3af-e97c74b47738/7e648ef6-c38f-45b4-9f17-5bab80e33baf","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-ef61a4b7-d070-4b4d-b3af-e97c74b47738"}
I1117 01:02:01.839484       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-ef61a4b7-d070-4b4d-b3af-e97c74b47738 on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-ef61a4b7-d070-4b4d-b3af-e97c74b47738/7e648ef6-c38f-45b4-9f17-5bab80e33baf
I1117 01:02:01.839516       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-ef61a4b7-d070-4b4d-b3af-e97c74b47738/7e648ef6-c38f-45b4-9f17-5bab80e33baf" is a mountpoint, unmounting
I1117 01:02:01.839528       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-ef61a4b7-d070-4b4d-b3af-e97c74b47738/7e648ef6-c38f-45b4-9f17-5bab80e33baf
... skipping 15 lines ...
I1117 01:02:03.356758       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:02:03.356782       1 utils.go:79] GRPC request: {}
I1117 01:02:03.356831       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:02:19.424526       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:02:19.424572       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:02:19.424616       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:02:19.424642       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:02:19.545628       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:02:32.710445       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:02:32Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="84392965-6613-11ed-ae03-000d3a2cfaa5"
I1117 01:02:32.718871       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 8 milliseconds
I1117 01:02:32.719004       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="84392965-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8614291
I1117 01:02:33.357482       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:02:33.357509       1 utils.go:79] GRPC request: {}
I1117 01:02:33.357556       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:02:41.410759       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 163 items received
I1117 01:02:41.412754       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=11573&timeout=7m47s&timeoutSeconds=467&watch=true 200 OK in 1 milliseconds
I1117 01:02:49.425433       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:02:49.425531       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:02:49.425554       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:02:49.425704       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:02:49.545897       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:02:58.812013       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:02:58.812092       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="69c19935-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-93f9fba4-0454-41a9-a120-684c9d272afa" "latency"=70505777365
I1117 01:02:58.812116       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="69c19935-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=70506050969
I1117 01:02:59.713335       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 01:02:59.713371       1 utils.go:79] GRPC request: {}
I1117 01:02:59.713336       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 01:02:59.713410       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 50 lines ...
I1117 01:03:02.721364       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="961aa8e8-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11861130
I1117 01:03:03.357703       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:03:03.357727       1 utils.go:79] GRPC request: {}
I1117 01:03:03.357773       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:03:10.442553       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 57 items received
I1117 01:03:10.465921       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=12058&timeout=5m4s&timeoutSeconds=304&watch=true 200 OK in 23 milliseconds
I1117 01:03:15.824251       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:03:15.824334       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="9451e1dc-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-98021c24-9d56-41ac-997f-11f86fd27d63" "latency"=16108102974
I1117 01:03:15.824389       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9451e1dc-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16108429177
I1117 01:03:15.827323       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:03:15.827392       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="945203ed-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-199eac27-a3f8-4625-b3f0-503fd0673d0f" "latency"=16110358798
I1117 01:03:15.827420       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="945203ed-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16110597200
I1117 01:03:16.448307       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/
I1117 01:03:16.448392       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1117 01:03:16.448408       1 utils.go:85] GRPC response: {}
I1117 01:03:16.448498       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/
... skipping 118 lines ...
I1117 01:04:32.709726       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:04:32Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="cbbf99b8-6613-11ed-ae03-000d3a2cfaa5"
I1117 01:04:32.716317       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 6 milliseconds
I1117 01:04:32.717041       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="cbbf99b8-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7330478
I1117 01:04:33.357408       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:04:33.357430       1 utils.go:79] GRPC request: {}
I1117 01:04:33.357471       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:04:38.627347       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:04:38.627418       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="c9c78f92-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6b4e1af0-f1ff-4628-afb1-1631edb4578d" "latency"=9220768145
I1117 01:04:38.627440       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c9c78f92-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9221026948
I1117 01:04:39.583849       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 01:04:39.583869       1 utils.go:79] GRPC request: {}
I1117 01:04:39.583901       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:04:39.586800       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 67 lines ...
I1117 01:04:45.522687       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6b4e1af0-f1ff-4628-afb1-1631edb4578d/globalmount
I1117 01:04:45.522723       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6b4e1af0-f1ff-4628-afb1-1631edb4578d/globalmount" is a mountpoint, unmounting
I1117 01:04:45.522736       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6b4e1af0-f1ff-4628-afb1-1631edb4578d/globalmount
W1117 01:04:45.535006       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6b4e1af0-f1ff-4628-afb1-1631edb4578d/globalmount" is not a mountpoint, deleting
I1117 01:04:45.535090       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6b4e1af0-f1ff-4628-afb1-1631edb4578d/globalmount successfully
I1117 01:04:45.535111       1 utils.go:85] GRPC response: {}
I1117 01:04:48.923861       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:04:48.923925       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="cfd93c54-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-896e0231-1fb2-4b5a-af9b-e9fdfc30c9fe" "latency"=9335109938
I1117 01:04:48.923946       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cfd93c54-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9335370041
I1117 01:04:48.927041       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:04:48.927093       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="cfe87f31-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8e0394b3-85e0-4a14-9636-e271dc5214b7" "latency"=9238297502
I1117 01:04:48.927119       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cfe87f31-6613-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9238522605
I1117 01:04:49.427447       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:04:49.427455       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:04:49.427516       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:04:49.537382       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/
... skipping 271 lines ...
I1117 01:06:02.715121       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 5 milliseconds
I1117 01:06:02.715999       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="01647b60-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6491870
I1117 01:06:03.357458       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:06:03.357478       1 utils.go:79] GRPC request: {}
I1117 01:06:03.357515       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:06:19.430108       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:06:19.430212       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:06:19.430219       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:06:19.430229       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:06:19.430242       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:06:19.551567       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:06:32.532778       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 20 items received
I1117 01:06:32.535430       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-18975173-vmss000002&resourceVersion=13992&timeout=9m54s&timeoutSeconds=594&watch=true 200 OK in 2 milliseconds
I1117 01:06:32.708844       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:06:32Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="13460286-6614-11ed-ae03-000d3a2cfaa5"
I1117 01:06:32.714711       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 5 milliseconds
I1117 01:06:32.714868       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="13460286-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6072266
I1117 01:06:33.358603       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:06:33.358627       1 utils.go:79] GRPC request: {}
I1117 01:06:33.358674       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:06:40.417948       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:06:40.418034       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="002e0a60-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c1b330b7-6f04-435d-bae9-03b2485dee37" "latency"=39742755395
I1117 01:06:40.418061       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="002e0a60-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=39743070499
I1117 01:06:40.418112       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:06:40.418173       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="000011e1-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9f864312-8dc2-44bc-80a2-e158b3843367" "latency"=40044233533
I1117 01:06:40.418205       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="000011e1-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=40044482535
I1117 01:06:41.023499       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1117 01:06:41.023561       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 01:06:41.023609       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9f864312-8dc2-44bc-80a2-e158b3843367/globalmount with mount options([nouuid])
I1117 01:06:41.023638       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
... skipping 113 lines ...
I1117 01:07:02.709385       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:07:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="2527b749-6614-11ed-ae03-000d3a2cfaa5"
I1117 01:07:02.716408       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 6 milliseconds
I1117 01:07:02.716804       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="2527b749-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7552183
I1117 01:07:03.356869       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:07:03.356896       1 utils.go:79] GRPC request: {}
I1117 01:07:03.356936       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:07:13.239212       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:07:13.239282       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="1f19a75c-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2d4bb2f3-ba7b-4beb-835d-7f849b46b549" "latency"=20688297256
I1117 01:07:13.239306       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1f19a75c-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20688553159
I1117 01:07:13.800815       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 01:07:13.800838       1 utils.go:79] GRPC request: {}
I1117 01:07:13.800922       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:07:13.804893       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 50 lines ...
I1117 01:07:14.343040       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2d4bb2f3-ba7b-4beb-835d-7f849b46b549/globalmount /var/lib/kubelet/pods/9ab45d0a-116e-4066-adb5-99d4a1c70d5d/volumes/kubernetes.io~csi/pvc-2d4bb2f3-ba7b-4beb-835d-7f849b46b549/mount)
I1117 01:07:14.343861       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2d4bb2f3-ba7b-4beb-835d-7f849b46b549/globalmount /var/lib/kubelet/pods/9ab45d0a-116e-4066-adb5-99d4a1c70d5d/volumes/kubernetes.io~csi/pvc-2d4bb2f3-ba7b-4beb-835d-7f849b46b549/mount)
I1117 01:07:14.344742       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2d4bb2f3-ba7b-4beb-835d-7f849b46b549/globalmount at /var/lib/kubelet/pods/9ab45d0a-116e-4066-adb5-99d4a1c70d5d/volumes/kubernetes.io~csi/pvc-2d4bb2f3-ba7b-4beb-835d-7f849b46b549/mount successfully
I1117 01:07:14.344767       1 utils.go:85] GRPC response: {}
I1117 01:07:19.431408       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:07:19.431497       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:07:19.431527       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:07:19.431563       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:07:19.431570       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:07:19.434663       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1117 01:07:19.434681       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/9ab45d0a-116e-4066-adb5-99d4a1c70d5d/volumes/kubernetes.io~csi/pvc-2d4bb2f3-ba7b-4beb-835d-7f849b46b549/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-2d4bb2f3-ba7b-4beb-835d-7f849b46b549"}
I1117 01:07:19.434747       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-2d4bb2f3-ba7b-4beb-835d-7f849b46b549 on /var/lib/kubelet/pods/9ab45d0a-116e-4066-adb5-99d4a1c70d5d/volumes/kubernetes.io~csi/pvc-2d4bb2f3-ba7b-4beb-835d-7f849b46b549/mount
I1117 01:07:19.434780       1 mount_helper_common.go:99] "/var/lib/kubelet/pods/9ab45d0a-116e-4066-adb5-99d4a1c70d5d/volumes/kubernetes.io~csi/pvc-2d4bb2f3-ba7b-4beb-835d-7f849b46b549/mount" is a mountpoint, unmounting
I1117 01:07:19.434794       1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/9ab45d0a-116e-4066-adb5-99d4a1c70d5d/volumes/kubernetes.io~csi/pvc-2d4bb2f3-ba7b-4beb-835d-7f849b46b549/mount
W1117 01:07:19.436315       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/pods/9ab45d0a-116e-4066-adb5-99d4a1c70d5d/volumes/kubernetes.io~csi/pvc-2d4bb2f3-ba7b-4beb-835d-7f849b46b549/mount" is not a mountpoint, deleting
... skipping 17 lines ...
I1117 01:07:33.357134       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:07:33.357167       1 utils.go:79] GRPC request: {}
I1117 01:07:33.357218       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:07:49.431730       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:07:49.431766       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:07:49.431753       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:07:49.431839       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:07:49.431882       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:07:49.554214       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:02.708937       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:08:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="48eaef47-6614-11ed-ae03-000d3a2cfaa5"
I1117 01:08:02.715234       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 6 milliseconds
I1117 01:08:02.715510       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="48eaef47-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6608070
I1117 01:08:03.356846       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:08:03.356866       1 utils.go:79] GRPC request: {}
I1117 01:08:03.356905       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:08:14.466107       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 47 items received
I1117 01:08:14.471730       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=14866&timeout=7m33s&timeoutSeconds=453&watch=true 200 OK in 5 milliseconds
I1117 01:08:19.432147       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:19.432188       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:19.432245       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:19.432264       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:08:19.432371       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:08:19.554962       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:24.326009       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:08:24.326079       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="2bd396c4-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5604c315-ddae-411d-bba0-ff70a6e9b171" "latency"=70423873843
I1117 01:08:24.326118       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2bd396c4-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=70424146245
I1117 01:08:24.328558       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:08:24.328614       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="2bc5035d-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a46cefb2-d98d-40f1-8679-02258d476416" "latency"=70521952318
I1117 01:08:24.328641       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2bc5035d-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=70522204222
I1117 01:08:24.956451       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/
I1117 01:08:24.956469       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdd under /dev/disk/azure/scsi1/
I1117 01:08:24.956495       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1117 01:08:24.956498       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType 
... skipping 210 lines ...
I1117 01:08:48.471251       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1117 01:08:48.471269       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-a46cefb2-d98d-40f1-8679-02258d476416","volume_path":"/var/lib/kubelet/pods/45950b36-32ad-4e3f-bdeb-0fb86cdcb149/volumes/kubernetes.io~csi/pvc-a46cefb2-d98d-40f1-8679-02258d476416/mount"}
I1117 01:08:48.471354       1 utils.go:85] GRPC response: {"usage":[{"available":5286957056,"total":5358223360,"unit":1,"used":71266304},{"available":2621436,"total":2621440,"unit":2,"used":4}]}
I1117 01:08:49.432869       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:49.432955       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:49.432961       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:08:49.433072       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:08:49.555333       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:09:02.485453       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1117 01:09:02.485492       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1117 01:09:02.485482       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/45950b36-32ad-4e3f-bdeb-0fb86cdcb149/volumes/kubernetes.io~csi/pvc-5604c315-ddae-411d-bba0-ff70a6e9b171/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-5604c315-ddae-411d-bba0-ff70a6e9b171"}
I1117 01:09:02.485533       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-5604c315-ddae-411d-bba0-ff70a6e9b171 on /var/lib/kubelet/pods/45950b36-32ad-4e3f-bdeb-0fb86cdcb149/volumes/kubernetes.io~csi/pvc-5604c315-ddae-411d-bba0-ff70a6e9b171/mount
I1117 01:09:02.485506       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/45950b36-32ad-4e3f-bdeb-0fb86cdcb149/volumes/kubernetes.io~csi/pvc-a46cefb2-d98d-40f1-8679-02258d476416/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-a46cefb2-d98d-40f1-8679-02258d476416"}
... skipping 33 lines ...
I1117 01:09:02.709191       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:09:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="6cae3e52-6614-11ed-ae03-000d3a2cfaa5"
I1117 01:09:02.715290       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 5 milliseconds
I1117 01:09:02.715531       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="6cae3e52-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6401869
I1117 01:09:03.357325       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:09:03.357345       1 utils.go:79] GRPC request: {}
I1117 01:09:03.357385       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:09:16.288537       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:09:16.288613       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="622ad4fa-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8d77d511-e96f-4a45-bb71-bd2982cac225" "latency"=31217605297
I1117 01:09:16.288637       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="622ad4fa-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31217957202
I1117 01:09:16.907572       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/
I1117 01:09:16.907617       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:09:16.907642       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8d77d511-e96f-4a45-bb71-bd2982cac225/globalmount with mount options([])
I1117 01:09:16.907651       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 78 lines ...
I1117 01:10:02.718077       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="90720079-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5762856
I1117 01:10:03.357687       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:10:03.357708       1 utils.go:79] GRPC request: {}
I1117 01:10:03.357757       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:10:19.434720       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:19.434787       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:19.434812       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:10:19.434907       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:19.557304       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:28.413710       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 299 items received
I1117 01:10:28.417511       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=16137&timeout=6m30s&timeoutSeconds=390&watch=true 200 OK in 3 milliseconds
I1117 01:10:32.709102       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:10:32Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="a253241f-6614-11ed-ae03-000d3a2cfaa5"
I1117 01:10:32.719422       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 10 milliseconds
I1117 01:10:32.720233       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="a253241f-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11185719
I1117 01:10:33.356610       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:10:33.356634       1 utils.go:79] GRPC request: {}
I1117 01:10:33.356672       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:10:41.333759       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 137 items received
I1117 01:10:41.338779       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=16325&timeout=5m5s&timeoutSeconds=305&watch=true 200 OK in 4 milliseconds
I1117 01:10:43.575565       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:10:43.575648       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="8cb62d50-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f243dc11-7088-4c52-be64-3b468a62173b" "latency"=47127084759
I1117 01:10:43.575681       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8cb62d50-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=47127473964
I1117 01:10:45.373698       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1117 01:10:45.373746       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 01:10:45.373774       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f243dc11-7088-4c52-be64-3b468a62173b/globalmount with mount options([])
I1117 01:10:45.373783       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 44 lines ...
I1117 01:10:48.470924       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1117 01:10:48.470943       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-f243dc11-7088-4c52-be64-3b468a62173b","volume_path":"/var/lib/kubelet/pods/f1bb6388-0256-426e-a131-cae290886790/volumes/kubernetes.io~csi/pvc-f243dc11-7088-4c52-be64-3b468a62173b/mount"}
I1117 01:10:48.471028       1 utils.go:85] GRPC response: {"usage":[{"available":5179576320,"total":5196382208,"unit":1,"used":28672},{"available":327668,"total":327680,"unit":2,"used":12}]}
I1117 01:10:49.435621       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:49.435694       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:49.435714       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:49.435799       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:10:49.558113       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:10:52.640976       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1117 01:10:52.640997       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/f1bb6388-0256-426e-a131-cae290886790/volumes/kubernetes.io~csi/pvc-f243dc11-7088-4c52-be64-3b468a62173b/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-f243dc11-7088-4c52-be64-3b468a62173b"}
I1117 01:10:52.641051       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-f243dc11-7088-4c52-be64-3b468a62173b on /var/lib/kubelet/pods/f1bb6388-0256-426e-a131-cae290886790/volumes/kubernetes.io~csi/pvc-f243dc11-7088-4c52-be64-3b468a62173b/mount
I1117 01:10:52.641079       1 mount_helper_common.go:99] "/var/lib/kubelet/pods/f1bb6388-0256-426e-a131-cae290886790/volumes/kubernetes.io~csi/pvc-f243dc11-7088-4c52-be64-3b468a62173b/mount" is a mountpoint, unmounting
I1117 01:10:52.641094       1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/f1bb6388-0256-426e-a131-cae290886790/volumes/kubernetes.io~csi/pvc-f243dc11-7088-4c52-be64-3b468a62173b/mount
... skipping 8 lines ...
I1117 01:10:52.740932       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f243dc11-7088-4c52-be64-3b468a62173b/globalmount
I1117 01:10:52.740960       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f243dc11-7088-4c52-be64-3b468a62173b/globalmount" is a mountpoint, unmounting
I1117 01:10:52.740968       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f243dc11-7088-4c52-be64-3b468a62173b/globalmount
W1117 01:10:52.753639       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f243dc11-7088-4c52-be64-3b468a62173b/globalmount" is not a mountpoint, deleting
I1117 01:10:52.753719       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f243dc11-7088-4c52-be64-3b468a62173b/globalmount successfully
I1117 01:10:52.753737       1 utils.go:85] GRPC response: {}
I1117 01:10:58.906138       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:10:58.906202       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="ab1755dc-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-adf4abfd-583d-46c7-a5bc-72a37b40dd1e" "latency"=11489388413
I1117 01:10:58.906224       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ab1755dc-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11489640117
I1117 01:10:59.518410       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1117 01:10:59.518462       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:10:59.518493       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-adf4abfd-583d-46c7-a5bc-72a37b40dd1e/globalmount with mount options([])
I1117 01:10:59.518503       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 63 lines ...
I1117 01:11:18.029847       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a05062d9-cf29-469c-a1dc-452d9110b924/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a05062d9-cf29-469c-a1dc-452d9110b924","csi.storage.k8s.io/pvc/name":"inline-volume-tester-zd2j7-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-1296","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-a05062d9-cf29-469c-a1dc-452d9110b924"}
I1117 01:11:18.031381       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-a05062d9-cf29-469c-a1dc-452d9110b924." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bd569399-6614-11ed-ae03-000d3a2cfaa5"
I1117 01:11:18.031720       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a05062d9-cf29-469c-a1dc-452d9110b924-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:11:19.436533       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:11:19.436623       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:11:19.436639       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:11:19.436703       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:11:19.558899       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:11:32.711429       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:11:32Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="c616c539-6614-11ed-ae03-000d3a2cfaa5"
I1117 01:11:32.720393       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 8 milliseconds
I1117 01:11:32.720603       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="c616c539-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9228201
I1117 01:11:33.357525       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:11:33.357549       1 utils.go:79] GRPC request: {}
I1117 01:11:33.357599       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:11:49.437078       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:11:49.437154       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:11:49.437171       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:11:49.437349       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:11:49.559477       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:11:55.143864       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:11:55.143934       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="bd569399-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a05062d9-cf29-469c-a1dc-452d9110b924" "latency"=37112127507
I1117 01:11:55.143960       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bd569399-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=37113929426
I1117 01:11:55.699856       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1117 01:11:55.699881       1 utils.go:79] GRPC request: {}
I1117 01:11:55.699919       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:11:55.703174       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 44 lines ...
I1117 01:12:03.359220       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:12:03.359246       1 utils.go:79] GRPC request: {}
I1117 01:12:03.359295       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:12:19.437738       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:12:19.437801       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:12:19.437807       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:12:19.437892       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:12:19.560387       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:12:20.752202       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:12:20.752269       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="d3cb4bca-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f7c84f11-600f-4d2e-8e5a-70db72455a1d" "latency"=25047179599
I1117 01:12:20.752292       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d3cb4bca-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=25047454601
I1117 01:12:21.387068       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1117 01:12:21.387119       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:12:21.387147       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f7c84f11-600f-4d2e-8e5a-70db72455a1d/globalmount with mount options([nouuid])
I1117 01:12:21.387159       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 99 lines ...
I1117 01:13:02.716818       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="fbbb5233-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7799774
I1117 01:13:03.356791       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:13:03.356815       1 utils.go:79] GRPC request: {}
I1117 01:13:03.356862       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:13:19.439321       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:13:19.439375       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:13:19.439460       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:13:19.439464       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:13:19.560722       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:13:32.708967       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:13:32Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="0d9cf134-6615-11ed-ae03-000d3a2cfaa5"
I1117 01:13:32.716168       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 6 milliseconds
I1117 01:13:32.716416       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="0d9cf134-6615-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7493181
I1117 01:13:33.357558       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:13:33.357584       1 utils.go:79] GRPC request: {}
I1117 01:13:33.357633       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:13:36.785156       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:13:36.785242       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="fa4a4e47-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d2eba6fa-cd15-462f-a975-a1811e68f904" "latency"=36494397430
I1117 01:13:36.785268       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="fa4a4e47-6614-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=36494637732
I1117 01:13:37.396664       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/
I1117 01:13:37.396721       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1117 01:13:37.396735       1 utils.go:85] GRPC response: {}
I1117 01:13:37.401570       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 70 lines ...
I1117 01:14:03.357615       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:14:03.357648       1 utils.go:79] GRPC request: {}
I1117 01:14:03.357694       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:14:19.439873       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:14:19.439961       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:14:19.440054       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:14:19.440136       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:14:19.561310       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:14:25.172861       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1117 01:14:25.172888       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/083373fb-e407-4e98-a58e-9a4336a45096/volumes/kubernetes.io~csi/pvc-a05062d9-cf29-469c-a1dc-452d9110b924/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-a05062d9-cf29-469c-a1dc-452d9110b924"}
I1117 01:14:25.172953       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-a05062d9-cf29-469c-a1dc-452d9110b924 on /var/lib/kubelet/pods/083373fb-e407-4e98-a58e-9a4336a45096/volumes/kubernetes.io~csi/pvc-a05062d9-cf29-469c-a1dc-452d9110b924/mount
I1117 01:14:25.172999       1 mount_helper_common.go:99] "/var/lib/kubelet/pods/083373fb-e407-4e98-a58e-9a4336a45096/volumes/kubernetes.io~csi/pvc-a05062d9-cf29-469c-a1dc-452d9110b924/mount" is a mountpoint, unmounting
I1117 01:14:25.173010       1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/083373fb-e407-4e98-a58e-9a4336a45096/volumes/kubernetes.io~csi/pvc-a05062d9-cf29-469c-a1dc-452d9110b924/mount
... skipping 14 lines ...
I1117 01:14:32.709184       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:14:32Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="31603e14-6615-11ed-ae03-000d3a2cfaa5"
I1117 01:14:32.717475       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 7 milliseconds
I1117 01:14:32.717687       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="31603e14-6615-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8592189
I1117 01:14:33.357404       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:14:33.357424       1 utils.go:79] GRPC request: {}
I1117 01:14:33.357468       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:14:37.552496       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:14:37.552566       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="1ee18ea2-6615-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2ee3ccdc-6998-4081-be91-3595fa724731" "latency"=35872480392
I1117 01:14:37.552589       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1ee18ea2-6615-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=35872741794
I1117 01:14:38.147698       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/
I1117 01:14:38.147748       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:14:38.147759       1 utils.go:85] GRPC response: {}
I1117 01:14:38.154558       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 73 lines ...
I1117 01:15:03.357272       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:15:03.357301       1 utils.go:79] GRPC request: {}
I1117 01:15:03.357354       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:15:19.441051       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:15:19.441139       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:15:19.441157       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:15:19.441261       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:15:19.441313       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:15:19.562623       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:15:23.544909       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:15:23.544983       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="439955c8-6615-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f5d602df-c5dc-4eaf-8763-1754ba83ff63" "latency"=20262132302
I1117 01:15:23.545008       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="439955c8-6615-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20262774109
I1117 01:15:23.550432       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:15:23.550495       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="43995582-6615-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e858c07c-44e2-44c9-b715-8f7dadbb58d4" "latency"=20267841859
I1117 01:15:23.550522       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="43995582-6615-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20268296064
I1117 01:15:24.158930       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdc under /dev/disk/azure/scsi1/
I1117 01:15:24.158981       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType 
I1117 01:15:24.158994       1 utils.go:85] GRPC response: {}
I1117 01:15:24.168663       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/
... skipping 135 lines ...
I1117 01:17:36.426034       1 utils.go:79] GRPC request: {}
I1117 01:17:36.426061       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:17:36.426845       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:17:36.426860       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-20d703fa-9f23-4a27-bbb8-8e3998406a4f/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-20d703fa-9f23-4a27-bbb8-8e3998406a4f","csi.storage.k8s.io/pvc/name":"test.csi.azure.comq6bmp","csi.storage.k8s.io/pvc/namespace":"snapshotting-8406","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-20d703fa-9f23-4a27-bbb8-8e3998406a4f"}
I1117 01:17:36.427136       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-20d703fa-9f23-4a27-bbb8-8e3998406a4f." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9ee15fa5-6615-11ed-ae03-000d3a2cfaa5"
I1117 01:17:36.427182       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-20d703fa-9f23-4a27-bbb8-8e3998406a4f-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:17:45.848871       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:17:45.848950       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="9ee15fa5-6615-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-20d703fa-9f23-4a27-bbb8-8e3998406a4f" "latency"=9421691627
I1117 01:17:45.848979       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9ee15fa5-6615-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9421964131
I1117 01:17:47.561962       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:17:47.562013       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:17:47.562054       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-20d703fa-9f23-4a27-bbb8-8e3998406a4f/globalmount with mount options([])
I1117 01:17:47.562073       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 109 lines ...
I1117 01:19:19.701612       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:19:19.701634       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e5076d3a-3a06-4888-8707-585bb14baa4b/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e5076d3a-3a06-4888-8707-585bb14baa4b","csi.storage.k8s.io/pvc/name":"pvc-s7vxl","csi.storage.k8s.io/pvc/namespace":"provisioning-5631","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-e5076d3a-3a06-4888-8707-585bb14baa4b"}
I1117 01:19:19.701914       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-e5076d3a-3a06-4888-8707-585bb14baa4b." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="dc6fdb1a-6615-11ed-ae03-000d3a2cfaa5"
I1117 01:19:19.701975       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e5076d3a-3a06-4888-8707-585bb14baa4b-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:19:27.412522       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 231 items received
I1117 01:19:27.415344       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=21073&timeout=8m54s&timeoutSeconds=534&watch=true 200 OK in 2 milliseconds
I1117 01:19:29.962949       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:19:29.963022       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="dc601c67-6615-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-403c02bc-e0e8-4a48-9036-e52fc81b9382" "latency"=10363979370
I1117 01:19:29.963048       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="dc601c67-6615-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10364434575
I1117 01:19:29.965933       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:19:29.965980       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="dc6fdb1a-6615-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e5076d3a-3a06-4888-8707-585bb14baa4b" "latency"=10263953101
I1117 01:19:29.966000       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="dc6fdb1a-6615-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10264203403
I1117 01:19:30.593934       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1117 01:19:30.593980       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 01:19:30.594010       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-403c02bc-e0e8-4a48-9036-e52fc81b9382/globalmount with mount options([])
I1117 01:19:30.594021       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 132 lines ...
I1117 01:20:33.356939       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:20:33.356966       1 utils.go:79] GRPC request: {}
I1117 01:20:33.357013       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:20:49.450243       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:20:49.450273       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:20:49.450361       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:20:49.450365       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:20:49.571699       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:20:50.253411       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:20:50.253482       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="06c048e6-6616-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1deb0604-15a9-4976-bf6c-bc4a9ca2397d" "latency"=19560065623
I1117 01:20:50.253508       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="06c048e6-6616-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19560300525
I1117 01:20:51.924957       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:20:51.925007       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:20:51.925034       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1deb0604-15a9-4976-bf6c-bc4a9ca2397d/globalmount with mount options([])
I1117 01:20:51.925044       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 82 lines ...
I1117 01:22:13.898953       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-96201739-a90f-47cf-b04b-cedc267d5ff1","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-96201739-a90f-47cf-b04b-cedc267d5ff1","csi.storage.k8s.io/pvc/name":"test.csi.azure.comn5qmp","csi.storage.k8s.io/pvc/namespace":"multivolume-285","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-96201739-a90f-47cf-b04b-cedc267d5ff1"}
I1117 01:22:13.899209       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-96201739-a90f-47cf-b04b-cedc267d5ff1." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="44444167-6616-11ed-ae03-000d3a2cfaa5"
I1117 01:22:13.899260       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-96201739-a90f-47cf-b04b-cedc267d5ff1-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:22:19.452000       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:22:19.452092       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:22:19.452092       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:22:19.452182       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:22:19.573434       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:22:23.493177       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:22:23.493266       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="44444167-6616-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-96201739-a90f-47cf-b04b-cedc267d5ff1" "latency"=9593911444
I1117 01:22:23.493296       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="44444167-6616-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9594189348
I1117 01:22:25.369993       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:22:25.370050       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:22:25.370064       1 utils.go:85] GRPC response: {}
I1117 01:22:25.376569       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 118 lines ...
I1117 01:24:43.279598       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4cd1982f-148b-467e-856c-edfcb13559ca/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4cd1982f-148b-467e-856c-edfcb13559ca","csi.storage.k8s.io/pvc/name":"test.csi.azure.comdfj6t","csi.storage.k8s.io/pvc/namespace":"multivolume-9387","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-4cd1982f-148b-467e-856c-edfcb13559ca"}
I1117 01:24:43.279877       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-4cd1982f-148b-467e-856c-edfcb13559ca." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9d4def45-6616-11ed-ae03-000d3a2cfaa5"
I1117 01:24:43.279934       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-4cd1982f-148b-467e-856c-edfcb13559ca-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:24:49.460312       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:24:49.460398       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:24:49.460407       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:24:49.460465       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:24:49.576911       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:25:02.712319       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:25:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="a8e31a27-6616-11ed-ae03-000d3a2cfaa5"
I1117 01:25:02.720075       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 7 milliseconds
I1117 01:25:02.720268       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="a8e31a27-6616-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7963893
I1117 01:25:03.357055       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:25:03.357083       1 utils.go:79] GRPC request: {}
I1117 01:25:03.357127       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:25:13.605622       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:25:13.605722       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="9d4def45-6616-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4cd1982f-148b-467e-856c-edfcb13559ca" "latency"=30325696589
I1117 01:25:13.605806       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9d4def45-6616-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=30326033093
I1117 01:25:15.297792       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:25:15.297843       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:25:15.297871       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4cd1982f-148b-467e-856c-edfcb13559ca/globalmount with mount options([])
I1117 01:25:15.297880       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 38 lines ...
I1117 01:25:27.202665       1 utils.go:79] GRPC request: {}
I1117 01:25:27.202702       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:25:27.203353       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:25:27.203375       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1148c0a5-9ae7-462c-a34e-328c59163696/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1148c0a5-9ae7-462c-a34e-328c59163696","csi.storage.k8s.io/pvc/name":"test.csi.azure.comdfj6t-restored","csi.storage.k8s.io/pvc/namespace":"multivolume-9387","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-1148c0a5-9ae7-462c-a34e-328c59163696"}
I1117 01:25:27.203646       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-1148c0a5-9ae7-462c-a34e-328c59163696." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b77c2b83-6616-11ed-ae03-000d3a2cfaa5"
I1117 01:25:27.203697       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-1148c0a5-9ae7-462c-a34e-328c59163696-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:25:30.472976       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:25:30.473068       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="b77c2b83-6616-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1148c0a5-9ae7-462c-a34e-328c59163696" "latency"=3269303202
I1117 01:25:30.473100       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b77c2b83-6616-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3269562705
I1117 01:25:31.087870       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1117 01:25:31.087926       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 01:25:31.087954       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1148c0a5-9ae7-462c-a34e-328c59163696/globalmount with mount options([])
I1117 01:25:31.087963       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 105 lines ...
I1117 01:27:02.711763       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:27:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="f0698fb8-6616-11ed-ae03-000d3a2cfaa5"
I1117 01:27:02.719621       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 7 milliseconds
I1117 01:27:02.720327       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="f0698fb8-6616-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8599387
I1117 01:27:03.357487       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:27:03.357514       1 utils.go:79] GRPC request: {}
I1117 01:27:03.357566       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:27:14.905470       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:27:14.905540       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="ee81e9ac-6616-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-41a4b67a-1711-44ab-8b37-7debc782eee2" "latency"=15389432771
I1117 01:27:14.905564       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ee81e9ac-6616-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15389709475
I1117 01:27:16.639620       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:27:16.639670       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:27:16.639682       1 utils.go:85] GRPC response: {}
I1117 01:27:16.645713       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 46 lines ...
I1117 01:27:49.296130       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-41a4b67a-1711-44ab-8b37-7debc782eee2","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-41a4b67a-1711-44ab-8b37-7debc782eee2","csi.storage.k8s.io/pvc/name":"test.csi.azure.com59g6v","csi.storage.k8s.io/pvc/namespace":"multivolume-1965","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-41a4b67a-1711-44ab-8b37-7debc782eee2"}
I1117 01:27:49.296402       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-41a4b67a-1711-44ab-8b37-7debc782eee2." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0c2dcd39-6617-11ed-ae03-000d3a2cfaa5"
I1117 01:27:49.296457       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-41a4b67a-1711-44ab-8b37-7debc782eee2-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:27:49.466485       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:27:49.466534       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:27:49.466599       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:27:49.466633       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:27:49.581015       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:27:58.892115       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:27:58.892199       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="0c2dcd39-6617-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-41a4b67a-1711-44ab-8b37-7debc782eee2" "latency"=9595669512
I1117 01:27:58.892229       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0c2dcd39-6617-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9595950115
I1117 01:28:00.685554       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:28:00.685621       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:28:00.685637       1 utils.go:85] GRPC response: {}
I1117 01:28:00.692876       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 22 lines ...
I1117 01:28:10.750004       1 utils.go:79] GRPC request: {}
I1117 01:28:10.750039       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:28:10.750772       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:28:10.750788       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-c167c53f-98c7-458a-803e-2b1a007db322","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c167c53f-98c7-458a-803e-2b1a007db322","csi.storage.k8s.io/pvc/name":"test.csi.azure.com59g6v-restored","csi.storage.k8s.io/pvc/namespace":"multivolume-1965","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-c167c53f-98c7-458a-803e-2b1a007db322"}
I1117 01:28:10.751047       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-c167c53f-98c7-458a-803e-2b1a007db322." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="18f785ff-6617-11ed-ae03-000d3a2cfaa5"
I1117 01:28:10.751101       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-c167c53f-98c7-458a-803e-2b1a007db322-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:28:17.338792       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:28:17.338877       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="18f785ff-6617-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c167c53f-98c7-458a-803e-2b1a007db322" "latency"=6587692744
I1117 01:28:17.338903       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="18f785ff-6617-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6587964746
I1117 01:28:17.962051       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1117 01:28:17.962109       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 01:28:17.962122       1 utils.go:85] GRPC response: {}
I1117 01:28:17.965598       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 147 lines ...
I1117 01:31:44.645485       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a717967c-db54-40d5-b3ab-334b2670365a-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:31:44.645602       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-8eb89517-0969-496d-aae3-3cfbc7938ab3." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="98753f13-6617-11ed-ae03-000d3a2cfaa5"
I1117 01:31:44.645648       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-8eb89517-0969-496d-aae3-3cfbc7938ab3-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:31:49.488082       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:31:49.488124       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:31:49.488148       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:31:49.488200       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:31:49.488225       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:31:49.585458       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:32:02.712494       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:32:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="a33a094f-6617-11ed-ae03-000d3a2cfaa5"
I1117 01:32:02.718214       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 5 milliseconds
I1117 01:32:02.718551       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="a33a094f-6617-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6122466
I1117 01:32:02.790084       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:32:02.790158       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="98753f13-6617-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8eb89517-0969-496d-aae3-3cfbc7938ab3" "latency"=18144456187
I1117 01:32:02.790186       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="98753f13-6617-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=18144665490
I1117 01:32:02.795670       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:32:02.795739       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="98753716-6617-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a717967c-db54-40d5-b3ab-334b2670365a" "latency"=18150179748
I1117 01:32:02.795771       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="98753716-6617-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=18150453151
I1117 01:32:03.356750       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:32:03.356773       1 utils.go:79] GRPC request: {}
I1117 01:32:03.356815       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:32:03.400099       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
... skipping 263 lines ...
I1117 01:33:47.187438       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1cb414fc-05bd-4083-9d65-9286f6463305/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1cb414fc-05bd-4083-9d65-9286f6463305","csi.storage.k8s.io/pvc/name":"test.csi.azure.commf7zx","csi.storage.k8s.io/pvc/namespace":"provisioning-7525","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-1cb414fc-05bd-4083-9d65-9286f6463305"}
I1117 01:33:47.187737       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-1cb414fc-05bd-4083-9d65-9286f6463305." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e17fafbf-6617-11ed-ae03-000d3a2cfaa5"
I1117 01:33:47.187788       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-1cb414fc-05bd-4083-9d65-9286f6463305-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:33:49.494145       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:33:49.494192       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:33:49.494215       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:33:49.494265       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:33:49.587553       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:33:59.757400       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:33:59.757476       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="e17fafbf-6617-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1cb414fc-05bd-4083-9d65-9286f6463305" "latency"=12569603253
I1117 01:33:59.757499       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e17fafbf-6617-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12569884755
I1117 01:34:01.520409       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:34:01.520464       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:34:01.520504       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1cb414fc-05bd-4083-9d65-9286f6463305/globalmount with mount options([])
I1117 01:34:01.520523       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 116 lines ...
I1117 01:36:21.687206       1 utils.go:79] GRPC request: {}
I1117 01:36:21.687239       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:36:21.688633       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:36:21.688652       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4e2aca38-c24f-4d5e-9ff5-3ae7d07589bf/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4e2aca38-c24f-4d5e-9ff5-3ae7d07589bf","csi.storage.k8s.io/pvc/name":"test.csi.azure.commpj6l","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-6687","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-4e2aca38-c24f-4d5e-9ff5-3ae7d07589bf"}
I1117 01:36:21.688970       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-4e2aca38-c24f-4d5e-9ff5-3ae7d07589bf." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3d96b327-6618-11ed-ae03-000d3a2cfaa5"
I1117 01:36:21.689027       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-4e2aca38-c24f-4d5e-9ff5-3ae7d07589bf-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:36:31.419390       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:36:31.419461       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="3d96b327-6618-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4e2aca38-c24f-4d5e-9ff5-3ae7d07589bf" "latency"=9730354325
I1117 01:36:31.419485       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3d96b327-6618-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9730662728
I1117 01:36:32.708993       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:36:32Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="44283be1-6618-11ed-ae03-000d3a2cfaa5"
I1117 01:36:32.716221       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 6 milliseconds
I1117 01:36:32.716834       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="44283be1-6618-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7899880
I1117 01:36:33.140022       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
... skipping 141 lines ...
I1117 01:37:41.950663       1 utils.go:79] GRPC request: {}
I1117 01:37:41.950697       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:37:41.951436       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:37:41.951449       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8a2e475f-39d2-46e8-92dc-7f57b6b3f9e3/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8a2e475f-39d2-46e8-92dc-7f57b6b3f9e3","csi.storage.k8s.io/pvc/name":"test.csi.azure.com5h8md","csi.storage.k8s.io/pvc/namespace":"multivolume-2002","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-8a2e475f-39d2-46e8-92dc-7f57b6b3f9e3"}
I1117 01:37:41.951744       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-8a2e475f-39d2-46e8-92dc-7f57b6b3f9e3." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6d6dd46b-6618-11ed-ae03-000d3a2cfaa5"
I1117 01:37:41.951797       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-8a2e475f-39d2-46e8-92dc-7f57b6b3f9e3-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:37:42.850668       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:37:42.850748       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="6d6dd46b-6618-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8a2e475f-39d2-46e8-92dc-7f57b6b3f9e3" "latency"=898854531
I1117 01:37:42.850781       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6d6dd46b-6618-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=899161934
I1117 01:37:44.579723       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:37:44.579772       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1117 01:37:44.579801       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8a2e475f-39d2-46e8-92dc-7f57b6b3f9e3/globalmount with mount options([])
I1117 01:37:44.579810       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 117 lines ...
I1117 01:39:34.206309       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c33d5085-5901-41e6-bb00-7c62089db7c8/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c33d5085-5901-41e6-bb00-7c62089db7c8","csi.storage.k8s.io/pvc/name":"test.csi.azure.comp8dpf","csi.storage.k8s.io/pvc/namespace":"multivolume-286","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-c33d5085-5901-41e6-bb00-7c62089db7c8"}
I1117 01:39:34.206380       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c7832345-a9ef-4eb9-8fdb-3128140a9723/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c7832345-a9ef-4eb9-8fdb-3128140a9723","csi.storage.k8s.io/pvc/name":"test.csi.azure.comcw47r","csi.storage.k8s.io/pvc/namespace":"multivolume-286","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-c7832345-a9ef-4eb9-8fdb-3128140a9723"}
I1117 01:39:34.206608       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-c33d5085-5901-41e6-bb00-7c62089db7c8." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b0568f9e-6618-11ed-ae03-000d3a2cfaa5"
I1117 01:39:34.206619       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-c7832345-a9ef-4eb9-8fdb-3128140a9723." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b05691f3-6618-11ed-ae03-000d3a2cfaa5"
I1117 01:39:34.206672       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-c7832345-a9ef-4eb9-8fdb-3128140a9723-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:39:34.206672       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-c33d5085-5901-41e6-bb00-7c62089db7c8-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:39:42.362318       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:39:42.362394       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="b05691f3-6618-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c7832345-a9ef-4eb9-8fdb-3128140a9723" "latency"=8155633197
I1117 01:39:42.362425       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b05691f3-6618-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8155892500
I1117 01:39:42.377840       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:39:42.377920       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="b0568f9e-6618-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c33d5085-5901-41e6-bb00-7c62089db7c8" "latency"=8171132268
I1117 01:39:42.377942       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b0568f9e-6618-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8171477273
I1117 01:39:43.001389       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1117 01:39:43.001436       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1117 01:39:43.001475       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c7832345-a9ef-4eb9-8fdb-3128140a9723/globalmount with mount options([])
I1117 01:39:43.001516       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 315 lines ...
I1117 01:43:02.711510       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-17T01:43:02Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="2c9de628-6619-11ed-ae03-000d3a2cfaa5"
I1117 01:43:02.718514       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-18975173-vmss000002/status 200 OK in 6 milliseconds
I1117 01:43:02.718920       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="2c9de628-6619-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7440280
I1117 01:43:03.358021       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1117 01:43:03.358048       1 utils.go:79] GRPC request: {}
I1117 01:43:03.358099       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1117 01:43:06.548319       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:43:06.548437       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="2a1b0e75-6619-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-cfbba5ca-1791-42ca-9b59-4f10fe1226c8" "latency"=8049685821
I1117 01:43:06.548472       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2a1b0e75-6619-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8049936924
I1117 01:43:08.316994       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1117 01:43:08.317041       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS
I1117 01:43:08.317065       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cfbba5ca-1791-42ca-9b59-4f10fe1226c8/globalmount with mount options([])
I1117 01:43:08.317075       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 39 lines ...
I1117 01:43:20.570219       1 utils.go:79] GRPC request: {}
I1117 01:43:20.570253       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1117 01:43:20.571007       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1117 01:43:20.571019       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-973b2da5-b977-423a-bd8d-a96864366eab/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-973b2da5-b977-423a-bd8d-a96864366eab","csi.storage.k8s.io/pvc/name":"nginx-azuredisk-ephemeral-azuredisk01","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1668645925807-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-973b2da5-b977-423a-bd8d-a96864366eab"}
I1117 01:43:20.571698       1 azure_disk_utils.go:1049]  "msg"="Label selector is: disk.csi.azure.com/node-name=k8s-agentpool1-18975173-vmss000002,disk.csi.azure.com/volume-name!=pvc-973b2da5-b977-423a-bd8d-a96864366eab." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="374319e9-6619-11ed-ae03-000d3a2cfaa5"
I1117 01:43:20.571816       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-973b2da5-b977-423a-bd8d-a96864366eab-k8s-agentpool1-18975173-vmss000002-attachment)
I1117 01:43:30.010655       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:43:30.010730       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="374319e9-6619-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-973b2da5-b977-423a-bd8d-a96864366eab" "latency"=9438837061
I1117 01:43:30.010760       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="374319e9-6619-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9439409067
I1117 01:43:30.650603       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1117 01:43:30.650658       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS
I1117 01:43:30.650695       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-973b2da5-b977-423a-bd8d-a96864366eab/globalmount with mount options([])
I1117 01:43:30.650712       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 56 lines ...
I1117 01:43:48.492151       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-qb0hkalz/providers/Microsoft.Compute/disks/pvc-cfbba5ca-1791-42ca-9b59-4f10fe1226c8","volume_path":"/var/lib/kubelet/pods/90a900c6-6d8a-4cea-8704-c04353905bb3/volumes/kubernetes.io~csi/pvc-cfbba5ca-1791-42ca-9b59-4f10fe1226c8/mount"}
I1117 01:43:48.492201       1 utils.go:85] GRPC response: {"usage":[{"available":10447216640,"total":10464022528,"unit":1,"used":28672},{"available":655348,"total":655360,"unit":2,"used":12}]}
I1117 01:43:48.492222       1 utils.go:85] GRPC response: {"usage":[{"available":10447216640,"total":10464022528,"unit":1,"used":28672},{"available":655348,"total":655360,"unit":2,"used":12}]}
I1117 01:43:49.543459       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:43:49.543547       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:43:49.543570       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:43:49.543621       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1117 01:43:49.600005       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1117 01:43:55.132162       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1117 01:43:55.132238       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-18975173-vmss000002" "disk.csi.azure.com/request-id"="4333acbb-6619-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1a267659-3be6-4f38-a06b-0ffbb4d69207" "latency"=14529079251
I1117 01:43:55.132268       1 crdprovisioner.go:699]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4333acbb-6619-11ed-ae03-000d3a2cfaa5" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14529354054
I1117 01:43:55.739847       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1117 01:43:55.739903       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType StandardSSD_LRS
I1117 01:43:55.739933       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1a267659-3be6-4f38-a06b-0ffbb4d69207/globalmount with mount options([])
I1117 01:43:55.739942       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 576 lines ...
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.77437184e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
make: *** [Makefile:341: e2e-test] Error 1
2022/11/17 01:45:30 process.go:155: Step 'make e2e-test' finished in 1h10m54.449117403s
2022/11/17 01:45:30 aksengine_helpers.go:425: downloading /root/tmp1926242230/log-dump.sh from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh
2022/11/17 01:45:30 util.go:70: curl https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh
2022/11/17 01:45:30 process.go:153: Running: chmod +x /root/tmp1926242230/log-dump.sh
2022/11/17 01:45:30 process.go:155: Step 'chmod +x /root/tmp1926242230/log-dump.sh' finished in 1.198084ms
2022/11/17 01:45:30 aksengine_helpers.go:425: downloading /root/tmp1926242230/log-dump-daemonset.yaml from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump-daemonset.yaml
... skipping 75 lines ...
ssh key file /root/.ssh/id_rsa does not exist. Exiting.
2022/11/17 01:46:28 process.go:155: Step 'bash -c /root/tmp1926242230/win-ci-logs-collector.sh kubetest-qb0hkalz.westeurope.cloudapp.azure.com /root/tmp1926242230 /root/.ssh/id_rsa' finished in 3.549154ms
2022/11/17 01:46:28 aksengine.go:1141: Deleting resource group: kubetest-qb0hkalz.
2022/11/17 01:53:39 process.go:96: Saved XML output to /logs/artifacts/junit_runner.xml.
2022/11/17 01:53:39 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"
2022/11/17 01:53:39 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 255.341512ms
2022/11/17 01:53:39 main.go:328: Something went wrong: encountered 1 errors: [error during make e2e-test: exit status 2]
+ EXIT_VALUE=1
+ set +o xtrace
Cleaning up after docker in docker.
================================================================================
Cleaning up after docker
8437d2653796
... skipping 4 lines ...
PR	sunpa93: [V2] bug: WaitForAttach call should also be able to detach replicas for primary
Result	FAILURE
Tests	1 failed / 13 succeeded
Started	2022-11-17 00:30
Elapsed	1h23m
Revision	467cd5247c921714ffbb4851ccea16337232b9e1
Refs	1532
job-version	v1.27.0-alpha.0.28+3f823c0daa0021
kubetest-version	v20221116-7c85504268
revision	v1.27.0-alpha.0.28+3f823c0daa0021
kubernetes-sigs/azuredisk-csi-driver PR #1532 pull-azuredisk-csi-driver-external-e2e-single-az-mainv2 #1593038299914571776

Test Failures

kubetest Test 1h10m

Passed

Error lines from build-log.txt
