This job view page is being replaced by Spyglass soon. Check out the new job view.
PRhccheng72: [V2] feat: watch CRD deletion to track driver uninstall
ResultFAILURE
Tests 1 failed / 13 succeeded
Started2022-11-11 01:51
Elapsed1h22m
Revision3b564021a6ed26e9b84e3489845c1621a19c3863
Refs 1570
job-versionv1.26.0-beta.0.7+520e2c84b36a65
kubetest-versionv20221109-489d560851
revisionv1.26.0-beta.0.7+520e2c84b36a65

Test Failures


kubetest Test 1h10m

error during make e2e-test: exit status 2
				from junit_runner.xml

Filter through log files | View test history on testgrid


Show 13 Passed Tests

Error lines from build-log.txt

... skipping 236 lines ...

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 11156  100 11156    0     0   172k      0 --:--:-- --:--:-- --:--:--  172k
Downloading https://get.helm.sh/helm-v3.10.2-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
docker pull k8sprow.azurecr.io/azuredisk-csi:latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 || make container-all push-manifest
Error response from daemon: manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 not found: manifest unknown: manifest tagged by "latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0" is not found
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=9ac0207e3b71fdd2858caea5dad8b15c528821d0 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2022-11-11T01:56:21Z -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/amd64/azurediskpluginv2.exe ./pkg/azurediskplugin
docker buildx rm container-builder || true
ERROR: no builder "container-builder" found
docker buildx create --use --name=container-builder
container-builder
# enable qemu for arm64 build
# https://github.com/docker/buildx/issues/464#issuecomment-741507760
docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64
Unable to find image 'tonistiigi/binfmt:latest' locally
... skipping 705 lines ...
         }
      }
   ]
}
make[2]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker pull k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 || make azdiskschedulerextender-all push-manifest-azdiskschedulerextender
Error response from daemon: manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 not found: manifest unknown: manifest tagged by "latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0" is not found
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker buildx rm container-builder || true
container-builder removed
docker buildx create --use --name=container-builder
container-builder
# enable qemu for arm64 build
... skipping 1030 lines ...
                    type: string
                type: object
                oneOf:
                - required: ["persistentVolumeClaimName"]
                - required: ["volumeSnapshotContentName"]
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 60 lines ...
                    type: string
                  volumeSnapshotContentName:
                    description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable.
                    type: string
                type: object
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 254 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 108 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 359 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status Required
            properties:
              detail:
                description: Status summarizes the current attachment state of the
                  volume attachment Nil Status indicates that the volume has not yet
                  been attached to the node
                properties:
... skipping 7 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 90 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 13 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 169 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              detail:
                description: Current status detail of the AzVolume Nil detail indicates
                  that the volume has not been created
                properties:
                  accessible_topology:
... skipping 28 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 154 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 34 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 1069 lines ...
          image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0"
          args:
            - "-csi-address=$(ADDRESS)"
            - "-v=2"
            - "-leader-election"
            - "--leader-election-namespace=kube-system"
            - '-handle-volume-inuse-error=false'
            - '-feature-gates=RecoverVolumeExpansionFailure=true'
            - "-timeout=240s"
          env:
            - name: ADDRESS
              value: /csi/csi.sock
          volumeMounts:
... skipping 323 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:269

    Distro debian doesn't support ntfs -- skipping

    test/e2e/storage/framework/testsuite.go:127
------------------------------
... skipping 81 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:258

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 69 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:280

    Distro debian doesn't support ntfs -- skipping

    test/e2e/storage/framework/testsuite.go:127
------------------------------
... skipping 313 lines ...
Nov 11 02:12:01.575: INFO: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 found and phase=Released (4m36.048010167s)
Nov 11 02:12:06.683: INFO: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 found and phase=Released (4m41.156643958s)
Nov 11 02:12:11.795: INFO: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 found and phase=Released (4m46.26857884s)
Nov 11 02:12:16.911: INFO: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 found and phase=Released (4m51.384404203s)
Nov 11 02:12:22.029: INFO: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 found and phase=Released (4m56.502878528s)
STEP: Deleting sc
Nov 11 02:12:27.141: FAIL: while cleaning up resource
Unexpected error:
    <errors.aggregate | len:1, cap:1>: [
        [
            {
                msg: "persistent Volume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 not deleted by dynamic provisioner: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 still exists within 5m0s",
                err: {
                    s: "PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 still exists within 5m0s",
... skipping 33 lines ...
Nov 11 02:12:27.253: INFO: At 2022-11-11 02:07:14 +0000 UTC - event for hostexec-k8s-agentpool1-30725374-vmss000001-bl4tc: {default-scheduler } Scheduled: Successfully assigned volumemode-3690/hostexec-k8s-agentpool1-30725374-vmss000001-bl4tc to k8s-agentpool1-30725374-vmss000001
Nov 11 02:12:27.253: INFO: At 2022-11-11 02:07:18 +0000 UTC - event for hostexec-k8s-agentpool1-30725374-vmss000001-bl4tc: {kubelet k8s-agentpool1-30725374-vmss000001} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/agnhost:2.36" in 3.794195202s
Nov 11 02:12:27.253: INFO: At 2022-11-11 02:07:18 +0000 UTC - event for hostexec-k8s-agentpool1-30725374-vmss000001-bl4tc: {kubelet k8s-agentpool1-30725374-vmss000001} Started: Started container agnhost-container
Nov 11 02:12:27.254: INFO: At 2022-11-11 02:07:18 +0000 UTC - event for hostexec-k8s-agentpool1-30725374-vmss000001-bl4tc: {kubelet k8s-agentpool1-30725374-vmss000001} Created: Created container agnhost-container
Nov 11 02:12:27.254: INFO: At 2022-11-11 02:07:23 +0000 UTC - event for hostexec-k8s-agentpool1-30725374-vmss000001-bl4tc: {kubelet k8s-agentpool1-30725374-vmss000001} Killing: Stopping container agnhost-container
Nov 11 02:12:27.254: INFO: At 2022-11-11 02:07:23 +0000 UTC - event for pod-3d455c00-a9bd-466e-b16b-83b2965556cd: {kubelet k8s-agentpool1-30725374-vmss000001} Killing: Stopping container write-pod
Nov 11 02:12:27.254: INFO: At 2022-11-11 02:07:25 +0000 UTC - event for pod-3d455c00-a9bd-466e-b16b-83b2965556cd: {kubelet k8s-agentpool1-30725374-vmss000001} FailedKillPod: error killing pod: failed to "KillContainer" for "write-pod" with KillContainerError: "rpc error: code = Unknown desc = Error response from daemon: No such container: d51e545125c0659d71cc059b1389f9cd7db490e82ff391d2861ec0cdcf0151a6"
Nov 11 02:12:27.254: INFO: At 2022-11-11 02:07:36 +0000 UTC - event for pod-3d455c00-a9bd-466e-b16b-83b2965556cd: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025" 
Nov 11 02:12:27.363: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov 11 02:12:27.363: INFO: 
Nov 11 02:12:27.579: INFO: 
Logging node info for node k8s-agentpool1-30725374-vmss000000
Nov 11 02:12:27.689: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-30725374-vmss000000    9e7f2eeb-24ce-481f-8f7f-b3379d8377a6 4487 0 2022-11-11 01:54:07 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D8s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westeurope failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-9juvoppw kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-30725374-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_D8s_v3 storageprofile:managed storagetier:Premium_LRS topology.kubernetes.io/region:westeurope topology.kubernetes.io/zone:0 topology.test.csi.azure.com/zone:] map[csi.volume.kubernetes.io/nodeid:{"test.csi.azure.com":"k8s-agentpool1-30725374-vmss000000"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2022-11-11 01:54:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{}}}} } {cloud-controller-manager Update v1 2022-11-11 01:54:22 +0000 UTC FieldsV1 {"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.3.0/24\"":{}}}} } {kubectl-label Update v1 2022-11-11 01:54:23 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kube-controller-manager Update v1 2022-11-11 01:54:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {cloud-controller-manager Update v1 2022-11-11 01:55:09 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {cloud-node-manager Update v1 2022-11-11 01:55:46 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2022-11-11 02:07:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.test.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2022-11-11 02:07:35 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-30725374-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{31025332224 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{33672720384 0} {<nil>} 32883516Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{27922798956 0} {<nil>} 27922798956 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{32886288384 0} {<nil>} 32115516Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-11 01:55:09 +0000 UTC,LastTransitionTime:2022-11-11 01:55:09 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-11 02:12:07 +0000 UTC,LastTransitionTime:2022-11-11 01:54:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-11 02:12:07 +0000 UTC,LastTransitionTime:2022-11-11 01:54:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-11 02:12:07 +0000 UTC,LastTransitionTime:2022-11-11 01:54:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-11 02:12:07 +0000 UTC,LastTransitionTime:2022-11-11 01:54:22 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool1-30725374-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:9dc7951faf284e71b31f4fbcc187bf1c,SystemUUID:04cd8ce5-2ce6-a04f-bb6b-13202e75538b,BootID:95f20e2d-1e4e-43d6-a374-9c013de158b4,KernelVersion:5.4.0-1094-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.14+azure-1,KubeletVersion:v1.23.13,KubeProxyVersion:v1.23.13,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:cbf9a288cfc596f962ae3401e7610f25d37321c217da7323c62a28d746eedea9 k8sprow.azurecr.io/azuredisk-csi:latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0],SizeBytes:142917021,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver@sha256:46c1541fe63e34dfc697e3acbf8d40661ba8af217bb5457b8ff5adb9af4b248e mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.23.13],SizeBytes:135172310,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager@sha256:041c9d6da46acbdbaaa937ef2f370a341cf86c77654304ce3b04ab82f28b4952 mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.23.13],SizeBytes:124990264,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:9aeab889b5d0623cd5ca7aceb506ed2bc34836ee99fec603bfd0e0abe95e983e mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.13],SizeBytes:112304283,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[k8sprow.azurecr.io/azdiskschedulerextender-csi@sha256:3ed520c42107e2374e4d039698ce45f2c7891721708c2d9534dc3a87181939af k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0],SizeBytes:58129683,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:95d70317663e1da01ef37d42cce2e35241292b777e934ec4e58089e3d69a7e10 mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.23.13],SizeBytes:53486364,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:949db4eea46ea31f9afa0b49ad0b55f7a31f91aa03b408e0ac9b7720c8ef7d29 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.11],SizeBytes:51376235,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:72fbc9d17947d1a54c09a16671e6810fd7c0d8f09c0502fdbdd212c2192451a7 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.14],SizeBytes:51044459,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:075ea1f8270312350f1396ab6677251e803e61a523822d5abfa5e6acd180cfab mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.11],SizeBytes:50806891,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:c35f45e8dfaa30363e589296415a057f9d32813920fc3820d6557491657f199a mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.2],SizeBytes:50607491,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:eb7d329559d954093f7a234bc0ff322764de48a0b2a1007229032fb53e7bb9eb mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.24.0],SizeBytes:49070187,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:2dfff4dc10f6d25b0a34c01bf4d166540cbf1ff2bccd52635350039138eb23c4 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.18],SizeBytes:48775275,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3e23b9ab6bfee42a1751019c21a60aa474964bd4cba00cbed5055430424242cc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.24.0],SizeBytes:48500843,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:acec9ee6626c2cdea89107ba191ae36e6f60339c9b3b3215b5a59bf0e2e730ba mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.14],SizeBytes:47853675,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:759ac7020e982884b99121583bd577658b5663b1c1ed69746356aa6ce9879da4 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.18],SizeBytes:45850731,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:2fbd1e1a0538a06f2061afd45975df70c942654aa7f86e920720169ee439c2d6 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.5.1],SizeBytes:19743433,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:31547791294872570393470991481c2477a311031d3a03e0ae54eb164347dc34 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0],SizeBytes:17857485,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-75b7eb16-398c-42f1-994f-7aa02a0ccdd8 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-d6226a7d-bdbf-4b17-a53a-8f979cb0ca11],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-75b7eb16-398c-42f1-994f-7aa02a0ccdd8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-d6226a7d-bdbf-4b17-a53a-8f979cb0ca11,DevicePath:,},},Config:nil,},}
... skipping 114 lines ...
  [Testpattern: Dynamic PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should not mount / map unused volumes in a pod [LinuxOnly] [Measurement]
    test/e2e/storage/testsuites/volumemode.go:354

    Nov 11 02:12:27.141: while cleaning up resource
    Unexpected error:
        <errors.aggregate | len:1, cap:1>: [
            [
                {
                    msg: "persistent Volume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 not deleted by dynamic provisioner: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 still exists within 5m0s",
                    err: {
                        s: "PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 still exists within 5m0s",
... skipping 3 lines ...
        ]
        persistent Volume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 not deleted by dynamic provisioner: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 still exists within 5m0s
    occurred

    test/e2e/storage/testsuites/volumemode.go:190
------------------------------
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":31,"completed":0,"skipped":7,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:12:30.828: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 3 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.002 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:280

    Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 44 lines ...
Nov 11 02:07:05.418: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comfdlbg] to have phase Bound
Nov 11 02:07:05.526: INFO: PersistentVolumeClaim test.csi.azure.comfdlbg found but phase is Pending instead of Bound.
Nov 11 02:07:07.635: INFO: PersistentVolumeClaim test.csi.azure.comfdlbg found but phase is Pending instead of Bound.
Nov 11 02:07:09.743: INFO: PersistentVolumeClaim test.csi.azure.comfdlbg found and phase=Bound (4.324617683s)
STEP: Creating pod exec-volume-test-dynamicpv-pp4s
STEP: Creating a pod to test exec-volume-test
Nov 11 02:07:10.084: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-pp4s" in namespace "volume-154" to be "Succeeded or Failed"
Nov 11 02:07:10.192: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 107.768391ms
Nov 11 02:07:12.301: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216969353s
Nov 11 02:07:14.410: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 4.325914972s
Nov 11 02:07:16.521: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 6.436256443s
Nov 11 02:07:18.629: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 8.544873882s
Nov 11 02:07:20.742: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 10.657486413s
... skipping 122 lines ...
Nov 11 02:11:40.236: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.151496323s
Nov 11 02:11:42.346: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.261111463s
Nov 11 02:11:44.455: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.370372229s
Nov 11 02:11:46.565: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.480825034s
Nov 11 02:11:48.675: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Succeeded", Reason="", readiness=false. Elapsed: 4m38.590370543s
STEP: Saw pod success
Nov 11 02:11:48.675: INFO: Pod "exec-volume-test-dynamicpv-pp4s" satisfied condition "Succeeded or Failed"
Nov 11 02:11:48.790: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000000 pod exec-volume-test-dynamicpv-pp4s container exec-container-dynamicpv-pp4s: <nil>
STEP: delete the pod
Nov 11 02:11:49.049: INFO: Waiting for pod exec-volume-test-dynamicpv-pp4s to disappear
Nov 11 02:11:49.157: INFO: Pod exec-volume-test-dynamicpv-pp4s no longer exists
STEP: Deleting pod exec-volume-test-dynamicpv-pp4s
Nov 11 02:11:49.157: INFO: Deleting pod "exec-volume-test-dynamicpv-pp4s" in namespace "volume-154"
... skipping 23 lines ...
  test/e2e/storage/framework/testsuite.go:50
    should allow exec of files on the volume
    test/e2e/storage/testsuites/volumes.go:198
------------------------------
S
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes should allow exec of files on the volume","total":27,"completed":1,"skipped":15,"failed":0}

SSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] volumeIO
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:12:30.914: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 232 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:138
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":28,"completed":1,"skipped":85,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy 
  (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents
  test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
... skipping 119 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents","total":38,"completed":1,"skipped":291,"failed":0}

SSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:13:22.271: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 300 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:138
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":41,"completed":1,"skipped":200,"failed":0}

SSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:13:50.066: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 104 lines ...
Nov 11 02:07:05.490: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com7t57r] to have phase Bound
Nov 11 02:07:05.598: INFO: PersistentVolumeClaim test.csi.azure.com7t57r found but phase is Pending instead of Bound.
Nov 11 02:07:07.707: INFO: PersistentVolumeClaim test.csi.azure.com7t57r found but phase is Pending instead of Bound.
Nov 11 02:07:09.815: INFO: PersistentVolumeClaim test.csi.azure.com7t57r found and phase=Bound (4.324955938s)
STEP: [init] starting a pod to use the claim
STEP: [init] check pod success
Nov 11 02:07:10.259: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-7jn44" in namespace "snapshotting-4409" to be "Succeeded or Failed"
Nov 11 02:07:10.368: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 108.238798ms
Nov 11 02:07:12.478: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218518842s
Nov 11 02:07:14.587: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327111958s
Nov 11 02:07:16.696: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 6.436368303s
Nov 11 02:07:18.809: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 8.5495259s
Nov 11 02:07:20.919: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 10.659881919s
... skipping 121 lines ...
Nov 11 02:11:38.329: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.069904174s
Nov 11 02:11:40.440: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.180749938s
Nov 11 02:11:42.549: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.289769913s
Nov 11 02:11:44.659: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.399784439s
Nov 11 02:11:46.770: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Succeeded", Reason="", readiness=false. Elapsed: 4m36.510388011s
STEP: Saw pod success
Nov 11 02:11:46.770: INFO: Pod "pvc-snapshottable-tester-7jn44" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
Nov 11 02:11:46.878: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com7t57r] to have phase Bound
Nov 11 02:11:46.987: INFO: PersistentVolumeClaim test.csi.azure.com7t57r found and phase=Bound (108.182636ms)
STEP: [init] checking the PV
STEP: [init] deleting the pod
Nov 11 02:11:47.342: INFO: Pod pvc-snapshottable-tester-7jn44 has the following logs: 
... skipping 14 lines ...
Nov 11 02:11:56.991: INFO: received snapshotStatus map[boundVolumeSnapshotContentName:snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc creationTime:2022-11-11T02:11:52Z readyToUse:true restoreSize:5Gi]
Nov 11 02:11:56.991: INFO: snapshotContentName snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Nov 11 02:11:57.428: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-4pdxb" in namespace "snapshotting-4409" to be "Succeeded or Failed"
Nov 11 02:11:57.538: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 109.048383ms
Nov 11 02:11:59.647: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218884206s
Nov 11 02:12:01.758: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 4.32956246s
Nov 11 02:12:03.867: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 6.438686918s
Nov 11 02:12:05.976: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547706872s
Nov 11 02:12:08.094: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 10.665147899s
... skipping 13 lines ...
Nov 11 02:12:37.632: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 40.203466877s
Nov 11 02:12:39.742: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 42.313432383s
Nov 11 02:12:41.852: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 44.4232079s
Nov 11 02:12:43.960: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 46.53157022s
Nov 11 02:12:46.073: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Succeeded", Reason="", readiness=false. Elapsed: 48.644922411s
STEP: Saw pod success
Nov 11 02:12:46.074: INFO: Pod "pvc-snapshottable-data-tester-4pdxb" satisfied condition "Succeeded or Failed"
Nov 11 02:12:46.295: INFO: Pod pvc-snapshottable-data-tester-4pdxb has the following logs: 
Nov 11 02:12:46.295: INFO: Deleting pod "pvc-snapshottable-data-tester-4pdxb" in namespace "snapshotting-4409"
Nov 11 02:12:46.411: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-4pdxb" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the snapshot
Nov 11 02:13:48.962: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-9juvoppw.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-4409 exec restored-pvc-tester-6pzsk --namespace=snapshotting-4409 -- cat /mnt/test/data'
... skipping 32 lines ...
Nov 11 02:14:14.376: INFO: volumesnapshotcontents snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc has been found and is not deleted
Nov 11 02:14:15.485: INFO: volumesnapshotcontents snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc has been found and is not deleted
Nov 11 02:14:16.595: INFO: volumesnapshotcontents snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc has been found and is not deleted
Nov 11 02:14:17.705: INFO: volumesnapshotcontents snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc has been found and is not deleted
Nov 11 02:14:18.816: INFO: volumesnapshotcontents snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc has been found and is not deleted
Nov 11 02:14:19.925: INFO: volumesnapshotcontents snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc has been found and is not deleted
Nov 11 02:14:20.926: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 11 02:14:21.131: INFO: Pod restored-pvc-tester-6pzsk has the following logs: 
Nov 11 02:14:21.131: INFO: Deleting pod "restored-pvc-tester-6pzsk" in namespace "snapshotting-4409"
Nov 11 02:14:21.239: INFO: Wait up to 5m0s for pod "restored-pvc-tester-6pzsk" to be fully deleted
Nov 11 02:14:23.459: INFO: deleting claim "snapshotting-4409"/"pvc-49v7h"
... skipping 31 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        test/e2e/storage/testsuites/snapshottable.go:278
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":27,"completed":1,"skipped":76,"failed":0}

SSSSS
------------------------------
[BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:14:41.672: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping
... skipping 176 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents","total":31,"completed":1,"skipped":148,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:14:42.666: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 62 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.002 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach]
    test/e2e/storage/testsuites/volumemode.go:299

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 246 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:168
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":38,"completed":1,"skipped":62,"failed":0}

SS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:14:48.309: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 80 lines ...
Nov 11 02:13:15.360: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comvxrcs] to have phase Bound
Nov 11 02:13:15.471: INFO: PersistentVolumeClaim test.csi.azure.comvxrcs found but phase is Pending instead of Bound.
Nov 11 02:13:17.578: INFO: PersistentVolumeClaim test.csi.azure.comvxrcs found but phase is Pending instead of Bound.
Nov 11 02:13:19.686: INFO: PersistentVolumeClaim test.csi.azure.comvxrcs found and phase=Bound (4.325950385s)
STEP: Creating pod pod-subpath-test-dynamicpv-gb7h
STEP: Creating a pod to test atomic-volume-subpath
Nov 11 02:13:20.011: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-gb7h" in namespace "provisioning-1040" to be "Succeeded or Failed"
Nov 11 02:13:20.118: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Pending", Reason="", readiness=false. Elapsed: 107.238006ms
Nov 11 02:13:22.230: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219190314s
Nov 11 02:13:24.339: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Pending", Reason="", readiness=false. Elapsed: 4.3280436s
Nov 11 02:13:26.449: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Pending", Reason="", readiness=false. Elapsed: 6.438010772s
Nov 11 02:13:28.558: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Pending", Reason="", readiness=false. Elapsed: 8.546728832s
Nov 11 02:13:30.667: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Pending", Reason="", readiness=false. Elapsed: 10.655537026s
... skipping 16 lines ...
Nov 11 02:14:06.532: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Running", Reason="", readiness=true. Elapsed: 46.521258636s
Nov 11 02:14:08.642: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Running", Reason="", readiness=true. Elapsed: 48.631266922s
Nov 11 02:14:10.756: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Running", Reason="", readiness=true. Elapsed: 50.744769319s
Nov 11 02:14:12.866: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Running", Reason="", readiness=false. Elapsed: 52.855259036s
Nov 11 02:14:14.977: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Succeeded", Reason="", readiness=false. Elapsed: 54.9654105s
STEP: Saw pod success
Nov 11 02:14:14.977: INFO: Pod "pod-subpath-test-dynamicpv-gb7h" satisfied condition "Succeeded or Failed"
Nov 11 02:14:15.086: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000001 pod pod-subpath-test-dynamicpv-gb7h container test-container-subpath-dynamicpv-gb7h: <nil>
STEP: delete the pod
Nov 11 02:14:15.321: INFO: Waiting for pod pod-subpath-test-dynamicpv-gb7h to disappear
Nov 11 02:14:15.429: INFO: Pod pod-subpath-test-dynamicpv-gb7h no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-gb7h
Nov 11 02:14:15.429: INFO: Deleting pod "pod-subpath-test-dynamicpv-gb7h" in namespace "provisioning-1040"
... skipping 29 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support file as subpath [LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:232
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support file as subpath [LinuxOnly]","total":28,"completed":2,"skipped":153,"failed":0}

SSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (ext4)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:15:27.977: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 190 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:323
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":27,"completed":2,"skipped":43,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:15:42.438: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 66 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should create read-only inline ephemeral volume
    test/e2e/storage/testsuites/ephemeral.go:175
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read-only inline ephemeral volume","total":41,"completed":2,"skipped":260,"failed":0}

SSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should support non-existent path
  test/e2e/storage/testsuites/subpath.go:196
... skipping 17 lines ...
Nov 11 02:14:42.794: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comg5mrf] to have phase Bound
Nov 11 02:14:42.905: INFO: PersistentVolumeClaim test.csi.azure.comg5mrf found but phase is Pending instead of Bound.
Nov 11 02:14:45.014: INFO: PersistentVolumeClaim test.csi.azure.comg5mrf found but phase is Pending instead of Bound.
Nov 11 02:14:47.123: INFO: PersistentVolumeClaim test.csi.azure.comg5mrf found and phase=Bound (4.328779386s)
STEP: Creating pod pod-subpath-test-dynamicpv-fkj5
STEP: Creating a pod to test subpath
Nov 11 02:14:47.453: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-fkj5" in namespace "provisioning-561" to be "Succeeded or Failed"
Nov 11 02:14:47.561: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 108.220041ms
Nov 11 02:14:49.670: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217369447s
Nov 11 02:14:51.779: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326264076s
Nov 11 02:14:53.889: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 6.435903032s
Nov 11 02:14:55.998: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 8.545229104s
Nov 11 02:14:58.108: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 10.655425395s
... skipping 30 lines ...
Nov 11 02:16:03.514: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.060601416s
Nov 11 02:16:05.632: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.17858206s
Nov 11 02:16:07.751: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.298271551s
Nov 11 02:16:09.861: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.408376505s
Nov 11 02:16:11.971: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m24.518055374s
STEP: Saw pod success
Nov 11 02:16:11.971: INFO: Pod "pod-subpath-test-dynamicpv-fkj5" satisfied condition "Succeeded or Failed"
Nov 11 02:16:12.078: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000001 pod pod-subpath-test-dynamicpv-fkj5 container test-container-volume-dynamicpv-fkj5: <nil>
STEP: delete the pod
Nov 11 02:16:12.328: INFO: Waiting for pod pod-subpath-test-dynamicpv-fkj5 to disappear
Nov 11 02:16:12.436: INFO: Pod pod-subpath-test-dynamicpv-fkj5 no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-fkj5
Nov 11 02:16:12.436: INFO: Deleting pod "pod-subpath-test-dynamicpv-fkj5" in namespace "provisioning-561"
... skipping 29 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support non-existent path
    test/e2e/storage/testsuites/subpath.go:196
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path","total":27,"completed":2,"skipped":166,"failed":0}

SSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] 
  should concurrently access the single volume from pods on the same node
  test/e2e/storage/testsuites/multivolume.go:298
... skipping 154 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:298
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":38,"completed":2,"skipped":226,"failed":0}

S
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should support readOnly directory specified in the volumeMount
  test/e2e/storage/testsuites/subpath.go:367
... skipping 17 lines ...
Nov 11 02:14:44.885: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com6blqb] to have phase Bound
Nov 11 02:14:44.995: INFO: PersistentVolumeClaim test.csi.azure.com6blqb found but phase is Pending instead of Bound.
Nov 11 02:14:47.104: INFO: PersistentVolumeClaim test.csi.azure.com6blqb found but phase is Pending instead of Bound.
Nov 11 02:14:49.213: INFO: PersistentVolumeClaim test.csi.azure.com6blqb found and phase=Bound (4.327944844s)
STEP: Creating pod pod-subpath-test-dynamicpv-zv5m
STEP: Creating a pod to test subpath
Nov 11 02:14:49.543: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-zv5m" in namespace "provisioning-5195" to be "Succeeded or Failed"
Nov 11 02:14:49.652: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 108.742694ms
Nov 11 02:14:51.763: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219215456s
Nov 11 02:14:53.873: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 4.329436267s
Nov 11 02:14:55.983: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 6.439035399s
Nov 11 02:14:58.093: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 8.549704534s
Nov 11 02:15:00.204: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 10.660151764s
... skipping 42 lines ...
Nov 11 02:16:30.965: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 1m41.421214198s
Nov 11 02:16:33.080: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 1m43.536673891s
Nov 11 02:16:35.189: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 1m45.645698135s
Nov 11 02:16:37.300: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 1m47.75611972s
Nov 11 02:16:39.410: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m49.866330048s
STEP: Saw pod success
Nov 11 02:16:39.410: INFO: Pod "pod-subpath-test-dynamicpv-zv5m" satisfied condition "Succeeded or Failed"
Nov 11 02:16:39.519: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000001 pod pod-subpath-test-dynamicpv-zv5m container test-container-subpath-dynamicpv-zv5m: <nil>
STEP: delete the pod
Nov 11 02:16:39.746: INFO: Waiting for pod pod-subpath-test-dynamicpv-zv5m to disappear
Nov 11 02:16:39.856: INFO: Pod pod-subpath-test-dynamicpv-zv5m no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-zv5m
Nov 11 02:16:39.856: INFO: Deleting pod "pod-subpath-test-dynamicpv-zv5m" in namespace "provisioning-5195"
... skipping 29 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support readOnly directory specified in the volumeMount
    test/e2e/storage/testsuites/subpath.go:367
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly directory specified in the volumeMount","total":31,"completed":2,"skipped":333,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] volumeIO
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:17:52.392: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 280 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should store data
    test/e2e/storage/testsuites/volumes.go:161
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data","total":38,"completed":2,"skipped":370,"failed":0}

SSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] 
  should concurrently access the single volume from pods on the same node
  test/e2e/storage/testsuites/multivolume.go:298
... skipping 153 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:298
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":27,"completed":3,"skipped":163,"failed":0}

SSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)(allowExpansion)][Feature:Windows] volume-expand
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:18:17.665: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 80 lines ...
Nov 11 02:17:53.688: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.commhdpp] to have phase Bound
Nov 11 02:17:53.796: INFO: PersistentVolumeClaim test.csi.azure.commhdpp found but phase is Pending instead of Bound.
Nov 11 02:17:55.905: INFO: PersistentVolumeClaim test.csi.azure.commhdpp found but phase is Pending instead of Bound.
Nov 11 02:17:58.013: INFO: PersistentVolumeClaim test.csi.azure.commhdpp found and phase=Bound (4.324674108s)
STEP: Creating pod pod-subpath-test-dynamicpv-gmmn
STEP: Creating a pod to test subpath
Nov 11 02:17:58.339: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-gmmn" in namespace "provisioning-3283" to be "Succeeded or Failed"
Nov 11 02:17:58.447: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 107.617605ms
Nov 11 02:18:00.558: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218849556s
Nov 11 02:18:02.668: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328471057s
Nov 11 02:18:04.776: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 6.436761697s
Nov 11 02:18:06.885: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 8.54552553s
Nov 11 02:18:08.997: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 10.65771461s
... skipping 8 lines ...
Nov 11 02:18:27.979: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 29.639358071s
Nov 11 02:18:30.087: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 31.747857383s
Nov 11 02:18:32.197: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 33.857489122s
Nov 11 02:18:34.306: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 35.966372651s
Nov 11 02:18:36.414: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.07499704s
STEP: Saw pod success
Nov 11 02:18:36.415: INFO: Pod "pod-subpath-test-dynamicpv-gmmn" satisfied condition "Succeeded or Failed"
Nov 11 02:18:36.523: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000002 pod pod-subpath-test-dynamicpv-gmmn container test-container-subpath-dynamicpv-gmmn: <nil>
STEP: delete the pod
Nov 11 02:18:36.779: INFO: Waiting for pod pod-subpath-test-dynamicpv-gmmn to disappear
Nov 11 02:18:36.886: INFO: Pod pod-subpath-test-dynamicpv-gmmn no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-gmmn
Nov 11 02:18:36.886: INFO: Deleting pod "pod-subpath-test-dynamicpv-gmmn" in namespace "provisioning-3283"
... skipping 23 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support existing single file [LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:221
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing single file [LinuxOnly]","total":31,"completed":3,"skipped":680,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes 
  should allow exec of files on the volume
  test/e2e/storage/testsuites/volumes.go:198
... skipping 17 lines ...
Nov 11 02:17:46.962: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comc7wqs] to have phase Bound
Nov 11 02:17:47.069: INFO: PersistentVolumeClaim test.csi.azure.comc7wqs found but phase is Pending instead of Bound.
Nov 11 02:17:49.177: INFO: PersistentVolumeClaim test.csi.azure.comc7wqs found but phase is Pending instead of Bound.
Nov 11 02:17:51.284: INFO: PersistentVolumeClaim test.csi.azure.comc7wqs found and phase=Bound (4.322189252s)
STEP: Creating pod exec-volume-test-dynamicpv-xj5t
STEP: Creating a pod to test exec-volume-test
Nov 11 02:17:51.606: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-xj5t" in namespace "volume-2914" to be "Succeeded or Failed"
Nov 11 02:17:51.713: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 106.946967ms
Nov 11 02:17:53.821: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 2.214463222s
Nov 11 02:17:55.929: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 4.322896132s
Nov 11 02:17:58.037: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 6.430980988s
Nov 11 02:18:00.148: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 8.541861668s
Nov 11 02:18:02.257: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 10.651195836s
... skipping 9 lines ...
Nov 11 02:18:23.342: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 31.736091741s
Nov 11 02:18:25.450: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 33.84435069s
Nov 11 02:18:27.559: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 35.952851506s
Nov 11 02:18:29.666: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 38.060430481s
Nov 11 02:18:31.776: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.169685421s
STEP: Saw pod success
Nov 11 02:18:31.776: INFO: Pod "exec-volume-test-dynamicpv-xj5t" satisfied condition "Succeeded or Failed"
Nov 11 02:18:31.883: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000001 pod exec-volume-test-dynamicpv-xj5t container exec-container-dynamicpv-xj5t: <nil>
STEP: delete the pod
Nov 11 02:18:32.109: INFO: Waiting for pod exec-volume-test-dynamicpv-xj5t to disappear
Nov 11 02:18:32.216: INFO: Pod exec-volume-test-dynamicpv-xj5t no longer exists
STEP: Deleting pod exec-volume-test-dynamicpv-xj5t
Nov 11 02:18:32.216: INFO: Deleting pod "exec-volume-test-dynamicpv-xj5t" in namespace "volume-2914"
... skipping 27 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should allow exec of files on the volume
    test/e2e/storage/testsuites/volumes.go:198
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should allow exec of files on the volume","total":38,"completed":3,"skipped":227,"failed":0}

SSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
... skipping 155 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should store data
    test/e2e/storage/testsuites/volumes.go:161
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data","total":28,"completed":3,"skipped":266,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral 
  should create read/write inline ephemeral volume
  test/e2e/storage/testsuites/ephemeral.go:196
... skipping 52 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should create read/write inline ephemeral volume
    test/e2e/storage/testsuites/ephemeral.go:196
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should create read/write inline ephemeral volume","total":41,"completed":3,"skipped":263,"failed":0}

SSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should fail if subpath file is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:258

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 11 02:18:17.759: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename provisioning
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if subpath file is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:258
Nov 11 02:18:18.517: INFO: Creating resource for dynamic PV
Nov 11 02:18:18.517: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-4691-e2e-scjrpvn
STEP: creating a claim
Nov 11 02:18:18.626: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 11 02:18:18.737: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comkxf5f] to have phase Bound
Nov 11 02:18:18.847: INFO: PersistentVolumeClaim test.csi.azure.comkxf5f found but phase is Pending instead of Bound.
Nov 11 02:18:20.955: INFO: PersistentVolumeClaim test.csi.azure.comkxf5f found but phase is Pending instead of Bound.
Nov 11 02:18:23.065: INFO: PersistentVolumeClaim test.csi.azure.comkxf5f found and phase=Bound (4.327903154s)
STEP: Creating pod pod-subpath-test-dynamicpv-2ldb
STEP: Checking for subpath error in container status
Nov 11 02:19:05.608: INFO: Deleting pod "pod-subpath-test-dynamicpv-2ldb" in namespace "provisioning-4691"
Nov 11 02:19:05.717: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-2ldb" to be fully deleted
STEP: Deleting pod
Nov 11 02:19:07.935: INFO: Deleting pod "pod-subpath-test-dynamicpv-2ldb" in namespace "provisioning-4691"
STEP: Deleting pvc
Nov 11 02:19:08.055: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comkxf5f"
... skipping 21 lines ...

• [SLOW TEST:117.482 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath file is outside the volume [Slow][LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:258
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow][LinuxOnly]","total":27,"completed":4,"skipped":254,"failed":0}

SSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should support readOnly file specified in the volumeMount [LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:382
... skipping 17 lines ...
Nov 11 02:18:04.946: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comq6h28] to have phase Bound
Nov 11 02:18:05.053: INFO: PersistentVolumeClaim test.csi.azure.comq6h28 found but phase is Pending instead of Bound.
Nov 11 02:18:07.162: INFO: PersistentVolumeClaim test.csi.azure.comq6h28 found but phase is Pending instead of Bound.
Nov 11 02:18:09.271: INFO: PersistentVolumeClaim test.csi.azure.comq6h28 found and phase=Bound (4.325051455s)
STEP: Creating pod pod-subpath-test-dynamicpv-5m7s
STEP: Creating a pod to test subpath
Nov 11 02:18:09.597: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-5m7s" in namespace "provisioning-9412" to be "Succeeded or Failed"
Nov 11 02:18:09.707: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 109.734964ms
Nov 11 02:18:11.816: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219040665s
Nov 11 02:18:13.926: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328360631s
Nov 11 02:18:16.035: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437920156s
Nov 11 02:18:18.144: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 8.5465115s
Nov 11 02:18:20.254: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 10.657182535s
... skipping 18 lines ...
Nov 11 02:19:00.330: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 50.732786849s
Nov 11 02:19:02.440: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 52.842873182s
Nov 11 02:19:04.549: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 54.951823676s
Nov 11 02:19:06.659: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 57.062034366s
Nov 11 02:19:08.768: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Succeeded", Reason="", readiness=false. Elapsed: 59.170776864s
STEP: Saw pod success
Nov 11 02:19:08.768: INFO: Pod "pod-subpath-test-dynamicpv-5m7s" satisfied condition "Succeeded or Failed"
Nov 11 02:19:08.876: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000000 pod pod-subpath-test-dynamicpv-5m7s container test-container-subpath-dynamicpv-5m7s: <nil>
STEP: delete the pod
Nov 11 02:19:09.129: INFO: Waiting for pod pod-subpath-test-dynamicpv-5m7s to disappear
Nov 11 02:19:09.236: INFO: Pod pod-subpath-test-dynamicpv-5m7s no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-5m7s
Nov 11 02:19:09.236: INFO: Deleting pod "pod-subpath-test-dynamicpv-5m7s" in namespace "provisioning-9412"
... skipping 29 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support readOnly file specified in the volumeMount [LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:382
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly]","total":38,"completed":3,"skipped":373,"failed":0}

S
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumeIO 
  should write files of various sizes, verify size, validate content [Slow]
  test/e2e/storage/testsuites/volume_io.go:149
... skipping 48 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] volumeIO
  test/e2e/storage/framework/testsuite.go:50
    should write files of various sizes, verify size, validate content [Slow]
    test/e2e/storage/testsuites/volume_io.go:149
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumeIO should write files of various sizes, verify size, validate content [Slow]","total":31,"completed":4,"skipped":737,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:20:53.039: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 171 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] volumes
  test/e2e/storage/framework/testsuite.go:50
    should store data
    test/e2e/storage/testsuites/volumes.go:161
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes should store data","total":27,"completed":3,"skipped":169,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should support existing directories when readOnly specified in the volumeSource
  test/e2e/storage/testsuites/subpath.go:397
... skipping 17 lines ...
Nov 11 02:19:51.060: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comlpk4k] to have phase Bound
Nov 11 02:19:51.167: INFO: PersistentVolumeClaim test.csi.azure.comlpk4k found but phase is Pending instead of Bound.
Nov 11 02:19:53.275: INFO: PersistentVolumeClaim test.csi.azure.comlpk4k found but phase is Pending instead of Bound.
Nov 11 02:19:55.382: INFO: PersistentVolumeClaim test.csi.azure.comlpk4k found and phase=Bound (4.321890674s)
STEP: Creating pod pod-subpath-test-dynamicpv-flfm
STEP: Creating a pod to test subpath
Nov 11 02:19:55.707: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-flfm" in namespace "provisioning-9676" to be "Succeeded or Failed"
Nov 11 02:19:55.815: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 107.309531ms
Nov 11 02:19:57.922: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 2.214677108s
Nov 11 02:20:00.030: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 4.323015759s
Nov 11 02:20:02.139: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 6.43195492s
Nov 11 02:20:04.247: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 8.539684753s
Nov 11 02:20:06.355: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 10.647568499s
Nov 11 02:20:08.463: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 12.75525108s
Nov 11 02:20:10.570: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 14.86273569s
Nov 11 02:20:12.679: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 16.971423048s
Nov 11 02:20:14.786: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 19.078377928s
Nov 11 02:20:16.893: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 21.185304565s
Nov 11 02:20:19.002: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Succeeded", Reason="", readiness=false. Elapsed: 23.294482383s
STEP: Saw pod success
Nov 11 02:20:19.002: INFO: Pod "pod-subpath-test-dynamicpv-flfm" satisfied condition "Succeeded or Failed"
Nov 11 02:20:19.109: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000000 pod pod-subpath-test-dynamicpv-flfm container test-container-subpath-dynamicpv-flfm: <nil>
STEP: delete the pod
Nov 11 02:20:19.337: INFO: Waiting for pod pod-subpath-test-dynamicpv-flfm to disappear
Nov 11 02:20:19.444: INFO: Pod pod-subpath-test-dynamicpv-flfm no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-flfm
Nov 11 02:20:19.444: INFO: Deleting pod "pod-subpath-test-dynamicpv-flfm" in namespace "provisioning-9676"
STEP: Creating pod pod-subpath-test-dynamicpv-flfm
STEP: Creating a pod to test subpath
Nov 11 02:20:19.659: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-flfm" in namespace "provisioning-9676" to be "Succeeded or Failed"
Nov 11 02:20:19.766: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 106.759893ms
Nov 11 02:20:21.881: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 2.221593371s
Nov 11 02:20:23.989: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 4.329607167s
Nov 11 02:20:26.097: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437639598s
Nov 11 02:20:28.205: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 8.545499816s
Nov 11 02:20:30.313: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 10.653403397s
... skipping 11 lines ...
Nov 11 02:20:55.622: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 35.962436064s
Nov 11 02:20:57.730: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 38.070574012s
Nov 11 02:20:59.839: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 40.179852682s
Nov 11 02:21:01.947: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 42.287459786s
Nov 11 02:21:04.055: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Succeeded", Reason="", readiness=false. Elapsed: 44.396166768s
STEP: Saw pod success
Nov 11 02:21:04.056: INFO: Pod "pod-subpath-test-dynamicpv-flfm" satisfied condition "Succeeded or Failed"
Nov 11 02:21:04.163: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000000 pod pod-subpath-test-dynamicpv-flfm container test-container-subpath-dynamicpv-flfm: <nil>
STEP: delete the pod
Nov 11 02:21:04.390: INFO: Waiting for pod pod-subpath-test-dynamicpv-flfm to disappear
Nov 11 02:21:04.497: INFO: Pod pod-subpath-test-dynamicpv-flfm no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-flfm
Nov 11 02:21:04.497: INFO: Deleting pod "pod-subpath-test-dynamicpv-flfm" in namespace "provisioning-9676"
... skipping 26 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support existing directories when readOnly specified in the volumeSource
    test/e2e/storage/testsuites/subpath.go:397
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource","total":28,"completed":4,"skipped":321,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] 
  should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
  test/e2e/storage/testsuites/multivolume.go:378
... skipping 79 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:378
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":41,"completed":4,"skipped":284,"failed":0}

SSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:22:29.136: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 3 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:280

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 221 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:209
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":27,"completed":5,"skipped":269,"failed":0}

SSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:23:36.893: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping
... skipping 197 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:378
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":27,"completed":4,"skipped":228,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:24:52.139: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 24 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:242

    Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 145 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
    test/e2e/storage/testsuites/provisioning.go:208
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":38,"completed":4,"skipped":374,"failed":0}

S
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral 
  should support two pods which have the same volume definition
  test/e2e/storage/testsuites/ephemeral.go:216
... skipping 63 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should support two pods which have the same volume definition
    test/e2e/storage/testsuites/ephemeral.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should support two pods which have the same volume definition","total":28,"completed":5,"skipped":350,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:25:15.530: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 216 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:209
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":41,"completed":5,"skipped":355,"failed":0}

SSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] 
  should access to two volumes with different volume mode and retain data across pod recreation on different node
  test/e2e/storage/testsuites/multivolume.go:248
... skipping 195 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:248
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":31,"completed":5,"skipped":833,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should support existing directory
  test/e2e/storage/testsuites/subpath.go:207
... skipping 17 lines ...
Nov 11 02:23:37.862: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comqd76d] to have phase Bound
Nov 11 02:23:37.969: INFO: PersistentVolumeClaim test.csi.azure.comqd76d found but phase is Pending instead of Bound.
Nov 11 02:23:40.077: INFO: PersistentVolumeClaim test.csi.azure.comqd76d found but phase is Pending instead of Bound.
Nov 11 02:23:42.184: INFO: PersistentVolumeClaim test.csi.azure.comqd76d found and phase=Bound (4.321599989s)
STEP: Creating pod pod-subpath-test-dynamicpv-292d
STEP: Creating a pod to test subpath
Nov 11 02:23:42.507: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-292d" in namespace "provisioning-9310" to be "Succeeded or Failed"
Nov 11 02:23:42.615: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 107.036851ms
Nov 11 02:23:44.723: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 2.215453988s
Nov 11 02:23:46.830: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 4.322997243s
Nov 11 02:23:48.938: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 6.430521575s
Nov 11 02:23:51.047: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 8.539623983s
Nov 11 02:23:53.154: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 10.646753219s
... skipping 6 lines ...
Nov 11 02:24:07.912: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 25.404866305s
Nov 11 02:24:10.021: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 27.51346437s
Nov 11 02:24:12.129: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 29.621556019s
Nov 11 02:24:14.238: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 31.730095719s
Nov 11 02:24:16.345: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Succeeded", Reason="", readiness=false. Elapsed: 33.837864371s
STEP: Saw pod success
Nov 11 02:24:16.345: INFO: Pod "pod-subpath-test-dynamicpv-292d" satisfied condition "Succeeded or Failed"
Nov 11 02:24:16.453: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000001 pod pod-subpath-test-dynamicpv-292d container test-container-volume-dynamicpv-292d: <nil>
STEP: delete the pod
Nov 11 02:24:16.711: INFO: Waiting for pod pod-subpath-test-dynamicpv-292d to disappear
Nov 11 02:24:16.818: INFO: Pod pod-subpath-test-dynamicpv-292d no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-292d
Nov 11 02:24:16.818: INFO: Deleting pod "pod-subpath-test-dynamicpv-292d" in namespace "provisioning-9310"
... skipping 29 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support existing directory
    test/e2e/storage/testsuites/subpath.go:207
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory","total":27,"completed":6,"skipped":284,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:25:29.371: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 157 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support restarting containers using file as subpath [Slow][LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:333
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using file as subpath [Slow][LinuxOnly]","total":27,"completed":5,"skipped":397,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:29:05.287: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 193 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:298
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":27,"completed":7,"skipped":506,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs)] volumeLimits 
  should verify that all csinodes have volume limits
  test/e2e/storage/testsuites/volumelimits.go:249
... skipping 16 lines ...
  test/e2e/framework/framework.go:188
Nov 11 02:29:32.948: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "volumelimits-159" for this suite.

•
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs)] volumeLimits should verify that all csinodes have volume limits","total":27,"completed":8,"skipped":578,"failed":0}

SSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:29:33.190: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 13 lines ...

    test/e2e/storage/external/external.go:262
------------------------------
SSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should fail if subpath directory is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:242

{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":38,"completed":4,"skipped":282,"failed":0}
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 11 02:23:56.680: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename provisioning
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if subpath directory is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:242
Nov 11 02:23:57.432: INFO: Creating resource for dynamic PV
Nov 11 02:23:57.432: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-2722-e2e-scjxc8x
STEP: creating a claim
Nov 11 02:23:57.540: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 11 02:23:57.651: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comjw8q9] to have phase Bound
Nov 11 02:23:57.758: INFO: PersistentVolumeClaim test.csi.azure.comjw8q9 found but phase is Pending instead of Bound.
Nov 11 02:23:59.867: INFO: PersistentVolumeClaim test.csi.azure.comjw8q9 found but phase is Pending instead of Bound.
Nov 11 02:24:01.975: INFO: PersistentVolumeClaim test.csi.azure.comjw8q9 found and phase=Bound (4.324576077s)
STEP: Creating pod pod-subpath-test-dynamicpv-qqgq
STEP: Checking for subpath error in container status
Nov 11 02:25:14.515: INFO: Deleting pod "pod-subpath-test-dynamicpv-qqgq" in namespace "provisioning-2722"
Nov 11 02:25:14.624: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-qqgq" to be fully deleted
STEP: Deleting pod
Nov 11 02:25:16.839: INFO: Deleting pod "pod-subpath-test-dynamicpv-qqgq" in namespace "provisioning-2722"
STEP: Deleting pvc
Nov 11 02:25:16.947: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comjw8q9"
... skipping 59 lines ...

• [SLOW TEST:341.605 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:242
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow][LinuxOnly]","total":38,"completed":5,"skipped":282,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral 
  should support multiple inline ephemeral volumes
  test/e2e/storage/testsuites/ephemeral.go:254
... skipping 51 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should support multiple inline ephemeral volumes
    test/e2e/storage/testsuites/ephemeral.go:254
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support multiple inline ephemeral volumes","total":28,"completed":6,"skipped":394,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:30:26.696: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 245 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:248
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":31,"completed":6,"skipped":850,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral 
  should create read-only inline ephemeral volume
  test/e2e/storage/testsuites/ephemeral.go:175
... skipping 52 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should create read-only inline ephemeral volume
    test/e2e/storage/testsuites/ephemeral.go:175
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read-only inline ephemeral volume","total":27,"completed":9,"skipped":610,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 11 02:31:49.004: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename topology
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  test/e2e/storage/testsuites/topology.go:194
Nov 11 02:31:49.759: INFO: Driver didn't provide topology keys -- skipping
[AfterEach] [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/framework/framework.go:188
Nov 11 02:31:49.759: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "topology-6842" for this suite.


S [SKIPPING] [1.080 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/storage/framework/testsuite.go:50
    should fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement]
    test/e2e/storage/testsuites/topology.go:194

    Driver didn't provide topology keys -- skipping

    test/e2e/storage/testsuites/topology.go:126
------------------------------
... skipping 104 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single read-only volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:423
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":27,"completed":6,"skipped":508,"failed":0}

SSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:32:32.163: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 227 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:168
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":41,"completed":6,"skipped":376,"failed":0}

SSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] 
  should access to two volumes with the same volume mode and retain data across pod recreation on the same node
  test/e2e/storage/testsuites/multivolume.go:138
... skipping 206 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:138
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":38,"completed":5,"skipped":375,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:33:33.292: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 192 lines ...
Nov 11 02:31:51.116: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com6n7j4] to have phase Bound
Nov 11 02:31:51.227: INFO: PersistentVolumeClaim test.csi.azure.com6n7j4 found but phase is Pending instead of Bound.
Nov 11 02:31:53.337: INFO: PersistentVolumeClaim test.csi.azure.com6n7j4 found but phase is Pending instead of Bound.
Nov 11 02:31:55.446: INFO: PersistentVolumeClaim test.csi.azure.com6n7j4 found and phase=Bound (4.329889029s)
STEP: Creating pod exec-volume-test-dynamicpv-sl65
STEP: Creating a pod to test exec-volume-test
Nov 11 02:31:55.774: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-sl65" in namespace "volume-7259" to be "Succeeded or Failed"
Nov 11 02:31:55.884: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 110.0421ms
Nov 11 02:31:57.993: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218667676s
Nov 11 02:32:00.102: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327999878s
Nov 11 02:32:02.212: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 6.43755528s
Nov 11 02:32:04.322: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 8.54759254s
Nov 11 02:32:06.432: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 10.657437246s
... skipping 29 lines ...
Nov 11 02:33:09.718: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 1m13.943878314s
Nov 11 02:33:11.829: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.054575768s
Nov 11 02:33:13.940: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.16533005s
Nov 11 02:33:16.048: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.274031565s
Nov 11 02:33:18.157: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m22.382642458s
STEP: Saw pod success
Nov 11 02:33:18.157: INFO: Pod "exec-volume-test-dynamicpv-sl65" satisfied condition "Succeeded or Failed"
Nov 11 02:33:18.265: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000001 pod exec-volume-test-dynamicpv-sl65 container exec-container-dynamicpv-sl65: <nil>
STEP: delete the pod
Nov 11 02:33:18.514: INFO: Waiting for pod exec-volume-test-dynamicpv-sl65 to disappear
Nov 11 02:33:18.623: INFO: Pod exec-volume-test-dynamicpv-sl65 no longer exists
STEP: Deleting pod exec-volume-test-dynamicpv-sl65
Nov 11 02:33:18.623: INFO: Deleting pod "exec-volume-test-dynamicpv-sl65" in namespace "volume-7259"
... skipping 27 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should allow exec of files on the volume
    test/e2e/storage/testsuites/volumes.go:198
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should allow exec of files on the volume","total":27,"completed":10,"skipped":704,"failed":0}

SSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (xfs)][Slow] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:34:31.034: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 253 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should not mount / map unused volumes in a pod [LinuxOnly]
    test/e2e/storage/testsuites/volumemode.go:354
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":31,"completed":7,"skipped":908,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeLimits 
  should verify that all csinodes have volume limits
  test/e2e/storage/testsuites/volumelimits.go:249
... skipping 16 lines ...
  test/e2e/framework/framework.go:188
Nov 11 02:35:35.688: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "volumelimits-2896" for this suite.

•
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeLimits should verify that all csinodes have volume limits","total":31,"completed":8,"skipped":942,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]}

SSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:35:35.925: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 175 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should store data
    test/e2e/storage/testsuites/volumes.go:161
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumes should store data","total":38,"completed":6,"skipped":646,"failed":0}

SSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:35:52.804: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 363 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with pvc data source in parallel [Slow]
    test/e2e/storage/testsuites/provisioning.go:459
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with pvc data source in parallel [Slow]","total":27,"completed":7,"skipped":576,"failed":0}

SSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:36:00.934: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 141 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:323
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":38,"completed":6,"skipped":310,"failed":0}

SSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:280

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 11 02:35:52.864: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename provisioning
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:280
Nov 11 02:35:53.620: INFO: Creating resource for dynamic PV
Nov 11 02:35:53.620: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-127-e2e-sccgj64
STEP: creating a claim
Nov 11 02:35:53.728: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 11 02:35:53.839: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comkk9cv] to have phase Bound
Nov 11 02:35:53.947: INFO: PersistentVolumeClaim test.csi.azure.comkk9cv found but phase is Pending instead of Bound.
Nov 11 02:35:56.056: INFO: PersistentVolumeClaim test.csi.azure.comkk9cv found but phase is Pending instead of Bound.
Nov 11 02:35:58.165: INFO: PersistentVolumeClaim test.csi.azure.comkk9cv found and phase=Bound (4.326052114s)
STEP: Creating pod pod-subpath-test-dynamicpv-t2s6
STEP: Checking for subpath error in container status
Nov 11 02:36:16.711: INFO: Deleting pod "pod-subpath-test-dynamicpv-t2s6" in namespace "provisioning-127"
Nov 11 02:36:16.822: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-t2s6" to be fully deleted
STEP: Deleting pod
Nov 11 02:36:19.039: INFO: Deleting pod "pod-subpath-test-dynamicpv-t2s6" in namespace "provisioning-127"
STEP: Deleting pvc
Nov 11 02:36:19.147: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comkk9cv"
... skipping 12 lines ...

• [SLOW TEST:47.477 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:280
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]","total":38,"completed":7,"skipped":712,"failed":0}

SSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  
  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
  test/e2e/storage/testsuites/snapshottable.go:278

{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":28,"completed":7,"skipped":614,"failed":0}
[BeforeEach] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 11 02:34:14.628: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json
... skipping 10 lines ...
Nov 11 02:34:15.599: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comwmppf] to have phase Bound
Nov 11 02:34:15.707: INFO: PersistentVolumeClaim test.csi.azure.comwmppf found but phase is Pending instead of Bound.
Nov 11 02:34:17.816: INFO: PersistentVolumeClaim test.csi.azure.comwmppf found but phase is Pending instead of Bound.
Nov 11 02:34:19.924: INFO: PersistentVolumeClaim test.csi.azure.comwmppf found and phase=Bound (4.324920315s)
STEP: [init] starting a pod to use the claim
STEP: [init] check pod success
Nov 11 02:34:20.359: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-ng9td" in namespace "snapshotting-4341" to be "Succeeded or Failed"
Nov 11 02:34:20.467: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 108.003837ms
Nov 11 02:34:22.576: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217358408s
Nov 11 02:34:24.685: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 4.325770563s
Nov 11 02:34:26.794: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 6.43501537s
Nov 11 02:34:28.905: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 8.545881275s
Nov 11 02:34:31.015: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 10.655694946s
Nov 11 02:34:33.123: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 12.763886413s
Nov 11 02:34:35.230: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 14.871507913s
Nov 11 02:34:37.347: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Succeeded", Reason="", readiness=false. Elapsed: 16.987807798s
STEP: Saw pod success
Nov 11 02:34:37.347: INFO: Pod "pvc-snapshottable-tester-ng9td" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
Nov 11 02:34:37.455: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comwmppf] to have phase Bound
Nov 11 02:34:37.562: INFO: PersistentVolumeClaim test.csi.azure.comwmppf found and phase=Bound (107.403429ms)
STEP: [init] checking the PV
STEP: [init] deleting the pod
Nov 11 02:34:37.912: INFO: Pod pvc-snapshottable-tester-ng9td has the following logs: 
... skipping 14 lines ...
Nov 11 02:34:47.552: INFO: received snapshotStatus map[boundVolumeSnapshotContentName:snapcontent-9cfb2f8c-b3c6-4d25-bfbe-0d82e90a57a2 creationTime:2022-11-11T02:34:43Z readyToUse:true restoreSize:5Gi]
Nov 11 02:34:47.552: INFO: snapshotContentName snapcontent-9cfb2f8c-b3c6-4d25-bfbe-0d82e90a57a2
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Nov 11 02:34:47.998: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-5wjng" in namespace "snapshotting-4341" to be "Succeeded or Failed"
Nov 11 02:34:48.106: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 107.821484ms
Nov 11 02:34:50.215: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216907877s
Nov 11 02:34:52.324: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326360323s
Nov 11 02:34:54.433: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 6.435187992s
Nov 11 02:34:56.542: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 8.543664525s
Nov 11 02:34:58.650: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 10.652380336s
... skipping 21 lines ...
Nov 11 02:35:45.067: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 57.06892718s
Nov 11 02:35:47.175: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 59.177239895s
Nov 11 02:35:49.283: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.285407094s
Nov 11 02:35:51.391: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.393288198s
Nov 11 02:35:53.500: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m5.501541883s
STEP: Saw pod success
Nov 11 02:35:53.500: INFO: Pod "pvc-snapshottable-data-tester-5wjng" satisfied condition "Succeeded or Failed"
Nov 11 02:35:53.749: INFO: Pod pvc-snapshottable-data-tester-5wjng has the following logs: 
Nov 11 02:35:53.749: INFO: Deleting pod "pvc-snapshottable-data-tester-5wjng" in namespace "snapshotting-4341"
Nov 11 02:35:53.867: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-5wjng" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the snapshot
Nov 11 02:36:18.415: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-9juvoppw.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-4341 exec restored-pvc-tester-frtr8 --namespace=snapshotting-4341 -- cat /mnt/test/data'
... skipping 47 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        test/e2e/storage/testsuites/snapshottable.go:278
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":28,"completed":8,"skipped":614,"failed":0}

SSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
... skipping 226 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:248
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":41,"completed":7,"skipped":396,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:37:14.929: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 187 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents","total":27,"completed":8,"skipped":648,"failed":0}

SSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:37:54.708: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 125 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:242

    Distro debian doesn't support ntfs -- skipping

    test/e2e/storage/framework/testsuite.go:127
------------------------------
... skipping 96 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":38,"completed":8,"skipped":719,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] volume-expand
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:38:18.646: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 155 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents","total":38,"completed":7,"skipped":314,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:38:55.060: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 13 lines ...

    test/e2e/storage/external/external.go:262
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:269

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 11 02:37:15.993: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename provisioning
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:269
Nov 11 02:37:16.740: INFO: Creating resource for dynamic PV
Nov 11 02:37:16.740: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-9449-e2e-sc55khs
STEP: creating a claim
Nov 11 02:37:16.848: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 11 02:37:16.957: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comjf52l] to have phase Bound
Nov 11 02:37:17.063: INFO: PersistentVolumeClaim test.csi.azure.comjf52l found but phase is Pending instead of Bound.
Nov 11 02:37:19.172: INFO: PersistentVolumeClaim test.csi.azure.comjf52l found but phase is Pending instead of Bound.
Nov 11 02:37:21.280: INFO: PersistentVolumeClaim test.csi.azure.comjf52l found and phase=Bound (4.322959341s)
STEP: Creating pod pod-subpath-test-dynamicpv-85l8
STEP: Checking for subpath error in container status
Nov 11 02:37:51.819: INFO: Deleting pod "pod-subpath-test-dynamicpv-85l8" in namespace "provisioning-9449"
Nov 11 02:37:51.927: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-85l8" to be fully deleted
STEP: Deleting pod
Nov 11 02:37:54.142: INFO: Deleting pod "pod-subpath-test-dynamicpv-85l8" in namespace "provisioning-9449"
STEP: Deleting pvc
Nov 11 02:37:54.250: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comjf52l"
... skipping 22 lines ...

• [SLOW TEST:110.566 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:269
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]","total":41,"completed":8,"skipped":519,"failed":0}

SSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 11 02:39:06.565: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename topology
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  test/e2e/storage/testsuites/topology.go:194
Nov 11 02:39:07.319: INFO: Driver didn't provide topology keys -- skipping
[AfterEach] [Testpattern: Dynamic PV (delayed binding)] topology
  test/e2e/framework/framework.go:188
Nov 11 02:39:07.319: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "topology-9817" for this suite.


S [SKIPPING] [1.077 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (delayed binding)] topology
  test/e2e/storage/framework/testsuite.go:50
    should fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement]
    test/e2e/storage/testsuites/topology.go:194

    Driver didn't provide topology keys -- skipping

    test/e2e/storage/testsuites/topology.go:126
------------------------------
... skipping 216 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:138
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":28,"completed":9,"skipped":651,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should be able to unmount after the subpath directory is deleted [LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:447
... skipping 57 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should be able to unmount after the subpath directory is deleted [LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:447
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [LinuxOnly]","total":38,"completed":9,"skipped":767,"failed":0}

SSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
... skipping 164 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single read-only volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:423
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":27,"completed":9,"skipped":833,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:40:31.306: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 50 lines ...
[It] should check snapshot fields, check restore correctly works, check deletion (ephemeral)
  test/e2e/storage/testsuites/snapshottable.go:177
Nov 11 02:38:55.905: INFO: Creating resource for dynamic PV
Nov 11 02:38:55.905: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass snapshotting-3890-e2e-sc8skz4
STEP: [init] starting a pod to use the claim
Nov 11 02:38:56.132: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-dcj5r" in namespace "snapshotting-3890" to be "Succeeded or Failed"
Nov 11 02:38:56.240: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 108.052397ms
Nov 11 02:38:58.349: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217528287s
Nov 11 02:39:00.459: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327553332s
Nov 11 02:39:02.568: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 6.436440015s
Nov 11 02:39:04.677: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 8.545542001s
Nov 11 02:39:06.786: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 10.653956435s
... skipping 3 lines ...
Nov 11 02:39:15.225: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 19.093501975s
Nov 11 02:39:17.335: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 21.203283437s
Nov 11 02:39:19.445: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 23.312934475s
Nov 11 02:39:21.554: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 25.421835726s
Nov 11 02:39:23.662: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Succeeded", Reason="", readiness=false. Elapsed: 27.530721068s
STEP: Saw pod success
Nov 11 02:39:23.662: INFO: Pod "pvc-snapshottable-tester-dcj5r" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
STEP: creating a SnapshotClass
STEP: creating a dynamic VolumeSnapshot
Nov 11 02:39:24.101: INFO: Waiting up to 5m0s for VolumeSnapshot snapshot-9hcb6 to become ready
Nov 11 02:39:24.209: INFO: VolumeSnapshot snapshot-9hcb6 found but is not ready.
Nov 11 02:39:26.319: INFO: VolumeSnapshot snapshot-9hcb6 found but is not ready.
... skipping 49 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works, check deletion (ephemeral)
        test/e2e/storage/testsuites/snapshottable.go:177
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works, check deletion (ephemeral)","total":38,"completed":8,"skipped":445,"failed":0}

SSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:40:53.854: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 24 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:242

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 156 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support restarting containers using directory as subpath [Slow]
    test/e2e/storage/testsuites/subpath.go:322
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using directory as subpath [Slow]","total":41,"completed":9,"skipped":637,"failed":0}

SSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:42:47.505: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 73 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should create read/write inline ephemeral volume
    test/e2e/storage/testsuites/ephemeral.go:196
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read/write inline ephemeral volume","total":38,"completed":9,"skipped":578,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  
  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
  test/e2e/storage/testsuites/snapshottable.go:278
... skipping 17 lines ...
Nov 11 02:39:15.258: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.combwbhs] to have phase Bound
Nov 11 02:39:15.366: INFO: PersistentVolumeClaim test.csi.azure.combwbhs found but phase is Pending instead of Bound.
Nov 11 02:39:17.474: INFO: PersistentVolumeClaim test.csi.azure.combwbhs found but phase is Pending instead of Bound.
Nov 11 02:39:19.583: INFO: PersistentVolumeClaim test.csi.azure.combwbhs found and phase=Bound (4.324754524s)
STEP: [init] starting a pod to use the claim
STEP: [init] check pod success
Nov 11 02:39:20.016: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-z7gpt" in namespace "snapshotting-955" to be "Succeeded or Failed"
Nov 11 02:39:20.124: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 107.828133ms
Nov 11 02:39:22.233: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21743853s
Nov 11 02:39:24.343: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327124906s
Nov 11 02:39:26.458: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 6.441766709s
Nov 11 02:39:28.567: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 8.551486557s
Nov 11 02:39:30.676: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 10.659882193s
... skipping 22 lines ...
Nov 11 02:40:19.215: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 59.199081097s
Nov 11 02:40:21.325: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.309019171s
Nov 11 02:40:23.433: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.417167317s
Nov 11 02:40:25.543: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.526579133s
Nov 11 02:40:27.652: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m7.635742965s
STEP: Saw pod success
Nov 11 02:40:27.652: INFO: Pod "pvc-snapshottable-tester-z7gpt" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
Nov 11 02:40:27.760: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.combwbhs] to have phase Bound
Nov 11 02:40:27.869: INFO: PersistentVolumeClaim test.csi.azure.combwbhs found and phase=Bound (108.928842ms)
STEP: [init] checking the PV
STEP: [init] deleting the pod
Nov 11 02:40:28.221: INFO: Pod pvc-snapshottable-tester-z7gpt has the following logs: 
... skipping 37 lines ...
Nov 11 02:40:45.485: INFO: WaitUntil finished successfully after 108.729202ms
STEP: getting the snapshot and snapshot content
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Nov 11 02:40:46.029: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-zbnw9" in namespace "snapshotting-955" to be "Succeeded or Failed"
Nov 11 02:40:46.136: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 107.702518ms
Nov 11 02:40:48.246: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217101798s
Nov 11 02:40:50.354: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 4.325598452s
Nov 11 02:40:52.462: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 6.433478998s
Nov 11 02:40:54.571: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 8.542553515s
Nov 11 02:40:56.681: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 10.652204926s
... skipping 33 lines ...
Nov 11 02:42:08.435: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.406259253s
Nov 11 02:42:10.544: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.514953457s
Nov 11 02:42:12.654: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.624900306s
Nov 11 02:42:14.763: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.733750842s
Nov 11 02:42:16.871: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m30.842305376s
STEP: Saw pod success
Nov 11 02:42:16.871: INFO: Pod "pvc-snapshottable-data-tester-zbnw9" satisfied condition "Succeeded or Failed"
Nov 11 02:42:17.091: INFO: Pod pvc-snapshottable-data-tester-zbnw9 has the following logs: 
Nov 11 02:42:17.091: INFO: Deleting pod "pvc-snapshottable-data-tester-zbnw9" in namespace "snapshotting-955"
Nov 11 02:42:17.201: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-zbnw9" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the snapshot
Nov 11 02:43:43.749: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-9juvoppw.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-955 exec restored-pvc-tester-g7lx8 --namespace=snapshotting-955 -- cat /mnt/test/data'
... skipping 48 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        test/e2e/storage/testsuites/snapshottable.go:278
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":28,"completed":10,"skipped":716,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (default fs)] capacity
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:44:33.169: INFO: Driver test.csi.azure.com doesn't publish storage capacity -- skipping
... skipping 142 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:323
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":38,"completed":10,"skipped":789,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource]
... skipping 30 lines ...

        test/e2e/storage/testsuites/snapshottable.go:179
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode 
  should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299

[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 11 02:42:47.535: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename volumemode
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299
Nov 11 02:42:48.288: INFO: Creating resource for dynamic PV
Nov 11 02:42:48.288: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass volumemode-2382-e2e-sc24hx2
STEP: creating a claim
Nov 11 02:42:48.507: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comlc6tg] to have phase Bound
Nov 11 02:42:48.615: INFO: PersistentVolumeClaim test.csi.azure.comlc6tg found but phase is Pending instead of Bound.
Nov 11 02:42:50.724: INFO: PersistentVolumeClaim test.csi.azure.comlc6tg found but phase is Pending instead of Bound.
Nov 11 02:42:52.834: INFO: PersistentVolumeClaim test.csi.azure.comlc6tg found and phase=Bound (4.32651136s)
STEP: Creating pod
STEP: Waiting for the pod to fail
Nov 11 02:42:55.484: INFO: Deleting pod "pod-a6809ccf-5ae9-4cef-a7ad-18cad857459b" in namespace "volumemode-2382"
Nov 11 02:42:55.593: INFO: Wait up to 5m0s for pod "pod-a6809ccf-5ae9-4cef-a7ad-18cad857459b" to be fully deleted
STEP: Deleting pvc
Nov 11 02:42:57.809: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comlc6tg"
Nov 11 02:42:57.918: INFO: Waiting up to 5m0s for PersistentVolume pvc-1fb3e23a-88f4-4267-af34-aeba0686f77f to get deleted
Nov 11 02:42:58.026: INFO: PersistentVolume pvc-1fb3e23a-88f4-4267-af34-aeba0686f77f found and phase=Released (107.671686ms)
... skipping 32 lines ...

• [SLOW TEST:143.878 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow]
    test/e2e/storage/testsuites/volumemode.go:299
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should fail to use a volume in a pod with mismatched mode [Slow]","total":41,"completed":10,"skipped":675,"failed":0}
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] volumeIO
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:45:11.414: INFO: Distro debian doesn't support ntfs -- skipping
[AfterEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] volumeIO
  test/e2e/framework/framework.go:188

... skipping 346 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with pvc data source in parallel [Slow]
    test/e2e/storage/testsuites/provisioning.go:459
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with pvc data source in parallel [Slow]","total":27,"completed":10,"skipped":993,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
Nov 11 02:46:07.971: INFO: Running AfterSuite actions on all nodes
Nov 11 02:46:07.971: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2
Nov 11 02:46:07.971: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2
... skipping 109 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with pvc data source
    test/e2e/storage/testsuites/provisioning.go:421
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with pvc data source","total":38,"completed":10,"skipped":602,"failed":0}

SSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning
... skipping 241 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:209
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":38,"completed":11,"skipped":896,"failed":0}

SSSSSSSS
------------------------------
[BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:46:56.693: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping
... skipping 24 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach]
    test/e2e/storage/testsuites/volumemode.go:299

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 43 lines ...
Nov 11 02:45:12.398: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comq58vv] to have phase Bound
Nov 11 02:45:12.506: INFO: PersistentVolumeClaim test.csi.azure.comq58vv found but phase is Pending instead of Bound.
Nov 11 02:45:14.615: INFO: PersistentVolumeClaim test.csi.azure.comq58vv found but phase is Pending instead of Bound.
Nov 11 02:45:16.723: INFO: PersistentVolumeClaim test.csi.azure.comq58vv found and phase=Bound (4.324922236s)
STEP: Creating pod pod-subpath-test-dynamicpv-6t86
STEP: Creating a pod to test multi_subpath
Nov 11 02:45:17.052: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-6t86" in namespace "provisioning-8993" to be "Succeeded or Failed"
Nov 11 02:45:17.162: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 109.339313ms
Nov 11 02:45:19.272: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219112808s
Nov 11 02:45:21.381: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328486719s
Nov 11 02:45:23.489: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 6.43697952s
Nov 11 02:45:25.598: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 8.545798581s
Nov 11 02:45:27.709: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 10.65659223s
... skipping 25 lines ...
Nov 11 02:46:22.557: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.504570081s
Nov 11 02:46:24.666: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.613713204s
Nov 11 02:46:26.779: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 1m9.726521029s
Nov 11 02:46:28.887: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 1m11.834745993s
Nov 11 02:46:30.997: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m13.944468758s
STEP: Saw pod success
Nov 11 02:46:30.997: INFO: Pod "pod-subpath-test-dynamicpv-6t86" satisfied condition "Succeeded or Failed"
Nov 11 02:46:31.105: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000001 pod pod-subpath-test-dynamicpv-6t86 container test-container-subpath-dynamicpv-6t86: <nil>
STEP: delete the pod
Nov 11 02:46:31.331: INFO: Waiting for pod pod-subpath-test-dynamicpv-6t86 to disappear
Nov 11 02:46:31.439: INFO: Pod pod-subpath-test-dynamicpv-6t86 no longer exists
STEP: Deleting pod
Nov 11 02:46:31.439: INFO: Deleting pod "pod-subpath-test-dynamicpv-6t86" in namespace "provisioning-8993"
... skipping 21 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support creating multiple subpath from same volumes [Slow]
    test/e2e/storage/testsuites/subpath.go:296
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support creating multiple subpath from same volumes [Slow]","total":41,"completed":11,"skipped":678,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (ext4)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:47:13.219: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 31 lines ...
[It] should check snapshot fields, check restore correctly works, check deletion (ephemeral)
  test/e2e/storage/testsuites/snapshottable.go:177
Nov 11 02:44:33.927: INFO: Creating resource for dynamic PV
Nov 11 02:44:33.927: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass snapshotting-4372-e2e-scgwfrv
STEP: [init] starting a pod to use the claim
Nov 11 02:44:34.146: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-ghnh8" in namespace "snapshotting-4372" to be "Succeeded or Failed"
Nov 11 02:44:34.254: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 108.071902ms
Nov 11 02:44:36.363: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216632772s
Nov 11 02:44:38.472: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326098977s
Nov 11 02:44:40.583: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.43637352s
Nov 11 02:44:42.692: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.545293875s
Nov 11 02:44:44.800: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.653902068s
... skipping 5 lines ...
Nov 11 02:44:57.455: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 23.308267129s
Nov 11 02:44:59.563: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 25.417045225s
Nov 11 02:45:01.673: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 27.527021534s
Nov 11 02:45:03.783: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 29.63646875s
Nov 11 02:45:05.892: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Succeeded", Reason="", readiness=false. Elapsed: 31.745463732s
STEP: Saw pod success
Nov 11 02:45:05.892: INFO: Pod "pvc-snapshottable-tester-ghnh8" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
STEP: creating a SnapshotClass
STEP: creating a dynamic VolumeSnapshot
Nov 11 02:45:06.328: INFO: Waiting up to 5m0s for VolumeSnapshot snapshot-lqqrv to become ready
Nov 11 02:45:06.437: INFO: VolumeSnapshot snapshot-lqqrv found but is not ready.
Nov 11 02:45:08.549: INFO: VolumeSnapshot snapshot-lqqrv found but is not ready.
... skipping 40 lines ...
Nov 11 02:46:22.479: INFO: volumesnapshotcontents snapcontent-8ca5136b-8b31-4b13-aaa2-702a532783cc has been found and is not deleted
Nov 11 02:46:23.589: INFO: volumesnapshotcontents snapcontent-8ca5136b-8b31-4b13-aaa2-702a532783cc has been found and is not deleted
Nov 11 02:46:24.699: INFO: volumesnapshotcontents snapcontent-8ca5136b-8b31-4b13-aaa2-702a532783cc has been found and is not deleted
Nov 11 02:46:25.807: INFO: volumesnapshotcontents snapcontent-8ca5136b-8b31-4b13-aaa2-702a532783cc has been found and is not deleted
Nov 11 02:46:26.916: INFO: volumesnapshotcontents snapcontent-8ca5136b-8b31-4b13-aaa2-702a532783cc has been found and is not deleted
Nov 11 02:46:28.025: INFO: volumesnapshotcontents snapcontent-8ca5136b-8b31-4b13-aaa2-702a532783cc has been found and is not deleted
Nov 11 02:46:29.025: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 11 02:46:29.160: INFO: Pod restored-pvc-tester-f8svc has the following logs: 
Nov 11 02:46:29.160: INFO: Deleting pod "restored-pvc-tester-f8svc" in namespace "snapshotting-4372"
Nov 11 02:46:29.270: INFO: Wait up to 5m0s for pod "restored-pvc-tester-f8svc" to be fully deleted
Nov 11 02:47:01.486: INFO: deleting snapshot "snapshotting-4372"/"snapshot-lqqrv"
... skipping 31 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works, check deletion (ephemeral)
        test/e2e/storage/testsuites/snapshottable.go:177
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Ephemeral Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works, check deletion (ephemeral)","total":28,"completed":11,"skipped":778,"failed":0}

SSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning 
  should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
  test/e2e/storage/testsuites/provisioning.go:208
... skipping 126 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
    test/e2e/storage/testsuites/provisioning.go:208
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":41,"completed":12,"skipped":733,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning 
  should provision storage with pvc data source
  test/e2e/storage/testsuites/provisioning.go:421
... skipping 103 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with pvc data source
    test/e2e/storage/testsuites/provisioning.go:421
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with pvc data source","total":38,"completed":12,"skipped":960,"failed":0}

SSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:49:55.395: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 3 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:258

    Distro debian doesn't support ntfs -- skipping

    test/e2e/storage/framework/testsuite.go:127
------------------------------
... skipping 193 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:168
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":28,"completed":12,"skipped":799,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral 
  should support two pods which have the same volume definition
  test/e2e/storage/testsuites/ephemeral.go:216
... skipping 70 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should support two pods which have the same volume definition
    test/e2e/storage/testsuites/ephemeral.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support two pods which have the same volume definition","total":38,"completed":11,"skipped":714,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:50:51.437: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 108 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should create read/write inline ephemeral volume
    test/e2e/storage/testsuites/ephemeral.go:196
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read/write inline ephemeral volume","total":41,"completed":13,"skipped":815,"failed":0}

SSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes 
  should store data
  test/e2e/storage/testsuites/volumes.go:161
... skipping 108 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should store data
    test/e2e/storage/testsuites/volumes.go:161
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should store data","total":38,"completed":13,"skipped":1012,"failed":0}

SSSSSSS
------------------------------
Nov 11 02:51:50.050: INFO: Running AfterSuite actions on all nodes
Nov 11 02:51:50.050: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2
Nov 11 02:51:50.050: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2
... skipping 26 lines ...
Nov 11 02:50:29.931: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 11 02:50:30.041: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.combjp2v] to have phase Bound
Nov 11 02:50:30.148: INFO: PersistentVolumeClaim test.csi.azure.combjp2v found but phase is Pending instead of Bound.
Nov 11 02:50:32.258: INFO: PersistentVolumeClaim test.csi.azure.combjp2v found but phase is Pending instead of Bound.
Nov 11 02:50:34.366: INFO: PersistentVolumeClaim test.csi.azure.combjp2v found and phase=Bound (4.32506118s)
STEP: Creating pod to format volume volume-prep-provisioning-4376
Nov 11 02:50:34.690: INFO: Waiting up to 5m0s for pod "volume-prep-provisioning-4376" in namespace "provisioning-4376" to be "Succeeded or Failed"
Nov 11 02:50:34.798: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 107.698244ms
Nov 11 02:50:36.907: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216602916s
Nov 11 02:50:39.016: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 4.325446657s
Nov 11 02:50:41.124: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 6.434378167s
Nov 11 02:50:43.237: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547263587s
Nov 11 02:50:45.346: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 10.656342627s
Nov 11 02:50:47.455: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 12.764506185s
Nov 11 02:50:49.564: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 14.873448023s
Nov 11 02:50:51.672: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 16.982095919s
Nov 11 02:50:53.783: INFO: Pod "volume-prep-provisioning-4376": Phase="Succeeded", Reason="", readiness=false. Elapsed: 19.092585067s
STEP: Saw pod success
Nov 11 02:50:53.783: INFO: Pod "volume-prep-provisioning-4376" satisfied condition "Succeeded or Failed"
Nov 11 02:50:53.783: INFO: Deleting pod "volume-prep-provisioning-4376" in namespace "provisioning-4376"
Nov 11 02:50:53.901: INFO: Wait up to 5m0s for pod "volume-prep-provisioning-4376" to be fully deleted
STEP: Creating pod pod-subpath-test-dynamicpv-v5fj
STEP: Checking for subpath error in container status
Nov 11 02:51:38.336: INFO: Deleting pod "pod-subpath-test-dynamicpv-v5fj" in namespace "provisioning-4376"
Nov 11 02:51:38.445: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-v5fj" to be fully deleted
STEP: Deleting pod
Nov 11 02:51:40.660: INFO: Deleting pod "pod-subpath-test-dynamicpv-v5fj" in namespace "provisioning-4376"
STEP: Deleting pvc
Nov 11 02:51:40.767: INFO: Deleting PersistentVolumeClaim "test.csi.azure.combjp2v"
... skipping 22 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should verify container cannot write to subpath readonly volumes [Slow]
    test/e2e/storage/testsuites/subpath.go:425
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should verify container cannot write to subpath readonly volumes [Slow]","total":28,"completed":13,"skipped":862,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:52:37.738: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 38 lines ...
Nov 11 02:50:52.465: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comh8ftg] to have phase Bound
Nov 11 02:50:52.573: INFO: PersistentVolumeClaim test.csi.azure.comh8ftg found but phase is Pending instead of Bound.
Nov 11 02:50:54.682: INFO: PersistentVolumeClaim test.csi.azure.comh8ftg found but phase is Pending instead of Bound.
Nov 11 02:50:56.791: INFO: PersistentVolumeClaim test.csi.azure.comh8ftg found and phase=Bound (4.326144809s)
STEP: [init] starting a pod to use the claim
STEP: [init] check pod success
Nov 11 02:50:57.229: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-ps5w4" in namespace "snapshotting-6604" to be "Succeeded or Failed"
Nov 11 02:50:57.338: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 108.013847ms
Nov 11 02:50:59.446: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216514311s
Nov 11 02:51:01.555: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 4.325971556s
Nov 11 02:51:03.665: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 6.435216522s
Nov 11 02:51:05.774: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 8.5440876s
Nov 11 02:51:07.882: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 10.652587059s
... skipping 6 lines ...
Nov 11 02:51:22.647: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 25.41727561s
Nov 11 02:51:24.756: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 27.52655733s
Nov 11 02:51:26.865: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 29.635544101s
Nov 11 02:51:28.974: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 31.744927803s
Nov 11 02:51:31.083: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Succeeded", Reason="", readiness=false. Elapsed: 33.853726164s
STEP: Saw pod success
Nov 11 02:51:31.083: INFO: Pod "pvc-snapshottable-tester-ps5w4" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
Nov 11 02:51:31.192: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comh8ftg] to have phase Bound
Nov 11 02:51:31.300: INFO: PersistentVolumeClaim test.csi.azure.comh8ftg found and phase=Bound (107.756862ms)
STEP: [init] checking the PV
STEP: [init] deleting the pod
Nov 11 02:51:31.648: INFO: Pod pvc-snapshottable-tester-ps5w4 has the following logs: 
... skipping 35 lines ...
Nov 11 02:51:44.478: INFO: WaitUntil finished successfully after 107.99982ms
STEP: getting the snapshot and snapshot content
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Nov 11 02:51:45.020: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-b97wp" in namespace "snapshotting-6604" to be "Succeeded or Failed"
Nov 11 02:51:45.127: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 107.517137ms
Nov 11 02:51:47.237: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217197023s
Nov 11 02:51:49.346: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326059788s
Nov 11 02:51:51.456: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.435722444s
Nov 11 02:51:53.565: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.544855964s
Nov 11 02:51:55.673: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.653123972s
... skipping 12 lines ...
Nov 11 02:52:23.096: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 38.075595941s
Nov 11 02:52:25.204: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 40.183722572s
Nov 11 02:52:27.313: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 42.293008582s
Nov 11 02:52:29.423: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 44.402577628s
Nov 11 02:52:31.533: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 46.51272373s
STEP: Saw pod success
Nov 11 02:52:31.533: INFO: Pod "pvc-snapshottable-data-tester-b97wp" satisfied condition "Succeeded or Failed"
Nov 11 02:52:31.753: INFO: Pod pvc-snapshottable-data-tester-b97wp has the following logs: 
Nov 11 02:52:31.753: INFO: Deleting pod "pvc-snapshottable-data-tester-b97wp" in namespace "snapshotting-6604"
Nov 11 02:52:31.867: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-b97wp" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the snapshot
Nov 11 02:53:04.412: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-9juvoppw.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-6604 exec restored-pvc-tester-8s5kn --namespace=snapshotting-6604 -- cat /mnt/test/data'
... skipping 33 lines ...
Nov 11 02:53:30.703: INFO: volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 has been found and is not deleted
Nov 11 02:53:31.812: INFO: volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 has been found and is not deleted
Nov 11 02:53:32.921: INFO: volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 has been found and is not deleted
Nov 11 02:53:34.030: INFO: volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 has been found and is not deleted
Nov 11 02:53:35.139: INFO: volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 has been found and is not deleted
Nov 11 02:53:36.248: INFO: volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 has been found and is not deleted
Nov 11 02:53:37.249: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 11 02:53:37.357: INFO: Error getting logs for pod restored-pvc-tester-8s5kn: the server could not find the requested resource (get pods restored-pvc-tester-8s5kn)
Nov 11 02:53:37.357: INFO: Deleting pod "restored-pvc-tester-8s5kn" in namespace "snapshotting-6604"
Nov 11 02:53:37.464: INFO: deleting claim "snapshotting-6604"/"pvc-psmfp"
Nov 11 02:53:37.572: INFO: deleting snapshot "snapshotting-6604"/"pre-provisioned-snapshot-f6460a03-ec05-4e57-9c9e-3609b18c3cc7"
Nov 11 02:53:37.680: INFO: deleting snapshot content "pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7"
Nov 11 02:53:38.009: INFO: Waiting up to 5m0s for volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 to be deleted
Nov 11 02:53:38.117: INFO: volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 has been found and is not deleted
... skipping 27 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        test/e2e/storage/testsuites/snapshottable.go:278
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":38,"completed":12,"skipped":834,"failed":0}

SSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:53:55.645: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 24 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:269

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 122 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should support two pods which have the same volume definition
    test/e2e/storage/testsuites/ephemeral.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support two pods which have the same volume definition","total":41,"completed":14,"skipped":819,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:54:41.192: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 102 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:378
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":28,"completed":14,"skipped":904,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (ntfs)][Feature:Windows] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:55:04.337: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 205 lines ...
Nov 11 02:54:42.194: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com5dqsl] to have phase Bound
Nov 11 02:54:42.302: INFO: PersistentVolumeClaim test.csi.azure.com5dqsl found but phase is Pending instead of Bound.
Nov 11 02:54:44.410: INFO: PersistentVolumeClaim test.csi.azure.com5dqsl found but phase is Pending instead of Bound.
Nov 11 02:54:46.518: INFO: PersistentVolumeClaim test.csi.azure.com5dqsl found and phase=Bound (4.323904229s)
STEP: Creating pod exec-volume-test-dynamicpv-znw6
STEP: Creating a pod to test exec-volume-test
Nov 11 02:54:46.840: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-znw6" in namespace "volume-4145" to be "Succeeded or Failed"
Nov 11 02:54:46.948: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 107.284994ms
Nov 11 02:54:49.056: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21599831s
Nov 11 02:54:51.164: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 4.324047074s
Nov 11 02:54:53.271: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 6.430772413s
Nov 11 02:54:55.379: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 8.539219222s
Nov 11 02:54:57.487: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 10.646830909s
Nov 11 02:54:59.595: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 12.7545404s
Nov 11 02:55:01.703: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 14.862930665s
Nov 11 02:55:03.811: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 16.970365321s
Nov 11 02:55:05.918: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Succeeded", Reason="", readiness=false. Elapsed: 19.077682475s
STEP: Saw pod success
Nov 11 02:55:05.918: INFO: Pod "exec-volume-test-dynamicpv-znw6" satisfied condition "Succeeded or Failed"
Nov 11 02:55:06.025: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000000 pod exec-volume-test-dynamicpv-znw6 container exec-container-dynamicpv-znw6: <nil>
STEP: delete the pod
Nov 11 02:55:06.277: INFO: Waiting for pod exec-volume-test-dynamicpv-znw6 to disappear
Nov 11 02:55:06.384: INFO: Pod exec-volume-test-dynamicpv-znw6 no longer exists
STEP: Deleting pod exec-volume-test-dynamicpv-znw6
Nov 11 02:55:06.385: INFO: Deleting pod "exec-volume-test-dynamicpv-znw6" in namespace "volume-4145"
... skipping 21 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should allow exec of files on the volume
    test/e2e/storage/testsuites/volumes.go:198
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should allow exec of files on the volume","total":41,"completed":15,"skipped":902,"failed":0}

SSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode 
  should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299

{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":38,"completed":13,"skipped":947,"failed":0}
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 11 02:55:36.147: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename volumemode
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299
Nov 11 02:55:36.900: INFO: Creating resource for dynamic PV
Nov 11 02:55:36.900: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass volumemode-9653-e2e-scdwfgx
STEP: creating a claim
Nov 11 02:55:37.118: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com86cmv] to have phase Bound
Nov 11 02:55:37.225: INFO: PersistentVolumeClaim test.csi.azure.com86cmv found but phase is Pending instead of Bound.
Nov 11 02:55:39.334: INFO: PersistentVolumeClaim test.csi.azure.com86cmv found but phase is Pending instead of Bound.
Nov 11 02:55:41.442: INFO: PersistentVolumeClaim test.csi.azure.com86cmv found and phase=Bound (4.32415817s)
STEP: Creating pod
STEP: Waiting for the pod to fail
Nov 11 02:55:44.092: INFO: Deleting pod "pod-6154b4b0-fdc9-4f5a-b3e8-dd37b3524c76" in namespace "volumemode-9653"
Nov 11 02:55:44.202: INFO: Wait up to 5m0s for pod "pod-6154b4b0-fdc9-4f5a-b3e8-dd37b3524c76" to be fully deleted
STEP: Deleting pvc
Nov 11 02:55:46.420: INFO: Deleting PersistentVolumeClaim "test.csi.azure.com86cmv"
Nov 11 02:55:46.529: INFO: Waiting up to 5m0s for PersistentVolume pvc-13c2eca2-4f19-45e4-9e72-66d5b8bc7261 to get deleted
Nov 11 02:55:46.640: INFO: PersistentVolume pvc-13c2eca2-4f19-45e4-9e72-66d5b8bc7261 found and phase=Released (110.799251ms)
... skipping 14 lines ...

• [SLOW TEST:51.899 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow]
    test/e2e/storage/testsuites/volumemode.go:299
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should fail to use a volume in a pod with mismatched mode [Slow]","total":38,"completed":14,"skipped":947,"failed":0}

S
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 11 02:56:28.050: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 3 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:258

    Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 8 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:269

    Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 80 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should support multiple inline ephemeral volumes
    test/e2e/storage/testsuites/ephemeral.go:254
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support multiple inline ephemeral volumes","total":41,"completed":16,"skipped":918,"failed":0}

SSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] 
  should concurrently access the single read-only volume from pods on the same node
  test/e2e/storage/testsuites/multivolume.go:423
... skipping 82 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single read-only volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:423
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":41,"completed":17,"skipped":923,"failed":0}

SSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy 
  (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
  test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
... skipping 91 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":41,"completed":18,"skipped":938,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] 
  should access to two volumes with the same volume mode and retain data across pod recreation on different node
  test/e2e/storage/testsuites/multivolume.go:168
... skipping 190 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:168
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":41,"completed":19,"skipped":978,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
Nov 11 03:02:43.306: INFO: Running AfterSuite actions on all nodes
Nov 11 03:02:43.306: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2
Nov 11 03:02:43.306: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2
... skipping 15 lines ...
Nov 11 03:02:43.339: INFO: Running AfterSuite actions on node 1



Summarizing 1 Failure:

[Fail] External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode [Measurement] should not mount / map unused volumes in a pod [LinuxOnly] 
test/e2e/storage/testsuites/volumemode.go:190

Ran 89 of 7227 Specs in 3341.042 seconds
FAIL! -- 88 Passed | 1 Failed | 0 Pending | 7138 Skipped 

Ginkgo ran 1 suite in 55m44.542035019s
Test Suite Failed
+ print_logs
+ sed -i s/disk.csi.azure.com/test.csi.azure.com/g deploy/example/storageclass-azuredisk-csi.yaml
+ bash ./hack/verify-examples.sh linux azurepubliccloud ephemeral test
begin to create deployment examples ...
storageclass.storage.k8s.io/managed-csi created
Applying config "deploy/example/deployment.yaml"
... skipping 102 lines ...
I1111 02:06:52.481911       1 reflector.go:257] Listing and watching *v1.Node from k8s.io/client-go/informers/factory.go:134
I1111 02:06:52.482015       1 reflector.go:221] Starting reflector *v1.CustomResourceDefinition (30s) from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117
I1111 02:06:52.482033       1 reflector.go:257] Listing and watching *v1.CustomResourceDefinition from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117
I1111 02:06:52.582045       1 shared_informer.go:285] caches populated
I1111 02:06:52.582088       1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 e2e-test
I1111 02:06:52.582120       1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider
I1111 02:06:52.584942       1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I1111 02:06:52.584962       1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider
I1111 02:06:52.584967       1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I1111 02:06:52.584996       1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully
I1111 02:06:52.585742       1 azure_auth.go:245] Using AzurePublicCloud environment
I1111 02:06:52.585765       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I1111 02:06:52.585769       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
... skipping 29 lines ...
I1111 02:06:52.586164       1 azure_vmasclient.go:73] Azure AvailabilitySetsClient  (write ops) using rate limit config: QPS=100, bucket=1000
I1111 02:06:52.586226       1 azure.go:1039] attach/detach disk operation rate limiter configuration - Enabled: true QPS: 1.333000, Bucket: 240
I1111 02:06:52.586272       1 azuredisk_v2.go:215] disable UseInstanceMetadata for controller
I1111 02:06:52.586286       1 azuredisk_v2.go:225] DisableAvailabilitySetNodes for controller since current VMType is vmss
I1111 02:06:52.586292       1 azuredisk_v2.go:231] cloud: AzurePublicCloud, location: westeurope, rg: kubetest-9juvoppw, VMType: vmss, PrimaryScaleSetName: k8s-agentpool1-30725374-vmss, PrimaryAvailabilitySetName: , DisableAvailabilitySetNodes: true
I1111 02:06:52.586298       1 skus.go:123] NewNodeInfo: Starting to populate node and disk sku information.
E1111 02:06:52.586313       1 azuredisk_v2.go:239] Failed to get node info. Error: NewNodeInfo: Failed to get instance type from Azure cloud provider, nodeName: , error: not a vmss instance
I1111 02:06:52.586418       1 mount_linux.go:208] Detected OS without systemd
I1111 02:06:52.586432       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I1111 02:06:52.586438       1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I1111 02:06:52.586443       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I1111 02:06:52.586448       1 driver.go:81] Enabling controller service capability: CLONE_VOLUME
I1111 02:06:52.586453       1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME
... skipping 273 lines ...
I1111 02:06:52.756016       1 shared_state.go:487]  "msg"="Storing pod cloud-node-manager-fqh5j and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb"
I1111 02:06:52.756032       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/cloud-node-manager-fqh5j." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-fqh5j"
I1111 02:06:52.756041       1 shared_state.go:311]  "msg"="Getting requested volumes for pod (kube-system/cloud-node-manager-fqh5j)." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-fqh5j"
I1111 02:06:52.756051       1 pod.go:99]  "msg"="Pod kube-system/cloud-node-manager-fqh5j has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-fqh5j"
I1111 02:06:52.756069       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-fqh5j" "latency"=36201
I1111 02:06:52.756081       1 shared_state.go:417]  "msg"="Adding pod csi-test-controller-78545b9dc-fqs6m to shared map with keyName kube-system/csi-test-controller-78545b9dc-fqs6m." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb"
I1111 02:06:52.757693       1 shared_state.go:423]  "msg"="Pod spec of pod csi-test-controller-78545b9dc-fqs6m is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-k892x VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-k892x ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-k892x ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-k892x ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-k892x ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-k892x ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 Command:[] Args:[--v=5 --config=/etc/csi-test-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-test-controller-config ReadOnly:false MountPath:/etc/csi-test-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-k892x ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000d6d190 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-agentpool1-30725374-vmss000001 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000d6d198} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000d6d1a0}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc000d6d1a8 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000d6d1ac PreemptionPolicy:0xc000946040 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil>}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-k892x VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb"
I1111 02:06:52.757994       1 shared_state.go:461]  "msg"="Pod csi-test-controller-78545b9dc-fqs6m: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb"
I1111 02:06:52.758046       1 shared_state.go:461]  "msg"="Pod csi-test-controller-78545b9dc-fqs6m: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb"
I1111 02:06:52.758082       1 shared_state.go:461]  "msg"="Pod csi-test-controller-78545b9dc-fqs6m: Skipping Volume {csi-test-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb"
I1111 02:06:52.758139       1 shared_state.go:461]  "msg"="Pod csi-test-controller-78545b9dc-fqs6m: Skipping Volume {kube-api-access-k892x {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb"
I1111 02:06:52.758161       1 shared_state.go:487]  "msg"="Storing pod csi-test-controller-78545b9dc-fqs6m and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb"
I1111 02:06:52.758183       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-test-controller-78545b9dc-fqs6m." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" "disk.csi/azure.com/pod-name"="kube-system/csi-test-controller-78545b9dc-fqs6m"
... skipping 140 lines ...
I1111 02:06:52.811498       1 shared_informer.go:285] caches populated
I1111 02:06:52.811515       1 controller.go:227] AzDiskControllerManager "msg"="Starting workers" "controller"="azvolumeattachment" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" "worker count"=10
I1111 02:06:52.811522       1 shared_informer.go:285] caches populated
I1111 02:06:52.811537       1 shared_informer.go:285] caches populated
I1111 02:06:52.811539       1 controller.go:227] AzDiskControllerManager "msg"="Starting workers" "controller"="azvolume" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" "worker count"=10
I1111 02:06:52.811545       1 shared_informer.go:285] caches populated
I1111 02:06:52.811620       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000000" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I1111 02:06:52.811644       1 shared_informer.go:285] caches populated
I1111 02:06:52.811662       1 controller.go:227] AzDiskControllerManager "msg"="Starting workers" "controller"="pod" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" "worker count"=10
I1111 02:06:52.811666       1 shared_state.go:1068]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="828b5ebb-6165-11ed-a60b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=8500
I1111 02:06:52.811674       1 shared_informer.go:285] caches populated
I1111 02:06:52.811697       1 controller.go:227] AzDiskControllerManager "msg"="Starting workers" "controller"="volumeattachment" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" "worker count"=10
I1111 02:06:52.811620       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I1111 02:06:52.811708       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I1111 02:06:52.811729       1 shared_informer.go:285] caches populated
I1111 02:06:52.811747       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="k8s-master-30725374-0" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I1111 02:06:52.811761       1 controller.go:227] AzDiskControllerManager "msg"="Starting workers" "controller"="replica" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" "worker count"=10
I1111 02:06:52.811759       1 shared_state.go:1068]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="828b61ef-6165-11ed-a60b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=11600
I1111 02:06:52.811781       1 shared_state.go:1068]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="828b635d-6165-11ed-a60b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=6800
I1111 02:06:52.811802       1 shared_state.go:417]  "msg"="Adding pod coredns-7cf78b68d8-zd9qd to shared map with keyName kube-system/coredns-7cf78b68d8-zd9qd." 
I1111 02:06:52.811813       1 shared_state.go:417]  "msg"="Adding pod csi-test-scheduler-extender-748844575-n8kpw to shared map with keyName kube-system/csi-test-scheduler-extender-748844575-n8kpw." 
I1111 02:06:52.811847       1 shared_state.go:417]  "msg"="Adding pod csi-test-node-lvncc to shared map with keyName kube-system/csi-test-node-lvncc." 
... skipping 34586 lines ...
I1111 02:06:47.667021       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=2556&timeout=9m52s&timeoutSeconds=592&watch=true 200 OK in 0 milliseconds
I1111 02:06:47.667839       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2332&timeout=5m24s&timeoutSeconds=324&watch=true 200 OK in 0 milliseconds
I1111 02:06:47.760082       1 shared_informer.go:285] caches populated
I1111 02:06:47.760132       1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 e2e-test
I1111 02:06:47.760157       1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider
I1111 02:06:47.763170       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I1111 02:06:47.763407       1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I1111 02:06:47.763424       1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider
I1111 02:06:47.763433       1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I1111 02:06:47.763453       1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully
I1111 02:06:47.764187       1 azure_auth.go:245] Using AzurePublicCloud environment
I1111 02:06:47.764207       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I1111 02:06:47.764213       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
... skipping 53 lines ...
I1111 02:06:47.888025       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes 201 Created in 4 milliseconds
I1111 02:06:47.892113       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 3 milliseconds
I1111 02:06:47.892271       1 azuredisk_v2.go:538]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7f9b50c0-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).registerAzDriverNode" "latency"=9262490
I1111 02:06:47.892363       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:06:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7f9cbd7c-6165-11ed-ae2b-6045bd88ebdb"
I1111 02:06:47.892507       1 server.go:117] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I1111 02:06:47.897506       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 409 Conflict in 5 milliseconds
E1111 02:06:47.897629       1 azuredisk_v2.go:597]  "msg"="Failed to update AzDriverNode status after creation" "error"="Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-agentpool1-30725374-vmss000001\": the object has been modified; please apply your changes to the latest version and try again" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7f9cbd7c-6165-11ed-ae2b-6045bd88ebdb"
E1111 02:06:47.897678       1 azuredisk_v2.go:549]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-agentpool1-30725374-vmss000001\": the object has been modified; please apply your changes to the latest version and try again]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7f9cbd7c-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5335152
I1111 02:06:47.897687       1 azuredisk_v2.go:512] Starting heartbeat loop with initial delay 29.777s and frequency 30s
I1111 02:06:48.316868       1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I1111 02:06:48.316891       1 utils.go:79] GRPC request: {}
I1111 02:06:48.319115       1 utils.go:85] GRPC response: {"name":"test.csi.azure.com","vendor_version":"latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0"}
I1111 02:06:48.782494       1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I1111 02:06:48.782517       1 utils.go:79] GRPC request: {}
... skipping 36 lines ...
I1111 02:07:37.131112       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:07:37.131132       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3af14f39-2800-4b53-9da0-715b511c77c6/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3af14f39-2800-4b53-9da0-715b511c77c6","csi.storage.k8s.io/pvc/name":"test.csi.azure.com7t57r","csi.storage.k8s.io/pvc/namespace":"snapshotting-4409","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-3af14f39-2800-4b53-9da0-715b511c77c6"}
I1111 02:07:37.131310       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-3af14f39-2800-4b53-9da0-715b511c77c6-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:07:47.595458       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:07:47.595591       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:07:47.595600       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:07:47.595866       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:07:47.595907       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:07:47.668259       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:07:47.675498       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:07:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a33eeaf2-6165-11ed-ae2b-6045bd88ebdb"
I1111 02:07:47.682507       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:07:47.682719       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a33eeaf2-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7279867
I1111 02:07:47.786685       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:07:59.199074       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:07:59.199100       1 utils.go:79] GRPC request: {}
I1111 02:07:59.199145       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:08:17.596366       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:17.596400       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:17.596384       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:17.596514       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:08:17.596550       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:08:17.668971       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:17.676239       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:08:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b520ac50-6165-11ed-ae2b-6045bd88ebdb"
I1111 02:08:17.685580       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 9 milliseconds
I1111 02:08:17.685720       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b520ac50-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9506393
I1111 02:08:17.786997       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:29.197595       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:08:29.197616       1 utils.go:79] GRPC request: {}
I1111 02:08:29.197659       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:08:47.597083       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:47.597181       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:47.597211       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:47.597246       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:08:47.597275       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:08:47.669150       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:47.676440       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:08:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c7025692-6165-11ed-ae2b-6045bd88ebdb"
I1111 02:08:47.681963       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:08:47.682177       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c7025692-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5766168
I1111 02:08:47.787921       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:59.197420       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:08:59.197443       1 utils.go:79] GRPC request: {}
I1111 02:08:59.197488       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:09:17.597963       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:17.598078       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:09:17.598096       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:17.598120       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:17.598110       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:09:17.669470       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:17.675749       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:09:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d8e3deef-6165-11ed-ae2b-6045bd88ebdb"
I1111 02:09:17.683035       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 7 milliseconds
I1111 02:09:17.683212       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d8e3deef-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7493859
I1111 02:09:17.788274       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:29.197530       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:09:29.197552       1 utils.go:79] GRPC request: {}
I1111 02:09:29.197588       1 utils.go:85] GRPC response: {"ready":{"value":true}}
E1111 02:09:36.725816       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9cb8bd86-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e10ddcaf-7585-41ea-b3e6-4403705103b6" "latency"=119995831254
E1111 02:09:36.725890       1 crdprovisioner.go:760]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9cb8bd86-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119996109556
E1111 02:09:36.726080       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-e10ddcaf-7585-41ea-b3e6-4403705103b6) to be attached to node (k8s-agentpool1-30725374-vmss000001): context canceled
E1111 02:09:37.126712       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9cf6029b-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3af14f39-2800-4b53-9da0-715b511c77c6" "latency"=119995297584
E1111 02:09:37.126781       1 crdprovisioner.go:760]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9cf6029b-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119995482285
E1111 02:09:37.126803       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-3af14f39-2800-4b53-9da0-715b511c77c6) to be attached to node (k8s-agentpool1-30725374-vmss000001): context canceled
I1111 02:09:37.285429       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:09:37.285449       1 utils.go:79] GRPC request: {}
I1111 02:09:37.285483       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:09:37.288612       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:09:37.288633       1 utils.go:79] GRPC request: {}
I1111 02:09:37.288664       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 15 lines ...
I1111 02:09:37.692077       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:09:37.692091       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3af14f39-2800-4b53-9da0-715b511c77c6/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3af14f39-2800-4b53-9da0-715b511c77c6","csi.storage.k8s.io/pvc/name":"test.csi.azure.com7t57r","csi.storage.k8s.io/pvc/namespace":"snapshotting-4409","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-3af14f39-2800-4b53-9da0-715b511c77c6"}
I1111 02:09:37.692262       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-3af14f39-2800-4b53-9da0-715b511c77c6-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:09:47.598982       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:47.599042       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:47.599093       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:47.599181       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:09:47.599202       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:09:47.670520       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:47.675809       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:09:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="eac583a6-6165-11ed-ae2b-6045bd88ebdb"
I1111 02:09:47.681810       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:09:47.682312       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="eac583a6-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6500957
I1111 02:09:47.789186       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:59.197387       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:09:59.197409       1 utils.go:79] GRPC request: {}
I1111 02:09:59.197447       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:10:17.599352       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:17.599374       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:17.599399       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:17.599459       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:10:17.599569       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:10:17.671060       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:17.676330       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:10:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="fca73b9d-6165-11ed-ae2b-6045bd88ebdb"
I1111 02:10:17.683059       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:10:17.683198       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="fca73b9d-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6896860
I1111 02:10:17.789292       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:29.197744       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:10:29.197769       1 utils.go:79] GRPC request: {}
I1111 02:10:29.197805       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:10:47.600202       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:47.600225       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:47.600258       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:47.600367       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:10:47.600387       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:10:47.671838       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:47.676131       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:10:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="0e88d62b-6166-11ed-ae2b-6045bd88ebdb"
I1111 02:10:47.681681       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:10:47.682506       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="0e88d62b-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6410856
I1111 02:10:47.789716       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:59.197178       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:10:59.197211       1 utils.go:79] GRPC request: {}
I1111 02:10:59.197252       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:11:17.601231       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:11:17.601312       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:11:17.601391       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:11:17.601418       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:11:17.601432       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:11:17.672843       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:11:17.676121       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:11:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="206a798e-6166-11ed-ae2b-6045bd88ebdb"
I1111 02:11:17.681295       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:11:17.681851       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="206a798e-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5758349
I1111 02:11:17.790821       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:11:29.197129       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:11:29.197151       1 utils.go:79] GRPC request: {}
I1111 02:11:29.197188       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:11:32.813354       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:11:32.813404       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="e4d226d8-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3af14f39-2800-4b53-9da0-715b511c77c6" "latency"=115121091923
I1111 02:11:32.813426       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e4d226d8-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=115121182623
I1111 02:11:32.815896       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:11:32.815940       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="e494d466-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e10ddcaf-7585-41ea-b3e6-4403705103b6" "latency"=115525504766
I1111 02:11:32.815963       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e494d466-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=115525598166
I1111 02:11:33.403235       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/
I1111 02:11:33.403251       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/
I1111 02:11:33.403281       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:11:33.403282       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
... skipping 181 lines ...
I1111 02:12:10.089312       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-3af14f39-2800-4b53-9da0-715b511c77c6-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:12:11.668738       1 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: Watch close - *v1.CustomResourceDefinition total 0 items received
I1111 02:12:11.672407       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2332&timeout=9m38s&timeoutSeconds=578&watch=true 200 OK in 3 milliseconds
I1111 02:12:17.602594       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:12:17.602631       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:12:17.602683       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:12:17.602722       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:12:17.674168       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:12:17.676391       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:12:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="442dc998-6166-11ed-ae2b-6045bd88ebdb"
I1111 02:12:17.682156       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:12:17.682375       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="442dc998-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6022248
I1111 02:12:17.792194       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:12:29.197398       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:12:29.197420       1 utils.go:79] GRPC request: {}
I1111 02:12:29.197458       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:12:39.596252       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 55 items received
I1111 02:12:39.598858       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=4577&timeout=8m39s&timeoutSeconds=519&watch=true 200 OK in 2 milliseconds
I1111 02:12:39.922853       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:12:39.922915       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="3fa8188f-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3af14f39-2800-4b53-9da0-715b511c77c6" "latency"=29833545015
I1111 02:12:39.922946       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3fa8188f-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29833645915
I1111 02:12:40.545135       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1111 02:12:40.545191       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:12:40.545223       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3af14f39-2800-4b53-9da0-715b511c77c6/globalmount with mount options([])
I1111 02:12:40.545236       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 55 lines ...
W1111 02:12:43.439835       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3af14f39-2800-4b53-9da0-715b511c77c6/globalmount" is not a mountpoint, deleting
I1111 02:12:43.439894       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3af14f39-2800-4b53-9da0-715b511c77c6/globalmount successfully
I1111 02:12:43.439903       1 utils.go:85] GRPC response: {}
I1111 02:12:47.603400       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:12:47.603487       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:12:47.603485       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:12:47.603581       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:12:47.675033       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:12:47.676196       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:12:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="560f64e3-6166-11ed-ae2b-6045bd88ebdb"
I1111 02:12:47.683516       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 7 milliseconds
I1111 02:12:47.683753       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="560f64e3-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7595360
I1111 02:12:47.792873       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:12:59.197996       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:12:59.198015       1 utils.go:79] GRPC request: {}
I1111 02:12:59.198052       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:13:17.604191       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:13:17.604285       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:13:17.604314       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:13:17.604391       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:13:17.675937       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:13:17.676059       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:13:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="67f10289-6166-11ed-ae2b-6045bd88ebdb"
I1111 02:13:17.694432       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 18 milliseconds
I1111 02:13:17.694644       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="67f10289-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18590845
I1111 02:13:17.793281       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:13:29.198146       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:13:29.198173       1 utils.go:79] GRPC request: {}
I1111 02:13:29.198219       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:13:30.983498       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:13:30.983571       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="52c09510-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1ee8f9fc-8188-4d77-ba26-dfc4169f7582" "latency"=48856998667
I1111 02:13:30.983601       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="52c09510-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=48857119468
I1111 02:13:31.499712       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:13:31.499732       1 utils.go:79] GRPC request: {}
I1111 02:13:31.499767       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:13:31.502614       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 65 lines ...
I1111 02:13:43.988863       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1ee8f9fc-8188-4d77-ba26-dfc4169f7582/globalmount
I1111 02:13:43.988890       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1ee8f9fc-8188-4d77-ba26-dfc4169f7582/globalmount" is a mountpoint, unmounting
I1111 02:13:43.988903       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1ee8f9fc-8188-4d77-ba26-dfc4169f7582/globalmount
W1111 02:13:44.022875       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1ee8f9fc-8188-4d77-ba26-dfc4169f7582/globalmount" is not a mountpoint, deleting
I1111 02:13:44.022953       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1ee8f9fc-8188-4d77-ba26-dfc4169f7582/globalmount successfully
I1111 02:13:44.022965       1 utils.go:85] GRPC response: {}
I1111 02:13:46.447704       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:13:46.447769       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="702f0db2-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6f481851-34b4-4af1-9f4f-03d165ba9b82" "latency"=14943298593
I1111 02:13:46.447808       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="702f0db2-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14943414495
I1111 02:13:46.451876       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:13:46.451944       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="71810849-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-008fe7f7-5797-41ca-addb-b2e8a842b067" "latency"=12732488065
I1111 02:13:46.451974       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="71810849-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12732607166
I1111 02:13:47.065882       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/
I1111 02:13:47.065932       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/
I1111 02:13:47.065969       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1111 02:13:47.066018       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6f481851-34b4-4af1-9f4f-03d165ba9b82/globalmount with mount options([])
... skipping 204 lines ...
I1111 02:14:59.197218       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:14:59.197244       1 utils.go:79] GRPC request: {}
I1111 02:14:59.197285       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:15:17.607023       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:15:17.607059       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:15:17.607151       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:15:17.607215       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:15:17.675481       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:15:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="af77784a-6166-11ed-ae2b-6045bd88ebdb"
I1111 02:15:17.678805       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:15:17.681218       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:15:17.681421       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="af77784a-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5970848
I1111 02:15:17.795721       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:15:29.197507       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:15:29.197530       1 utils.go:79] GRPC request: {}
I1111 02:15:29.197571       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:15:47.607906       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:15:47.608061       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:15:47.608084       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:15:47.608161       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:15:47.675518       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:15:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c1591cf7-6166-11ed-ae2b-6045bd88ebdb"
I1111 02:15:47.679215       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:15:47.681296       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:15:47.681937       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c1591cf7-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6447950
I1111 02:15:47.796843       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:15:59.197076       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:15:59.197109       1 utils.go:79] GRPC request: {}
I1111 02:15:59.197176       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:16:01.397375       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:16:01.397436       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a1b50433-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-644fc064-7724-4609-ade0-52427c92efcb" "latency"=66806668987
I1111 02:16:01.397460       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a1b50433-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=66806773689
I1111 02:16:03.110174       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1111 02:16:03.110214       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:16:03.110241       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-644fc064-7724-4609-ade0-52427c92efcb/globalmount with mount options([])
I1111 02:16:03.110249       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 58 lines ...
I1111 02:16:09.427493       1 utils.go:85] GRPC response: {}
I1111 02:16:12.596579       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 210 items received
I1111 02:16:12.599499       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=6769&timeout=9m50s&timeoutSeconds=590&watch=true 200 OK in 2 milliseconds
I1111 02:16:17.608547       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:16:17.608628       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:16:17.608537       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:16:17.608841       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:16:17.676399       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:16:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d33ae121-6166-11ed-ae2b-6045bd88ebdb"
I1111 02:16:17.679419       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:16:17.681191       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds
I1111 02:16:17.681413       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d33ae121-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5075940
I1111 02:16:17.797558       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:16:29.198086       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:16:29.198115       1 utils.go:79] GRPC request: {}
I1111 02:16:29.198163       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:16:32.014304       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:16:32.014361       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="cbc8c5a1-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-822db309-9767-4898-b4da-11a98999ff89" "latency"=26829800152
I1111 02:16:32.014381       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cbc8c5a1-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=26829919253
I1111 02:16:32.380456       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:16:32.380477       1 utils.go:79] GRPC request: {}
I1111 02:16:32.380512       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:16:32.381385       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 55 lines ...
I1111 02:16:38.748063       1 utils.go:85] GRPC response: {}
I1111 02:16:39.668167       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 102 items received
I1111 02:16:39.670702       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=7036&timeout=8m9s&timeoutSeconds=489&watch=true 200 OK in 2 milliseconds
I1111 02:16:47.609736       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:16:47.609817       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:16:47.609831       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:16:47.609897       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:16:47.676265       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:16:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="e51c7f79-6166-11ed-ae2b-6045bd88ebdb"
I1111 02:16:47.680172       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:16:47.681699       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:16:47.681912       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="e51c7f79-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5687244
I1111 02:16:47.798146       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:16:59.197386       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:16:59.197410       1 utils.go:79] GRPC request: {}
I1111 02:16:59.197451       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:17:17.610017       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:17:17.610049       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:17:17.610088       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:17:17.610153       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:17:17.676484       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:17:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f6fe2ace-6166-11ed-ae2b-6045bd88ebdb"
I1111 02:17:17.681042       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:17:17.681584       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds
I1111 02:17:17.681757       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f6fe2ace-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5327042
I1111 02:17:17.798345       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:17:29.197602       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:17:29.197622       1 utils.go:79] GRPC request: {}
I1111 02:17:29.197656       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:17:47.610160       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:17:47.610262       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:17:47.610280       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:17:47.610297       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:17:47.675615       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:17:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="08dfab7a-6167-11ed-ae2b-6045bd88ebdb"
I1111 02:17:47.681903       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:17:47.682078       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:17:47.682285       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="08dfab7a-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6730852
I1111 02:17:47.799416       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:17:58.664418       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:17:58.664492       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="dbfedce3-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-955327db-3ec7-467b-8608-a885eb36800a" "latency"=86281914489
I1111 02:17:58.664516       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="dbfedce3-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=86282017389
I1111 02:17:59.174060       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:17:59.174089       1 utils.go:79] GRPC request: {}
I1111 02:17:59.174137       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:17:59.177997       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 42 lines ...
I1111 02:18:07.402929       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1111 02:18:07.402944       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-955327db-3ec7-467b-8608-a885eb36800a","volume_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-955327db-3ec7-467b-8608-a885eb36800a/dev/4534e6eb-4029-4bc9-bdd7-9381e077f6a1"}
I1111 02:18:07.403631       1 utils.go:85] GRPC response: {"usage":[{"total":5368709120,"unit":1}]}
I1111 02:18:17.610661       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:18:17.610674       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:18:17.610684       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:18:17.610889       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:18:17.610922       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:18:17.675962       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:18:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1ac15ca5-6167-11ed-ae2b-6045bd88ebdb"
I1111 02:18:17.682811       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:18:17.682834       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:18:17.684585       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1ac15ca5-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8661568
I1111 02:18:17.799668       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:18:24.144708       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:18:24.144766       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="0fbb34cf-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8b06a43e-3ded-4b4c-8798-35687a3c310e" "latency"=24964068568
I1111 02:18:24.144795       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0fbb34cf-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24964161368
I1111 02:18:24.148508       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:18:24.148568       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1026d4f1-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-adf86ba4-0150-4820-bc17-b65dadcd974f" "latency"=24262529739
I1111 02:18:24.148596       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1026d4f1-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24262634041
I1111 02:18:24.739652       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1111 02:18:24.739672       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/
I1111 02:18:24.739700       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:18:24.739700       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
... skipping 126 lines ...
W1111 02:18:37.972033       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-955327db-3ec7-467b-8608-a885eb36800a" is not a mountpoint, deleting
I1111 02:18:37.972112       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-955327db-3ec7-467b-8608-a885eb36800a successfully
I1111 02:18:37.972126       1 utils.go:85] GRPC response: {}
I1111 02:18:47.611234       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:18:47.611277       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:18:47.611334       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:18:47.611398       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:18:47.675729       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:18:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="2ca2f63d-6167-11ed-ae2b-6045bd88ebdb"
I1111 02:18:47.681983       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:18:47.682222       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="2ca2f63d-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6535851
I1111 02:18:47.683356       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:18:47.800684       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:18:59.198001       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:18:59.198029       1 utils.go:79] GRPC request: {}
I1111 02:18:59.198078       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:19:00.193051       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:19:00.193120       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="21d155b0-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3a8f8efe-d680-482a-bd4a-8b620166b0c6" "latency"=30668404491
I1111 02:19:00.193149       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="21d155b0-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=30668507992
I1111 02:19:00.790570       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/
I1111 02:19:00.790623       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1111 02:19:00.790652       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3a8f8efe-d680-482a-bd4a-8b620166b0c6/globalmount with mount options([])
I1111 02:19:00.790663       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3])
... skipping 66 lines ...
I1111 02:19:20.825920       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-48a25eef-bd2b-459f-911a-136e5183b043-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:19:29.197121       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:19:29.197144       1 utils.go:79] GRPC request: {}
I1111 02:19:29.197179       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:19:47.612491       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:19:47.612506       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:19:47.612610       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:19:47.612518       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:19:47.675994       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:19:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5066478c-6167-11ed-ae2b-6045bd88ebdb"
I1111 02:19:47.682049       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:19:47.682714       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5066478c-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6759137
I1111 02:19:47.684232       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:19:47.801550       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:19:59.197957       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:19:59.197987       1 utils.go:79] GRPC request: {}
I1111 02:19:59.198030       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:20:01.453183       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:20:01.453259       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="40654957-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-48a25eef-bd2b-459f-911a-136e5183b043" "latency"=40627263447
I1111 02:20:01.453304       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="40654957-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=40627388649
I1111 02:20:01.921933       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:20:01.921961       1 utils.go:79] GRPC request: {}
I1111 02:20:01.922008       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:20:01.926380       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 42 lines ...
I1111 02:20:03.491237       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-48a25eef-bd2b-459f-911a-136e5183b043/globalmount","target_path":"/var/lib/kubelet/pods/4ebf2513-33d4-4c65-bbc6-f84c21b4b53b/volumes/kubernetes.io~csi/pvc-48a25eef-bd2b-459f-911a-136e5183b043/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-48a25eef-bd2b-459f-911a-136e5183b043","csi.storage.k8s.io/pvc/name":"test.csi.azure.comfptp8","csi.storage.k8s.io/pvc/namespace":"volume-9791","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-48a25eef-bd2b-459f-911a-136e5183b043"}
I1111 02:20:03.491392       1 nodeserver_v2.go:348] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-48a25eef-bd2b-459f-911a-136e5183b043/globalmount at /var/lib/kubelet/pods/4ebf2513-33d4-4c65-bbc6-f84c21b4b53b/volumes/kubernetes.io~csi/pvc-48a25eef-bd2b-459f-911a-136e5183b043/mount
I1111 02:20:03.491406       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-48a25eef-bd2b-459f-911a-136e5183b043/globalmount /var/lib/kubelet/pods/4ebf2513-33d4-4c65-bbc6-f84c21b4b53b/volumes/kubernetes.io~csi/pvc-48a25eef-bd2b-459f-911a-136e5183b043/mount)
I1111 02:20:03.492255       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-48a25eef-bd2b-459f-911a-136e5183b043/globalmount /var/lib/kubelet/pods/4ebf2513-33d4-4c65-bbc6-f84c21b4b53b/volumes/kubernetes.io~csi/pvc-48a25eef-bd2b-459f-911a-136e5183b043/mount)
I1111 02:20:03.493157       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-48a25eef-bd2b-459f-911a-136e5183b043/globalmount at /var/lib/kubelet/pods/4ebf2513-33d4-4c65-bbc6-f84c21b4b53b/volumes/kubernetes.io~csi/pvc-48a25eef-bd2b-459f-911a-136e5183b043/mount successfully
I1111 02:20:03.493180       1 utils.go:85] GRPC response: {}
I1111 02:20:11.719580       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:20:11.719634       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="58e5281e-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-50522e90-872b-46df-8acf-7267e686b23e" "latency"=9790326818
I1111 02:20:11.719665       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="58e5281e-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9790428619
I1111 02:20:11.720962       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:20:11.721036       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5902fbb9-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-874c6d1e-232f-4c47-b057-d06057d49f52" "latency"=9596239326
I1111 02:20:11.721063       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5902fbb9-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9596361827
I1111 02:20:12.314211       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/
I1111 02:20:12.314327       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/
I1111 02:20:12.314373       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1111 02:20:12.314423       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-50522e90-872b-46df-8acf-7267e686b23e/globalmount with mount options([])
... skipping 161 lines ...
I1111 02:20:37.063948       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1111 02:20:37.063963       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-874c6d1e-232f-4c47-b057-d06057d49f52","volume_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-874c6d1e-232f-4c47-b057-d06057d49f52/dev/20f9a18f-690a-4809-9181-abc75739c042"}
I1111 02:20:37.065262       1 utils.go:85] GRPC response: {"usage":[{"total":5368709120,"unit":1}]}
I1111 02:20:47.613997       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:20:47.614063       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:20:47.614071       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:20:47.614181       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:20:47.614227       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:20:47.675613       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:20:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="74297b89-6167-11ed-ae2b-6045bd88ebdb"
I1111 02:20:47.682098       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:20:47.682707       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="74297b89-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7203857
I1111 02:20:47.684591       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:20:47.802860       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:20:59.198891       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:20:59.198918       1 utils.go:79] GRPC request: {}
I1111 02:20:59.198968       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:21:17.614708       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:21:17.614750       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:21:17.614717       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:21:17.614828       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:21:17.614844       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:21:17.676073       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:21:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="860b32e4-6167-11ed-ae2b-6045bd88ebdb"
I1111 02:21:17.681621       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:21:17.681871       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="860b32e4-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5833344
I1111 02:21:17.684908       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:21:17.803228       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:21:18.599155       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 80 items received
I1111 02:21:18.602927       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=9739&timeout=6m1s&timeoutSeconds=361&watch=true 200 OK in 3 milliseconds
I1111 02:21:29.197364       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:21:29.197391       1 utils.go:79] GRPC request: {}
I1111 02:21:29.197439       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:21:29.499738       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:21:29.499786       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="6a9c78a1-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b61cfee1-08a3-45b0-a959-f7416124b164" "latency"=57847858537
I1111 02:21:29.499806       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6a9c78a1-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=57847940239
I1111 02:21:29.500246       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:21:29.500323       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="6a9c754f-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d4da29a0-a8d5-47e6-a4bb-af8d60bcc7e6" "latency"=57848428742
I1111 02:21:29.500355       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6a9c754f-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=57848568543
I1111 02:21:30.107279       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdc under /dev/disk/azure/scsi1/
I1111 02:21:30.107282       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdd under /dev/disk/azure/scsi1/
I1111 02:21:30.107332       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType 
I1111 02:21:30.107354       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
... skipping 75 lines ...
I1111 02:21:30.607536       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"4"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/globalmount","target_path":"/var/lib/kubelet/pods/c48844af-3f59-4185-a183-9434c8cbf6df/volumes/kubernetes.io~csi/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b61cfee1-08a3-45b0-a959-f7416124b164","csi.storage.k8s.io/pvc/name":"test.csi.azure.comw27rm","csi.storage.k8s.io/pvc/namespace":"multivolume-6915","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-b61cfee1-08a3-45b0-a959-f7416124b164"}
I1111 02:21:30.607711       1 nodeserver_v2.go:348] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/globalmount at /var/lib/kubelet/pods/c48844af-3f59-4185-a183-9434c8cbf6df/volumes/kubernetes.io~csi/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/mount
I1111 02:21:30.607728       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/globalmount /var/lib/kubelet/pods/c48844af-3f59-4185-a183-9434c8cbf6df/volumes/kubernetes.io~csi/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/mount)
I1111 02:21:30.608486       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/globalmount /var/lib/kubelet/pods/c48844af-3f59-4185-a183-9434c8cbf6df/volumes/kubernetes.io~csi/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/mount)
I1111 02:21:30.609300       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/globalmount at /var/lib/kubelet/pods/c48844af-3f59-4185-a183-9434c8cbf6df/volumes/kubernetes.io~csi/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/mount successfully
I1111 02:21:30.609316       1 utils.go:85] GRPC response: {}
I1111 02:21:39.847584       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:21:39.847653       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="8d976b8c-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-df212b9f-8dc2-4e69-92ad-d118022abf03" "latency"=9508560846
I1111 02:21:39.847683       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8d976b8c-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9508657348
I1111 02:21:39.849920       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:21:39.849994       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="8d9734b6-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7fece55f-2db4-48a2-acbc-d5a038b3fb09" "latency"=9512283975
I1111 02:21:39.850031       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8d9734b6-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9512408976
I1111 02:21:40.476973       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdg under /dev/disk/azure/scsi1/
I1111 02:21:40.477051       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1111 02:21:40.477066       1 utils.go:85] GRPC response: {}
I1111 02:21:40.477185       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdf under /dev/disk/azure/scsi1/
... skipping 278 lines ...
I1111 02:22:13.147633       1 utils.go:85] GRPC response: {}
W1111 02:22:13.153641       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d4da29a0-a8d5-47e6-a4bb-af8d60bcc7e6/globalmount" is not a mountpoint, deleting
I1111 02:22:13.153699       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d4da29a0-a8d5-47e6-a4bb-af8d60bcc7e6/globalmount successfully
I1111 02:22:13.153713       1 utils.go:85] GRPC response: {}
I1111 02:22:17.615810       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:22:17.615930       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:22:17.615971       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:22:17.616004       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:22:17.676353       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:22:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a9ce833b-6167-11ed-ae2b-6045bd88ebdb"
I1111 02:22:17.683818       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 7 milliseconds
I1111 02:22:17.684035       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a9ce833b-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7739167
I1111 02:22:17.685990       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:22:17.804505       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:22:22.790270       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 25 items received
I1111 02:22:22.791495       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-30725374-vmss000001&resourceVersion=10328&timeout=7m6s&timeoutSeconds=426&watch=true 200 OK in 1 milliseconds
I1111 02:22:29.198073       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:22:29.198099       1 utils.go:79] GRPC request: {}
I1111 02:22:29.198142       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:22:35.278352       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:22:35.278425       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a18354b2-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0a32ef96-6d0c-415d-9fff-f5f376d27c96" "latency"=31516533271
I1111 02:22:35.278453       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a18354b2-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31516654372
I1111 02:22:35.867803       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun5 by sde under /dev/disk/azure/scsi1/
I1111 02:22:35.867860       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun5. perfProfile none accountType 
I1111 02:22:35.867890       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun5 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0a32ef96-6d0c-415d-9fff-f5f376d27c96/globalmount with mount options([])
I1111 02:22:35.867904       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun5" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun5])
... skipping 54 lines ...
I1111 02:22:44.358492       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:22:44.358509       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4d5cd466-8cab-4c69-beee-d69091dce53a/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4d5cd466-8cab-4c69-beee-d69091dce53a","csi.storage.k8s.io/pvc/name":"test.csi.azure.combqk8w","csi.storage.k8s.io/pvc/namespace":"multivolume-5806","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-4d5cd466-8cab-4c69-beee-d69091dce53a"}
I1111 02:22:44.358643       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-4d5cd466-8cab-4c69-beee-d69091dce53a-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:22:47.616361       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:22:47.616448       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:22:47.616596       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:22:47.616739       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:22:47.616811       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:22:47.676029       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:22:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="bbb01a84-6167-11ed-ae2b-6045bd88ebdb"
I1111 02:22:47.681174       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds
I1111 02:22:47.681728       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="bbb01a84-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5729845
I1111 02:22:47.687101       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:22:47.805505       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:22:48.882491       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
... skipping 15 lines ...
W1111 02:22:49.002679       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0a32ef96-6d0c-415d-9fff-f5f376d27c96/globalmount" is not a mountpoint, deleting
I1111 02:22:49.002769       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0a32ef96-6d0c-415d-9fff-f5f376d27c96/globalmount successfully
I1111 02:22:49.002789       1 utils.go:85] GRPC response: {}
I1111 02:22:59.197695       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:22:59.197717       1 utils.go:79] GRPC request: {}
I1111 02:22:59.197761       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:23:16.098127       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:23:16.098180       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b9b5ea77-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4d5cd466-8cab-4c69-beee-d69091dce53a" "latency"=31739488132
I1111 02:23:16.098205       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b9b5ea77-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31739574532
I1111 02:23:16.112044       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:23:16.112099       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b9b5c7f4-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-03509e51-f46f-4dc7-b1e7-06600d90c1d9" "latency"=31754249348
I1111 02:23:16.112127       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b9b5c7f4-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31754378950
I1111 02:23:16.695833       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1111 02:23:16.695901       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:23:16.695944       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-03509e51-f46f-4dc7-b1e7-06600d90c1d9/globalmount with mount options([nouuid])
I1111 02:23:16.695971       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 211 lines ...
I1111 02:23:35.890399       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-03509e51-f46f-4dc7-b1e7-06600d90c1d9/globalmount /var/lib/kubelet/pods/b5abdb58-4e93-42fe-b59f-edc67b96cab2/volumes/kubernetes.io~csi/pvc-03509e51-f46f-4dc7-b1e7-06600d90c1d9/mount)
I1111 02:23:35.890622       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4d5cd466-8cab-4c69-beee-d69091dce53a/globalmount /var/lib/kubelet/pods/b5abdb58-4e93-42fe-b59f-edc67b96cab2/volumes/kubernetes.io~csi/pvc-4d5cd466-8cab-4c69-beee-d69091dce53a/mount)
I1111 02:23:35.891267       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-03509e51-f46f-4dc7-b1e7-06600d90c1d9/globalmount at /var/lib/kubelet/pods/b5abdb58-4e93-42fe-b59f-edc67b96cab2/volumes/kubernetes.io~csi/pvc-03509e51-f46f-4dc7-b1e7-06600d90c1d9/mount successfully
I1111 02:23:35.891294       1 utils.go:85] GRPC response: {}
I1111 02:23:35.891450       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4d5cd466-8cab-4c69-beee-d69091dce53a/globalmount at /var/lib/kubelet/pods/b5abdb58-4e93-42fe-b59f-edc67b96cab2/volumes/kubernetes.io~csi/pvc-4d5cd466-8cab-4c69-beee-d69091dce53a/mount successfully
I1111 02:23:35.891470       1 utils.go:85] GRPC response: {}
I1111 02:23:36.476020       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:23:36.476081       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="cda59559-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-77bbd751-4bbe-4ed0-9bf3-4596c75901a5" "latency"=18669982492
I1111 02:23:36.476112       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cda59559-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=18670082392
I1111 02:23:36.491758       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:23:36.491832       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="cd9653e7-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-41381644-1141-4bd5-8764-c59bc24c27f1" "latency"=18785696563
I1111 02:23:36.491877       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cd9653e7-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=18785826563
I1111 02:23:37.064830       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:23:37.064850       1 utils.go:79] GRPC request: {}
I1111 02:23:37.064890       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:23:37.065792       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
... skipping 78 lines ...
I1111 02:23:45.306261       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:23:45.307147       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:23:45.307167       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"4"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2f81cf66-89b0-419b-b85d-493960e6afba/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2f81cf66-89b0-419b-b85d-493960e6afba","csi.storage.k8s.io/pvc/name":"test.csi.azure.comqd76d","csi.storage.k8s.io/pvc/namespace":"provisioning-9310","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-2f81cf66-89b0-419b-b85d-493960e6afba"}
I1111 02:23:45.307361       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-2f81cf66-89b0-419b-b85d-493960e6afba-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:23:47.618559       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:23:47.618576       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:23:47.618670       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:23:47.618717       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:23:47.676022       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:23:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="df735eea-6167-11ed-ae2b-6045bd88ebdb"
I1111 02:23:47.682330       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:23:47.682568       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="df735eea-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6620868
I1111 02:23:47.688842       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:23:47.806147       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 75 lines ...
I1111 02:23:53.879004       1 utils.go:85] GRPC response: {}
I1111 02:23:57.597776       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 195 items received
I1111 02:23:57.601152       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=11183&timeout=8m27s&timeoutSeconds=507&watch=true 200 OK in 3 milliseconds
I1111 02:23:59.197039       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:23:59.197059       1 utils.go:79] GRPC request: {}
I1111 02:23:59.197104       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:24:07.164877       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:24:07.164945       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="de09f30e-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2f81cf66-89b0-419b-b85d-493960e6afba" "latency"=21857505339
I1111 02:24:07.164973       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="de09f30e-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=21857642741
I1111 02:24:07.557659       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:24:07.557682       1 utils.go:79] GRPC request: {}
I1111 02:24:07.557727       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:24:07.560728       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 56 lines ...
W1111 02:24:14.628877       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2f81cf66-89b0-419b-b85d-493960e6afba/globalmount" is not a mountpoint, deleting
I1111 02:24:14.628955       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2f81cf66-89b0-419b-b85d-493960e6afba/globalmount successfully
I1111 02:24:14.628983       1 utils.go:85] GRPC response: {}
I1111 02:24:17.619375       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:24:17.619585       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:24:17.619649       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:24:17.619836       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:24:17.675958       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:24:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f15500aa-6167-11ed-ae2b-6045bd88ebdb"
I1111 02:24:17.684081       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 7 milliseconds
I1111 02:24:17.684330       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f15500aa-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8408067
I1111 02:24:17.689544       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:24:17.806922       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:24:29.198181       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:24:29.198201       1 utils.go:79] GRPC request: {}
I1111 02:24:29.198247       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:24:47.620093       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:24:47.620187       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:24:47.620197       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:24:47.620281       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:24:47.675530       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:24:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="0336928c-6168-11ed-ae2b-6045bd88ebdb"
I1111 02:24:47.682057       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:24:47.682338       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="0336928c-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6833360
I1111 02:24:47.690547       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:24:47.807946       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:24:48.672071       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 127 items received
I1111 02:24:48.673983       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=11626&timeout=7m43s&timeoutSeconds=463&watch=true 200 OK in 1 milliseconds
I1111 02:24:59.197932       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:24:59.197954       1 utils.go:79] GRPC request: {}
I1111 02:24:59.197998       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:25:08.433254       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:25:08.433322       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="eb4dd336-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a72d9a1e-5acf-466a-98ee-4d8302dca0fe" "latency"=60870683747
I1111 02:25:08.433360       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="eb4dd336-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=60870817948
I1111 02:25:10.276048       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun5 by sdc under /dev/disk/azure/scsi1/
I1111 02:25:10.276098       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun5. perfProfile none accountType 
I1111 02:25:10.276134       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun5 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a72d9a1e-5acf-466a-98ee-4d8302dca0fe/globalmount with mount options([])
I1111 02:25:10.276153       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun5" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun5])
... skipping 91 lines ...
I1111 02:25:29.197773       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:25:29.197792       1 utils.go:79] GRPC request: {}
I1111 02:25:29.197834       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:25:47.621625       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:25:47.621649       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:25:47.621767       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:25:47.621795       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:25:47.621818       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:25:47.621848       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:25:47.621863       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:25:47.676116       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:25:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="26f9ef70-6168-11ed-ae2b-6045bd88ebdb"
I1111 02:25:47.682279       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:25:47.683408       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="26f9ef70-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7343864
I1111 02:25:47.691654       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:25:47.809076       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:25:59.197661       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:25:59.197691       1 utils.go:79] GRPC request: {}
I1111 02:25:59.197738       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:26:02.600491       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 367 items received
I1111 02:26:02.607035       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=12534&timeout=5m28s&timeoutSeconds=328&watch=true 200 OK in 6 milliseconds
I1111 02:26:17.622698       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:17.622795       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:17.622826       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:17.622924       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:26:17.622956       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:26:17.622975       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:26:17.623008       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:26:17.676209       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:26:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="38db9657-6168-11ed-ae2b-6045bd88ebdb"
I1111 02:26:17.683540       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 7 milliseconds
I1111 02:26:17.683797       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="38db9657-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7622563
I1111 02:26:17.692043       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:17.809477       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:29.197613       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:26:29.197638       1 utils.go:79] GRPC request: {}
I1111 02:26:29.197685       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:26:47.623187       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:47.623177       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:47.623318       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:47.623401       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:26:47.623431       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:26:47.623480       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:26:47.623518       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:26:47.675601       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:26:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="4abd211c-6168-11ed-ae2b-6045bd88ebdb"
I1111 02:26:47.682181       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:26:47.682976       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="4abd211c-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7430265
I1111 02:26:47.692221       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:47.810562       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:59.198477       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:26:59.198496       1 utils.go:79] GRPC request: {}
I1111 02:26:59.198536       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:27:17.624308       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:17.624379       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:17.624386       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:17.624420       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:27:17.624435       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:27:17.624446       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:27:17.624456       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:27:17.675760       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:27:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5c9ecb2a-6168-11ed-ae2b-6045bd88ebdb"
I1111 02:27:17.685721       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 9 milliseconds
I1111 02:27:17.686620       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5c9ecb2a-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10894991
I1111 02:27:17.692786       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:17.811129       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:19.604400       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 55 items received
I1111 02:27:19.607937       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=12978&timeout=7m58s&timeoutSeconds=478&watch=true 200 OK in 3 milliseconds
E1111 02:27:27.967902       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1b3b10b1-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845" "latency"=119997513172
E1111 02:27:27.967947       1 crdprovisioner.go:760]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1b3b10b1-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119997712473
E1111 02:27:27.967962       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845) to be attached to node (k8s-agentpool1-30725374-vmss000001): context canceled
E1111 02:27:28.167188       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1b596c4a-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-20e42213-59b8-41b3-85a4-683078bfbb02" "latency"=119997867735
E1111 02:27:28.167256       1 crdprovisioner.go:760]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1b596c4a-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119998060136
E1111 02:27:28.167280       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-20e42213-59b8-41b3-85a4-683078bfbb02) to be attached to node (k8s-agentpool1-30725374-vmss000001): context deadline exceeded
E1111 02:27:28.368615       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1b7815af-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e496bae9-5f84-4a37-bc4d-5d411c91560f" "latency"=119998370603
E1111 02:27:28.368676       1 crdprovisioner.go:760]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1b7815af-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119998534704
E1111 02:27:28.368694       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-e496bae9-5f84-4a37-bc4d-5d411c91560f) to be attached to node (k8s-agentpool1-30725374-vmss000001): context canceled
I1111 02:27:28.551412       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:27:28.551433       1 utils.go:79] GRPC request: {}
I1111 02:27:28.551473       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:27:28.552197       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:27:28.552219       1 utils.go:79] GRPC request: {}
I1111 02:27:28.552251       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:27:28.553032       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:27:28.553052       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845","csi.storage.k8s.io/pvc/name":"inline-volume-tester-w9jm8-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-6041","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845"}
I1111 02:27:28.553242       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845-k8s-agentpool1-30725374-vmss000001-attachment)
E1111 02:27:28.567365       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1b966d78-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d13af50c-86a9-4b8c-81f7-7f377bda5063" "latency"=119998233261
E1111 02:27:28.567423       1 crdprovisioner.go:760]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1b966d78-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119998426763
E1111 02:27:28.567443       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-d13af50c-86a9-4b8c-81f7-7f377bda5063) to be attached to node (k8s-agentpool1-30725374-vmss000001): context deadline exceeded
I1111 02:27:28.752926       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:27:28.752949       1 utils.go:79] GRPC request: {}
I1111 02:27:28.752982       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:27:28.753757       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:27:28.753774       1 utils.go:79] GRPC request: {}
I1111 02:27:28.753803       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 21 lines ...
I1111 02:27:29.197613       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:27:29.197633       1 utils.go:79] GRPC request: {}
I1111 02:27:29.197670       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:27:47.624433       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:47.624499       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:47.624558       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:47.624604       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:27:47.624621       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:27:47.624653       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:27:47.624669       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:27:47.676110       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:27:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="6e807a87-6168-11ed-ae2b-6045bd88ebdb"
I1111 02:27:47.682547       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:27:47.682773       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="6e807a87-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6725757
I1111 02:27:47.693243       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:47.811563       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:59.198165       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:27:59.198190       1 utils.go:79] GRPC request: {}
I1111 02:27:59.198231       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:28:17.624540       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:28:17.624664       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:28:17.624712       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:28:17.624733       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:28:17.624753       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:28:17.624755       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:28:17.624770       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:28:17.676089       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:28:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="80621d78-6168-11ed-ae2b-6045bd88ebdb"
I1111 02:28:17.681761       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:28:17.681971       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="80621d78-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5928848
I1111 02:28:17.693626       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:28:17.812042       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:28:29.198007       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:28:29.198035       1 utils.go:79] GRPC request: {}
I1111 02:28:29.198083       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:28:35.367448       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:28:35.367531       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="637737be-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d13af50c-86a9-4b8c-81f7-7f377bda5063" "latency"=66207023744
I1111 02:28:35.367564       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="637737be-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=66207160045
I1111 02:28:35.373227       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:28:35.373276       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="631a91da-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845" "latency"=66819974678
I1111 02:28:35.373298       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="631a91da-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=66820074579
I1111 02:28:35.374991       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:28:35.375031       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="635861a7-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e496bae9-5f84-4a37-bc4d-5d411c91560f" "latency"=66416647834
I1111 02:28:35.375048       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="635861a7-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=66416739135
I1111 02:28:35.375441       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:28:35.375492       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="63395269-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-20e42213-59b8-41b3-85a4-683078bfbb02" "latency"=66620659576
I1111 02:28:35.375511       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="63395269-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=66620752577
I1111 02:28:36.313150       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1111 02:28:36.313207       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1111 02:28:36.313221       1 utils.go:85] GRPC response: {}
I1111 02:28:36.313238       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/
... skipping 74 lines ...
I1111 02:28:38.652800       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:28:38.653480       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:28:38.653499       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"4"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a85b59b6-6ba5-4d7a-bdef-ef29b3752261","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a85b59b6-6ba5-4d7a-bdef-ef29b3752261","csi.storage.k8s.io/pvc/name":"test.csi.azure.com72qjx","csi.storage.k8s.io/pvc/namespace":"multivolume-7189","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-a85b59b6-6ba5-4d7a-bdef-ef29b3752261"}
I1111 02:28:38.653654       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a85b59b6-6ba5-4d7a-bdef-ef29b3752261-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:28:47.625506       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:28:47.625555       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:28:47.625600       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:28:47.625641       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:28:47.625659       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:28:47.675901       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:28:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9243b9bc-6168-11ed-ae2b-6045bd88ebdb"
I1111 02:28:47.694253       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:28:47.696990       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 20 milliseconds
I1111 02:28:47.698626       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9243b9bc-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=22737601
I1111 02:28:47.812999       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 28 lines ...
I1111 02:28:50.274481       1 utils.go:85] GRPC response: {}
I1111 02:28:50.274445       1 utils.go:79] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-20e42213-59b8-41b3-85a4-683078bfbb02","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-20e42213-59b8-41b3-85a4-683078bfbb02"}
I1111 02:28:50.274529       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-20e42213-59b8-41b3-85a4-683078bfbb02
W1111 02:28:50.274781       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-20e42213-59b8-41b3-85a4-683078bfbb02" is not a mountpoint, deleting
I1111 02:28:50.274825       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-20e42213-59b8-41b3-85a4-683078bfbb02 successfully
I1111 02:28:50.274831       1 utils.go:85] GRPC response: {}
I1111 02:28:50.957229       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:28:50.957291       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="8cb4b6c5-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1466ecb7-5bce-4d42-9de7-fc40b1aaf05c" "latency"=12607198905
I1111 02:28:50.957326       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8cb4b6c5-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12607319205
I1111 02:28:50.959528       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:28:50.959605       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="8ce30b46-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a85b59b6-6ba5-4d7a-bdef-ef29b3752261" "latency"=12305889018
I1111 02:28:50.959646       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8ce30b46-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12305996920
I1111 02:28:51.541216       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun5 by sdg under /dev/disk/azure/scsi1/
I1111 02:28:51.541216       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdh under /dev/disk/azure/scsi1/
I1111 02:28:51.541301       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType 
I1111 02:28:51.541321       1 utils.go:85] GRPC response: {}
... skipping 114 lines ...
W1111 02:29:11.976814       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845" is not a mountpoint, deleting
I1111 02:29:11.976873       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845 successfully
I1111 02:29:11.976882       1 utils.go:85] GRPC response: {}
W1111 02:29:11.976957       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-d13af50c-86a9-4b8c-81f7-7f377bda5063" is not a mountpoint, deleting
I1111 02:29:11.977046       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-d13af50c-86a9-4b8c-81f7-7f377bda5063 successfully
I1111 02:29:11.977066       1 utils.go:85] GRPC response: {}
I1111 02:29:13.238482       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:29:13.238529       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="95cdd07d-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-09de6e98-ef97-4118-9359-0dcf58025169" "latency"=19624459441
I1111 02:29:13.238549       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="95cdd07d-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19624548341
I1111 02:29:13.239363       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:29:13.239413       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="95cd9d08-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-65b99319-536b-4adb-acd1-8410dc77d4c5" "latency"=19626631161
I1111 02:29:13.239438       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="95cd9d08-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19626756762
I1111 02:29:13.851839       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun7 by sdj under /dev/disk/azure/scsi1/
I1111 02:29:13.851840       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun6 by sdi under /dev/disk/azure/scsi1/
I1111 02:29:13.851899       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun7. perfProfile none accountType 
I1111 02:29:13.851931       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun6. perfProfile none accountType 
... skipping 101 lines ...
I1111 02:29:43.537312       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:29:43.537328       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"8"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-71fe6150-d0bf-4346-b3c2-a1365614f344/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-71fe6150-d0bf-4346-b3c2-a1365614f344","csi.storage.k8s.io/pvc/name":"inline-volume-tester-bxf4c-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-696","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-71fe6150-d0bf-4346-b3c2-a1365614f344"}
I1111 02:29:43.537500       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-71fe6150-d0bf-4346-b3c2-a1365614f344-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:29:47.626745       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:29:47.626761       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:29:47.626771       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:29:47.626956       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:29:47.676250       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:29:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b6070ad5-6168-11ed-ae2b-6045bd88ebdb"
I1111 02:29:47.682119       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:29:47.682338       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b6070ad5-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6187447
I1111 02:29:47.696305       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:29:47.814619       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:29:59.043105       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:29:59.043160       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b38f8871-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-71fe6150-d0bf-4346-b3c2-a1365614f344" "latency"=15505594536
I1111 02:29:59.043185       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b38f8871-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15505701738
I1111 02:29:59.197710       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:29:59.197733       1 utils.go:79] GRPC request: {}
I1111 02:29:59.197780       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:29:59.580145       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 55 lines ...
I1111 02:30:00.373465       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:30:00.373481       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"9"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-59e83969-58ae-46ee-b1d2-b773a4527216","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-59e83969-58ae-46ee-b1d2-b773a4527216","csi.storage.k8s.io/pvc/name":"test.csi.azure.comzxmlz","csi.storage.k8s.io/pvc/namespace":"multivolume-2130","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-59e83969-58ae-46ee-b1d2-b773a4527216"}
I1111 02:30:00.373640       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-59e83969-58ae-46ee-b1d2-b773a4527216-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:30:17.626860       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:30:17.626943       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:30:17.626956       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:30:17.627055       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:30:17.627082       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:30:17.627111       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:30:17.676460       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:30:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c7e8b745-6168-11ed-ae2b-6045bd88ebdb"
I1111 02:30:17.681647       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds
I1111 02:30:17.682065       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c7e8b745-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5635942
I1111 02:30:17.696914       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:30:17.815287       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:30:29.197767       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
... skipping 18 lines ...
W1111 02:30:35.189670       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-71fe6150-d0bf-4346-b3c2-a1365614f344/globalmount" is not a mountpoint, deleting
I1111 02:30:35.189735       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-71fe6150-d0bf-4346-b3c2-a1365614f344/globalmount successfully
I1111 02:30:35.189748       1 utils.go:85] GRPC response: {}
I1111 02:30:47.627862       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:30:47.627977       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:30:47.628028       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:30:47.628209       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:30:47.628324       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:30:47.628364       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:30:47.675829       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:30:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d9ca425c-6168-11ed-ae2b-6045bd88ebdb"
I1111 02:30:47.682589       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:30:47.682808       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d9ca425c-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7025357
I1111 02:30:47.697971       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:30:47.815701       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:30:50.026098       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:30:50.026180       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="bd1fbd26-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-20e42213-59b8-41b3-85a4-683078bfbb02" "latency"=50444039056
I1111 02:30:50.026219       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bd1fbd26-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=50444168858
I1111 02:30:50.030993       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:30:50.031036       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="bd988621-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-59e83969-58ae-46ee-b1d2-b773a4527216" "latency"=49657349381
I1111 02:30:50.031056       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bd988621-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=49657435482
I1111 02:30:50.031201       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:30:50.031252       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="bd2c3c15-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e496bae9-5f84-4a37-bc4d-5d411c91560f" "latency"=50367226629
I1111 02:30:50.031284       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bd2c3c15-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=50367343731
I1111 02:30:50.644325       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdf under /dev/disk/azure/scsi1/
I1111 02:30:50.644375       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:30:50.644387       1 utils.go:85] GRPC response: {}
I1111 02:30:50.644972       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun9 by sde under /dev/disk/azure/scsi1/
... skipping 165 lines ...
I1111 02:33:03.488882       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:33:03.488899       1 utils.go:79] GRPC request: {}
I1111 02:33:03.488937       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:33:03.489703       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:33:03.489717       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-59e83969-58ae-46ee-b1d2-b773a4527216","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-59e83969-58ae-46ee-b1d2-b773a4527216","csi.storage.k8s.io/pvc/name":"test.csi.azure.comzxmlz","csi.storage.k8s.io/pvc/namespace":"multivolume-2130","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-59e83969-58ae-46ee-b1d2-b773a4527216"}
I1111 02:33:03.489901       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-59e83969-58ae-46ee-b1d2-b773a4527216-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:33:12.166239       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:33:12.166302       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="2a333978-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6f13450e-7108-4e37-91bc-2507a8873ea1" "latency"=9584815902
I1111 02:33:12.166330       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2a333978-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9584928204
I1111 02:33:12.166370       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:33:12.166417       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="2abdd922-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-59e83969-58ae-46ee-b1d2-b773a4527216" "latency"=8676467213
I1111 02:33:12.166435       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2abdd922-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8676554415
I1111 02:33:12.759431       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/
I1111 02:33:12.759475       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:33:12.759486       1 utils.go:85] GRPC response: {}
I1111 02:33:12.759805       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
... skipping 80 lines ...
I1111 02:33:29.197889       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:33:29.197916       1 utils.go:79] GRPC request: {}
I1111 02:33:29.197970       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:33:47.631134       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:33:47.631162       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:33:47.631258       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:33:47.631383       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:33:47.675579       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:33:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="45140a59-6169-11ed-ae2b-6045bd88ebdb"
I1111 02:33:47.680727       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds
I1111 02:33:47.681461       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="45140a59-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5932447
I1111 02:33:47.700905       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:33:47.819161       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:33:48.991335       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 2 lines ...
I1111 02:33:48.992238       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1111 02:33:48.992252       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-59e83969-58ae-46ee-b1d2-b773a4527216","volume_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-59e83969-58ae-46ee-b1d2-b773a4527216/dev/000ca66c-7dd5-443f-93e5-13ea038fc628"}
I1111 02:33:48.992951       1 utils.go:85] GRPC response: {"usage":[{"total":5368709120,"unit":1}]}
I1111 02:33:59.197080       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:33:59.197099       1 utils.go:79] GRPC request: {}
I1111 02:33:59.197141       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:34:09.635014       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:34:09.635073       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="397528c4-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154" "latency"=41455642678
I1111 02:34:09.635096       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="397528c4-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=41455763478
I1111 02:34:10.223876       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/
I1111 02:34:10.223923       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1111 02:34:10.223936       1 utils.go:85] GRPC response: {}
I1111 02:34:10.239022       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 33 lines ...
I1111 02:34:11.571268       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:34:11.571287       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a67a0ecf-f127-4252-9abc-8f1327f68862/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a67a0ecf-f127-4252-9abc-8f1327f68862","csi.storage.k8s.io/pvc/name":"pvc-5g7jp","csi.storage.k8s.io/pvc/namespace":"provisioning-3371","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-a67a0ecf-f127-4252-9abc-8f1327f68862"}
I1111 02:34:11.571461       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a67a0ecf-f127-4252-9abc-8f1327f68862-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:34:17.631881       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:34:17.631893       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:34:17.631936       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:34:17.632026       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:34:17.632064       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:34:17.640274       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1111 02:34:17.640291       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154/98ba362c-4b5e-4135-b34a-974e5e16c80f","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154"}
I1111 02:34:17.640342       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154 on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154/98ba362c-4b5e-4135-b34a-974e5e16c80f
I1111 02:34:17.640367       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154/98ba362c-4b5e-4135-b34a-974e5e16c80f" is a mountpoint, unmounting
I1111 02:34:17.640380       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154/98ba362c-4b5e-4135-b34a-974e5e16c80f
W1111 02:34:17.642064       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154/98ba362c-4b5e-4135-b34a-974e5e16c80f" is not a mountpoint, deleting
... skipping 30 lines ...
W1111 02:34:20.928631       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-59e83969-58ae-46ee-b1d2-b773a4527216" is not a mountpoint, deleting
I1111 02:34:20.928692       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-59e83969-58ae-46ee-b1d2-b773a4527216 successfully
I1111 02:34:20.928709       1 utils.go:85] GRPC response: {}
I1111 02:34:29.197465       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:34:29.197488       1 utils.go:79] GRPC request: {}
I1111 02:34:29.197534       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:34:30.281679       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:34:30.281749       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="535244c0-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a67a0ecf-f127-4252-9abc-8f1327f68862" "latency"=18710229470
I1111 02:34:30.281784       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="535244c0-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=18710342471
I1111 02:34:30.285101       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:34:30.285157       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="53523a06-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c968c265-3bba-493f-bae7-b67ba22bcf87" "latency"=18713889699
I1111 02:34:30.285178       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="53523a06-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=18714019500
I1111 02:34:30.868542       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1111 02:34:30.868581       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:34:30.868612       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c968c265-3bba-493f-bae7-b67ba22bcf87/globalmount with mount options([])
I1111 02:34:30.868622       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 142 lines ...
I1111 02:35:15.744602       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:35:17.607950       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 70 items received
I1111 02:35:17.610401       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=17121&timeout=6m5s&timeoutSeconds=365&watch=true 200 OK in 2 milliseconds
I1111 02:35:17.633650       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:35:17.633676       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:35:17.633745       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:35:17.633781       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:35:17.633827       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:35:17.676231       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:35:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7ab90d96-6169-11ed-ae2b-6045bd88ebdb"
I1111 02:35:17.681561       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:35:17.682356       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7ab90d96-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6156049
I1111 02:35:17.703809       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:35:17.821310       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:35:29.198078       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:35:29.198103       1 utils.go:79] GRPC request: {}
I1111 02:35:29.198146       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:35:38.686295       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:35:38.686359       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7992507a-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c" "latency"=22941705024
I1111 02:35:38.686384       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7992507a-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=22941795624
I1111 02:35:38.687348       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:35:38.687410       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7945787f-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c4cfb065-dbf9-4356-ac29-6e2c98a476c6" "latency"=23446299025
I1111 02:35:38.687452       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7945787f-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=23446461726
I1111 02:35:39.273167       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdd under /dev/disk/azure/scsi1/
I1111 02:35:39.273207       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1111 02:35:39.273236       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c/globalmount with mount options([])
I1111 02:35:39.273246       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3])
... skipping 66 lines ...
I1111 02:35:39.725482       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:35:39.726151       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:35:39.726166       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a","csi.storage.k8s.io/pvc/name":"test.csi.azure.comwmppf","csi.storage.k8s.io/pvc/namespace":"snapshotting-4341","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a"}
I1111 02:35:39.726318       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:35:47.634500       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:35:47.634577       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:35:47.634589       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:35:47.634602       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:35:47.676038       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:35:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="8c9aa8cf-6169-11ed-ae2b-6045bd88ebdb"
I1111 02:35:47.682123       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:35:47.682576       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="8c9aa8cf-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6577732
I1111 02:35:47.703875       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:35:47.822339       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:35:48.937606       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:35:48.937710       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="87dda225-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a" "latency"=9211323755
I1111 02:35:48.937732       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="87dda225-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9211434455
I1111 02:35:49.534289       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/
I1111 02:35:49.534327       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:35:49.534356       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a/globalmount with mount options([])
I1111 02:35:49.534364       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 93 lines ...
I1111 02:36:01.779940       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:36:01.779959       1 utils.go:79] GRPC request: {}
I1111 02:36:01.779997       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:36:01.780702       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:36:01.780717       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94","csi.storage.k8s.io/pvc/name":"pvc-mjhpr","csi.storage.k8s.io/pvc/namespace":"snapshotting-4341","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94"}
I1111 02:36:01.780889       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:36:13.844527       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:36:13.844600       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9502e4e3-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94" "latency"=12063638934
I1111 02:36:13.844634       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9502e4e3-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12063756135
I1111 02:36:14.355472       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:36:14.355500       1 utils.go:79] GRPC request: {}
I1111 02:36:14.355545       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:36:14.358858       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 36 lines ...
I1111 02:36:14.565177       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94/globalmount /var/lib/kubelet/pods/72c9660b-69ed-458e-98bf-ec8c51fa7209/volumes/kubernetes.io~csi/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94/mount)
I1111 02:36:14.565943       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94/globalmount at /var/lib/kubelet/pods/72c9660b-69ed-458e-98bf-ec8c51fa7209/volumes/kubernetes.io~csi/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94/mount successfully
I1111 02:36:14.565963       1 utils.go:85] GRPC response: {}
I1111 02:36:17.635055       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:36:17.635114       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:36:17.635191       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:36:17.635341       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:36:17.675811       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:36:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9e7c4392-6169-11ed-ae2b-6045bd88ebdb"
I1111 02:36:17.681330       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:36:17.681498       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9e7c4392-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5714245
I1111 02:36:17.704821       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:36:17.823154       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:36:29.197712       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
... skipping 2 lines ...
I1111 02:36:37.073249       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:36:37.073276       1 utils.go:79] GRPC request: {}
I1111 02:36:37.073324       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:36:37.074061       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1111 02:36:37.074082       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94","volume_path":"/var/lib/kubelet/pods/72c9660b-69ed-458e-98bf-ec8c51fa7209/volumes/kubernetes.io~csi/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94/mount"}
I1111 02:36:37.074180       1 utils.go:85] GRPC response: {"usage":[{"available":5179576320,"total":5196382208,"unit":1,"used":28672},{"available":327668,"total":327680,"unit":2,"used":12}]}
I1111 02:36:44.583573       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:36:44.583647       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9c8268e4-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-34903c00-ca40-488f-a7e7-3fe1806b3674" "latency"=30222956335
I1111 02:36:44.583683       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9c8268e4-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=30223066137
I1111 02:36:45.172999       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/
I1111 02:36:45.173090       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1111 02:36:45.173131       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-34903c00-ca40-488f-a7e7-3fe1806b3674/globalmount with mount options([])
I1111 02:36:45.173153       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 169 lines ...
I1111 02:37:23.459935       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-00efa424-3c77-4a8e-9dc1-a1e7b3b1f847-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:37:24.613122       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 220 items received
I1111 02:37:24.614762       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=18592&timeout=7m17s&timeoutSeconds=437&watch=true 200 OK in 1 milliseconds
I1111 02:37:29.197522       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:37:29.197545       1 utils.go:79] GRPC request: {}
I1111 02:37:29.197589       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:37:45.314216       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:37:45.314287       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c5b21633-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a51dc9b0-7b91-42ed-abe4-206458934d79" "latency"=21854527623
I1111 02:37:45.314317       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c5b21633-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=21854666724
I1111 02:37:45.315924       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:37:45.315991       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c5b22096-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-00efa424-3c77-4a8e-9dc1-a1e7b3b1f847" "latency"=21855993035
I1111 02:37:45.316021       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c5b22096-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=21856097836
I1111 02:37:45.824206       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:37:45.824231       1 utils.go:79] GRPC request: {}
I1111 02:37:45.824275       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:37:45.827291       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 69 lines ...
I1111 02:37:46.414070       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/globalmount /var/lib/kubelet/pods/a743d533-5855-4339-af5a-0d139613a95c/volumes/kubernetes.io~csi/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/mount)
I1111 02:37:46.414889       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/globalmount at /var/lib/kubelet/pods/a743d533-5855-4339-af5a-0d139613a95c/volumes/kubernetes.io~csi/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/mount successfully
I1111 02:37:46.414903       1 utils.go:85] GRPC response: {}
I1111 02:37:47.637308       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:37:47.637353       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:37:47.637315       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:37:47.637437       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:37:47.675595       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:37:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d421229a-6169-11ed-ae2b-6045bd88ebdb"
I1111 02:37:47.681500       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:37:47.682451       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d421229a-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6908857
I1111 02:37:47.706612       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:37:47.824866       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:37:59.197472       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
... skipping 118 lines ...
I1111 02:38:04.733385       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-00efa424-3c77-4a8e-9dc1-a1e7b3b1f847/globalmount /var/lib/kubelet/pods/dd3b6b97-012e-47ef-a2a8-33a6a1d70be9/volumes/kubernetes.io~csi/pvc-00efa424-3c77-4a8e-9dc1-a1e7b3b1f847/mount)
I1111 02:38:04.733662       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/globalmount /var/lib/kubelet/pods/dd3b6b97-012e-47ef-a2a8-33a6a1d70be9/volumes/kubernetes.io~csi/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/mount)
I1111 02:38:04.734106       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-00efa424-3c77-4a8e-9dc1-a1e7b3b1f847/globalmount at /var/lib/kubelet/pods/dd3b6b97-012e-47ef-a2a8-33a6a1d70be9/volumes/kubernetes.io~csi/pvc-00efa424-3c77-4a8e-9dc1-a1e7b3b1f847/mount successfully
I1111 02:38:04.734116       1 utils.go:85] GRPC response: {}
I1111 02:38:04.734578       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/globalmount at /var/lib/kubelet/pods/dd3b6b97-012e-47ef-a2a8-33a6a1d70be9/volumes/kubernetes.io~csi/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/mount successfully
I1111 02:38:04.734594       1 utils.go:85] GRPC response: {}
I1111 02:38:05.674369       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:38:05.674443       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d3075d90-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-50c4b81c-d020-4dad-a795-c865b1da561d" "latency"=19845441292
I1111 02:38:05.674469       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d3075d90-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19845551493
I1111 02:38:06.272374       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1111 02:38:06.272420       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1111 02:38:06.272454       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-50c4b81c-d020-4dad-a795-c865b1da561d/globalmount with mount options([])
I1111 02:38:06.272467       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 104 lines ...
I1111 02:38:29.197805       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:38:29.197824       1 utils.go:79] GRPC request: {}
I1111 02:38:29.197861       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:38:47.638803       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:38:47.638851       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:38:47.638830       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:38:47.638999       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:38:47.676473       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:38:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f7e48aa3-6169-11ed-ae2b-6045bd88ebdb"
I1111 02:38:47.682424       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:38:47.682641       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f7e48aa3-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6270047
I1111 02:38:47.708407       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:38:47.826829       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:38:59.197790       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:38:59.197809       1 utils.go:79] GRPC request: {}
I1111 02:38:59.197857       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:39:03.797139       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 29 items received
I1111 02:39:03.800459       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-30725374-vmss000001&resourceVersion=19263&timeout=7m44s&timeoutSeconds=464&watch=true 200 OK in 3 milliseconds
I1111 02:39:06.585081       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:39:06.585146       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="eb197197-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-41e04970-9406-4ef2-8242-676cbc074a5d" "latency"=40372320520
I1111 02:39:06.585179       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="eb197197-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=40372462220
I1111 02:39:07.216035       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:39:07.216059       1 utils.go:79] GRPC request: {}
I1111 02:39:07.216098       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:39:07.219424       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 53 lines ...
I1111 02:39:15.570173       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-41e04970-9406-4ef2-8242-676cbc074a5d/globalmount
I1111 02:39:15.570214       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-41e04970-9406-4ef2-8242-676cbc074a5d/globalmount" is a mountpoint, unmounting
I1111 02:39:15.570229       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-41e04970-9406-4ef2-8242-676cbc074a5d/globalmount
W1111 02:39:15.592605       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-41e04970-9406-4ef2-8242-676cbc074a5d/globalmount" is not a mountpoint, deleting
I1111 02:39:15.592698       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-41e04970-9406-4ef2-8242-676cbc074a5d/globalmount successfully
I1111 02:39:15.592720       1 utils.go:85] GRPC response: {}
I1111 02:39:16.891232       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:39:16.891293       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="038ad6e0-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-938bb352-9562-491f-b7d6-05b0976e093b" "latency"=9670033046
I1111 02:39:16.891323       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="038ad6e0-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9670137546
I1111 02:39:17.486641       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1111 02:39:17.486693       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:39:17.486733       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-938bb352-9562-491f-b7d6-05b0976e093b/globalmount with mount options([])
I1111 02:39:17.486751       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 68 lines ...
I1111 02:39:29.197233       1 utils.go:79] GRPC request: {}
I1111 02:39:29.197281       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:39:45.686082       1 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: Watch close - *v1.CustomResourceDefinition total 7 items received
I1111 02:39:45.688535       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=16328&timeout=7m3s&timeoutSeconds=423&watch=true 200 OK in 2 milliseconds
I1111 02:39:47.640001       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:39:47.640082       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:39:47.640115       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:39:47.640159       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:39:47.676460       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:39:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1ba7d192-616a-11ed-ae2b-6045bd88ebdb"
I1111 02:39:47.681858       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:39:47.682400       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1ba7d192-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5973306
I1111 02:39:47.709575       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:39:47.828945       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:39:59.197924       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:39:59.197947       1 utils.go:79] GRPC request: {}
I1111 02:39:59.197997       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:40:17.640474       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:40:17.640497       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:40:17.640541       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:40:17.640626       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:40:17.676023       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:40:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="2d8962c8-616a-11ed-ae2b-6045bd88ebdb"
I1111 02:40:17.681968       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:40:17.682398       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="2d8962c8-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6430456
I1111 02:40:17.710675       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:40:17.830015       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:40:20.303453       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:40:20.303521       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="0f03fd1a-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c790a21e-6466-4989-a38c-3c9f1bb85b5e" "latency"=53833342957
I1111 02:40:20.303545       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0f03fd1a-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=53833462158
I1111 02:40:22.161787       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1111 02:40:22.161835       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:40:22.161877       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c790a21e-6466-4989-a38c-3c9f1bb85b5e/globalmount with mount options([])
I1111 02:40:22.161892       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 60 lines ...
I1111 02:40:37.480437       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:40:37.480452       1 utils.go:79] GRPC request: {}
I1111 02:40:37.480480       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:40:37.481273       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:40:37.481293       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1","csi.storage.k8s.io/pvc/name":"test.csi.azure.com4h6ww","csi.storage.k8s.io/pvc/namespace":"multivolume-8476","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1"}
I1111 02:40:37.481461       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:40:46.708563       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:40:46.708638       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="3957766e-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1" "latency"=9227095568
I1111 02:40:46.708679       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3957766e-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9227225470
I1111 02:40:46.951275       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:40:46.951296       1 utils.go:79] GRPC request: {}
I1111 02:40:46.951337       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:40:46.952077       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 8 lines ...
I1111 02:40:47.300295       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
I1111 02:40:47.309763       1 mount_linux.go:490] Output: ""
I1111 02:40:47.309787       1 mount_linux.go:449] Disk "/dev/disk/azure/scsi1/lun0" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/disk/azure/scsi1/lun0]
I1111 02:40:47.641154       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:40:47.641199       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:40:47.641218       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:40:47.641291       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:40:47.675650       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:40:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="3f6af5ff-616a-11ed-ae2b-6045bd88ebdb"
I1111 02:40:47.681574       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:40:47.681903       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="3f6af5ff-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6339366
I1111 02:40:47.711090       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:40:47.789098       1 mount_linux.go:459] Disk successfully formatted (mkfs): ext4 - /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/globalmount
I1111 02:40:47.789131       1 mount_linux.go:477] Attempting to mount disk /dev/disk/azure/scsi1/lun0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/globalmount
... skipping 86 lines ...
I1111 02:41:09.879781       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/globalmount /var/lib/kubelet/pods/02ecbfaf-5b90-41b3-8999-13e662be968e/volumes/kubernetes.io~csi/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/mount)
I1111 02:41:09.880639       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/globalmount at /var/lib/kubelet/pods/02ecbfaf-5b90-41b3-8999-13e662be968e/volumes/kubernetes.io~csi/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/mount successfully
I1111 02:41:09.880661       1 utils.go:85] GRPC response: {}
I1111 02:41:17.641809       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:41:17.641969       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:41:17.641997       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:41:17.642178       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:41:17.676462       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:41:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="514cba10-616a-11ed-ae2b-6045bd88ebdb"
I1111 02:41:17.693381       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 16 milliseconds
I1111 02:41:17.695089       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="514cba10-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18673684
I1111 02:41:17.711524       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:41:17.831884       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:41:22.611394       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 58 items received
... skipping 4 lines ...
I1111 02:41:37.075269       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:41:37.075290       1 utils.go:79] GRPC request: {}
I1111 02:41:37.075325       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:41:37.076096       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1111 02:41:37.076116       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1","volume_path":"/var/lib/kubelet/pods/02ecbfaf-5b90-41b3-8999-13e662be968e/volumes/kubernetes.io~csi/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/mount"}
I1111 02:41:37.076192       1 utils.go:85] GRPC response: {"usage":[{"available":5179576320,"total":5196382208,"unit":1,"used":28672},{"available":327668,"total":327680,"unit":2,"used":12}]}
I1111 02:41:37.568416       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:41:37.568489       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="3efce76b-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f7af547e-1ffd-4b0e-8a57-45ce93934d64" "latency"=50614128239
I1111 02:41:37.568519       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3efce76b-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=50614226342
I1111 02:41:38.160858       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/
I1111 02:41:38.160908       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1111 02:41:38.160920       1 utils.go:85] GRPC response: {}
I1111 02:41:38.164623       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 38 lines ...
W1111 02:41:46.986634       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-f7af547e-1ffd-4b0e-8a57-45ce93934d64" is not a mountpoint, deleting
I1111 02:41:46.986696       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-f7af547e-1ffd-4b0e-8a57-45ce93934d64 successfully
I1111 02:41:46.986704       1 utils.go:85] GRPC response: {}
I1111 02:41:47.642487       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:41:47.642516       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:41:47.642506       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:41:47.642618       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:41:47.675938       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:41:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="632e4849-616a-11ed-ae2b-6045bd88ebdb"
I1111 02:41:47.681431       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:41:47.681666       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="632e4849-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5788962
I1111 02:41:47.711786       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:41:47.832051       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:41:59.197845       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:41:59.197870       1 utils.go:79] GRPC request: {}
I1111 02:41:59.197917       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:42:13.103262       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:42:13.103337       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5e074f14-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d4853aa0-c900-469a-b015-4429594b1f96" "latency"=34071376356
I1111 02:42:13.103370       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5e074f14-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=34071515857
I1111 02:42:13.696273       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/
I1111 02:42:13.696325       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:42:13.696352       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d4853aa0-c900-469a-b015-4429594b1f96/globalmount with mount options([])
I1111 02:42:13.696361       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 83 lines ...
I1111 02:42:30.093856       1 utils.go:79] GRPC request: {}
I1111 02:42:30.093888       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:42:30.094520       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:42:30.094534       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"3"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e","csi.storage.k8s.io/pvc/name":"pvc-2bwlb","csi.storage.k8s.io/pvc/namespace":"snapshotting-955","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e"}
I1111 02:42:30.094672       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:42:47.643677       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:42:47.643783       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:42:47.643806       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:42:47.643831       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:42:47.676189       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:42:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="86f198fd-616a-11ed-ae2b-6045bd88ebdb"
I1111 02:42:47.681114       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds
I1111 02:42:47.681337       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="86f198fd-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5182946
I1111 02:42:47.713116       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:42:47.833519       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:42:59.198104       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:42:59.198132       1 utils.go:79] GRPC request: {}
I1111 02:42:59.198178       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:43:17.644506       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:43:17.644583       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:43:17.644581       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:43:17.644706       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:43:17.676042       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:43:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="98d335a5-616a-11ed-ae2b-6045bd88ebdb"
I1111 02:43:17.681984       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:43:17.682539       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="98d335a5-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6547355
I1111 02:43:17.713988       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:43:17.834385       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:43:29.197262       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:43:29.197286       1 utils.go:79] GRPC request: {}
I1111 02:43:29.197336       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:43:39.270059       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:43:39.270114       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7c76dff2-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e" "latency"=69175367359
I1111 02:43:39.270140       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7c76dff2-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=69175484960
I1111 02:43:39.863159       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/
I1111 02:43:39.863211       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1111 02:43:39.863246       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e/globalmount with mount options([])
I1111 02:43:39.863258       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3])
... skipping 54 lines ...
I1111 02:43:41.532248       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:43:41.532263       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"5"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cacecee8-1f4b-4286-a42c-6c1dc2988e3c/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-cacecee8-1f4b-4286-a42c-6c1dc2988e3c","csi.storage.k8s.io/pvc/name":"pvc-tfmgp","csi.storage.k8s.io/pvc/namespace":"provisioning-6901","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-cacecee8-1f4b-4286-a42c-6c1dc2988e3c"}
I1111 02:43:41.532421       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-cacecee8-1f4b-4286-a42c-6c1dc2988e3c-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:43:47.644918       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:43:47.645061       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:43:47.645095       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:43:47.645279       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:43:47.645355       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:43:47.645429       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:43:47.676518       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:43:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="aab4ec19-616a-11ed-ae2b-6045bd88ebdb"
I1111 02:43:47.683633       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:43:47.683903       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="aab4ec19-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7400278
I1111 02:43:47.714459       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:43:47.834876       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:43:59.198047       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
... skipping 23 lines ...
I1111 02:44:15.983455       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e/globalmount
W1111 02:44:15.999503       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e/globalmount" is not a mountpoint, deleting
I1111 02:44:15.999556       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e/globalmount successfully
I1111 02:44:15.999565       1 utils.go:85] GRPC response: {}
I1111 02:44:17.645759       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:44:17.645848       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:44:17.645869       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:44:17.645879       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:44:17.645903       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:44:17.645918       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:44:17.676323       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:44:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="bc968785-616a-11ed-ae2b-6045bd88ebdb"
I1111 02:44:17.683160       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:44:17.683329       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="bc968785-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7024458
I1111 02:44:17.714662       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:44:17.835046       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:44:29.197881       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:44:29.197903       1 utils.go:79] GRPC request: {}
I1111 02:44:29.197950       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:44:41.024011       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:44:41.024068       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a70b690b-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-cacecee8-1f4b-4286-a42c-6c1dc2988e3c" "latency"=59491589335
I1111 02:44:41.024095       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a70b690b-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=59491691435
I1111 02:44:41.028282       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:44:41.028326       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a6be1b8e-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d021c5d3-9653-4df4-b6f0-2b4b051c35d0" "latency"=60002448197
I1111 02:44:41.028346       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a6be1b8e-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=60002555798
I1111 02:44:41.029088       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:44:41.029161       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a6be1cc1-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-78e6944f-19bb-4f8e-81ef-8dbae9ae0778" "latency"=60003269204
I1111 02:44:41.029196       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a6be1cc1-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=60003369004
I1111 02:44:41.615896       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 190 items received
I1111 02:44:41.619564       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=22232&timeout=6m23s&timeoutSeconds=383&watch=true 200 OK in 3 milliseconds
I1111 02:44:41.980003       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdf under /dev/disk/azure/scsi1/
I1111 02:44:41.980072       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
... skipping 146 lines ...
I1111 02:45:29.197154       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:45:29.197178       1 utils.go:79] GRPC request: {}
I1111 02:45:29.197215       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:45:47.646772       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:45:47.646924       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:45:47.646965       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:45:47.647038       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:45:47.676340       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:45:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f23b70b5-616a-11ed-ae2b-6045bd88ebdb"
I1111 02:45:47.681764       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:45:47.682012       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f23b70b5-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5709548
I1111 02:45:47.715813       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:45:47.836054       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:45:52.807778       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:45:52.807860       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="e4173e7a-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1216e252-79f7-49d1-8737-6722ac45923d" "latency"=28856781576
I1111 02:45:52.807902       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e4173e7a-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=28856921278
I1111 02:45:53.871356       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:45:53.871381       1 utils.go:79] GRPC request: {}
I1111 02:45:53.871418       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:45:53.874540       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 41 lines ...
I1111 02:45:59.197925       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:46:13.607155       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 118 items received
I1111 02:46:13.609604       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=23172&timeout=7m31s&timeoutSeconds=451&watch=true 200 OK in 2 milliseconds
I1111 02:46:17.647722       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:46:17.647778       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:46:17.647804       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:46:17.647903       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:46:17.676158       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:46:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="041d0b99-616b-11ed-ae2b-6045bd88ebdb"
I1111 02:46:17.680774       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds
I1111 02:46:17.681091       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="041d0b99-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=4995241
I1111 02:46:17.716211       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:46:17.836627       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:46:23.433289       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:46:23.433369       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f5ed7bc3-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1d184be2-3854-4618-a760-98f325e099c3" "latency"=29557007399
I1111 02:46:23.433411       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f5ed7bc3-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29557122600
I1111 02:46:24.028133       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1111 02:46:24.028191       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:46:24.028224       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1d184be2-3854-4618-a760-98f325e099c3/globalmount with mount options([])
I1111 02:46:24.028240       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 89 lines ...
I1111 02:47:05.879394       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:47:05.879410       1 utils.go:79] GRPC request: {}
I1111 02:47:05.879446       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:47:05.880257       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:47:05.880276       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-341c5962-95ed-4733-8b4a-433fbb9dd5aa","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-341c5962-95ed-4733-8b4a-433fbb9dd5aa","csi.storage.k8s.io/pvc/name":"pvc-mclzr","csi.storage.k8s.io/pvc/namespace":"provisioning-200","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-341c5962-95ed-4733-8b4a-433fbb9dd5aa"}
I1111 02:47:05.880450       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-341c5962-95ed-4733-8b4a-433fbb9dd5aa-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:47:14.288439       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:47:14.288505       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="20d871ed-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-341c5962-95ed-4733-8b4a-433fbb9dd5aa" "latency"=8407969841
I1111 02:47:14.288529       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="20d871ed-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8408095842
I1111 02:47:14.911051       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1111 02:47:14.911100       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:47:14.911111       1 utils.go:85] GRPC response: {}
I1111 02:47:14.916648       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 21 lines ...
I1111 02:47:16.166352       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:47:16.166371       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f505dad6-786d-4e08-a7ce-a993beed5714/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f505dad6-786d-4e08-a7ce-a993beed5714","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-2bf67-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-5208","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-f505dad6-786d-4e08-a7ce-a993beed5714"}
I1111 02:47:16.166558       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-f505dad6-786d-4e08-a7ce-a993beed5714-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:47:17.648341       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:47:17.648408       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:47:17.648421       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:47:17.648481       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:47:17.675868       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:47:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="27e0466e-616b-11ed-ae2b-6045bd88ebdb"
I1111 02:47:17.681965       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:47:17.682205       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="27e0466e-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6398052
I1111 02:47:17.718262       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:47:17.837650       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:47:24.252667       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
... skipping 16 lines ...
I1111 02:47:29.197342       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:47:29.197362       1 utils.go:79] GRPC request: {}
I1111 02:47:29.197401       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:47:47.648801       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:47:47.648870       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:47:47.648897       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:47:47.648944       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:47:47.676245       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:47:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="39c1f935-616b-11ed-ae2b-6045bd88ebdb"
I1111 02:47:47.682660       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:47:47.682879       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="39c1f935-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6658355
I1111 02:47:47.718926       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:47:47.838301       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:47:50.356817       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:47:50.356890       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="26f9fad4-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f505dad6-786d-4e08-a7ce-a993beed5714" "latency"=34190270190
I1111 02:47:50.356941       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="26f9fad4-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=34190379891
I1111 02:47:50.811862       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:47:50.811882       1 utils.go:79] GRPC request: {}
I1111 02:47:50.811915       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:47:50.815153       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 49 lines ...
I1111 02:47:52.855754       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f505dad6-786d-4e08-a7ce-a993beed5714/globalmount /var/lib/kubelet/pods/6f927703-35aa-46f6-97fd-c04319cb012e/volumes/kubernetes.io~csi/pvc-f505dad6-786d-4e08-a7ce-a993beed5714/mount)
I1111 02:47:52.856524       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f505dad6-786d-4e08-a7ce-a993beed5714/globalmount at /var/lib/kubelet/pods/6f927703-35aa-46f6-97fd-c04319cb012e/volumes/kubernetes.io~csi/pvc-f505dad6-786d-4e08-a7ce-a993beed5714/mount successfully
I1111 02:47:52.856538       1 utils.go:85] GRPC response: {}
I1111 02:47:59.198028       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:47:59.198054       1 utils.go:79] GRPC request: {}
I1111 02:47:59.198100       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:48:05.817481       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:48:05.817536       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="3ba12dc6-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ce77a2e2-f9fb-4c4f-b3d4-b38e7d635385" "latency"=15000724925
I1111 02:48:05.817563       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3ba12dc6-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15000836426
I1111 02:48:05.822252       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:48:05.822313       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="3bc04777-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f42a016a-7ee3-409b-ba03-5dc001702368" "latency"=14801703889
I1111 02:48:05.822349       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3bc04777-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14801801789
I1111 02:48:06.234737       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:48:06.234763       1 utils.go:79] GRPC request: {}
I1111 02:48:06.234800       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:48:06.235627       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 66 lines ...
I1111 02:48:06.922137       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f42a016a-7ee3-409b-ba03-5dc001702368/globalmount /var/lib/kubelet/pods/4dee0fad-d284-4c05-93a1-b000ddac5a20/volumes/kubernetes.io~csi/pvc-f42a016a-7ee3-409b-ba03-5dc001702368/mount)
I1111 02:48:06.922892       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f42a016a-7ee3-409b-ba03-5dc001702368/globalmount at /var/lib/kubelet/pods/4dee0fad-d284-4c05-93a1-b000ddac5a20/volumes/kubernetes.io~csi/pvc-f42a016a-7ee3-409b-ba03-5dc001702368/mount successfully
I1111 02:48:06.922909       1 utils.go:85] GRPC response: {}
I1111 02:48:17.649601       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:48:17.649655       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:48:17.649655       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:48:17.649786       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:48:17.676008       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:48:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="4ba39327-616b-11ed-ae2b-6045bd88ebdb"
I1111 02:48:17.682573       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:48:17.682765       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="4ba39327-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6779956
I1111 02:48:17.720029       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:48:17.839365       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:48:19.649436       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
... skipping 31 lines ...
W1111 02:48:19.771899       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ce77a2e2-f9fb-4c4f-b3d4-b38e7d635385/globalmount" is not a mountpoint, deleting
I1111 02:48:19.771955       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ce77a2e2-f9fb-4c4f-b3d4-b38e7d635385/globalmount successfully
I1111 02:48:19.771965       1 utils.go:85] GRPC response: {}
W1111 02:48:19.774155       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f42a016a-7ee3-409b-ba03-5dc001702368/globalmount" is not a mountpoint, deleting
I1111 02:48:19.774226       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f42a016a-7ee3-409b-ba03-5dc001702368/globalmount successfully
I1111 02:48:19.774245       1 utils.go:85] GRPC response: {}
I1111 02:48:21.160281       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:48:21.160361       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="44d21287-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8a0ffac8-d2fc-4d10-acf5-5eb4cafd289d" "latency"=14923623833
I1111 02:48:21.160405       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="44d21287-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14923746833
I1111 02:48:21.760677       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/
I1111 02:48:21.760725       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1111 02:48:21.760771       1 utils.go:85] GRPC response: {}
I1111 02:48:21.766358       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 90 lines ...
I1111 02:50:01.294998       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:50:01.295013       1 utils.go:79] GRPC request: {}
I1111 02:50:01.295045       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:50:01.295837       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:50:01.295852       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef","csi.storage.k8s.io/pvc/name":"inline-volume-tester-p586b-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-3195","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef"}
I1111 02:50:01.296019       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:50:11.287653       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:50:11.287729       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="8966bc8d-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef" "latency"=9991620697
I1111 02:50:11.287754       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8966bc8d-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9991753799
I1111 02:50:12.474401       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:50:12.474421       1 utils.go:79] GRPC request: {}
I1111 02:50:12.474461       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:50:12.480156       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 37 lines ...
I1111 02:50:13.522996       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef/globalmount /var/lib/kubelet/pods/608bbc53-8ef5-4e5b-a1d1-c7922e2afb5e/volumes/kubernetes.io~csi/pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef/mount)
I1111 02:50:13.523764       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef/globalmount at /var/lib/kubelet/pods/608bbc53-8ef5-4e5b-a1d1-c7922e2afb5e/volumes/kubernetes.io~csi/pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef/mount successfully
I1111 02:50:13.523780       1 utils.go:85] GRPC response: {}
I1111 02:50:17.651137       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:50:17.651238       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:50:17.651269       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:50:17.651353       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:50:17.675511       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:50:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="932a0b5b-616b-11ed-ae2b-6045bd88ebdb"
I1111 02:50:17.683290       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 7 milliseconds
I1111 02:50:17.683520       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="932a0b5b-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8044764
I1111 02:50:17.723693       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:50:17.842117       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:50:29.198160       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:50:29.198184       1 utils.go:79] GRPC request: {}
I1111 02:50:29.198226       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:50:31.720779       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:50:31.720841       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="90119477-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-98ed955e-36b4-479d-976c-421724b1c02d" "latency"=19238797558
I1111 02:50:31.720866       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="90119477-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19238896959
I1111 02:50:32.315522       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1111 02:50:32.315572       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:50:32.315613       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-98ed955e-36b4-479d-976c-421724b1c02d/globalmount with mount options([])
I1111 02:50:32.315634       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 163 lines ...
I1111 02:51:14.918768       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:51:14.918785       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e1731f59-32e9-439e-adef-6837d3c212ca/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e1731f59-32e9-439e-adef-6837d3c212ca","csi.storage.k8s.io/pvc/name":"test.csi.azure.combjp2v","csi.storage.k8s.io/pvc/namespace":"provisioning-4376","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-e1731f59-32e9-439e-adef-6837d3c212ca"}
I1111 02:51:14.918942       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e1731f59-32e9-439e-adef-6837d3c212ca-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:51:17.652949       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:51:17.652986       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:51:17.653051       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:51:17.653102       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:51:17.676521       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:51:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b6ed7644-616b-11ed-ae2b-6045bd88ebdb"
I1111 02:51:17.682545       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:51:17.683057       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b6ed7644-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6593638
I1111 02:51:17.725505       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:51:17.843835       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:51:29.198088       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:51:29.198109       1 utils.go:79] GRPC request: {}
I1111 02:51:29.198152       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:51:34.010043       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:51:34.010105       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b548b422-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e1731f59-32e9-439e-adef-6837d3c212ca" "latency"=19091092102
I1111 02:51:34.010128       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b548b422-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19091202002
I1111 02:51:35.688508       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1111 02:51:35.688830       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:51:35.688882       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e1731f59-32e9-439e-adef-6837d3c212ca/globalmount with mount options([])
I1111 02:51:35.688897       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 66 lines ...
I1111 02:52:03.442629       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:52:03.442646       1 utils.go:79] GRPC request: {}
I1111 02:52:03.442681       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:52:03.443530       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:52:03.443546       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3e25c7e8-8885-4454-ac16-3b390acd3669","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-7fl89-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-57","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669"}
I1111 02:52:03.443735       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-3e25c7e8-8885-4454-ac16-3b390acd3669-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:52:14.831526       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:52:14.831597       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d234ffbb-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3e25c7e8-8885-4454-ac16-3b390acd3669" "latency"=11387803437
I1111 02:52:14.831631       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d234ffbb-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11387913837
I1111 02:52:15.341329       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:52:15.341352       1 utils.go:79] GRPC request: {}
I1111 02:52:15.341393       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:52:15.345158       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 13 lines ...
I1111 02:52:16.734579       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:52:16.735460       1 utils.go:78] GRPC call: /csi.v1.Node/NodePublishVolume
I1111 02:52:16.735480       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669","target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669/07458416-4d83-4cc7-84ab-bb4c024e4018","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3e25c7e8-8885-4454-ac16-3b390acd3669","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-7fl89-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-57","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669"}
I1111 02:52:17.653385       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:52:17.653422       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:52:17.653405       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:52:17.653492       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:52:17.675917       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:52:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="dab0a701-616b-11ed-ae2b-6045bd88ebdb"
I1111 02:52:17.683547       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 7 milliseconds
I1111 02:52:17.683763       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="dab0a701-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7885962
I1111 02:52:17.726869       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:52:17.845119       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:52:18.538167       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1111 02:52:18.538208       1 nodeserver_v2.go:330] NodePublishVolume [block]: found device path /dev/disk/azure/scsi1/lun1 with LUN 1
I1111 02:52:18.538235       1 nodeprovisioner.go:164] EnsureBlockTargetReady [block]: making target file /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669/07458416-4d83-4cc7-84ab-bb4c024e4018
I1111 02:52:18.538300       1 nodeserver_v2.go:348] NodePublishVolume: mounting /dev/disk/azure/scsi1/lun1 at /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669/07458416-4d83-4cc7-84ab-bb4c024e4018
I1111 02:52:18.538314       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /dev/disk/azure/scsi1/lun1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669/07458416-4d83-4cc7-84ab-bb4c024e4018)
I1111 02:52:18.539317       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /dev/disk/azure/scsi1/lun1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669/07458416-4d83-4cc7-84ab-bb4c024e4018)
I1111 02:52:18.540163       1 nodeserver_v2.go:353] NodePublishVolume: mount /dev/disk/azure/scsi1/lun1 at /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669/07458416-4d83-4cc7-84ab-bb4c024e4018 successfully
I1111 02:52:18.540183       1 utils.go:85] GRPC response: {}
I1111 02:52:25.076479       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:52:25.076577       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d94d415c-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fee094c3-c0b8-4fc3-92db-e1aa52d27d18" "latency"=9729752897
I1111 02:52:25.076628       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d94d415c-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9729889097
I1111 02:52:25.668584       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1111 02:52:25.668630       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:52:25.668658       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fee094c3-c0b8-4fc3-92db-e1aa52d27d18/globalmount with mount options([])
I1111 02:52:25.668669       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 65 lines ...
I1111 02:52:43.838317       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:52:43.838953       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:52:43.838971       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719","csi.storage.k8s.io/pvc/name":"pvc-psmfp","csi.storage.k8s.io/pvc/namespace":"snapshotting-6604","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719"}
I1111 02:52:43.839112       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:52:47.654612       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:52:47.654760       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:52:47.654863       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:52:47.654744       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:52:47.676203       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:52:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="ec92556d-616b-11ed-ae2b-6045bd88ebdb"
I1111 02:52:47.682538       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds
I1111 02:52:47.682760       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="ec92556d-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6592747
I1111 02:52:47.727929       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:52:47.845263       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 14 lines ...
W1111 02:52:53.405164       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669" is not a mountpoint, deleting
I1111 02:52:53.405238       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669 successfully
I1111 02:52:53.405253       1 utils.go:85] GRPC response: {}
I1111 02:52:59.197920       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:52:59.197945       1 utils.go:79] GRPC request: {}
I1111 02:52:59.197982       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:53:00.598540       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:53:00.598612       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="ea48d83b-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719" "latency"=16759424433
I1111 02:53:00.598643       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ea48d83b-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16759540935
I1111 02:53:01.189824       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:53:01.189847       1 utils.go:79] GRPC request: {}
I1111 02:53:01.189891       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:53:01.191493       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/
... skipping 36 lines ...
I1111 02:53:01.325347       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719/globalmount /var/lib/kubelet/pods/8a80a59a-598c-4fd2-9c73-3ce7038d93d1/volumes/kubernetes.io~csi/pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719/mount)
I1111 02:53:01.326087       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719/globalmount at /var/lib/kubelet/pods/8a80a59a-598c-4fd2-9c73-3ce7038d93d1/volumes/kubernetes.io~csi/pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719/mount successfully
I1111 02:53:01.326100       1 utils.go:85] GRPC response: {}
I1111 02:53:17.655489       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:53:17.655516       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:53:17.655574       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:53:17.655611       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:53:17.675895       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:53:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="fe73ec6a-616b-11ed-ae2b-6045bd88ebdb"
I1111 02:53:17.681444       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:53:17.682326       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="fe73ec6a-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6465151
I1111 02:53:17.728701       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:53:17.846078       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:53:27.803274       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 21 items received
I1111 02:53:27.806673       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-30725374-vmss000001&resourceVersion=26749&timeout=5m15s&timeoutSeconds=315&watch=true 200 OK in 3 milliseconds
I1111 02:53:29.197573       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:53:29.197600       1 utils.go:79] GRPC request: {}
I1111 02:53:29.197650       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:53:31.297788       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:53:31.297865       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f4a131ee-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b78f5c34-fd0f-4e65-be44-ef642570638f" "latency"=30102456566
I1111 02:53:31.297901       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f4a131ee-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=30102565567
I1111 02:53:31.957538       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1111 02:53:31.957590       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:53:31.957619       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b78f5c34-fd0f-4e65-be44-ef642570638f/globalmount with mount options([nouuid])
I1111 02:53:31.957631       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 135 lines ...
I1111 02:53:55.166591       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:53:55.166604       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3ba666f2-fc79-47b2-8e01-9da74ac598e4/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3ba666f2-fc79-47b2-8e01-9da74ac598e4","csi.storage.k8s.io/pvc/name":"test.csi.azure.com6jt2w-cloned","csi.storage.k8s.io/pvc/namespace":"multivolume-8159","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-3ba666f2-fc79-47b2-8e01-9da74ac598e4"}
I1111 02:53:55.166785       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-3ba666f2-fc79-47b2-8e01-9da74ac598e4-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:53:59.197353       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:53:59.197377       1 utils.go:79] GRPC request: {}
I1111 02:53:59.197424       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:54:11.826122       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:54:11.826194       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="14cc9563-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3ba666f2-fc79-47b2-8e01-9da74ac598e4" "latency"=16659333518
I1111 02:54:11.826226       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="14cc9563-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16659452619
I1111 02:54:12.438084       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1111 02:54:12.438134       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:54:12.438163       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3ba666f2-fc79-47b2-8e01-9da74ac598e4/globalmount with mount options([nouuid])
I1111 02:54:12.438181       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 36 lines ...
I1111 02:54:12.894071       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:54:12.894084       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-740cfa70-55fa-4647-975f-052c29c541b6/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-740cfa70-55fa-4647-975f-052c29c541b6","csi.storage.k8s.io/pvc/name":"test.csi.azure.comnnhwr","csi.storage.k8s.io/pvc/namespace":"multivolume-7367","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-740cfa70-55fa-4647-975f-052c29c541b6"}
I1111 02:54:12.894248       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-740cfa70-55fa-4647-975f-052c29c541b6-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:54:17.657075       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:54:17.657161       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:54:17.657182       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:54:17.657240       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:54:17.676414       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:54:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="223746f5-616c-11ed-ae2b-6045bd88ebdb"
I1111 02:54:17.681436       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds
I1111 02:54:17.681637       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="223746f5-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5254842
I1111 02:54:17.730633       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:54:17.848142       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:54:18.425976       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
... skipping 31 lines ...
I1111 02:54:21.757252       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b78f5c34-fd0f-4e65-be44-ef642570638f/globalmount
I1111 02:54:21.757282       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b78f5c34-fd0f-4e65-be44-ef642570638f/globalmount" is a mountpoint, unmounting
I1111 02:54:21.757299       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b78f5c34-fd0f-4e65-be44-ef642570638f/globalmount
W1111 02:54:21.767838       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b78f5c34-fd0f-4e65-be44-ef642570638f/globalmount" is not a mountpoint, deleting
I1111 02:54:21.767914       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b78f5c34-fd0f-4e65-be44-ef642570638f/globalmount successfully
I1111 02:54:21.767929       1 utils.go:85] GRPC response: {}
I1111 02:54:24.525426       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:54:24.525502       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1f5d9482-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-740cfa70-55fa-4647-975f-052c29c541b6" "latency"=11631171048
I1111 02:54:24.525537       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1f5d9482-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11631301350
I1111 02:54:25.138312       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1111 02:54:25.138360       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1111 02:54:25.138389       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-740cfa70-55fa-4647-975f-052c29c541b6/globalmount with mount options([nouuid])
I1111 02:54:25.138430       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 140 lines ...
I1111 02:55:59.198339       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:55:59.198367       1 utils.go:79] GRPC request: {}
I1111 02:55:59.198414       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:56:17.658617       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:56:17.658655       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:56:17.658711       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:56:17.658730       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:56:17.658763       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:56:17.676166       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:56:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="69bdc888-616c-11ed-ae2b-6045bd88ebdb"
I1111 02:56:17.682222       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds
I1111 02:56:17.683690       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="69bdc888-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7531659
I1111 02:56:17.733067       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:56:17.851459       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:56:20.941438       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:56:20.941514       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5dacd066-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-52bced0b-008e-49ff-af52-1a26d04fa440" "latency"=23509166838
I1111 02:56:20.941554       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5dacd066-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=23509311040
I1111 02:56:20.944922       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:56:20.944992       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5dacd30f-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1c0f985f-487a-4c3d-b317-db9166d07e68" "latency"=23512603665
I1111 02:56:20.945041       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5dacd30f-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=23512727865
I1111 02:56:21.527299       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1111 02:56:21.527350       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:56:21.527305       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/
I1111 02:56:21.527378       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1c0f985f-487a-4c3d-b317-db9166d07e68/globalmount with mount options([])
... skipping 148 lines ...
I1111 02:57:49.742174       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:57:49.742193       1 utils.go:79] GRPC request: {}
I1111 02:57:49.742223       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:57:49.742875       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:57:49.742892       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-db5d0648-7c4f-424b-8318-0c2c1ef85268","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-db5d0648-7c4f-424b-8318-0c2c1ef85268","csi.storage.k8s.io/pvc/name":"test.csi.azure.comd9fxt","csi.storage.k8s.io/pvc/namespace":"multivolume-1811","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-db5d0648-7c4f-424b-8318-0c2c1ef85268"}
I1111 02:57:49.743064       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-db5d0648-7c4f-424b-8318-0c2c1ef85268-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:57:54.114569       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:57:54.114642       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a09e1567-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-db5d0648-7c4f-424b-8318-0c2c1ef85268" "latency"=4371485793
I1111 02:57:54.114675       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a09e1567-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4371618795
I1111 02:57:55.834047       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1111 02:57:55.834094       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:57:55.834112       1 utils.go:85] GRPC response: {}
I1111 02:57:55.845058       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 83 lines ...
I1111 02:59:11.238739       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:59:11.238755       1 utils.go:79] GRPC request: {}
I1111 02:59:11.238791       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:59:11.239545       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:59:11.239561       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1e3dc67b-5da5-448f-a098-30547a547891/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1e3dc67b-5da5-448f-a098-30547a547891","csi.storage.k8s.io/pvc/name":"test.csi.azure.combnndp","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-2932","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-1e3dc67b-5da5-448f-a098-30547a547891"}
I1111 02:59:11.239727       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-1e3dc67b-5da5-448f-a098-30547a547891-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 02:59:15.221824       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:59:15.221900       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d1317cd7-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1e3dc67b-5da5-448f-a098-30547a547891" "latency"=3982114445
I1111 02:59:15.221928       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d1317cd7-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3982211547
I1111 02:59:16.909709       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1111 02:59:16.909759       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:59:16.909788       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1e3dc67b-5da5-448f-a098-30547a547891/globalmount with mount options([])
I1111 02:59:16.909797       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 169 lines ...
I1111 03:00:34.834385       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 03:00:34.834403       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-645c9610-0f87-46c0-afce-8091ae8a2a72/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-645c9610-0f87-46c0-afce-8091ae8a2a72","csi.storage.k8s.io/pvc/name":"test.csi.azure.comv7sqt","csi.storage.k8s.io/pvc/namespace":"multivolume-1004","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-645c9610-0f87-46c0-afce-8091ae8a2a72"}
I1111 03:00:34.834561       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 03:00:34.834605       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-645c9610-0f87-46c0-afce-8091ae8a2a72-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 03:00:34.834571       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-77f3f92f-9208-4a3f-8daa-d489e3800757/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-77f3f92f-9208-4a3f-8daa-d489e3800757","csi.storage.k8s.io/pvc/name":"test.csi.azure.com5jgjc","csi.storage.k8s.io/pvc/namespace":"multivolume-1004","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-77f3f92f-9208-4a3f-8daa-d489e3800757"}
I1111 03:00:34.834732       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-77f3f92f-9208-4a3f-8daa-d489e3800757-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 03:00:44.562890       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 03:00:44.562951       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="03050d50-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-645c9610-0f87-46c0-afce-8091ae8a2a72" "latency"=9728274011
I1111 03:00:44.562971       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="03050d50-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9728394611
I1111 03:00:44.568327       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 03:00:44.568399       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="030512a0-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-77f3f92f-9208-4a3f-8daa-d489e3800757" "latency"=9733603358
I1111 03:00:44.568438       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="030512a0-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9733716259
I1111 03:00:45.145814       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1111 03:00:45.145869       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 03:00:45.145906       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-77f3f92f-9208-4a3f-8daa-d489e3800757/globalmount with mount options([])
I1111 03:00:45.145917       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 163 lines ...
I1111 03:02:56.526887       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 03:02:56.526904       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1beddae0-e39c-412c-b950-597e7ac0d449/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1beddae0-e39c-412c-b950-597e7ac0d449","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-1beddae0-e39c-412c-b950-597e7ac0d449"}
I1111 03:02:56.527101       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-1beddae0-e39c-412c-b950-597e7ac0d449-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 03:02:59.197838       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 03:02:59.197865       1 utils.go:79] GRPC request: {}
I1111 03:02:59.197910       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 03:03:00.210517       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 03:03:00.210582       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="57799c70-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1beddae0-e39c-412c-b950-597e7ac0d449" "latency"=3683389763
I1111 03:03:00.210613       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="57799c70-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3683531864
I1111 03:03:01.895244       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1111 03:03:01.895308       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS
I1111 03:03:01.895337       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1beddae0-e39c-412c-b950-597e7ac0d449/globalmount with mount options([])
I1111 03:03:01.895355       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 62 lines ...
I1111 03:03:57.738974       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 03:03:57.738989       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-29f57298-87bb-4966-8af3-4495005617ba/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-29f57298-87bb-4966-8af3-4495005617ba","csi.storage.k8s.io/pvc/name":"nginx-azuredisk-ephemeral-azuredisk01","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-29f57298-87bb-4966-8af3-4495005617ba"}
I1111 03:03:57.739156       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-29f57298-87bb-4966-8af3-4495005617ba-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 03:03:59.197698       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 03:03:59.197722       1 utils.go:79] GRPC request: {}
I1111 03:03:59.197766       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 03:04:07.793887       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 03:04:07.793958       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7bf5d4bb-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-29f57298-87bb-4966-8af3-4495005617ba" "latency"=10054739161
I1111 03:04:07.793988       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7bf5d4bb-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10054843662
I1111 03:04:08.384698       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1111 03:04:08.384751       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS
I1111 03:04:08.384783       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-29f57298-87bb-4966-8af3-4495005617ba/globalmount with mount options([])
I1111 03:04:08.384802       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 42 lines ...
I1111 03:04:17.782401       1 utils.go:79] GRPC request: {}
I1111 03:04:17.782431       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 03:04:17.783263       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 03:04:17.783280       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cdaad550-6d6a-492a-a447-eaf824cd5686/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-cdaad550-6d6a-492a-a447-eaf824cd5686","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-hh9gg-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-cdaad550-6d6a-492a-a447-eaf824cd5686"}
I1111 03:04:17.783444       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-cdaad550-6d6a-492a-a447-eaf824cd5686-k8s-agentpool1-30725374-vmss000001-attachment)
I1111 03:04:17.861572       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 03:04:27.692091       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 03:04:27.692152       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="87e8589b-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-cdaad550-6d6a-492a-a447-eaf824cd5686" "latency"=9908619871
I1111 03:04:27.692184       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="87e8589b-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9908750872
I1111 03:04:28.291768       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1111 03:04:28.291818       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType StandardSSD_LRS
I1111 03:04:28.291850       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cdaad550-6d6a-492a-a447-eaf824cd5686/globalmount with mount options([])
I1111 03:04:28.291860       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 143 lines ...
I1111 02:06:43.300895       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=2523&timeout=9m44s&timeoutSeconds=584&watch=true 200 OK in 1 milliseconds
I1111 02:06:43.301758       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2332&timeout=6m22s&timeoutSeconds=382&watch=true 200 OK in 0 milliseconds
I1111 02:06:43.391085       1 shared_informer.go:285] caches populated
I1111 02:06:43.391111       1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 e2e-test
I1111 02:06:43.391135       1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider
I1111 02:06:43.394192       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I1111 02:06:43.394460       1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I1111 02:06:43.394479       1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider
I1111 02:06:43.394488       1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I1111 02:06:43.394509       1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully
I1111 02:06:43.395201       1 azure_auth.go:245] Using AzurePublicCloud environment
I1111 02:06:43.395247       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I1111 02:06:43.395270       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
... skipping 103 lines ...
I1111 02:07:36.571411       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-ea425b32-c65a-4964-9a98-97164117dec3-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:07:36.571516       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:07:40.322007       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:07:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="9edcdd77-6165-11ed-9710-6045bd88e3ef"
I1111 02:07:40.330148       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 7 milliseconds
I1111 02:07:40.330368       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="9edcdd77-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8417184
I1111 02:07:43.085071       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:07:43.085171       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:07:43.085213       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:07:43.158495       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:07:43.239442       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:07:43.301981       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:07:43.418162       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:07:59.222991       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:07:59.223010       1 utils.go:79] GRPC request: {}
I1111 02:07:59.223050       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:08:10.329736       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:08:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="b0bfaf02-6165-11ed-9710-6045bd88e3ef"
I1111 02:08:10.340166       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 10 milliseconds
I1111 02:08:10.340365       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="b0bfaf02-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10661832
I1111 02:08:13.087043       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:13.087208       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:08:13.087266       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:08:13.159295       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:13.240571       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:13.302096       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:13.418747       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:29.224994       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:08:29.225020       1 utils.go:79] GRPC request: {}
I1111 02:08:29.225071       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:08:40.326722       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:08:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="c2a0dc87-6165-11ed-9710-6045bd88e3ef"
I1111 02:08:40.335404       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 8 milliseconds
I1111 02:08:40.335574       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="c2a0dc87-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8873293
I1111 02:08:43.087472       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:43.087800       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:08:43.087924       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:08:43.159973       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:43.241352       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:43.302430       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:43.418906       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:08:59.223353       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:08:59.223373       1 utils.go:79] GRPC request: {}
I1111 02:08:59.223447       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:09:10.319122       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:09:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="d48154bb-6165-11ed-9710-6045bd88e3ef"
I1111 02:09:10.337125       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 17 milliseconds
I1111 02:09:10.337549       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="d48154bb-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18506304
I1111 02:09:13.087903       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:13.088129       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:09:13.088160       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:09:13.160529       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:13.241825       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:13.303254       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:13.419667       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:29.222979       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:09:29.223002       1 utils.go:79] GRPC request: {}
I1111 02:09:29.223049       1 utils.go:85] GRPC response: {"ready":{"value":true}}
E1111 02:09:36.565388       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="9ca0936c-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ea425b32-c65a-4964-9a98-97164117dec3" "latency"=119993476327
E1111 02:09:36.565565       1 crdprovisioner.go:760]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9ca0936c-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119994114040
E1111 02:09:36.565389       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="9ca0973d-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d" "latency"=119993532828
E1111 02:09:36.565587       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-ea425b32-c65a-4964-9a98-97164117dec3) to be attached to node (k8s-agentpool1-30725374-vmss000002): context canceled
E1111 02:09:36.565636       1 crdprovisioner.go:760]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9ca0973d-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119994124340
E1111 02:09:36.565669       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d) to be attached to node (k8s-agentpool1-30725374-vmss000002): context deadline exceeded
I1111 02:09:37.162067       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:09:37.162093       1 utils.go:79] GRPC request: {}
I1111 02:09:37.162136       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:09:37.162146       1 utils.go:79] GRPC request: {}
I1111 02:09:37.162138       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:09:37.162190       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 16 lines ...
I1111 02:09:37.167009       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d","csi.storage.k8s.io/pvc/name":"test.csi.azure.coms6mwg","csi.storage.k8s.io/pvc/namespace":"multivolume-5754","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d"}
I1111 02:09:37.167162       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:09:40.321861       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:09:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="e66363b7-6165-11ed-9710-6045bd88e3ef"
I1111 02:09:40.329717       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 7 milliseconds
I1111 02:09:40.329944       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="e66363b7-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8138271
I1111 02:09:43.090328       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:43.090452       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:09:43.090475       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:09:43.160759       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:43.242796       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:43.304091       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:43.420446       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:09:59.223193       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:09:59.223232       1 utils.go:79] GRPC request: {}
I1111 02:09:59.223281       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:10:10.326088       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:10:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="f845ac8d-6165-11ed-9710-6045bd88e3ef"
I1111 02:10:10.333498       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 7 milliseconds
I1111 02:10:10.333794       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="f845ac8d-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7732768
I1111 02:10:13.092505       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:13.092627       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:10:13.092683       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:10:13.160941       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:13.243294       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:13.304627       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:13.421033       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:29.222916       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:10:29.222941       1 utils.go:79] GRPC request: {}
I1111 02:10:29.222988       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:10:40.323396       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:10:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="0a26e677-6166-11ed-9710-6045bd88e3ef"
I1111 02:10:40.331732       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 8 milliseconds
I1111 02:10:40.331919       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="0a26e677-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8553185
I1111 02:10:43.095310       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:43.095611       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:10:43.095686       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:10:43.161573       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:43.243834       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:43.305262       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:43.421829       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:10:59.223591       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:10:59.223612       1 utils.go:79] GRPC request: {}
I1111 02:10:59.223662       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:11:10.326446       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:11:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1c0900e3-6166-11ed-9710-6045bd88e3ef"
I1111 02:11:10.334295       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 7 milliseconds
I1111 02:11:10.334448       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1c0900e3-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8031575
I1111 02:11:13.098204       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:11:13.098327       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:11:13.098341       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:11:13.162751       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:11:13.243945       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:11:13.306243       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:11:13.422721       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:11:29.223196       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:11:29.223233       1 utils.go:79] GRPC request: {}
I1111 02:11:29.223283       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:11:32.719657       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:11:32.719723       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="e4820202-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ea425b32-c65a-4964-9a98-97164117dec3" "latency"=115552610284
I1111 02:11:32.719759       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e4820202-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=115552739286
I1111 02:11:32.719771       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:11:32.719846       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="e4820740-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d" "latency"=115552621585
I1111 02:11:32.719877       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e4820740-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=115552723887
I1111 02:11:33.311953       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1111 02:11:33.312014       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:11:33.312045       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ea425b32-c65a-4964-9a98-97164117dec3/globalmount with mount options([nouuid])
I1111 02:11:33.312067       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 266 lines ...
I1111 02:12:42.398657       1 utils.go:79] GRPC request: {}
I1111 02:12:42.398689       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:12:42.399414       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:12:42.399429       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e8718e1c-528b-4fd6-a400-43aa6a13394f/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e8718e1c-528b-4fd6-a400-43aa6a13394f","csi.storage.k8s.io/pvc/name":"test.csi.azure.com6k9wb","csi.storage.k8s.io/pvc/namespace":"multivolume-7685","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-e8718e1c-528b-4fd6-a400-43aa6a13394f"}
I1111 02:12:42.399599       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e8718e1c-528b-4fd6-a400-43aa6a13394f-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:12:43.102293       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:12:43.102417       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:12:43.102484       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:12:43.164662       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:12:43.245941       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:12:43.307381       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:12:43.423793       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:12:59.223293       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:12:59.223311       1 utils.go:79] GRPC request: {}
I1111 02:12:59.223349       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:13:05.302389       1 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: Watch close - *v1.CustomResourceDefinition total 0 items received
I1111 02:13:05.304753       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2332&timeout=6m20s&timeoutSeconds=380&watch=true 200 OK in 2 milliseconds
I1111 02:13:07.254435       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:13:07.254501       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="52bc2568-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3337fe34-dd9f-4493-8a58-4f4fa00d239b" "latency"=25157018542
I1111 02:13:07.254524       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="52bc2568-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=25157119844
I1111 02:13:07.262631       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:13:07.262680       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="52ea40d4-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e8718e1c-528b-4fd6-a400-43aa6a13394f" "latency"=24863031150
I1111 02:13:07.262704       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="52ea40d4-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24863129652
I1111 02:13:07.853579       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1111 02:13:07.853626       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:13:07.853653       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e8718e1c-528b-4fd6-a400-43aa6a13394f/globalmount with mount options([nouuid])
I1111 02:13:07.853666       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 119 lines ...
I1111 02:13:33.380220       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e7215e08-5df4-4a62-9969-7a3c177b9e40/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext3"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e7215e08-5df4-4a62-9969-7a3c177b9e40","csi.storage.k8s.io/pvc/name":"test.csi.azure.comn4n42","csi.storage.k8s.io/pvc/namespace":"volume-6500","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-e7215e08-5df4-4a62-9969-7a3c177b9e40"}
I1111 02:13:33.380379       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e7215e08-5df4-4a62-9969-7a3c177b9e40-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:13:40.320209       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:13:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="7570393f-6166-11ed-9710-6045bd88e3ef"
I1111 02:13:40.328033       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 7 milliseconds
I1111 02:13:40.328336       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="7570393f-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8234285
I1111 02:13:43.105229       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:13:43.105488       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:13:43.165581       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:13:43.246890       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:13:43.308316       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:13:43.424691       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:13:46.426301       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:13:46.426376       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="714d4df2-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e7215e08-5df4-4a62-9969-7a3c177b9e40" "latency"=13045933037
I1111 02:13:46.426406       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="714d4df2-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=13046038440
I1111 02:13:47.024198       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1111 02:13:47.024247       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1111 02:13:47.024275       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e7215e08-5df4-4a62-9969-7a3c177b9e40/globalmount with mount options([])
I1111 02:13:47.024283       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 62 lines ...
I1111 02:14:10.324282       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:14:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="87527c9e-6166-11ed-9710-6045bd88e3ef"
I1111 02:14:10.333738       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 9 milliseconds
I1111 02:14:10.333912       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="87527c9e-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9699413
I1111 02:14:12.419349       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 26 items received
I1111 02:14:12.425273       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-30725374-vmss000002&resourceVersion=5743&timeout=6m23s&timeoutSeconds=383&watch=true 200 OK in 5 milliseconds
I1111 02:14:13.106090       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:14:13.106248       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:14:13.159336       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 75 items received
I1111 02:14:13.162081       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=5743&timeout=7m23s&timeoutSeconds=443&watch=true 200 OK in 2 milliseconds
I1111 02:14:13.166427       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:14:13.247764       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:14:13.309026       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:14:13.425287       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:14:29.222810       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:14:29.222832       1 utils.go:79] GRPC request: {}
I1111 02:14:29.222880       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:14:40.321004       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:14:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="9933a079-6166-11ed-9710-6045bd88e3ef"
I1111 02:14:40.327232       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 5 milliseconds
I1111 02:14:40.327435       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="9933a079-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6479541
I1111 02:14:43.106365       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:14:43.106462       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:14:43.167033       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:14:43.248349       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:14:43.309701       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:14:43.426244       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:14:57.386712       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:14:57.386772       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="83b5cd8d-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-98672851-fd48-4b89-9310-e3e3f222e658" "latency"=53122505119
I1111 02:14:57.386793       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="83b5cd8d-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=53122602220
I1111 02:14:57.994043       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/
I1111 02:14:57.994090       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1111 02:14:57.994117       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-98672851-fd48-4b89-9310-e3e3f222e658/globalmount with mount options([])
I1111 02:14:57.994126       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3])
... skipping 128 lines ...
I1111 02:18:00.598857       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a3d9ddeb-2aba-4efa-a438-b77ee53d5936/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a3d9ddeb-2aba-4efa-a438-b77ee53d5936","csi.storage.k8s.io/pvc/name":"test.csi.azure.commhdpp","csi.storage.k8s.io/pvc/namespace":"provisioning-3283","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-a3d9ddeb-2aba-4efa-a438-b77ee53d5936"}
I1111 02:18:00.599036       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a3d9ddeb-2aba-4efa-a438-b77ee53d5936-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:18:10.319836       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:18:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="165ee779-6167-11ed-9710-6045bd88e3ef"
I1111 02:18:10.328306       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 8 milliseconds
I1111 02:18:10.328941       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="165ee779-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9100299
I1111 02:18:13.109870       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:18:13.110180       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:18:13.172173       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:18:13.254533       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:18:13.316088       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:18:13.432535       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:18:24.818396       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:18:24.818468       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1093a2a4-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a3d9ddeb-2aba-4efa-a438-b77ee53d5936" "latency"=24219373128
I1111 02:18:24.818497       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1093a2a4-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24219478830
I1111 02:18:26.539430       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1111 02:18:26.539483       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:18:26.539513       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a3d9ddeb-2aba-4efa-a438-b77ee53d5936/globalmount with mount options([])
I1111 02:18:26.539523       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 125 lines ...
I1111 02:21:14.709929       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:21:14.710635       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:21:14.710649       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a42f45dd-17b3-4305-8325-781135ac7133/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a42f45dd-17b3-4305-8325-781135ac7133","csi.storage.k8s.io/pvc/name":"test.csi.azure.comxjsrj","csi.storage.k8s.io/pvc/namespace":"multivolume-6620","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-a42f45dd-17b3-4305-8325-781135ac7133"}
I1111 02:21:14.710828       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a42f45dd-17b3-4305-8325-781135ac7133-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:21:16.085586       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 311 items received
I1111 02:21:16.093447       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=9697&timeout=5m31s&timeoutSeconds=331&watch=true 200 OK in 7 milliseconds
I1111 02:21:28.634056       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:21:28.634124       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="8446be2f-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a42f45dd-17b3-4305-8325-781135ac7133" "latency"=13923233041
I1111 02:21:28.634161       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8446be2f-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=13923346444
I1111 02:21:29.222822       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:21:29.222846       1 utils.go:79] GRPC request: {}
I1111 02:21:29.222893       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:21:30.362473       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
... skipping 88 lines ...
I1111 02:22:16.031785       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-aaa672b4-088a-40de-bf83-32888502d696-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:22:29.224687       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:22:29.224712       1 utils.go:79] GRPC request: {}
I1111 02:22:29.224760       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:22:31.243708       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 207 items received
I1111 02:22:31.246889       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=10423&timeout=9m53s&timeoutSeconds=593&watch=true 200 OK in 2 milliseconds
I1111 02:22:32.349762       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:22:32.349837       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="a8d3948d-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-aaa672b4-088a-40de-bf83-32888502d696" "latency"=16317989526
I1111 02:22:32.349867       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a8d3948d-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16318091929
I1111 02:22:34.068122       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1111 02:22:34.068166       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:22:34.068197       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-aaa672b4-088a-40de-bf83-32888502d696/globalmount with mount options([])
I1111 02:22:34.068207       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 48 lines ...
I1111 02:22:46.859071       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:22:46.859085       1 utils.go:79] GRPC request: {}
I1111 02:22:46.859115       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:22:46.859888       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:22:46.859905       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cae9a0f6-2ca6-41b2-b348-493a8f4daa82/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-cae9a0f6-2ca6-41b2-b348-493a8f4daa82","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-2wzwm-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-6566","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-cae9a0f6-2ca6-41b2-b348-493a8f4daa82"}
I1111 02:22:46.860085       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-cae9a0f6-2ca6-41b2-b348-493a8f4daa82-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:22:52.290793       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:22:52.290861       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="bb339a5d-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-cae9a0f6-2ca6-41b2-b348-493a8f4daa82" "latency"=5430710235
I1111 02:22:52.290886       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bb339a5d-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5430832638
I1111 02:22:52.900154       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1111 02:22:52.900202       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:22:52.900238       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cae9a0f6-2ca6-41b2-b348-493a8f4daa82/globalmount with mount options([])
I1111 02:22:52.900252       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 153 lines ...
I1111 02:25:30.708508       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-65b99319-536b-4adb-acd1-8410dc77d4c5","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-65b99319-536b-4adb-acd1-8410dc77d4c5","csi.storage.k8s.io/pvc/name":"test.csi.azure.comhdrpr","csi.storage.k8s.io/pvc/namespace":"multivolume-8220","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-65b99319-536b-4adb-acd1-8410dc77d4c5"}
I1111 02:25:30.708729       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-65b99319-536b-4adb-acd1-8410dc77d4c5-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:25:40.321313       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:25:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="2297aeb2-6168-11ed-9710-6045bd88e3ef"
I1111 02:25:40.329460       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 7 milliseconds
I1111 02:25:40.329690       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="2297aeb2-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8420785
I1111 02:25:43.126045       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:25:43.126210       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:25:43.126273       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:25:43.181364       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:25:43.265576       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:25:43.326002       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:25:43.443407       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:25:59.223417       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:25:59.223439       1 utils.go:79] GRPC request: {}
I1111 02:25:59.223481       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:26:10.326425       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:26:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="347a19f2-6168-11ed-9710-6045bd88e3ef"
I1111 02:26:10.337808       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 11 milliseconds
I1111 02:26:10.338018       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="347a19f2-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11621653
I1111 02:26:13.126740       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:13.126906       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:26:13.126938       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:26:13.182052       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:13.266348       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:13.326860       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:13.444430       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:29.222953       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:26:29.222977       1 utils.go:79] GRPC request: {}
I1111 02:26:29.223027       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:26:40.321362       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:26:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="465af610-6168-11ed-9710-6045bd88e3ef"
I1111 02:26:40.329830       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 8 milliseconds
I1111 02:26:40.330059       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="465af610-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8754891
I1111 02:26:43.126983       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:43.127169       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:26:43.127280       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:26:43.182176       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:43.267487       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:43.327792       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:43.445243       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:26:47.094388       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 200 items received
I1111 02:26:47.103549       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=12534&timeout=6m42s&timeoutSeconds=402&watch=true 200 OK in 8 milliseconds
I1111 02:26:59.223301       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:26:59.223326       1 utils.go:79] GRPC request: {}
I1111 02:26:59.223363       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:27:10.329034       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:27:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="583dc5f2-6168-11ed-9710-6045bd88e3ef"
I1111 02:27:10.336000       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 6 milliseconds
I1111 02:27:10.336200       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="583dc5f2-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7193757
I1111 02:27:13.128016       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:13.128350       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:27:13.128395       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:27:13.182299       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:13.267705       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:13.328084       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:13.445380       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:24.430287       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 22 items received
I1111 02:27:24.433866       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-30725374-vmss000002&resourceVersion=12978&timeout=7m19s&timeoutSeconds=439&watch=true 200 OK in 3 milliseconds
I1111 02:27:29.223721       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:27:29.223746       1 utils.go:79] GRPC request: {}
I1111 02:27:29.223782       1 utils.go:85] GRPC response: {"ready":{"value":true}}
E1111 02:27:30.501404       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1cbe22da-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-09de6e98-ef97-4118-9359-0dcf58025169" "latency"=119994337430
E1111 02:27:30.501465       1 crdprovisioner.go:760]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1cbe22da-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119994513434
E1111 02:27:30.501483       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-09de6e98-ef97-4118-9359-0dcf58025169) to be attached to node (k8s-agentpool1-30725374-vmss000002): context canceled
E1111 02:27:30.706013       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1cdceca8-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-65b99319-536b-4adb-acd1-8410dc77d4c5" "latency"=119997140092
E1111 02:27:30.706095       1 crdprovisioner.go:760]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1cdceca8-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119997378297
E1111 02:27:30.706113       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-65b99319-536b-4adb-acd1-8410dc77d4c5) to be attached to node (k8s-agentpool1-30725374-vmss000002): context deadline exceeded
I1111 02:27:31.091833       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:27:31.091855       1 utils.go:79] GRPC request: {}
I1111 02:27:31.091902       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:27:31.094866       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:27:31.094890       1 utils.go:79] GRPC request: {}
I1111 02:27:31.094936       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 13 lines ...
I1111 02:27:31.296685       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-65b99319-536b-4adb-acd1-8410dc77d4c5","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-65b99319-536b-4adb-acd1-8410dc77d4c5","csi.storage.k8s.io/pvc/name":"test.csi.azure.comhdrpr","csi.storage.k8s.io/pvc/namespace":"multivolume-8220","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-65b99319-536b-4adb-acd1-8410dc77d4c5"}
I1111 02:27:31.296851       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-65b99319-536b-4adb-acd1-8410dc77d4c5-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:27:40.320999       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:27:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="6a1e2e2a-6168-11ed-9710-6045bd88e3ef"
I1111 02:27:40.328169       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 6 milliseconds
I1111 02:27:40.328438       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="6a1e2e2a-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7487063
I1111 02:27:43.131071       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:43.131191       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:27:43.131205       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:27:43.182507       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:43.268855       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:43.329259       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:43.445672       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:27:59.223625       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:27:59.223653       1 utils.go:79] GRPC request: {}
I1111 02:27:59.223702       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:28:10.326706       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:28:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="7c00b0ec-6168-11ed-9710-6045bd88e3ef"
I1111 02:28:10.333882       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 7 milliseconds
I1111 02:28:10.334461       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="7c00b0ec-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7777569
I1111 02:28:13.131519       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:28:13.131630       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:28:13.131644       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:28:13.182798       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:28:13.269287       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:28:13.329712       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:28:13.446295       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:28:16.303744       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:28:16.303814       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="64bd35c6-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-65b99319-536b-4adb-acd1-8410dc77d4c5" "latency"=45006901273
I1111 02:28:16.303839       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="64bd35c6-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=45007013376
I1111 02:28:16.306273       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:28:16.306335       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="649eb14f-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-09de6e98-ef97-4118-9359-0dcf58025169" "latency"=45209430088
I1111 02:28:16.306367       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="649eb14f-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=45209534291
I1111 02:28:16.908805       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1111 02:28:16.908861       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:28:16.908899       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-09de6e98-ef97-4118-9359-0dcf58025169/globalmount with mount options([])
I1111 02:28:16.908919       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 189 lines ...
I1111 02:32:40.068042       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-06727e11-d7e2-4848-b27e-049c575ae4b3/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-06727e11-d7e2-4848-b27e-049c575ae4b3","csi.storage.k8s.io/pvc/name":"pvc-lsnxb","csi.storage.k8s.io/pvc/namespace":"provisioning-3371","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-06727e11-d7e2-4848-b27e-049c575ae4b3"}
I1111 02:32:40.068202       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-06727e11-d7e2-4848-b27e-049c575ae4b3-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:32:40.319581       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:32:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1cee5550-6169-11ed-9710-6045bd88e3ef"
I1111 02:32:40.325103       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 5 milliseconds
I1111 02:32:40.325317       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1cee5550-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5769326
I1111 02:32:43.139292       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:32:43.139389       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:32:43.189747       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:32:43.276144       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:32:43.335766       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:32:43.453085       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:32:59.222577       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:32:59.222598       1 utils.go:79] GRPC request: {}
I1111 02:32:59.222644       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:32:59.630858       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:32:59.630941       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1cc7fb1f-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-06727e11-d7e2-4848-b27e-049c575ae4b3" "latency"=19562669815
I1111 02:32:59.630965       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1cc7fb1f-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19562777918
I1111 02:33:01.329684       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1111 02:33:01.329731       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:33:01.329758       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-06727e11-d7e2-4848-b27e-049c575ae4b3/globalmount with mount options([])
I1111 02:33:01.329767       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 86 lines ...
I1111 02:33:50.265685       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:33:50.265760       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:33:50.265701       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c","csi.storage.k8s.io/pvc/name":"test.csi.azure.com2w94g","csi.storage.k8s.io/pvc/namespace":"multivolume-5107","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c"}
I1111 02:33:50.265771       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c4cfb065-dbf9-4356-ac29-6e2c98a476c6/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c4cfb065-dbf9-4356-ac29-6e2c98a476c6","csi.storage.k8s.io/pvc/name":"test.csi.azure.comnnjhh","csi.storage.k8s.io/pvc/namespace":"multivolume-5107","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-c4cfb065-dbf9-4356-ac29-6e2c98a476c6"}
I1111 02:33:50.265853       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:33:50.265897       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-c4cfb065-dbf9-4356-ac29-6e2c98a476c6-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:33:58.446198       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:33:58.446261       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="469f4ca8-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c4cfb065-dbf9-4356-ac29-6e2c98a476c6" "latency"=8180314923
I1111 02:33:58.446283       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="469f4ca8-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8180398424
I1111 02:33:58.446528       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:33:58.446609       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="469f4ad4-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c" "latency"=8180678930
I1111 02:33:58.446644       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="469f4ad4-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8180804133
I1111 02:33:58.832316       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:33:58.832337       1 utils.go:79] GRPC request: {}
I1111 02:33:58.832373       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:33:58.835736       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 111 lines ...
I1111 02:34:12.137565       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c4cfb065-dbf9-4356-ac29-6e2c98a476c6/globalmount successfully
I1111 02:34:12.137579       1 utils.go:85] GRPC response: {}
W1111 02:34:12.138833       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c/globalmount" is not a mountpoint, deleting
I1111 02:34:12.138881       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c/globalmount successfully
I1111 02:34:12.138889       1 utils.go:85] GRPC response: {}
I1111 02:34:13.141425       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:34:13.141553       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:34:13.191684       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:34:13.278074       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:34:13.337395       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:34:13.454834       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:34:13.862789       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:34:13.862909       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="4bbb3c27-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-eacb38a6-1de7-4c1b-b1ec-64badef707a4" "latency"=15025234167
I1111 02:34:13.862960       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4bbb3c27-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15025369170
I1111 02:34:14.473964       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1111 02:34:14.474012       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1111 02:34:14.474044       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-eacb38a6-1de7-4c1b-b1ec-64badef707a4/globalmount with mount options([])
I1111 02:34:14.474054       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 57 lines ...
W1111 02:34:27.357906       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-eacb38a6-1de7-4c1b-b1ec-64badef707a4/globalmount" is not a mountpoint, deleting
I1111 02:34:27.357982       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-eacb38a6-1de7-4c1b-b1ec-64badef707a4/globalmount successfully
I1111 02:34:27.357998       1 utils.go:85] GRPC response: {}
I1111 02:34:29.223792       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:34:29.223815       1 utils.go:79] GRPC request: {}
I1111 02:34:29.223857       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:34:31.717899       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:34:31.717957       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="59986e7d-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a" "latency"=9620292013
I1111 02:34:31.717981       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="59986e7d-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9620393615
I1111 02:34:32.315771       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/
I1111 02:34:32.315822       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1111 02:34:32.315853       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a/globalmount with mount options([])
I1111 02:34:32.315862       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3])
... skipping 173 lines ...
I1111 02:39:29.223296       1 utils.go:79] GRPC request: {}
I1111 02:39:29.223343       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:39:40.324869       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:39:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="17460aef-616a-11ed-9710-6045bd88e3ef"
I1111 02:39:40.333132       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 8 milliseconds
I1111 02:39:40.333319       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="17460aef-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8550786
I1111 02:39:43.152863       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:39:43.153045       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:39:43.201123       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:39:43.286644       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:39:43.343953       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:39:43.462253       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:39:54.865588       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:39:54.865679       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="09924135-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1ec298d8-dc40-4bbc-b1a4-f6d8c48e69b6" "latency"=37529480391
I1111 02:39:54.865738       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="09924135-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=37529621294
I1111 02:39:56.523433       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1111 02:39:56.523487       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:39:56.523527       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1ec298d8-dc40-4bbc-b1a4-f6d8c48e69b6/globalmount with mount options([])
I1111 02:39:56.523551       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 149 lines ...
I1111 02:43:29.224024       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:43:40.319926       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:43:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="a6526299-616a-11ed-9710-6045bd88e3ef"
I1111 02:43:40.328632       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 8 milliseconds
I1111 02:43:40.328784       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="a6526299-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8955396
I1111 02:43:41.189429       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 83 items received
I1111 02:43:41.191964       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=21753&timeout=6m37s&timeoutSeconds=397&watch=true 200 OK in 2 milliseconds
I1111 02:43:42.830727       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:43:42.830801       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="9b931bdd-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-71243770-1990-468c-869a-16af87cb006e" "latency"=20541648497
I1111 02:43:42.830845       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9b931bdd-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20541778100
I1111 02:43:43.163926       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:43:43.206499       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:43:43.293831       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:43:43.349194       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
... skipping 80 lines ...
I1111 02:44:44.195841       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:44:44.195855       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f8425869-8e2d-42e7-855e-5535677289bc/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f8425869-8e2d-42e7-855e-5535677289bc","csi.storage.k8s.io/pvc/name":"pvc-snapshottable-tester-ghnh8-my-volume","csi.storage.k8s.io/pvc/namespace":"snapshotting-4372","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-f8425869-8e2d-42e7-855e-5535677289bc"}
I1111 02:44:44.196001       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-f8425869-8e2d-42e7-855e-5535677289bc-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:44:59.223049       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:44:59.223076       1 utils.go:79] GRPC request: {}
I1111 02:44:59.223120       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:45:00.014601       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:45:00.014664       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="cc651d37-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f8425869-8e2d-42e7-855e-5535677289bc" "latency"=15818605571
I1111 02:45:00.014685       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cc651d37-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15818700473
I1111 02:45:00.929076       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 02:45:00.929096       1 utils.go:79] GRPC request: {}
I1111 02:45:00.929128       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 02:45:00.929907       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 66 lines ...
I1111 02:45:05.168675       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f8425869-8e2d-42e7-855e-5535677289bc/globalmount successfully
I1111 02:45:05.168700       1 utils.go:85] GRPC response: {}
I1111 02:45:10.319657       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:45:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="dbf740c3-616a-11ed-9710-6045bd88e3ef"
I1111 02:45:10.326206       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 6 milliseconds
I1111 02:45:10.326408       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="dbf740c3-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6853449
I1111 02:45:13.167042       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:45:13.167311       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:45:13.167398       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:45:13.209321       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:45:13.296696       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:45:13.352137       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:45:13.470582       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:45:20.363515       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:45:20.363588       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="d65eaad6-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-177f539f-785c-4890-b732-6a8a4198f060" "latency"=19432548778
I1111 02:45:20.363612       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d65eaad6-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19432661080
I1111 02:45:20.369407       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:45:20.369464       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="d66d1850-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c34630bc-09fb-41e2-93ae-9313570c8666" "latency"=19343883242
I1111 02:45:20.369500       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d66d1850-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19343987144
I1111 02:45:20.970647       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1111 02:45:20.970698       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1111 02:45:20.970713       1 utils.go:85] GRPC response: {}
I1111 02:45:20.970783       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
... skipping 234 lines ...
I1111 02:47:27.858275       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 02:47:27.858287       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-c7fa64df-0ec6-434e-aa00-c57cbe9c3e1b","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c7fa64df-0ec6-434e-aa00-c57cbe9c3e1b","csi.storage.k8s.io/pvc/name":"pvc-g9tk8","csi.storage.k8s.io/pvc/namespace":"provisioning-4550","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-c7fa64df-0ec6-434e-aa00-c57cbe9c3e1b"}
I1111 02:47:27.858442       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-c7fa64df-0ec6-434e-aa00-c57cbe9c3e1b-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 02:47:29.223566       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1111 02:47:29.223594       1 utils.go:79] GRPC request: {}
I1111 02:47:29.223634       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:47:31.252842       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:47:31.252901       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="2df20518-616b-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c7fa64df-0ec6-434e-aa00-c57cbe9c3e1b" "latency"=3394396810
I1111 02:47:31.252934       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2df20518-616b-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3394502913
I1111 02:47:33.029255       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1111 02:47:33.029302       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1111 02:47:33.029313       1 utils.go:85] GRPC response: {}
I1111 02:47:33.037272       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 57 lines ...
I1111 02:47:59.223998       1 utils.go:79] GRPC request: {}
I1111 02:47:59.224045       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1111 02:48:10.319160       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:48:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="4740ffda-616b-11ed-9710-6045bd88e3ef"
I1111 02:48:10.337695       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 18 milliseconds
I1111 02:48:10.339188       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="4740ffda-616b-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=20088738
I1111 02:48:13.174925       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:48:13.175167       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 02:48:13.214238       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:48:13.301622       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:48:13.356939       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:48:13.474466       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 02:48:25.035881       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 02:48:25.035954       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="40245ff0-616b-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-99a107cb-e783-49cd-af6b-31646664aaec" "latency"=26648444098
I1111 02:48:25.035982       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="40245ff0-616b-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=26648566601
I1111 02:48:26.794915       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1111 02:48:26.794967       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1111 02:48:26.794980       1 utils.go:85] GRPC response: {}
I1111 02:48:26.804257       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 404 lines ...
I1111 03:03:40.320865       1 azuredisk_v2.go:562]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T03:03:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="7193ff42-616d-11ed-9710-6045bd88e3ef"
I1111 03:03:40.341532       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 20 milliseconds
I1111 03:03:40.341732       1 azuredisk_v2.go:549]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="7193ff42-616d-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=20964657
I1111 03:03:41.331687       1 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: Watch close - *v1.CustomResourceDefinition total 0 items received
I1111 03:03:41.338140       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=27092&timeout=8m17s&timeoutSeconds=497&watch=true 200 OK in 6 milliseconds
I1111 03:03:43.212389       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 03:03:43.212649       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1111 03:03:43.231642       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 03:03:43.319931       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 03:03:43.377543       1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync
I1111 03:03:43.493093       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1111 03:03:43.732714       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 03:03:43.732791       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="703f88fe-616d-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1f84464b-1ed5-465c-8635-350592edd63c" "latency"=5643183529
I1111 03:03:43.732814       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="703f88fe-616d-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5643305831
I1111 03:03:45.393473       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1111 03:03:45.393529       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS
I1111 03:03:45.393576       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1f84464b-1ed5-465c-8635-350592edd63c/globalmount with mount options([])
I1111 03:03:45.393595       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 45 lines ...
I1111 03:04:18.577314       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1111 03:04:18.577334       1 utils.go:79] GRPC request: {}
I1111 03:04:18.577366       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1111 03:04:18.578253       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 03:04:18.578272       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-49387867-afa8-4df9-b167-425fff340720/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-49387867-afa8-4df9-b167-425fff340720","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-t8cxp-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-49387867-afa8-4df9-b167-425fff340720"}
I1111 03:04:18.578509       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-49387867-afa8-4df9-b167-425fff340720-k8s-agentpool1-30725374-vmss000002-attachment)
I1111 03:04:26.933884       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1111 03:04:26.933946       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="8861a925-616d-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-49387867-afa8-4df9-b167-425fff340720" "latency"=8355361283
I1111 03:04:26.933969       1 crdprovisioner.go:760]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8861a925-616d-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8355496586
I1111 03:04:27.525413       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1111 03:04:27.525467       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS
I1111 03:04:27.525502       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-49387867-afa8-4df9-b167-425fff340720/globalmount with mount options([])
I1111 03:04:27.525521       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 565 lines ...
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.77207808e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
make: *** [Makefile:341: e2e-test] Error 1
2022/11/11 03:05:59 process.go:155: Step 'make e2e-test' finished in 1h10m0.136321513s
2022/11/11 03:05:59 aksengine_helpers.go:425: downloading /root/tmp3853355860/log-dump.sh from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh
2022/11/11 03:05:59 util.go:70: curl https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh
2022/11/11 03:05:59 process.go:153: Running: chmod +x /root/tmp3853355860/log-dump.sh
2022/11/11 03:05:59 process.go:155: Step 'chmod +x /root/tmp3853355860/log-dump.sh' finished in 2.156435ms
2022/11/11 03:05:59 aksengine_helpers.go:425: downloading /root/tmp3853355860/log-dump-daemonset.yaml from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump-daemonset.yaml
... skipping 75 lines ...
ssh key file /root/.ssh/id_rsa does not exist. Exiting.
2022/11/11 03:06:58 process.go:155: Step 'bash -c /root/tmp3853355860/win-ci-logs-collector.sh kubetest-9juvoppw.westeurope.cloudapp.azure.com /root/tmp3853355860 /root/.ssh/id_rsa' finished in 3.627918ms
2022/11/11 03:06:58 aksengine.go:1141: Deleting resource group: kubetest-9juvoppw.
2022/11/11 03:14:08 process.go:96: Saved XML output to /logs/artifacts/junit_runner.xml.
2022/11/11 03:14:08 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"
2022/11/11 03:14:09 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 271.712235ms
2022/11/11 03:14:09 main.go:328: Something went wrong: encountered 1 errors: [error during make e2e-test: exit status 2]
+ EXIT_VALUE=1
+ set +o xtrace
Cleaning up after docker in docker.
================================================================================
Cleaning up after docker
e03554d77005
... skipping 4 lines ...