This job view page is being replaced by Spyglass soon. Check out the new job view.

PR	sunpa93: [V2] fix: handle pre-provisioned volume deletion at the right time
Result	FAILURE
Tests	1 failed / 13 succeeded
Started	2022-11-29 19:18
Elapsed	1h11m
Revision	8c8b4a1288a32eea961fc8b4257321345159c905
Refs	1599
job-version	v1.27.0-alpha.0.50+70617042976dc1
kubetest-version	v20221116-7c85504268
revision	v1.27.0-alpha.0.50+70617042976dc1

• artifacts
• build log

Test Failures

---

kubetest Test 58m27s

error during make e2e-test: exit status 2
				from junit_runner.xml

Filter through log files | View test history on testgrid

---

Error lines from build-log.txt

• Expand Skipped Lines
• Raw build-log.txt

... skipping 1063 lines ...
                    type: string
                type: object
                oneOf:
                - required: ["persistentVolumeClaimName"]
                - required: ["volumeSnapshotContentName"]
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 60 lines ...
                    type: string
                  volumeSnapshotContentName:
                    description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable.
                    type: string
                type: object
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 254 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 108 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 359 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status Required
            properties:
              detail:
                description: Status summarizes the current attachment state of the
                  volume attachment Nil Status indicates that the volume has not yet
                  been attached to the node
                properties:
... skipping 7 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 90 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 13 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 169 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              detail:
                description: Current status detail of the AzVolume Nil detail indicates
                  that the volume has not been created
                properties:
                  accessible_topology:
... skipping 28 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 154 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 34 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 1063 lines ...
          image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0"
          args:
            - "-csi-address=$(ADDRESS)"
            - "-v=2"
            - "-leader-election"
            - "--leader-election-namespace=kube-system"
            - '-handle-volume-inuse-error=false'
            - '-feature-gates=RecoverVolumeExpansionFailure=true'
            - "-timeout=240s"
          env:
            - name: ADDRESS
              value: /csi/csi.sock
          volumeMounts:
... skipping 405 lines ...

    test/e2e/storage/external/external.go:262
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:280

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
... skipping 2 lines ...
STEP: Building a namespace api object, basename provisioning
W1129 19:23:57.420251    1007 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
Nov 29 19:23:57.420: INFO: Found PodSecurityPolicies; testing pod creation to see if PodSecurityPolicy is enabled
Nov 29 19:23:57.533: INFO: No PSP annotation exists on dry run pod; assuming PodSecurityPolicy is disabled
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:280
Nov 29 19:23:57.970: INFO: Creating resource for dynamic PV
Nov 29 19:23:57.970: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-9367-e2e-scxc786
STEP: creating a claim
Nov 29 19:23:58.081: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 29 19:23:58.194: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comnzcm4] to have phase Bound
Nov 29 19:23:58.304: INFO: PersistentVolumeClaim test.csi.azure.comnzcm4 found but phase is Pending instead of Bound.
Nov 29 19:24:00.438: INFO: PersistentVolumeClaim test.csi.azure.comnzcm4 found but phase is Pending instead of Bound.
Nov 29 19:24:02.549: INFO: PersistentVolumeClaim test.csi.azure.comnzcm4 found and phase=Bound (4.354275821s)
STEP: Creating pod pod-subpath-test-dynamicpv-7b8z
STEP: Checking for subpath error in container status
Nov 29 19:24:25.114: INFO: Deleting pod "pod-subpath-test-dynamicpv-7b8z" in namespace "provisioning-9367"
Nov 29 19:24:25.226: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-7b8z" to be fully deleted
STEP: Deleting pod
Nov 29 19:24:27.451: INFO: Deleting pod "pod-subpath-test-dynamicpv-7b8z" in namespace "provisioning-9367"
STEP: Deleting pvc
Nov 29 19:24:27.562: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comnzcm4"
... skipping 22 lines ...

• [SLOW TEST:102.916 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:280
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] 
  should concurrently access the single read-only volume from pods on the same node
  test/e2e/storage/testsuites/multivolume.go:423

... skipping 90 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single read-only volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:423
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":49,"completed":1,"skipped":16,"failed":0}

SSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral 
  should support multiple inline ephemeral volumes
  test/e2e/storage/testsuites/ephemeral.go:254
... skipping 54 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should support multiple inline ephemeral volumes
    test/e2e/storage/testsuites/ephemeral.go:254
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support multiple inline ephemeral volumes","total":32,"completed":1,"skipped":45,"failed":0}

SSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
... skipping 148 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents","total":29,"completed":1,"skipped":85,"failed":0}

SSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] 
  should access to two volumes with the same volume mode and retain data across pod recreation on different node
  test/e2e/storage/testsuites/multivolume.go:168
... skipping 193 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:168
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":27,"completed":1,"skipped":28,"failed":0}

SSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:27:00.846: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 148 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with pvc data source
    test/e2e/storage/testsuites/provisioning.go:421
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with pvc data source","total":30,"completed":1,"skipped":174,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs)] volumeLimits 
  should verify that all csinodes have volume limits
  test/e2e/storage/testsuites/volumelimits.go:249
... skipping 16 lines ...
  test/e2e/framework/framework.go:188
Nov 29 19:27:52.862: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "volumelimits-7641" for this suite.

•
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs)] volumeLimits should verify that all csinodes have volume limits","total":30,"completed":2,"skipped":199,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] 
  should access to two volumes with the same volume mode and retain data across pod recreation on the same node
  test/e2e/storage/testsuites/multivolume.go:138

{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]","total":39,"completed":1,"skipped":94,"failed":0}
[BeforeEach] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 19:25:39.896: INFO: >>> kubeConfig: /root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json
... skipping 183 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:138
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":39,"completed":2,"skipped":94,"failed":0}

SS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:27:56.649: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 132 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:378
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":24,"completed":1,"skipped":70,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
... skipping 48 lines ...
[It] should check snapshot fields, check restore correctly works, check deletion (ephemeral)
  test/e2e/storage/testsuites/snapshottable.go:177
Nov 29 19:26:08.937: INFO: Creating resource for dynamic PV
Nov 29 19:26:08.937: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass snapshotting-4791-e2e-sc9ttqp
STEP: [init] starting a pod to use the claim
Nov 29 19:26:09.159: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-gfq89" in namespace "snapshotting-4791" to be "Succeeded or Failed"
Nov 29 19:26:09.268: INFO: Pod "pvc-snapshottable-tester-gfq89": Phase="Pending", Reason="", readiness=false. Elapsed: 108.775885ms
Nov 29 19:26:11.378: INFO: Pod "pvc-snapshottable-tester-gfq89": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219066255s
Nov 29 19:26:13.489: INFO: Pod "pvc-snapshottable-tester-gfq89": Phase="Pending", Reason="", readiness=false. Elapsed: 4.329597293s
Nov 29 19:26:15.600: INFO: Pod "pvc-snapshottable-tester-gfq89": Phase="Pending", Reason="", readiness=false. Elapsed: 6.440985978s
Nov 29 19:26:17.709: INFO: Pod "pvc-snapshottable-tester-gfq89": Phase="Pending", Reason="", readiness=false. Elapsed: 8.550206991s
Nov 29 19:26:19.819: INFO: Pod "pvc-snapshottable-tester-gfq89": Phase="Pending", Reason="", readiness=false. Elapsed: 10.659738081s
... skipping 7 lines ...
Nov 29 19:26:36.699: INFO: Pod "pvc-snapshottable-tester-gfq89": Phase="Pending", Reason="", readiness=false. Elapsed: 27.54023022s
Nov 29 19:26:38.808: INFO: Pod "pvc-snapshottable-tester-gfq89": Phase="Pending", Reason="", readiness=false. Elapsed: 29.648772632s
Nov 29 19:26:40.917: INFO: Pod "pvc-snapshottable-tester-gfq89": Phase="Pending", Reason="", readiness=false. Elapsed: 31.758221028s
Nov 29 19:26:43.029: INFO: Pod "pvc-snapshottable-tester-gfq89": Phase="Pending", Reason="", readiness=false. Elapsed: 33.869410132s
Nov 29 19:26:45.142: INFO: Pod "pvc-snapshottable-tester-gfq89": Phase="Succeeded", Reason="", readiness=false. Elapsed: 35.982890799s
STEP: Saw pod success
Nov 29 19:26:45.142: INFO: Pod "pvc-snapshottable-tester-gfq89" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
STEP: creating a SnapshotClass
STEP: creating a dynamic VolumeSnapshot
Nov 29 19:26:45.586: INFO: Waiting up to 5m0s for VolumeSnapshot snapshot-489fq to become ready
Nov 29 19:26:45.695: INFO: VolumeSnapshot snapshot-489fq found but is not ready.
Nov 29 19:26:47.807: INFO: VolumeSnapshot snapshot-489fq found but is not ready.
... skipping 49 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works, check deletion (ephemeral)
        test/e2e/storage/testsuites/snapshottable.go:177
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works, check deletion (ephemeral)","total":32,"completed":2,"skipped":126,"failed":0}
[BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 19:28:31.440: INFO: >>> kubeConfig: /root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename topology
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  test/e2e/storage/testsuites/topology.go:194
Nov 29 19:28:32.200: INFO: Driver didn't provide topology keys -- skipping
[AfterEach] [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/framework/framework.go:188
Nov 29 19:28:32.200: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "topology-3085" for this suite.


S [SKIPPING] [0.981 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/storage/framework/testsuite.go:50
    should fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement]
    test/e2e/storage/testsuites/topology.go:194

    Driver didn't provide topology keys -- skipping

    test/e2e/storage/testsuites/topology.go:126
------------------------------
... skipping 81 lines ...

    test/e2e/storage/external/external.go:262
------------------------------
SSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should fail if subpath directory is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:242

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 19:27:01.153: INFO: >>> kubeConfig: /root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename provisioning
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if subpath directory is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:242
Nov 29 19:27:01.917: INFO: Creating resource for dynamic PV
Nov 29 19:27:01.917: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-6278-e2e-sc9qtnd
STEP: creating a claim
Nov 29 19:27:02.027: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 29 19:27:02.141: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com4zjn5] to have phase Bound
Nov 29 19:27:02.250: INFO: PersistentVolumeClaim test.csi.azure.com4zjn5 found but phase is Pending instead of Bound.
Nov 29 19:27:04.358: INFO: PersistentVolumeClaim test.csi.azure.com4zjn5 found but phase is Pending instead of Bound.
Nov 29 19:27:06.467: INFO: PersistentVolumeClaim test.csi.azure.com4zjn5 found and phase=Bound (4.326687591s)
STEP: Creating pod pod-subpath-test-dynamicpv-grhm
STEP: Checking for subpath error in container status
Nov 29 19:28:01.018: INFO: Deleting pod "pod-subpath-test-dynamicpv-grhm" in namespace "provisioning-6278"
Nov 29 19:28:01.135: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-grhm" to be fully deleted
STEP: Deleting pod
Nov 29 19:28:03.354: INFO: Deleting pod "pod-subpath-test-dynamicpv-grhm" in namespace "provisioning-6278"
STEP: Deleting pvc
Nov 29 19:28:03.463: INFO: Deleting PersistentVolumeClaim "test.csi.azure.com4zjn5"
... skipping 22 lines ...

• [SLOW TEST:134.630 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:242
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow][LinuxOnly]","total":27,"completed":2,"skipped":151,"failed":0}

SSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
... skipping 103 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support restarting containers using file as subpath [Slow][LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:333
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using file as subpath [Slow][LinuxOnly]","total":49,"completed":2,"skipped":27,"failed":0}

S
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:29:30.875: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 115 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should create read/write inline ephemeral volume
    test/e2e/storage/testsuites/ephemeral.go:196
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read/write inline ephemeral volume","total":30,"completed":3,"skipped":243,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:30:21.176: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 3 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:242

    Distro debian doesn't support ntfs -- skipping

    test/e2e/storage/framework/testsuite.go:127
------------------------------
... skipping 77 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should support multiple inline ephemeral volumes
    test/e2e/storage/testsuites/ephemeral.go:254
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support multiple inline ephemeral volumes","total":39,"completed":3,"skipped":167,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:30:24.294: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 76 lines ...
Nov 29 19:28:33.521: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comxsg4l] to have phase Bound
Nov 29 19:28:33.632: INFO: PersistentVolumeClaim test.csi.azure.comxsg4l found but phase is Pending instead of Bound.
Nov 29 19:28:35.741: INFO: PersistentVolumeClaim test.csi.azure.comxsg4l found but phase is Pending instead of Bound.
Nov 29 19:28:37.850: INFO: PersistentVolumeClaim test.csi.azure.comxsg4l found and phase=Bound (4.328466186s)
STEP: Creating pod pod-subpath-test-dynamicpv-bplg
STEP: Creating a pod to test atomic-volume-subpath
Nov 29 19:28:38.178: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-bplg" in namespace "provisioning-9666" to be "Succeeded or Failed"
Nov 29 19:28:38.286: INFO: Pod "pod-subpath-test-dynamicpv-bplg": Phase="Pending", Reason="", readiness=false. Elapsed: 108.277034ms
Nov 29 19:28:40.397: INFO: Pod "pod-subpath-test-dynamicpv-bplg": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218834506s
Nov 29 19:28:42.507: INFO: Pod "pod-subpath-test-dynamicpv-bplg": Phase="Pending", Reason="", readiness=false. Elapsed: 4.329189083s
Nov 29 19:28:44.617: INFO: Pod "pod-subpath-test-dynamicpv-bplg": Phase="Pending", Reason="", readiness=false. Elapsed: 6.439061523s
Nov 29 19:28:46.726: INFO: Pod "pod-subpath-test-dynamicpv-bplg": Phase="Pending", Reason="", readiness=false. Elapsed: 8.548480884s
Nov 29 19:28:48.837: INFO: Pod "pod-subpath-test-dynamicpv-bplg": Phase="Pending", Reason="", readiness=false. Elapsed: 10.658781559s
... skipping 21 lines ...
Nov 29 19:29:35.266: INFO: Pod "pod-subpath-test-dynamicpv-bplg": Phase="Running", Reason="", readiness=true. Elapsed: 57.088052664s
Nov 29 19:29:37.376: INFO: Pod "pod-subpath-test-dynamicpv-bplg": Phase="Running", Reason="", readiness=true. Elapsed: 59.197748171s
Nov 29 19:29:39.485: INFO: Pod "pod-subpath-test-dynamicpv-bplg": Phase="Running", Reason="", readiness=true. Elapsed: 1m1.306786356s
Nov 29 19:29:41.594: INFO: Pod "pod-subpath-test-dynamicpv-bplg": Phase="Running", Reason="", readiness=false. Elapsed: 1m3.416227963s
Nov 29 19:29:43.704: INFO: Pod "pod-subpath-test-dynamicpv-bplg": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m5.526188823s
STEP: Saw pod success
Nov 29 19:29:43.704: INFO: Pod "pod-subpath-test-dynamicpv-bplg" satisfied condition "Succeeded or Failed"
Nov 29 19:29:43.813: INFO: Trying to get logs from node k8s-agentpool1-16172903-vmss000001 pod pod-subpath-test-dynamicpv-bplg container test-container-subpath-dynamicpv-bplg: <nil>
STEP: delete the pod
Nov 29 19:29:44.049: INFO: Waiting for pod pod-subpath-test-dynamicpv-bplg to disappear
Nov 29 19:29:44.160: INFO: Pod pod-subpath-test-dynamicpv-bplg no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-bplg
Nov 29 19:29:44.160: INFO: Deleting pod "pod-subpath-test-dynamicpv-bplg" in namespace "provisioning-9666"
... skipping 23 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support file as subpath [LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:232
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support file as subpath [LinuxOnly]","total":32,"completed":3,"skipped":205,"failed":0}
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:30:26.045: INFO: Distro debian doesn't support ntfs -- skipping
[AfterEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning
  test/e2e/framework/framework.go:188

... skipping 120 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:323
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":29,"completed":2,"skipped":107,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (ntfs)][Feature:Windows] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:30:34.423: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 87 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should create read-only inline ephemeral volume
    test/e2e/storage/testsuites/ephemeral.go:175
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read-only inline ephemeral volume","total":27,"completed":3,"skipped":207,"failed":0}
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:31:20.878: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
[AfterEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/framework/framework.go:188


S [SKIPPING] in Spec Setup (BeforeEach) [0.002 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:269

    Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 18 lines ...

    test/e2e/storage/external/external.go:262
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should fail if subpath file is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:258

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 19:30:25.457: INFO: >>> kubeConfig: /root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename provisioning
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if subpath file is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:258
Nov 29 19:30:26.228: INFO: Creating resource for dynamic PV
Nov 29 19:30:26.228: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-4888-e2e-scskxcm
STEP: creating a claim
Nov 29 19:30:26.341: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 29 19:30:26.454: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comvltrl] to have phase Bound
Nov 29 19:30:26.563: INFO: PersistentVolumeClaim test.csi.azure.comvltrl found but phase is Pending instead of Bound.
Nov 29 19:30:28.672: INFO: PersistentVolumeClaim test.csi.azure.comvltrl found but phase is Pending instead of Bound.
Nov 29 19:30:30.782: INFO: PersistentVolumeClaim test.csi.azure.comvltrl found and phase=Bound (4.327947425s)
STEP: Creating pod pod-subpath-test-dynamicpv-fdkt
STEP: Checking for subpath error in container status
Nov 29 19:31:19.333: INFO: Deleting pod "pod-subpath-test-dynamicpv-fdkt" in namespace "provisioning-4888"
Nov 29 19:31:19.447: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-fdkt" to be fully deleted
STEP: Deleting pod
Nov 29 19:31:21.668: INFO: Deleting pod "pod-subpath-test-dynamicpv-fdkt" in namespace "provisioning-4888"
STEP: Deleting pvc
Nov 29 19:31:21.782: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comvltrl"
... skipping 22 lines ...

• [SLOW TEST:128.672 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath file is outside the volume [Slow][LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:258
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow][LinuxOnly]","total":39,"completed":4,"skipped":272,"failed":0}

SS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:32:34.162: INFO: Driver "test.csi.azure.com" does not support volume expansion - skipping
... skipping 77 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.003 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:280

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 212 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:248
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":24,"completed":2,"skipped":156,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (ext4)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:32:50.226: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 38 lines ...
Nov 29 19:30:27.030: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com6jgf6] to have phase Bound
Nov 29 19:30:27.139: INFO: PersistentVolumeClaim test.csi.azure.com6jgf6 found but phase is Pending instead of Bound.
Nov 29 19:30:29.247: INFO: PersistentVolumeClaim test.csi.azure.com6jgf6 found but phase is Pending instead of Bound.
Nov 29 19:30:31.357: INFO: PersistentVolumeClaim test.csi.azure.com6jgf6 found and phase=Bound (4.326453346s)
STEP: Creating pod pod-subpath-test-dynamicpv-6r66
STEP: Creating a pod to test subpath
Nov 29 19:30:31.685: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-6r66" in namespace "provisioning-9949" to be "Succeeded or Failed"
Nov 29 19:30:31.794: INFO: Pod "pod-subpath-test-dynamicpv-6r66": Phase="Pending", Reason="", readiness=false. Elapsed: 109.106127ms
Nov 29 19:30:33.904: INFO: Pod "pod-subpath-test-dynamicpv-6r66": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218544319s
Nov 29 19:30:36.013: INFO: Pod "pod-subpath-test-dynamicpv-6r66": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327888387s
Nov 29 19:30:38.123: INFO: Pod "pod-subpath-test-dynamicpv-6r66": Phase="Pending", Reason="", readiness=false. Elapsed: 6.43810689s
Nov 29 19:30:40.237: INFO: Pod "pod-subpath-test-dynamicpv-6r66": Phase="Pending", Reason="", readiness=false. Elapsed: 8.551396688s
Nov 29 19:30:42.346: INFO: Pod "pod-subpath-test-dynamicpv-6r66": Phase="Pending", Reason="", readiness=false. Elapsed: 10.661164805s
... skipping 38 lines ...
Nov 29 19:32:04.645: INFO: Pod "pod-subpath-test-dynamicpv-6r66": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.95964905s
Nov 29 19:32:06.756: INFO: Pod "pod-subpath-test-dynamicpv-6r66": Phase="Pending", Reason="", readiness=false. Elapsed: 1m35.071128993s
Nov 29 19:32:08.868: INFO: Pod "pod-subpath-test-dynamicpv-6r66": Phase="Pending", Reason="", readiness=false. Elapsed: 1m37.182715812s
Nov 29 19:32:10.978: INFO: Pod "pod-subpath-test-dynamicpv-6r66": Phase="Pending", Reason="", readiness=false. Elapsed: 1m39.293031299s
Nov 29 19:32:13.090: INFO: Pod "pod-subpath-test-dynamicpv-6r66": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m41.404533959s
STEP: Saw pod success
Nov 29 19:32:13.090: INFO: Pod "pod-subpath-test-dynamicpv-6r66" satisfied condition "Succeeded or Failed"
Nov 29 19:32:13.199: INFO: Trying to get logs from node k8s-agentpool1-16172903-vmss000000 pod pod-subpath-test-dynamicpv-6r66 container test-container-volume-dynamicpv-6r66: <nil>
STEP: delete the pod
Nov 29 19:32:13.451: INFO: Waiting for pod pod-subpath-test-dynamicpv-6r66 to disappear
Nov 29 19:32:13.559: INFO: Pod pod-subpath-test-dynamicpv-6r66 no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-6r66
Nov 29 19:32:13.559: INFO: Deleting pod "pod-subpath-test-dynamicpv-6r66" in namespace "provisioning-9949"
... skipping 29 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support non-existent path
    test/e2e/storage/testsuites/subpath.go:196
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path","total":32,"completed":4,"skipped":209,"failed":0}

SSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should support readOnly directory specified in the volumeMount
  test/e2e/storage/testsuites/subpath.go:367
... skipping 17 lines ...
Nov 29 19:32:36.874: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com57fb9] to have phase Bound
Nov 29 19:32:36.984: INFO: PersistentVolumeClaim test.csi.azure.com57fb9 found but phase is Pending instead of Bound.
Nov 29 19:32:39.094: INFO: PersistentVolumeClaim test.csi.azure.com57fb9 found but phase is Pending instead of Bound.
Nov 29 19:32:41.214: INFO: PersistentVolumeClaim test.csi.azure.com57fb9 found and phase=Bound (4.33903283s)
STEP: Creating pod pod-subpath-test-dynamicpv-k5hr
STEP: Creating a pod to test subpath
Nov 29 19:32:41.547: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-k5hr" in namespace "provisioning-8901" to be "Succeeded or Failed"
Nov 29 19:32:41.660: INFO: Pod "pod-subpath-test-dynamicpv-k5hr": Phase="Pending", Reason="", readiness=false. Elapsed: 112.52923ms
Nov 29 19:32:43.772: INFO: Pod "pod-subpath-test-dynamicpv-k5hr": Phase="Pending", Reason="", readiness=false. Elapsed: 2.224913618s
Nov 29 19:32:45.886: INFO: Pod "pod-subpath-test-dynamicpv-k5hr": Phase="Pending", Reason="", readiness=false. Elapsed: 4.338381068s
Nov 29 19:32:47.997: INFO: Pod "pod-subpath-test-dynamicpv-k5hr": Phase="Pending", Reason="", readiness=false. Elapsed: 6.449391207s
Nov 29 19:32:50.111: INFO: Pod "pod-subpath-test-dynamicpv-k5hr": Phase="Pending", Reason="", readiness=false. Elapsed: 8.56413686s
Nov 29 19:32:52.223: INFO: Pod "pod-subpath-test-dynamicpv-k5hr": Phase="Pending", Reason="", readiness=false. Elapsed: 10.67576791s
... skipping 18 lines ...
Nov 29 19:33:32.359: INFO: Pod "pod-subpath-test-dynamicpv-k5hr": Phase="Pending", Reason="", readiness=false. Elapsed: 50.811923931s
Nov 29 19:33:34.475: INFO: Pod "pod-subpath-test-dynamicpv-k5hr": Phase="Pending", Reason="", readiness=false. Elapsed: 52.927495163s
Nov 29 19:33:36.587: INFO: Pod "pod-subpath-test-dynamicpv-k5hr": Phase="Pending", Reason="", readiness=false. Elapsed: 55.039843591s
Nov 29 19:33:38.698: INFO: Pod "pod-subpath-test-dynamicpv-k5hr": Phase="Pending", Reason="", readiness=false. Elapsed: 57.150851422s
Nov 29 19:33:40.810: INFO: Pod "pod-subpath-test-dynamicpv-k5hr": Phase="Succeeded", Reason="", readiness=false. Elapsed: 59.262299766s
STEP: Saw pod success
Nov 29 19:33:40.810: INFO: Pod "pod-subpath-test-dynamicpv-k5hr" satisfied condition "Succeeded or Failed"
Nov 29 19:33:40.920: INFO: Trying to get logs from node k8s-agentpool1-16172903-vmss000000 pod pod-subpath-test-dynamicpv-k5hr container test-container-subpath-dynamicpv-k5hr: <nil>
STEP: delete the pod
Nov 29 19:33:41.153: INFO: Waiting for pod pod-subpath-test-dynamicpv-k5hr to disappear
Nov 29 19:33:41.269: INFO: Pod pod-subpath-test-dynamicpv-k5hr no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-k5hr
Nov 29 19:33:41.269: INFO: Deleting pod "pod-subpath-test-dynamicpv-k5hr" in namespace "provisioning-8901"
... skipping 29 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support readOnly directory specified in the volumeMount
    test/e2e/storage/testsuites/subpath.go:367
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly directory specified in the volumeMount","total":39,"completed":5,"skipped":403,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:34:54.008: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 110 lines ...
Nov 29 19:30:35.488: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comp8ll8] to have phase Bound
Nov 29 19:30:35.597: INFO: PersistentVolumeClaim test.csi.azure.comp8ll8 found but phase is Pending instead of Bound.
Nov 29 19:30:37.707: INFO: PersistentVolumeClaim test.csi.azure.comp8ll8 found but phase is Pending instead of Bound.
Nov 29 19:30:39.818: INFO: PersistentVolumeClaim test.csi.azure.comp8ll8 found and phase=Bound (4.329803521s)
STEP: Creating pod pod-subpath-test-dynamicpv-9c9k
STEP: Creating a pod to test subpath
Nov 29 19:30:40.146: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-9c9k" in namespace "provisioning-549" to be "Succeeded or Failed"
Nov 29 19:30:40.258: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 112.32339ms
Nov 29 19:30:42.368: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 2.221538488s
Nov 29 19:30:44.477: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 4.330655417s
Nov 29 19:30:46.586: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 6.440280832s
Nov 29 19:30:48.696: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 8.549347345s
Nov 29 19:30:50.806: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 10.660024392s
... skipping 23 lines ...
Nov 29 19:31:41.472: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.325638259s
Nov 29 19:31:43.581: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.435073261s
Nov 29 19:31:45.691: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.544398109s
Nov 29 19:31:47.800: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.653566843s
Nov 29 19:31:49.909: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m9.763252218s
STEP: Saw pod success
Nov 29 19:31:49.909: INFO: Pod "pod-subpath-test-dynamicpv-9c9k" satisfied condition "Succeeded or Failed"
Nov 29 19:31:50.019: INFO: Trying to get logs from node k8s-agentpool1-16172903-vmss000001 pod pod-subpath-test-dynamicpv-9c9k container test-container-subpath-dynamicpv-9c9k: <nil>
STEP: delete the pod
Nov 29 19:31:50.247: INFO: Waiting for pod pod-subpath-test-dynamicpv-9c9k to disappear
Nov 29 19:31:50.356: INFO: Pod pod-subpath-test-dynamicpv-9c9k no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-9c9k
Nov 29 19:31:50.356: INFO: Deleting pod "pod-subpath-test-dynamicpv-9c9k" in namespace "provisioning-549"
STEP: Creating pod pod-subpath-test-dynamicpv-9c9k
STEP: Creating a pod to test subpath
Nov 29 19:31:50.581: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-9c9k" in namespace "provisioning-549" to be "Succeeded or Failed"
Nov 29 19:31:50.689: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 108.376945ms
Nov 29 19:31:52.799: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217582136s
Nov 29 19:31:54.908: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326926823s
Nov 29 19:31:57.018: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 6.43639557s
Nov 29 19:31:59.127: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 8.546299139s
Nov 29 19:32:01.238: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 10.656460888s
... skipping 64 lines ...
Nov 29 19:34:18.482: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 2m27.901219654s
Nov 29 19:34:20.593: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.011574592s
Nov 29 19:34:22.703: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.122258894s
Nov 29 19:34:24.815: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.234084787s
Nov 29 19:34:26.926: INFO: Pod "pod-subpath-test-dynamicpv-9c9k": Phase="Succeeded", Reason="", readiness=false. Elapsed: 2m36.345097555s
STEP: Saw pod success
Nov 29 19:34:26.926: INFO: Pod "pod-subpath-test-dynamicpv-9c9k" satisfied condition "Succeeded or Failed"
Nov 29 19:34:27.036: INFO: Trying to get logs from node k8s-agentpool1-16172903-vmss000001 pod pod-subpath-test-dynamicpv-9c9k container test-container-subpath-dynamicpv-9c9k: <nil>
STEP: delete the pod
Nov 29 19:34:27.284: INFO: Waiting for pod pod-subpath-test-dynamicpv-9c9k to disappear
Nov 29 19:34:27.392: INFO: Pod pod-subpath-test-dynamicpv-9c9k no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-9c9k
Nov 29 19:34:27.392: INFO: Deleting pod "pod-subpath-test-dynamicpv-9c9k" in namespace "provisioning-549"
... skipping 23 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support existing directories when readOnly specified in the volumeSource
    test/e2e/storage/testsuites/subpath.go:397
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource","total":29,"completed":3,"skipped":211,"failed":0}

SSS
------------------------------
[BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:35:09.253: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping
... skipping 357 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with pvc data source in parallel [Slow]
    test/e2e/storage/testsuites/provisioning.go:459
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with pvc data source in parallel [Slow]","total":30,"completed":4,"skipped":301,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] 
  should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
  test/e2e/storage/testsuites/multivolume.go:378
... skipping 84 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:378
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":27,"completed":4,"skipped":264,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral 
  should support two pods which have the same volume definition
  test/e2e/storage/testsuites/ephemeral.go:216
... skipping 89 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should support two pods which have the same volume definition
    test/e2e/storage/testsuites/ephemeral.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should support two pods which have the same volume definition","total":49,"completed":3,"skipped":53,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:36:19.767: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 175 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:298
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":32,"completed":5,"skipped":223,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes 
  should store data
  test/e2e/storage/testsuites/volumes.go:161
... skipping 115 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should store data
    test/e2e/storage/testsuites/volumes.go:161
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data","total":24,"completed":3,"skipped":322,"failed":0}

SSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode 
  should not mount / map unused volumes in a pod [LinuxOnly]
  test/e2e/storage/testsuites/volumemode.go:354
... skipping 98 lines ...
Nov 29 19:35:49.509: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comk6nqj] to have phase Bound
Nov 29 19:35:49.616: INFO: PersistentVolumeClaim test.csi.azure.comk6nqj found but phase is Pending instead of Bound.
Nov 29 19:35:51.725: INFO: PersistentVolumeClaim test.csi.azure.comk6nqj found but phase is Pending instead of Bound.
Nov 29 19:35:53.839: INFO: PersistentVolumeClaim test.csi.azure.comk6nqj found and phase=Bound (4.330168124s)
STEP: Creating pod exec-volume-test-dynamicpv-6zjp
STEP: Creating a pod to test exec-volume-test
Nov 29 19:35:54.172: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-6zjp" in namespace "volume-9944" to be "Succeeded or Failed"
Nov 29 19:35:54.281: INFO: Pod "exec-volume-test-dynamicpv-6zjp": Phase="Pending", Reason="", readiness=false. Elapsed: 108.368192ms
Nov 29 19:35:56.390: INFO: Pod "exec-volume-test-dynamicpv-6zjp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217720391s
Nov 29 19:35:58.501: INFO: Pod "exec-volume-test-dynamicpv-6zjp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328635305s
Nov 29 19:36:00.611: INFO: Pod "exec-volume-test-dynamicpv-6zjp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.438455627s
Nov 29 19:36:02.721: INFO: Pod "exec-volume-test-dynamicpv-6zjp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.549116834s
Nov 29 19:36:04.831: INFO: Pod "exec-volume-test-dynamicpv-6zjp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.658915093s
... skipping 8 lines ...
Nov 29 19:36:23.821: INFO: Pod "exec-volume-test-dynamicpv-6zjp": Phase="Pending", Reason="", readiness=false. Elapsed: 29.648380824s
Nov 29 19:36:25.930: INFO: Pod "exec-volume-test-dynamicpv-6zjp": Phase="Pending", Reason="", readiness=false. Elapsed: 31.758094476s
Nov 29 19:36:28.042: INFO: Pod "exec-volume-test-dynamicpv-6zjp": Phase="Pending", Reason="", readiness=false. Elapsed: 33.869599099s
Nov 29 19:36:30.150: INFO: Pod "exec-volume-test-dynamicpv-6zjp": Phase="Pending", Reason="", readiness=false. Elapsed: 35.978264961s
Nov 29 19:36:32.259: INFO: Pod "exec-volume-test-dynamicpv-6zjp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.087074548s
STEP: Saw pod success
Nov 29 19:36:32.259: INFO: Pod "exec-volume-test-dynamicpv-6zjp" satisfied condition "Succeeded or Failed"
Nov 29 19:36:32.368: INFO: Trying to get logs from node k8s-agentpool1-16172903-vmss000001 pod exec-volume-test-dynamicpv-6zjp container exec-container-dynamicpv-6zjp: <nil>
STEP: delete the pod
Nov 29 19:36:32.633: INFO: Waiting for pod exec-volume-test-dynamicpv-6zjp to disappear
Nov 29 19:36:32.741: INFO: Pod exec-volume-test-dynamicpv-6zjp no longer exists
STEP: Deleting pod exec-volume-test-dynamicpv-6zjp
Nov 29 19:36:32.741: INFO: Deleting pod "exec-volume-test-dynamicpv-6zjp" in namespace "volume-9944"
... skipping 27 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should allow exec of files on the volume
    test/e2e/storage/testsuites/volumes.go:198
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should allow exec of files on the volume","total":30,"completed":5,"skipped":361,"failed":0}

SSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:37:45.155: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 38 lines ...
Nov 29 19:36:14.837: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com42kz2] to have phase Bound
Nov 29 19:36:14.945: INFO: PersistentVolumeClaim test.csi.azure.com42kz2 found but phase is Pending instead of Bound.
Nov 29 19:36:17.055: INFO: PersistentVolumeClaim test.csi.azure.com42kz2 found but phase is Pending instead of Bound.
Nov 29 19:36:19.164: INFO: PersistentVolumeClaim test.csi.azure.com42kz2 found and phase=Bound (4.326817926s)
STEP: Creating pod exec-volume-test-dynamicpv-8mdz
STEP: Creating a pod to test exec-volume-test
Nov 29 19:36:19.492: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-8mdz" in namespace "volume-3731" to be "Succeeded or Failed"
Nov 29 19:36:19.601: INFO: Pod "exec-volume-test-dynamicpv-8mdz": Phase="Pending", Reason="", readiness=false. Elapsed: 108.760718ms
Nov 29 19:36:21.710: INFO: Pod "exec-volume-test-dynamicpv-8mdz": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218410408s
Nov 29 19:36:23.820: INFO: Pod "exec-volume-test-dynamicpv-8mdz": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328031533s
Nov 29 19:36:25.929: INFO: Pod "exec-volume-test-dynamicpv-8mdz": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437179865s
Nov 29 19:36:28.039: INFO: Pod "exec-volume-test-dynamicpv-8mdz": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547010692s
Nov 29 19:36:30.147: INFO: Pod "exec-volume-test-dynamicpv-8mdz": Phase="Pending", Reason="", readiness=false. Elapsed: 10.655606763s
... skipping 12 lines ...
Nov 29 19:36:57.582: INFO: Pod "exec-volume-test-dynamicpv-8mdz": Phase="Pending", Reason="", readiness=false. Elapsed: 38.089844761s
Nov 29 19:36:59.691: INFO: Pod "exec-volume-test-dynamicpv-8mdz": Phase="Pending", Reason="", readiness=false. Elapsed: 40.199356661s
Nov 29 19:37:01.801: INFO: Pod "exec-volume-test-dynamicpv-8mdz": Phase="Pending", Reason="", readiness=false. Elapsed: 42.308794744s
Nov 29 19:37:03.910: INFO: Pod "exec-volume-test-dynamicpv-8mdz": Phase="Pending", Reason="", readiness=false. Elapsed: 44.417907592s
Nov 29 19:37:06.018: INFO: Pod "exec-volume-test-dynamicpv-8mdz": Phase="Succeeded", Reason="", readiness=false. Elapsed: 46.526450848s
STEP: Saw pod success
Nov 29 19:37:06.018: INFO: Pod "exec-volume-test-dynamicpv-8mdz" satisfied condition "Succeeded or Failed"
Nov 29 19:37:06.128: INFO: Trying to get logs from node k8s-agentpool1-16172903-vmss000000 pod exec-volume-test-dynamicpv-8mdz container exec-container-dynamicpv-8mdz: <nil>
STEP: delete the pod
Nov 29 19:37:06.372: INFO: Waiting for pod exec-volume-test-dynamicpv-8mdz to disappear
Nov 29 19:37:06.480: INFO: Pod exec-volume-test-dynamicpv-8mdz no longer exists
STEP: Deleting pod exec-volume-test-dynamicpv-8mdz
Nov 29 19:37:06.480: INFO: Deleting pod "exec-volume-test-dynamicpv-8mdz" in namespace "volume-3731"
... skipping 33 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should allow exec of files on the volume
    test/e2e/storage/testsuites/volumes.go:198
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should allow exec of files on the volume","total":27,"completed":5,"skipped":310,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:38:49.727: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 24 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach]
    test/e2e/storage/testsuites/volumemode.go:299

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
SSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode 
  should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299

[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 19:38:49.918: INFO: >>> kubeConfig: /root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename volumemode
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299
Nov 29 19:38:50.684: INFO: Creating resource for dynamic PV
Nov 29 19:38:50.684: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass volumemode-9017-e2e-scr7kng
STEP: creating a claim
Nov 29 19:38:50.907: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comdfvd4] to have phase Bound
Nov 29 19:38:51.016: INFO: PersistentVolumeClaim test.csi.azure.comdfvd4 found but phase is Pending instead of Bound.
Nov 29 19:38:53.126: INFO: PersistentVolumeClaim test.csi.azure.comdfvd4 found but phase is Pending instead of Bound.
Nov 29 19:38:55.235: INFO: PersistentVolumeClaim test.csi.azure.comdfvd4 found and phase=Bound (4.328632031s)
STEP: Creating pod
STEP: Waiting for the pod to fail
Nov 29 19:38:57.899: INFO: Deleting pod "pod-4ce57f42-cea5-4852-8b5a-0e5013a09f90" in namespace "volumemode-9017"
Nov 29 19:38:58.011: INFO: Wait up to 5m0s for pod "pod-4ce57f42-cea5-4852-8b5a-0e5013a09f90" to be fully deleted
STEP: Deleting pvc
Nov 29 19:39:00.231: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comdfvd4"
Nov 29 19:39:00.342: INFO: Waiting up to 5m0s for PersistentVolume pvc-f69ad472-78df-4a2d-82b5-1a7dcbb7a4f6 to get deleted
Nov 29 19:39:00.451: INFO: PersistentVolume pvc-f69ad472-78df-4a2d-82b5-1a7dcbb7a4f6 found and phase=Released (108.694109ms)
... skipping 14 lines ...

• [SLOW TEST:51.969 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow]
    test/e2e/storage/testsuites/volumemode.go:299
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should fail to use a volume in a pod with mismatched mode [Slow]","total":27,"completed":6,"skipped":608,"failed":0}

SSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:39:41.907: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 37 lines ...
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should verify container cannot write to subpath readonly volumes [Slow]
  test/e2e/storage/testsuites/subpath.go:425

{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":29,"completed":4,"skipped":222,"failed":0}
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 19:36:56.260: INFO: >>> kubeConfig: /root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json
... skipping 9 lines ...
Nov 29 19:36:57.129: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 29 19:36:57.244: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com9pwz7] to have phase Bound
Nov 29 19:36:57.352: INFO: PersistentVolumeClaim test.csi.azure.com9pwz7 found but phase is Pending instead of Bound.
Nov 29 19:36:59.462: INFO: PersistentVolumeClaim test.csi.azure.com9pwz7 found but phase is Pending instead of Bound.
Nov 29 19:37:01.572: INFO: PersistentVolumeClaim test.csi.azure.com9pwz7 found and phase=Bound (4.327682383s)
STEP: Creating pod to format volume volume-prep-provisioning-1110
Nov 29 19:37:01.897: INFO: Waiting up to 5m0s for pod "volume-prep-provisioning-1110" in namespace "provisioning-1110" to be "Succeeded or Failed"
Nov 29 19:37:02.008: INFO: Pod "volume-prep-provisioning-1110": Phase="Pending", Reason="", readiness=false. Elapsed: 110.414276ms
Nov 29 19:37:04.138: INFO: Pod "volume-prep-provisioning-1110": Phase="Pending", Reason="", readiness=false. Elapsed: 2.241077339s
Nov 29 19:37:06.257: INFO: Pod "volume-prep-provisioning-1110": Phase="Pending", Reason="", readiness=false. Elapsed: 4.360125006s
Nov 29 19:37:08.367: INFO: Pod "volume-prep-provisioning-1110": Phase="Pending", Reason="", readiness=false. Elapsed: 6.469313708s
Nov 29 19:37:10.475: INFO: Pod "volume-prep-provisioning-1110": Phase="Pending", Reason="", readiness=false. Elapsed: 8.578044709s
Nov 29 19:37:12.586: INFO: Pod "volume-prep-provisioning-1110": Phase="Pending", Reason="", readiness=false. Elapsed: 10.68864855s
... skipping 24 lines ...
Nov 29 19:38:05.334: INFO: Pod "volume-prep-provisioning-1110": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.436558853s
Nov 29 19:38:07.444: INFO: Pod "volume-prep-provisioning-1110": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.546856666s
Nov 29 19:38:09.554: INFO: Pod "volume-prep-provisioning-1110": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.656245933s
Nov 29 19:38:11.664: INFO: Pod "volume-prep-provisioning-1110": Phase="Pending", Reason="", readiness=false. Elapsed: 1m9.76660303s
Nov 29 19:38:13.773: INFO: Pod "volume-prep-provisioning-1110": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m11.876090628s
STEP: Saw pod success
Nov 29 19:38:13.773: INFO: Pod "volume-prep-provisioning-1110" satisfied condition "Succeeded or Failed"
Nov 29 19:38:13.773: INFO: Deleting pod "volume-prep-provisioning-1110" in namespace "provisioning-1110"
Nov 29 19:38:13.888: INFO: Wait up to 5m0s for pod "volume-prep-provisioning-1110" to be fully deleted
STEP: Creating pod pod-subpath-test-dynamicpv-mrpm
STEP: Checking for subpath error in container status
Nov 29 19:38:22.332: INFO: Deleting pod "pod-subpath-test-dynamicpv-mrpm" in namespace "provisioning-1110"
Nov 29 19:38:22.444: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-mrpm" to be fully deleted
STEP: Deleting pod
Nov 29 19:38:24.662: INFO: Deleting pod "pod-subpath-test-dynamicpv-mrpm" in namespace "provisioning-1110"
STEP: Deleting pvc
Nov 29 19:38:24.770: INFO: Deleting PersistentVolumeClaim "test.csi.azure.com9pwz7"
... skipping 31 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should verify container cannot write to subpath readonly volumes [Slow]
    test/e2e/storage/testsuites/subpath.go:425
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should verify container cannot write to subpath readonly volumes [Slow]","total":29,"completed":5,"skipped":222,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] volumeIO
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:40:07.816: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 73 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should create read/write inline ephemeral volume
    test/e2e/storage/testsuites/ephemeral.go:196
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should create read/write inline ephemeral volume","total":30,"completed":6,"skipped":403,"failed":0}

SSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (xfs)][Slow] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:40:56.996: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 173 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] volumes
  test/e2e/storage/framework/testsuite.go:50
    should store data
    test/e2e/storage/testsuites/volumes.go:161
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes should store data","total":24,"completed":4,"skipped":326,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:41:01.054: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 390 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:168
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":32,"completed":6,"skipped":261,"failed":0}

SSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] 
  should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
  test/e2e/storage/testsuites/multivolume.go:378
... skipping 214 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents","total":27,"completed":7,"skipped":691,"failed":0}

SSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning 
  should provision storage with pvc data source in parallel [Slow]
  test/e2e/storage/testsuites/provisioning.go:459
... skipping 482 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with pvc data source in parallel [Slow]
    test/e2e/storage/testsuites/provisioning.go:459
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with pvc data source in parallel [Slow]","total":39,"completed":6,"skipped":520,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] 
  should concurrently access the single read-only volume from pods on the same node
  test/e2e/storage/testsuites/multivolume.go:423

{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":49,"completed":4,"skipped":86,"failed":0}
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 19:41:16.988: INFO: >>> kubeConfig: /root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json
... skipping 110 lines ...
Nov 29 19:46:46.200: INFO: PersistentVolume pvc-d9209029-cc60-45be-8d4f-411d67de4846 found and phase=Released (4m36.107187576s)
Nov 29 19:46:51.310: INFO: PersistentVolume pvc-d9209029-cc60-45be-8d4f-411d67de4846 found and phase=Released (4m41.217026198s)
Nov 29 19:46:56.419: INFO: PersistentVolume pvc-d9209029-cc60-45be-8d4f-411d67de4846 found and phase=Released (4m46.326411823s)
Nov 29 19:47:01.527: INFO: PersistentVolume pvc-d9209029-cc60-45be-8d4f-411d67de4846 found and phase=Released (4m51.434845732s)
Nov 29 19:47:06.639: INFO: PersistentVolume pvc-d9209029-cc60-45be-8d4f-411d67de4846 found and phase=Released (4m56.546584627s)
STEP: Deleting sc
Nov 29 19:47:11.752: FAIL: while cleanup resource
Unexpected error:
    <errors.aggregate | len:1, cap:1>: [
        [
            {
                msg: "persistent Volume pvc-d9209029-cc60-45be-8d4f-411d67de4846 not deleted by dynamic provisioner: PersistentVolume pvc-d9209029-cc60-45be-8d4f-411d67de4846 still exists within 5m0s",
                err: {
                    s: "PersistentVolume pvc-d9209029-cc60-45be-8d4f-411d67de4846 still exists within 5m0s",
... skipping 30 lines ...
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:46 +0000 UTC - event for pod-05d0a192-11f7-4d76-9c1f-fa4498f414a9: {kubelet k8s-agentpool1-16172903-vmss000000} Created: Created container write-pod
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:46 +0000 UTC - event for pod-05d0a192-11f7-4d76-9c1f-fa4498f414a9: {kubelet k8s-agentpool1-16172903-vmss000000} Started: Started container write-pod
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:46 +0000 UTC - event for pod-05d0a192-11f7-4d76-9c1f-fa4498f414a9: {kubelet k8s-agentpool1-16172903-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2"
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:49 +0000 UTC - event for pod-05d0a192-11f7-4d76-9c1f-fa4498f414a9: {kubelet k8s-agentpool1-16172903-vmss000000} Killing: Stopping container write-pod
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:51 +0000 UTC - event for pod-da363ca2-25da-45ba-8006-729cf2699a32: {default-scheduler } Scheduled: Successfully assigned multivolume-5550/pod-da363ca2-25da-45ba-8006-729cf2699a32 to k8s-agentpool1-16172903-vmss000000
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:55 +0000 UTC - event for pod-da363ca2-25da-45ba-8006-729cf2699a32: {kubelet k8s-agentpool1-16172903-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5"
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:55 +0000 UTC - event for pod-da363ca2-25da-45ba-8006-729cf2699a32: {kubelet k8s-agentpool1-16172903-vmss000000} FileSystemResizeFailed: MountVolume.NodeExpandVolume failed for volume "pvc-d9209029-cc60-45be-8d4f-411d67de4846" requested read-only file system
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:55 +0000 UTC - event for test.csi.azure.comg47lv: {kubelet k8s-agentpool1-16172903-vmss000000} FileSystemResizeFailed: MountVolume.NodeExpandVolume failed for volume "pvc-d9209029-cc60-45be-8d4f-411d67de4846" requested read-only file system
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:56 +0000 UTC - event for pod-da363ca2-25da-45ba-8006-729cf2699a32: {kubelet k8s-agentpool1-16172903-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5" in 284.143061ms
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:56 +0000 UTC - event for pod-da363ca2-25da-45ba-8006-729cf2699a32: {kubelet k8s-agentpool1-16172903-vmss000000} Created: Created container write-pod
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:56 +0000 UTC - event for pod-da363ca2-25da-45ba-8006-729cf2699a32: {kubelet k8s-agentpool1-16172903-vmss000000} Started: Started container write-pod
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:57 +0000 UTC - event for pod-6be01a85-9e31-46d2-b048-1a3933476321: {default-scheduler } Scheduled: Successfully assigned multivolume-5550/pod-6be01a85-9e31-46d2-b048-1a3933476321 to k8s-agentpool1-16172903-vmss000000
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:58 +0000 UTC - event for pod-6be01a85-9e31-46d2-b048-1a3933476321: {kubelet k8s-agentpool1-16172903-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5"
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:58 +0000 UTC - event for pod-6be01a85-9e31-46d2-b048-1a3933476321: {kubelet k8s-agentpool1-16172903-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5" in 330.70133ms
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:58 +0000 UTC - event for pod-6be01a85-9e31-46d2-b048-1a3933476321: {kubelet k8s-agentpool1-16172903-vmss000000} FileSystemResizeFailed: MountVolume.NodeExpandVolume failed for volume "pvc-d9209029-cc60-45be-8d4f-411d67de4846" requested read-only file system
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:59 +0000 UTC - event for pod-6be01a85-9e31-46d2-b048-1a3933476321: {kubelet k8s-agentpool1-16172903-vmss000000} Created: Created container write-pod
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:41:59 +0000 UTC - event for pod-6be01a85-9e31-46d2-b048-1a3933476321: {kubelet k8s-agentpool1-16172903-vmss000000} Started: Started container write-pod
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:42:03 +0000 UTC - event for pod-6be01a85-9e31-46d2-b048-1a3933476321: {kubelet k8s-agentpool1-16172903-vmss000000} Killing: Stopping container write-pod
Nov 29 19:47:11.971: INFO: At 2022-11-29 19:42:07 +0000 UTC - event for pod-da363ca2-25da-45ba-8006-729cf2699a32: {kubelet k8s-agentpool1-16172903-vmss000000} Killing: Stopping container write-pod
Nov 29 19:47:12.080: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov 29 19:47:12.080: INFO: 
... skipping 123 lines ...
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single read-only volume from pods on the same node [Measurement]
    test/e2e/storage/testsuites/multivolume.go:423

    Nov 29 19:47:11.752: while cleanup resource
    Unexpected error:
        <errors.aggregate | len:1, cap:1>: [
            [
                {
                    msg: "persistent Volume pvc-d9209029-cc60-45be-8d4f-411d67de4846 not deleted by dynamic provisioner: PersistentVolume pvc-d9209029-cc60-45be-8d4f-411d67de4846 still exists within 5m0s",
                    err: {
                        s: "PersistentVolume pvc-d9209029-cc60-45be-8d4f-411d67de4846 still exists within 5m0s",
... skipping 3 lines ...
        ]
        persistent Volume pvc-d9209029-cc60-45be-8d4f-411d67de4846 not deleted by dynamic provisioner: PersistentVolume pvc-d9209029-cc60-45be-8d4f-411d67de4846 still exists within 5m0s
    occurred

    test/e2e/storage/testsuites/multivolume.go:129
------------------------------
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":49,"completed":4,"skipped":86,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:47:15.770: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 66 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:280

    Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 125 lines ...
Nov 29 19:47:17.772: INFO: PersistentVolume pvc-34b85d7b-09c7-4287-8d4c-27c4b7130660 found and phase=Released (4m36.104764613s)
Nov 29 19:47:22.881: INFO: PersistentVolume pvc-34b85d7b-09c7-4287-8d4c-27c4b7130660 found and phase=Released (4m41.214105265s)
Nov 29 19:47:27.992: INFO: PersistentVolume pvc-34b85d7b-09c7-4287-8d4c-27c4b7130660 found and phase=Released (4m46.324421793s)
Nov 29 19:47:33.103: INFO: PersistentVolume pvc-34b85d7b-09c7-4287-8d4c-27c4b7130660 found and phase=Released (4m51.436022742s)
Nov 29 19:47:38.213: INFO: PersistentVolume pvc-34b85d7b-09c7-4287-8d4c-27c4b7130660 found and phase=Released (4m56.545578556s)
STEP: Deleting sc
Nov 29 19:47:43.326: FAIL: while cleaning up resource
Unexpected error:
    <errors.aggregate | len:1, cap:1>: [
        [
            {
                msg: "persistent Volume pvc-34b85d7b-09c7-4287-8d4c-27c4b7130660 not deleted by dynamic provisioner: PersistentVolume pvc-34b85d7b-09c7-4287-8d4c-27c4b7130660 still exists within 5m0s",
                err: {
                    s: "PersistentVolume pvc-34b85d7b-09c7-4287-8d4c-27c4b7130660 still exists within 5m0s",
... skipping 35 lines ...
Nov 29 19:47:43.543: INFO: At 2022-11-29 19:41:07 +0000 UTC - event for pod-subpath-test-dynamicpv-xzbl: {kubelet k8s-agentpool1-16172903-vmss000001} Started: Started container test-container-subpath-dynamicpv-xzbl
Nov 29 19:47:43.543: INFO: At 2022-11-29 19:41:07 +0000 UTC - event for pod-subpath-test-dynamicpv-xzbl: {kubelet k8s-agentpool1-16172903-vmss000001} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 332.634962ms
Nov 29 19:47:43.543: INFO: At 2022-11-29 19:41:07 +0000 UTC - event for pod-subpath-test-dynamicpv-xzbl: {kubelet k8s-agentpool1-16172903-vmss000001} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 362.363327ms
Nov 29 19:47:43.543: INFO: At 2022-11-29 19:41:07 +0000 UTC - event for pod-subpath-test-dynamicpv-xzbl: {kubelet k8s-agentpool1-16172903-vmss000001} Created: Created container test-container-volume-dynamicpv-xzbl
Nov 29 19:47:43.543: INFO: At 2022-11-29 19:41:07 +0000 UTC - event for pod-subpath-test-dynamicpv-xzbl: {kubelet k8s-agentpool1-16172903-vmss000001} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2"
Nov 29 19:47:43.543: INFO: At 2022-11-29 19:41:08 +0000 UTC - event for pod-subpath-test-dynamicpv-xzbl: {kubelet k8s-agentpool1-16172903-vmss000001} Started: Started container test-container-volume-dynamicpv-xzbl
Nov 29 19:47:43.543: INFO: At 2022-11-29 19:41:11 +0000 UTC - event for pod-subpath-test-dynamicpv-xzbl: {kubelet k8s-agentpool1-16172903-vmss000001} Killing: Container test-container-subpath-dynamicpv-xzbl failed liveness probe, will be restarted
Nov 29 19:47:43.543: INFO: At 2022-11-29 19:41:11 +0000 UTC - event for pod-subpath-test-dynamicpv-xzbl: {kubelet k8s-agentpool1-16172903-vmss000001} Unhealthy: Liveness probe failed: cat: can't open '/probe-volume/probe-file': No such file or directory

Nov 29 19:47:43.543: INFO: At 2022-11-29 19:41:13 +0000 UTC - event for pod-subpath-test-dynamicpv-xzbl: {kubelet k8s-agentpool1-16172903-vmss000001} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 410.069318ms
Nov 29 19:47:43.543: INFO: At 2022-11-29 19:41:18 +0000 UTC - event for pod-subpath-test-dynamicpv-xzbl: {kubelet k8s-agentpool1-16172903-vmss000001} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 302.090165ms
Nov 29 19:47:43.652: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov 29 19:47:43.652: INFO: 
Nov 29 19:47:43.870: INFO: 
... skipping 124 lines ...
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support restarting containers using directory as subpath [Slow] [Measurement]
    test/e2e/storage/testsuites/subpath.go:322

    Nov 29 19:47:43.326: while cleaning up resource
    Unexpected error:
        <errors.aggregate | len:1, cap:1>: [
            [
                {
                    msg: "persistent Volume pvc-34b85d7b-09c7-4287-8d4c-27c4b7130660 not deleted by dynamic provisioner: PersistentVolume pvc-34b85d7b-09c7-4287-8d4c-27c4b7130660 still exists within 5m0s",
                    err: {
                        s: "PersistentVolume pvc-34b85d7b-09c7-4287-8d4c-27c4b7130660 still exists within 5m0s",
... skipping 3 lines ...
        ]
        persistent Volume pvc-34b85d7b-09c7-4287-8d4c-27c4b7130660 not deleted by dynamic provisioner: PersistentVolume pvc-34b85d7b-09c7-4287-8d4c-27c4b7130660 still exists within 5m0s
    occurred

    test/e2e/storage/testsuites/subpath.go:185
------------------------------
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using directory as subpath [Slow]","total":29,"completed":5,"skipped":450,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using directory as subpath [Slow]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] 
  should access to two volumes with the same volume mode and retain data across pod recreation on different node
  test/e2e/storage/testsuites/multivolume.go:168
... skipping 80 lines ...
Nov 29 19:41:32.722: INFO: >>> kubeConfig: /root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json
Nov 29 19:41:32.724: INFO: ExecWithOptions: Clientset creation
Nov 29 19:41:32.724: INFO: ExecWithOptions: execute(POST https://kubetest-2hnyryib.westeurope.cloudapp.azure.com/api/v1/namespaces/multivolume-7015/pods/pod-97f1429b-37dd-406f-9602-1791035092e3/exec?command=%2Fbin%2Fsh&command=-c&command=dd+if%3D%2Fmnt%2Fvolume2++bs%3D64+count%3D1+%7C+sha256sum+%7C+grep+-Fq+cb579715233846050c6409a3c72e8d15ee344218fef68fd8f0d04362cc99f8e7&container=write-pod&container=write-pod&stderr=true&stdout=true)
Nov 29 19:41:33.647: INFO: Deleting pod "pod-97f1429b-37dd-406f-9602-1791035092e3" in namespace "multivolume-7015"
Nov 29 19:41:33.759: INFO: Wait up to 5m0s for pod "pod-97f1429b-37dd-406f-9602-1791035092e3" to be fully deleted
STEP: Creating pod on {Name: Selector:map[] Affinity:&Affinity{NodeAffinity:&NodeAffinity{RequiredDuringSchedulingIgnoredDuringExecution:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{},MatchFields:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:metadata.name,Operator:NotIn,Values:[k8s-agentpool1-16172903-vmss000000],},},},},},PreferredDuringSchedulingIgnoredDuringExecution:[]PreferredSchedulingTerm{},},PodAffinity:nil,PodAntiAffinity:nil,}} with multiple volumes
Nov 29 19:46:36.429: FAIL: Unexpected error:
    <*errors.errorString | 0xc0027a81b0>: {
        s: "pod \"pod-645b1ce5-b71a-4174-930c-ae3f9a252a52\" is not Running: timed out waiting for the condition",
    }
    pod "pod-645b1ce5-b71a-4174-930c-ae3f9a252a52" is not Running: timed out waiting for the condition
occurred

... skipping 59 lines ...
Nov 29 19:49:35.423: INFO: At 2022-11-29 19:41:20 +0000 UTC - event for pod-97f1429b-37dd-406f-9602-1791035092e3: {kubelet k8s-agentpool1-16172903-vmss000000} SuccessfulMountVolume: MapVolume.MapPodDevice succeeded for volume "pvc-040f12d8-ebf6-49cd-b236-67c7e8d63e0a" volumeMapPath "/var/lib/kubelet/pods/6cca6a9a-44e5-49ea-afdc-6d33e2fcb89f/volumeDevices/kubernetes.io~csi"
Nov 29 19:49:35.423: INFO: At 2022-11-29 19:41:21 +0000 UTC - event for pod-97f1429b-37dd-406f-9602-1791035092e3: {kubelet k8s-agentpool1-16172903-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 315.163596ms
Nov 29 19:49:35.423: INFO: At 2022-11-29 19:41:21 +0000 UTC - event for pod-97f1429b-37dd-406f-9602-1791035092e3: {kubelet k8s-agentpool1-16172903-vmss000000} Created: Created container write-pod
Nov 29 19:49:35.423: INFO: At 2022-11-29 19:41:21 +0000 UTC - event for pod-97f1429b-37dd-406f-9602-1791035092e3: {kubelet k8s-agentpool1-16172903-vmss000000} Started: Started container write-pod
Nov 29 19:49:35.423: INFO: At 2022-11-29 19:41:21 +0000 UTC - event for pod-97f1429b-37dd-406f-9602-1791035092e3: {kubelet k8s-agentpool1-16172903-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2"
Nov 29 19:49:35.423: INFO: At 2022-11-29 19:41:33 +0000 UTC - event for pod-97f1429b-37dd-406f-9602-1791035092e3: {kubelet k8s-agentpool1-16172903-vmss000000} Killing: Stopping container write-pod
Nov 29 19:49:35.423: INFO: At 2022-11-29 19:41:36 +0000 UTC - event for pod-645b1ce5-b71a-4174-930c-ae3f9a252a52: {attachdetach-controller } FailedAttachVolume: Multi-Attach error for volume "pvc-8ed24491-1106-4093-a2e3-7f686d1a1913" Volume is already exclusively attached to one node and can't be attached to another
Nov 29 19:49:35.423: INFO: At 2022-11-29 19:41:36 +0000 UTC - event for pod-645b1ce5-b71a-4174-930c-ae3f9a252a52: {attachdetach-controller } FailedAttachVolume: Multi-Attach error for volume "pvc-040f12d8-ebf6-49cd-b236-67c7e8d63e0a" Volume is already exclusively attached to one node and can't be attached to another
Nov 29 19:49:35.423: INFO: At 2022-11-29 19:41:36 +0000 UTC - event for pod-645b1ce5-b71a-4174-930c-ae3f9a252a52: {default-scheduler } Scheduled: Successfully assigned multivolume-7015/pod-645b1ce5-b71a-4174-930c-ae3f9a252a52 to k8s-agentpool1-16172903-vmss000001
Nov 29 19:49:35.423: INFO: At 2022-11-29 19:43:39 +0000 UTC - event for pod-645b1ce5-b71a-4174-930c-ae3f9a252a52: {kubelet k8s-agentpool1-16172903-vmss000001} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1 volume2], unattached volumes=[kube-api-access-htgnt volume1 volume2]: timed out waiting for the condition
Nov 29 19:49:35.532: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov 29 19:49:35.532: INFO: 
Nov 29 19:49:35.749: INFO: 
Logging node info for node k8s-agentpool1-16172903-vmss000000
... skipping 116 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on different node [Measurement]
    test/e2e/storage/testsuites/multivolume.go:168

    Nov 29 19:46:36.429: Unexpected error:
        <*errors.errorString | 0xc0027a81b0>: {
            s: "pod \"pod-645b1ce5-b71a-4174-930c-ae3f9a252a52\" is not Running: timed out waiting for the condition",
        }
        pod "pod-645b1ce5-b71a-4174-930c-ae3f9a252a52" is not Running: timed out waiting for the condition
    occurred

    test/e2e/storage/testsuites/multivolume.go:497
------------------------------
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":30,"completed":6,"skipped":510,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (xfs)][Slow] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:49:39.201: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 206 lines ...
Nov 29 19:41:41.884: INFO: >>> kubeConfig: /root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json
Nov 29 19:41:41.886: INFO: ExecWithOptions: Clientset creation
Nov 29 19:41:41.886: INFO: ExecWithOptions: execute(POST https://kubetest-2hnyryib.westeurope.cloudapp.azure.com/api/v1/namespaces/multivolume-5172/pods/pod-53925a62-26f0-480e-9296-556c6a39ddb5/exec?command=%2Fbin%2Fsh&command=-c&command=dd+if%3D%2Fmnt%2Fvolume2%2Ffile1.txt++bs%3D64+count%3D1+%7C+sha256sum+%7C+grep+-Fq+c1d89bc05d61179d2deebf9fcc170bce5cbf5cf740dda2bdb0f0478e780e819e&container=write-pod&container=write-pod&stderr=true&stdout=true)
Nov 29 19:41:42.819: INFO: Deleting pod "pod-53925a62-26f0-480e-9296-556c6a39ddb5" in namespace "multivolume-5172"
Nov 29 19:41:42.932: INFO: Wait up to 5m0s for pod "pod-53925a62-26f0-480e-9296-556c6a39ddb5" to be fully deleted
STEP: Creating pod on {Name: Selector:map[] Affinity:&Affinity{NodeAffinity:&NodeAffinity{RequiredDuringSchedulingIgnoredDuringExecution:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{},MatchFields:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:metadata.name,Operator:NotIn,Values:[k8s-agentpool1-16172903-vmss000000],},},},},},PreferredDuringSchedulingIgnoredDuringExecution:[]PreferredSchedulingTerm{},},PodAffinity:nil,PodAntiAffinity:nil,}} with multiple volumes
Nov 29 19:46:45.599: FAIL: Unexpected error:
    <*errors.errorString | 0xc0035265c0>: {
        s: "pod \"pod-086ae0de-de52-460e-b2a0-8361a6064797\" is not Running: timed out waiting for the condition",
    }
    pod "pod-086ae0de-de52-460e-b2a0-8361a6064797" is not Running: timed out waiting for the condition
occurred

... skipping 57 lines ...
Nov 29 19:49:44.626: INFO: At 2022-11-29 19:41:30 +0000 UTC - event for pod-53925a62-26f0-480e-9296-556c6a39ddb5: {kubelet k8s-agentpool1-16172903-vmss000000} SuccessfulMountVolume: MapVolume.MapPodDevice succeeded for volume "pvc-36ce4cf6-a4ff-4673-aec4-c1e246ee26e5" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-36ce4cf6-a4ff-4673-aec4-c1e246ee26e5/dev"
Nov 29 19:49:44.626: INFO: At 2022-11-29 19:41:31 +0000 UTC - event for pod-53925a62-26f0-480e-9296-556c6a39ddb5: {kubelet k8s-agentpool1-16172903-vmss000000} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/busybox:1.29-2"
Nov 29 19:49:44.626: INFO: At 2022-11-29 19:41:31 +0000 UTC - event for pod-53925a62-26f0-480e-9296-556c6a39ddb5: {kubelet k8s-agentpool1-16172903-vmss000000} Created: Created container write-pod
Nov 29 19:49:44.626: INFO: At 2022-11-29 19:41:31 +0000 UTC - event for pod-53925a62-26f0-480e-9296-556c6a39ddb5: {kubelet k8s-agentpool1-16172903-vmss000000} Started: Started container write-pod
Nov 29 19:49:44.626: INFO: At 2022-11-29 19:41:31 +0000 UTC - event for pod-53925a62-26f0-480e-9296-556c6a39ddb5: {kubelet k8s-agentpool1-16172903-vmss000000} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" in 275.913551ms
Nov 29 19:49:44.626: INFO: At 2022-11-29 19:41:42 +0000 UTC - event for pod-53925a62-26f0-480e-9296-556c6a39ddb5: {kubelet k8s-agentpool1-16172903-vmss000000} Killing: Stopping container write-pod
Nov 29 19:49:44.626: INFO: At 2022-11-29 19:41:44 +0000 UTC - event for pod-53925a62-26f0-480e-9296-556c6a39ddb5: {kubelet k8s-agentpool1-16172903-vmss000000} FailedKillPod: error killing pod: failed to "KillContainer" for "write-pod" with KillContainerError: "rpc error: code = Unknown desc = Error response from daemon: No such container: 0bb8e65c0fcb19657928ed226545b4153a5eb90413c4fff54f2c6e4996be817a"
Nov 29 19:49:44.626: INFO: At 2022-11-29 19:41:45 +0000 UTC - event for pod-086ae0de-de52-460e-b2a0-8361a6064797: {attachdetach-controller } FailedAttachVolume: Multi-Attach error for volume "pvc-e5567487-d946-4ff9-be0c-74254c5a7896" Volume is already exclusively attached to one node and can't be attached to another
Nov 29 19:49:44.626: INFO: At 2022-11-29 19:41:45 +0000 UTC - event for pod-086ae0de-de52-460e-b2a0-8361a6064797: {attachdetach-controller } FailedAttachVolume: Multi-Attach error for volume "pvc-36ce4cf6-a4ff-4673-aec4-c1e246ee26e5" Volume is already exclusively attached to one node and can't be attached to another
Nov 29 19:49:44.626: INFO: At 2022-11-29 19:41:45 +0000 UTC - event for pod-086ae0de-de52-460e-b2a0-8361a6064797: {default-scheduler } Scheduled: Successfully assigned multivolume-5172/pod-086ae0de-de52-460e-b2a0-8361a6064797 to k8s-agentpool1-16172903-vmss000002
Nov 29 19:49:44.626: INFO: At 2022-11-29 19:43:48 +0000 UTC - event for pod-086ae0de-de52-460e-b2a0-8361a6064797: {kubelet k8s-agentpool1-16172903-vmss000002} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume2 volume1], unattached volumes=[volume2 kube-api-access-45zqj volume1]: timed out waiting for the condition
Nov 29 19:49:44.736: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov 29 19:49:44.736: INFO: 
Nov 29 19:49:44.956: INFO: 
Logging node info for node k8s-agentpool1-16172903-vmss000000
... skipping 118 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on different node [Measurement]
    test/e2e/storage/testsuites/multivolume.go:248

    Nov 29 19:46:45.599: Unexpected error:
        <*errors.errorString | 0xc0035265c0>: {
            s: "pod \"pod-086ae0de-de52-460e-b2a0-8361a6064797\" is not Running: timed out waiting for the condition",
        }
        pod "pod-086ae0de-de52-460e-b2a0-8361a6064797" is not Running: timed out waiting for the condition
    occurred

    test/e2e/storage/testsuites/multivolume.go:497
------------------------------
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":24,"completed":4,"skipped":576,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:49:48.336: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 87 lines ...
STEP: creating a claim
Nov 29 19:41:48.071: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comlxpvs] to have phase Bound
Nov 29 19:41:48.182: INFO: PersistentVolumeClaim test.csi.azure.comlxpvs found but phase is Pending instead of Bound.
Nov 29 19:41:50.292: INFO: PersistentVolumeClaim test.csi.azure.comlxpvs found but phase is Pending instead of Bound.
Nov 29 19:41:52.403: INFO: PersistentVolumeClaim test.csi.azure.comlxpvs found and phase=Bound (4.332108047s)
STEP: Creating pod on {Name: Selector:map[] Affinity:nil} with multiple volumes
Nov 29 19:46:53.069: FAIL: Unexpected error:
    <*errors.errorString | 0xc0024f0af0>: {
        s: "pod \"pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90\" is not Running: timed out waiting for the condition",
    }
    pod "pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90" is not Running: timed out waiting for the condition
occurred

... skipping 51 lines ...
Nov 29 19:49:50.087: INFO: At 2022-11-29 19:41:48 +0000 UTC - event for test.csi.azure.comlxpvs: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "test.csi.azure.com" or manually created by system administrator
Nov 29 19:49:50.087: INFO: At 2022-11-29 19:41:50 +0000 UTC - event for test.csi.azure.comlxpvs: {test.csi.azure.com_k8s-agentpool1-16172903-vmss000002_0b832c14-6302-465a-aef7-e7c24ed830e6 } ProvisioningSucceeded: Successfully provisioned volume pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210
Nov 29 19:49:50.087: INFO: At 2022-11-29 19:41:52 +0000 UTC - event for pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90: {default-scheduler } Scheduled: Successfully assigned multivolume-4110/pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90 to k8s-agentpool1-16172903-vmss000002
Nov 29 19:49:50.087: INFO: At 2022-11-29 19:41:53 +0000 UTC - event for pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210" 
Nov 29 19:49:50.087: INFO: At 2022-11-29 19:41:54 +0000 UTC - event for pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c" 
Nov 29 19:49:50.087: INFO: At 2022-11-29 19:43:55 +0000 UTC - event for pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90: {kubelet k8s-agentpool1-16172903-vmss000002} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1 volume2], unattached volumes=[volume1 volume2 kube-api-access-dbr7x]: timed out waiting for the condition
Nov 29 19:49:50.087: INFO: At 2022-11-29 19:43:55 +0000 UTC - event for pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90: {kubelet k8s-agentpool1-16172903-vmss000002} FailedMount: MountVolume.MountDevice failed for volume "pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 29 19:49:50.087: INFO: At 2022-11-29 19:43:55 +0000 UTC - event for pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90: {kubelet k8s-agentpool1-16172903-vmss000002} FailedMount: MountVolume.MountDevice failed for volume "pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 29 19:49:50.087: INFO: At 2022-11-29 19:46:09 +0000 UTC - event for pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90: {kubelet k8s-agentpool1-16172903-vmss000002} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1 volume2], unattached volumes=[kube-api-access-dbr7x volume1 volume2]: timed out waiting for the condition
Nov 29 19:49:50.087: INFO: At 2022-11-29 19:46:52 +0000 UTC - event for pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90: {kubelet k8s-agentpool1-16172903-vmss000002} FailedMount: MountVolume.MountDevice failed for volume "pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210) to be attached to node (k8s-agentpool1-16172903-vmss000002): rpc error: code = Unknown desc = context deadline exceeded
Nov 29 19:49:50.087: INFO: At 2022-11-29 19:46:52 +0000 UTC - event for pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90: {kubelet k8s-agentpool1-16172903-vmss000002} FailedMount: MountVolume.MountDevice failed for volume "pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c" : rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c) to be attached to node (k8s-agentpool1-16172903-vmss000002): rpc error: code = Unknown desc = context deadline exceeded
Nov 29 19:49:50.087: INFO: At 2022-11-29 19:48:31 +0000 UTC - event for pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90: {kubelet k8s-agentpool1-16172903-vmss000002} FailedMount: MountVolume.MountDevice failed while expanding volume for volume "pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c" : mountVolume.NodeExpandVolume get PVC failed : persistentvolumeclaims "test.csi.azure.comm2mx6" is forbidden: User "system:node:k8s-agentpool1-16172903-vmss000002" cannot get resource "persistentvolumeclaims" in API group "" in the namespace "multivolume-4110": no relationship found between node 'k8s-agentpool1-16172903-vmss000002' and this object
Nov 29 19:49:50.087: INFO: At 2022-11-29 19:48:31 +0000 UTC - event for pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90: {kubelet k8s-agentpool1-16172903-vmss000002} FailedMount: MountVolume.MountDevice failed while expanding volume for volume "pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210" : mountVolume.NodeExpandVolume get PVC failed : persistentvolumeclaims "test.csi.azure.comlxpvs" is forbidden: User "system:node:k8s-agentpool1-16172903-vmss000002" cannot get resource "persistentvolumeclaims" in API group "" in the namespace "multivolume-4110": no relationship found between node 'k8s-agentpool1-16172903-vmss000002' and this object
Nov 29 19:49:50.198: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov 29 19:49:50.198: INFO: 
Nov 29 19:49:50.416: INFO: 
Logging node info for node k8s-agentpool1-16172903-vmss000000
Nov 29 19:49:50.526: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-16172903-vmss000000    940202f6-ff3a-43ea-9cda-b2ab8d693901 14838 0 2022-11-29 19:20:56 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D8s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westeurope failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-2hnyryib kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-16172903-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_D8s_v3 storageprofile:managed storagetier:Premium_LRS topology.kubernetes.io/region:westeurope topology.kubernetes.io/zone:0 topology.test.csi.azure.com/zone:] map[csi.volume.kubernetes.io/nodeid:{"test.csi.azure.com":"k8s-agentpool1-16172903-vmss000000"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2022-11-29 19:20:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{}}}} } {kubectl-label Update v1 2022-11-29 19:21:07 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kube-controller-manager Update v1 2022-11-29 19:21:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.1.0/24\"":{}}}} } {cloud-controller-manager Update v1 2022-11-29 19:22:06 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {cloud-node-manager Update v1 2022-11-29 19:22:29 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2022-11-29 19:41:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.test.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2022-11-29 19:43:50 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-16172903-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{31025332224 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{33672720384 0} {<nil>} 32883516Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{27922798956 0} {<nil>} 27922798956 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{32886288384 0} {<nil>} 32115516Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-29 19:22:06 +0000 UTC,LastTransitionTime:2022-11-29 19:22:06 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-29 19:48:43 +0000 UTC,LastTransitionTime:2022-11-29 19:20:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-29 19:48:43 +0000 UTC,LastTransitionTime:2022-11-29 19:20:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-29 19:48:43 +0000 UTC,LastTransitionTime:2022-11-29 19:20:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-29 19:48:43 +0000 UTC,LastTransitionTime:2022-11-29 19:21:27 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool1-16172903-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:c7b774489805482ebad138ac4327e3ac,SystemUUID:adde590a-c6db-fe44-a9c1-9107b736d660,BootID:be9e71cf-5fad-4226-93ec-30e3071df7b1,KernelVersion:5.4.0-1094-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.14+azure-1,KubeletVersion:v1.23.14,KubeProxyVersion:v1.23.14,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:253346057,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:301ecdd545f77e58d25b13c391e8db2de9842f836906f7daad90154d3506007c k8sprow.azurecr.io/azuredisk-csi:latest-v2-c13cf58a5ca48aed0e92b8b1ecf1b82eae2479e6],SizeBytes:142874872,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:f5241226198f5a54d22540acf2b3933ea0f49458f90c51fc75833d0c428687b8 k8s.gcr.io/e2e-test-images/agnhost:2.36],SizeBytes:125426639,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:efa8e4b7776fa9fb2279745008780cefec632ee5fdb081e0f6e234ea4b8bbf61 mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.14],SizeBytes:112312488,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:949db4eea46ea31f9afa0b49ad0b55f7a31f91aa03b408e0ac9b7720c8ef7d29 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.11],SizeBytes:51376235,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:72fbc9d17947d1a54c09a16671e6810fd7c0d8f09c0502fdbdd212c2192451a7 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.14],SizeBytes:51044459,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:075ea1f8270312350f1396ab6677251e803e61a523822d5abfa5e6acd180cfab mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.11],SizeBytes:50806891,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:eb7d329559d954093f7a234bc0ff322764de48a0b2a1007229032fb53e7bb9eb mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.24.0],SizeBytes:49070187,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:2dfff4dc10f6d25b0a34c01bf4d166540cbf1ff2bccd52635350039138eb23c4 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.18],SizeBytes:48775275,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3e23b9ab6bfee42a1751019c21a60aa474964bd4cba00cbed5055430424242cc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.24.0],SizeBytes:48500843,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:acec9ee6626c2cdea89107ba191ae36e6f60339c9b3b3215b5a59bf0e2e730ba mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.14],SizeBytes:47853675,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:759ac7020e982884b99121583bd577658b5663b1c1ed69746356aa6ce9879da4 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.18],SizeBytes:45850731,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:2fbd1e1a0538a06f2061afd45975df70c942654aa7f86e920720169ee439c2d6 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.5.1],SizeBytes:19743433,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:31547791294872570393470991481c2477a311031d3a03e0ae54eb164347dc34 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0],SizeBytes:17857485,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-24ce38c8-1b18-4a71-95f6-396fd8f3daa4 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-edb50ce2-6d0c-4bed-846a-8b90f0310d48],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-e5567487-d946-4ff9-be0c-74254c5a7896,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-d9209029-cc60-45be-8d4f-411d67de4846,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-24ce38c8-1b18-4a71-95f6-396fd8f3daa4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-8ed24491-1106-4093-a2e3-7f686d1a1913,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-040f12d8-ebf6-49cd-b236-67c7e8d63e0a,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-36ce4cf6-a4ff-4673-aec4-c1e246ee26e5,DevicePath:,},},Config:nil,},}
Nov 29 19:49:50.527: INFO: 
... skipping 116 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on the same node [Measurement]
    test/e2e/storage/testsuites/multivolume.go:209

    Nov 29 19:46:53.069: Unexpected error:
        <*errors.errorString | 0xc0024f0af0>: {
            s: "pod \"pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90\" is not Running: timed out waiting for the condition",
        }
        pod "pod-0e1f904a-a092-4c6c-8dd3-a12e32708e90" is not Running: timed out waiting for the condition
    occurred

    test/e2e/storage/testsuites/multivolume.go:497
------------------------------
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":27,"completed":7,"skipped":704,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:49:53.750: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 91 lines ...
Nov 29 19:43:44.024: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comdqtqw] to have phase Bound
Nov 29 19:43:44.134: INFO: PersistentVolumeClaim test.csi.azure.comdqtqw found but phase is Pending instead of Bound.
Nov 29 19:43:46.245: INFO: PersistentVolumeClaim test.csi.azure.comdqtqw found but phase is Pending instead of Bound.
Nov 29 19:43:48.358: INFO: PersistentVolumeClaim test.csi.azure.comdqtqw found and phase=Bound (4.334042326s)
STEP: Creating pod exec-volume-test-dynamicpv-9t2r
STEP: Creating a pod to test exec-volume-test
Nov 29 19:43:48.689: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-9t2r" in namespace "volume-5543" to be "Succeeded or Failed"
Nov 29 19:43:48.799: INFO: Pod "exec-volume-test-dynamicpv-9t2r": Phase="Pending", Reason="", readiness=false. Elapsed: 109.107296ms
Nov 29 19:43:50.909: INFO: Pod "exec-volume-test-dynamicpv-9t2r": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219738917s
Nov 29 19:43:53.020: INFO: Pod "exec-volume-test-dynamicpv-9t2r": Phase="Pending", Reason="", readiness=false. Elapsed: 4.330618844s
Nov 29 19:43:55.131: INFO: Pod "exec-volume-test-dynamicpv-9t2r": Phase="Pending", Reason="", readiness=false. Elapsed: 6.441504311s
Nov 29 19:43:57.241: INFO: Pod "exec-volume-test-dynamicpv-9t2r": Phase="Pending", Reason="", readiness=false. Elapsed: 8.551910356s
Nov 29 19:43:59.352: INFO: Pod "exec-volume-test-dynamicpv-9t2r": Phase="Pending", Reason="", readiness=false. Elapsed: 10.662657372s
... skipping 129 lines ...
Nov 29 19:48:33.941: INFO: Pod "exec-volume-test-dynamicpv-9t2r": Phase="Pending", Reason="", readiness=false. Elapsed: 4m45.25146001s
Nov 29 19:48:36.052: INFO: Pod "exec-volume-test-dynamicpv-9t2r": Phase="Pending", Reason="", readiness=false. Elapsed: 4m47.363001842s
Nov 29 19:48:38.165: INFO: Pod "exec-volume-test-dynamicpv-9t2r": Phase="Pending", Reason="", readiness=false. Elapsed: 4m49.475173027s
Nov 29 19:48:40.276: INFO: Pod "exec-volume-test-dynamicpv-9t2r": Phase="Pending", Reason="", readiness=false. Elapsed: 4m51.586681671s
Nov 29 19:48:42.387: INFO: Pod "exec-volume-test-dynamicpv-9t2r": Phase="Succeeded", Reason="", readiness=false. Elapsed: 4m53.697034645s
STEP: Saw pod success
Nov 29 19:48:42.387: INFO: Pod "exec-volume-test-dynamicpv-9t2r" satisfied condition "Succeeded or Failed"
Nov 29 19:48:42.497: INFO: Trying to get logs from node k8s-agentpool1-16172903-vmss000000 pod exec-volume-test-dynamicpv-9t2r container exec-container-dynamicpv-9t2r: <nil>
STEP: delete the pod
Nov 29 19:48:42.735: INFO: Waiting for pod exec-volume-test-dynamicpv-9t2r to disappear
Nov 29 19:48:42.849: INFO: Pod exec-volume-test-dynamicpv-9t2r no longer exists
STEP: Deleting pod exec-volume-test-dynamicpv-9t2r
Nov 29 19:48:42.849: INFO: Deleting pod "exec-volume-test-dynamicpv-9t2r" in namespace "volume-5543"
... skipping 27 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should allow exec of files on the volume
    test/e2e/storage/testsuites/volumes.go:198
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should allow exec of files on the volume","total":39,"completed":7,"skipped":549,"failed":0}

SS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 19:49:54.962: INFO: >>> kubeConfig: /root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename topology
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  test/e2e/storage/testsuites/topology.go:194
Nov 29 19:49:55.728: INFO: Driver didn't provide topology keys -- skipping
[AfterEach] [Testpattern: Dynamic PV (delayed binding)] topology
  test/e2e/framework/framework.go:188
Nov 29 19:49:55.728: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "topology-6389" for this suite.


S [SKIPPING] [0.987 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (delayed binding)] topology
  test/e2e/storage/framework/testsuite.go:50
    should fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement]
    test/e2e/storage/testsuites/topology.go:194

    Driver didn't provide topology keys -- skipping

    test/e2e/storage/testsuites/topology.go:126
------------------------------
... skipping 237 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:168
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":29,"completed":6,"skipped":595,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using directory as subpath [Slow]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  
  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
  test/e2e/storage/testsuites/snapshottable.go:278
... skipping 17 lines ...
Nov 29 19:41:11.490: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.compnhvt] to have phase Bound
Nov 29 19:41:11.599: INFO: PersistentVolumeClaim test.csi.azure.compnhvt found but phase is Pending instead of Bound.
Nov 29 19:41:13.708: INFO: PersistentVolumeClaim test.csi.azure.compnhvt found but phase is Pending instead of Bound.
Nov 29 19:41:15.816: INFO: PersistentVolumeClaim test.csi.azure.compnhvt found and phase=Bound (4.326542598s)
STEP: [init] starting a pod to use the claim
STEP: [init] check pod success
Nov 29 19:41:16.255: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-ws7dz" in namespace "snapshotting-3992" to be "Succeeded or Failed"
Nov 29 19:41:16.363: INFO: Pod "pvc-snapshottable-tester-ws7dz": Phase="Pending", Reason="", readiness=false. Elapsed: 108.544382ms
Nov 29 19:41:18.474: INFO: Pod "pvc-snapshottable-tester-ws7dz": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218779032s
Nov 29 19:41:20.583: INFO: Pod "pvc-snapshottable-tester-ws7dz": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327996216s
Nov 29 19:41:22.695: INFO: Pod "pvc-snapshottable-tester-ws7dz": Phase="Pending", Reason="", readiness=false. Elapsed: 6.440420479s
Nov 29 19:41:24.806: INFO: Pod "pvc-snapshottable-tester-ws7dz": Phase="Pending", Reason="", readiness=false. Elapsed: 8.551437889s
Nov 29 19:41:26.915: INFO: Pod "pvc-snapshottable-tester-ws7dz": Phase="Pending", Reason="", readiness=false. Elapsed: 10.660369372s
Nov 29 19:41:29.030: INFO: Pod "pvc-snapshottable-tester-ws7dz": Phase="Pending", Reason="", readiness=false. Elapsed: 12.775222654s
Nov 29 19:41:31.139: INFO: Pod "pvc-snapshottable-tester-ws7dz": Phase="Pending", Reason="", readiness=false. Elapsed: 14.884630389s
Nov 29 19:41:33.254: INFO: Pod "pvc-snapshottable-tester-ws7dz": Phase="Pending", Reason="", readiness=false. Elapsed: 16.999596202s
Nov 29 19:41:35.363: INFO: Pod "pvc-snapshottable-tester-ws7dz": Phase="Pending", Reason="", readiness=false. Elapsed: 19.108579523s
Nov 29 19:41:37.474: INFO: Pod "pvc-snapshottable-tester-ws7dz": Phase="Pending", Reason="", readiness=false. Elapsed: 21.219501054s
Nov 29 19:41:39.586: INFO: Pod "pvc-snapshottable-tester-ws7dz": Phase="Succeeded", Reason="", readiness=false. Elapsed: 23.331618403s
STEP: Saw pod success
Nov 29 19:41:39.586: INFO: Pod "pvc-snapshottable-tester-ws7dz" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
Nov 29 19:41:39.698: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.compnhvt] to have phase Bound
Nov 29 19:41:39.809: INFO: PersistentVolumeClaim test.csi.azure.compnhvt found and phase=Bound (110.478546ms)
STEP: [init] checking the PV
STEP: [init] deleting the pod
Nov 29 19:41:40.164: INFO: Pod pvc-snapshottable-tester-ws7dz has the following logs: 
... skipping 15 lines ...
Nov 29 19:41:52.041: INFO: received snapshotStatus map[boundVolumeSnapshotContentName:snapcontent-806955ed-4d66-4876-8a87-5be14abdfe10 creationTime:2022-11-29T19:41:47Z readyToUse:true restoreSize:5Gi]
Nov 29 19:41:52.041: INFO: snapshotContentName snapcontent-806955ed-4d66-4876-8a87-5be14abdfe10
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Nov 29 19:41:52.480: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-mxxgw" in namespace "snapshotting-3992" to be "Succeeded or Failed"
Nov 29 19:41:52.589: INFO: Pod "pvc-snapshottable-data-tester-mxxgw": Phase="Pending", Reason="", readiness=false. Elapsed: 108.389718ms
Nov 29 19:41:54.699: INFO: Pod "pvc-snapshottable-data-tester-mxxgw": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218338321s
Nov 29 19:41:56.808: INFO: Pod "pvc-snapshottable-data-tester-mxxgw": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327723007s
Nov 29 19:41:58.918: INFO: Pod "pvc-snapshottable-data-tester-mxxgw": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437556319s
Nov 29 19:42:01.030: INFO: Pod "pvc-snapshottable-data-tester-mxxgw": Phase="Pending", Reason="", readiness=false. Elapsed: 8.54996338s
Nov 29 19:42:03.140: INFO: Pod "pvc-snapshottable-data-tester-mxxgw": Phase="Pending", Reason="", readiness=false. Elapsed: 10.660074257s
... skipping 227 lines ...
Nov 29 19:50:04.457: INFO: Pod "pvc-snapshottable-data-tester-mxxgw": Phase="Pending", Reason="", readiness=false. Elapsed: 8m11.976635994s
Nov 29 19:50:06.567: INFO: Pod "pvc-snapshottable-data-tester-mxxgw": Phase="Pending", Reason="", readiness=false. Elapsed: 8m14.086320134s
Nov 29 19:50:08.677: INFO: Pod "pvc-snapshottable-data-tester-mxxgw": Phase="Pending", Reason="", readiness=false. Elapsed: 8m16.196265976s
Nov 29 19:50:10.788: INFO: Pod "pvc-snapshottable-data-tester-mxxgw": Phase="Pending", Reason="", readiness=false. Elapsed: 8m18.307712743s
Nov 29 19:50:12.898: INFO: Pod "pvc-snapshottable-data-tester-mxxgw": Phase="Succeeded", Reason="", readiness=false. Elapsed: 8m20.418187756s
STEP: Saw pod success
Nov 29 19:50:12.899: INFO: Pod "pvc-snapshottable-data-tester-mxxgw" satisfied condition "Succeeded or Failed"
Nov 29 19:50:13.121: INFO: Pod pvc-snapshottable-data-tester-mxxgw has the following logs: 
Nov 29 19:50:13.121: INFO: Deleting pod "pvc-snapshottable-data-tester-mxxgw" in namespace "snapshotting-3992"
Nov 29 19:50:13.238: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-mxxgw" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the snapshot
Nov 29 19:51:47.807: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-2hnyryib.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-3992 exec restored-pvc-tester-mrprx --namespace=snapshotting-3992 -- cat /mnt/test/data'
... skipping 47 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        test/e2e/storage/testsuites/snapshottable.go:278
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":32,"completed":7,"skipped":273,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:52:32.529: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 90 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should be able to unmount after the subpath directory is deleted [LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:447
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [LinuxOnly]","total":39,"completed":8,"skipped":551,"failed":0}

SSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should support existing directory
  test/e2e/storage/testsuites/subpath.go:207
... skipping 17 lines ...
Nov 29 19:49:49.885: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.compf5v8] to have phase Bound
Nov 29 19:49:49.994: INFO: PersistentVolumeClaim test.csi.azure.compf5v8 found but phase is Pending instead of Bound.
Nov 29 19:49:52.114: INFO: PersistentVolumeClaim test.csi.azure.compf5v8 found but phase is Pending instead of Bound.
Nov 29 19:49:54.225: INFO: PersistentVolumeClaim test.csi.azure.compf5v8 found and phase=Bound (4.340314837s)
STEP: Creating pod pod-subpath-test-dynamicpv-9prl
STEP: Creating a pod to test subpath
Nov 29 19:49:54.560: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-9prl" in namespace "provisioning-5472" to be "Succeeded or Failed"
Nov 29 19:49:54.677: INFO: Pod "pod-subpath-test-dynamicpv-9prl": Phase="Pending", Reason="", readiness=false. Elapsed: 117.084713ms
Nov 29 19:49:56.788: INFO: Pod "pod-subpath-test-dynamicpv-9prl": Phase="Pending", Reason="", readiness=false. Elapsed: 2.228496344s
Nov 29 19:49:58.899: INFO: Pod "pod-subpath-test-dynamicpv-9prl": Phase="Pending", Reason="", readiness=false. Elapsed: 4.339085996s
Nov 29 19:50:01.015: INFO: Pod "pod-subpath-test-dynamicpv-9prl": Phase="Pending", Reason="", readiness=false. Elapsed: 6.454926282s
Nov 29 19:50:03.130: INFO: Pod "pod-subpath-test-dynamicpv-9prl": Phase="Pending", Reason="", readiness=false. Elapsed: 8.570191795s
Nov 29 19:50:05.241: INFO: Pod "pod-subpath-test-dynamicpv-9prl": Phase="Pending", Reason="", readiness=false. Elapsed: 10.681674438s
... skipping 7 lines ...
Nov 29 19:50:22.145: INFO: Pod "pod-subpath-test-dynamicpv-9prl": Phase="Pending", Reason="", readiness=false. Elapsed: 27.585570025s
Nov 29 19:50:24.256: INFO: Pod "pod-subpath-test-dynamicpv-9prl": Phase="Pending", Reason="", readiness=false. Elapsed: 29.69618814s
Nov 29 19:50:26.366: INFO: Pod "pod-subpath-test-dynamicpv-9prl": Phase="Pending", Reason="", readiness=false. Elapsed: 31.806606679s
Nov 29 19:50:28.477: INFO: Pod "pod-subpath-test-dynamicpv-9prl": Phase="Pending", Reason="", readiness=false. Elapsed: 33.917581367s
Nov 29 19:50:30.588: INFO: Pod "pod-subpath-test-dynamicpv-9prl": Phase="Succeeded", Reason="", readiness=false. Elapsed: 36.028178475s
STEP: Saw pod success
Nov 29 19:50:30.593: INFO: Pod "pod-subpath-test-dynamicpv-9prl" satisfied condition "Succeeded or Failed"
Nov 29 19:50:30.710: INFO: Trying to get logs from node k8s-agentpool1-16172903-vmss000001 pod pod-subpath-test-dynamicpv-9prl container test-container-volume-dynamicpv-9prl: <nil>
STEP: delete the pod
Nov 29 19:50:30.951: INFO: Waiting for pod pod-subpath-test-dynamicpv-9prl to disappear
Nov 29 19:50:31.061: INFO: Pod pod-subpath-test-dynamicpv-9prl no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-9prl
Nov 29 19:50:31.061: INFO: Deleting pod "pod-subpath-test-dynamicpv-9prl" in namespace "provisioning-5472"
... skipping 41 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support existing directory
    test/e2e/storage/testsuites/subpath.go:207
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory","total":24,"completed":5,"skipped":767,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:52:44.975: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 87 lines ...
  test/e2e/storage/framework/testsuite.go:50
    should create read/write inline ephemeral volume
    test/e2e/storage/testsuites/ephemeral.go:196
------------------------------
SS
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read/write inline ephemeral volume","total":49,"completed":5,"skipped":208,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]}
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:52:45.033: INFO: Distro debian doesn't support ntfs -- skipping
[AfterEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/framework/framework.go:188


S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:280

    Distro debian doesn't support ntfs -- skipping

    test/e2e/storage/framework/testsuite.go:127
------------------------------
... skipping 187 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should store data
    test/e2e/storage/testsuites/volumes.go:161
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data","total":27,"completed":8,"skipped":881,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning 
  should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
  test/e2e/storage/testsuites/provisioning.go:208
... skipping 120 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
    test/e2e/storage/testsuites/provisioning.go:208
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":30,"completed":7,"skipped":787,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node"]}

SSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
... skipping 67 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:242

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 64 lines ...
Nov 29 19:52:45.554: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comd55mn] to have phase Bound
Nov 29 19:52:45.666: INFO: PersistentVolumeClaim test.csi.azure.comd55mn found but phase is Pending instead of Bound.
Nov 29 19:52:47.777: INFO: PersistentVolumeClaim test.csi.azure.comd55mn found but phase is Pending instead of Bound.
Nov 29 19:52:49.888: INFO: PersistentVolumeClaim test.csi.azure.comd55mn found and phase=Bound (4.333638287s)
STEP: Creating pod pod-subpath-test-dynamicpv-gj4q
STEP: Creating a pod to test multi_subpath
Nov 29 19:52:50.222: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-gj4q" in namespace "provisioning-2713" to be "Succeeded or Failed"
Nov 29 19:52:50.336: INFO: Pod "pod-subpath-test-dynamicpv-gj4q": Phase="Pending", Reason="", readiness=false. Elapsed: 113.154925ms
Nov 29 19:52:52.448: INFO: Pod "pod-subpath-test-dynamicpv-gj4q": Phase="Pending", Reason="", readiness=false. Elapsed: 2.225688839s
Nov 29 19:52:54.559: INFO: Pod "pod-subpath-test-dynamicpv-gj4q": Phase="Pending", Reason="", readiness=false. Elapsed: 4.336094783s
Nov 29 19:52:56.669: INFO: Pod "pod-subpath-test-dynamicpv-gj4q": Phase="Pending", Reason="", readiness=false. Elapsed: 6.446602289s
Nov 29 19:52:58.779: INFO: Pod "pod-subpath-test-dynamicpv-gj4q": Phase="Pending", Reason="", readiness=false. Elapsed: 8.556878319s
Nov 29 19:53:00.889: INFO: Pod "pod-subpath-test-dynamicpv-gj4q": Phase="Pending", Reason="", readiness=false. Elapsed: 10.666833247s
... skipping 15 lines ...
Nov 29 19:53:34.656: INFO: Pod "pod-subpath-test-dynamicpv-gj4q": Phase="Pending", Reason="", readiness=false. Elapsed: 44.433678204s
Nov 29 19:53:36.766: INFO: Pod "pod-subpath-test-dynamicpv-gj4q": Phase="Pending", Reason="", readiness=false. Elapsed: 46.543585848s
Nov 29 19:53:38.876: INFO: Pod "pod-subpath-test-dynamicpv-gj4q": Phase="Pending", Reason="", readiness=false. Elapsed: 48.653279011s
Nov 29 19:53:40.986: INFO: Pod "pod-subpath-test-dynamicpv-gj4q": Phase="Pending", Reason="", readiness=false. Elapsed: 50.763649614s
Nov 29 19:53:43.096: INFO: Pod "pod-subpath-test-dynamicpv-gj4q": Phase="Succeeded", Reason="", readiness=false. Elapsed: 52.873201782s
STEP: Saw pod success
Nov 29 19:53:43.096: INFO: Pod "pod-subpath-test-dynamicpv-gj4q" satisfied condition "Succeeded or Failed"
Nov 29 19:53:43.205: INFO: Trying to get logs from node k8s-agentpool1-16172903-vmss000000 pod pod-subpath-test-dynamicpv-gj4q container test-container-subpath-dynamicpv-gj4q: <nil>
STEP: delete the pod
Nov 29 19:53:43.458: INFO: Waiting for pod pod-subpath-test-dynamicpv-gj4q to disappear
Nov 29 19:53:43.567: INFO: Pod pod-subpath-test-dynamicpv-gj4q no longer exists
STEP: Deleting pod
Nov 29 19:53:43.567: INFO: Deleting pod "pod-subpath-test-dynamicpv-gj4q" in namespace "provisioning-2713"
... skipping 27 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support creating multiple subpath from same volumes [Slow]
    test/e2e/storage/testsuites/subpath.go:296
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support creating multiple subpath from same volumes [Slow]","total":39,"completed":9,"skipped":568,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] 
  should concurrently access the single read-only volume from pods on the same node
  test/e2e/storage/testsuites/multivolume.go:423
... skipping 88 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single read-only volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:423
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":30,"completed":8,"skipped":905,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] 
  should access to two volumes with the same volume mode and retain data across pod recreation on the same node
  test/e2e/storage/testsuites/multivolume.go:138
... skipping 219 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:138
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":29,"completed":7,"skipped":633,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using directory as subpath [Slow]"]}

SSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource]
... skipping 62 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:258

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 8 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:258

    Distro debian doesn't support ntfs -- skipping

    test/e2e/storage/framework/testsuite.go:127
------------------------------
... skipping 75 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should support two pods which have the same volume definition
    test/e2e/storage/testsuites/ephemeral.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support two pods which have the same volume definition","total":32,"completed":8,"skipped":344,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] 
  should access to two volumes with different volume mode and retain data across pod recreation on different node
  test/e2e/storage/testsuites/multivolume.go:248
... skipping 189 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on different node
    test/e2e/storage/testsuites/multivolume.go:248
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":27,"completed":9,"skipped":909,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node"]}

SS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:56:54.679: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 180 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:323
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":24,"completed":6,"skipped":902,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
Nov 29 19:57:06.394: INFO: Running AfterSuite actions on all nodes
Nov 29 19:57:06.394: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2
Nov 29 19:57:06.394: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2
... skipping 27 lines ...
Nov 29 19:54:57.060: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comg6vcx] to have phase Bound
Nov 29 19:54:57.170: INFO: PersistentVolumeClaim test.csi.azure.comg6vcx found but phase is Pending instead of Bound.
Nov 29 19:54:59.279: INFO: PersistentVolumeClaim test.csi.azure.comg6vcx found but phase is Pending instead of Bound.
Nov 29 19:55:01.389: INFO: PersistentVolumeClaim test.csi.azure.comg6vcx found and phase=Bound (4.328859787s)
STEP: Creating pod exec-volume-test-dynamicpv-pdhs
STEP: Creating a pod to test exec-volume-test
Nov 29 19:55:01.721: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-pdhs" in namespace "volume-6866" to be "Succeeded or Failed"
Nov 29 19:55:01.834: INFO: Pod "exec-volume-test-dynamicpv-pdhs": Phase="Pending", Reason="", readiness=false. Elapsed: 112.81891ms
Nov 29 19:55:03.947: INFO: Pod "exec-volume-test-dynamicpv-pdhs": Phase="Pending", Reason="", readiness=false. Elapsed: 2.226207902s
Nov 29 19:55:06.058: INFO: Pod "exec-volume-test-dynamicpv-pdhs": Phase="Pending", Reason="", readiness=false. Elapsed: 4.336491741s
Nov 29 19:55:08.167: INFO: Pod "exec-volume-test-dynamicpv-pdhs": Phase="Pending", Reason="", readiness=false. Elapsed: 6.445777767s
Nov 29 19:55:10.277: INFO: Pod "exec-volume-test-dynamicpv-pdhs": Phase="Pending", Reason="", readiness=false. Elapsed: 8.555646617s
Nov 29 19:55:12.390: INFO: Pod "exec-volume-test-dynamicpv-pdhs": Phase="Pending", Reason="", readiness=false. Elapsed: 10.66867398s
... skipping 34 lines ...
Nov 29 19:56:26.256: INFO: Pod "exec-volume-test-dynamicpv-pdhs": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.534953417s
Nov 29 19:56:28.366: INFO: Pod "exec-volume-test-dynamicpv-pdhs": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.645115541s
Nov 29 19:56:30.480: INFO: Pod "exec-volume-test-dynamicpv-pdhs": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.759269934s
Nov 29 19:56:32.590: INFO: Pod "exec-volume-test-dynamicpv-pdhs": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.868915503s
Nov 29 19:56:34.699: INFO: Pod "exec-volume-test-dynamicpv-pdhs": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m32.978415659s
STEP: Saw pod success
Nov 29 19:56:34.700: INFO: Pod "exec-volume-test-dynamicpv-pdhs" satisfied condition "Succeeded or Failed"
Nov 29 19:56:34.812: INFO: Trying to get logs from node k8s-agentpool1-16172903-vmss000000 pod exec-volume-test-dynamicpv-pdhs container exec-container-dynamicpv-pdhs: <nil>
STEP: delete the pod
Nov 29 19:56:35.081: INFO: Waiting for pod exec-volume-test-dynamicpv-pdhs to disappear
Nov 29 19:56:35.191: INFO: Pod exec-volume-test-dynamicpv-pdhs no longer exists
STEP: Deleting pod exec-volume-test-dynamicpv-pdhs
Nov 29 19:56:35.191: INFO: Deleting pod "exec-volume-test-dynamicpv-pdhs" in namespace "volume-6866"
... skipping 27 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] volumes
  test/e2e/storage/framework/testsuite.go:50
    should allow exec of files on the volume
    test/e2e/storage/testsuites/volumes.go:198
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes should allow exec of files on the volume","total":39,"completed":10,"skipped":617,"failed":0}

SSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:57:47.652: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 169 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:298
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":30,"completed":9,"skipped":950,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node"]}

SSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] volume-expand
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:58:05.160: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 139 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:323
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":49,"completed":6,"skipped":260,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
... skipping 195 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:298
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":32,"completed":9,"skipped":370,"failed":0}

SSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 19:59:06.898: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 3 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:269

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 93 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single read-only volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:423
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":29,"completed":8,"skipped":775,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using directory as subpath [Slow]"]}

SSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 20:00:08.634: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 3 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:269

    Distro debian doesn't support ntfs -- skipping

    test/e2e/storage/framework/testsuite.go:127
------------------------------
... skipping 82 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.002 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:258

    Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 124 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents","total":49,"completed":7,"skipped":304,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]}

SSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (ntfs)][Feature:Windows] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 29 20:01:37.682: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 89 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] volumeIO
  test/e2e/storage/framework/testsuite.go:50
    should write files of various sizes, verify size, validate content [Slow]
    test/e2e/storage/testsuites/volume_io.go:149
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumeIO should write files of various sizes, verify size, validate content [Slow]","total":29,"completed":9,"skipped":865,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using directory as subpath [Slow]"]}

SSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Ephemeral Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  
  should check snapshot fields, check restore correctly works, check deletion (ephemeral)
  test/e2e/storage/testsuites/snapshottable.go:177
... skipping 10 lines ...
[It] should check snapshot fields, check restore correctly works, check deletion (ephemeral)
  test/e2e/storage/testsuites/snapshottable.go:177
Nov 29 19:59:07.731: INFO: Creating resource for dynamic PV
Nov 29 19:59:07.731: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass snapshotting-5313-e2e-scm84g6
STEP: [init] starting a pod to use the claim
Nov 29 19:59:07.954: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-b5rkw" in namespace "snapshotting-5313" to be "Succeeded or Failed"
Nov 29 19:59:08.063: INFO: Pod "pvc-snapshottable-tester-b5rkw": Phase="Pending", Reason="", readiness=false. Elapsed: 108.545885ms
Nov 29 19:59:10.173: INFO: Pod "pvc-snapshottable-tester-b5rkw": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218260401s
Nov 29 19:59:12.285: INFO: Pod "pvc-snapshottable-tester-b5rkw": Phase="Pending", Reason="", readiness=false. Elapsed: 4.330776382s
Nov 29 19:59:14.396: INFO: Pod "pvc-snapshottable-tester-b5rkw": Phase="Pending", Reason="", readiness=false. Elapsed: 6.441422657s
Nov 29 19:59:16.505: INFO: Pod "pvc-snapshottable-tester-b5rkw": Phase="Pending", Reason="", readiness=false. Elapsed: 8.550765589s
Nov 29 19:59:18.615: INFO: Pod "pvc-snapshottable-tester-b5rkw": Phase="Pending", Reason="", readiness=false. Elapsed: 10.660502997s
... skipping 24 lines ...
Nov 29 20:00:11.368: INFO: Pod "pvc-snapshottable-tester-b5rkw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.413590097s
Nov 29 20:00:13.478: INFO: Pod "pvc-snapshottable-tester-b5rkw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.523791703s
Nov 29 20:00:15.589: INFO: Pod "pvc-snapshottable-tester-b5rkw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.634516853s
Nov 29 20:00:17.701: INFO: Pod "pvc-snapshottable-tester-b5rkw": Phase="Pending", Reason="", readiness=false. Elapsed: 1m9.746008797s
Nov 29 20:00:19.810: INFO: Pod "pvc-snapshottable-tester-b5rkw": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m11.855466944s
STEP: Saw pod success
Nov 29 20:00:19.810: INFO: Pod "pvc-snapshottable-tester-b5rkw" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
STEP: creating a SnapshotClass
STEP: creating a dynamic VolumeSnapshot
Nov 29 20:00:20.250: INFO: Waiting up to 5m0s for VolumeSnapshot snapshot-9qp2k to become ready
Nov 29 20:00:20.359: INFO: VolumeSnapshot snapshot-9qp2k found but is not ready.
Nov 29 20:00:22.470: INFO: VolumeSnapshot snapshot-9qp2k found but is not ready.
... skipping 40 lines ...
Nov 29 20:01:18.451: INFO: volumesnapshotcontents snapcontent-3637ef01-9989-4891-87b9-0ee4b85c059a has been found and is not deleted
Nov 29 20:01:19.561: INFO: volumesnapshotcontents snapcontent-3637ef01-9989-4891-87b9-0ee4b85c059a has been found and is not deleted
Nov 29 20:01:20.672: INFO: volumesnapshotcontents snapcontent-3637ef01-9989-4891-87b9-0ee4b85c059a has been found and is not deleted
Nov 29 20:01:21.782: INFO: volumesnapshotcontents snapcontent-3637ef01-9989-4891-87b9-0ee4b85c059a has been found and is not deleted
Nov 29 20:01:22.893: INFO: volumesnapshotcontents snapcontent-3637ef01-9989-4891-87b9-0ee4b85c059a has been found and is not deleted
Nov 29 20:01:24.002: INFO: volumesnapshotcontents snapcontent-3637ef01-9989-4891-87b9-0ee4b85c059a has been found and is not deleted
Nov 29 20:01:25.003: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 29 20:01:25.140: INFO: Pod restored-pvc-tester-llzn4 has the following logs: 
Nov 29 20:01:25.140: INFO: Deleting pod "restored-pvc-tester-llzn4" in namespace "snapshotting-5313"
Nov 29 20:01:25.255: INFO: Wait up to 5m0s for pod "restored-pvc-tester-llzn4" to be fully deleted
Nov 29 20:01:57.495: INFO: deleting snapshot "snapshotting-5313"/"snapshot-9qp2k"
... skipping 26 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works, check deletion (ephemeral)
        test/e2e/storage/testsuites/snapshottable.go:177
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Ephemeral Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works, check deletion (ephemeral)","total":32,"completed":10,"skipped":462,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning
  test/e2e/storage/framework/testsuite.go:51
Nov 29 20:02:05.578: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 80 lines ...
Nov 29 19:57:48.677: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com5jqhv] to have phase Bound
Nov 29 19:57:48.786: INFO: PersistentVolumeClaim test.csi.azure.com5jqhv found but phase is Pending instead of Bound.
Nov 29 19:57:50.908: INFO: PersistentVolumeClaim test.csi.azure.com5jqhv found but phase is Pending instead of Bound.
Nov 29 19:57:53.018: INFO: PersistentVolumeClaim test.csi.azure.com5jqhv found and phase=Bound (4.341764922s)
STEP: [init] starting a pod to use the claim
STEP: [init] check pod success
Nov 29 19:57:53.780: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-lq45x" in namespace "snapshotting-887" to be "Succeeded or Failed"
Nov 29 19:57:53.895: INFO: Pod "pvc-snapshottable-tester-lq45x": Phase="Pending", Reason="", readiness=false. Elapsed: 115.380509ms
Nov 29 19:57:56.005: INFO: Pod "pvc-snapshottable-tester-lq45x": Phase="Pending", Reason="", readiness=false. Elapsed: 2.225195299s
Nov 29 19:57:58.115: INFO: Pod "pvc-snapshottable-tester-lq45x": Phase="Pending", Reason="", readiness=false. Elapsed: 4.3350718s
Nov 29 19:58:00.228: INFO: Pod "pvc-snapshottable-tester-lq45x": Phase="Pending", Reason="", readiness=false. Elapsed: 6.447642323s
Nov 29 19:58:02.338: INFO: Pod "pvc-snapshottable-tester-lq45x": Phase="Pending", Reason="", readiness=false. Elapsed: 8.558284838s
Nov 29 19:58:04.449: INFO: Pod "pvc-snapshottable-tester-lq45x": Phase="Pending", Reason="", readiness=false. Elapsed: 10.669213592s
... skipping 21 lines ...
Nov 29 19:58:50.882: INFO: Pod "pvc-snapshottable-tester-lq45x": Phase="Pending", Reason="", readiness=false. Elapsed: 57.102440537s
Nov 29 19:58:52.993: INFO: Pod "pvc-snapshottable-tester-lq45x": Phase="Pending", Reason="", readiness=false. Elapsed: 59.21285637s
Nov 29 19:58:55.104: INFO: Pod "pvc-snapshottable-tester-lq45x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.324057724s
Nov 29 19:58:57.215: INFO: Pod "pvc-snapshottable-tester-lq45x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.434989031s
Nov 29 19:58:59.325: INFO: Pod "pvc-snapshottable-tester-lq45x": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m5.545220136s
STEP: Saw pod success
Nov 29 19:58:59.325: INFO: Pod "pvc-snapshottable-tester-lq45x" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
Nov 29 19:58:59.434: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com5jqhv] to have phase Bound
Nov 29 19:58:59.544: INFO: PersistentVolumeClaim test.csi.azure.com5jqhv found and phase=Bound (109.146949ms)
STEP: [init] checking the PV
STEP: [init] deleting the pod
Nov 29 19:58:59.898: INFO: Pod pvc-snapshottable-tester-lq45x has the following logs: 
... skipping 36 lines ...
Nov 29 19:59:15.584: INFO: WaitUntil finished successfully after 109.196883ms
STEP: getting the snapshot and snapshot content
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Nov 29 19:59:16.131: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-r7b2m" in namespace "snapshotting-887" to be "Succeeded or Failed"
Nov 29 19:59:16.240: INFO: Pod "pvc-snapshottable-data-tester-r7b2m": Phase="Pending", Reason="", readiness=false. Elapsed: 108.791388ms
Nov 29 19:59:18.350: INFO: Pod "pvc-snapshottable-data-tester-r7b2m": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21922598s
Nov 29 19:59:20.461: INFO: Pod "pvc-snapshottable-data-tester-r7b2m": Phase="Pending", Reason="", readiness=false. Elapsed: 4.330435631s
Nov 29 19:59:22.572: INFO: Pod "pvc-snapshottable-data-tester-r7b2m": Phase="Pending", Reason="", readiness=false. Elapsed: 6.441442828s
Nov 29 19:59:24.683: INFO: Pod "pvc-snapshottable-data-tester-r7b2m": Phase="Pending", Reason="", readiness=false. Elapsed: 8.552414118s
Nov 29 19:59:26.794: INFO: Pod "pvc-snapshottable-data-tester-r7b2m": Phase="Pending", Reason="", readiness=false. Elapsed: 10.663225896s
... skipping 30 lines ...
Nov 29 20:00:32.295: INFO: Pod "pvc-snapshottable-data-tester-r7b2m": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.164103973s
Nov 29 20:00:34.405: INFO: Pod "pvc-snapshottable-data-tester-r7b2m": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.274411461s
Nov 29 20:00:36.516: INFO: Pod "pvc-snapshottable-data-tester-r7b2m": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.38523272s
Nov 29 20:00:38.627: INFO: Pod "pvc-snapshottable-data-tester-r7b2m": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.496160631s
Nov 29 20:00:40.738: INFO: Pod "pvc-snapshottable-data-tester-r7b2m": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m24.607625859s
STEP: Saw pod success
Nov 29 20:00:40.739: INFO: Pod "pvc-snapshottable-data-tester-r7b2m" satisfied condition "Succeeded or Failed"
Nov 29 20:00:40.991: INFO: Pod pvc-snapshottable-data-tester-r7b2m has the following logs: 
Nov 29 20:00:40.991: INFO: Deleting pod "pvc-snapshottable-data-tester-r7b2m" in namespace "snapshotting-887"
Nov 29 20:00:41.107: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-r7b2m" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the snapshot
Nov 29 20:01:33.678: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-2hnyryib.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-887 exec restored-pvc-tester-gzt74 --namespace=snapshotting-887 -- cat /mnt/test/data'
... skipping 47 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        test/e2e/storage/testsuites/snapshottable.go:278
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":39,"completed":11,"skipped":636,"failed":0}

SSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy 
  (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents
  test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
... skipping 119 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents","total":49,"completed":8,"skipped":361,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]}

SSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] 
  should access to two volumes with the same volume mode and retain data across pod recreation on the same node
  test/e2e/storage/testsuites/multivolume.go:138
... skipping 194 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:138
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":29,"completed":10,"skipped":872,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using directory as subpath [Slow]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 20:04:32.132: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 215 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:209
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":32,"completed":11,"skipped":624,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] 
  should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
  test/e2e/storage/testsuites/multivolume.go:378
... skipping 83 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    test/e2e/storage/testsuites/multivolume.go:378
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":39,"completed":12,"skipped":641,"failed":0}

SSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning 
  should provision storage with pvc data source
  test/e2e/storage/testsuites/provisioning.go:421
... skipping 99 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with pvc data source
    test/e2e/storage/testsuites/provisioning.go:421
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with pvc data source","total":49,"completed":9,"skipped":377,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral
  test/e2e/storage/framework/testsuite.go:51
Nov 29 20:06:07.859: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping
... skipping 78 lines ...
Nov 29 20:05:15.560: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com5jtz7] to have phase Bound
Nov 29 20:05:15.669: INFO: PersistentVolumeClaim test.csi.azure.com5jtz7 found but phase is Pending instead of Bound.
Nov 29 20:05:17.779: INFO: PersistentVolumeClaim test.csi.azure.com5jtz7 found but phase is Pending instead of Bound.
Nov 29 20:05:19.889: INFO: PersistentVolumeClaim test.csi.azure.com5jtz7 found and phase=Bound (4.328525356s)
STEP: Creating pod pod-subpath-test-dynamicpv-6vtk
STEP: Creating a pod to test subpath
Nov 29 20:05:20.229: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-6vtk" in namespace "provisioning-1304" to be "Succeeded or Failed"
Nov 29 20:05:20.347: INFO: Pod "pod-subpath-test-dynamicpv-6vtk": Phase="Pending", Reason="", readiness=false. Elapsed: 117.600113ms
Nov 29 20:05:22.461: INFO: Pod "pod-subpath-test-dynamicpv-6vtk": Phase="Pending", Reason="", readiness=false. Elapsed: 2.231345618s
Nov 29 20:05:24.573: INFO: Pod "pod-subpath-test-dynamicpv-6vtk": Phase="Pending", Reason="", readiness=false. Elapsed: 4.343949655s
Nov 29 20:05:26.684: INFO: Pod "pod-subpath-test-dynamicpv-6vtk": Phase="Pending", Reason="", readiness=false. Elapsed: 6.454373628s
Nov 29 20:05:28.794: INFO: Pod "pod-subpath-test-dynamicpv-6vtk": Phase="Pending", Reason="", readiness=false. Elapsed: 8.564664875s
Nov 29 20:05:30.905: INFO: Pod "pod-subpath-test-dynamicpv-6vtk": Phase="Pending", Reason="", readiness=false. Elapsed: 10.676039809s
Nov 29 20:05:33.016: INFO: Pod "pod-subpath-test-dynamicpv-6vtk": Phase="Pending", Reason="", readiness=false. Elapsed: 12.786290199s
Nov 29 20:05:35.126: INFO: Pod "pod-subpath-test-dynamicpv-6vtk": Phase="Pending", Reason="", readiness=false. Elapsed: 14.89663261s
Nov 29 20:05:37.236: INFO: Pod "pod-subpath-test-dynamicpv-6vtk": Phase="Pending", Reason="", readiness=false. Elapsed: 17.006635416s
Nov 29 20:05:39.347: INFO: Pod "pod-subpath-test-dynamicpv-6vtk": Phase="Succeeded", Reason="", readiness=false. Elapsed: 19.117741047s
STEP: Saw pod success
Nov 29 20:05:39.347: INFO: Pod "pod-subpath-test-dynamicpv-6vtk" satisfied condition "Succeeded or Failed"
Nov 29 20:05:39.457: INFO: Trying to get logs from node k8s-agentpool1-16172903-vmss000000 pod pod-subpath-test-dynamicpv-6vtk container test-container-subpath-dynamicpv-6vtk: <nil>
STEP: delete the pod
Nov 29 20:05:39.709: INFO: Waiting for pod pod-subpath-test-dynamicpv-6vtk to disappear
Nov 29 20:05:39.818: INFO: Pod pod-subpath-test-dynamicpv-6vtk no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-6vtk
Nov 29 20:05:39.818: INFO: Deleting pod "pod-subpath-test-dynamicpv-6vtk" in namespace "provisioning-1304"
... skipping 23 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support existing single file [LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:221
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing single file [LinuxOnly]","total":39,"completed":13,"skipped":660,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
... skipping 70 lines ...
Nov 29 20:06:08.883: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comwck6k] to have phase Bound
Nov 29 20:06:08.992: INFO: PersistentVolumeClaim test.csi.azure.comwck6k found but phase is Pending instead of Bound.
Nov 29 20:06:11.102: INFO: PersistentVolumeClaim test.csi.azure.comwck6k found but phase is Pending instead of Bound.
Nov 29 20:06:13.211: INFO: PersistentVolumeClaim test.csi.azure.comwck6k found and phase=Bound (4.328334058s)
STEP: Creating pod pod-subpath-test-dynamicpv-w48g
STEP: Creating a pod to test subpath
Nov 29 20:06:13.606: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-w48g" in namespace "provisioning-8841" to be "Succeeded or Failed"
Nov 29 20:06:13.733: INFO: Pod "pod-subpath-test-dynamicpv-w48g": Phase="Pending", Reason="", readiness=false. Elapsed: 126.460549ms
Nov 29 20:06:15.841: INFO: Pod "pod-subpath-test-dynamicpv-w48g": Phase="Pending", Reason="", readiness=false. Elapsed: 2.23529608s
Nov 29 20:06:17.952: INFO: Pod "pod-subpath-test-dynamicpv-w48g": Phase="Pending", Reason="", readiness=false. Elapsed: 4.346331102s
Nov 29 20:06:20.062: INFO: Pod "pod-subpath-test-dynamicpv-w48g": Phase="Pending", Reason="", readiness=false. Elapsed: 6.45625511s
Nov 29 20:06:22.171: INFO: Pod "pod-subpath-test-dynamicpv-w48g": Phase="Pending", Reason="", readiness=false. Elapsed: 8.564815922s
Nov 29 20:06:24.280: INFO: Pod "pod-subpath-test-dynamicpv-w48g": Phase="Pending", Reason="", readiness=false. Elapsed: 10.673628411s
Nov 29 20:06:26.391: INFO: Pod "pod-subpath-test-dynamicpv-w48g": Phase="Pending", Reason="", readiness=false. Elapsed: 12.78465429s
Nov 29 20:06:28.500: INFO: Pod "pod-subpath-test-dynamicpv-w48g": Phase="Pending", Reason="", readiness=false. Elapsed: 14.894203948s
Nov 29 20:06:30.611: INFO: Pod "pod-subpath-test-dynamicpv-w48g": Phase="Pending", Reason="", readiness=false. Elapsed: 17.004416755s
Nov 29 20:06:32.720: INFO: Pod "pod-subpath-test-dynamicpv-w48g": Phase="Succeeded", Reason="", readiness=false. Elapsed: 19.114170293s
STEP: Saw pod success
Nov 29 20:06:32.720: INFO: Pod "pod-subpath-test-dynamicpv-w48g" satisfied condition "Succeeded or Failed"
Nov 29 20:06:32.828: INFO: Trying to get logs from node k8s-agentpool1-16172903-vmss000001 pod pod-subpath-test-dynamicpv-w48g container test-container-subpath-dynamicpv-w48g: <nil>
STEP: delete the pod
Nov 29 20:06:33.065: INFO: Waiting for pod pod-subpath-test-dynamicpv-w48g to disappear
Nov 29 20:06:33.173: INFO: Pod pod-subpath-test-dynamicpv-w48g no longer exists
STEP: Deleting pod pod-subpath-test-dynamicpv-w48g
Nov 29 20:06:33.173: INFO: Deleting pod "pod-subpath-test-dynamicpv-w48g" in namespace "provisioning-8841"
... skipping 22 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should support readOnly file specified in the volumeMount [LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:382
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly]","total":49,"completed":10,"skipped":532,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]}

SSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 29 20:07:09.921: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 45 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Pre-provisioned PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach]
    test/e2e/storage/testsuites/volumemode.go:299

    Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 75 lines ...
Nov 29 20:04:33.112: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comjwnb8] to have phase Bound
Nov 29 20:04:33.220: INFO: PersistentVolumeClaim test.csi.azure.comjwnb8 found but phase is Pending instead of Bound.
Nov 29 20:04:35.335: INFO: PersistentVolumeClaim test.csi.azure.comjwnb8 found but phase is Pending instead of Bound.
Nov 29 20:04:37.444: INFO: PersistentVolumeClaim test.csi.azure.comjwnb8 found and phase=Bound (4.331544547s)
STEP: [init] starting a pod to use the claim
STEP: [init] check pod success
Nov 29 20:04:37.881: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-t89ch" in namespace "snapshotting-9843" to be "Succeeded or Failed"
Nov 29 20:04:37.994: INFO: Pod "pvc-snapshottable-tester-t89ch": Phase="Pending", Reason="", readiness=false. Elapsed: 112.937698ms
Nov 29 20:04:40.104: INFO: Pod "pvc-snapshottable-tester-t89ch": Phase="Pending", Reason="", readiness=false. Elapsed: 2.222559449s
Nov 29 20:04:42.217: INFO: Pod "pvc-snapshottable-tester-t89ch": Phase="Pending", Reason="", readiness=false. Elapsed: 4.335325129s
Nov 29 20:04:44.326: INFO: Pod "pvc-snapshottable-tester-t89ch": Phase="Pending", Reason="", readiness=false. Elapsed: 6.444344825s
Nov 29 20:04:46.435: INFO: Pod "pvc-snapshottable-tester-t89ch": Phase="Pending", Reason="", readiness=false. Elapsed: 8.553792687s
Nov 29 20:04:48.544: INFO: Pod "pvc-snapshottable-tester-t89ch": Phase="Pending", Reason="", readiness=false. Elapsed: 10.662243464s
... skipping 4 lines ...
Nov 29 20:04:59.093: INFO: Pod "pvc-snapshottable-tester-t89ch": Phase="Pending", Reason="", readiness=false. Elapsed: 21.21113779s
Nov 29 20:05:01.203: INFO: Pod "pvc-snapshottable-tester-t89ch": Phase="Pending", Reason="", readiness=false. Elapsed: 23.322004412s
Nov 29 20:05:03.312: INFO: Pod "pvc-snapshottable-tester-t89ch": Phase="Pending", Reason="", readiness=false. Elapsed: 25.430880208s
Nov 29 20:05:05.422: INFO: Pod "pvc-snapshottable-tester-t89ch": Phase="Pending", Reason="", readiness=false. Elapsed: 27.540427997s
Nov 29 20:05:07.532: INFO: Pod "pvc-snapshottable-tester-t89ch": Phase="Succeeded", Reason="", readiness=false. Elapsed: 29.650451978s
STEP: Saw pod success
Nov 29 20:05:07.532: INFO: Pod "pvc-snapshottable-tester-t89ch" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
Nov 29 20:05:07.642: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comjwnb8] to have phase Bound
Nov 29 20:05:07.749: INFO: PersistentVolumeClaim test.csi.azure.comjwnb8 found and phase=Bound (107.592216ms)
STEP: [init] checking the PV
STEP: [init] deleting the pod
Nov 29 20:05:08.104: INFO: Pod pvc-snapshottable-tester-t89ch has the following logs: 
... skipping 15 lines ...
Nov 29 20:05:19.972: INFO: received snapshotStatus map[boundVolumeSnapshotContentName:snapcontent-8fb2478c-1861-4a43-8e05-ad98afbed39c creationTime:2022-11-29T20:05:15Z readyToUse:true restoreSize:5Gi]
Nov 29 20:05:19.972: INFO: snapshotContentName snapcontent-8fb2478c-1861-4a43-8e05-ad98afbed39c
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Nov 29 20:05:20.408: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-6vl82" in namespace "snapshotting-9843" to be "Succeeded or Failed"
Nov 29 20:05:20.517: INFO: Pod "pvc-snapshottable-data-tester-6vl82": Phase="Pending", Reason="", readiness=false. Elapsed: 108.472376ms
Nov 29 20:05:22.626: INFO: Pod "pvc-snapshottable-data-tester-6vl82": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21771218s
Nov 29 20:05:24.735: INFO: Pod "pvc-snapshottable-data-tester-6vl82": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326631811s
Nov 29 20:05:26.846: INFO: Pod "pvc-snapshottable-data-tester-6vl82": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437686239s
Nov 29 20:05:28.955: INFO: Pod "pvc-snapshottable-data-tester-6vl82": Phase="Pending", Reason="", readiness=false. Elapsed: 8.546503091s
Nov 29 20:05:31.065: INFO: Pod "pvc-snapshottable-data-tester-6vl82": Phase="Pending", Reason="", readiness=false. Elapsed: 10.65666828s
... skipping 9 lines ...
Nov 29 20:05:52.159: INFO: Pod "pvc-snapshottable-data-tester-6vl82": Phase="Pending", Reason="", readiness=false. Elapsed: 31.750654768s
Nov 29 20:05:54.269: INFO: Pod "pvc-snapshottable-data-tester-6vl82": Phase="Pending", Reason="", readiness=false. Elapsed: 33.860166123s
Nov 29 20:05:56.378: INFO: Pod "pvc-snapshottable-data-tester-6vl82": Phase="Pending", Reason="", readiness=false. Elapsed: 35.969725514s
Nov 29 20:05:58.488: INFO: Pod "pvc-snapshottable-data-tester-6vl82": Phase="Pending", Reason="", readiness=false. Elapsed: 38.07923459s
Nov 29 20:06:00.598: INFO: Pod "pvc-snapshottable-data-tester-6vl82": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.189096453s
STEP: Saw pod success
Nov 29 20:06:00.598: INFO: Pod "pvc-snapshottable-data-tester-6vl82" satisfied condition "Succeeded or Failed"
Nov 29 20:06:00.849: INFO: Pod pvc-snapshottable-data-tester-6vl82 has the following logs: 
Nov 29 20:06:00.849: INFO: Deleting pod "pvc-snapshottable-data-tester-6vl82" in namespace "snapshotting-9843"
Nov 29 20:06:00.966: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-6vl82" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the snapshot
Nov 29 20:06:31.515: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-2hnyryib.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-9843 exec restored-pvc-tester-46gwg --namespace=snapshotting-9843 -- cat /mnt/test/data'
... skipping 33 lines ...
Nov 29 20:06:57.812: INFO: volumesnapshotcontents snapcontent-8fb2478c-1861-4a43-8e05-ad98afbed39c has been found and is not deleted
Nov 29 20:06:58.921: INFO: volumesnapshotcontents snapcontent-8fb2478c-1861-4a43-8e05-ad98afbed39c has been found and is not deleted
Nov 29 20:07:00.031: INFO: volumesnapshotcontents snapcontent-8fb2478c-1861-4a43-8e05-ad98afbed39c has been found and is not deleted
Nov 29 20:07:01.140: INFO: volumesnapshotcontents snapcontent-8fb2478c-1861-4a43-8e05-ad98afbed39c has been found and is not deleted
Nov 29 20:07:02.249: INFO: volumesnapshotcontents snapcontent-8fb2478c-1861-4a43-8e05-ad98afbed39c has been found and is not deleted
Nov 29 20:07:03.358: INFO: volumesnapshotcontents snapcontent-8fb2478c-1861-4a43-8e05-ad98afbed39c has been found and is not deleted
Nov 29 20:07:04.359: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 29 20:07:04.467: INFO: Error getting logs for pod restored-pvc-tester-46gwg: the server could not find the requested resource (get pods restored-pvc-tester-46gwg)
Nov 29 20:07:04.467: INFO: Deleting pod "restored-pvc-tester-46gwg" in namespace "snapshotting-9843"
Nov 29 20:07:04.575: INFO: deleting claim "snapshotting-9843"/"pvc-hwhbv"
Nov 29 20:07:04.684: INFO: deleting snapshot "snapshotting-9843"/"snapshot-dj7xf"
Nov 29 20:07:04.792: INFO: deleting snapshot content "snapcontent-8fb2478c-1861-4a43-8e05-ad98afbed39c"
Nov 29 20:07:05.126: INFO: Waiting up to 5m0s for volumesnapshotcontents snapcontent-8fb2478c-1861-4a43-8e05-ad98afbed39c to be deleted
Nov 29 20:07:05.236: INFO: volumesnapshotcontents snapcontent-8fb2478c-1861-4a43-8e05-ad98afbed39c has been found and is not deleted
... skipping 27 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        test/e2e/storage/testsuites/snapshottable.go:278
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":29,"completed":11,"skipped":949,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using directory as subpath [Slow]"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 20:07:22.833: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 113 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should support two pods which have the same volume definition
    test/e2e/storage/testsuites/ephemeral.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support two pods which have the same volume definition","total":32,"completed":12,"skipped":656,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes 
  should store data
  test/e2e/storage/testsuites/volumes.go:161
... skipping 110 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should store data
    test/e2e/storage/testsuites/volumes.go:161
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should store data","total":39,"completed":14,"skipped":758,"failed":0}

SSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 20:09:07.417: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 53 lines ...

    Distro debian doesn't support ntfs -- skipping

    test/e2e/storage/framework/testsuite.go:127
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode 
  should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299

[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 20:07:11.065: INFO: >>> kubeConfig: /root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename volumemode
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299
Nov 29 20:07:11.820: INFO: Creating resource for dynamic PV
Nov 29 20:07:11.821: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass volumemode-1726-e2e-schk9x9
STEP: creating a claim
Nov 29 20:07:12.050: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comlq47q] to have phase Bound
Nov 29 20:07:12.158: INFO: PersistentVolumeClaim test.csi.azure.comlq47q found but phase is Pending instead of Bound.
Nov 29 20:07:14.269: INFO: PersistentVolumeClaim test.csi.azure.comlq47q found but phase is Pending instead of Bound.
Nov 29 20:07:16.379: INFO: PersistentVolumeClaim test.csi.azure.comlq47q found and phase=Bound (4.328940237s)
STEP: Creating pod
STEP: Waiting for the pod to fail
Nov 29 20:07:16.936: INFO: Deleting pod "pod-2e363b45-5dde-43b9-8371-36507f86b1fa" in namespace "volumemode-1726"
Nov 29 20:07:17.046: INFO: Wait up to 5m0s for pod "pod-2e363b45-5dde-43b9-8371-36507f86b1fa" to be fully deleted
STEP: Deleting pvc
Nov 29 20:07:19.263: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comlq47q"
Nov 29 20:07:19.373: INFO: Waiting up to 5m0s for PersistentVolume pvc-dd4a33e7-5ee3-4af9-9ad0-67f325594d4f to get deleted
Nov 29 20:07:19.482: INFO: PersistentVolume pvc-dd4a33e7-5ee3-4af9-9ad0-67f325594d4f found and phase=Released (108.493461ms)
... skipping 32 lines ...

• [SLOW TEST:141.819 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow]
    test/e2e/storage/testsuites/volumemode.go:299
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should fail to use a volume in a pod with mismatched mode [Slow]","total":49,"completed":11,"skipped":674,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode 
  should not mount / map unused volumes in a pod [LinuxOnly]
  test/e2e/storage/testsuites/volumemode.go:354
... skipping 63 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:50
    should not mount / map unused volumes in a pod [LinuxOnly]
    test/e2e/storage/testsuites/volumemode.go:354
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":49,"completed":12,"skipped":705,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]}

SS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeLimits 
  should verify that all csinodes have volume limits
  test/e2e/storage/testsuites/volumelimits.go:249
... skipping 16 lines ...
  test/e2e/framework/framework.go:188
Nov 29 20:10:31.103: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "volumelimits-3956" for this suite.

•
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeLimits should verify that all csinodes have volume limits","total":49,"completed":13,"skipped":707,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 29 20:10:31.435: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 119 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach]
    test/e2e/storage/testsuites/subpath.go:242

    Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping

    test/e2e/storage/external/external.go:262
------------------------------
... skipping 103 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:50
    should create read-only inline ephemeral volume
    test/e2e/storage/testsuites/ephemeral.go:175
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read-only inline ephemeral volume","total":39,"completed":15,"skipped":854,"failed":0}

SSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] 
  should access to two volumes with different volume mode and retain data across pod recreation on the same node
  test/e2e/storage/testsuites/multivolume.go:209
... skipping 189 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with different volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:209
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":32,"completed":13,"skipped":696,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning
  test/e2e/storage/framework/testsuite.go:51
Nov 29 20:10:41.110: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 133 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":32,"completed":14,"skipped":858,"failed":0}

SSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] 
  should access to two volumes with the same volume mode and retain data across pod recreation on the same node
  test/e2e/storage/testsuites/multivolume.go:138
... skipping 190 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    test/e2e/storage/testsuites/multivolume.go:138
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":39,"completed":16,"skipped":867,"failed":0}

SSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 29 20:12:53.898: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 59 lines ...
Nov 29 20:10:34.590: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com6pdbh] to have phase Bound
Nov 29 20:10:34.698: INFO: PersistentVolumeClaim test.csi.azure.com6pdbh found but phase is Pending instead of Bound.
Nov 29 20:10:36.807: INFO: PersistentVolumeClaim test.csi.azure.com6pdbh found but phase is Pending instead of Bound.
Nov 29 20:10:38.915: INFO: PersistentVolumeClaim test.csi.azure.com6pdbh found and phase=Bound (4.325546353s)
STEP: [init] starting a pod to use the claim
STEP: [init] check pod success
Nov 29 20:10:39.353: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-gx4tc" in namespace "snapshotting-5256" to be "Succeeded or Failed"
Nov 29 20:10:39.461: INFO: Pod "pvc-snapshottable-tester-gx4tc": Phase="Pending", Reason="", readiness=false. Elapsed: 108.319818ms
Nov 29 20:10:41.571: INFO: Pod "pvc-snapshottable-tester-gx4tc": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218323998s
Nov 29 20:10:43.682: INFO: Pod "pvc-snapshottable-tester-gx4tc": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328954989s
Nov 29 20:10:45.792: INFO: Pod "pvc-snapshottable-tester-gx4tc": Phase="Pending", Reason="", readiness=false. Elapsed: 6.439298728s
Nov 29 20:10:47.902: INFO: Pod "pvc-snapshottable-tester-gx4tc": Phase="Pending", Reason="", readiness=false. Elapsed: 8.549689781s
Nov 29 20:10:50.011: INFO: Pod "pvc-snapshottable-tester-gx4tc": Phase="Pending", Reason="", readiness=false. Elapsed: 10.657813678s
Nov 29 20:10:52.122: INFO: Pod "pvc-snapshottable-tester-gx4tc": Phase="Pending", Reason="", readiness=false. Elapsed: 12.769275054s
Nov 29 20:10:54.232: INFO: Pod "pvc-snapshottable-tester-gx4tc": Phase="Pending", Reason="", readiness=false. Elapsed: 14.87927586s
Nov 29 20:10:56.343: INFO: Pod "pvc-snapshottable-tester-gx4tc": Phase="Pending", Reason="", readiness=false. Elapsed: 16.99040418s
Nov 29 20:10:58.453: INFO: Pod "pvc-snapshottable-tester-gx4tc": Phase="Succeeded", Reason="", readiness=false. Elapsed: 19.099770793s
STEP: Saw pod success
Nov 29 20:10:58.453: INFO: Pod "pvc-snapshottable-tester-gx4tc" satisfied condition "Succeeded or Failed"
STEP: [init] checking the claim
Nov 29 20:10:58.561: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com6pdbh] to have phase Bound
Nov 29 20:10:58.670: INFO: PersistentVolumeClaim test.csi.azure.com6pdbh found and phase=Bound (108.58178ms)
STEP: [init] checking the PV
STEP: [init] deleting the pod
Nov 29 20:10:59.030: INFO: Pod pvc-snapshottable-tester-gx4tc has the following logs: 
... skipping 34 lines ...
Nov 29 20:11:09.674: INFO: WaitUntil finished successfully after 108.726515ms
STEP: getting the snapshot and snapshot content
STEP: checking the snapshot
STEP: checking the SnapshotContent
STEP: Modifying source data test
STEP: modifying the data in the source PVC
Nov 29 20:11:10.220: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-gqvvq" in namespace "snapshotting-5256" to be "Succeeded or Failed"
Nov 29 20:11:10.329: INFO: Pod "pvc-snapshottable-data-tester-gqvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 108.208223ms
Nov 29 20:11:12.439: INFO: Pod "pvc-snapshottable-data-tester-gqvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218165453s
Nov 29 20:11:14.548: INFO: Pod "pvc-snapshottable-data-tester-gqvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327997923s
Nov 29 20:11:16.659: INFO: Pod "pvc-snapshottable-data-tester-gqvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 6.438247476s
Nov 29 20:11:18.769: INFO: Pod "pvc-snapshottable-data-tester-gqvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 8.548541851s
Nov 29 20:11:20.878: INFO: Pod "pvc-snapshottable-data-tester-gqvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 10.658132686s
... skipping 16 lines ...
Nov 29 20:11:56.744: INFO: Pod "pvc-snapshottable-data-tester-gqvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 46.523413205s
Nov 29 20:11:58.853: INFO: Pod "pvc-snapshottable-data-tester-gqvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 48.632217186s
Nov 29 20:12:00.962: INFO: Pod "pvc-snapshottable-data-tester-gqvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 50.741735022s
Nov 29 20:12:03.071: INFO: Pod "pvc-snapshottable-data-tester-gqvvq": Phase="Pending", Reason="", readiness=false. Elapsed: 52.850720274s
Nov 29 20:12:05.181: INFO: Pod "pvc-snapshottable-data-tester-gqvvq": Phase="Succeeded", Reason="", readiness=false. Elapsed: 54.961080217s
STEP: Saw pod success
Nov 29 20:12:05.181: INFO: Pod "pvc-snapshottable-data-tester-gqvvq" satisfied condition "Succeeded or Failed"
Nov 29 20:12:05.403: INFO: Pod pvc-snapshottable-data-tester-gqvvq has the following logs: 
Nov 29 20:12:05.403: INFO: Deleting pod "pvc-snapshottable-data-tester-gqvvq" in namespace "snapshotting-5256"
Nov 29 20:12:05.517: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-gqvvq" to be fully deleted
STEP: creating a pvc from the snapshot
STEP: starting a pod to use the snapshot
Nov 29 20:12:32.069: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-2hnyryib.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-5256 exec restored-pvc-tester-kdkrp --namespace=snapshotting-5256 -- cat /mnt/test/data'
... skipping 33 lines ...
Nov 29 20:12:58.350: INFO: volumesnapshotcontents pre-provisioned-snapcontent-88842128-bccd-4667-8435-92b9361515ab has been found and is not deleted
Nov 29 20:12:59.460: INFO: volumesnapshotcontents pre-provisioned-snapcontent-88842128-bccd-4667-8435-92b9361515ab has been found and is not deleted
Nov 29 20:13:00.574: INFO: volumesnapshotcontents pre-provisioned-snapcontent-88842128-bccd-4667-8435-92b9361515ab has been found and is not deleted
Nov 29 20:13:01.684: INFO: volumesnapshotcontents pre-provisioned-snapcontent-88842128-bccd-4667-8435-92b9361515ab has been found and is not deleted
Nov 29 20:13:02.793: INFO: volumesnapshotcontents pre-provisioned-snapcontent-88842128-bccd-4667-8435-92b9361515ab has been found and is not deleted
Nov 29 20:13:03.903: INFO: volumesnapshotcontents pre-provisioned-snapcontent-88842128-bccd-4667-8435-92b9361515ab has been found and is not deleted
Nov 29 20:13:04.903: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 29 20:13:05.011: INFO: Error getting logs for pod restored-pvc-tester-kdkrp: the server could not find the requested resource (get pods restored-pvc-tester-kdkrp)
Nov 29 20:13:05.011: INFO: Deleting pod "restored-pvc-tester-kdkrp" in namespace "snapshotting-5256"
Nov 29 20:13:05.121: INFO: deleting claim "snapshotting-5256"/"pvc-txqmp"
Nov 29 20:13:05.232: INFO: deleting snapshot "snapshotting-5256"/"pre-provisioned-snapshot-88842128-bccd-4667-8435-92b9361515ab"
Nov 29 20:13:05.349: INFO: deleting snapshot content "pre-provisioned-snapcontent-88842128-bccd-4667-8435-92b9361515ab"
Nov 29 20:13:05.683: INFO: Waiting up to 5m0s for volumesnapshotcontents pre-provisioned-snapcontent-88842128-bccd-4667-8435-92b9361515ab to be deleted
Nov 29 20:13:05.792: INFO: volumesnapshotcontents pre-provisioned-snapcontent-88842128-bccd-4667-8435-92b9361515ab has been found and is not deleted
... skipping 26 lines ...
    test/e2e/storage/testsuites/snapshottable.go:113
      
      test/e2e/storage/testsuites/snapshottable.go:176
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        test/e2e/storage/testsuites/snapshottable.go:278
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":49,"completed":14,"skipped":970,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]}

SSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 20:13:18.250: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 11 lines ...

    Distro debian doesn't support ntfs -- skipping

    test/e2e/storage/framework/testsuite.go:127
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:269

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
STEP: Creating a kubernetes client
Nov 29 20:12:23.561: INFO: >>> kubeConfig: /root/tmp2345232819/kubeconfig/kubeconfig.westeurope.json
STEP: Building a namespace api object, basename provisioning
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:269
Nov 29 20:12:24.321: INFO: Creating resource for dynamic PV
Nov 29 20:12:24.322: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-7450-e2e-scx2kpg
STEP: creating a claim
Nov 29 20:12:24.469: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 29 20:12:24.580: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comj7b2r] to have phase Bound
Nov 29 20:12:24.688: INFO: PersistentVolumeClaim test.csi.azure.comj7b2r found but phase is Pending instead of Bound.
Nov 29 20:12:26.798: INFO: PersistentVolumeClaim test.csi.azure.comj7b2r found but phase is Pending instead of Bound.
Nov 29 20:12:28.908: INFO: PersistentVolumeClaim test.csi.azure.comj7b2r found and phase=Bound (4.327345453s)
STEP: Creating pod pod-subpath-test-dynamicpv-p7b4
STEP: Checking for subpath error in container status
Nov 29 20:13:03.462: INFO: Deleting pod "pod-subpath-test-dynamicpv-p7b4" in namespace "provisioning-7450"
Nov 29 20:13:03.573: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-p7b4" to be fully deleted
STEP: Deleting pod
Nov 29 20:13:05.792: INFO: Deleting pod "pod-subpath-test-dynamicpv-p7b4" in namespace "provisioning-7450"
STEP: Deleting pvc
Nov 29 20:13:05.901: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comj7b2r"
... skipping 22 lines ...

• [SLOW TEST:114.659 seconds]
External Storage [Driver: test.csi.azure.com]
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:50
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]
    test/e2e/storage/testsuites/subpath.go:269
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]","total":32,"completed":15,"skipped":875,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 29 20:14:18.254: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 122 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (block volmode)] volumes
  test/e2e/storage/framework/testsuite.go:50
    should store data
    test/e2e/storage/testsuites/volumes.go:161
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumes should store data","total":39,"completed":17,"skipped":911,"failed":0}

SSSSSSSS
------------------------------
[BeforeEach] [Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource]
... skipping 150 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  test/e2e/storage/framework/testsuite.go:50
    (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
    test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":32,"completed":16,"skipped":936,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning 
  should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
  test/e2e/storage/testsuites/provisioning.go:208
... skipping 122 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (default fs)] provisioning
  test/e2e/storage/framework/testsuite.go:50
    should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
    test/e2e/storage/testsuites/provisioning.go:208
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":49,"completed":15,"skipped":989,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node"]}

SSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
Nov 29 20:16:21.303: INFO: Running AfterSuite actions on all nodes
Nov 29 20:16:21.304: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2
Nov 29 20:16:21.304: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2
... skipping 157 lines ...
test/e2e/storage/external/external.go:174
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single volume from pods on the same node
    test/e2e/storage/testsuites/multivolume.go:298
------------------------------
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":32,"completed":17,"skipped":966,"failed":0}

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
Nov 29 20:17:31.155: INFO: Running AfterSuite actions on all nodes
Nov 29 20:17:31.155: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2
Nov 29 20:17:31.155: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2
... skipping 15 lines ...
Nov 29 20:17:31.201: INFO: Running AfterSuite actions on node 1



Summarizing 5 Failures:

[Fail] External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] [Measurement] should concurrently access the single read-only volume from pods on the same node 
test/e2e/storage/testsuites/multivolume.go:129

[Fail] External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath [Measurement] should support restarting containers using directory as subpath [Slow] 
test/e2e/storage/testsuites/subpath.go:185

[Fail] External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [Measurement] should access to two volumes with the same volume mode and retain data across pod recreation on different node 
test/e2e/storage/testsuites/multivolume.go:497

[Fail] External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [Measurement] should access to two volumes with different volume mode and retain data across pod recreation on different node 
test/e2e/storage/testsuites/multivolume.go:497

[Fail] External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [Measurement] should access to two volumes with different volume mode and retain data across pod recreation on the same node 
test/e2e/storage/testsuites/multivolume.go:497

Ran 89 of 7227 Specs in 3216.345 seconds
FAIL! -- 84 Passed | 5 Failed | 0 Pending | 7138 Skipped 

Ginkgo ran 1 suite in 53m43.896133004s
Test Suite Failed
+ print_logs
+ sed -i s/disk.csi.azure.com/test.csi.azure.com/g deploy/example/storageclass-azuredisk-csi.yaml
+ bash ./hack/verify-examples.sh linux azurepubliccloud ephemeral test
begin to create deployment examples ...
storageclass.storage.k8s.io/managed-csi created
Applying config "deploy/example/deployment.yaml"
... skipping 100 lines ...
I1129 19:23:42.571778       1 reflector.go:221] Starting reflector *v1beta2.AzVolumeAttachment (30s) from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117
I1129 19:23:42.571795       1 reflector.go:257] Listing and watching *v1beta2.AzDriverNode from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117
I1129 19:23:42.571799       1 reflector.go:257] Listing and watching *v1beta2.AzVolumeAttachment from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117
I1129 19:23:42.671897       1 shared_informer.go:285] caches populated
I1129 19:23:42.671960       1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-c13cf58a5ca48aed0e92b8b1ecf1b82eae2479e6 e2e-test
I1129 19:23:42.671980       1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider
I1129 19:23:42.675258       1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I1129 19:23:42.675276       1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider
I1129 19:23:42.675281       1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I1129 19:23:42.675304       1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully
I1129 19:23:42.675917       1 azure_auth.go:245] Using AzurePublicCloud environment
I1129 19:23:42.675937       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I1129 19:23:42.675942       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
... skipping 29 lines ...
I1129 19:23:42.676342       1 azure_vmasclient.go:73] Azure AvailabilitySetsClient  (write ops) using rate limit config: QPS=100, bucket=1000
I1129 19:23:42.676410       1 azure.go:1039] attach/detach disk operation rate limiter configuration - Enabled: true QPS: 1.333000, Bucket: 240
I1129 19:23:42.676453       1 azuredisk_v2.go:215] disable UseInstanceMetadata for controller
I1129 19:23:42.676459       1 azuredisk_v2.go:225] DisableAvailabilitySetNodes for controller since current VMType is vmss
I1129 19:23:42.676468       1 azuredisk_v2.go:231] cloud: AzurePublicCloud, location: westeurope, rg: kubetest-2hnyryib, VMType: vmss, PrimaryScaleSetName: k8s-agentpool1-16172903-vmss, PrimaryAvailabilitySetName: , DisableAvailabilitySetNodes: true
I1129 19:23:42.676481       1 skus.go:123] NewNodeInfo: Starting to populate node and disk sku information.
E1129 19:23:42.676496       1 azuredisk_v2.go:239] Failed to get node info. Error: NewNodeInfo: Failed to get instance type from Azure cloud provider, nodeName: , error: not a vmss instance
I1129 19:23:42.676600       1 mount_linux.go:208] Detected OS without systemd
I1129 19:23:42.676618       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I1129 19:23:42.676623       1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I1129 19:23:42.676626       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I1129 19:23:42.676630       1 driver.go:81] Enabling controller service capability: CLONE_VOLUME
I1129 19:23:42.676633       1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME
... skipping 135 lines ...
I1129 19:23:42.851111       1 shared_state.go:485]  "msg"="Storing pod azure-ip-masq-agent-jkwdg and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="560683b4-701b-11ed-8695-000d3a29806d"
I1129 19:23:42.851125       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/azure-ip-masq-agent-jkwdg." "disk.csi.azure.com/request-id"="560683b4-701b-11ed-8695-000d3a29806d" "disk.csi/azure.com/pod-name"="kube-system/azure-ip-masq-agent-jkwdg"
I1129 19:23:42.851134       1 shared_state.go:309]  "msg"="Getting requested volumes for pod (kube-system/azure-ip-masq-agent-jkwdg)." "disk.csi.azure.com/request-id"="560683b4-701b-11ed-8695-000d3a29806d" "disk.csi/azure.com/pod-name"="kube-system/azure-ip-masq-agent-jkwdg"
I1129 19:23:42.851144       1 pod.go:99]  "msg"="Pod kube-system/azure-ip-masq-agent-jkwdg has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="560683b4-701b-11ed-8695-000d3a29806d" "disk.csi/azure.com/pod-name"="kube-system/azure-ip-masq-agent-jkwdg"
I1129 19:23:42.851157       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="560683b4-701b-11ed-8695-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/azure-ip-masq-agent-jkwdg" "latency"=31001
I1129 19:23:42.851167       1 shared_state.go:415]  "msg"="Adding pod csi-test-controller-7898947974-hmrmh to shared map with keyName kube-system/csi-test-controller-7898947974-hmrmh." "disk.csi.azure.com/request-id"="560683b4-701b-11ed-8695-000d3a29806d"
I1129 19:23:42.851470       1 shared_state.go:421]  "msg"="Pod spec of pod csi-test-controller-7898947974-hmrmh is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-pwfbf VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-pwfbf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-pwfbf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-pwfbf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-pwfbf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-pwfbf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-c13cf58a5ca48aed0e92b8b1ecf1b82eae2479e6 Command:[] Args:[--v=5 --config=/etc/csi-test-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-test-controller-config ReadOnly:false MountPath:/etc/csi-test-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-pwfbf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc0004adea8 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-agentpool1-16172903-vmss000002 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc0004adeb0} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc0004adeb8}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc0004adec0 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc0004adec4 PreemptionPolicy:0xc000a42e30 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil>}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-pwfbf VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" "disk.csi.azure.com/request-id"="560683b4-701b-11ed-8695-000d3a29806d"
I1129 19:23:42.851506       1 shared_state.go:459]  "msg"="Pod csi-test-controller-7898947974-hmrmh: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="560683b4-701b-11ed-8695-000d3a29806d"
I1129 19:23:42.851523       1 shared_state.go:459]  "msg"="Pod csi-test-controller-7898947974-hmrmh: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="560683b4-701b-11ed-8695-000d3a29806d"
I1129 19:23:42.851541       1 shared_state.go:459]  "msg"="Pod csi-test-controller-7898947974-hmrmh: Skipping Volume {csi-test-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="560683b4-701b-11ed-8695-000d3a29806d"
I1129 19:23:42.851568       1 shared_state.go:459]  "msg"="Pod csi-test-controller-7898947974-hmrmh: Skipping Volume {kube-api-access-pwfbf {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="560683b4-701b-11ed-8695-000d3a29806d"
I1129 19:23:42.851580       1 shared_state.go:485]  "msg"="Storing pod csi-test-controller-7898947974-hmrmh and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="560683b4-701b-11ed-8695-000d3a29806d"
I1129 19:23:42.851596       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-test-controller-7898947974-hmrmh." "disk.csi.azure.com/request-id"="560683b4-701b-11ed-8695-000d3a29806d" "disk.csi/azure.com/pod-name"="kube-system/csi-test-controller-7898947974-hmrmh"
... skipping 32074 lines ...
I1129 20:16:08.515411       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:16:08.515425       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5c7a35ae-aed4-4e9d-9387-7f606d0bff83/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5c7a35ae-aed4-4e9d-9387-7f606d0bff83","csi.storage.k8s.io/pvc/name":"test.csi.azure.comgzkqt","csi.storage.k8s.io/pvc/namespace":"multivolume-9459","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-5c7a35ae-aed4-4e9d-9387-7f606d0bff83"}
I1129 20:16:08.515865       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-5c7a35ae-aed4-4e9d-9387-7f606d0bff83-k8s-agentpool1-16172903-vmss000001-attachment)
I1129 20:16:11.012144       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 20:16:11.012171       1 utils.go:79] GRPC request: {}
I1129 20:16:11.012234       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:16:20.622610       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:16:20.622734       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000001" "disk.csi.azure.com/request-id"="a903e861-7022-11ed-ad3f-000d3a298392" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5c7a35ae-aed4-4e9d-9387-7f606d0bff83" "latency"=12105942341
I1129 20:16:20.622779       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a903e861-7022-11ed-ad3f-000d3a298392" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12106957350
I1129 20:16:22.314304       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1129 20:16:22.314363       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 20:16:22.314397       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5c7a35ae-aed4-4e9d-9387-7f606d0bff83/globalmount with mount options([])
I1129 20:16:22.314410       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 121 lines ...
I1129 20:18:01.565891       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:18:01.565907       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8d7c495c-094b-47ae-83f6-d6af599f4183/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8d7c495c-094b-47ae-83f6-d6af599f4183","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-8d7c495c-094b-47ae-83f6-d6af599f4183"}
I1129 20:18:01.566136       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-8d7c495c-094b-47ae-83f6-d6af599f4183-k8s-agentpool1-16172903-vmss000001-attachment)
I1129 20:18:11.012285       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 20:18:11.012315       1 utils.go:79] GRPC request: {}
I1129 20:18:11.012373       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:18:13.518854       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:18:13.518968       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000001" "disk.csi.azure.com/request-id"="ec6603d3-7022-11ed-ad3f-000d3a298392" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8d7c495c-094b-47ae-83f6-d6af599f4183" "latency"=11952715527
I1129 20:18:13.519018       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ec6603d3-7022-11ed-ad3f-000d3a298392" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11952893929
I1129 20:18:15.176734       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1129 20:18:15.176794       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS
I1129 20:18:15.176835       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8d7c495c-094b-47ae-83f6-d6af599f4183/globalmount with mount options([])
I1129 20:18:15.176849       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 54 lines ...
I1129 20:19:13.724604       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 20:19:13.724618       1 utils.go:79] GRPC request: {}
I1129 20:19:13.724647       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 20:19:13.725433       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:19:13.725446       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5aaa5059-37d4-4ec0-955f-24b7c7fb68c1/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5aaa5059-37d4-4ec0-955f-24b7c7fb68c1","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-nd5mq-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-5aaa5059-37d4-4ec0-955f-24b7c7fb68c1"}
I1129 20:19:13.725629       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-5aaa5059-37d4-4ec0-955f-24b7c7fb68c1-k8s-agentpool1-16172903-vmss000001-attachment)
I1129 20:19:23.366097       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:19:23.366188       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000001" "disk.csi.azure.com/request-id"="1768ae2c-7023-11ed-ad3f-000d3a298392" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5aaa5059-37d4-4ec0-955f-24b7c7fb68c1" "latency"=9640457701
I1129 20:19:23.366221       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1768ae2c-7023-11ed-ad3f-000d3a298392" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9640603803
I1129 20:19:23.943049       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1129 20:19:23.943107       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS
I1129 20:19:23.943141       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5aaa5059-37d4-4ec0-955f-24b7c7fb68c1/globalmount with mount options([])
I1129 20:19:23.943152       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 112 lines ...
I1129 19:23:27.602326       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=1208&timeout=9m28s&timeoutSeconds=568&watch=true 200 OK in 0 milliseconds
I1129 19:23:27.602693       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=1213&timeout=5m35s&timeoutSeconds=335&watch=true 200 OK in 1 milliseconds
I1129 19:23:27.696877       1 shared_informer.go:285] caches populated
I1129 19:23:27.696913       1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-c13cf58a5ca48aed0e92b8b1ecf1b82eae2479e6 e2e-test
I1129 19:23:27.696929       1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider
I1129 19:23:27.699717       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I1129 19:23:27.700127       1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I1129 19:23:27.700143       1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider
I1129 19:23:27.700151       1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I1129 19:23:27.700288       1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully
I1129 19:23:27.701022       1 azure_auth.go:245] Using AzurePublicCloud environment
I1129 19:23:27.701046       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I1129 19:23:27.701073       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
... skipping 1307 lines ...
I1129 19:23:39.000365       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=1208&timeout=6m43s&timeoutSeconds=403&watch=true 200 OK in 1 milliseconds
I1129 19:23:39.000899       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=1343&timeout=6m2s&timeoutSeconds=362&watch=true 200 OK in 1 milliseconds
I1129 19:23:39.096614       1 shared_informer.go:285] caches populated
I1129 19:23:39.096681       1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-c13cf58a5ca48aed0e92b8b1ecf1b82eae2479e6 e2e-test
I1129 19:23:39.096704       1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider
I1129 19:23:39.099236       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I1129 19:23:39.099489       1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I1129 19:23:39.099503       1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider
I1129 19:23:39.099509       1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I1129 19:23:39.099531       1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully
I1129 19:23:39.100172       1 azure_auth.go:245] Using AzurePublicCloud environment
I1129 19:23:39.100201       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I1129 19:23:39.100207       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
... skipping 53 lines ...
I1129 19:23:39.225379       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes 201 Created in 5 milliseconds
I1129 19:23:39.229694       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 3 milliseconds
I1129 19:23:39.230209       1 azuredisk_v2.go:532]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="53e360e9-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).registerAzDriverNode" "latency"=10987406
I1129 19:23:39.230353       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:23:39Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="53e513a9-701b-11ed-b321-000d3a29806d"
I1129 19:23:39.230482       1 server.go:117] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I1129 19:23:39.234321       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 409 Conflict in 3 milliseconds
E1129 19:23:39.234459       1 azuredisk_v2.go:591]  "msg"="Failed to update AzDriverNode status after creation" "error"="Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-agentpool1-16172903-vmss000002\": the object has been modified; please apply your changes to the latest version and try again" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="53e513a9-701b-11ed-b321-000d3a29806d"
E1129 19:23:39.234508       1 azuredisk_v2.go:543]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-agentpool1-16172903-vmss000002\": the object has been modified; please apply your changes to the latest version and try again]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="53e513a9-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=4173440
I1129 19:23:39.234516       1 azuredisk_v2.go:506] Starting heartbeat loop with initial delay 3.731s and frequency 30s
I1129 19:23:39.495054       1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I1129 19:23:39.495082       1 utils.go:79] GRPC request: {}
I1129 19:23:39.496342       1 utils.go:85] GRPC response: {"name":"test.csi.azure.com","vendor_version":"latest-v2-c13cf58a5ca48aed0e92b8b1ecf1b82eae2479e6"}
I1129 19:23:39.512375       1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I1129 19:23:39.512395       1 utils.go:79] GRPC request: {}
... skipping 23 lines ...
I1129 19:24:11.644902       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:24:11.644918       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bfe977e6-3b2c-48a9-8806-892dc05697e7/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-bfe977e6-3b2c-48a9-8806-892dc05697e7","csi.storage.k8s.io/pvc/name":"test.csi.azure.comvqs7w","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-7799","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-bfe977e6-3b2c-48a9-8806-892dc05697e7"}
I1129 19:24:11.645200       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-bfe977e6-3b2c-48a9-8806-892dc05697e7-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:24:12.967334       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:24:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="6800ec88-701b-11ed-b321-000d3a29806d"
I1129 19:24:12.976119       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 8 milliseconds
I1129 19:24:12.976305       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="6800ec88-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9057080
I1129 19:24:15.160233       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:24:15.160323       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="6737311d-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-bfe977e6-3b2c-48a9-8806-892dc05697e7" "latency"=3515031569
I1129 19:24:15.160363       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6737311d-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3515184170
I1129 19:24:17.042293       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1129 19:24:17.042340       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 19:24:17.042368       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bfe977e6-3b2c-48a9-8806-892dc05697e7/globalmount with mount options([])
I1129 19:24:17.042377       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 88 lines ...
I1129 19:25:36.238367       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:25:36.238383       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e47da8b5-14b0-4067-8981-9b953b10be9f/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e47da8b5-14b0-4067-8981-9b953b10be9f","csi.storage.k8s.io/pvc/name":"test.csi.azure.combql7k","csi.storage.k8s.io/pvc/namespace":"multivolume-6147","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-e47da8b5-14b0-4067-8981-9b953b10be9f"}
I1129 19:25:36.238534       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e47da8b5-14b0-4067-8981-9b953b10be9f-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:25:39.002541       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:25:39.002679       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:25:39.002721       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:25:39.002795       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:25:39.002827       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:25:39.126144       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:25:41.108081       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:25:41.108101       1 utils.go:79] GRPC request: {}
I1129 19:25:41.108148       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:25:42.966058       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:25:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="9da5a539-701b-11ed-b321-000d3a29806d"
I1129 19:25:42.975777       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 9 milliseconds
I1129 19:25:42.975970       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="9da5a539-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9949889
I1129 19:25:50.722963       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:25:50.723027       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="99a2dadc-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d2694ce1-f079-4c09-8713-ad4288f0abe6" "latency"=14486124437
I1129 19:25:50.723057       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="99a2dadc-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14486218938
I1129 19:25:50.724785       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:25:50.724859       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="99a31ca6-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e47da8b5-14b0-4067-8981-9b953b10be9f" "latency"=14486263438
I1129 19:25:50.724892       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="99a31ca6-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14486367640
I1129 19:25:51.349438       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1129 19:25:51.349489       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 19:25:51.349517       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e47da8b5-14b0-4067-8981-9b953b10be9f/globalmount with mount options([])
I1129 19:25:51.349527       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 117 lines ...
W1129 19:26:06.942453       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e47da8b5-14b0-4067-8981-9b953b10be9f/globalmount" is not a mountpoint, deleting
I1129 19:26:06.942614       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e47da8b5-14b0-4067-8981-9b953b10be9f/globalmount successfully
I1129 19:26:06.942640       1 utils.go:85] GRPC response: {}
I1129 19:26:09.003538       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:26:09.003590       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:26:09.003651       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:26:09.003699       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:26:09.003721       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:26:09.126981       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:26:11.108095       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:26:11.108127       1 utils.go:79] GRPC request: {}
I1129 19:26:11.108179       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:26:12.966437       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:26:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="af875746-701b-11ed-b321-000d3a29806d"
I1129 19:26:12.972547       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 5 milliseconds
I1129 19:26:12.973193       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="af875746-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6780763
I1129 19:26:26.102866       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:26:26.102940       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="a46dbfb3-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1da56f63-e850-42aa-adc8-941bee85124c" "latency"=31759115238
I1129 19:26:26.102970       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a46dbfb3-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31759233039
I1129 19:26:26.103327       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:26:26.103413       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="a46dc1b3-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-00346b25-349c-4797-aeb3-7a76ae86c2d8" "latency"=31759518842
I1129 19:26:26.103444       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a46dc1b3-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31759652743
I1129 19:26:26.733653       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sde under /dev/disk/azure/scsi1/
I1129 19:26:26.733653       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdf under /dev/disk/azure/scsi1/
I1129 19:26:26.733709       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1129 19:26:26.733717       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
... skipping 239 lines ...
W1129 19:27:03.032310       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1da56f63-e850-42aa-adc8-941bee85124c/globalmount" is not a mountpoint, deleting
I1129 19:27:03.032372       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1da56f63-e850-42aa-adc8-941bee85124c/globalmount successfully
I1129 19:27:03.032384       1 utils.go:85] GRPC response: {}
I1129 19:27:09.004739       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:27:09.004766       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:27:09.004807       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:27:09.004912       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:27:09.128083       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:27:11.108271       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:27:11.108292       1 utils.go:79] GRPC request: {}
I1129 19:27:11.108332       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:27:11.436478       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:27:11.436557       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="c3c30fb9-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fe4e168d-ef99-4b47-b77a-9e98455b8e9d" "latency"=24524251114
I1129 19:27:11.436596       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c3c30fb9-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24524384815
I1129 19:27:12.042956       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1129 19:27:12.043017       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 19:27:12.043052       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fe4e168d-ef99-4b47-b77a-9e98455b8e9d/globalmount with mount options([])
I1129 19:27:12.043064       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 81 lines ...
I1129 19:28:07.476946       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:28:07.476961       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5042c66c-27d3-4322-90e5-d4d43afeaf7c/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5042c66c-27d3-4322-90e5-d4d43afeaf7c","csi.storage.k8s.io/pvc/name":"inline-volume-tester-vxs45-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-144","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-5042c66c-27d3-4322-90e5-d4d43afeaf7c"}
I1129 19:28:07.477095       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-5042c66c-27d3-4322-90e5-d4d43afeaf7c-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:28:09.005590       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:28:09.005644       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:28:09.005604       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:28:09.005754       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:28:09.005792       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:28:09.128885       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:28:11.107827       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:28:11.107848       1 utils.go:79] GRPC request: {}
I1129 19:28:11.107891       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:28:12.967160       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:28:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="f70dff5c-701b-11ed-b321-000d3a29806d"
I1129 19:28:12.975863       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 8 milliseconds
I1129 19:28:12.976488       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="f70dff5c-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9355193
I1129 19:28:34.372533       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:28:34.372597       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="f3c83daa-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-cae64499-050b-434d-9d63-c28851f7cb82" "latency"=26895716673
I1129 19:28:34.372633       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f3c83daa-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=26895837975
I1129 19:28:34.380188       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:28:34.380241       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="f3c848b6-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5042c66c-27d3-4322-90e5-d4d43afeaf7c" "latency"=26903094136
I1129 19:28:34.380261       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f3c848b6-701b-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=26903189338
I1129 19:28:34.679539       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:28:34.679566       1 utils.go:79] GRPC request: {}
I1129 19:28:34.679616       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:28:34.683219       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 81 lines ...
I1129 19:28:35.795621       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:28:35.795643       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a6a1a1e-278f-4268-9d6f-564dc46d1b60/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4a6a1a1e-278f-4268-9d6f-564dc46d1b60","csi.storage.k8s.io/pvc/name":"test.csi.azure.comknz5d","csi.storage.k8s.io/pvc/namespace":"multivolume-1381","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-4a6a1a1e-278f-4268-9d6f-564dc46d1b60"}
I1129 19:28:35.795853       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-4a6a1a1e-278f-4268-9d6f-564dc46d1b60-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:28:39.006371       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:28:39.006406       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:28:39.006442       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:28:39.006527       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:28:39.006551       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:28:39.129963       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:28:41.107701       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:28:41.107723       1 utils.go:79] GRPC request: {}
I1129 19:28:41.107759       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:28:42.967334       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:28:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="08efa89a-701c-11ed-b321-000d3a29806d"
I1129 19:28:42.974412       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 6 milliseconds
I1129 19:28:42.974619       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="08efa89a-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7331069
I1129 19:28:59.711182       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:28:59.711240       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="03ffdea0-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8f9d1fd7-dc0c-4ada-980f-21c504bd6e36" "latency"=25026257681
I1129 19:28:59.711264       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="03ffdea0-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=25026357882
I1129 19:28:59.714867       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:28:59.714925       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="04a961db-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4a6a1a1e-278f-4268-9d6f-564dc46d1b60" "latency"=23919007315
I1129 19:28:59.714966       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="04a961db-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=23919135117
I1129 19:29:00.368297       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdf under /dev/disk/azure/scsi1/
I1129 19:29:00.368341       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1129 19:29:00.368364       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a6a1a1e-278f-4268-9d6f-564dc46d1b60/globalmount with mount options([])
I1129 19:29:00.368373       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 169 lines ...
I1129 19:29:29.171874       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:29:29.171895       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"4"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6b6b9c8a-c056-47c1-865c-5a1e03ec1853/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6b6b9c8a-c056-47c1-865c-5a1e03ec1853","csi.storage.k8s.io/pvc/name":"inline-volume-tester-ztjmg-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-2304","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-6b6b9c8a-c056-47c1-865c-5a1e03ec1853"}
I1129 19:29:29.172060       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-6b6b9c8a-c056-47c1-865c-5a1e03ec1853-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:29:39.007768       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:29:39.007796       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:29:39.007834       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:29:39.007895       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:29:39.131277       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:29:41.003683       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 58 items received
I1129 19:29:41.006230       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=5086&timeout=5m59s&timeoutSeconds=359&watch=true 200 OK in 2 milliseconds
I1129 19:29:41.108426       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:29:41.108451       1 utils.go:79] GRPC request: {}
I1129 19:29:41.108496       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:29:42.966935       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:29:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="2cb2dfac-701c-11ed-b321-000d3a29806d"
I1129 19:29:42.975130       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 7 milliseconds
I1129 19:29:42.975323       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="2cb2dfac-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8419779
I1129 19:30:05.710138       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:30:05.710228       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="2479f24e-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6b6b9c8a-c056-47c1-865c-5a1e03ec1853" "latency"=36538087254
I1129 19:30:05.710265       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2479f24e-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=36538217155
I1129 19:30:06.650342       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:30:06.650370       1 utils.go:79] GRPC request: {}
I1129 19:30:06.650417       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:30:06.653620       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 37 lines ...
I1129 19:30:07.987990       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6b6b9c8a-c056-47c1-865c-5a1e03ec1853/globalmount /var/lib/kubelet/pods/8542912e-7ab6-4729-ba29-53e69df60c74/volumes/kubernetes.io~csi/pvc-6b6b9c8a-c056-47c1-865c-5a1e03ec1853/mount)
I1129 19:30:07.988776       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6b6b9c8a-c056-47c1-865c-5a1e03ec1853/globalmount at /var/lib/kubelet/pods/8542912e-7ab6-4729-ba29-53e69df60c74/volumes/kubernetes.io~csi/pvc-6b6b9c8a-c056-47c1-865c-5a1e03ec1853/mount successfully
I1129 19:30:07.988794       1 utils.go:85] GRPC response: {}
I1129 19:30:09.008910       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:30:09.008997       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:30:09.009023       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:30:09.009621       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:30:09.131625       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:30:09.631284       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:30:09.631301       1 utils.go:79] GRPC request: {}
I1129 19:30:09.631340       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:30:09.632164       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1129 19:30:09.632179       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-6b6b9c8a-c056-47c1-865c-5a1e03ec1853","volume_path":"/var/lib/kubelet/pods/8542912e-7ab6-4729-ba29-53e69df60c74/volumes/kubernetes.io~csi/pvc-6b6b9c8a-c056-47c1-865c-5a1e03ec1853/mount"}
... skipping 3 lines ...
I1129 19:30:11.108086       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:30:12.967461       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:30:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="3e94954a-701c-11ed-b321-000d3a29806d"
I1129 19:30:12.974236       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 6 milliseconds
I1129 19:30:12.974793       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="3e94954a-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7402257
I1129 19:30:22.002393       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 172 items received
I1129 19:30:22.004039       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=5490&timeout=8m46s&timeoutSeconds=526&watch=true 200 OK in 1 milliseconds
I1129 19:30:26.162801       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:30:26.162886       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="3ad177c1-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-662eab62-4f6a-4a86-acf6-054589b4cf61" "latency"=19507242795
I1129 19:30:26.162919       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3ad177c1-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19507419696
I1129 19:30:26.793123       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1129 19:30:26.793172       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 19:30:26.793199       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-662eab62-4f6a-4a86-acf6-054589b4cf61/globalmount with mount options([])
I1129 19:30:26.793208       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 113 lines ...
I1129 19:33:03.789433       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:33:03.790113       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:33:03.790129       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0bac95a0-4d04-44ce-9a19-2680b519c39f/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0bac95a0-4d04-44ce-9a19-2680b519c39f","csi.storage.k8s.io/pvc/name":"pvc-smpsr","csi.storage.k8s.io/pvc/namespace":"provisioning-8915","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-0bac95a0-4d04-44ce-9a19-2680b519c39f"}
I1129 19:33:03.790290       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-0bac95a0-4d04-44ce-9a19-2680b519c39f-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:33:09.012308       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:33:09.012343       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:33:09.012423       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:33:09.012464       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:33:09.135788       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:33:11.108512       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:33:11.108534       1 utils.go:79] GRPC request: {}
I1129 19:33:11.108583       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:33:12.966718       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:33:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="a9de4bbd-701c-11ed-b321-000d3a29806d"
... skipping 4 lines ...
I1129 19:33:25.627419       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:33:25.628849       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1129 19:33:25.628869       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-662eab62-4f6a-4a86-acf6-054589b4cf61","volume_path":"/var/lib/kubelet/pods/97ef061b-0012-42c3-a564-697bd8de5488/volumes/kubernetes.io~csi/pvc-662eab62-4f6a-4a86-acf6-054589b4cf61/mount"}
I1129 19:33:25.628966       1 utils.go:85] GRPC response: {"usage":[{"available":5179580416,"total":5196382208,"unit":1,"used":24576},{"available":327668,"total":327680,"unit":2,"used":12}]}
I1129 19:33:26.919933       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 156 items received
I1129 19:33:26.922561       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=7143&timeout=6m29s&timeoutSeconds=389&watch=true 200 OK in 2 milliseconds
I1129 19:33:31.150343       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:33:31.150408       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="a4661728-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0bac95a0-4d04-44ce-9a19-2680b519c39f" "latency"=27360056024
I1129 19:33:31.150434       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a4661728-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=27360156026
I1129 19:33:31.763040       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1129 19:33:31.763087       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 19:33:31.763121       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0bac95a0-4d04-44ce-9a19-2680b519c39f/globalmount with mount options([])
I1129 19:33:31.763133       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 110 lines ...
W1129 19:35:06.066984       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-662eab62-4f6a-4a86-acf6-054589b4cf61/globalmount" is not a mountpoint, deleting
I1129 19:35:06.067060       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-662eab62-4f6a-4a86-acf6-054589b4cf61/globalmount successfully
I1129 19:35:06.067071       1 utils.go:85] GRPC response: {}
I1129 19:35:09.014908       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:35:09.014999       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:35:09.015013       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:35:09.015114       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:35:09.137400       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:35:11.108133       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:35:11.108155       1 utils.go:79] GRPC request: {}
I1129 19:35:11.108199       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:35:12.966406       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:35:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="f164cb15-701c-11ed-b321-000d3a29806d"
I1129 19:35:12.972027       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 5 milliseconds
I1129 19:35:12.972207       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="f164cb15-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5871756
I1129 19:35:21.170321       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:35:21.170388       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="eaa4dea0-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a43b172f-56cf-4547-8641-e0fa6cb4e5e8" "latency"=19528072790
I1129 19:35:21.170416       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="eaa4dea0-701c-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19528196590
I1129 19:35:21.870276       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1129 19:35:21.870322       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 19:35:21.870350       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a43b172f-56cf-4547-8641-e0fa6cb4e5e8/globalmount with mount options([])
I1129 19:35:21.870359       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 72 lines ...
I1129 19:36:27.722226       1 utils.go:79] GRPC request: {}
I1129 19:36:27.722257       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:36:27.723037       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:36:27.723054       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a8f78b74-6cdd-46d1-a2d4-a80e216adc10","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a8f78b74-6cdd-46d1-a2d4-a80e216adc10","csi.storage.k8s.io/pvc/name":"pvc-kk7vj","csi.storage.k8s.io/pvc/namespace":"provisioning-7654","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-a8f78b74-6cdd-46d1-a2d4-a80e216adc10"}
I1129 19:36:27.723281       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a8f78b74-6cdd-46d1-a2d4-a80e216adc10-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:36:39.016104       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:36:39.016221       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:36:39.016217       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:36:39.016251       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:36:39.139542       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:36:41.108661       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:36:41.108690       1 utils.go:79] GRPC request: {}
I1129 19:36:41.108737       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:36:42.967086       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:36:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="2709cfbf-701d-11ed-b321-000d3a29806d"
I1129 19:36:42.974417       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 7 milliseconds
I1129 19:36:42.974765       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="2709cfbf-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7682711
I1129 19:36:56.872258       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:36:56.872317       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="1df3cb76-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a8f78b74-6cdd-46d1-a2d4-a80e216adc10" "latency"=29148971711
I1129 19:36:56.872338       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1df3cb76-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29149079211
I1129 19:36:58.663943       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1129 19:36:58.663993       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 19:36:58.664009       1 utils.go:85] GRPC response: {}
I1129 19:36:58.667917       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 21 lines ...
I1129 19:37:00.590564       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-a8f78b74-6cdd-46d1-a2d4-a80e216adc10/74f09ebe-55d5-49ae-bb33-4ebf91926d86)
I1129 19:37:00.591553       1 nodeserver_v2.go:353] NodePublishVolume: mount /dev/disk/azure/scsi1/lun0 at /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-a8f78b74-6cdd-46d1-a2d4-a80e216adc10/74f09ebe-55d5-49ae-bb33-4ebf91926d86 successfully
I1129 19:37:00.591572       1 utils.go:85] GRPC response: {}
I1129 19:37:09.016864       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:37:09.017027       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:37:09.017051       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:37:09.017178       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:37:09.140521       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:37:09.636309       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:37:09.636330       1 utils.go:79] GRPC request: {}
I1129 19:37:09.636376       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:37:09.637116       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1129 19:37:09.637143       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-a8f78b74-6cdd-46d1-a2d4-a80e216adc10","volume_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-a8f78b74-6cdd-46d1-a2d4-a80e216adc10/dev/74f09ebe-55d5-49ae-bb33-4ebf91926d86"}
... skipping 18 lines ...
W1129 19:37:12.382416       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a8f78b74-6cdd-46d1-a2d4-a80e216adc10" is not a mountpoint, deleting
I1129 19:37:12.382504       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a8f78b74-6cdd-46d1-a2d4-a80e216adc10 successfully
I1129 19:37:12.382517       1 utils.go:85] GRPC response: {}
I1129 19:37:12.966753       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:37:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="38eb6578-701d-11ed-b321-000d3a29806d"
I1129 19:37:12.978269       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 11 milliseconds
I1129 19:37:12.978454       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="38eb6578-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11732696
I1129 19:37:37.281123       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:37:37.281196       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="30692219-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5cbfe1b4-dcdb-4a49-8638-1d442ecb6cc8" "latency"=38589841954
I1129 19:37:37.281226       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="30692219-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=38589981354
I1129 19:37:37.891239       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1129 19:37:37.891295       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 19:37:37.891326       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5cbfe1b4-dcdb-4a49-8638-1d442ecb6cc8/globalmount with mount options([])
I1129 19:37:37.891336       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 109 lines ...
I1129 19:39:41.108573       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:39:41.108601       1 utils.go:79] GRPC request: {}
I1129 19:39:41.108651       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:39:42.966320       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:39:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="925383ff-701d-11ed-b321-000d3a29806d"
I1129 19:39:42.973497       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 6 milliseconds
I1129 19:39:42.977410       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="925383ff-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11114007
I1129 19:39:50.399830       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:39:50.399907       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="90b81d56-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5628ba37-3794-4dce-b73b-d4a33c769be4" "latency"=10129694066
I1129 19:39:50.399942       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="90b81d56-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10129808466
I1129 19:39:52.185888       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1129 19:39:52.185938       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 19:39:52.185969       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5628ba37-3794-4dce-b73b-d4a33c769be4/globalmount with mount options([nouuid])
I1129 19:39:52.185993       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 84 lines ...
I1129 19:41:25.284257       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:41:25.284276       1 utils.go:79] GRPC request: {}
I1129 19:41:25.284310       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:41:25.285099       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:41:25.285113       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-edb50ce2-6d0c-4bed-846a-8b90f0310d48/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-edb50ce2-6d0c-4bed-846a-8b90f0310d48","csi.storage.k8s.io/pvc/name":"test.csi.azure.compnhvt","csi.storage.k8s.io/pvc/namespace":"snapshotting-3992","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-edb50ce2-6d0c-4bed-846a-8b90f0310d48"}
I1129 19:41:25.285275       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-edb50ce2-6d0c-4bed-846a-8b90f0310d48-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:41:32.510749       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:41:32.510814       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="cf5026dd-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-edb50ce2-6d0c-4bed-846a-8b90f0310d48" "latency"=7225478799
I1129 19:41:32.510839       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cf5026dd-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7225582401
I1129 19:41:34.245952       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1129 19:41:34.246007       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 19:41:34.246036       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-edb50ce2-6d0c-4bed-846a-8b90f0310d48/globalmount with mount options([])
I1129 19:41:34.246046       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 78 lines ...
I1129 19:41:55.935123       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:41:55.935137       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210","csi.storage.k8s.io/pvc/name":"test.csi.azure.comlxpvs","csi.storage.k8s.io/pvc/namespace":"multivolume-4110","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210"}
I1129 19:41:55.935278       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:42:09.022741       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:42:09.022825       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:42:09.022855       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:42:09.022955       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:42:09.022978       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:42:09.145204       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:42:11.108269       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:42:11.108295       1 utils.go:79] GRPC request: {}
I1129 19:42:11.108346       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:42:12.966100       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:42:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="ebbba903-701d-11ed-b321-000d3a29806d"
I1129 19:42:12.973350       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 7 milliseconds
I1129 19:42:12.974778       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="ebbba903-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8740881
I1129 19:42:39.023202       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:42:39.023292       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:42:39.023388       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:42:39.023427       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:42:39.023439       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:42:39.145748       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:42:41.108084       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:42:41.108108       1 utils.go:79] GRPC request: {}
I1129 19:42:41.108154       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:42:42.967037       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:42:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="fd9d71d3-701d-11ed-b321-000d3a29806d"
I1129 19:42:42.975481       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 8 milliseconds
I1129 19:42:42.975750       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="fd9d71d3-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8733182
I1129 19:43:09.023646       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:43:09.023737       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:43:09.023766       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:43:09.023826       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:43:09.023853       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:43:09.146149       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:43:11.108627       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:43:11.108649       1 utils.go:79] GRPC request: {}
I1129 19:43:11.108693       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:43:12.967071       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:43:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="0f7f1648-701e-11ed-b321-000d3a29806d"
I1129 19:43:12.974928       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 7 milliseconds
I1129 19:43:12.975669       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="0f7f1648-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8615880
I1129 19:43:39.024364       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:43:39.024524       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:43:39.024540       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:43:39.024647       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:43:39.024673       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:43:39.146913       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:43:41.107793       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:43:41.107816       1 utils.go:79] GRPC request: {}
I1129 19:43:41.107855       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:43:42.966210       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:43:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="216096e6-701e-11ed-b321-000d3a29806d"
I1129 19:43:42.973867       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 7 milliseconds
I1129 19:43:42.974114       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="216096e6-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7945676
E1129 19:43:55.929286       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="e194f8c8-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210" "latency"=119993911204
E1129 19:43:55.929350       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="e194db3d-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c" "latency"=119994708711
E1129 19:43:55.929363       1 crdprovisioner.go:740]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e194f8c8-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119994086705
E1129 19:43:55.929402       1 crdprovisioner.go:740]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e194db3d-701d-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119994876013
E1129 19:43:55.929423       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c) to be attached to node (k8s-agentpool1-16172903-vmss000002): context canceled
E1129 19:43:55.929381       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210) to be attached to node (k8s-agentpool1-16172903-vmss000002): context canceled
I1129 19:43:56.436229       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:43:56.436258       1 utils.go:79] GRPC request: {}
I1129 19:43:56.436325       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:43:56.436338       1 utils.go:79] GRPC request: {}
I1129 19:43:56.436304       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:43:56.436377       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 15 lines ...
I1129 19:43:56.441362       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210","csi.storage.k8s.io/pvc/name":"test.csi.azure.comlxpvs","csi.storage.k8s.io/pvc/namespace":"multivolume-4110","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210"}
I1129 19:43:56.441514       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:43:56.441535       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:44:09.024851       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:44:09.024914       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:44:09.024942       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:44:09.024991       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:44:09.025047       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:44:09.147335       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:44:11.108120       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:44:11.108148       1 utils.go:79] GRPC request: {}
I1129 19:44:11.108198       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:44:12.967561       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:44:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="33426f0a-701e-11ed-b321-000d3a29806d"
I1129 19:44:12.974542       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 6 milliseconds
I1129 19:44:12.975644       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="33426f0a-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8110277
I1129 19:44:39.025774       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:44:39.025815       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:44:39.025891       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:44:39.025911       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:44:39.025923       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:44:39.148438       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:44:41.108521       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:44:41.108543       1 utils.go:79] GRPC request: {}
I1129 19:44:41.108588       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:44:42.965939       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:44:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="4523d2f0-701e-11ed-b321-000d3a29806d"
I1129 19:44:42.972796       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 6 milliseconds
I1129 19:44:42.973027       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="4523d2f0-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7104069
I1129 19:45:09.026035       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:09.026132       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:09.026136       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:09.026313       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:45:09.026359       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:45:09.149585       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:11.108257       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:45:11.108283       1 utils.go:79] GRPC request: {}
I1129 19:45:11.108329       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:45:12.967797       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:45:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="5705be56-701e-11ed-b321-000d3a29806d"
I1129 19:45:12.977434       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 9 milliseconds
... skipping 2 lines ...
I1129 19:45:27.015673       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=12985&timeout=7m44s&timeoutSeconds=464&watch=true 200 OK in 2 milliseconds
I1129 19:45:29.011100       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 89 items received
I1129 19:45:29.013413       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=13413&timeout=5m43s&timeoutSeconds=343&watch=true 200 OK in 2 milliseconds
I1129 19:45:39.027073       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:39.027152       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:39.027190       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:39.027273       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:45:39.027331       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:45:39.150515       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:41.108465       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:45:41.108489       1 utils.go:79] GRPC request: {}
I1129 19:45:41.108534       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:45:42.966812       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:45:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="68e73aee-701e-11ed-b321-000d3a29806d"
I1129 19:45:42.973594       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 6 milliseconds
I1129 19:45:42.973792       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="68e73aee-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7004557
I1129 19:45:54.134192       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 20 items received
I1129 19:45:54.182275       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-16172903-vmss000002&resourceVersion=13478&timeout=9m21s&timeoutSeconds=561&watch=true 200 OK in 47 milliseconds
E1129 19:45:56.436963       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="2968c453-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210" "latency"=119995323870
E1129 19:45:56.437027       1 crdprovisioner.go:740]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2968c453-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119995497273
E1129 19:45:56.437045       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="2968c392-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c" "latency"=119995347071
E1129 19:45:56.437092       1 crdprovisioner.go:740]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2968c392-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119995582273
E1129 19:45:56.437123       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c) to be attached to node (k8s-agentpool1-16172903-vmss000002): context deadline exceeded
E1129 19:45:56.437047       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210) to be attached to node (k8s-agentpool1-16172903-vmss000002): context canceled
I1129 19:45:57.474674       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:45:57.474709       1 utils.go:79] GRPC request: {}
I1129 19:45:57.474677       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:45:57.474755       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:45:57.474820       1 utils.go:79] GRPC request: {}
I1129 19:45:57.474869       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 15 lines ...
I1129 19:45:57.480219       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:45:57.480212       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210","csi.storage.k8s.io/pvc/name":"test.csi.azure.comlxpvs","csi.storage.k8s.io/pvc/namespace":"multivolume-4110","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210"}
I1129 19:45:57.480362       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:46:09.028039       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:09.028086       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:09.028125       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:09.028174       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:46:09.028205       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:46:09.151697       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:11.108556       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:46:11.108577       1 utils.go:79] GRPC request: {}
I1129 19:46:11.108619       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:46:12.966950       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:46:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="7ac8e310-701e-11ed-b321-000d3a29806d"
I1129 19:46:12.975710       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 8 milliseconds
I1129 19:46:12.976194       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="7ac8e310-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9280485
I1129 19:46:39.028162       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:39.028238       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:39.028265       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:39.028312       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:46:39.028341       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:46:39.152641       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:41.108655       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:46:41.108680       1 utils.go:79] GRPC request: {}
I1129 19:46:41.108728       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:46:42.966457       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:46:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="8caa72b6-701e-11ed-b321-000d3a29806d"
I1129 19:46:42.973395       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 6 milliseconds
I1129 19:46:42.973549       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="8caa72b6-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7131967
I1129 19:46:45.007476       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 98 items received
I1129 19:46:45.011182       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=12851&timeout=5m21s&timeoutSeconds=321&watch=true 200 OK in 3 milliseconds
I1129 19:46:52.799898       1 conditionwatcher.go:187] condition result: succeeded: false, error: rpc error: code = Unknown desc = context deadline exceeded
E1129 19:46:52.800023       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [rpc error: code = Unknown desc = context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="718dce09-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c" "latency"=55319711738
E1129 19:46:52.800055       1 crdprovisioner.go:740]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [rpc error: code = Unknown desc = context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="718dce09-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=55319843139
E1129 19:46:52.800072       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c) to be attached to node (k8s-agentpool1-16172903-vmss000002): rpc error: code = Unknown desc = context deadline exceeded
I1129 19:46:52.800302       1 conditionwatcher.go:187] condition result: succeeded: false, error: rpc error: code = Unknown desc = context deadline exceeded
E1129 19:46:52.800380       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [rpc error: code = Unknown desc = context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="718dd3ab-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210" "latency"=55319939137
E1129 19:46:52.800413       1 crdprovisioner.go:740]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [rpc error: code = Unknown desc = context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="718dd3ab-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=55320055137
E1129 19:46:52.800427       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210) to be attached to node (k8s-agentpool1-16172903-vmss000002): rpc error: code = Unknown desc = context deadline exceeded
I1129 19:46:54.905411       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:46:54.905431       1 utils.go:79] GRPC request: {}
I1129 19:46:54.905475       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:46:54.905490       1 utils.go:79] GRPC request: {}
I1129 19:46:54.905470       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:46:54.905535       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 16 lines ...
I1129 19:46:54.921398       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c-k8s-agentpool1-16172903-vmss000002-attachment/status 200 OK in 9 milliseconds
I1129 19:46:54.921987       1 azure_disk_utils.go:1084]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/request-id"="93c91608-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "latency"=10628094
I1129 19:46:54.922006       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:46:54.927155       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210-k8s-agentpool1-16172903-vmss000002-attachment/status 200 OK in 14 milliseconds
I1129 19:46:54.927327       1 azure_disk_utils.go:1084]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/request-id"="93c92c2e-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "latency"=15448536
I1129 19:46:54.927345       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:46:54.928692       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:46:54.931028       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:46:54.936099       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:46:54.960372       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:46:54.968721       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:46:54.974340       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:46:54.974394       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="93c92c2e-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c6ab73a3-1ca8-413d-96fd-61d13fa45210" "latency"=46972014
I1129 19:46:54.974415       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="93c92c2e-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=62605452
I1129 19:46:55.976049       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:46:55.976123       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="93c91608-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-809fb1d1-3b87-4657-b5ce-1c4a618d5f5c" "latency"=1054058790
I1129 19:46:55.976159       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="93c91608-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1064904786
I1129 19:47:09.028917       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:47:09.029010       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:47:09.029036       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:47:09.153484       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 130 lines ...
I1129 19:49:53.425045       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:49:53.425067       1 utils.go:79] GRPC request: {}
I1129 19:49:53.425114       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:49:53.425806       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:49:53.425821       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7a99d8b0-095d-457c-927a-0edc4f0a5198/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7a99d8b0-095d-457c-927a-0edc4f0a5198","csi.storage.k8s.io/pvc/name":"test.csi.azure.com98xjw","csi.storage.k8s.io/pvc/namespace":"multivolume-426","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-7a99d8b0-095d-457c-927a-0edc4f0a5198"}
I1129 19:49:53.425980       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7a99d8b0-095d-457c-927a-0edc4f0a5198-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:50:08.448773       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:50:08.448800       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:50:08.448837       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="fe1e177a-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5c96045b-ee19-4c8a-9d17-61a7a9df19c4" "latency"=15141931608
I1129 19:50:08.448860       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="fe1e177a-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15142031908
I1129 19:50:08.449145       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="fe304534-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7a99d8b0-095d-457c-927a-0edc4f0a5198" "latency"=15023003922
I1129 19:50:08.449205       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="fe304534-701e-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15023233625
I1129 19:50:08.911597       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:50:08.911621       1 utils.go:79] GRPC request: {}
... skipping 7 lines ...
I1129 19:50:08.919340       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:50:08.919355       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d2216ba4-e59d-4c5b-a1da-72637b806f25/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-d2216ba4-e59d-4c5b-a1da-72637b806f25","csi.storage.k8s.io/pvc/name":"test.csi.azure.com9fmrr","csi.storage.k8s.io/pvc/namespace":"volume-1105","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-d2216ba4-e59d-4c5b-a1da-72637b806f25"}
I1129 19:50:08.919526       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-d2216ba4-e59d-4c5b-a1da-72637b806f25-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:50:09.034056       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:50:09.034224       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:50:09.034256       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:50:09.034348       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:50:09.084324       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1129 19:50:09.084374       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 19:50:09.084409       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5c96045b-ee19-4c8a-9d17-61a7a9df19c4/globalmount with mount options([])
I1129 19:50:09.084430       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
I1129 19:50:09.084501       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1129 19:50:09.084544       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
... skipping 69 lines ...
I1129 19:50:11.108681       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:50:11.108706       1 utils.go:79] GRPC request: {}
I1129 19:50:11.108751       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:50:12.967216       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:50:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="09d6055d-701f-11ed-b321-000d3a29806d"
I1129 19:50:12.973405       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 5 milliseconds
I1129 19:50:12.973678       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="09d6055d-701f-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6481558
I1129 19:50:23.792271       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:50:23.792363       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="076c65e6-701f-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d2216ba4-e59d-4c5b-a1da-72637b806f25" "latency"=14872762057
I1129 19:50:23.792397       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="076c65e6-701f-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14872879859
I1129 19:50:24.413061       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1129 19:50:24.413105       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1129 19:50:24.413133       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d2216ba4-e59d-4c5b-a1da-72637b806f25/globalmount with mount options([])
I1129 19:50:24.413142       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 159 lines ...
I1129 19:53:21.333990       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:53:21.334003       1 utils.go:79] GRPC request: {}
I1129 19:53:21.334028       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:53:21.334785       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:53:21.334802       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5d364888-256b-4ec5-9309-af7be594ad61/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5d364888-256b-4ec5-9309-af7be594ad61","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-tzzlg-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-2269","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-5d364888-256b-4ec5-9309-af7be594ad61"}
I1129 19:53:21.334976       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-5d364888-256b-4ec5-9309-af7be594ad61-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:53:38.222832       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:53:38.222927       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="7a1caa73-701f-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5d364888-256b-4ec5-9309-af7be594ad61" "latency"=16887871917
I1129 19:53:38.222959       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7a1caa73-701f-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16887991917
I1129 19:53:39.038596       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:53:39.038609       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:53:39.038622       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:53:39.162891       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 116 lines ...
I1129 19:55:19.161174       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:55:19.161196       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6334d105-0733-476d-9592-ee734b7e4581/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6334d105-0733-476d-9592-ee734b7e4581","csi.storage.k8s.io/pvc/name":"test.csi.azure.com2zx7z","csi.storage.k8s.io/pvc/namespace":"multivolume-4975","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-6334d105-0733-476d-9592-ee734b7e4581"}
I1129 19:55:19.161377       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-6334d105-0733-476d-9592-ee734b7e4581-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 19:55:39.042762       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:55:39.042847       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:55:39.042832       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:55:39.042919       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:55:39.042960       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:55:39.165115       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:55:41.108639       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:55:41.108665       1 utils.go:79] GRPC request: {}
I1129 19:55:41.108717       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:55:42.966634       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:55:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="ce87efe5-701f-11ed-b321-000d3a29806d"
I1129 19:55:42.972889       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 5 milliseconds
I1129 19:55:42.973753       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="ce87efe5-701f-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7151969
I1129 19:55:43.780828       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:55:43.780875       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="c01abc0a-701f-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fd1ac11f-2bf8-4663-88cf-7654d93e7d4a" "latency"=25017978800
I1129 19:55:43.780905       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c01abc0a-701f-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=25018091702
I1129 19:55:43.781680       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:55:43.781743       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="c0578c38-701f-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6334d105-0733-476d-9592-ee734b7e4581" "latency"=24620306609
I1129 19:55:43.781944       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c0578c38-701f-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24620531912
I1129 19:55:44.402153       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1129 19:55:44.402199       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 19:55:44.402225       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6334d105-0733-476d-9592-ee734b7e4581/globalmount with mount options([nouuid])
I1129 19:55:44.402238       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 309 lines ...
I1129 20:05:33.643574       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:05:33.643591       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2c3e05f1-c75a-46fc-b676-4d41e67dab63/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2c3e05f1-c75a-46fc-b676-4d41e67dab63","csi.storage.k8s.io/pvc/name":"test.csi.azure.comjwnb8","csi.storage.k8s.io/pvc/namespace":"snapshotting-9843","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-2c3e05f1-c75a-46fc-b676-4d41e67dab63"}
I1129 20:05:33.643741       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-2c3e05f1-c75a-46fc-b676-4d41e67dab63-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 20:05:39.058070       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:05:39.058122       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:05:39.058161       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:05:39.058207       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:05:39.180487       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:05:41.108626       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 20:05:41.108649       1 utils.go:79] GRPC request: {}
I1129 20:05:41.108697       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:05:42.967218       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T20:05:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="3428c211-7021-11ed-b321-000d3a29806d"
I1129 20:05:42.979135       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 11 milliseconds
I1129 20:05:42.979339       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="3428c211-7021-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12167783
I1129 20:05:53.150947       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:05:53.151005       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="2e9a1d7e-7021-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2c3e05f1-c75a-46fc-b676-4d41e67dab63" "latency"=19507196882
I1129 20:05:53.151033       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2e9a1d7e-7021-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19507305982
I1129 20:05:54.842044       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1129 20:05:54.842090       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 20:05:54.842115       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2c3e05f1-c75a-46fc-b676-4d41e67dab63/globalmount with mount options([])
I1129 20:05:54.842123       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 64 lines ...
I1129 20:06:38.328012       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 20:06:38.328741       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:06:38.328755       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-780aa584-a242-410f-9d9c-dfd1c0e914c6/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-780aa584-a242-410f-9d9c-dfd1c0e914c6","csi.storage.k8s.io/pvc/name":"test.csi.azure.comc67rh","csi.storage.k8s.io/pvc/namespace":"volume-1571","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-780aa584-a242-410f-9d9c-dfd1c0e914c6"}
I1129 20:06:38.328915       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-780aa584-a242-410f-9d9c-dfd1c0e914c6-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 20:06:39.060019       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:06:39.060040       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:06:39.060170       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:06:39.060055       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:06:39.182401       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:06:41.108265       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 20:06:41.108285       1 utils.go:79] GRPC request: {}
I1129 20:06:41.108323       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:06:42.967787       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T20:06:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="57ec1ede-7021-11ed-b321-000d3a29806d"
I1129 20:06:42.979091       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000002/status 200 OK in 11 milliseconds
I1129 20:06:42.979315       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="57ec1ede-7021-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11559010
I1129 20:06:55.098441       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:06:55.098505       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="55284a31-7021-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-780aa584-a242-410f-9d9c-dfd1c0e914c6" "latency"=16769518363
I1129 20:06:55.098532       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="55284a31-7021-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16769629764
I1129 20:06:56.791143       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1129 20:06:56.791193       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 20:06:56.791222       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-780aa584-a242-410f-9d9c-dfd1c0e914c6/globalmount with mount options([])
I1129 20:06:56.791231       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 299 lines ...
I1129 20:18:34.151346       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 20:18:34.151386       1 utils.go:79] GRPC request: {}
I1129 20:18:34.151421       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 20:18:34.152256       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:18:34.152280       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1c1f7073-cb52-4fad-aedf-614a99ef8c20/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1c1f7073-cb52-4fad-aedf-614a99ef8c20","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-nonroot-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-1c1f7073-cb52-4fad-aedf-614a99ef8c20"}
I1129 20:18:34.152477       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-1c1f7073-cb52-4fad-aedf-614a99ef8c20-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 20:18:35.639662       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:18:35.639739       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="ffd24b5d-7022-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1c1f7073-cb52-4fad-aedf-614a99ef8c20" "latency"=1487188407
I1129 20:18:35.639777       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ffd24b5d-7022-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1487328309
I1129 20:18:37.419710       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1129 20:18:37.419766       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS
I1129 20:18:37.419800       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1c1f7073-cb52-4fad-aedf-614a99ef8c20/globalmount with mount options([])
I1129 20:18:37.419817       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 60 lines ...
I1129 20:19:14.923328       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 20:19:14.923342       1 utils.go:79] GRPC request: {}
I1129 20:19:14.923398       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 20:19:14.924149       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:19:14.924162       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d737e238-a991-46f1-a4c5-685341d7a396/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-d737e238-a991-46f1-a4c5-685341d7a396","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-fbtdn-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-d737e238-a991-46f1-a4c5-685341d7a396"}
I1129 20:19:14.924343       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-d737e238-a991-46f1-a4c5-685341d7a396-k8s-agentpool1-16172903-vmss000002-attachment)
I1129 20:19:23.891186       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:19:23.891248       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000002" "disk.csi.azure.com/request-id"="181f969c-7023-11ed-b321-000d3a29806d" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d737e238-a991-46f1-a4c5-685341d7a396" "latency"=8966837016
I1129 20:19:23.891269       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="181f969c-7023-11ed-b321-000d3a29806d" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8966954018
I1129 20:19:24.540018       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1129 20:19:24.540075       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS
I1129 20:19:24.540105       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d737e238-a991-46f1-a4c5-685341d7a396/globalmount with mount options([])
I1129 20:19:24.540114       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 116 lines ...
I1129 19:23:25.525824       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=1212&timeout=7m40s&timeoutSeconds=460&watch=true 200 OK in 1 milliseconds
I1129 19:23:25.525883       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=1213&timeout=9m42s&timeoutSeconds=582&watch=true 200 OK in 1 milliseconds
I1129 19:23:25.622210       1 shared_informer.go:285] caches populated
I1129 19:23:25.622262       1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-c13cf58a5ca48aed0e92b8b1ecf1b82eae2479e6 e2e-test
I1129 19:23:25.622273       1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider
I1129 19:23:25.625673       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 3 milliseconds
I1129 19:23:25.625912       1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I1129 19:23:25.625928       1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider
I1129 19:23:25.625934       1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I1129 19:23:25.625955       1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully
I1129 19:23:25.626644       1 azure_auth.go:245] Using AzurePublicCloud environment
I1129 19:23:25.626665       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I1129 19:23:25.626671       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
... skipping 101 lines ...
I1129 19:24:11.305109       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:24:11.305329       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:24:11.305128       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9ee7b1b9-4f00-4f61-8479-7aa20265181f/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9ee7b1b9-4f00-4f61-8479-7aa20265181f","csi.storage.k8s.io/pvc/name":"test.csi.azure.comdcvmf","csi.storage.k8s.io/pvc/namespace":"multivolume-3336","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-9ee7b1b9-4f00-4f61-8479-7aa20265181f"}
I1129 19:24:11.305355       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a","csi.storage.k8s.io/pvc/name":"test.csi.azure.comnzcm4","csi.storage.k8s.io/pvc/namespace":"provisioning-9367","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a"}
I1129 19:24:11.305506       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-9ee7b1b9-4f00-4f61-8479-7aa20265181f-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 19:24:11.305513       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 19:24:15.957708       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:24:15.957810       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="67035c02-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a" "latency"=4652175774
I1129 19:24:15.957838       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="67035c02-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4652354376
I1129 19:24:15.963235       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:24:15.963283       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="67035bf8-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9ee7b1b9-4f00-4f61-8479-7aa20265181f" "latency"=4657716061
I1129 19:24:15.963304       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="67035bf8-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4657823463
I1129 19:24:16.537328       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1129 19:24:16.537394       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 19:24:16.537428       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9ee7b1b9-4f00-4f61-8479-7aa20265181f/globalmount with mount options([nouuid])
I1129 19:24:16.537479       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 100 lines ...
W1129 19:24:22.985081       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9ee7b1b9-4f00-4f61-8479-7aa20265181f/globalmount" is not a mountpoint, deleting
I1129 19:24:22.985155       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9ee7b1b9-4f00-4f61-8479-7aa20265181f/globalmount successfully
I1129 19:24:22.985167       1 utils.go:85] GRPC response: {}
I1129 19:24:25.526255       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:24:25.526263       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:24:25.526276       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:24:25.526429       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:24:25.526482       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:24:25.591731       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1129 19:24:25.591757       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/3a0401e7-f73c-41a0-abc1-05c6d512682a/volumes/kubernetes.io~csi/pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a"}
I1129 19:24:25.591806       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a on /var/lib/kubelet/pods/3a0401e7-f73c-41a0-abc1-05c6d512682a/volumes/kubernetes.io~csi/pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a/mount
I1129 19:24:25.591841       1 mount_helper_common.go:99] "/var/lib/kubelet/pods/3a0401e7-f73c-41a0-abc1-05c6d512682a/volumes/kubernetes.io~csi/pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a/mount" is a mountpoint, unmounting
I1129 19:24:25.591858       1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/3a0401e7-f73c-41a0-abc1-05c6d512682a/volumes/kubernetes.io~csi/pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a/mount
W1129 19:24:25.593105       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/pods/3a0401e7-f73c-41a0-abc1-05c6d512682a/volumes/kubernetes.io~csi/pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a/mount" is not a mountpoint, deleting
... skipping 8 lines ...
I1129 19:24:25.694811       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a/globalmount
I1129 19:24:25.694843       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a/globalmount" is a mountpoint, unmounting
I1129 19:24:25.694854       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a/globalmount
W1129 19:24:25.707137       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a/globalmount" is not a mountpoint, deleting
I1129 19:24:25.707202       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-092eb803-4180-4cc9-b0ba-030d1ebeb12a/globalmount successfully
I1129 19:24:25.707213       1 utils.go:85] GRPC response: {}
I1129 19:24:26.255024       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:24:26.255101       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="6a5fc16c-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d2694ce1-f079-4c09-8713-ad4288f0abe6" "latency"=9310843711
I1129 19:24:26.255124       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6a5fc16c-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9310948613
I1129 19:24:26.255785       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:24:26.255842       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="6a503743-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e47da8b5-14b0-4067-8981-9b953b10be9f" "latency"=9413405936
I1129 19:24:26.255864       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6a503743-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9413529738
I1129 19:24:26.863154       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sde under /dev/disk/azure/scsi1/
I1129 19:24:26.863208       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1129 19:24:26.863245       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d2694ce1-f079-4c09-8713-ad4288f0abe6/globalmount with mount options([])
I1129 19:24:26.863257       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3])
... skipping 208 lines ...
I1129 19:25:14.461324       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:25:14.461343       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-468fecb6-04d4-4fb0-9397-b6b1f54e741f/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-468fecb6-04d4-4fb0-9397-b6b1f54e741f","csi.storage.k8s.io/pvc/name":"test.csi.azure.comtjrnn","csi.storage.k8s.io/pvc/namespace":"multivolume-8875","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-468fecb6-04d4-4fb0-9397-b6b1f54e741f"}
I1129 19:25:14.461538       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-468fecb6-04d4-4fb0-9397-b6b1f54e741f-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 19:25:25.527567       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:25:25.527649       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:25:25.527682       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:25:25.527781       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:25:25.651908       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:25:34.038493       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:25:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="9853673a-701b-11ed-ba40-000d3a2987eb"
I1129 19:25:34.046095       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 7 milliseconds
I1129 19:25:34.046258       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="9853673a-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7826720
I1129 19:25:40.958622       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:25:40.958651       1 utils.go:79] GRPC request: {}
I1129 19:25:40.958696       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:25:49.078688       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:25:49.078761       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="8ca834a8-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-468fecb6-04d4-4fb0-9397-b6b1f54e741f" "latency"=34617153919
I1129 19:25:49.078783       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8ca834a8-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=34617264421
I1129 19:25:50.013469       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:25:50.013492       1 utils.go:79] GRPC request: {}
I1129 19:25:50.013532       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:25:50.014301       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 33 lines ...
I1129 19:25:51.014064       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-468fecb6-04d4-4fb0-9397-b6b1f54e741f/globalmount /var/lib/kubelet/pods/2dce165a-0367-4603-8273-0b5afc3ab83b/volumes/kubernetes.io~csi/pvc-468fecb6-04d4-4fb0-9397-b6b1f54e741f/mount)
I1129 19:25:51.019275       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-468fecb6-04d4-4fb0-9397-b6b1f54e741f/globalmount at /var/lib/kubelet/pods/2dce165a-0367-4603-8273-0b5afc3ab83b/volumes/kubernetes.io~csi/pvc-468fecb6-04d4-4fb0-9397-b6b1f54e741f/mount successfully
I1129 19:25:51.019295       1 utils.go:85] GRPC response: {}
I1129 19:25:55.527772       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:25:55.527852       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:25:55.527869       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:25:55.527941       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:25:55.652199       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:26:04.038688       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:26:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="aa351297-701b-11ed-ba40-000d3a2987eb"
I1129 19:26:04.050624       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 11 milliseconds
I1129 19:26:04.051524       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="aa351297-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12882011
I1129 19:26:10.958748       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:26:10.958776       1 utils.go:79] GRPC request: {}
I1129 19:26:10.958827       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:26:24.520100       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:26:24.520162       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="a1d94d9a-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a9ca3536-8843-4396-aa9d-55e398cd974e" "latency"=34504622781
I1129 19:26:24.520186       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a1d94d9a-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=34504747583
I1129 19:26:25.101855       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1129 19:26:25.101914       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 19:26:25.101938       1 utils.go:85] GRPC response: {}
I1129 19:26:25.108785       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 48 lines ...
I1129 19:26:38.104872       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
I1129 19:26:38.104886       1 utils.go:79] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a9ca3536-8843-4396-aa9d-55e398cd974e","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-a9ca3536-8843-4396-aa9d-55e398cd974e"}
I1129 19:26:38.104929       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a9ca3536-8843-4396-aa9d-55e398cd974e
W1129 19:26:38.105247       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a9ca3536-8843-4396-aa9d-55e398cd974e" is not a mountpoint, deleting
I1129 19:26:38.105308       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a9ca3536-8843-4396-aa9d-55e398cd974e successfully
I1129 19:26:38.105317       1 utils.go:85] GRPC response: {}
I1129 19:26:39.858990       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:26:39.859066       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="b750e9cc-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b348dfc3-61b9-48ba-9bba-779f5868ec13" "latency"=13827513700
I1129 19:26:39.859092       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b750e9cc-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=13827623302
I1129 19:26:40.453499       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1129 19:26:40.453563       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1129 19:26:40.453592       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b348dfc3-61b9-48ba-9bba-779f5868ec13/globalmount with mount options([nouuid])
I1129 19:26:40.453605       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 87 lines ...
I1129 19:27:10.959101       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:27:10.959126       1 utils.go:79] GRPC request: {}
I1129 19:27:10.959167       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:27:25.530307       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:27:25.530370       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:27:25.530407       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:27:25.530532       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:27:25.654127       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:27:34.040219       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:27:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="dfda3707-701b-11ed-ba40-000d3a2987eb"
I1129 19:27:34.051270       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 10 milliseconds
I1129 19:27:34.051409       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="dfda3707-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11246985
I1129 19:27:40.958132       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:27:40.958160       1 utils.go:79] GRPC request: {}
I1129 19:27:40.958207       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:27:46.921872       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:27:46.921941       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="cea81b25-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0553f8b6-f717-40b6-a431-11917896339d" "latency"=41731368725
I1129 19:27:46.921962       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cea81b25-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=41731470927
I1129 19:27:48.702552       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/
I1129 19:27:48.702607       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1129 19:27:48.702642       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0553f8b6-f717-40b6-a431-11917896339d/globalmount with mount options([])
I1129 19:27:48.702655       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3])
... skipping 36 lines ...
I1129 19:27:49.005313       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:27:49.005331       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9690fd2f-864c-4f90-8761-6948ee2c57ae/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9690fd2f-864c-4f90-8761-6948ee2c57ae","csi.storage.k8s.io/pvc/name":"test.csi.azure.com4zjn5","csi.storage.k8s.io/pvc/namespace":"provisioning-6278","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-9690fd2f-864c-4f90-8761-6948ee2c57ae"}
I1129 19:27:49.005493       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-9690fd2f-864c-4f90-8761-6948ee2c57ae-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 19:27:55.530760       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:27:55.530786       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:27:55.530819       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:27:55.530905       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:27:55.655148       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:27:57.276629       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:27:57.276705       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="e8c5bdfa-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9690fd2f-864c-4f90-8761-6948ee2c57ae" "latency"=8271144192
I1129 19:27:57.276728       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e8c5bdfa-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8271251994
I1129 19:27:57.470094       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:27:57.470121       1 utils.go:79] GRPC request: {}
I1129 19:27:57.470170       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:27:57.473334       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 62 lines ...
I1129 19:28:10.958533       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:28:10.958557       1 utils.go:79] GRPC request: {}
I1129 19:28:10.958608       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:28:25.531013       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:28:25.531086       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:28:25.531130       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:28:25.531316       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:28:25.655375       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:28:28.467502       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1129 19:28:28.467530       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/2178f0b1-9341-4bdf-86c4-bf56a3bc4129/volumes/kubernetes.io~csi/pvc-0553f8b6-f717-40b6-a431-11917896339d/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-0553f8b6-f717-40b6-a431-11917896339d"}
I1129 19:28:28.467581       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-0553f8b6-f717-40b6-a431-11917896339d on /var/lib/kubelet/pods/2178f0b1-9341-4bdf-86c4-bf56a3bc4129/volumes/kubernetes.io~csi/pvc-0553f8b6-f717-40b6-a431-11917896339d/mount
I1129 19:28:28.467609       1 mount_helper_common.go:99] "/var/lib/kubelet/pods/2178f0b1-9341-4bdf-86c4-bf56a3bc4129/volumes/kubernetes.io~csi/pvc-0553f8b6-f717-40b6-a431-11917896339d/mount" is a mountpoint, unmounting
I1129 19:28:28.467625       1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/2178f0b1-9341-4bdf-86c4-bf56a3bc4129/volumes/kubernetes.io~csi/pvc-0553f8b6-f717-40b6-a431-11917896339d/mount
... skipping 8 lines ...
I1129 19:28:28.569644       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0553f8b6-f717-40b6-a431-11917896339d/globalmount
I1129 19:28:28.569677       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0553f8b6-f717-40b6-a431-11917896339d/globalmount" is a mountpoint, unmounting
I1129 19:28:28.569696       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0553f8b6-f717-40b6-a431-11917896339d/globalmount
W1129 19:28:28.576369       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0553f8b6-f717-40b6-a431-11917896339d/globalmount" is not a mountpoint, deleting
I1129 19:28:28.576430       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0553f8b6-f717-40b6-a431-11917896339d/globalmount successfully
I1129 19:28:28.576445       1 utils.go:85] GRPC response: {}
I1129 19:28:32.698877       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:28:32.698960       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="edd21b31-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fe4e168d-ef99-4b47-b77a-9e98455b8e9d" "latency"=35223751271
I1129 19:28:32.698987       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="edd21b31-701b-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=35223869172
I1129 19:28:33.284922       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/
I1129 19:28:33.284973       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 19:28:33.285004       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fe4e168d-ef99-4b47-b77a-9e98455b8e9d/globalmount with mount options([])
I1129 19:28:33.285013       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 42 lines ...
I1129 19:28:40.958182       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:28:40.958205       1 utils.go:79] GRPC request: {}
I1129 19:28:40.958242       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:28:55.531874       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:28:55.531911       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:28:55.531961       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:28:55.532040       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:28:55.656370       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:29:04.038943       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:29:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="157eee8d-701c-11ed-ba40-000d3a2987eb"
I1129 19:29:04.046002       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 6 milliseconds
I1129 19:29:04.046161       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="157eee8d-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7263319
I1129 19:29:05.527926       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 137 items received
I1129 19:29:05.532977       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=4553&timeout=5m30s&timeoutSeconds=330&watch=true 200 OK in 4 milliseconds
I1129 19:29:10.958625       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:29:10.958647       1 utils.go:79] GRPC request: {}
I1129 19:29:10.958683       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:29:25.532280       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:29:25.532364       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:29:25.532418       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:29:25.532435       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:29:25.656824       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:29:34.039476       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:29:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="2760a64e-701c-11ed-ba40-000d3a2987eb"
I1129 19:29:34.050391       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 10 milliseconds
I1129 19:29:34.050621       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="2760a64e-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11189990
I1129 19:29:40.958049       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:29:40.958072       1 utils.go:79] GRPC request: {}
I1129 19:29:40.958110       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:29:43.893805       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:29:43.893890       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="0555f339-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-27d2cba7-a35e-45bd-ba19-d6471872096c" "latency"=66967030848
I1129 19:29:43.893914       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0555f339-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=66967151650
I1129 19:29:44.467689       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/
I1129 19:29:44.467741       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1129 19:29:44.467771       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-27d2cba7-a35e-45bd-ba19-d6471872096c/globalmount with mount options([])
I1129 19:29:44.467780       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 84 lines ...
I1129 19:29:53.984826       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fe4e168d-ef99-4b47-b77a-9e98455b8e9d/globalmount" is a mountpoint, unmounting
I1129 19:29:53.984838       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fe4e168d-ef99-4b47-b77a-9e98455b8e9d/globalmount
W1129 19:29:53.994683       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fe4e168d-ef99-4b47-b77a-9e98455b8e9d/globalmount" is not a mountpoint, deleting
I1129 19:29:53.994746       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fe4e168d-ef99-4b47-b77a-9e98455b8e9d/globalmount successfully
I1129 19:29:53.994761       1 utils.go:85] GRPC response: {}
I1129 19:29:55.532987       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:29:55.533112       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:29:55.533138       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:29:55.533184       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:29:55.533201       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:29:55.657675       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:30:04.042631       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:30:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="3942c382-701c-11ed-ba40-000d3a2987eb"
I1129 19:30:04.052290       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 9 milliseconds
I1129 19:30:04.052509       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="3942c382-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9951066
I1129 19:30:07.037930       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:30:07.038004       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="2f203e39-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4a6a1a1e-278f-4268-9d6f-564dc46d1b60" "latency"=19998834653
I1129 19:30:07.038029       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2f203e39-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19998931254
I1129 19:30:07.042180       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:30:07.042244       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="2f10ffe9-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8f9d1fd7-dc0c-4ada-980f-21c504bd6e36" "latency"=20102957586
I1129 19:30:07.042295       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2f10ffe9-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20103089888
I1129 19:30:07.635806       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/
I1129 19:30:07.635869       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1129 19:30:07.635906       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a6a1a1e-278f-4268-9d6f-564dc46d1b60/globalmount with mount options([])
I1129 19:30:07.635918       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3])
... skipping 129 lines ...
I1129 19:30:34.044432       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="4b23c9f9-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5870692
I1129 19:30:40.958624       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:30:40.958650       1 utils.go:79] GRPC request: {}
I1129 19:30:40.958687       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:30:55.533708       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:30:55.533749       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:30:55.533843       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:30:55.533857       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:30:55.658343       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:31:04.038510       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:31:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="5d05682c-701c-11ed-ba40-000d3a2987eb"
I1129 19:31:04.052746       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 14 milliseconds
I1129 19:31:04.052960       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="5d05682c-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14522339
I1129 19:31:05.527399       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 82 items received
I1129 19:31:05.540293       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=5996&timeout=8m43s&timeoutSeconds=523&watch=true 200 OK in 12 milliseconds
I1129 19:31:09.543089       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:31:09.543183       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="4894405c-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6708960b-f29c-40e0-a394-07c66fc77fba" "latency"=39800655388
I1129 19:31:09.543227       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4894405c-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=39800810091
I1129 19:31:10.127615       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1129 19:31:10.127679       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 19:31:10.127720       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6708960b-f29c-40e0-a394-07c66fc77fba/globalmount with mount options([])
I1129 19:31:10.127738       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 61 lines ...
W1129 19:31:18.982853       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6708960b-f29c-40e0-a394-07c66fc77fba/globalmount" is not a mountpoint, deleting
I1129 19:31:18.982927       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6708960b-f29c-40e0-a394-07c66fc77fba/globalmount successfully
I1129 19:31:18.982947       1 utils.go:85] GRPC response: {}
I1129 19:31:25.534050       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:31:25.534149       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:31:25.534168       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:31:25.534255       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:31:25.659492       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:31:34.041773       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:31:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="6ee78c60-701c-11ed-ba40-000d3a2987eb"
I1129 19:31:34.050572       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 8 milliseconds
I1129 19:31:34.050761       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="6ee78c60-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9027643
I1129 19:31:40.958681       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:31:40.958706       1 utils.go:79] GRPC request: {}
I1129 19:31:40.958752       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:31:55.534634       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:31:55.534672       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:31:55.534710       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:31:55.534795       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:31:55.660104       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:32:04.039068       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:32:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="80c8c4f4-701c-11ed-ba40-000d3a2987eb"
I1129 19:32:04.050674       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 11 milliseconds
I1129 19:32:04.050950       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="80c8c4f4-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11936897
I1129 19:32:05.123210       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:32:05.123295       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="618ab7be-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2f4df2e0-a564-4328-bbd5-f82711ff9deb" "latency"=53500197429
I1129 19:32:05.123325       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="618ab7be-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=53500356931
I1129 19:32:05.700870       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/
I1129 19:32:05.700923       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1129 19:32:05.700948       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2f4df2e0-a564-4328-bbd5-f82711ff9deb/globalmount with mount options([])
I1129 19:32:05.700958       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 71 lines ...
I1129 19:32:51.620570       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:32:51.620586       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5d8792e4-d78a-4839-9f3d-2117603faab6/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5d8792e4-d78a-4839-9f3d-2117603faab6","csi.storage.k8s.io/pvc/name":"test.csi.azure.com57fb9","csi.storage.k8s.io/pvc/namespace":"provisioning-8901","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-5d8792e4-d78a-4839-9f3d-2117603faab6"}
I1129 19:32:51.620742       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-5d8792e4-d78a-4839-9f3d-2117603faab6-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 19:32:55.535844       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:32:55.535971       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:32:55.536061       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:32:55.536016       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:32:55.660642       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:33:04.038988       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:33:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="a48c081b-701c-11ed-ba40-000d3a2987eb"
I1129 19:33:04.046299       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 7 milliseconds
I1129 19:33:04.046740       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="a48c081b-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7802064
I1129 19:33:07.527352       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 341 items received
I1129 19:33:07.531374       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=6989&timeout=6m55s&timeoutSeconds=415&watch=true 200 OK in 3 milliseconds
I1129 19:33:10.958213       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:33:10.958237       1 utils.go:79] GRPC request: {}
I1129 19:33:10.958277       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:33:25.536991       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:33:25.537074       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:33:25.537081       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:33:25.537211       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:33:25.661519       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:33:31.696192       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:33:31.696253       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="9d252a1b-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5d8792e4-d78a-4839-9f3d-2117603faab6" "latency"=40075445440
I1129 19:33:31.696273       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9d252a1b-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=40075550342
I1129 19:33:33.325050       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1129 19:33:33.325100       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 19:33:33.325135       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5d8792e4-d78a-4839-9f3d-2117603faab6/globalmount with mount options([])
I1129 19:33:33.325144       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 95 lines ...
W1129 19:33:39.188885       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5d8792e4-d78a-4839-9f3d-2117603faab6/globalmount" is not a mountpoint, deleting
I1129 19:33:39.188959       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5d8792e4-d78a-4839-9f3d-2117603faab6/globalmount successfully
I1129 19:33:39.188979       1 utils.go:85] GRPC response: {}
I1129 19:33:40.958368       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:33:40.958395       1 utils.go:79] GRPC request: {}
I1129 19:33:40.958467       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:33:52.224091       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:33:52.224179       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="b621ac88-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a43b172f-56cf-4547-8641-e0fa6cb4e5e8" "latency"=18683201055
I1129 19:33:52.224211       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b621ac88-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=18683315656
I1129 19:33:52.226628       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:33:52.226687       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="b64fbf5d-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7caaba16-cf57-4b40-a37b-ca6af544d434" "latency"=18383781613
I1129 19:33:52.226719       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b64fbf5d-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=18383872614
I1129 19:33:52.226832       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:33:52.226895       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="b68c2336-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-df59479a-cc80-4b97-b7a0-1afa8ddd80c1" "latency"=17988214149
I1129 19:33:52.226938       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b68c2336-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=17988313150
I1129 19:33:52.235334       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:33:52.235391       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="b640934b-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a797a718-97ba-4d94-b46b-e901ba0a2103" "latency"=18491907857
I1129 19:33:52.235419       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b640934b-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=18492015858
I1129 19:33:52.888766       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:33:52.888789       1 utils.go:79] GRPC request: {}
I1129 19:33:52.888823       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:33:52.889598       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 127 lines ...
I1129 19:33:54.295097       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a43b172f-56cf-4547-8641-e0fa6cb4e5e8/globalmount /var/lib/kubelet/pods/82c5229a-f325-4690-aa3d-d6f90daf0e3c/volumes/kubernetes.io~csi/pvc-a43b172f-56cf-4547-8641-e0fa6cb4e5e8/mount)
I1129 19:33:54.295915       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a43b172f-56cf-4547-8641-e0fa6cb4e5e8/globalmount at /var/lib/kubelet/pods/82c5229a-f325-4690-aa3d-d6f90daf0e3c/volumes/kubernetes.io~csi/pvc-a43b172f-56cf-4547-8641-e0fa6cb4e5e8/mount successfully
I1129 19:33:54.295929       1 utils.go:85] GRPC response: {}
I1129 19:33:55.538102       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:33:55.538137       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:33:55.538168       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:33:55.538252       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:33:55.662504       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:34:03.169236       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1129 19:34:03.169258       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/82c5229a-f325-4690-aa3d-d6f90daf0e3c/volumes/kubernetes.io~csi/pvc-a43b172f-56cf-4547-8641-e0fa6cb4e5e8/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-a43b172f-56cf-4547-8641-e0fa6cb4e5e8"}
I1129 19:34:03.169327       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-a43b172f-56cf-4547-8641-e0fa6cb4e5e8 on /var/lib/kubelet/pods/82c5229a-f325-4690-aa3d-d6f90daf0e3c/volumes/kubernetes.io~csi/pvc-a43b172f-56cf-4547-8641-e0fa6cb4e5e8/mount
I1129 19:34:03.169374       1 mount_helper_common.go:99] "/var/lib/kubelet/pods/82c5229a-f325-4690-aa3d-d6f90daf0e3c/volumes/kubernetes.io~csi/pvc-a43b172f-56cf-4547-8641-e0fa6cb4e5e8/mount" is a mountpoint, unmounting
I1129 19:34:03.169392       1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/82c5229a-f325-4690-aa3d-d6f90daf0e3c/volumes/kubernetes.io~csi/pvc-a43b172f-56cf-4547-8641-e0fa6cb4e5e8/mount
... skipping 74 lines ...
I1129 19:34:10.958671       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:34:10.958699       1 utils.go:79] GRPC request: {}
I1129 19:34:10.958746       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:34:25.539058       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:34:25.539107       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:34:25.539068       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:34:25.539198       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:34:25.663430       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:34:34.038584       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:34:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="da30e181-701c-11ed-ba40-000d3a2987eb"
I1129 19:34:34.047518       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 8 milliseconds
I1129 19:34:34.047729       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="da30e181-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9191934
I1129 19:34:35.532263       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 118 items received
I1129 19:34:35.534692       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=7745&timeout=8m56s&timeoutSeconds=536&watch=true 200 OK in 2 milliseconds
I1129 19:34:37.959130       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:34:37.959207       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="c1aa2ffd-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-121506df-0938-4e65-83cb-d6351eabaf87" "latency"=45068634682
I1129 19:34:37.959238       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c1aa2ffd-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=45068751782
I1129 19:34:38.552175       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun5 by sdc under /dev/disk/azure/scsi1/
I1129 19:34:38.552232       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun5. perfProfile none accountType 
I1129 19:34:38.552399       1 utils.go:85] GRPC response: {}
I1129 19:34:38.561345       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 21 lines ...
I1129 19:34:44.844783       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:34:44.844799       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-5c4e43ff-34c5-40d1-aa97-c55ae9668165","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5c4e43ff-34c5-40d1-aa97-c55ae9668165","csi.storage.k8s.io/pvc/name":"test.csi.azure.comjqnhq-cloned","csi.storage.k8s.io/pvc/namespace":"multivolume-1349","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-5c4e43ff-34c5-40d1-aa97-c55ae9668165"}
I1129 19:34:44.844944       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-5c4e43ff-34c5-40d1-aa97-c55ae9668165-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 19:34:55.539538       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:34:55.539561       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:34:55.539587       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:34:55.539687       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:34:55.664019       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:35:04.039266       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:35:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="ec129f21-701c-11ed-ba40-000d3a2987eb"
I1129 19:35:04.045009       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 5 milliseconds
I1129 19:35:04.045222       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="ec129f21-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5996298
I1129 19:35:10.958777       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:35:10.958799       1 utils.go:79] GRPC request: {}
I1129 19:35:10.958847       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:35:18.159561       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:35:18.159587       1 utils.go:79] GRPC request: {}
I1129 19:35:18.159630       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:35:18.160414       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1129 19:35:18.160433       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-121506df-0938-4e65-83cb-d6351eabaf87","volume_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-121506df-0938-4e65-83cb-d6351eabaf87/dev/6112ae56-c4ca-4a19-b44b-2a49c5aef9f9"}
I1129 19:35:18.161164       1 utils.go:85] GRPC response: {"usage":[{"total":5368709120,"unit":1}]}
I1129 19:35:18.903129       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:35:18.903239       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="e0a1ce90-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5c4e43ff-34c5-40d1-aa97-c55ae9668165" "latency"=34058213524
I1129 19:35:18.903274       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e0a1ce90-701c-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=34058342026
I1129 19:35:19.498817       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1129 19:35:19.498861       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 19:35:19.498871       1 utils.go:85] GRPC response: {}
I1129 19:35:19.504990       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 103 lines ...
I1129 19:36:40.958370       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:36:40.958399       1 utils.go:79] GRPC request: {}
I1129 19:36:40.958471       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:36:55.541676       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:36:55.541704       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:36:55.541759       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:36:55.541865       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:36:55.541902       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:36:55.541922       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:36:55.666414       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:36:56.131756       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:36:56.131831       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="1dae9dcb-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e142fb1a-ebf8-4f43-8f7d-a3db94bd7c16" "latency"=28861838622
I1129 19:36:56.131857       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1dae9dcb-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=28861962924
I1129 19:36:56.138755       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:36:56.138809       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="1daea040-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f0e4e81d-cdf2-4169-9add-abfbf085cf9d" "latency"=28868788239
I1129 19:36:56.138831       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1daea040-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=28868873641
I1129 19:36:56.139602       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:36:56.139674       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="1dae9de6-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-bd540ea4-e5ac-46f3-934b-8c59ee6f28c8" "latency"=28869663355
I1129 19:36:56.139707       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1dae9de6-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=28869801856
I1129 19:36:57.087731       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/
I1129 19:36:57.087806       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 19:36:57.087822       1 utils.go:85] GRPC response: {}
I1129 19:36:57.088201       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
... skipping 140 lines ...
W1129 19:37:12.468910       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e142fb1a-ebf8-4f43-8f7d-a3db94bd7c16" is not a mountpoint, deleting
I1129 19:37:12.468981       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-e142fb1a-ebf8-4f43-8f7d-a3db94bd7c16 successfully
I1129 19:37:12.468997       1 utils.go:85] GRPC response: {}
I1129 19:37:25.542624       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:37:25.542661       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:37:25.542705       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:37:25.542761       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:37:25.667099       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:37:34.043280       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:37:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="457b6b41-701d-11ed-ba40-000d3a2987eb"
I1129 19:37:34.053806       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 10 milliseconds
I1129 19:37:34.054003       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="457b6b41-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10756974
I1129 19:37:36.741374       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:37:36.741455       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="3657f2fb-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5628ba37-3794-4dce-b73b-d4a33c769be4" "latency"=28096406305
I1129 19:37:36.741487       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3657f2fb-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=28096527906
I1129 19:37:37.261698       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:37:37.261722       1 utils.go:79] GRPC request: {}
I1129 19:37:37.261764       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:37:37.265073       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 58 lines ...
I1129 19:37:47.424106       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5628ba37-3794-4dce-b73b-d4a33c769be4/globalmount
W1129 19:37:47.437362       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5628ba37-3794-4dce-b73b-d4a33c769be4/globalmount" is not a mountpoint, deleting
I1129 19:37:47.437429       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5628ba37-3794-4dce-b73b-d4a33c769be4/globalmount successfully
I1129 19:37:47.437440       1 utils.go:85] GRPC response: {}
I1129 19:37:55.543644       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:37:55.543735       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:37:55.543775       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:37:55.543795       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:37:55.668199       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:38:04.042568       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:38:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="575cf2b2-701d-11ed-ba40-000d3a2987eb"
I1129 19:38:04.057932       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 15 milliseconds
I1129 19:38:04.058160       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="575cf2b2-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15621863
I1129 19:38:07.168330       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:38:07.168398       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="476747bd-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7f2e2c43-0206-419a-ac12-5b8579de7a51" "latency"=29901614493
I1129 19:38:07.168420       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="476747bd-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29901727494
I1129 19:38:07.594674       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:38:07.594697       1 utils.go:79] GRPC request: {}
I1129 19:38:07.594742       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:38:07.595633       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 117 lines ...
W1129 19:38:23.621666       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7f2e2c43-0206-419a-ac12-5b8579de7a51/globalmount" is not a mountpoint, deleting
I1129 19:38:23.621742       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7f2e2c43-0206-419a-ac12-5b8579de7a51/globalmount successfully
I1129 19:38:23.621759       1 utils.go:85] GRPC response: {}
I1129 19:38:25.544361       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:38:25.544377       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:38:25.544387       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:38:25.544621       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:38:25.668728       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:38:34.042251       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:38:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="693e8951-701d-11ed-ba40-000d3a2987eb"
I1129 19:38:34.054312       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 11 milliseconds
I1129 19:38:34.054557       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="693e8951-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12330700
I1129 19:38:40.958789       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:38:40.958814       1 utils.go:79] GRPC request: {}
I1129 19:38:40.958856       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:38:44.656733       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 18 items received
I1129 19:38:44.659034       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-16172903-vmss000000&resourceVersion=10392&timeout=6m42s&timeoutSeconds=402&watch=true 200 OK in 2 milliseconds
I1129 19:38:55.544905       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:38:55.544894       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:38:55.545016       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:38:55.545114       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:38:55.668948       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:39:04.042844       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:39:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="7b20438c-701d-11ed-ba40-000d3a2987eb"
I1129 19:39:04.052256       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 9 milliseconds
I1129 19:39:04.052417       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="7b20438c-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9599756
I1129 19:39:07.922755       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:39:07.922825       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="597b4595-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f90a5dfd-0487-4f7b-a3e8-6f51824f9c11" "latency"=60326038177
I1129 19:39:07.922850       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="597b4595-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=60326150679
I1129 19:39:08.509542       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun5 by sdc under /dev/disk/azure/scsi1/
I1129 19:39:08.509598       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun5. perfProfile none accountType 
I1129 19:39:08.509610       1 utils.go:85] GRPC response: {}
I1129 19:39:08.524182       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 30 lines ...
I1129 19:39:14.506214       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1129 19:39:14.506233       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-f90a5dfd-0487-4f7b-a3e8-6f51824f9c11","volume_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-f90a5dfd-0487-4f7b-a3e8-6f51824f9c11/dev/33fff1e2-509c-456a-8a58-e52bc6a8345c"}
I1129 19:39:14.506969       1 utils.go:85] GRPC response: {"usage":[{"total":5368709120,"unit":1}]}
I1129 19:39:25.545234       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:39:25.545325       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:39:25.545342       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:39:25.545403       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:39:25.669733       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:39:34.038459       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:39:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="8d01399d-701d-11ed-ba40-000d3a2987eb"
I1129 19:39:34.046148       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 7 milliseconds
I1129 19:39:34.046393       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="8d01399d-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7996531
I1129 19:39:40.958673       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:39:40.958695       1 utils.go:79] GRPC request: {}
... skipping 14 lines ...
I1129 19:39:43.162305       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-f90a5dfd-0487-4f7b-a3e8-6f51824f9c11
W1129 19:39:43.162679       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-f90a5dfd-0487-4f7b-a3e8-6f51824f9c11" is not a mountpoint, deleting
I1129 19:39:43.162758       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-f90a5dfd-0487-4f7b-a3e8-6f51824f9c11 successfully
I1129 19:39:43.162773       1 utils.go:85] GRPC response: {}
I1129 19:39:48.540070       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 80 items received
I1129 19:39:48.544099       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=10873&timeout=6m3s&timeoutSeconds=363&watch=true 200 OK in 3 milliseconds
I1129 19:39:49.190149       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:39:49.190228       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="7ddb8027-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4109b778-dfb5-4502-a935-004f50460025" "latency"=40564811314
I1129 19:39:49.190263       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7ddb8027-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=40564933416
I1129 19:39:49.783306       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1129 19:39:49.783371       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 19:39:49.783411       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4109b778-dfb5-4502-a935-004f50460025/globalmount with mount options([nouuid])
I1129 19:39:49.783428       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 48 lines ...
I1129 19:39:51.933119       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:39:51.933134       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-745003db-41b5-4588-bb4f-fe02fc965fbc/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-745003db-41b5-4588-bb4f-fe02fc965fbc","csi.storage.k8s.io/pvc/name":"test.csi.azure.comp48tc","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-6782","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-745003db-41b5-4588-bb4f-fe02fc965fbc"}
I1129 19:39:51.933330       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-745003db-41b5-4588-bb4f-fe02fc965fbc-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 19:39:55.545780       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:39:55.545868       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:39:55.545914       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:39:55.545996       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:39:55.546037       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:39:55.670262       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:39:59.474126       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:39:59.474193       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="97abc69b-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-745003db-41b5-4588-bb4f-fe02fc965fbc" "latency"=7540803394
I1129 19:39:59.474218       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="97abc69b-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7540902996
I1129 19:39:59.479007       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:39:59.479070       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="96f36ea5-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-13aeadac-81f4-4a45-b590-7d84364fc7f6" "latency"=8753780526
I1129 19:39:59.479106       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="96f36ea5-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8753895728
I1129 19:40:00.068667       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdf under /dev/disk/azure/scsi1/
I1129 19:40:00.068696       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1129 19:40:00.068716       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 19:40:00.068730       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
... skipping 225 lines ...
I1129 19:41:13.537620       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:41:13.537636       1 utils.go:79] GRPC request: {}
I1129 19:41:13.537664       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:41:13.538424       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:41:13.538469       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-040f12d8-ebf6-49cd-b236-67c7e8d63e0a","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-040f12d8-ebf6-49cd-b236-67c7e8d63e0a","csi.storage.k8s.io/pvc/name":"test.csi.azure.com9pc4x","csi.storage.k8s.io/pvc/namespace":"multivolume-7015","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-040f12d8-ebf6-49cd-b236-67c7e8d63e0a"}
I1129 19:41:13.538656       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-040f12d8-ebf6-49cd-b236-67c7e8d63e0a-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 19:41:19.243689       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:41:19.243765       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="c836172f-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8ed24491-1106-4093-a2e3-7f686d1a1913" "latency"=5873238595
I1129 19:41:19.243788       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c836172f-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5873379997
I1129 19:41:19.248431       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:41:19.248484       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="c84fc26f-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-040f12d8-ebf6-49cd-b236-67c7e8d63e0a" "latency"=5709772198
I1129 19:41:19.248530       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c84fc26f-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5709889900
I1129 19:41:19.632121       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:41:19.632145       1 utils.go:79] GRPC request: {}
I1129 19:41:19.632191       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:41:19.633978       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 46 lines ...
I1129 19:41:20.424887       1 utils.go:85] GRPC response: {}
I1129 19:41:20.424915       1 nodeserver_v2.go:353] NodePublishVolume: mount /dev/disk/azure/scsi1/lun0 at /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-8ed24491-1106-4093-a2e3-7f686d1a1913/6cca6a9a-44e5-49ea-afdc-6d33e2fcb89f successfully
I1129 19:41:20.424956       1 utils.go:85] GRPC response: {}
I1129 19:41:25.548092       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:41:25.548187       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:41:25.548218       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:41:25.548289       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:41:25.548308       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:41:25.671522       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:41:29.543598       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:41:29.543665       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="cbf250c3-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e5567487-d946-4ff9-be0c-74254c5a7896" "latency"=9906461245
I1129 19:41:29.543686       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cbf250c3-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9906560447
I1129 19:41:29.552461       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:41:29.552515       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="cbf21ede-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-36ce4cf6-a4ff-4673-aec4-c1e246ee26e5" "latency"=9916586212
I1129 19:41:29.552536       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cbf21ede-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9916688814
I1129 19:41:30.142735       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1129 19:41:30.142791       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1129 19:41:30.142806       1 utils.go:85] GRPC response: {}
I1129 19:41:30.142907       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/
... skipping 126 lines ...
I1129 19:41:44.526074       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e5567487-d946-4ff9-be0c-74254c5a7896/globalmount
I1129 19:41:44.526095       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e5567487-d946-4ff9-be0c-74254c5a7896/globalmount" is a mountpoint, unmounting
I1129 19:41:44.526108       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e5567487-d946-4ff9-be0c-74254c5a7896/globalmount
W1129 19:41:44.537614       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e5567487-d946-4ff9-be0c-74254c5a7896/globalmount" is not a mountpoint, deleting
I1129 19:41:44.537682       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e5567487-d946-4ff9-be0c-74254c5a7896/globalmount successfully
I1129 19:41:44.537696       1 utils.go:85] GRPC response: {}
I1129 19:41:44.786785       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:41:44.786849       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="d2522b5e-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d9209029-cc60-45be-8d4f-411d67de4846" "latency"=14455122794
I1129 19:41:44.786872       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d2522b5e-701d-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14455228896
I1129 19:41:45.371231       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdg under /dev/disk/azure/scsi1/
I1129 19:41:45.371282       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType 
I1129 19:41:45.371471       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun4 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d9209029-cc60-45be-8d4f-411d67de4846/globalmount with mount options([])
I1129 19:41:45.371489       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun4" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun4])
... skipping 199 lines ...
I1129 19:44:10.959773       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:44:10.959799       1 utils.go:79] GRPC request: {}
I1129 19:44:10.959851       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:44:25.555845       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:44:25.555948       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:44:25.555963       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:44:25.556035       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:44:25.675343       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:44:34.039221       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:44:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="3fd1b60b-701e-11ed-ba40-000d3a2987eb"
I1129 19:44:34.054598       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 15 milliseconds
I1129 19:44:34.054839       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="3fd1b60b-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15673756
I1129 19:44:40.958585       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:44:40.958608       1 utils.go:79] GRPC request: {}
I1129 19:44:40.958654       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:44:55.563175       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:44:55.563264       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:44:55.563309       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:44:55.563330       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:44:55.675567       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:04.039607       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:45:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="51b36860-701e-11ed-ba40-000d3a2987eb"
I1129 19:45:04.062025       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 22 milliseconds
I1129 19:45:04.063061       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="51b36860-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=23495786
I1129 19:45:10.958954       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:45:10.958977       1 utils.go:79] GRPC request: {}
I1129 19:45:10.959017       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:45:25.563936       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:25.564071       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:25.564109       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:25.564243       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:45:25.676515       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:26.660802       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 20 items received
I1129 19:45:26.664805       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-16172903-vmss000000&resourceVersion=13353&timeout=5m0s&timeoutSeconds=300&watch=true 200 OK in 3 milliseconds
I1129 19:45:34.038656       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:45:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="6394e654-701e-11ed-ba40-000d3a2987eb"
I1129 19:45:34.057786       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 18 milliseconds
I1129 19:45:34.057999       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="6394e654-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=19377519
... skipping 2 lines ...
I1129 19:45:40.958221       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:45:51.546076       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 56 items received
I1129 19:45:51.548267       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=13478&timeout=5m46s&timeoutSeconds=346&watch=true 200 OK in 2 milliseconds
I1129 19:45:55.564066       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:55.564167       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:55.564228       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:45:55.564265       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:45:55.677627       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
E1129 19:45:57.208899       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="29de812a-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-51940753-833a-447f-bfb9-2dd92417a0e4" "latency"=119995612537
E1129 19:45:57.208981       1 crdprovisioner.go:740]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="29de812a-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119995838841
E1129 19:45:57.209002       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-51940753-833a-447f-bfb9-2dd92417a0e4) to be attached to node (k8s-agentpool1-16172903-vmss000000): context canceled
I1129 19:45:57.754177       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:45:57.754197       1 utils.go:79] GRPC request: {}
I1129 19:45:57.754233       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:45:57.757305       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:45:57.757323       1 utils.go:79] GRPC request: {}
I1129 19:45:57.757356       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 8 lines ...
I1129 19:46:04.055536       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="7576af0d-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15944962
I1129 19:46:10.958681       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:46:10.958706       1 utils.go:79] GRPC request: {}
I1129 19:46:10.958760       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:46:25.570118       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:25.570240       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:25.570251       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:46:25.570270       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:25.678650       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:34.038630       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:46:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="87582aeb-701e-11ed-ba40-000d3a2987eb"
I1129 19:46:34.055669       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 16 milliseconds
I1129 19:46:34.056309       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="87582aeb-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=17736492
I1129 19:46:40.541684       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 85 items received
I1129 19:46:40.557022       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=12985&timeout=5m9s&timeoutSeconds=309&watch=true 200 OK in 15 milliseconds
I1129 19:46:40.958628       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:46:40.958655       1 utils.go:79] GRPC request: {}
I1129 19:46:40.958703       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:46:55.570489       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:55.570526       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:55.570601       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:55.570608       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:46:55.679077       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:46:59.462870       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 81 items received
I1129 19:46:59.467032       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=13863&timeout=5m0s&timeoutSeconds=300&watch=true 200 OK in 4 milliseconds
I1129 19:47:04.039135       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:47:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="9939e1a5-701e-11ed-ba40-000d3a2987eb"
I1129 19:47:04.045849       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 6 milliseconds
I1129 19:47:04.046089       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="9939e1a5-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7001215
I1129 19:47:10.958384       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:47:10.958413       1 utils.go:79] GRPC request: {}
I1129 19:47:10.958485       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:47:25.570774       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:47:25.570872       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:47:25.570888       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:47:25.571030       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:47:25.679178       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:47:34.040052       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:47:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="ab1ba95b-701e-11ed-ba40-000d3a2987eb"
I1129 19:47:34.054318       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 14 milliseconds
I1129 19:47:34.054607       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="ab1ba95b-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14576040
I1129 19:47:40.958402       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:47:40.958424       1 utils.go:79] GRPC request: {}
I1129 19:47:40.958483       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:47:55.572596       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:47:55.572697       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:47:55.572712       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:47:55.572881       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:47:55.680213       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
E1129 19:47:57.753917       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="71b85985-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-51940753-833a-447f-bfb9-2dd92417a0e4" "latency"=119994783342
E1129 19:47:57.754008       1 crdprovisioner.go:740]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="71b85985-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119994963845
E1129 19:47:57.754029       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-51940753-833a-447f-bfb9-2dd92417a0e4) to be attached to node (k8s-agentpool1-16172903-vmss000000): context deadline exceeded
I1129 19:47:58.825468       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:47:58.825498       1 utils.go:79] GRPC request: {}
I1129 19:47:58.825546       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:47:58.829438       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:47:58.829457       1 utils.go:79] GRPC request: {}
I1129 19:47:58.829496       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 9 lines ...
I1129 19:48:10.958217       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:48:10.958245       1 utils.go:79] GRPC request: {}
I1129 19:48:10.958297       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:48:25.573636       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:48:25.573689       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:48:25.573670       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:48:25.573806       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:48:25.681005       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:48:34.042855       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:48:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="cedf5cea-701e-11ed-ba40-000d3a2987eb"
I1129 19:48:34.058034       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 14 milliseconds
I1129 19:48:34.058226       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="cedf5cea-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15397251
I1129 19:48:36.254511       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:48:36.254579       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="b9e27b56-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-51940753-833a-447f-bfb9-2dd92417a0e4" "latency"=37423401826
I1129 19:48:36.254603       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b9e27b56-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=37423504028
I1129 19:48:38.115686       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun5 by sdc under /dev/disk/azure/scsi1/
I1129 19:48:38.115731       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun5. perfProfile none accountType 
I1129 19:48:38.115756       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun5 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-51940753-833a-447f-bfb9-2dd92417a0e4/globalmount with mount options([])
I1129 19:48:38.115764       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun5" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun5])
... skipping 59 lines ...
W1129 19:48:41.745820       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-51940753-833a-447f-bfb9-2dd92417a0e4/globalmount" is not a mountpoint, deleting
I1129 19:48:41.745883       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-51940753-833a-447f-bfb9-2dd92417a0e4/globalmount successfully
I1129 19:48:41.745892       1 utils.go:85] GRPC response: {}
I1129 19:48:55.573846       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:48:55.573991       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:48:55.574014       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:48:55.574168       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:48:55.681521       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:49:04.038683       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:49:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="e0c05c6f-701e-11ed-ba40-000d3a2987eb"
I1129 19:49:04.049426       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 10 milliseconds
I1129 19:49:04.049599       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="e0c05c6f-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10965096
I1129 19:49:10.959029       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:49:10.959052       1 utils.go:79] GRPC request: {}
I1129 19:49:10.959089       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:49:25.574179       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:49:25.574231       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:49:25.574274       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:49:25.574349       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:49:25.682665       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:49:34.038870       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:49:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="f2a20750-701e-11ed-ba40-000d3a2987eb"
I1129 19:49:34.045765       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 6 milliseconds
I1129 19:49:34.045931       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="f2a20750-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7093416
I1129 19:49:40.959394       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:49:40.959424       1 utils.go:79] GRPC request: {}
I1129 19:49:40.959476       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:49:52.415571       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:49:52.415650       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="d270d6ad-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-24ce38c8-1b18-4a71-95f6-396fd8f3daa4" "latency"=72386185319
I1129 19:49:52.415675       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d270d6ad-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=72386308422
I1129 19:49:53.041056       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:49:53.041088       1 utils.go:79] GRPC request: {}
I1129 19:49:53.041135       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:49:53.044378       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 37 lines ...
I1129 19:49:54.571882       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-24ce38c8-1b18-4a71-95f6-396fd8f3daa4/globalmount /var/lib/kubelet/pods/34ed0b1c-1f3f-4c55-95ce-4b3bde6026c1/volumes/kubernetes.io~csi/pvc-24ce38c8-1b18-4a71-95f6-396fd8f3daa4/mount)
I1129 19:49:54.572692       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-24ce38c8-1b18-4a71-95f6-396fd8f3daa4/globalmount at /var/lib/kubelet/pods/34ed0b1c-1f3f-4c55-95ce-4b3bde6026c1/volumes/kubernetes.io~csi/pvc-24ce38c8-1b18-4a71-95f6-396fd8f3daa4/mount successfully
I1129 19:49:54.572715       1 utils.go:85] GRPC response: {}
I1129 19:49:55.574505       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:49:55.574547       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:49:55.574548       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:49:55.574710       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:49:55.683810       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:49:58.541554       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 69 items received
I1129 19:49:58.544600       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=15253&timeout=6m48s&timeoutSeconds=408&watch=true 200 OK in 2 milliseconds
I1129 19:50:04.039407       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:50:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="0483bf24-701f-11ed-ba40-000d3a2987eb"
I1129 19:50:04.047314       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 7 milliseconds
I1129 19:50:04.047782       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="0483bf24-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8407337
I1129 19:50:07.665254       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:50:07.665334       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="fdf653a2-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-edb50ce2-6d0c-4bed-846a-8b90f0310d48" "latency"=14619012955
I1129 19:50:07.665357       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="fdf653a2-701e-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14619135157
I1129 19:50:08.264939       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1129 19:50:08.265385       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 19:50:08.265421       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-edb50ce2-6d0c-4bed-846a-8b90f0310d48/globalmount with mount options([])
I1129 19:50:08.265431       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 55 lines ...
I1129 19:50:11.256982       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-edb50ce2-6d0c-4bed-846a-8b90f0310d48/globalmount
I1129 19:50:11.257018       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-edb50ce2-6d0c-4bed-846a-8b90f0310d48/globalmount" is a mountpoint, unmounting
I1129 19:50:11.257037       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-edb50ce2-6d0c-4bed-846a-8b90f0310d48/globalmount
W1129 19:50:11.266163       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-edb50ce2-6d0c-4bed-846a-8b90f0310d48/globalmount" is not a mountpoint, deleting
I1129 19:50:11.266229       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-edb50ce2-6d0c-4bed-846a-8b90f0310d48/globalmount successfully
I1129 19:50:11.266239       1 utils.go:85] GRPC response: {}
I1129 19:50:22.956402       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:50:22.956486       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="0734569d-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b8a9b8fb-204d-421e-a62d-ee149d76ee0b" "latency"=14404267403
I1129 19:50:22.956519       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0734569d-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14404397005
I1129 19:50:23.536744       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1129 19:50:23.536794       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1129 19:50:23.536822       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b8a9b8fb-204d-421e-a62d-ee149d76ee0b/globalmount with mount options([])
I1129 19:50:23.536832       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 37 lines ...
I1129 19:50:25.366251       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:50:25.366266       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"3"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-13e3fb59-9d0c-423b-860a-9c44c7372bba/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-13e3fb59-9d0c-423b-860a-9c44c7372bba","csi.storage.k8s.io/pvc/name":"pvc-xd2qb","csi.storage.k8s.io/pvc/namespace":"snapshotting-3992","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-13e3fb59-9d0c-423b-860a-9c44c7372bba"}
I1129 19:50:25.366408       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-13e3fb59-9d0c-423b-860a-9c44c7372bba-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 19:50:25.574612       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:50:25.574640       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:50:25.574659       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:50:25.574732       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:50:25.684599       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:50:26.665578       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 15 items received
I1129 19:50:26.668693       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-16172903-vmss000000&resourceVersion=15460&timeout=9m52s&timeoutSeconds=592&watch=true 200 OK in 2 milliseconds
I1129 19:50:27.078744       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:50:27.078765       1 utils.go:79] GRPC request: {}
I1129 19:50:27.078800       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 43 lines ...
I1129 19:50:34.048741       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="16659827-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7975231
I1129 19:50:40.958824       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:50:40.958843       1 utils.go:79] GRPC request: {}
I1129 19:50:40.958881       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:50:55.574850       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:50:55.574863       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:50:55.575001       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:50:55.574879       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:50:55.685216       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:51:04.039262       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:51:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="2846ff3f-701f-11ed-ba40-000d3a2987eb"
I1129 19:51:04.048202       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 8 milliseconds
I1129 19:51:04.052948       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="2846ff3f-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13721224
I1129 19:51:10.958078       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:51:10.958100       1 utils.go:79] GRPC request: {}
I1129 19:51:10.958147       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:51:25.582137       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:51:25.582170       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:51:25.582196       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:51:25.582270       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:51:25.685730       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:51:34.040689       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:51:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="3a28d934-701f-11ed-ba40-000d3a2987eb"
I1129 19:51:34.046997       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 6 milliseconds
I1129 19:51:34.047200       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="3a28d934-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6569107
I1129 19:51:37.547259       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 52 items received
I1129 19:51:37.549160       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=16031&timeout=7m41s&timeoutSeconds=461&watch=true 200 OK in 1 milliseconds
I1129 19:51:40.958699       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:51:40.958719       1 utils.go:79] GRPC request: {}
I1129 19:51:40.958759       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:51:43.934587       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:51:43.934653       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="1139fe46-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-13e3fb59-9d0c-423b-860a-9c44c7372bba" "latency"=78568189050
I1129 19:51:43.934680       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1139fe46-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=78568283952
I1129 19:51:44.529884       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdd under /dev/disk/azure/scsi1/
I1129 19:51:44.529931       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1129 19:51:44.529958       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-13e3fb59-9d0c-423b-860a-9c44c7372bba/globalmount with mount options([])
I1129 19:51:44.529968       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3])
... skipping 35 lines ...
I1129 19:51:44.823982       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:51:44.824662       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:51:44.824677       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d2216ba4-e59d-4c5b-a1da-72637b806f25/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-d2216ba4-e59d-4c5b-a1da-72637b806f25","csi.storage.k8s.io/pvc/name":"test.csi.azure.com9fmrr","csi.storage.k8s.io/pvc/namespace":"volume-1105","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-d2216ba4-e59d-4c5b-a1da-72637b806f25"}
I1129 19:51:44.824821       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-d2216ba4-e59d-4c5b-a1da-72637b806f25-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 19:51:49.556299       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 151 items received
I1129 19:51:49.562611       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=16098&timeout=8m51s&timeoutSeconds=531&watch=true 200 OK in 6 milliseconds
I1129 19:51:54.283438       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:51:54.283507       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="40966284-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d2216ba4-e59d-4c5b-a1da-72637b806f25" "latency"=9458617549
I1129 19:51:54.283529       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="40966284-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9458724551
I1129 19:51:54.871014       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdf under /dev/disk/azure/scsi1/
I1129 19:51:54.871080       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 19:51:54.871120       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d2216ba4-e59d-4c5b-a1da-72637b806f25/globalmount with mount options([])
I1129 19:51:54.871143       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 70 lines ...
W1129 19:52:07.585274       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d2216ba4-e59d-4c5b-a1da-72637b806f25/globalmount" is not a mountpoint, deleting
I1129 19:52:07.585343       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d2216ba4-e59d-4c5b-a1da-72637b806f25/globalmount successfully
I1129 19:52:07.585363       1 utils.go:85] GRPC response: {}
I1129 19:52:10.958794       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:52:10.958815       1 utils.go:79] GRPC request: {}
I1129 19:52:10.958861       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:52:20.078272       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:52:20.078568       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="4e17c614-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1f724373-4240-4d3b-a71a-6ea60bdf4475" "latency"=12595119038
I1129 19:52:20.078630       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4e17c614-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12595458343
I1129 19:52:20.082730       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:52:20.082787       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="4e17c698-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2f03295b-cf9a-462d-af2b-09ef6bf21bb5" "latency"=12599557610
I1129 19:52:20.082816       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4e17c698-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12599648512
I1129 19:52:20.296323       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1129 19:52:20.296349       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/2affec37-b1eb-407c-be28-d395490a8ae5/volumes/kubernetes.io~csi/pvc-13e3fb59-9d0c-423b-860a-9c44c7372bba/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-13e3fb59-9d0c-423b-860a-9c44c7372bba"}
I1129 19:52:20.296387       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-13e3fb59-9d0c-423b-860a-9c44c7372bba on /var/lib/kubelet/pods/2affec37-b1eb-407c-be28-d395490a8ae5/volumes/kubernetes.io~csi/pvc-13e3fb59-9d0c-423b-860a-9c44c7372bba/mount
I1129 19:52:20.296412       1 mount_helper_common.go:99] "/var/lib/kubelet/pods/2affec37-b1eb-407c-be28-d395490a8ae5/volumes/kubernetes.io~csi/pvc-13e3fb59-9d0c-423b-860a-9c44c7372bba/mount" is a mountpoint, unmounting
... skipping 204 lines ...
I1129 19:53:10.958683       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:53:10.958709       1 utils.go:79] GRPC request: {}
I1129 19:53:10.958756       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:53:25.589488       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:53:25.589536       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:53:25.589553       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:53:25.589643       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:53:25.589667       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:53:25.688935       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:53:34.043148       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:53:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="81afc677-701f-11ed-ba40-000d3a2987eb"
I1129 19:53:34.057645       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 14 milliseconds
I1129 19:53:34.058214       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="81afc677-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15090546
I1129 19:53:35.318145       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:53:35.318225       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="6c6e4be2-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8ff03583-7255-4979-b8ed-19cf2aad4ef0" "latency"=36936288128
I1129 19:53:35.318248       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6c6e4be2-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=36936416231
I1129 19:53:35.318357       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:53:35.318417       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="6c6e5384-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8f45960f-0226-4b72-a1f6-dad302128dd2" "latency"=36936313629
I1129 19:53:35.318475       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6c6e5384-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=36936419231
I1129 19:53:35.657596       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:53:35.657629       1 utils.go:79] GRPC request: {}
I1129 19:53:35.657681       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:53:35.661194       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 122 lines ...
W1129 19:53:44.827289       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8f45960f-0226-4b72-a1f6-dad302128dd2/globalmount" is not a mountpoint, deleting
I1129 19:53:44.827358       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8f45960f-0226-4b72-a1f6-dad302128dd2/globalmount successfully
I1129 19:53:44.827376       1 utils.go:85] GRPC response: {}
I1129 19:53:55.590223       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:53:55.590305       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:53:55.590314       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:53:55.590425       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:53:55.590503       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:53:55.689951       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:54:04.042048       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:54:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="93913e71-701f-11ed-ba40-000d3a2987eb"
I1129 19:54:04.056437       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 14 milliseconds
I1129 19:54:04.056657       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="93913e71-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14637839
I1129 19:54:10.959269       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:54:10.959295       1 utils.go:79] GRPC request: {}
I1129 19:54:10.959347       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:54:21.006118       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:54:21.006196       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="82a6f922-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6334d105-0733-476d-9592-ee734b7e4581" "latency"=45342944283
I1129 19:54:21.006220       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="82a6f922-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=45343075085
I1129 19:54:21.011261       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:54:21.011325       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="83e97de6-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fd1ac11f-2bf8-4663-88cf-7654d93e7d4a" "latency"=43234438517
I1129 19:54:21.011357       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="83e97de6-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=43234553319
I1129 19:54:21.619199       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/
I1129 19:54:21.619258       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1129 19:54:21.619297       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6334d105-0733-476d-9592-ee734b7e4581/globalmount with mount options([nouuid])
I1129 19:54:21.619313       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 117 lines ...
I1129 19:54:44.459384       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 19:54:44.459400       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8f45960f-0226-4b72-a1f6-dad302128dd2/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8f45960f-0226-4b72-a1f6-dad302128dd2","csi.storage.k8s.io/pvc/name":"test.csi.azure.comm87cf","csi.storage.k8s.io/pvc/namespace":"multivolume-2375","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-8f45960f-0226-4b72-a1f6-dad302128dd2"}
I1129 19:54:44.459543       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-8f45960f-0226-4b72-a1f6-dad302128dd2-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 19:54:55.594284       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:54:55.594401       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:54:55.594432       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:54:55.594597       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:54:55.690854       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:55:04.039650       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:55:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="b754267a-701f-11ed-ba40-000d3a2987eb"
I1129 19:55:04.050429       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 10 milliseconds
I1129 19:55:04.050685       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="b754267a-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11067281
I1129 19:55:10.958125       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:55:10.958147       1 utils.go:79] GRPC request: {}
I1129 19:55:10.958185       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:55:25.600513       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:55:25.600603       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:55:25.600633       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:55:25.600716       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:55:25.691996       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:55:34.039298       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:55:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="c935bb9d-701f-11ed-ba40-000d3a2987eb"
I1129 19:55:34.049268       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 9 milliseconds
I1129 19:55:34.049452       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="c935bb9d-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10181167
I1129 19:55:40.958604       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:55:40.958633       1 utils.go:79] GRPC request: {}
I1129 19:55:40.958682       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:55:42.644374       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:55:42.644447       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="aba877e3-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8f45960f-0226-4b72-a1f6-dad302128dd2" "latency"=58184844886
I1129 19:55:42.644479       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="aba877e3-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=58184942486
I1129 19:55:44.054847       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:55:44.054868       1 utils.go:79] GRPC request: {}
I1129 19:55:44.054906       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:55:44.063768       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 34 lines ...
I1129 19:55:44.397260       1 nodeserver_v2.go:348] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8f45960f-0226-4b72-a1f6-dad302128dd2/globalmount at /var/lib/kubelet/pods/df3947c1-1e2c-431c-9206-875e7b38f55b/volumes/kubernetes.io~csi/pvc-8f45960f-0226-4b72-a1f6-dad302128dd2/mount
I1129 19:55:44.397277       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8f45960f-0226-4b72-a1f6-dad302128dd2/globalmount /var/lib/kubelet/pods/df3947c1-1e2c-431c-9206-875e7b38f55b/volumes/kubernetes.io~csi/pvc-8f45960f-0226-4b72-a1f6-dad302128dd2/mount)
I1129 19:55:44.398170       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8f45960f-0226-4b72-a1f6-dad302128dd2/globalmount /var/lib/kubelet/pods/df3947c1-1e2c-431c-9206-875e7b38f55b/volumes/kubernetes.io~csi/pvc-8f45960f-0226-4b72-a1f6-dad302128dd2/mount)
I1129 19:55:44.398963       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8f45960f-0226-4b72-a1f6-dad302128dd2/globalmount at /var/lib/kubelet/pods/df3947c1-1e2c-431c-9206-875e7b38f55b/volumes/kubernetes.io~csi/pvc-8f45960f-0226-4b72-a1f6-dad302128dd2/mount successfully
I1129 19:55:44.398980       1 utils.go:85] GRPC response: {}
I1129 19:55:55.603540       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:55:55.603686       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:55:55.603723       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:55:55.603752       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:55:55.693144       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:56:04.038570       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:56:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="db1741e4-701f-11ed-ba40-000d3a2987eb"
I1129 19:56:04.045197       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 6 milliseconds
I1129 19:56:04.045405       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="db1741e4-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6876412
I1129 19:56:10.959066       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:56:10.959094       1 utils.go:79] GRPC request: {}
I1129 19:56:10.959140       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:56:25.604297       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:56:25.604331       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:56:25.604314       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:56:25.604435       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:56:25.693656       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:56:28.182987       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:56:28.183065       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="cf2f9ff1-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a08a5e5d-3aa9-4dcb-b4f0-3943ba90aead" "latency"=44117410403
I1129 19:56:28.183096       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cf2f9ff1-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=44117544706
I1129 19:56:28.775897       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1129 19:56:28.775945       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 19:56:28.775987       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a08a5e5d-3aa9-4dcb-b4f0-3943ba90aead/globalmount with mount options([nouuid])
I1129 19:56:28.776002       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 73 lines ...
I1129 19:56:34.045327       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="ecf900fc-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6086399
I1129 19:56:40.958968       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:56:40.958990       1 utils.go:79] GRPC request: {}
I1129 19:56:40.959029       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:56:46.545099       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 142 items received
I1129 19:56:46.553073       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=18940&timeout=9m24s&timeoutSeconds=564&watch=true 200 OK in 7 milliseconds
I1129 19:56:53.459413       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:56:53.459493       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="ea075c56-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ffcbe004-7eb3-43ec-8fdb-e52545c9d885" "latency"=24359261998
I1129 19:56:53.459527       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ea075c56-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24359365200
I1129 19:56:53.463333       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:56:53.463404       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="e9ec1306-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3c498753-1f1c-4b6f-9c1a-3c7d4ee06161" "latency"=24541978187
I1129 19:56:53.463443       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e9ec1306-701f-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24542104789
I1129 19:56:54.040933       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1129 19:56:54.040933       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/
I1129 19:56:54.040989       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1129 19:56:54.041006       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
... skipping 191 lines ...
I1129 19:58:10.958923       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:58:10.958947       1 utils.go:79] GRPC request: {}
I1129 19:58:10.959005       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:58:25.606033       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:58:25.606047       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:58:25.606058       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:58:25.606185       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:58:25.695369       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:58:34.043382       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:58:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="34802d45-7020-11ed-ba40-000d3a2987eb"
I1129 19:58:34.056478       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 12 milliseconds
I1129 19:58:34.057187       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="34802d45-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13833827
I1129 19:58:39.246281       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:58:39.246356       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="1fb15f9f-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-797bbddf-a9a5-4d3e-ac3d-e5180fc417f8" "latency"=40112655165
I1129 19:58:39.246378       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1fb15f9f-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=40112783067
I1129 19:58:40.020090       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 19:58:40.020116       1 utils.go:79] GRPC request: {}
I1129 19:58:40.020162       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 19:58:40.023315       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 52 lines ...
I1129 19:58:44.256683       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"readonly":true,"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-797bbddf-a9a5-4d3e-ac3d-e5180fc417f8/globalmount","target_path":"/var/lib/kubelet/pods/caa2d687-d17a-4b02-9847-b06195984dbf/volumes/kubernetes.io~csi/pvc-797bbddf-a9a5-4d3e-ac3d-e5180fc417f8/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-797bbddf-a9a5-4d3e-ac3d-e5180fc417f8","csi.storage.k8s.io/pvc/name":"test.csi.azure.comdns7w","csi.storage.k8s.io/pvc/namespace":"multivolume-8564","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-797bbddf-a9a5-4d3e-ac3d-e5180fc417f8"}
I1129 19:58:44.256882       1 nodeserver_v2.go:348] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-797bbddf-a9a5-4d3e-ac3d-e5180fc417f8/globalmount at /var/lib/kubelet/pods/caa2d687-d17a-4b02-9847-b06195984dbf/volumes/kubernetes.io~csi/pvc-797bbddf-a9a5-4d3e-ac3d-e5180fc417f8/mount
I1129 19:58:44.256904       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-797bbddf-a9a5-4d3e-ac3d-e5180fc417f8/globalmount /var/lib/kubelet/pods/caa2d687-d17a-4b02-9847-b06195984dbf/volumes/kubernetes.io~csi/pvc-797bbddf-a9a5-4d3e-ac3d-e5180fc417f8/mount)
I1129 19:58:44.257943       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount,ro /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-797bbddf-a9a5-4d3e-ac3d-e5180fc417f8/globalmount /var/lib/kubelet/pods/caa2d687-d17a-4b02-9847-b06195984dbf/volumes/kubernetes.io~csi/pvc-797bbddf-a9a5-4d3e-ac3d-e5180fc417f8/mount)
I1129 19:58:44.258889       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-797bbddf-a9a5-4d3e-ac3d-e5180fc417f8/globalmount at /var/lib/kubelet/pods/caa2d687-d17a-4b02-9847-b06195984dbf/volumes/kubernetes.io~csi/pvc-797bbddf-a9a5-4d3e-ac3d-e5180fc417f8/mount successfully
I1129 19:58:44.258908       1 utils.go:85] GRPC response: {}
I1129 19:58:49.533468       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 19:58:49.533547       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="3810e762-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3dc636c1-3795-42ef-8d7e-bd7aceff7d82" "latency"=9508473627
I1129 19:58:49.533575       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3810e762-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9508593529
I1129 19:58:50.113999       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1129 19:58:50.114049       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 19:58:50.114080       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3dc636c1-3795-42ef-8d7e-bd7aceff7d82/globalmount with mount options([])
I1129 19:58:50.114090       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 95 lines ...
I1129 19:59:16.780604       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7d385a3c-b43b-47ba-b907-2027924895b5-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 19:59:18.550765       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 71 items received
I1129 19:59:18.556638       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=20224&timeout=9m19s&timeoutSeconds=559&watch=true 200 OK in 5 milliseconds
I1129 19:59:25.609709       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:59:25.609796       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:59:25.609810       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:59:25.609889       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:59:25.696192       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:59:34.042261       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T19:59:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="58434721-7020-11ed-ba40-000d3a2987eb"
I1129 19:59:34.050862       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 8 milliseconds
I1129 19:59:34.051095       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="58434721-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8867547
I1129 19:59:40.958640       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 19:59:40.958666       1 utils.go:79] GRPC request: {}
I1129 19:59:40.958718       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 19:59:55.616123       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:59:55.616159       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:59:55.616226       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 19:59:55.616262       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 19:59:55.696639       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:00:04.041852       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T20:00:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="6a24d9fd-7020-11ed-ba40-000d3a2987eb"
I1129 20:00:04.053633       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 11 milliseconds
I1129 20:00:04.053840       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="6a24d9fd-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12039797
I1129 20:00:10.958359       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 20:00:10.958384       1 utils.go:79] GRPC request: {}
I1129 20:00:10.958429       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:00:13.739188       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:00:13.739275       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="4df95ca9-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7d385a3c-b43b-47ba-b907-2027924895b5" "latency"=56958571910
I1129 20:00:13.739311       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4df95ca9-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=56958732513
I1129 20:00:14.694375       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 20:00:14.694398       1 utils.go:79] GRPC request: {}
I1129 20:00:14.694450       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 20:00:14.698094       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 58 lines ...
W1129 20:00:19.212100       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7d385a3c-b43b-47ba-b907-2027924895b5/globalmount" is not a mountpoint, deleting
I1129 20:00:19.212193       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7d385a3c-b43b-47ba-b907-2027924895b5/globalmount successfully
I1129 20:00:19.212211       1 utils.go:85] GRPC response: {}
I1129 20:00:25.616646       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:00:25.616742       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:00:25.616768       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:00:25.616838       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:00:25.697033       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:00:34.038568       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T20:00:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="7c05fdb9-7020-11ed-ba40-000d3a2987eb"
I1129 20:00:34.049351       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 10 milliseconds
I1129 20:00:34.049512       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="7c05fdb9-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10962378
I1129 20:00:34.242212       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:00:34.242282       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="707f1e07-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-97b66164-fcc0-4de5-8301-1aeb32fe176b" "latency"=19542522531
I1129 20:00:34.242310       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="707f1e07-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19542620533
I1129 20:00:34.839695       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1129 20:00:34.839749       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 20:00:34.839777       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-97b66164-fcc0-4de5-8301-1aeb32fe176b/globalmount with mount options([])
I1129 20:00:34.839786       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 62 lines ...
I1129 20:00:40.958968       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:00:43.469677       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 106 items received
I1129 20:00:43.471250       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=20769&timeout=5m31s&timeoutSeconds=331&watch=true 200 OK in 1 milliseconds
I1129 20:00:55.621762       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:00:55.621868       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:00:55.621878       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:00:55.621986       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:00:55.698156       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:01:04.043320       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T20:01:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="8de859d2-7020-11ed-ba40-000d3a2987eb"
I1129 20:01:04.056518       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 13 milliseconds
I1129 20:01:04.056745       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="8de859d2-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13454016
I1129 20:01:10.959084       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 20:01:10.959111       1 utils.go:79] GRPC request: {}
I1129 20:01:10.959153       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:01:15.926681       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:01:15.926756       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="7d18a4fb-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6d78997a-307c-4276-b712-c4fa0737cf19" "latency"=40088154907
I1129 20:01:15.926782       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7d18a4fb-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=40088282308
I1129 20:01:17.223653       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 20:01:17.223673       1 utils.go:79] GRPC request: {}
I1129 20:01:17.223714       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 20:01:17.233218       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 55 lines ...
I1129 20:01:24.771656       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6d78997a-307c-4276-b712-c4fa0737cf19/globalmount
W1129 20:01:24.783893       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6d78997a-307c-4276-b712-c4fa0737cf19/globalmount" is not a mountpoint, deleting
I1129 20:01:24.783968       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6d78997a-307c-4276-b712-c4fa0737cf19/globalmount successfully
I1129 20:01:24.783979       1 utils.go:85] GRPC response: {}
I1129 20:01:25.622237       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:01:25.622292       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:01:25.622339       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:01:25.622402       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:01:25.698627       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:01:31.374145       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:01:31.374232       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="95c53a62-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f01001be-9350-49eb-b01c-fa6fd09ca524" "latency"=14139284116
I1129 20:01:31.374269       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="95c53a62-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14139397918
I1129 20:01:31.960124       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1129 20:01:31.960185       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 20:01:31.960223       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f01001be-9350-49eb-b01c-fa6fd09ca524/globalmount with mount options([])
I1129 20:01:31.960242       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 40 lines ...
I1129 20:01:50.564362       1 utils.go:79] GRPC request: {}
I1129 20:01:50.564400       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 20:01:50.565230       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:01:50.565252       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5516e0aa-0e54-4edf-bd80-bce63d3f58fa/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5516e0aa-0e54-4edf-bd80-bce63d3f58fa","csi.storage.k8s.io/pvc/name":"test.csi.azure.comkkq42","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-8904","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-5516e0aa-0e54-4edf-bd80-bce63d3f58fa"}
I1129 20:01:50.565418       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-5516e0aa-0e54-4edf-bd80-bce63d3f58fa-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 20:01:55.623394       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:01:55.623509       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:01:55.623523       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:01:55.623528       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:01:55.698753       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:02:04.039135       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T20:02:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="b1aafc1d-7020-11ed-ba40-000d3a2987eb"
I1129 20:02:04.045901       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 6 milliseconds
I1129 20:02:04.046471       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="b1aafc1d-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7357320
... skipping 16 lines ...
W1129 20:02:06.069641       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f01001be-9350-49eb-b01c-fa6fd09ca524/globalmount" is not a mountpoint, deleting
I1129 20:02:06.069716       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f01001be-9350-49eb-b01c-fa6fd09ca524/globalmount successfully
I1129 20:02:06.069732       1 utils.go:85] GRPC response: {}
I1129 20:02:10.958523       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 20:02:10.958541       1 utils.go:79] GRPC request: {}
I1129 20:02:10.958578       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:02:17.138311       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:02:17.138387       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="a9a31031-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5516e0aa-0e54-4edf-bd80-bce63d3f58fa" "latency"=26572883800
I1129 20:02:17.138452       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a9a31031-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=26573021902
I1129 20:02:17.720322       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/
I1129 20:02:17.720373       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1129 20:02:17.720413       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5516e0aa-0e54-4edf-bd80-bce63d3f58fa/globalmount with mount options([])
I1129 20:02:17.720435       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 45 lines ...
I1129 20:02:21.278608       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 20:02:21.279300       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:02:21.279316       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"3"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-610ed1b4-95a0-49e1-80ee-d1c1f0c29148/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-610ed1b4-95a0-49e1-80ee-d1c1f0c29148","csi.storage.k8s.io/pvc/name":"test.csi.azure.comvdn2h","csi.storage.k8s.io/pvc/namespace":"multivolume-7394","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-610ed1b4-95a0-49e1-80ee-d1c1f0c29148"}
I1129 20:02:21.279486       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-610ed1b4-95a0-49e1-80ee-d1c1f0c29148-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 20:02:25.625367       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:02:25.625455       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:02:25.625509       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:02:25.625523       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:02:25.625581       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:02:25.699777       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:02:29.656523       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1129 20:02:29.656547       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/d32365d0-42bf-4508-a274-86881435abd9/volumes/kubernetes.io~csi/pvc-5516e0aa-0e54-4edf-bd80-bce63d3f58fa/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-5516e0aa-0e54-4edf-bd80-bce63d3f58fa"}
I1129 20:02:29.656609       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-5516e0aa-0e54-4edf-bd80-bce63d3f58fa on /var/lib/kubelet/pods/d32365d0-42bf-4508-a274-86881435abd9/volumes/kubernetes.io~csi/pvc-5516e0aa-0e54-4edf-bd80-bce63d3f58fa/mount
I1129 20:02:29.656637       1 mount_helper_common.go:99] "/var/lib/kubelet/pods/d32365d0-42bf-4508-a274-86881435abd9/volumes/kubernetes.io~csi/pvc-5516e0aa-0e54-4edf-bd80-bce63d3f58fa/mount" is a mountpoint, unmounting
I1129 20:02:29.656650       1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/d32365d0-42bf-4508-a274-86881435abd9/volumes/kubernetes.io~csi/pvc-5516e0aa-0e54-4edf-bd80-bce63d3f58fa/mount
... skipping 75 lines ...
W1129 20:02:38.037869       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5516e0aa-0e54-4edf-bd80-bce63d3f58fa/globalmount" is not a mountpoint, deleting
I1129 20:02:38.037928       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5516e0aa-0e54-4edf-bd80-bce63d3f58fa/globalmount successfully
I1129 20:02:38.037937       1 utils.go:85] GRPC response: {}
I1129 20:02:40.958591       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 20:02:40.958619       1 utils.go:79] GRPC request: {}
I1129 20:02:40.958671       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:02:48.306567       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:02:48.306643       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="bbf14645-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-610c60cb-2f9b-4508-ba44-811879c7d024" "latency"=27029532292
I1129 20:02:48.306668       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bbf14645-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=27029721695
I1129 20:02:48.316060       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:02:48.316127       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="bbf1a8d7-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-610ed1b4-95a0-49e1-80ee-d1c1f0c29148" "latency"=27036585502
I1129 20:02:48.316150       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bbf1a8d7-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=27036681803
I1129 20:02:48.886094       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1129 20:02:48.886094       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sde under /dev/disk/azure/scsi1/
I1129 20:02:48.886141       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 20:02:48.886158       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
... skipping 165 lines ...
I1129 20:03:20.961849       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:03:20.961873       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-778cdb97-659b-4673-9c08-092e3feb8856/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-778cdb97-659b-4673-9c08-092e3feb8856","csi.storage.k8s.io/pvc/name":"test.csi.azure.com9zhfw","csi.storage.k8s.io/pvc/namespace":"multivolume-5738","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-778cdb97-659b-4673-9c08-092e3feb8856"}
I1129 20:03:20.962066       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-778cdb97-659b-4673-9c08-092e3feb8856-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 20:03:25.635039       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:03:25.635087       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:03:25.635113       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:03:25.635178       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:03:25.700493       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:03:30.110980       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1129 20:03:30.111003       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/1a36aca3-4ff7-4933-abba-8dd35be01eb4/volumes/kubernetes.io~csi/pvc-610ed1b4-95a0-49e1-80ee-d1c1f0c29148/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-610ed1b4-95a0-49e1-80ee-d1c1f0c29148"}
I1129 20:03:30.111067       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-610ed1b4-95a0-49e1-80ee-d1c1f0c29148 on /var/lib/kubelet/pods/1a36aca3-4ff7-4933-abba-8dd35be01eb4/volumes/kubernetes.io~csi/pvc-610ed1b4-95a0-49e1-80ee-d1c1f0c29148/mount
I1129 20:03:30.111108       1 mount_helper_common.go:99] "/var/lib/kubelet/pods/1a36aca3-4ff7-4933-abba-8dd35be01eb4/volumes/kubernetes.io~csi/pvc-610ed1b4-95a0-49e1-80ee-d1c1f0c29148/mount" is a mountpoint, unmounting
I1129 20:03:30.111124       1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/1a36aca3-4ff7-4933-abba-8dd35be01eb4/volumes/kubernetes.io~csi/pvc-610ed1b4-95a0-49e1-80ee-d1c1f0c29148/mount
... skipping 31 lines ...
I1129 20:03:34.039757       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T20:03:34Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="e74ffd04-7020-11ed-ba40-000d3a2987eb"
I1129 20:03:34.054309       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 14 milliseconds
I1129 20:03:34.055903       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="e74ffd04-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=16187669
I1129 20:03:40.958762       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 20:03:40.958791       1 utils.go:79] GRPC request: {}
I1129 20:03:40.958844       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:03:45.869770       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:03:45.869845       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="df847f01-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-778cdb97-659b-4673-9c08-092e3feb8856" "latency"=24907689098
I1129 20:03:45.869870       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="df847f01-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24907836201
I1129 20:03:46.440570       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1129 20:03:46.440621       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 20:03:46.440651       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-778cdb97-659b-4673-9c08-092e3feb8856/globalmount with mount options([])
I1129 20:03:46.440660       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 34 lines ...
I1129 20:03:52.882575       1 utils.go:79] GRPC request: {}
I1129 20:03:52.882605       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 20:03:52.883297       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:03:52.883311       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2f37d517-7caf-4c06-9064-11b83f93c22a/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2f37d517-7caf-4c06-9064-11b83f93c22a","csi.storage.k8s.io/pvc/name":"test.csi.azure.com9zhfw-cloned","csi.storage.k8s.io/pvc/namespace":"multivolume-5738","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-2f37d517-7caf-4c06-9064-11b83f93c22a"}
I1129 20:03:52.883479       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-2f37d517-7caf-4c06-9064-11b83f93c22a-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 20:03:55.635862       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:03:55.635955       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:03:55.635986       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:03:55.636013       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:03:55.701299       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:04:04.038521       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T20:04:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="f9316e90-7020-11ed-ba40-000d3a2987eb"
I1129 20:04:04.046594       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 7 milliseconds
I1129 20:04:04.047855       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="f9316e90-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9416558
I1129 20:04:10.958145       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 20:04:10.958170       1 utils.go:79] GRPC request: {}
I1129 20:04:10.958220       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:04:21.507992       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:04:21.508106       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="f28b51e1-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2f37d517-7caf-4c06-9064-11b83f93c22a" "latency"=28624540898
I1129 20:04:21.508148       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f28b51e1-7020-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=28624675801
I1129 20:04:22.120825       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/
I1129 20:04:22.120875       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1129 20:04:22.120902       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2f37d517-7caf-4c06-9064-11b83f93c22a/globalmount with mount options([])
I1129 20:04:22.120911       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 100 lines ...
I1129 20:05:04.590881       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:05:04.590900       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-255ae2b0-008f-478d-b21b-07e7b06d6768/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-255ae2b0-008f-478d-b21b-07e7b06d6768","csi.storage.k8s.io/pvc/name":"pvc-cl6sm","csi.storage.k8s.io/pvc/namespace":"provisioning-39","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-255ae2b0-008f-478d-b21b-07e7b06d6768"}
I1129 20:05:04.591078       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-255ae2b0-008f-478d-b21b-07e7b06d6768-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 20:05:10.959179       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 20:05:10.959207       1 utils.go:79] GRPC request: {}
I1129 20:05:10.959258       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:05:11.785958       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:05:11.786032       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="1d490791-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-255ae2b0-008f-478d-b21b-07e7b06d6768" "latency"=7194889808
I1129 20:05:11.786057       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1d490791-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7195001310
I1129 20:05:11.792717       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:05:11.792796       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="1d489ac1-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8515b15c-0dd4-4898-ab17-73c30affdbb2" "latency"=7204404465
I1129 20:05:11.792838       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1d489ac1-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7204562068
I1129 20:05:12.394584       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1129 20:05:12.394641       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 20:05:12.394674       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-255ae2b0-008f-478d-b21b-07e7b06d6768/globalmount with mount options([])
I1129 20:05:12.394687       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 71 lines ...
W1129 20:05:25.043828       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-255ae2b0-008f-478d-b21b-07e7b06d6768/globalmount" is not a mountpoint, deleting
I1129 20:05:25.043886       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-255ae2b0-008f-478d-b21b-07e7b06d6768/globalmount successfully
I1129 20:05:25.043899       1 utils.go:85] GRPC response: {}
I1129 20:05:25.644987       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:05:25.645011       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:05:25.645071       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:05:25.645093       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:05:25.703487       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:05:31.691431       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:05:31.691494       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="2979599c-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1ff0229a-1618-4616-826c-29be8b45d178" "latency"=6651036383
I1129 20:05:31.691523       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2979599c-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6651131285
I1129 20:05:32.299568       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1129 20:05:32.299619       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1129 20:05:32.299660       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1ff0229a-1618-4616-826c-29be8b45d178/globalmount with mount options([])
I1129 20:05:32.299672       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 77 lines ...
I1129 20:06:12.693559       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 18 items received
I1129 20:06:12.695037       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-16172903-vmss000000&resourceVersion=23465&timeout=7m53s&timeoutSeconds=473&watch=true 200 OK in 1 milliseconds
I1129 20:06:14.471990       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 66 items received
I1129 20:06:14.476240       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=23501&timeout=5m48s&timeoutSeconds=348&watch=true 200 OK in 4 milliseconds
I1129 20:06:25.647321       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:06:25.647430       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:06:25.647494       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:06:25.647546       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:06:25.704928       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:06:26.565711       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 149 items received
I1129 20:06:26.568170       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=23677&timeout=6m16s&timeoutSeconds=376&watch=true 200 OK in 2 milliseconds
I1129 20:06:27.546026       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:06:27.546102       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="42241fad-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0baa4f2f-4516-402d-9114-e8387873fbd6" "latency"=21121114697
I1129 20:06:27.546129       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="42241fad-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=21121239800
I1129 20:06:28.137608       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1129 20:06:28.137660       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 20:06:28.137689       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0baa4f2f-4516-402d-9114-e8387873fbd6/globalmount with mount options([])
I1129 20:06:28.137698       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 145 lines ...
I1129 20:09:10.455815       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:09:10.455830       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-00635490-cad3-47ef-9ff5-1a5f5b3bc1b2/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-00635490-cad3-47ef-9ff5-1a5f5b3bc1b2","csi.storage.k8s.io/pvc/name":"test.csi.azure.comd5cjv","csi.storage.k8s.io/pvc/namespace":"multivolume-6442","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-00635490-cad3-47ef-9ff5-1a5f5b3bc1b2"}
I1129 20:09:10.456000       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-00635490-cad3-47ef-9ff5-1a5f5b3bc1b2-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 20:09:10.958997       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 20:09:10.959018       1 utils.go:79] GRPC request: {}
I1129 20:09:10.959056       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:09:14.533017       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:09:14.533089       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="afd4d565-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1949ad05-547e-4e04-a575-e1d0ef0db1b3" "latency"=4078350638
I1129 20:09:14.533121       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="afd4d565-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4078488741
I1129 20:09:14.533225       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:09:14.533293       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="afd50a0a-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-00635490-cad3-47ef-9ff5-1a5f5b3bc1b2" "latency"=4077225520
I1129 20:09:14.533332       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="afd50a0a-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4077355222
I1129 20:09:15.129866       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1129 20:09:15.129925       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 20:09:15.129967       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1949ad05-547e-4e04-a575-e1d0ef0db1b3/globalmount with mount options([])
I1129 20:09:15.129993       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 256 lines ...
I1129 20:10:42.651838       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 20:10:42.651857       1 utils.go:79] GRPC request: {}
I1129 20:10:42.651891       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 20:10:42.652690       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:10:42.652711       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297","csi.storage.k8s.io/pvc/name":"test.csi.azure.com6pdbh","csi.storage.k8s.io/pvc/namespace":"snapshotting-5256","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297"}
I1129 20:10:42.652890       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 20:10:50.717294       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:10:50.717369       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="e6c92b28-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297" "latency"=8064384726
I1129 20:10:50.717406       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e6c92b28-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8064533628
I1129 20:10:52.593387       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1129 20:10:52.593454       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 20:10:52.593493       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297/globalmount with mount options([])
I1129 20:10:52.593508       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 49 lines ...
I1129 20:10:53.110330       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297/globalmount /var/lib/kubelet/pods/f37c23ba-1647-4842-8dd3-ae5af2fb418b/volumes/kubernetes.io~csi/pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297/mount)
I1129 20:10:53.111161       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297/globalmount at /var/lib/kubelet/pods/f37c23ba-1647-4842-8dd3-ae5af2fb418b/volumes/kubernetes.io~csi/pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297/mount successfully
I1129 20:10:53.111178       1 utils.go:85] GRPC response: {}
I1129 20:10:55.666583       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:10:55.666615       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:10:55.666614       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:10:55.666727       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:10:55.666757       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:10:55.708948       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:10:56.136385       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1129 20:10:56.136408       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/f37c23ba-1647-4842-8dd3-ae5af2fb418b/volumes/kubernetes.io~csi/pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297"}
I1129 20:10:56.136466       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297 on /var/lib/kubelet/pods/f37c23ba-1647-4842-8dd3-ae5af2fb418b/volumes/kubernetes.io~csi/pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297/mount
I1129 20:10:56.136504       1 mount_helper_common.go:99] "/var/lib/kubelet/pods/f37c23ba-1647-4842-8dd3-ae5af2fb418b/volumes/kubernetes.io~csi/pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297/mount" is a mountpoint, unmounting
I1129 20:10:56.136517       1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/f37c23ba-1647-4842-8dd3-ae5af2fb418b/volumes/kubernetes.io~csi/pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297/mount
... skipping 14 lines ...
I1129 20:11:04.038747       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T20:11:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="f3886249-7021-11ed-ba40-000d3a2987eb"
I1129 20:11:04.046751       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 7 milliseconds
I1129 20:11:04.046991       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="f3886249-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8306135
I1129 20:11:10.958804       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 20:11:10.958829       1 utils.go:79] GRPC request: {}
I1129 20:11:10.958876       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:11:21.021801       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:11:21.021867       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="ece8fb19-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3c9fc891-f2b7-4b73-8f2c-0f75a1b9b206" "latency"=28094096977
I1129 20:11:21.021893       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ece8fb19-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=28094211479
I1129 20:11:21.024696       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:11:21.024766       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="ece90b5d-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-575c20ef-64f1-46aa-af5b-063c255ae824" "latency"=28096579718
I1129 20:11:21.024796       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ece90b5d-7021-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=28096692919
I1129 20:11:21.620240       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1129 20:11:21.620293       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1129 20:11:21.620331       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-575c20ef-64f1-46aa-af5b-063c255ae824/globalmount with mount options([nouuid])
I1129 20:11:21.620347       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 201 lines ...
I1129 20:11:44.330726       1 utils.go:85] GRPC response: {}
I1129 20:11:44.330863       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-575c20ef-64f1-46aa-af5b-063c255ae824/globalmount at /var/lib/kubelet/pods/921f1503-a74a-432b-9229-c1497b2cac43/volumes/kubernetes.io~csi/pvc-575c20ef-64f1-46aa-af5b-063c255ae824/mount successfully
I1129 20:11:44.330880       1 utils.go:85] GRPC response: {}
I1129 20:11:55.667630       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:11:55.667674       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:11:55.667653       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:11:55.667768       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:11:55.710134       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:11:59.852715       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:11:59.852821       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="0911f4cf-7022-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-04c0d3be-2c4b-48f7-b65b-0cc41c5ba297" "latency"=19680296565
I1129 20:11:59.852854       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0911f4cf-7022-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19680431067
I1129 20:12:00.099323       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1129 20:12:00.099380       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1129 20:12:00.099366       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/921f1503-a74a-432b-9229-c1497b2cac43/volumes/kubernetes.io~csi/pvc-3c9fc891-f2b7-4b73-8f2c-0f75a1b9b206/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-3c9fc891-f2b7-4b73-8f2c-0f75a1b9b206"}
I1129 20:12:00.099416       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-3c9fc891-f2b7-4b73-8f2c-0f75a1b9b206 on /var/lib/kubelet/pods/921f1503-a74a-432b-9229-c1497b2cac43/volumes/kubernetes.io~csi/pvc-3c9fc891-f2b7-4b73-8f2c-0f75a1b9b206/mount
... skipping 112 lines ...
I1129 20:12:40.958367       1 utils.go:79] GRPC request: {}
I1129 20:12:40.958417       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:12:42.568536       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 127 items received
I1129 20:12:42.579086       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=26453&timeout=8m6s&timeoutSeconds=486&watch=true 200 OK in 10 milliseconds
I1129 20:12:55.668430       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:12:55.668492       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:12:55.668527       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:12:55.668638       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:12:55.711905       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:12:56.650910       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:12:56.651007       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="298871a1-7022-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-75f03d50-438f-4eea-b4fd-3d86da4cd102" "latency"=22014864779
I1129 20:12:56.651044       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="298871a1-7022-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=22015005682
I1129 20:12:58.419465       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1129 20:12:58.419517       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 20:12:58.419555       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-75f03d50-438f-4eea-b4fd-3d86da4cd102/globalmount with mount options([])
I1129 20:12:58.419574       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 56 lines ...
I1129 20:13:05.353755       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:13:05.353771       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-ae1a3db7-b443-4228-8c63-cd48d623f9d2","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ae1a3db7-b443-4228-8c63-cd48d623f9d2","csi.storage.k8s.io/pvc/name":"test.csi.azure.comspmpd","csi.storage.k8s.io/pvc/namespace":"volume-993","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-ae1a3db7-b443-4228-8c63-cd48d623f9d2"}
I1129 20:13:05.353965       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-ae1a3db7-b443-4228-8c63-cd48d623f9d2-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 20:13:10.958380       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 20:13:10.958405       1 utils.go:79] GRPC request: {}
I1129 20:13:10.958476       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:13:11.093154       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:13:11.093224       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="3bd79fc4-7022-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ae1a3db7-b443-4228-8c63-cd48d623f9d2" "latency"=5739197283
I1129 20:13:11.093247       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3bd79fc4-7022-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5739308984
I1129 20:13:11.683404       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1129 20:13:11.683455       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 20:13:11.683469       1 utils.go:85] GRPC response: {}
I1129 20:13:11.694689       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 110 lines ...
I1129 20:14:34.051314       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="70b48bf8-7022-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7995631
I1129 20:14:40.958580       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1129 20:14:40.958603       1 utils.go:79] GRPC request: {}
I1129 20:14:40.958652       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1129 20:14:44.556910       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 131 items received
I1129 20:14:44.560595       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=27369&timeout=7m58s&timeoutSeconds=478&watch=true 200 OK in 3 milliseconds
I1129 20:14:52.494209       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:14:52.494296       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="6ca3bf97-7022-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-952c678d-2a67-4952-9dcf-77606f3ef71c" "latency"=25271863021
I1129 20:14:52.494333       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6ca3bf97-7022-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=25271999023
I1129 20:14:54.270002       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1129 20:14:54.270065       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1129 20:14:54.270107       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-952c678d-2a67-4952-9dcf-77606f3ef71c/globalmount with mount options([])
I1129 20:14:54.270126       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 124 lines ...
I1129 20:15:15.752685       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-952c678d-2a67-4952-9dcf-77606f3ef71c/globalmount
I1129 20:15:15.752713       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-952c678d-2a67-4952-9dcf-77606f3ef71c/globalmount" is a mountpoint, unmounting
I1129 20:15:15.752721       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-952c678d-2a67-4952-9dcf-77606f3ef71c/globalmount
W1129 20:15:15.761721       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-952c678d-2a67-4952-9dcf-77606f3ef71c/globalmount" is not a mountpoint, deleting
I1129 20:15:15.761788       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-952c678d-2a67-4952-9dcf-77606f3ef71c/globalmount successfully
I1129 20:15:15.761803       1 utils.go:85] GRPC response: {}
I1129 20:15:18.303053       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:15:18.303116       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="854f0e2d-7022-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-56c6dccc-e6ca-4c52-b4de-719271262de5" "latency"=9692726447
I1129 20:15:18.303145       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="854f0e2d-7022-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9692820548
I1129 20:15:18.892068       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1129 20:15:18.892258       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1129 20:15:18.892290       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-56c6dccc-e6ca-4c52-b4de-719271262de5/globalmount with mount options([])
I1129 20:15:18.892298       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 104 lines ...
I1129 20:17:41.855989       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 20:17:41.856003       1 utils.go:79] GRPC request: {}
I1129 20:17:41.856029       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 20:17:41.856853       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:17:41.856875       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8606b875-bc8b-47be-968b-f6f467bfdb35/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8606b875-bc8b-47be-968b-f6f467bfdb35","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-8606b875-bc8b-47be-968b-f6f467bfdb35"}
I1129 20:17:41.857054       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-8606b875-bc8b-47be-968b-f6f467bfdb35-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 20:17:48.046002       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:17:48.046095       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="e0a6a5f8-7022-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8606b875-bc8b-47be-968b-f6f467bfdb35" "latency"=6188936086
I1129 20:17:48.046141       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e0a6a5f8-7022-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6189104990
I1129 20:17:49.741744       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1129 20:17:49.741789       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS
I1129 20:17:49.741814       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8606b875-bc8b-47be-968b-f6f467bfdb35/globalmount with mount options([])
I1129 20:17:49.741823       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 65 lines ...
I1129 20:18:53.408949       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:18:53.408964       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-092fc965-2669-4e26-b04e-bbb2ac88eb5d/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-092fc965-2669-4e26-b04e-bbb2ac88eb5d","csi.storage.k8s.io/pvc/name":"nginx-azuredisk-ephemeral-azuredisk01","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-092fc965-2669-4e26-b04e-bbb2ac88eb5d"}
I1129 20:18:53.409146       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-092fc965-2669-4e26-b04e-bbb2ac88eb5d-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 20:18:55.695395       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:18:55.695441       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:18:55.695476       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:18:55.695564       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1129 20:18:55.720707       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1129 20:19:04.043592       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-29T20:19:04Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="11a34fd8-7023-11ed-ba40-000d3a2987eb"
I1129 20:19:04.059178       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-16172903-vmss000000/status 200 OK in 15 milliseconds
I1129 20:19:04.059439       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="11a34fd8-7023-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15890259
I1129 20:19:04.158184       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:19:04.158262       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="0b4ca1af-7023-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-092fc965-2669-4e26-b04e-bbb2ac88eb5d" "latency"=10749055079
I1129 20:19:04.158294       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0b4ca1af-7023-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10749156581
I1129 20:19:04.741700       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1129 20:19:04.741753       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS
I1129 20:19:04.741782       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-092fc965-2669-4e26-b04e-bbb2ac88eb5d/globalmount with mount options([])
I1129 20:19:04.741793       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 43 lines ...
I1129 20:19:14.155049       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1129 20:19:14.155069       1 utils.go:79] GRPC request: {}
I1129 20:19:14.155100       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1129 20:19:14.155734       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1129 20:19:14.155747       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-984aaab8-85db-4943-8d74-7d99d87909ff/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-984aaab8-85db-4943-8d74-7d99d87909ff","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-q6htl-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1669749823560-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-2hnyryib/providers/Microsoft.Compute/disks/pvc-984aaab8-85db-4943-8d74-7d99d87909ff"}
I1129 20:19:14.155916       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-984aaab8-85db-4943-8d74-7d99d87909ff-k8s-agentpool1-16172903-vmss000000-attachment)
I1129 20:19:22.304352       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1129 20:19:22.304412       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-16172903-vmss000000" "disk.csi.azure.com/request-id"="17aa564c-7023-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-984aaab8-85db-4943-8d74-7d99d87909ff" "latency"=8148408182
I1129 20:19:22.304434       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="17aa564c-7023-11ed-ba40-000d3a2987eb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8148537185
I1129 20:19:22.897329       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1129 20:19:22.897378       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType StandardSSD_LRS
I1129 20:19:22.897405       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-984aaab8-85db-4943-8d74-7d99d87909ff/globalmount with mount options([])
I1129 20:19:22.897423       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 469 lines ...
cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-2hnyryib",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="300"} 223
cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-2hnyryib",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="600"} 230
cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-2hnyryib",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="1200"} 230
cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-2hnyryib",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="+Inf"} 230
cloudprovider_azure_op_duration_seconds_sum{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-2hnyryib",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 8848.046354390995
cloudprovider_azure_op_duration_seconds_count{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-2hnyryib",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 230
# HELP cloudprovider_azure_op_failure_count [ALPHA] Number of failed Azure service operations
# TYPE cloudprovider_azure_op_failure_count counter
cloudprovider_azure_op_failure_count{request="azuredisk_csi_driver_attach_volume_latency",resource_group="kubetest-2hnyryib",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 2
cloudprovider_azure_op_failure_count{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-2hnyryib",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 7
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 4.62e-05
... skipping 103 lines ...
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.77183232e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
make: *** [Makefile:341: e2e-test] Error 1
2022/11/29 20:20:54 process.go:155: Step 'make e2e-test' finished in 58m27.702490631s
2022/11/29 20:20:54 aksengine_helpers.go:425: downloading /root/tmp2345232819/log-dump.sh from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh
2022/11/29 20:20:54 util.go:70: curl https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh
2022/11/29 20:20:54 process.go:153: Running: chmod +x /root/tmp2345232819/log-dump.sh
2022/11/29 20:20:54 process.go:155: Step 'chmod +x /root/tmp2345232819/log-dump.sh' finished in 1.190871ms
2022/11/29 20:20:54 aksengine_helpers.go:425: downloading /root/tmp2345232819/log-dump-daemonset.yaml from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump-daemonset.yaml
... skipping 75 lines ...
ssh key file /root/.ssh/id_rsa does not exist. Exiting.
2022/11/29 20:21:54 process.go:155: Step 'bash -c /root/tmp2345232819/win-ci-logs-collector.sh kubetest-2hnyryib.westeurope.cloudapp.azure.com /root/tmp2345232819 /root/.ssh/id_rsa' finished in 4.212256ms
2022/11/29 20:21:54 aksengine.go:1141: Deleting resource group: kubetest-2hnyryib.
2022/11/29 20:29:05 process.go:96: Saved XML output to /logs/artifacts/junit_runner.xml.
2022/11/29 20:29:05 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"
2022/11/29 20:29:07 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 1.821198752s
2022/11/29 20:29:07 main.go:328: Something went wrong: encountered 1 errors: [error during make e2e-test: exit status 2]
+ EXIT_VALUE=1
+ set +o xtrace
Cleaning up after docker in docker.
================================================================================
Cleaning up after docker
Stopping Docker: dockerProgram process in pidfile '/var/run/docker-ssd.pid', 1 process(es), refused to die.
... skipping 3 lines ...